reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
816 commits 1 branch 0 tags 16 MiB
Commit graph

23 commits

Author SHA1 Message Date
Nicolas Dandrimont
e785e67315 Hook up recently introduced options to all listers 
Hopefully one day we'll be able to replace all of this mess with PEP692
TypedDict kwargs, but that's only on track for Python 3.12.
 2022-12-05 16:33:45 +01:00
Antoine Lambert
9c55acd286 Use generic HTTP retry policy by default and rename dedicated decorator 
Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.

Rename throttling_retry decorator to http_retry to reflect this change.
 2022-09-26 10:48:40 +02:00
Antoine Lambert
d38e05cff7 python: Reformat code with black 22.3.0 
Related to T3922
 2022-04-08 15:15:09 +02:00
Antoine R. Dumont (@ardumont)
2568ecc7c2
launchpad: Ignore erratic page and continue listing next page 
The decorator is dropped on `get_origins_from_page` as we cannot retry an iterator
consumption anyway.

Related to T3948
 2022-02-18 10:36:54 +01:00
Antoine R. Dumont (@ardumont)
4265e5dd77
launchpad: Drop extra filtering step which is no longer necessary 
as the scheduler is now able to deduplicate it when recording listed origins.

Related to T3945
 2022-02-17 12:11:25 +01:00
Antoine R. Dumont (@ardumont)
c86e4b43f4
launchpad: Use tuple instead of list 
Related to T3945
 2022-02-17 12:11:24 +01:00
Antoine R. Dumont (@ardumont)
fc2edd24aa
launchpad: Manage unhandled exceptions when listing 
Prior to this commit, the listing could fail when either reading a page or the page of
results (lauchpad api raises RestfulError). This now retries when those kind of
exceptions happen. If the error persists (after multiple tryouts and exponential
backoff), the listing continues nonetheless (with warning logs).

Note that if the page ends up being empty, it's no longer accounted for.

This actually allows the listing to finish in case of issues.

Related to T3945
 2022-02-17 12:11:09 +01:00
Antoine R. Dumont (@ardumont)
262f9369c8
launchpad: Allow bzr origins listing 
Related to T3945
 2022-02-16 17:56:13 +01:00
Antoine Lambert
5aa7c8f2b2 launchpad: Remove call to dataclasses.asdict on lister state 
This generates an error due to the datetime type field, so manually build
the dict instead.

Related to T3003#57551
 2021-01-28 19:17:58 +01:00
Antoine Lambert
46f5a50099 launchpad: Prevent error due to origin listed twice 
launchpadlib can list the last modified repository twice so ensure to yield
a single ListedOrigin model for that special case.

Related to T3003#57551
 2021-01-28 19:09:44 +01:00
Antoine Lambert
e8725eb247 launchpad/tasks: Fix ping task function name 
An exception is raised when registering task types in scheduler database otherwise.
 2021-01-28 17:35:40 +01:00
Antoine Lambert
f862004700 launchpad: Reimplement lister using new Lister API 
Port launchpad lister to the swh.lister.pattern.Lister API.

Last update date of each listed git repositories is now sent to the scheduler.

The lister can work in incremental mode, only modified repositories since
the last listing operation will be returned in that case.

Closes T2992
 2021-01-28 15:22:40 +01:00
Antoine R. Dumont (@ardumont)
b11b4d1001
launchpad: Actually mock the anonymous login to launchpad 
This current test was failing in a debian chroot with connection error.
 2021-01-27 16:14:01 +01:00
Antoine R. Dumont (@ardumont)
f09b9f8cfa
lister.launchpad.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b90ffa4bdd
tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b35dff1266
Create listing task with a default of 3 if unspecified 
This will allow to retry task if they do not specify it.
 2020-10-30 09:10:27 +01:00
Antoine Lambert
22f7181294 python: Reorder imports with isort 
Related to T2610
 2020-09-17 17:48:27 +02:00
Antoine R. Dumont (@ardumont)
7d44bc2e75
launchpad.tasks: Update copyright headers 2020-09-08 14:42:37 +02:00
Vincent SELLIER
74ca3d0b87
Launchpad: rename task name to match conventions 
Related to T2358
 2020-09-08 14:21:47 +02:00
Antoine R. Dumont (@ardumont)
9437a643ad
pytest: Define plugin and declare it in the root conftest 
Then drop all unneeded and indirect imports
 2020-09-02 12:25:15 +02:00
Antoine R. Dumont (@ardumont)
250160ad75
launchpad: Add missing copyright headers 2020-09-01 15:53:26 +02:00
Nicolas Dandrimont
c9963d4302 Use the new names for the swh.scheduler test fixtures 2020-07-09 17:06:50 +02:00
Léni Gauffier
58ef08b083 Added LaunchpadLister 
Summary:
Related to T1734

From abandonned D2799

Reviewers: ardumont

Reviewed By: ardumont

Differential Revision: https://forge.softwareheritage.org/D2974
 2020-04-12 01:00:12 +02:00