Commit graph

21 commits

Author SHA1 Message Date
Antoine Lambert
d1fbccd988 lister: Add utility decorator to ease HTTP requests rate limit handling
Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.
2021-01-18 11:28:51 +01:00
Léni Gauffier
58ef08b083 Added LaunchpadLister
Summary:
Related to T1734

From abandonned D2799

Reviewers: ardumont

Reviewed By: ardumont

Differential Revision: https://forge.softwareheritage.org/D2974
2020-04-12 01:00:12 +02:00
Antoine R. Dumont (@ardumont)
c6372eea7e
gnu.lister: Unify timestamp formats to isoformat date in model
Related T2023
2019-11-04 10:08:01 +01:00
Archit Agrawal
b972a2a88d swh.lister.cgit
Implemented a lister to list the repos for a given CGit instance.

Closes T1659
2019-06-28 19:27:25 +05:30
David Douard
c2c26d7e46 Fix the bitbucket lister; handle properly the date-like bounds 2019-02-01 15:38:11 +01:00
Nicolas Dandrimont
2922b68570 Clean up dependencies to enable tests on build 2017-10-30 17:04:49 +01:00
Stefano Zacchiroli
164922afe2 requirements.txt: add missing dependency on "arrow" 2017-09-05 10:54:54 +02:00
Antoine Pietri
c81c7de88c requirements: remove celery (already required by swh.scheduler) 2017-04-12 15:21:22 +02:00
Avi Kelman (fiendish)
68d77fd43f Refactor lister code
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
2017-03-06 12:35:49 +01:00
Antoine Pietri
ede9e5048c requirements: split internal and external requirements in two separate files 2017-02-09 14:32:02 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading
Related T613
2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d2483e7893 requirements.txt: use proper syntax 2016-10-20 17:27:26 +02:00
Nicolas Dandrimont
9ba8fedc4c base: add implementation for adding origins 2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
ca4d346451 requirements.txt: Add inter-swh dependencies 2016-10-19 15:41:26 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
52f9fd157e sync packaging metadata 2016-03-17 19:01:10 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
c7871e44e8 requirements.txt: add redis 2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
533f6fa1a3 swh.lister.github: Refactor to use swh.storage instead of sqlalchemy 2016-03-09 19:03:35 +01:00
Stefano Zacchiroli
ecca87dccf requirements.txt: add dependency on requests 2015-09-21 21:13:04 +02:00
Stefano Zacchiroli
376141397d add requirements.txt, listing sqlalchemy as dependency 2015-09-21 21:11:45 +02:00