Antoine Lambert
82ab96ad06
gnu: Remove dependency on pytz
...
UTC timezone settings can be obtained from the datetime.timezone
module from Python standard library so remove dependency on external
pytz module.
2021-02-02 13:19:04 +01:00
Antoine Lambert
d1fbccd988
lister: Add utility decorator to ease HTTP requests rate limit handling
...
Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.
The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.
The default wait strategy is based on exponential backoff.
The default max number of attempts is set to 5, HTTPError exception
will then be reraised.
All tenacity.retry parameters can also be overridden in client code.
2021-01-18 11:28:51 +01:00
Léni Gauffier
58ef08b083
Added LaunchpadLister
...
Summary:
Related to T1734
From abandonned D2799
Reviewers: ardumont
Reviewed By: ardumont
Differential Revision: https://forge.softwareheritage.org/D2974
2020-04-12 01:00:12 +02:00
Antoine R. Dumont (@ardumont)
c6372eea7e
gnu.lister: Unify timestamp formats to isoformat date in model
...
Related T2023
2019-11-04 10:08:01 +01:00
Archit Agrawal
b972a2a88d
swh.lister.cgit
...
Implemented a lister to list the repos for a given CGit instance.
Closes T1659
2019-06-28 19:27:25 +05:30
David Douard
c2c26d7e46
Fix the bitbucket lister; handle properly the date-like bounds
2019-02-01 15:38:11 +01:00
Nicolas Dandrimont
2922b68570
Clean up dependencies to enable tests on build
2017-10-30 17:04:49 +01:00
Stefano Zacchiroli
164922afe2
requirements.txt: add missing dependency on "arrow"
2017-09-05 10:54:54 +02:00
Antoine Pietri
c81c7de88c
requirements: remove celery (already required by swh.scheduler)
2017-04-12 15:21:22 +02:00
Avi Kelman (fiendish)
68d77fd43f
Refactor lister code
...
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
2017-03-06 12:35:49 +01:00
Antoine Pietri
ede9e5048c
requirements: split internal and external requirements in two separate files
2017-02-09 14:32:02 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading
...
Related T613
2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d2483e7893
requirements.txt: use proper syntax
2016-10-20 17:27:26 +02:00
Nicolas Dandrimont
9ba8fedc4c
base: add implementation for adding origins
2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
ca4d346451
requirements.txt: Add inter-swh dependencies
2016-10-19 15:41:26 +02:00
Nicolas Dandrimont
2a62db6827
Revert to the pre-qless refactoring version
2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
52f9fd157e
sync packaging metadata
2016-03-17 19:01:10 +01:00
Nicolas Dandrimont
6fbabbe586
req_queue: use qless instead of a handmade queue
2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
c7871e44e8
requirements.txt: add redis
2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
533f6fa1a3
swh.lister.github: Refactor to use swh.storage instead of sqlalchemy
2016-03-09 19:03:35 +01:00
Stefano Zacchiroli
ecca87dccf
requirements.txt: add dependency on requests
2015-09-21 21:13:04 +02:00
Stefano Zacchiroli
376141397d
add requirements.txt, listing sqlalchemy as dependency
2015-09-21 21:11:45 +02:00
