swh-lister/swh/lister
Antoine Lambert d1fbccd988 lister: Add utility decorator to ease HTTP requests rate limit handling
Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.
2021-01-18 11:28:51 +01:00
..
bitbucket lister.bitbucket.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
cgit lister.cgit.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
core Create listing task with a default of 3 if unspecified 2020-10-30 09:10:27 +01:00
cran lister.cran.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
debian tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
gitea lister.gitea.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
github Reimplement the GitHub lister using the new pattern class 2021-01-11 11:00:29 +01:00
gitlab lister.gitlab.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
gnu lister.gnu.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
launchpad lister.launchpad.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
npm lister.npm.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
packagist python: Reorder imports with isort 2020-09-17 17:48:27 +02:00
phabricator phabricator/tasks: Fix task function return type 2021-01-11 17:59:24 +01:00
pypi python: Reorder imports with isort 2020-09-17 17:48:27 +02:00
tests lister: Add utility decorator to ease HTTP requests rate limit handling 2021-01-18 11:28:51 +01:00
__init__.py Hook up listers implemented with the new pattern to the CLI 2021-01-11 11:00:29 +01:00
cli.py Run isort after the CLI import changes 2020-09-25 14:19:21 +02:00
pattern.py Implement a base pattern for listers with no state storage 2021-01-11 11:00:29 +01:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
pytest_plugin.py tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
utils.py lister: Add utility decorator to ease HTTP requests rate limit handling 2021-01-18 11:28:51 +01:00