Use generic HTTP retry policy by default and rename dedicated decorator
Instead of retrying HTTP requests only for 429 status code by default, prefer to use the generic retry policy enabling to also retry for status codes >= 500 but also on ConnectionError exceptions. Rename throttling_retry decorator to http_retry to reflect this change.
This commit is contained in:
parent
9b3e565cf7
commit
9c55acd286
23 changed files with 71 additions and 62 deletions
|
@ -11,7 +11,7 @@ from urllib.parse import urljoin
|
|||
import requests
|
||||
from tenacity.before_sleep import before_sleep_log
|
||||
|
||||
from swh.lister.utils import throttling_retry
|
||||
from swh.lister.utils import http_retry
|
||||
from swh.scheduler.interface import SchedulerInterface
|
||||
from swh.scheduler.model import ListedOrigin
|
||||
|
||||
|
@ -77,11 +77,11 @@ class NewForgeLister(Lister[NewForgeListerState, NewForgeListerPage]):
|
|||
def state_to_dict(self, state: NewForgeListerState) -> Dict[str, Any]:
|
||||
return asdict(state)
|
||||
|
||||
@throttling_retry(before_sleep=before_sleep_log(logger, logging.WARNING))
|
||||
@http_retry(before_sleep=before_sleep_log(logger, logging.WARNING))
|
||||
def page_request(self, url, params) -> requests.Response:
|
||||
# Do the network resource request under a retrying decorator
|
||||
# to handle rate limiting and transient errors up to a limit.
|
||||
# `throttling_retry` by default use the `requests` library to check
|
||||
# `http_retry` by default use the `requests` library to check
|
||||
# only for rate-limit and a base-10 exponential waiting strategy.
|
||||
# This can be customized by passed waiting, retrying and logging strategies
|
||||
# as functions. See the `tenacity` library documentation.
|
||||
|
|
|
@ -317,7 +317,7 @@ We generally recommend logging every unhandleable error with the response conten
|
|||
then immediately stop the listing by doing an equivalent of
|
||||
:py:meth:`Response.raise_for_status` from the ``requests`` library. As for rate-limiting
|
||||
errors, we have a strategy of using a flexible decorator to handle the retrying for us.
|
||||
It is based on the ``tenacity`` library and accessible as :py:func:`throttling_retry` from
|
||||
It is based on the ``tenacity`` library and accessible as :py:func:`http_retry` from
|
||||
:py:mod:`swh.lister.utils`.
|
||||
|
||||
Pagination
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue