Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.
Rename throttling_retry decorator to http_retry to reflect this change.
Since this lister is doing a lot more requests than most other, it makes
sense that issues would arise more often. We want the lister to continue
even if the website is having issues and not break on the first 500 or
closed connection it encounters.
This change introduces a mechanism to retry all exceptions worth
retrying and uses it for the SourceForge lister. Other listers might
benefit from this, but this is out of scope here.
Tests had to be adjusted to stub the sleep function since retries happened
way more often.
Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.
The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.
The default wait strategy is based on exponential backoff.
The default max number of attempts is set to 5, HTTPError exception
will then be reraised.
All tenacity.retry parameters can also be overridden in client code.
