Commit graph

7 commits

Author SHA1 Message Date
Antoine Lambert
9c55acd286 Use generic HTTP retry policy by default and rename dedicated decorator
Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.

Rename throttling_retry decorator to http_retry to reflect this change.
2022-09-26 10:48:40 +02:00
KShivendu
bd35d54398 gogs: Skip pages with error 500
This also affects the gitea lister
2022-09-20 19:05:20 +05:30
Valentin Lorentz
b7ec6cb120 tests: Simplify origin comparison and improve pytest diff on failure
By using a single equality instead of checking len() then zip()
to check one by one, pytest can find the common/missing elements
and print them nicely when the two lists are unequal.
2022-08-24 17:21:24 +02:00
Valentin Lorentz
31c44330e8 gogs: Lower unnecessarily verbose logging statement 2022-08-23 13:40:19 +02:00
Valentin Lorentz
17a219ece0 gitea: Inherit from Gogs lister
This removes code and adds support for incremental pagination.

While both are essentially the same lister now, it still makes sense to
keep the Gitea lister separate, in order to:

1. display them in different categories on https://archive.softwareheritage.org/
2. support possible divergence of APIs in the future
2022-08-23 13:38:32 +02:00
KShivendu
6a53a6ad06 feat: Make the Gogs lister incremental 2022-08-17 15:01:32 +05:30
KShivendu
d34a6232a6 gogs: Introduce Gogs lister 2022-08-03 16:22:06 +05:30