swh-lister/swh/lister/arch
Antoine Lambert 8d85b2e4e8 pattern: Ensure accurate origin counts returned by run method
Previously, the run method was returning the total count of ListedOrigin
objects sent to scheduler database.

However, some listers can send multiple ListedOrigin objects for a given
origin URL during the listing process, for instance when an origin is
contained in multiple pages (e.g. gogs listing) or when the listing
is gathering multiple versions of an origin spread across multiple
pages (e.g. maven listing).

This changes ensures an accurate count of listed origins by maintaining
a set of origin URLs associated to the sent ListedOrigin objects.
2022-09-29 11:14:08 +02:00
..
tests pattern: Ensure accurate origin counts returned by run method 2022-09-29 11:14:08 +02:00
__init__.py arch: Fix broken ref 2022-08-19 19:07:55 +02:00
lister.py Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
tasks.py Add arch lister module (origins from archives). 2022-06-15 09:11:57 +02:00