swh-lister/swh
Antoine Lambert 35871896b2 pattern: Improve handling of max_origins_per_page parameter
Instead of fully consuming the get_origins_from_page generator into
a list and truncate it, prefer to consume the generator origin per
origin and abort the process when the max number of origin per page
is reached.

Indeed some non trivial listers like the cgit one can perform costly
processing, HTTP request for instance, for each origin in a page.
So better not consuming the full generator in a row to avoid such
side effects.
2023-03-21 16:56:48 +01:00
..
lister pattern: Improve handling of max_origins_per_page parameter 2023-03-21 16:56:48 +01:00
__init__.py Pin mypy and drop type annotations which makes mypy unhappy 2021-12-16 16:10:20 +01:00