packagist: Yield pages of origins to regularly record origins

Instead of sending one page with all origins listed which is britle.
When something goes wrong during the listing, the lister currently records nothing.
This commit is contained in:
Antoine R. Dumont (@ardumont) 2023-08-03 16:12:13 +02:00
parent 15a4c4cdb4
commit b02144b4f9
No known key found for this signature in database
GPG key ID: 52E2E9840D10C3B8
2 changed files with 5 additions and 4 deletions

View file

@ -356,6 +356,7 @@ class Lister(Generic[StateType, PageType]):
else:
logger.warning("Skipping invalid origin: %s", origin.url)
logger.debug("Record valid %s origins in the scheduler", len(valid_origins))
ret = self.scheduler.record_listed_origins(valid_origins)
recorded_origins.extend(origin.url for origin in ret)