packagist: Yield pages of origins to regularly record origins
Instead of sending one page with all origins listed which is britle. When something goes wrong during the listing, the lister currently records nothing.
This commit is contained in:
parent
15a4c4cdb4
commit
b02144b4f9
2 changed files with 5 additions and 4 deletions
|
@ -356,6 +356,7 @@ class Lister(Generic[StateType, PageType]):
|
|||
else:
|
||||
logger.warning("Skipping invalid origin: %s", origin.url)
|
||||
|
||||
logger.debug("Record valid %s origins in the scheduler", len(valid_origins))
|
||||
ret = self.scheduler.record_listed_origins(valid_origins)
|
||||
recorded_origins.extend(origin.url for origin in ret)
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue