Commit graph

6 commits

Author SHA1 Message Date
Antoine Lambert
eadb704494 pattern: Ensure termination date is set at the end of listing process
Previously it could be set by any call to the `set_state_in_scheduler`
method.

This was leading to side effects on the save bulk lister while updating
the scheduler state when encountering an invalid or not found origin,
and thus the listing failed.

Fixes #4712.
2024-10-24 12:33:40 +02:00
Antoine Lambert
7609ebf7e1 pattern: Store termination date to scheduler database at end of listing
It enables to track last lister execution date and will be used to schedule
first visits with high priority for listed origins.

Related to swh/devel/swh-scheduler#4687.
2024-10-14 15:03:28 +02:00
Antoine Lambert
6e7bc49ec7 Harmonize listers parameters and add test to check mandatory ones
Ensure that all lister classes have the same set of mandatory parameters
in their constructors, notably: scheduler, url, instance and credentials.

Add a new test checking listers classes have mandatory parameters declared
in their constructors. The purpose is to avoid deployment issues on staging
or production environment as celery tasks can fail to be executed if mandatory
parameters are not handled by listers.

Reated to swh/infra/sysadm-environment#5030.
2023-09-06 11:55:34 +02:00
Nicolas Dandrimont
e785e67315 Hook up recently introduced options to all listers
Hopefully one day we'll be able to replace all of this mess with PEP692
TypedDict kwargs, but that's only on track for Python 3.12.
2022-12-05 16:33:45 +01:00
Franck Bret
065b3f81a1 Hackage: Implement incremental mode
Use http api lastUpload argument in search query to retrieve new or
updated origins since last run

Related to T4597
2022-11-18 13:48:45 +01:00
Franck Bret
6696a8424a Hackage: List origins from hackage.haskell.org, The Haskell Package Repository
Use http api point to get package names and build origin urls.
2022-09-27 14:22:03 +02:00