Commit graph

8 commits

Author SHA1 Message Date
Antoine Lambert
bde37867d8 docs: Fix broken external links
Those were spotted thanks to the sphinx linkcheck builder
2025-02-20 10:15:54 +00:00
David Douard
714fccc3c7 python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
Antoine Lambert
6e7bc49ec7 Harmonize listers parameters and add test to check mandatory ones
Ensure that all lister classes have the same set of mandatory parameters
in their constructors, notably: scheduler, url, instance and credentials.

Add a new test checking listers classes have mandatory parameters declared
in their constructors. The purpose is to avoid deployment issues on staging
or production environment as celery tasks can fail to be executed if mandatory
parameters are not handled by listers.

Reated to swh/infra/sysadm-environment#5030.
2023-09-06 11:55:34 +02:00
Nicolas Dandrimont
e785e67315 Hook up recently introduced options to all listers
Hopefully one day we'll be able to replace all of this mess with PEP692
TypedDict kwargs, but that's only on track for Python 3.12.
2022-12-05 16:33:45 +01:00
Antoine Lambert
05cd1de1cd cpan: Fix module version extraction for some edge cases
CPAN API can return versions that are not of str type: either
int or float.

When version equals 0, it means that version failed to be parsed
by CPAN so we try to extract it from release name in that case.

Otherwise we ensure to convert the version to str type.

Related to T2833
2022-10-11 15:24:34 +02:00
Antoine Lambert
f57b8f3a2c cpan: Improve listing process by querying the metacpan release endpoint
Instead of querying the metacpan distribution endpoint to list origins,
prefer to use the release endpoint instead enabling to list all artifacts
associated to CPAN packages by scrolling results.

Compared to previous implementation, it enables to compute a last_update
date for all CPAN packages but also to obtain artifact sha256 checksums
that will be used by the CPAN loader to check downloads integrity.

As the multiple versions of a module are spread across multiple pages
from the CPAN API, origins are sent to the scheduler once all pages
processed, it is also faster to proceed that way.

Related to T2833
2022-10-11 15:24:34 +02:00
Antoine Lambert
fa1205c4df Send package artifact checksums to loaders when info is available
In listers collecting artifacts for each package to load, add artifacts
checksums, when that info is available, in parameters sent to loaders
in order to check downloaded artifact integrity.
2022-09-30 18:44:11 +02:00
Franck Bret
a4aec3894e Cpan: List Perl module origins from cpan.org
Related T2833
2022-09-27 14:29:33 +02:00