reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swh-lister/swh/lister
History
Antoine R. Dumont (@ardumont) f1ae6825e5
packagist: Improve extract package metadata information algorithm 
The current lister implementation lists very few metadata with the hard-coded /p/ base
url (404 on mostly all packages). The packagist api implementation must have evolved
since the initial implementation of the lister (and the first deployment on staging).

Following the upstream documentation [1], it's sensible to first use the /p2/ as it's
performant from the packagist api side. It's then fallbacking to use /p2/+~dev url
scheme, then the /p/ scheme and finally the /packages/ base url if previous result are
either not found or empty (different than no modification since the last visit).

It keeps the initial implementation behavior of stopping immediately if a 304
NotModifiedSince is returned by the server.

[1] https://repo.packagist.org/apidoc
2023-08-02 10:34:55 +02:00
..
arch Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
aur Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
bitbucket Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00
bower Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
cgit cgit: Allow url to be optional 2023-05-23 11:47:51 +02:00
conda Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
cpan Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
cran Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
crates crates: Don't extract unused files 2023-06-20 16:06:21 +02:00
debian debian: refactor inner loop slightly to help mypy 2023-06-21 13:57:27 +02:00
fedora Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
gitea gogs, gitea: Fix task execution to pass along extra kwargs 2022-12-14 16:09:56 +01:00
github github: Fix fixtures use in tests 2023-01-02 18:06:26 +01:00
gitiles Add gitiles lister 2023-07-13 10:30:51 +02:00
gitlab lister: Allow lister to build url out of the instance parameter 2023-05-19 15:03:49 +02:00
gitweb Add Gitweb lister 2023-07-10 16:50:41 +02:00
gnu Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
gogs gogs: Ensure to list all repositories 2023-06-26 15:16:48 +00:00
golang Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00
hackage Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
hex hex: Update loader arguments 2023-03-22 08:45:41 +00:00
launchpad Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00
maven mypy: Bump to 1.0.1 and fix new typing errors 2023-02-17 17:56:07 +01:00
nixguix nixguix/lister: Rename listed origin visit type to tarball-directory 2023-06-08 11:24:38 +02:00
npm Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
nuget Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
opam opam: Fix 'opam init' error when relisting an opam instance 2023-06-29 17:49:21 +02:00
packagist packagist: Improve extract package metadata information algorithm 2023-08-02 10:34:55 +02:00
pagure pagure/tasks: Add missing docstring for list_pagure task function 2023-06-23 14:29:17 +02:00
phabricator Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
pubdev Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
puppet Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
pypi Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00
rubygems Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
sourceforge Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00
stagit Add stagit lister 2023-07-13 11:50:51 +02:00
tests lister.pattern: Make batch record parametric and test it 2023-08-01 15:06:21 +02:00
tuleap Hook up recently introduced options to all listers 2022-12-05 16:33:45 +01:00
__init__.py Add support for more tarball recognition based on extensions 2022-10-25 09:50:31 +02:00
cli.py Fix crash of 'swh lister run' when called without -l 2023-05-10 10:19:26 +02:00
pattern.py lister.pattern: Make batch record parametric and test it 2023-08-01 15:06:21 +02:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
utils.py Use http_retry decorator from swh.core.retry module 2023-04-13 14:19:57 +02:00