swh-lister/swh/lister
Antoine Lambert ff05191b7d packagist: Reimplement lister using new Lister API
The previous implementation was generating tasks for a non implemented
Packagist loader.

The new implementation extracts source repository URL, VCS type and
last update date for each package referenced by Packagist and send
those info to the scheduler.

Packages metadata are retrieved using Packagist API endpoints whose
responses are served from static files, which are guaranteed to be
efficient on the Packagist side (no dymamic queries).
Furthermore, subsequent listing will send the "If-Modified-Since" HTTP
header to only retrieve packages metadata updated since the previous
listing operation in order to save bandwidth and return only origins
which might have new released versions.

Closes T2991
2021-02-02 14:48:47 +01:00
..
bitbucket tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
cgit cgit: remove the repository urls's trailing / 2021-02-01 17:31:08 +01:00
core Drop no longer swh.lister.core.{indexing,page_by_page}_lister 2021-01-27 15:42:57 +01:00
cran Make stateless lister constructors compatible with credentials 2021-01-28 14:42:56 +01:00
debian Make debian lister constructors compatible with credentials 2021-01-28 18:46:52 +01:00
gitea tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
github tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
gitlab gitlab: Deal with missing or trailing / in url input 2021-01-28 10:46:58 +01:00
gnu gnu: Remove dependency on pytz 2021-02-02 13:19:04 +01:00
launchpad launchpad: Remove call to dataclasses.asdict on lister state 2021-01-28 19:17:58 +01:00
npm tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
packagist packagist: Reimplement lister using new Lister API 2021-02-02 14:48:47 +01:00
phabricator tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
pypi Make stateless lister constructors compatible with credentials 2021-01-28 14:42:56 +01:00
tests launchpad: Reimplement lister using new Lister API 2021-01-28 15:22:40 +01:00
__init__.py Hook up listers implemented with the new pattern to the CLI 2021-01-11 11:00:29 +01:00
cli.py cli: Make cli work with new lister 2021-01-28 09:12:56 +01:00
pattern.py pattern: Bump packet split to chunk of 1000 records 2021-01-29 16:55:29 +01:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
pytest_plugin.py tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
utils.py lister: Add utility decorator to ease HTTP requests rate limit handling 2021-01-18 11:28:51 +01:00