reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

rubygems: Use gems database dump to improve listing output

Browse source 
Instead of using an undocumented rubygems HTTP endpoint that only
gives us the names of the gems, prefer to exploit the daily PostgreSQL
dump of the rubygems.org database.

It enables to list all gems but also all versions of a gem and its
release artifacts. For each relase artifact, the following info are
extracted: version, download URL, sha256 checksum, release date
plus a couple of extra metadata.

The lister will now set list of artifacts and list of metadata as extra
loader arguments when sending a listed origin to the scheduler database.
A last_update date is also computed which should ensure loading tasks
for rubygems will be scheduled only when new releases are available since
last loadings.

To be noted, the lister will spawn a temporary postgres instance so this
require the initdb executable from postgres server installation to be
available in the execution environment.

Related to T1777
This commit is contained in:
Antoine Lambert 2022-10-06 17:51:27 +02:00
parent c22f41a6d7
commit 108816f232
8 changed files with 378 additions and 50 deletions
Download patch file Download diff file Expand all files Collapse all files

6
mypy.ini
View file

@ -42,3 +42,9 @@ ignore_missing_imports = True
[mypy-dulwich.*]
ignore_missing_imports = True
[mypy-testing.postgresql.*]
ignore_missing_imports = True
[mypy-psycopg2.*]
ignore_missing_imports = True