reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
776 commits 1 branch 0 tags 16 MiB
Find a file
Antoine Lambert 108816f232 rubygems: Use gems database dump to improve listing output 
Instead of using an undocumented rubygems HTTP endpoint that only
gives us the names of the gems, prefer to exploit the daily PostgreSQL
dump of the rubygems.org database.

It enables to list all gems but also all versions of a gem and its
release artifacts. For each relase artifact, the following info are
extracted: version, download URL, sha256 checksum, release date
plus a couple of extra metadata.

The lister will now set list of artifacts and list of metadata as extra
loader arguments when sending a listed origin to the scheduler database.
A last_update date is also computed which should ensure loading tasks
for rubygems will be scheduled only when new releases are available since
last loadings.

To be noted, the lister will spawn a temporary postgres instance so this
require the initdb executable from postgres server installation to be
available in the execution environment.

Related to T1777
2022-10-07 16:54:48 +02:00
docs Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
sql Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
swh rubygems: Use gems database dump to improve listing output 2022-10-07 16:54:48 +02:00
.git-blame-ignore-revs Add .git-blame-ignore-revs file with automatic reformatting commits 2022-04-08 15:15:26 +02:00
.gitignore Use pkg_resources to get the package version instead of vcversioner 2019-11-22 15:49:23 +01:00
.pre-commit-config.yaml pre-commit: Remove codespell commit-msg hook 2022-04-21 13:39:42 +02:00
ACKNOWLEDGEMENTS add ACKNOWLEDGEMENTS 2015-04-26 15:54:25 +02:00
CODE_OF_CONDUCT.md add code of conduct document 2019-07-11 16:29:36 +02:00
conftest.py Use swh.core.github.pytest_plugin in github tests 2022-05-20 16:06:11 +02:00
CONTRIBUTORS gogs: Introduce Gogs lister 2022-08-03 16:22:06 +05:30
LICENSE add license information 2015-04-26 16:24:32 +02:00
Makefile Makefile: add from swh-py-template 2015-10-27 14:35:54 +01:00
MANIFEST.in pytest: Define plugin and declare it in the root conftest 2020-09-02 12:25:15 +02:00
mypy.ini rubygems: Use gems database dump to improve listing output 2022-10-07 16:54:48 +02:00
pyproject.toml pre-commit: Add isort hook and configuration 2020-09-17 17:48:25 +02:00
pytest.ini add strict asyncio_mode in pytest.ini 2022-05-09 12:13:28 +02:00
README.md Add non-incremental Golang modules lister 2022-08-30 14:32:02 +02:00
requirements-swh.txt nixguix: Add lister 2022-10-03 18:26:36 +02:00
requirements-test.txt requirements-test: Remove pytest pinning to < 7 2022-04-06 17:14:45 +02:00
requirements.txt rubygems: Use gems database dump to improve listing output 2022-10-07 16:54:48 +02:00
setup.cfg pre-commit, tox: Bump black from 19.10b0 to 22.3.0 2022-04-08 15:13:41 +02:00
setup.py nixguix: Register task 2022-10-03 18:26:36 +02:00
tox.ini Bump mypy to v0.942 2022-04-26 13:05:44 +02:00

README.md

swh-lister

This component from the Software Heritage stack aims to produce listings of software origins and their urls hosted on various public developer platforms or package managers. As these operations are quite similar, it provides a set of Python modules abstracting common software origins listing behaviors.

It also provides several lister implementations, contained in the following Python modules:

  • swh.lister.bitbucket
  • swh.lister.cgit
  • swh.lister.cran
  • swh.lister.debian
  • swh.lister.gitea
  • swh.lister.github
  • swh.lister.gitlab
  • swh.lister.gnu
  • swh.lister.golang
  • swh.lister.launchpad
  • swh.lister.maven
  • swh.lister.npm
  • swh.lister.packagist
  • swh.lister.phabricator
  • swh.lister.pypi
  • swh.lister.tuleap
  • swh.lister.gogs

Dependencies

All required dependencies can be found in the requirements*.txt files located at the root of the repository.

Local deployment

lister configuration

Each lister implemented so far by Software Heritage (bitbucket, cgit, cran, debian, gitea, github, gitlab, gnu, golang, launchpad, npm, packagist, phabricator, pypi, tuleap, maven) must be configured by following the instructions below (please note that you have to replace <lister_name> by one of the lister name introduced above).

Preparation steps

  1. mkdir ~/.config/swh/
  2. create configuration file ~/.config/swh/listers.yml

Configuration file sample

Minimalistic configuration shared by all listers to add in file ~/.config/swh/listers.yml:

scheduler:
  cls: 'remote'
  args:
    url: 'http://localhost:5008/'

credentials: {}

Note: This expects scheduler (5008) service to run locally

Executing a lister

Once configured, a lister can be executed by using the swh CLI tool with the following options and commands:

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]

Examples:

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://codeberg.org/api/v1/

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm

$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi

Licensing

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

See top-level LICENSE file for the full text of the GNU General Public License along with this program.