04c0a50706
tuleap-lister: fix args in test_task. tuleap-lister: Add rate-limiting test + fix debug and typo. tuleap-lister: code review: fix mocker + tests/setup_cli. tuleap-lister: code review: fix relister > lister. tuleap-lister: code review: fix test_task kwargs. tuleap-lister: code review: Remove authentication useless lines + fix typos. tuleap-lister: code review: improve results_simplified for svn repos. tuleap-lister: code review: add name to CONTRIBUTORS file. tuleap-lister: code review: Update tutorial for misc files to edit. tuleap-lister: code review: Update copyright to 2021 exactly. tuleap-lister: code review: Update py files perms -X. tuleap-lister: code review: minimise json files. tuleap-lister: code review: fix chmod on json files. tuleap-lister: code review: fix var names + add tests. tuleap-lister: code review: fix useless indirection. tuleap-lister: code review: Add empty repo test, minor typo fixes.
101 lines
3 KiB
Markdown
101 lines
3 KiB
Markdown
swh-lister
|
|
==========
|
|
|
|
This component from the Software Heritage stack aims to produce listings
|
|
of software origins and their urls hosted on various public developer platforms
|
|
or package managers. As these operations are quite similar, it provides a set of
|
|
Python modules abstracting common software origins listing behaviors.
|
|
|
|
It also provides several lister implementations, contained in the
|
|
following Python modules:
|
|
|
|
- `swh.lister.bitbucket`
|
|
- `swh.lister.cgit`
|
|
- `swh.lister.cran`
|
|
- `swh.lister.debian`
|
|
- `swh.lister.gitea`
|
|
- `swh.lister.github`
|
|
- `swh.lister.gitlab`
|
|
- `swh.lister.gnu`
|
|
- `swh.lister.launchpad`
|
|
- `swh.lister.npm`
|
|
- `swh.lister.packagist`
|
|
- `swh.lister.phabricator`
|
|
- `swh.lister.pypi`
|
|
- `swh.lister.tuleap`
|
|
|
|
Dependencies
|
|
------------
|
|
|
|
All required dependencies can be found in the `requirements*.txt` files located
|
|
at the root of the repository.
|
|
|
|
Local deployment
|
|
----------------
|
|
|
|
## lister configuration
|
|
|
|
Each lister implemented so far by Software Heritage (`bitbucket`, `cgit`, `cran`, `debian`,
|
|
`gitea`, `github`, `gitlab`, `gnu`, `launchpad`, `npm`, `packagist`, `phabricator`, `pypi`, `tuleap`)
|
|
must be configured by following the instructions below (please note that you have to replace
|
|
`<lister_name>` by one of the lister name introduced above).
|
|
|
|
### Preparation steps
|
|
|
|
1. `mkdir ~/.config/swh/`
|
|
2. create configuration file `~/.config/swh/listers.yml`
|
|
|
|
### Configuration file sample
|
|
|
|
Minimalistic configuration shared by all listers to add in file `~/.config/swh/listers.yml`:
|
|
|
|
```lang=yml
|
|
scheduler:
|
|
cls: 'remote'
|
|
args:
|
|
url: 'http://localhost:5008/'
|
|
|
|
credentials: {}
|
|
```
|
|
|
|
Note: This expects scheduler (5008) service to run locally
|
|
|
|
## Executing a lister
|
|
|
|
Once configured, a lister can be executed by using the `swh` CLI tool with the
|
|
following options and commands:
|
|
|
|
```
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister <lister_name> [lister_parameters]
|
|
```
|
|
|
|
Examples:
|
|
|
|
```
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister bitbucket
|
|
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister cran
|
|
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitea url=https://codeberg.org/api/v1/
|
|
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister gitlab url=https://salsa.debian.org/api/v4/
|
|
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister npm
|
|
|
|
$ swh --log-level DEBUG lister -C ~/.config/swh/listers.yml run --lister pypi
|
|
```
|
|
|
|
Licensing
|
|
---------
|
|
|
|
This program is free software: you can redistribute it and/or modify it under
|
|
the terms of the GNU General Public License as published by the Free Software
|
|
Foundation, either version 3 of the License, or (at your option) any later
|
|
version.
|
|
|
|
This program is distributed in the hope that it will be useful, but WITHOUT ANY
|
|
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
|
|
PARTICULAR PURPOSE. See the GNU General Public License for more details.
|
|
|
|
See top-level LICENSE file for the full text of the GNU General Public License
|
|
along with this program.
|