implement listers as plugins
Listers are declared as plugins via the `swh.workers` entry_point. As such, the registry function is expected to return a dict with the `task_modules` field (as for generic worker plugins), plus: - `lister`: the lister class, - `models`: list of SQLAlchemy models used by this lister, - `init` (optionnal): hook (callable) used to initialize the lister's state (typically, create/initialize the database for this lister). If not set, the default implementation creates database tables (after optionally having deleted exisintg ones) according to models declared in the `models` register field. There is no need for explicitely add lister task modules in the main `conftest` module, but any new/extra lister to be tested must be registered (the tested lister module must be properly installed in the test environment). Also refactor a bit the cli tools: - add support for the standard --config-file option at the 'lister' group level, - move the --db-url to the 'lister' group, - drop the --lister option for the `swh lister db-init` cli tool: initializing (especially with --drop-tables) the database for a single lister is unreliable, since all tables are created using a sibgle MetaData (in the same namespace).
This commit is contained in:
parent
c67a926f26
commit
e3c0ea9d90
18 changed files with 279 additions and 216 deletions
12
setup.py
12
setup.py
|
@ -55,6 +55,18 @@ setup(
|
|||
swh-lister=swh.lister.cli:cli
|
||||
[swh.cli.subcommands]
|
||||
lister=swh.lister.cli:lister
|
||||
[swh.workers]
|
||||
lister.bitbucket=swh.lister.bitbucket:register
|
||||
lister.cgit=swh.lister.cgit:register
|
||||
lister.cran=swh.lister.cran:register
|
||||
lister.debian=swh.lister.debian:register
|
||||
lister.github=swh.lister.github:register
|
||||
lister.gitlab=swh.lister.gitlab:register
|
||||
lister.gnu=swh.lister.gnu:register
|
||||
lister.npm=swh.lister.npm:register
|
||||
lister.packagist=swh.lister.packagist:register
|
||||
lister.phabricator=swh.lister.phabricator:register
|
||||
lister.pypi=swh.lister.pypi:register
|
||||
''',
|
||||
classifiers=[
|
||||
"Programming Language :: Python :: 3",
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue