swh-lister/swh/lister
David Douard e3c0ea9d90 implement listers as plugins
Listers are declared as plugins via the `swh.workers` entry_point.

As such, the registry function is expected to return a dict with the
`task_modules` field (as for generic worker plugins), plus:

- `lister`: the lister class,
- `models`: list of SQLAlchemy models used by this lister,
- `init` (optionnal): hook (callable) used to initialize the lister's state
  (typically, create/initialize the database for this lister).
  If not set, the default implementation creates database tables (after
  optionally having deleted exisintg ones) according to models declared in
  the `models` register field.

There is no need for explicitely add lister task modules in the main
`conftest` module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:
- add support for the standard --config-file option at the 'lister' group
  level,
- move the --db-url to the 'lister' group,
- drop the --lister option for the `swh lister db-init` cli tool:
  initializing (especially with --drop-tables) the database for a single
  lister is unreliable, since all tables are created using a sibgle MetaData
  (in the same namespace).
2019-09-03 15:02:24 +02:00
..
bitbucket implement listers as plugins 2019-09-03 15:02:24 +02:00
cgit implement listers as plugins 2019-09-03 15:02:24 +02:00
core implement listers as plugins 2019-09-03 15:02:24 +02:00
cran implement listers as plugins 2019-09-03 15:02:24 +02:00
debian implement listers as plugins 2019-09-03 15:02:24 +02:00
github implement listers as plugins 2019-09-03 15:02:24 +02:00
gitlab implement listers as plugins 2019-09-03 15:02:24 +02:00
gnu implement listers as plugins 2019-09-03 15:02:24 +02:00
npm implement listers as plugins 2019-09-03 15:02:24 +02:00
packagist implement listers as plugins 2019-09-03 15:02:24 +02:00
phabricator implement listers as plugins 2019-09-03 15:02:24 +02:00
pypi implement listers as plugins 2019-09-03 15:02:24 +02:00
tests implement listers as plugins 2019-09-03 15:02:24 +02:00
__init__.py Add missing __init__.py files 2017-10-30 16:38:27 +01:00
cli.py implement listers as plugins 2019-09-03 15:02:24 +02:00
utils.py gitlab.lister: Simplify retrieving headers information 2018-07-16 13:57:34 +02:00