Commit graph

11 commits

Author SHA1 Message Date
David Douard
e3c0ea9d90 implement listers as plugins
Listers are declared as plugins via the `swh.workers` entry_point.

As such, the registry function is expected to return a dict with the
`task_modules` field (as for generic worker plugins), plus:

- `lister`: the lister class,
- `models`: list of SQLAlchemy models used by this lister,
- `init` (optionnal): hook (callable) used to initialize the lister's state
  (typically, create/initialize the database for this lister).
  If not set, the default implementation creates database tables (after
  optionally having deleted exisintg ones) according to models declared in
  the `models` register field.

There is no need for explicitely add lister task modules in the main
`conftest` module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:
- add support for the standard --config-file option at the 'lister' group
  level,
- move the --db-url to the 'lister' group,
- drop the --lister option for the `swh lister db-init` cli tool:
  initializing (especially with --drop-tables) the database for a single
  lister is unreliable, since all tables are created using a sibgle MetaData
  (in the same namespace).
2019-09-03 15:02:24 +02:00
David Douard
c1a3c10c5c Bump dependencies 2019-02-06 15:38:11 +01:00
David Douard
6957f3c435 update dep on swh-scheduler>0.0.39 and pytest<4 (tests)
pytest<4 because of https://github.com/pytest-dev/pytest/issues/4641
2019-01-16 16:39:03 +01:00
David Douard
94c1eaf402 Fix: TaskType has been removed from scheduler 0.0.38 2018-12-20 15:06:19 +01:00
Antoine R. Dumont (@ardumont)
8b2ee221ac
core.lister_base: Batch create origins (storage) & tasks (scheduler) 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
b272a36237
core.lister_base: Refactor task creation 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
91f05745ac
core/lister: Make the listers' scheduler configuration adaptable
Related T1138
2018-07-18 12:19:21 +02:00
Nicolas Dandrimont
082f415952 swh.storage is the requirement with the schemata stuff 2017-10-30 17:06:02 +01:00
Sushant Sushant
83ebb95705 Add a lister for Debian-like package archives
This work is based on Sushant's internship and D229.
2017-10-04 12:43:09 +02:00
Nicolas Dandrimont
af60301f3a tasks: update to new swh.scheduler API 2017-06-12 15:40:28 +02:00
Antoine Pietri
ede9e5048c requirements: split internal and external requirements in two separate files 2017-02-09 14:32:02 +01:00