Commit graph

16 commits

Author SHA1 Message Date
David Douard
d10f78d80c Adapt cli declaration entrypoint to swh.core 0.3 2020-09-23 17:42:00 +02:00
Antoine R. Dumont (@ardumont)
81a31f3c06
tests: Bump dependency on latest swh-core
This also modifies the test dataset to filename with url decoded filename.
As this is what the latest pytest plugin requires.
2019-11-06 15:01:05 +01:00
Antoine R. Dumont (@ardumont)
56d7cff6e1
debian/model: Install lister model within the lister repository
This is no longer shared between the new debian loader and the lister.

The swh.storage.schemata module is still part of the swh.storage module though.
As this is still a dependency for the current swh.loader.debian production
loader. This will be cleaned up later.

Related D2135
2019-11-04 10:00:54 +01:00
Antoine R. Dumont (@ardumont)
a8cde12d72
tests: Update pytest_plugin according to latest version change 2019-10-14 18:20:15 +02:00
Antoine R. Dumont (@ardumont)
394658e53b
cgit.tests: Check the tasks from the scheduler 2019-10-09 17:57:57 +02:00
David Douard
e3c0ea9d90 implement listers as plugins
Listers are declared as plugins via the `swh.workers` entry_point.

As such, the registry function is expected to return a dict with the
`task_modules` field (as for generic worker plugins), plus:

- `lister`: the lister class,
- `models`: list of SQLAlchemy models used by this lister,
- `init` (optionnal): hook (callable) used to initialize the lister's state
  (typically, create/initialize the database for this lister).
  If not set, the default implementation creates database tables (after
  optionally having deleted exisintg ones) according to models declared in
  the `models` register field.

There is no need for explicitely add lister task modules in the main
`conftest` module, but any new/extra lister to be tested must be registered
(the tested lister module must be properly installed in the test environment).

Also refactor a bit the cli tools:
- add support for the standard --config-file option at the 'lister' group
  level,
- move the --db-url to the 'lister' group,
- drop the --lister option for the `swh lister db-init` cli tool:
  initializing (especially with --drop-tables) the database for a single
  lister is unreliable, since all tables are created using a sibgle MetaData
  (in the same namespace).
2019-09-03 15:02:24 +02:00
David Douard
c1a3c10c5c Bump dependencies 2019-02-06 15:38:11 +01:00
David Douard
6957f3c435 update dep on swh-scheduler>0.0.39 and pytest<4 (tests)
pytest<4 because of https://github.com/pytest-dev/pytest/issues/4641
2019-01-16 16:39:03 +01:00
David Douard
94c1eaf402 Fix: TaskType has been removed from scheduler 0.0.38 2018-12-20 15:06:19 +01:00
Antoine R. Dumont (@ardumont)
8b2ee221ac
core.lister_base: Batch create origins (storage) & tasks (scheduler) 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
b272a36237
core.lister_base: Refactor task creation 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
91f05745ac
core/lister: Make the listers' scheduler configuration adaptable
Related T1138
2018-07-18 12:19:21 +02:00
Nicolas Dandrimont
082f415952 swh.storage is the requirement with the schemata stuff 2017-10-30 17:06:02 +01:00
Sushant Sushant
83ebb95705 Add a lister for Debian-like package archives
This work is based on Sushant's internship and D229.
2017-10-04 12:43:09 +02:00
Nicolas Dandrimont
af60301f3a tasks: update to new swh.scheduler API 2017-06-12 15:40:28 +02:00
Antoine Pietri
ede9e5048c requirements: split internal and external requirements in two separate files 2017-02-09 14:32:02 +01:00