plugins: add support for scheduler's task-type declaration

Add a new register-task-types cli that will create missing task-type entries in the
scheduler according to:

- only create missing task-types (do not update them), but check that the
  backend_name field is consistent,
- each SWHTask-based task declared in a module listed in the 'task_modules'
  plugin registry field will be checked and added if needed; tasks which name
  start wit an underscore will not be added,
- added task-type will have:
  - the 'type' field is derived from the task's function name (with underscores
    replaced with dashes),
  - the description field is the first line of that function's docstring,
  - default values as provided by the swh.lister.cli.DEFAULT_TASK_TYPE (with
    a simple pattern matching to have decent default values for full/incremental
    tasks),
  - these default values can be overloaded via the 'task_type' plugin registry
    entry.

For this, we had to rename all tasks names (eg. `cran_lister` -> `list_cran`).

Comes with some tests.
This commit is contained in:
David Douard 2019-09-03 14:39:06 +02:00
parent e3c0ea9d90
commit 8d9deeb8f8
14 changed files with 233 additions and 41 deletions

View file

@ -17,20 +17,21 @@ def new_lister(api_baseurl='https://api.bitbucket.org/2.0', per_page=100):
@app.task(name=__name__ + '.IncrementalBitBucketLister')
def incremental_bitbucket_lister(**lister_args):
def list_bitbucket_incremental(**lister_args):
'''Incremental update of the BitBucket forge'''
lister = new_lister(**lister_args)
lister.run(min_bound=lister.db_last_index(), max_bound=None)
@app.task(name=__name__ + '.RangeBitBucketLister')
def range_bitbucket_lister(start, end, **lister_args):
def _range_bitbucket_lister(start, end, **lister_args):
lister = new_lister(**lister_args)
lister.run(min_bound=start, max_bound=end)
@app.task(name=__name__ + '.FullBitBucketRelister', bind=True)
def full_bitbucket_relister(self, split=None, **lister_args):
"""Relist from the beginning of what's already been listed.
def list_bitbucket_full(self, split=None, **lister_args):
"""Full update of the BitBucket forge
It's not to be called for an initial listing.
@ -42,7 +43,7 @@ def full_bitbucket_relister(self, split=None, **lister_args):
return
random.shuffle(ranges)
promise = group(range_bitbucket_lister.s(minv, maxv, **lister_args)
promise = group(_range_bitbucket_lister.s(minv, maxv, **lister_args)
for minv, maxv in ranges)()
self.log.debug('%s OK (spawned %s subtasks)', (self.name, len(ranges)))
try:
@ -53,5 +54,5 @@ def full_bitbucket_relister(self, split=None, **lister_args):
@app.task(name=__name__ + '.ping')
def ping():
def _ping():
return 'OK'