reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
310 commits 1 branch 0 tags 16 MiB
Commit graph

61 commits

Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
b81621274b
lister: Unify credentials structure between listers 
This becomes a dictionary of key <lister-name>, value a dict of key
<instance-name>, value list of dict username/password.

Related T1772
 2019-05-29 14:00:11 +02:00
David Douard
e5c3559033 tasks: fix handling of unsupported promise.save() calls 
the exception can also be an AttributeError.

Also do not reraise this exception (in github/tasks.py). This promise
saving feature is used for tests.
 2019-04-11 11:03:48 +02:00
David Douard
f670de298f Remove debug logging from tasks' code 
since this is now handled by the SWHTask itself.
 2019-01-17 13:58:29 +01:00
David Douard
e31b61bee1 Do not crash range tasks if celery result backend does not support saving the group's state 2019-01-15 15:32:07 +01:00
David Douard
4fc1968f1f Rename the bitbucket and github listers to remove the 'tld' part 
so that we can easily manage its configuration (especially in the docker
environment) by referring to this lister as only 'bitbucket' everywhere
(ie. python package name and config file names).
 2019-01-14 12:07:57 +01:00
David Douard
f46f3e2015 Remove explicit setting of the task base class 
since it's now the default base class in swh-scheduler (>= 0.0.39)
 2019-01-10 09:55:17 +01:00
David Douard
b7139619fd Add tests for github tasks 
in order to be able to run unit tests using celery pytest fixtures, we
use a dedicated swh_app fixture that ensure the "main" celery app
is the test app (otherwise subtasks won't work).
 2019-01-08 10:35:33 +01:00
David Douard
0583b0e685 Add a 'ping' task for every lister. 2019-01-08 10:35:33 +01:00
David Douard
2d1f0643ff Heavy refactor of the task system 
Get rid of the class based task definition in favor of decorator-based
task declarations.

Doing so, we can get rid of core/tasks.py

Task names are explicitely set to keep compatibility with task
definitions in schedulers' database.

This also add debug statements at the beginning and end of each lister
task.
 2019-01-08 10:33:32 +01:00
David Douard
5ff8093c5d Simplify listers Model constructor 
the default implementation of SQLAlchemy's declarative API should
work just fine.
 2018-12-12 18:27:11 +01:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
4db15aaf16
swh.lister.gitlab: Remove indexable column from gitlab lister 2018-07-12 13:41:47 +02:00
Antoine R. Dumont (@ardumont)
d640fdcc96
swh.lister.gitlab.tests: Separate properly tests per lister 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
4c4aa0ead2
swh.lister: Make LISTER_NAME a class attribute 
swh.lister.gitlab: make the 'instance' a constructor parameter
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
7954e03627
swh.lister: Document swh.lister.tasks's intent 
And remove uneeded indexing name from the RangeListerTask
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
ba146376d6
swh.lister: Add tests around the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting 2018-07-03 12:17:46 +02:00
Nicolas Dandrimont
e477a46c60 Add missing __init__.py files 
Helps with tests autodetection
 2017-10-30 16:38:27 +01:00
Nicolas Dandrimont
cf3220d1fb github.models: handle the fork argument 2017-09-05 11:38:27 +02:00
Nicolas Dandrimont
4b56b6037c github.models: add fork information to repos 2017-09-04 19:40:45 +02:00
Nicolas Dandrimont
f6f077b789 github.tasks: the github api is rooted at api.github.com 2017-09-04 17:42:20 +02:00
Nicolas Dandrimont
75ba12e395 remove useless __init__.py file 2017-06-12 18:22:04 +02:00
Avi Kelman (fiendish)
68d77fd43f Refactor lister code 
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
 2017-03-06 12:35:49 +01:00
Antoine Pietri
a6e43f2777 config: use 5002 as the default storage port 2017-02-21 17:23:58 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading 
Related T613
 2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d47905b0a1 tasks: add tasks for incremental and full updates 2016-10-20 17:19:39 +02:00
Nicolas Dandrimont
6fd0184229 lister: update Copyright 2016-10-20 16:59:28 +02:00
Nicolas Dandrimont
7fa507e6ff lister: disable tasks for deleted repositories 
When operating on a range of repositories, and we notice that a
repository has disappeared, we disable the task associated with that
repository.
 2016-10-20 16:28:19 +02:00
Nicolas Dandrimont
a1a6228e05 lister: retrieve old task and origin id if a full_name has been recycled 
If a repo changed hands, it is possible that a full_name is recycled. In
that case, we reuse the task_id and origin_id from the old repository
instead of recreating them.
 2016-10-20 16:26:53 +02:00
Nicolas Dandrimont
ab25599cca lister: create origins and tasks for repositories 2016-10-20 16:26:27 +02:00
Nicolas Dandrimont
1eb5d5cf79 lister: move "next" next_id computation earlier 2016-10-20 16:23:05 +02:00
Nicolas Dandrimont
9587b6e591 lister: swap inject_repo arguments to be consistent with rest of file 2016-10-20 15:36:26 +02:00
Nicolas Dandrimont
5fe5abb688 lister: fix wrong reindentation for last_seen update 2016-10-20 15:25:45 +02:00
Nicolas Dandrimont
4b6c899f39 base: the scheduler API returns dicts, not lists 2016-10-20 15:23:25 +02:00
Nicolas Dandrimont
f1a9158b39 lister.py: class-ify the GitHub lister 2016-10-20 14:22:31 +02:00
Nicolas Dandrimont
9ba8fedc4c base: add implementation for adding origins 2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
4c0699eddc base: Create a new base class for the github loader 2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff generate_tasks: remove now useless file 2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239 models: add task_id and origin_id columns to the repos table 2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683 lister: instantiate a requests session all the time 2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383 lister: only commit every 20 queries 2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8 lister: only one transaction per github API query 2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95 lister: allow sharing the connection between requests 2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936 lister: properly select credentials for each request 2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575 lister: update last_seen on updates 2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108 lister: Support updates 2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b lister: Support multiple credentials 2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c Actually revert to the pre-qless version 2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd storage_utils: Add constants to a constants module 2016-03-17 20:35:08 +01:00