reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
884 commits 1 branch 0 tags 16 MiB
Commit graph

704 commits

Author SHA1 Message Date
Nicolas Dandrimont
5eca777c75 lister_transports: filter known http response codes 2017-09-05 15:21:17 +02:00
Nicolas Dandrimont
de01552ac5 indexing_lister: actually filter objects by start and end of range 2017-09-05 14:51:35 +02:00
Nicolas Dandrimont
cf3220d1fb github.models: handle the fork argument 2017-09-05 11:38:27 +02:00
Nicolas Dandrimont
4b56b6037c github.models: add fork information to repos 2017-09-04 19:40:45 +02:00
Nicolas Dandrimont
f6f077b789 github.tasks: the github api is rooted at api.github.com 2017-09-04 17:42:20 +02:00
Nicolas Dandrimont
04c22dd56e lister_base: when storing responses for debugging, use a timestamp as filename 2017-09-04 17:41:54 +02:00
Nicolas Dandrimont
3fe5619bcc lister_base: use timezone-aware datetimes everywhere 2017-09-04 17:41:36 +02:00
Nicolas Dandrimont
a85f55dbf9 lister_transports: add headers to debug data 2017-09-04 17:41:06 +02:00
Nicolas Dandrimont
eb47600461 lister_transports: requests credentials are a tuple (username, password) 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
7c6ac9df0d lister_base: cache all responses 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
3c7cfe7d78 lister_base: prepare_folders takes its arguments flattened 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
b6bdab70c6 lister_base: create_tasks returns a list of tasks 2017-06-28 16:44:52 +02:00
Nicolas Dandrimont
c677ad6067 indexing_lister: use task_ids to disable tasks 2017-06-28 16:44:27 +02:00
Nicolas Dandrimont
75ba12e395 remove useless __init__.py file 2017-06-12 18:22:04 +02:00
Nicolas Dandrimont
af60301f3a tasks: update to new swh.scheduler API 2017-06-12 15:40:28 +02:00
Avi Kelman (fiendish)
68d77fd43f Refactor lister code 
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
 2017-03-06 12:35:49 +01:00
Antoine Pietri
a6e43f2777 config: use 5002 as the default storage port 2017-02-21 17:23:58 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading 
Related T613
 2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d47905b0a1 tasks: add tasks for incremental and full updates 2016-10-20 17:19:39 +02:00
Nicolas Dandrimont
6fd0184229 lister: update Copyright 2016-10-20 16:59:28 +02:00
Nicolas Dandrimont
7fa507e6ff lister: disable tasks for deleted repositories 
When operating on a range of repositories, and we notice that a
repository has disappeared, we disable the task associated with that
repository.
 2016-10-20 16:28:19 +02:00
Nicolas Dandrimont
a1a6228e05 lister: retrieve old task and origin id if a full_name has been recycled 
If a repo changed hands, it is possible that a full_name is recycled. In
that case, we reuse the task_id and origin_id from the old repository
instead of recreating them.
 2016-10-20 16:26:53 +02:00
Nicolas Dandrimont
ab25599cca lister: create origins and tasks for repositories 2016-10-20 16:26:27 +02:00
Nicolas Dandrimont
1eb5d5cf79 lister: move "next" next_id computation earlier 2016-10-20 16:23:05 +02:00
Nicolas Dandrimont
9587b6e591 lister: swap inject_repo arguments to be consistent with rest of file 2016-10-20 15:36:26 +02:00
Nicolas Dandrimont
5fe5abb688 lister: fix wrong reindentation for last_seen update 2016-10-20 15:25:45 +02:00
Nicolas Dandrimont
4b6c899f39 base: the scheduler API returns dicts, not lists 2016-10-20 15:23:25 +02:00
Nicolas Dandrimont
f1a9158b39 lister.py: class-ify the GitHub lister 2016-10-20 14:22:31 +02:00
Nicolas Dandrimont
9ba8fedc4c base: add implementation for adding origins 2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
4c0699eddc base: Create a new base class for the github loader 2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff generate_tasks: remove now useless file 2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239 models: add task_id and origin_id columns to the repos table 2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683 lister: instantiate a requests session all the time 2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383 lister: only commit every 20 queries 2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8 lister: only one transaction per github API query 2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95 lister: allow sharing the connection between requests 2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936 lister: properly select credentials for each request 2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575 lister: update last_seen on updates 2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108 lister: Support updates 2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b lister: Support multiple credentials 2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c Actually revert to the pre-qless version 2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd storage_utils: Add constants to a constants module 2016-03-17 20:35:08 +01:00
Nicolas Dandrimont
124ebf34df processors: Schedule jobs for users 2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e processors: use proper key when adding a user 2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec lister: Initialize cache from config 2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
5e06110570 cache: use redis instead of a module-level dict 2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0 storage_utils: store user uuids in proper cache 2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250 processors: be more eager in scheduling jobs 2016-03-14 11:13:43 +01:00