Commit graph

325 commits

Author SHA1 Message Date
Nicolas Dandrimont
4c0699eddc base: Create a new base class for the github loader 2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff generate_tasks: remove now useless file 2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239 models: add task_id and origin_id columns to the repos table 2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683 lister: instantiate a requests session all the time 2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383 lister: only commit every 20 queries 2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8 lister: only one transaction per github API query 2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95 lister: allow sharing the connection between requests 2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936 lister: properly select credentials for each request 2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575 lister: update last_seen on updates 2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108 lister: Support updates 2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b lister: Support multiple credentials 2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c Actually revert to the pre-qless version 2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd storage_utils: Add constants to a constants module 2016-03-17 20:35:08 +01:00
Nicolas Dandrimont
124ebf34df processors: Schedule jobs for users 2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e processors: use proper key when adding a user 2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec lister: Initialize cache from config 2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
5e06110570 cache: use redis instead of a module-level dict 2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0 storage_utils: store user uuids in proper cache 2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250 processors: be more eager in scheduling jobs 2016-03-14 11:13:43 +01:00
Nicolas Dandrimont
b6974d49c7 lister: actually run something when the module is loaded 2016-03-14 10:48:09 +01:00
Nicolas Dandrimont
533f6fa1a3 swh.lister.github: Refactor to use swh.storage instead of sqlalchemy 2016-03-09 19:03:35 +01:00
Nicolas Dandrimont
f13764ba36 Add task generation for Celery 2015-11-25 14:45:19 +01:00
Nicolas Dandrimont
cabf83613c ghlister: move to swh.lister.github as per swh convention
Closes T30
2015-10-27 14:36:20 +01:00