reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
140 commits 1 branch 0 tags 16 MiB
Commit graph

140 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nicolas Dandrimont
9809deec00 setup.py: install ghlister script 2016-10-20 17:28:22 +02:00
Nicolas Dandrimont
d2483e7893 requirements.txt: use proper syntax 2016-10-20 17:27:26 +02:00
Nicolas Dandrimont
d47905b0a1 tasks: add tasks for incremental and full updates 2016-10-20 17:19:39 +02:00
Nicolas Dandrimont
6fd0184229 lister: update Copyright 2016-10-20 16:59:28 +02:00
Nicolas Dandrimont
7fa507e6ff lister: disable tasks for deleted repositories 
When operating on a range of repositories, and we notice that a
repository has disappeared, we disable the task associated with that
repository.
 2016-10-20 16:28:19 +02:00
Nicolas Dandrimont
a1a6228e05 lister: retrieve old task and origin id if a full_name has been recycled 
If a repo changed hands, it is possible that a full_name is recycled. In
that case, we reuse the task_id and origin_id from the old repository
instead of recreating them.
 2016-10-20 16:26:53 +02:00
Nicolas Dandrimont
ab25599cca lister: create origins and tasks for repositories 2016-10-20 16:26:27 +02:00
Nicolas Dandrimont
1eb5d5cf79 lister: move "next" next_id computation earlier 2016-10-20 16:23:05 +02:00
Nicolas Dandrimont
9587b6e591 lister: swap inject_repo arguments to be consistent with rest of file 2016-10-20 15:36:26 +02:00
Nicolas Dandrimont
5fe5abb688 lister: fix wrong reindentation for last_seen update 2016-10-20 15:25:45 +02:00
Nicolas Dandrimont
4b6c899f39 base: the scheduler API returns dicts, not lists 2016-10-20 15:23:25 +02:00
Nicolas Dandrimont
f1a9158b39 lister.py: class-ify the GitHub lister 2016-10-20 14:22:31 +02:00
Nicolas Dandrimont
9ba8fedc4c base: add implementation for adding origins 2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
bc05a0f0a7 d/control: remove unused dependencies on qless and redis 2016-10-19 15:41:52 +02:00
Nicolas Dandrimont
ca4d346451 requirements.txt: Add inter-swh dependencies 2016-10-19 15:41:26 +02:00
Nicolas Dandrimont
4c0699eddc base: Create a new base class for the github loader 2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff generate_tasks: remove now useless file 2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239 models: add task_id and origin_id columns to the repos table 2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683 lister: instantiate a requests session all the time 2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383 lister: only commit every 20 queries 2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8 lister: only one transaction per github API query 2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95 lister: allow sharing the connection between requests 2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936 lister: properly select credentials for each request 2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575 lister: update last_seen on updates 2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108 lister: Support updates 2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b lister: Support multiple credentials 2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c Actually revert to the pre-qless version 2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd storage_utils: Add constants to a constants module 2016-03-17 20:35:08 +01:00
Nicolas Dandrimont
52f9fd157e sync packaging metadata 2016-03-17 19:01:10 +01:00
Nicolas Dandrimont
c20bbab4ef Add python packaging metadata 2016-03-17 18:02:45 +01:00
Nicolas Dandrimont
04fde18fec debian: initial debian packaging 2016-03-17 17:59:34 +01:00
Nicolas Dandrimont
124ebf34df processors: Schedule jobs for users 2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e processors: use proper key when adding a user 2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec lister: Initialize cache from config 2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
c7871e44e8 requirements.txt: add redis 2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
5e06110570 cache: use redis instead of a module-level dict 2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0 storage_utils: store user uuids in proper cache 2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250 processors: be more eager in scheduling jobs 2016-03-14 11:13:43 +01:00
Nicolas Dandrimont
b6974d49c7 lister: actually run something when the module is loaded 2016-03-14 10:48:09 +01:00
Nicolas Dandrimont
533f6fa1a3 swh.lister.github: Refactor to use swh.storage instead of sqlalchemy 2016-03-09 19:03:35 +01:00
Nicolas Dandrimont
f13764ba36 Add task generation for Celery 2015-11-25 14:45:19 +01:00
Nicolas Dandrimont
cabf83613c ghlister: move to swh.lister.github as per swh convention 
Closes T30
 2015-10-27 14:36:20 +01:00
Nicolas Dandrimont
0f46452f6b Makefile: add from swh-py-template 2015-10-27 14:35:54 +01:00
Nicolas Dandrimont
8d366bb80c Flake8 fixes 2015-10-27 14:35:37 +01:00
Stefano Zacchiroli
e46f1e9dff pimp_db.sql: lower case SQL keyword 
Rationale: no need to use upper case keywords in a .sql file. Also, this
is more coherent with the other SQL files in SWH.
 2015-09-23 15:00:10 +02:00
Stefano Zacchiroli
0cd63531ef crawler.sql: add views/indexes to inspect missing repos 2015-09-23 14:57:49 +02:00
Stefano Zacchiroli
ecca87dccf requirements.txt: add dependency on requests 2015-09-21 21:13:04 +02:00
Stefano Zacchiroli
376141397d add requirements.txt, listing sqlalchemy as dependency 2015-09-21 21:11:45 +02:00