reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
931 commits 1 branch 0 tags 16 MiB
Commit graph

931 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nicolas Dandrimont
5fe5abb688 lister: fix wrong reindentation for last_seen update 2016-10-20 15:25:45 +02:00
Nicolas Dandrimont
4b6c899f39 base: the scheduler API returns dicts, not lists 2016-10-20 15:23:25 +02:00
Nicolas Dandrimont
f1a9158b39 lister.py: class-ify the GitHub lister 2016-10-20 14:22:31 +02:00
Nicolas Dandrimont
9ba8fedc4c base: add implementation for adding origins 2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
bc05a0f0a7 d/control: remove unused dependencies on qless and redis 2016-10-19 15:41:52 +02:00
Nicolas Dandrimont
ca4d346451 requirements.txt: Add inter-swh dependencies 2016-10-19 15:41:26 +02:00
Nicolas Dandrimont
4c0699eddc base: Create a new base class for the github loader 2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff generate_tasks: remove now useless file 2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239 models: add task_id and origin_id columns to the repos table 2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683 lister: instantiate a requests session all the time 2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383 lister: only commit every 20 queries 2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8 lister: only one transaction per github API query 2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95 lister: allow sharing the connection between requests 2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936 lister: properly select credentials for each request 2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575 lister: update last_seen on updates 2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108 lister: Support updates 2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b lister: Support multiple credentials 2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c Actually revert to the pre-qless version 2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827 Revert to the pre-qless refactoring version 2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd storage_utils: Add constants to a constants module 2016-03-17 20:35:08 +01:00
Nicolas Dandrimont
52f9fd157e sync packaging metadata 2016-03-17 19:01:10 +01:00
Nicolas Dandrimont
c20bbab4ef Add python packaging metadata 2016-03-17 18:02:45 +01:00
Nicolas Dandrimont
04fde18fec debian: initial debian packaging 2016-03-17 17:59:34 +01:00
Nicolas Dandrimont
124ebf34df processors: Schedule jobs for users 2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e processors: use proper key when adding a user 2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec lister: Initialize cache from config 2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
c7871e44e8 requirements.txt: add redis 2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
5e06110570 cache: use redis instead of a module-level dict 2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0 storage_utils: store user uuids in proper cache 2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250 processors: be more eager in scheduling jobs 2016-03-14 11:13:43 +01:00
Nicolas Dandrimont
b6974d49c7 lister: actually run something when the module is loaded 2016-03-14 10:48:09 +01:00
Nicolas Dandrimont
533f6fa1a3 swh.lister.github: Refactor to use swh.storage instead of sqlalchemy 2016-03-09 19:03:35 +01:00
Nicolas Dandrimont
f13764ba36 Add task generation for Celery 2015-11-25 14:45:19 +01:00
Nicolas Dandrimont
cabf83613c ghlister: move to swh.lister.github as per swh convention 
Closes T30
 2015-10-27 14:36:20 +01:00
Nicolas Dandrimont
0f46452f6b Makefile: add from swh-py-template 2015-10-27 14:35:54 +01:00
Nicolas Dandrimont
8d366bb80c Flake8 fixes 2015-10-27 14:35:37 +01:00
Stefano Zacchiroli
e46f1e9dff pimp_db.sql: lower case SQL keyword 
Rationale: no need to use upper case keywords in a .sql file. Also, this
is more coherent with the other SQL files in SWH.
 2015-09-23 15:00:10 +02:00
Stefano Zacchiroli
0cd63531ef crawler.sql: add views/indexes to inspect missing repos 2015-09-23 14:57:49 +02:00
Stefano Zacchiroli
ecca87dccf requirements.txt: add dependency on requests 2015-09-21 21:13:04 +02:00
Stefano Zacchiroli
376141397d add requirements.txt, listing sqlalchemy as dependency 2015-09-21 21:11:45 +02:00
Stefano Zacchiroli
94b837d4c3 crontab: do not use ~ in *cron* variables (doesntwork) 2015-09-18 09:21:59 +02:00
Stefano Zacchiroli
5499396109 crontab: force SHELL=bash, to ensure tilde-expansion works 2015-09-16 12:16:47 +02:00
Stefano Zacchiroli
35a855ba32 get rid of driver shell script, to simplify deployment 
update README and crontab to match
 2015-09-15 08:57:16 +02:00
Stefano Zacchiroli
8711fad9fa gitignore: ignore usual python cruft 2015-09-13 13:14:38 +02:00
Stefano Zacchiroli
28aa210348 crontab: ease the customization of path/pythonpath 
now only the variable GHLISTER_ROOT at the beginning of the crontab needs to be
properly customized
 2015-09-11 10:07:05 +02:00
Stefano Zacchiroli
fb4d0a3b7c README: fix conffile location in the example 2015-09-10 23:33:55 +02:00
Stefano Zacchiroli
2be420e772 ghlister: change conffile location to match SWH convention 2015-09-10 18:20:02 +02:00
Stefano Zacchiroli
d3e542ab7e lister.py: add conffile setting to en/disable JSON caching 2015-09-10 18:19:41 +02:00
Stefano Zacchiroli
7f706e4fa8 update-github-list: parse log_file from (Python) conffile 2015-09-10 18:18:47 +02:00