reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
958 commits 1 branch 0 tags 16 MiB
Commit graph

958 commits

This branch
This branch
All branches
Author SHA1 Message Date
Nicolas Dandrimont
124ebf34df processors: Schedule jobs for users 2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586 req_queue: use qless instead of a handmade queue 2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e processors: use proper key when adding a user 2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec lister: Initialize cache from config 2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
c7871e44e8 requirements.txt: add redis 2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
5e06110570 cache: use redis instead of a module-level dict 2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0 storage_utils: store user uuids in proper cache 2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250 processors: be more eager in scheduling jobs 2016-03-14 11:13:43 +01:00
Nicolas Dandrimont
b6974d49c7 lister: actually run something when the module is loaded 2016-03-14 10:48:09 +01:00
Nicolas Dandrimont
533f6fa1a3 swh.lister.github: Refactor to use swh.storage instead of sqlalchemy 2016-03-09 19:03:35 +01:00
Nicolas Dandrimont
f13764ba36 Add task generation for Celery 2015-11-25 14:45:19 +01:00
Nicolas Dandrimont
cabf83613c ghlister: move to swh.lister.github as per swh convention 
Closes T30
 2015-10-27 14:36:20 +01:00
Nicolas Dandrimont
0f46452f6b Makefile: add from swh-py-template 2015-10-27 14:35:54 +01:00
Nicolas Dandrimont
8d366bb80c Flake8 fixes 2015-10-27 14:35:37 +01:00
Stefano Zacchiroli
e46f1e9dff pimp_db.sql: lower case SQL keyword 
Rationale: no need to use upper case keywords in a .sql file. Also, this
is more coherent with the other SQL files in SWH.
 2015-09-23 15:00:10 +02:00
Stefano Zacchiroli
0cd63531ef crawler.sql: add views/indexes to inspect missing repos 2015-09-23 14:57:49 +02:00
Stefano Zacchiroli
ecca87dccf requirements.txt: add dependency on requests 2015-09-21 21:13:04 +02:00
Stefano Zacchiroli
376141397d add requirements.txt, listing sqlalchemy as dependency 2015-09-21 21:11:45 +02:00
Stefano Zacchiroli
94b837d4c3 crontab: do not use ~ in *cron* variables (doesntwork) 2015-09-18 09:21:59 +02:00
Stefano Zacchiroli
5499396109 crontab: force SHELL=bash, to ensure tilde-expansion works 2015-09-16 12:16:47 +02:00
Stefano Zacchiroli
35a855ba32 get rid of driver shell script, to simplify deployment 
update README and crontab to match
 2015-09-15 08:57:16 +02:00
Stefano Zacchiroli
8711fad9fa gitignore: ignore usual python cruft 2015-09-13 13:14:38 +02:00
Stefano Zacchiroli
28aa210348 crontab: ease the customization of path/pythonpath 
now only the variable GHLISTER_ROOT at the beginning of the crontab needs to be
properly customized
 2015-09-11 10:07:05 +02:00
Stefano Zacchiroli
fb4d0a3b7c README: fix conffile location in the example 2015-09-10 23:33:55 +02:00
Stefano Zacchiroli
2be420e772 ghlister: change conffile location to match SWH convention 2015-09-10 18:20:02 +02:00
Stefano Zacchiroli
d3e542ab7e lister.py: add conffile setting to en/disable JSON caching 2015-09-10 18:19:41 +02:00
Stefano Zacchiroli
7f706e4fa8 update-github-list: parse log_file from (Python) conffile 2015-09-10 18:18:47 +02:00
Stefano Zacchiroli
49aff6e3d6 update-github-list: do not override PYTHONPATH 
setting it properly is the responsibility of the caller
 2015-09-10 18:18:26 +02:00
Stefano Zacchiroli
df469d01d2 update-github-list: do not compute stats after update 
they forced configuration duplication, and were redundant anyhow
 2015-09-10 18:16:29 +02:00
Stefano Zacchiroli
fca6397cdf ghlister: PEP8 fixes 
also: prefer lexer-time string concatenations to runtime ones
 2015-09-10 18:12:28 +02:00
Stefano Zacchiroli
a97a2953c8 ghlister: remove unused constant CACHE_DIR 
since quite a while cache dir is set via the conffile only
 2015-09-10 18:11:31 +02:00
Stefano Zacchiroli
8768c60ddf etc/: check-in sample crontab 2015-09-10 18:10:32 +02:00
Stefano Zacchiroli
802baf6425 crawler.sql: add crawl_history table 2015-07-07 11:31:21 +02:00
Stefano Zacchiroli
752e23eb4b SQL: move stuff not strictly related to ghlister to crawler.sql 2015-07-07 11:07:19 +02:00
Stefano Zacchiroli
df959dd584 pimp_db: add plpgsql function list some well-known repos 2015-07-03 00:15:36 +02:00
Stefano Zacchiroli
f1adec6fde pimp_db: add plpgsql function to extract random repos sample 2015-07-02 18:04:30 +02:00
Stefano Zacchiroli
48f93a9307 sql: add view to monitor daily repository creations 2015-05-08 15:11:17 +02:00
Stefano Zacchiroli
ce3f9fd17f bin/update-github-list: cron updater script 2015-04-30 12:39:44 +02:00
Stefano Zacchiroli
5bd3ab91f3 sql: add repos_history table 2015-04-30 12:39:32 +02:00
Stefano Zacchiroli
c093e81c23 README: document dependency on psycopg2 2015-04-29 20:13:10 +02:00
Stefano Zacchiroli
059b323341 new TODO item: split cache dir in subdirs 2015-04-29 12:00:45 +02:00
Stefano Zacchiroli
011ce383e2 ghlister: fail when no action is given 2015-04-29 11:08:28 +02:00
Stefano Zacchiroli
944d4a9c49 ghlister: add "catchup" action to continue from last time 2015-04-29 11:05:28 +02:00
Stefano Zacchiroli
e564aa3fa5 add TODO list 2015-04-29 08:55:13 +02:00
Stefano Zacchiroli
647a780764 add SQL info on how to pimp the DB 2015-04-29 08:54:46 +02:00
Stefano Zacchiroli
c51c86a735 status: local change, connect to the right pgsql cluster 2015-04-28 10:31:56 +02:00
Stefano Zacchiroli
a65ab0cf2a bin/batch: add support for continuing from last id 2015-04-28 10:31:30 +02:00
Stefano Zacchiroli
99f2c06c95 gitignore: ignore cache dir even if it is a symlink 2015-04-28 10:31:04 +02:00
Stefano Zacchiroli
9a78c80b5c lister.py: handle temporary connection error 
increase MAX_SLEEP a bit, as it is now used for both connection-level and
throttle-level retries
 2015-04-26 16:29:36 +02:00
Stefano Zacchiroli
8cedd574ab add license information 2015-04-26 16:24:32 +02:00