Nicolas Dandrimont
90d4855414
debian: update for swh-lister-github -> swh-lister rename
2017-06-12 18:11:39 +02:00
Nicolas Dandrimont
d3d13261e2
debian/control: prepare for release
2017-06-12 17:18:54 +02:00
Nicolas Dandrimont
af60301f3a
tasks: update to new swh.scheduler API
2017-06-12 15:40:28 +02:00
Antoine Pietri
c81c7de88c
requirements: remove celery (already required by swh.scheduler)
2017-04-12 15:21:22 +02:00
Avi Kelman (fiendish)
68d77fd43f
Refactor lister code
...
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
2017-03-06 12:35:49 +01:00
Antoine Pietri
a6e43f2777
config: use 5002 as the default storage port
2017-02-21 17:23:58 +01:00
Antoine Pietri
ede9e5048c
requirements: split internal and external requirements in two separate files
2017-02-09 14:32:02 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading
...
Related T613
2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
9809deec00
setup.py: install ghlister script
2016-10-20 17:28:22 +02:00
Nicolas Dandrimont
d2483e7893
requirements.txt: use proper syntax
2016-10-20 17:27:26 +02:00
Nicolas Dandrimont
d47905b0a1
tasks: add tasks for incremental and full updates
2016-10-20 17:19:39 +02:00
Nicolas Dandrimont
6fd0184229
lister: update Copyright
2016-10-20 16:59:28 +02:00
Nicolas Dandrimont
7fa507e6ff
lister: disable tasks for deleted repositories
...
When operating on a range of repositories, and we notice that a
repository has disappeared, we disable the task associated with that
repository.
2016-10-20 16:28:19 +02:00
Nicolas Dandrimont
a1a6228e05
lister: retrieve old task and origin id if a full_name has been recycled
...
If a repo changed hands, it is possible that a full_name is recycled. In
that case, we reuse the task_id and origin_id from the old repository
instead of recreating them.
2016-10-20 16:26:53 +02:00
Nicolas Dandrimont
ab25599cca
lister: create origins and tasks for repositories
2016-10-20 16:26:27 +02:00
Nicolas Dandrimont
1eb5d5cf79
lister: move "next" next_id computation earlier
2016-10-20 16:23:05 +02:00
Nicolas Dandrimont
9587b6e591
lister: swap inject_repo arguments to be consistent with rest of file
2016-10-20 15:36:26 +02:00
Nicolas Dandrimont
5fe5abb688
lister: fix wrong reindentation for last_seen update
2016-10-20 15:25:45 +02:00
Nicolas Dandrimont
4b6c899f39
base: the scheduler API returns dicts, not lists
2016-10-20 15:23:25 +02:00
Nicolas Dandrimont
f1a9158b39
lister.py: class-ify the GitHub lister
2016-10-20 14:22:31 +02:00
Nicolas Dandrimont
9ba8fedc4c
base: add implementation for adding origins
2016-10-19 16:53:32 +02:00
Nicolas Dandrimont
bc05a0f0a7
d/control: remove unused dependencies on qless and redis
2016-10-19 15:41:52 +02:00
Nicolas Dandrimont
ca4d346451
requirements.txt: Add inter-swh dependencies
2016-10-19 15:41:26 +02:00
Nicolas Dandrimont
4c0699eddc
base: Create a new base class for the github loader
2016-10-19 15:40:45 +02:00
Nicolas Dandrimont
3954b5d2ff
generate_tasks: remove now useless file
2016-10-19 14:42:15 +02:00
Nicolas Dandrimont
971c253239
models: add task_id and origin_id columns to the repos table
2016-10-13 16:36:25 +02:00
Nicolas Dandrimont
033ef98683
lister: instantiate a requests session all the time
2016-10-13 16:36:03 +02:00
Nicolas Dandrimont
e2a96f1383
lister: only commit every 20 queries
2016-09-13 17:19:11 +02:00
Nicolas Dandrimont
8f5110a6e8
lister: only one transaction per github API query
2016-09-13 16:25:04 +02:00
Nicolas Dandrimont
5b83dbbc95
lister: allow sharing the connection between requests
2016-09-13 16:15:48 +02:00
Nicolas Dandrimont
736f6de936
lister: properly select credentials for each request
2016-09-13 16:14:45 +02:00
Nicolas Dandrimont
0254d72575
lister: update last_seen on updates
2016-09-13 16:11:51 +02:00
Nicolas Dandrimont
6332c6d108
lister: Support updates
2016-09-13 16:05:16 +02:00
Nicolas Dandrimont
0331ba4c7b
lister: Support multiple credentials
2016-09-13 16:05:04 +02:00
Nicolas Dandrimont
4d53974e0c
Actually revert to the pre-qless version
2016-09-13 15:20:09 +02:00
Nicolas Dandrimont
2a62db6827
Revert to the pre-qless refactoring version
2016-09-13 14:57:26 +02:00
Nicolas Dandrimont
9550ca3edd
storage_utils: Add constants to a constants module
2016-03-17 20:35:08 +01:00
Nicolas Dandrimont
52f9fd157e
sync packaging metadata
2016-03-17 19:01:10 +01:00
Nicolas Dandrimont
c20bbab4ef
Add python packaging metadata
2016-03-17 18:02:45 +01:00
Nicolas Dandrimont
04fde18fec
debian: initial debian packaging
2016-03-17 17:59:34 +01:00
Nicolas Dandrimont
124ebf34df
processors: Schedule jobs for users
2016-03-17 17:50:27 +01:00
Nicolas Dandrimont
6fbabbe586
req_queue: use qless instead of a handmade queue
2016-03-17 17:50:03 +01:00
Nicolas Dandrimont
401d37190e
processors: use proper key when adding a user
2016-03-17 17:48:41 +01:00
Nicolas Dandrimont
2fb9eca7ec
lister: Initialize cache from config
2016-03-17 17:47:59 +01:00
Nicolas Dandrimont
c7871e44e8
requirements.txt: add redis
2016-03-17 17:45:59 +01:00
Nicolas Dandrimont
5e06110570
cache: use redis instead of a module-level dict
2016-03-17 17:45:18 +01:00
Nicolas Dandrimont
cf3dfc62c0
storage_utils: store user uuids in proper cache
2016-03-17 17:44:45 +01:00
Nicolas Dandrimont
50d562e250
processors: be more eager in scheduling jobs
2016-03-14 11:13:43 +01:00
Nicolas Dandrimont
b6974d49c7
lister: actually run something when the module is loaded
2016-03-14 10:48:09 +01:00
Nicolas Dandrimont
533f6fa1a3
swh.lister.github: Refactor to use swh.storage instead of sqlalchemy
2016-03-09 19:03:35 +01:00
