reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
754 commits 1 branch 0 tags 16 MiB
Commit graph

585 commits

Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00
Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id) 
And not 'project' id
 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister 
Related T989
 2018-07-03 12:23:11 +02:00
Antoine R. Dumont (@ardumont)
c350434127
swh.lister.core.lister_transports: Fix typo and pep8 violations 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting 2018-07-03 12:17:46 +02:00
Stefano Zacchiroli
4f2361ab0e fix typo in docstrings/comments (tnx codespell) 2018-03-12 12:02:30 +01:00
Nicolas Dandrimont
16e40151dd core.tasks: use proper TaskType import 2018-01-19 14:24:17 +01:00
Nicolas Dandrimont
77585de20c Force utf-8 encoding for tests 2017-10-30 17:05:18 +01:00
Nicolas Dandrimont
e477a46c60 Add missing __init__.py files 
Helps with tests autodetection
 2017-10-30 16:38:27 +01:00
Nicolas Dandrimont
6a7b0f802e swh.lister.debian.tasks: add task_queue attribute 2017-10-30 15:08:37 +01:00
Nicolas Dandrimont
458a9e6733 Add a new DebianListerTask 2017-10-30 14:19:43 +01:00
Nicolas Dandrimont
2857dcf926 Cleanup packaging 2017-10-12 17:16:49 +02:00
Nicolas Dandrimont
c6e455ce9b debian.lister: use get_packages() method on the snapshot to list packages 
Allows to list the packages outside of lister context (e.g. for loader tests)
 2017-10-10 16:37:15 +02:00
Nicolas Dandrimont
d2a71ac980 debian.lister: make file size an integer rather than a string 2017-10-10 16:36:58 +02:00
Nicolas Dandrimont
3efd0e33bf lister_transports: load version by using vcversioner's export module 2017-10-10 16:36:38 +02:00
Sushant Sushant
83ebb95705 Add a lister for Debian-like package archives 
This work is based on Sushant's internship and D229.
 2017-10-04 12:43:09 +02:00
Nicolas Dandrimont
6c8e16aae9 lister_transports: allow overriding the parameters to requests 2017-09-11 15:23:17 +02:00
Nicolas Dandrimont
52b5c140c1 lister_base: use the softwareheritage-scheduler-dev db by default 2017-09-11 15:21:56 +02:00
Nicolas Dandrimont
7476f5d533 lister_base: make the default filter return all entries 2017-09-11 15:21:56 +02:00
Stefano Zacchiroli
8a519af8f5 docstring: drop useless heading ":py" domain in crossrefs 2017-09-06 20:27:34 +02:00
Stefano Zacchiroli
e3c59f8fa9 sanitize docstrings for sphinx 2017-09-05 17:15:58 +02:00
Nicolas Dandrimont
5a08bb7278 lister_trasnports: send a custom http user agent 2017-09-05 15:21:55 +02:00
Nicolas Dandrimont
5eca777c75 lister_transports: filter known http response codes 2017-09-05 15:21:17 +02:00
Nicolas Dandrimont
de01552ac5 indexing_lister: actually filter objects by start and end of range 2017-09-05 14:51:35 +02:00
Nicolas Dandrimont
cf3220d1fb github.models: handle the fork argument 2017-09-05 11:38:27 +02:00
Nicolas Dandrimont
4b56b6037c github.models: add fork information to repos 2017-09-04 19:40:45 +02:00
Nicolas Dandrimont
f6f077b789 github.tasks: the github api is rooted at api.github.com 2017-09-04 17:42:20 +02:00
Nicolas Dandrimont
04c22dd56e lister_base: when storing responses for debugging, use a timestamp as filename 2017-09-04 17:41:54 +02:00
Nicolas Dandrimont
3fe5619bcc lister_base: use timezone-aware datetimes everywhere 2017-09-04 17:41:36 +02:00
Nicolas Dandrimont
a85f55dbf9 lister_transports: add headers to debug data 2017-09-04 17:41:06 +02:00
Nicolas Dandrimont
eb47600461 lister_transports: requests credentials are a tuple (username, password) 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
7c6ac9df0d lister_base: cache all responses 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
3c7cfe7d78 lister_base: prepare_folders takes its arguments flattened 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
b6bdab70c6 lister_base: create_tasks returns a list of tasks 2017-06-28 16:44:52 +02:00
Nicolas Dandrimont
c677ad6067 indexing_lister: use task_ids to disable tasks 2017-06-28 16:44:27 +02:00
Nicolas Dandrimont
75ba12e395 remove useless __init__.py file 2017-06-12 18:22:04 +02:00
Nicolas Dandrimont
af60301f3a tasks: update to new swh.scheduler API 2017-06-12 15:40:28 +02:00
Avi Kelman (fiendish)
68d77fd43f Refactor lister code 
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
 2017-03-06 12:35:49 +01:00
Antoine Pietri
a6e43f2777 config: use 5002 as the default storage port 2017-02-21 17:23:58 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading 
Related T613
 2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d47905b0a1 tasks: add tasks for incremental and full updates 2016-10-20 17:19:39 +02:00