reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
848 commits 1 branch 0 tags 16 MiB
Commit graph

848 commits

This branch
This branch
All branches
Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
f7eb3e529e
README.md: Simplify the readme to use the default configuration 
Related T1138
 2018-07-18 12:22:47 +02:00
Antoine R. Dumont (@ardumont)
91f05745ac
core/lister: Make the listers' scheduler configuration adaptable 
Related T1138
 2018-07-18 12:19:21 +02:00
Antoine R. Dumont (@ardumont)
7fadb1f609
README.md: Align github sample run 2018-07-17 15:48:49 +02:00
Antoine R. Dumont (@ardumont)
63cff5b337
lister.cli: Fix broken imports 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
d08ab241f5
gitlab/lister: Remove unused import 2018-07-17 11:45:07 +02:00
Antoine R. Dumont (@ardumont)
0292bd8cd4
core/lister: Rename module paging_lister to page_by_page_lister 2018-07-17 11:40:21 +02:00
Antoine R. Dumont (@ardumont)
5003a0475b
gitlab.lister: Fix buggy call to self._get_int 
Was missing the dict to read from
 2018-07-17 11:37:53 +02:00
Antoine R. Dumont (@ardumont)
e24cf1d4f1
gitlab.lister: Simplify retrieving headers information 
As response headers' keys are case-insensitive and requests does the
aggregation magic.

[1] http://docs.python-requests.org/en/master/user/quickstart/#response-headers
 2018-07-16 13:57:34 +02:00
Antoine R. Dumont (@ardumont)
ec6968e31a
swh.lister.core.paging_lister: Fix page initialization 2018-07-12 14:34:25 +02:00
Antoine R. Dumont (@ardumont)
81fd5f9c5d
swh.lister.gitlab.tasks: Fix range computations 2018-07-12 14:23:14 +02:00
Antoine R. Dumont (@ardumont)
a69e576c85
swh.lister.gitlab: Fix the total pages reading instruction 2018-07-12 13:52:27 +02:00
Antoine R. Dumont (@ardumont)
cd98af7705
swh.lister.gitlab: Change uid format 2018-07-12 13:46:03 +02:00
Antoine R. Dumont (@ardumont)
4db15aaf16
swh.lister.gitlab: Remove indexable column from gitlab lister 2018-07-12 13:41:47 +02:00
Antoine R. Dumont (@ardumont)
2648f1ae2e
swh.lister.gitlab: Read next page from headers 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
d640fdcc96
swh.lister.gitlab.tests: Separate properly tests per lister 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
b9544c77f4
swh.lister.gitlab: Add the presence check for the incremental lister 2018-07-11 19:15:54 +02:00
Antoine R. Dumont (@ardumont)
d520891547
swh.lister.core.paging_lister: Adding comments 2018-07-11 18:29:38 +02:00
Antoine R. Dumont (@ardumont)
74d8375261
swh.lister.gitlab.tasks: Remove spurious comma 2018-07-11 18:10:35 +02:00
Antoine R. Dumont (@ardumont)
13bb7aca58
swh.lister.gitlab: Improve headers extraction 2018-07-11 18:09:18 +02:00
Antoine R. Dumont (@ardumont)
847a8d341a
swh.lister.gitlab: Add Incremental lister behavior 
Related T989
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
ccd0525c9b
swh.lister: Do not hardcode the index notion into parameter names 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
b6c5865ab1
swh.lister.paging_lister: Improve lister's base class name 
Also drop the SWH prefix as this is redundant.
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
4c4aa0ead2
swh.lister: Make LISTER_NAME a class attribute 
swh.lister.gitlab: make the 'instance' a constructor parameter
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
581028cfc5
swh.lister.cli: Fix cli docstring 2018-07-11 15:56:33 +02:00
Antoine R. Dumont (@ardumont)
a51c36194e
swh.lister.gitlab: Add full gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
7954e03627
swh.lister: Document swh.lister.tasks's intent 
And remove uneeded indexing name from the RangeListerTask
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
ba146376d6
swh.lister: Add tests around the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
e1a460caa5
swh.lister.gitlab: Improve docstring 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
a527977b6c
README.md: Add a section to demo how to run the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
3ca566776f
swh.lister.gitlab: Make rate limit check optional 
Samples:
- https://0xacab.org/api/v4/projects/
- https://framagit.org/api/v4/projects/
- https://salsa.debian.org/api/v4/projects/
- https://gitlab.com/api/v4/projects/
- https://gitlab.freedesktop.org/api/v4/projects/
- https://gitlab.gnome.org/api/v4/projects/
- https://gitlab.inria.fr/api/v4/projects/

Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
79cd00737f
swh.lister.gitlab: Remove TODO about the 403 response code 
Multiple issues wish for the api to converge on 429 but nothing is
clear nor documented yet:
- https://gitlab.com/gitlab-com/infrastructure/issues/348
- https://gitlab.com/gitlab-org/gitlab-ce/issues/41309
- https://gitlab.com/gitlab-org/gitlab-ce/issues/46522

The only response code mentioned in the documentation is
403 (https://docs.gitlab.com/ee/api/README.html#status-codes).
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
1fd44207eb
README: Add a run section to demonstrate how to run a github lister 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
935b9cd24f
swh.lister.core: Make gitlab lister a paging lister instance 
Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
db36c499fe
swh.lister.gitlab: Do not store information we cannot have 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00
Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id) 
And not 'project' id
 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister 
Related T989
 2018-07-03 12:23:11 +02:00
Antoine R. Dumont (@ardumont)
c350434127
swh.lister.core.lister_transports: Fix typo and pep8 violations 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
1540c989b5
README: Use the right default configuration sample 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting 2018-07-03 12:17:46 +02:00