reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
245 commits 1 branch 0 tags 16 MiB
Commit graph

42 commits

Author SHA1 Message Date
David Douard
849b909a52 Fix rst syntax in docstrings 2018-11-09 11:20:31 +01:00
Nicolas Dandrimont
b3e815bef8 Rename test methods to test_ to allow collection by pytest 
Part of T1261
 2018-10-15 10:11:07 +02:00
Antoine R. Dumont (@ardumont)
8f5b10b3a3
gitlab.lister: Break asap when problem exists during fetch info 
Prior to this, when wrong url injection happens, no information is
fetched because we have bad input in the first place (404, 400).  This
makes the debugging session quite hard.
 2018-10-08 13:55:33 +02:00
Antoine R. Dumont (@ardumont)
ae514a84c4
gitlab.lister: Do not expect gitlab instances to have credentials 
Only the main gitlab.com instance supports a rate limit (thus the
credentials need), others do not (as per summer 2018).
 2018-10-08 13:55:33 +02:00
Stefano Zacchiroli
0417c0c61a gitlab/models.py: drop unused import 2018-10-01 15:53:04 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
bbef4bdeae
swh.lister.gitlab.tasks: Use gitlab as instance name for gitlab.com 2018-07-19 11:28:51 +02:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
d08ab241f5
gitlab/lister: Remove unused import 2018-07-17 11:45:07 +02:00
Antoine R. Dumont (@ardumont)
0292bd8cd4
core/lister: Rename module paging_lister to page_by_page_lister 2018-07-17 11:40:21 +02:00
Antoine R. Dumont (@ardumont)
5003a0475b
gitlab.lister: Fix buggy call to self._get_int 
Was missing the dict to read from
 2018-07-17 11:37:53 +02:00
Antoine R. Dumont (@ardumont)
e24cf1d4f1
gitlab.lister: Simplify retrieving headers information 
As response headers' keys are case-insensitive and requests does the
aggregation magic.

[1] http://docs.python-requests.org/en/master/user/quickstart/#response-headers
 2018-07-16 13:57:34 +02:00
Antoine R. Dumont (@ardumont)
81fd5f9c5d
swh.lister.gitlab.tasks: Fix range computations 2018-07-12 14:23:14 +02:00
Antoine R. Dumont (@ardumont)
a69e576c85
swh.lister.gitlab: Fix the total pages reading instruction 2018-07-12 13:52:27 +02:00
Antoine R. Dumont (@ardumont)
cd98af7705
swh.lister.gitlab: Change uid format 2018-07-12 13:46:03 +02:00
Antoine R. Dumont (@ardumont)
4db15aaf16
swh.lister.gitlab: Remove indexable column from gitlab lister 2018-07-12 13:41:47 +02:00
Antoine R. Dumont (@ardumont)
2648f1ae2e
swh.lister.gitlab: Read next page from headers 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
d640fdcc96
swh.lister.gitlab.tests: Separate properly tests per lister 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
d520891547
swh.lister.core.paging_lister: Adding comments 2018-07-11 18:29:38 +02:00
Antoine R. Dumont (@ardumont)
74d8375261
swh.lister.gitlab.tasks: Remove spurious comma 2018-07-11 18:10:35 +02:00
Antoine R. Dumont (@ardumont)
13bb7aca58
swh.lister.gitlab: Improve headers extraction 2018-07-11 18:09:18 +02:00
Antoine R. Dumont (@ardumont)
847a8d341a
swh.lister.gitlab: Add Incremental lister behavior 
Related T989
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
b6c5865ab1
swh.lister.paging_lister: Improve lister's base class name 
Also drop the SWH prefix as this is redundant.
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
4c4aa0ead2
swh.lister: Make LISTER_NAME a class attribute 
swh.lister.gitlab: make the 'instance' a constructor parameter
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
a51c36194e
swh.lister.gitlab: Add full gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
7954e03627
swh.lister: Document swh.lister.tasks's intent 
And remove uneeded indexing name from the RangeListerTask
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
ba146376d6
swh.lister: Add tests around the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
e1a460caa5
swh.lister.gitlab: Improve docstring 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
3ca566776f
swh.lister.gitlab: Make rate limit check optional 
Samples:
- https://0xacab.org/api/v4/projects/
- https://framagit.org/api/v4/projects/
- https://salsa.debian.org/api/v4/projects/
- https://gitlab.com/api/v4/projects/
- https://gitlab.freedesktop.org/api/v4/projects/
- https://gitlab.gnome.org/api/v4/projects/
- https://gitlab.inria.fr/api/v4/projects/

Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
79cd00737f
swh.lister.gitlab: Remove TODO about the 403 response code 
Multiple issues wish for the api to converge on 429 but nothing is
clear nor documented yet:
- https://gitlab.com/gitlab-com/infrastructure/issues/348
- https://gitlab.com/gitlab-org/gitlab-ce/issues/41309
- https://gitlab.com/gitlab-org/gitlab-ce/issues/46522

The only response code mentioned in the documentation is
403 (https://docs.gitlab.com/ee/api/README.html#status-codes).
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
935b9cd24f
swh.lister.core: Make gitlab lister a paging lister instance 
Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
db36c499fe
swh.lister.gitlab: Do not store information we cannot have 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00
Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id) 
And not 'project' id
 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister 
Related T989
 2018-07-03 12:23:11 +02:00