reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
958 commits 1 branch 0 tags 16 MiB
Commit graph

958 commits

This branch
This branch
All branches
Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
2c69b586bc
Revert "lister.cli: Insert optional flag to permit post insert data" 
This reverts commit e61512afc7.

This commit contains one section not supposed to be there yet (pypi is
not ready)
 2018-07-27 11:12:01 +02:00
Antoine R. Dumont (@ardumont)
e61512afc7
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:06:22 +02:00
Antoine R. Dumont (@ardumont)
726d45b182
swh.lister.cli: Factorize supported listers 2018-07-27 10:21:38 +02:00
Antoine R. Dumont (@ardumont)
6c54b64a8f
swh.lister.cli: Add debian lister to the list of supported listers 2018-07-27 10:19:48 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
ff3afe391c
lister/core: Fix missing use case about no response from api server 
UnboundLocalError could happen otherwise
 2018-07-20 13:33:36 +02:00
Antoine R. Dumont (@ardumont)
bbef4bdeae
swh.lister.gitlab.tasks: Use gitlab as instance name for gitlab.com 2018-07-19 11:28:51 +02:00
Antoine R. Dumont (@ardumont)
30e14677e7
core/lister_base: Remove unused import 2018-07-18 16:05:16 +02:00
Antoine R. Dumont (@ardumont)
2dfa8bdb26
README.md: Add gitlab to the lister implementations referenced 2018-07-18 14:25:07 +02:00
Antoine R. Dumont (@ardumont)
d6694a8cb6
debian: Fix tests during debian packaging 
The api response json files used in tests were missing from the sdist
tarball created for package purposes, resulting in test failures.

Related T1137
Related T1138
 2018-07-18 14:15:09 +02:00
Antoine R. Dumont (@ardumont)
f7eb3e529e
README.md: Simplify the readme to use the default configuration 
Related T1138
 2018-07-18 12:22:47 +02:00
Antoine R. Dumont (@ardumont)
91f05745ac
core/lister: Make the listers' scheduler configuration adaptable 
Related T1138
 2018-07-18 12:19:21 +02:00
Antoine R. Dumont (@ardumont)
7fadb1f609
README.md: Align github sample run 2018-07-17 15:48:49 +02:00
Antoine R. Dumont (@ardumont)
63cff5b337
lister.cli: Fix broken imports 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
d08ab241f5
gitlab/lister: Remove unused import 2018-07-17 11:45:07 +02:00
Antoine R. Dumont (@ardumont)
0292bd8cd4
core/lister: Rename module paging_lister to page_by_page_lister 2018-07-17 11:40:21 +02:00
Antoine R. Dumont (@ardumont)
5003a0475b
gitlab.lister: Fix buggy call to self._get_int 
Was missing the dict to read from
 2018-07-17 11:37:53 +02:00
Antoine R. Dumont (@ardumont)
e24cf1d4f1
gitlab.lister: Simplify retrieving headers information 
As response headers' keys are case-insensitive and requests does the
aggregation magic.

[1] http://docs.python-requests.org/en/master/user/quickstart/#response-headers
 2018-07-16 13:57:34 +02:00
Antoine R. Dumont (@ardumont)
ec6968e31a
swh.lister.core.paging_lister: Fix page initialization 2018-07-12 14:34:25 +02:00
Antoine R. Dumont (@ardumont)
81fd5f9c5d
swh.lister.gitlab.tasks: Fix range computations 2018-07-12 14:23:14 +02:00
Antoine R. Dumont (@ardumont)
a69e576c85
swh.lister.gitlab: Fix the total pages reading instruction 2018-07-12 13:52:27 +02:00
Antoine R. Dumont (@ardumont)
cd98af7705
swh.lister.gitlab: Change uid format 2018-07-12 13:46:03 +02:00
Antoine R. Dumont (@ardumont)
4db15aaf16
swh.lister.gitlab: Remove indexable column from gitlab lister 2018-07-12 13:41:47 +02:00
Antoine R. Dumont (@ardumont)
2648f1ae2e
swh.lister.gitlab: Read next page from headers 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
d640fdcc96
swh.lister.gitlab.tests: Separate properly tests per lister 2018-07-12 12:23:46 +02:00
Antoine R. Dumont (@ardumont)
b9544c77f4
swh.lister.gitlab: Add the presence check for the incremental lister 2018-07-11 19:15:54 +02:00
Antoine R. Dumont (@ardumont)
d520891547
swh.lister.core.paging_lister: Adding comments 2018-07-11 18:29:38 +02:00
Antoine R. Dumont (@ardumont)
74d8375261
swh.lister.gitlab.tasks: Remove spurious comma 2018-07-11 18:10:35 +02:00
Antoine R. Dumont (@ardumont)
13bb7aca58
swh.lister.gitlab: Improve headers extraction 2018-07-11 18:09:18 +02:00
Antoine R. Dumont (@ardumont)
847a8d341a
swh.lister.gitlab: Add Incremental lister behavior 
Related T989
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
ccd0525c9b
swh.lister: Do not hardcode the index notion into parameter names 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
b6c5865ab1
swh.lister.paging_lister: Improve lister's base class name 
Also drop the SWH prefix as this is redundant.
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
4c4aa0ead2
swh.lister: Make LISTER_NAME a class attribute 
swh.lister.gitlab: make the 'instance' a constructor parameter
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
581028cfc5
swh.lister.cli: Fix cli docstring 2018-07-11 15:56:33 +02:00
Antoine R. Dumont (@ardumont)
a51c36194e
swh.lister.gitlab: Add full gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
7954e03627
swh.lister: Document swh.lister.tasks's intent 
And remove uneeded indexing name from the RangeListerTask
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
ba146376d6
swh.lister: Add tests around the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
e1a460caa5
swh.lister.gitlab: Improve docstring 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
a527977b6c
README.md: Add a section to demo how to run the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
3ca566776f
swh.lister.gitlab: Make rate limit check optional 
Samples:
- https://0xacab.org/api/v4/projects/
- https://framagit.org/api/v4/projects/
- https://salsa.debian.org/api/v4/projects/
- https://gitlab.com/api/v4/projects/
- https://gitlab.freedesktop.org/api/v4/projects/
- https://gitlab.gnome.org/api/v4/projects/
- https://gitlab.inria.fr/api/v4/projects/

Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
79cd00737f
swh.lister.gitlab: Remove TODO about the 403 response code 
Multiple issues wish for the api to converge on 429 but nothing is
clear nor documented yet:
- https://gitlab.com/gitlab-com/infrastructure/issues/348
- https://gitlab.com/gitlab-org/gitlab-ce/issues/41309
- https://gitlab.com/gitlab-org/gitlab-ce/issues/46522

The only response code mentioned in the documentation is
403 (https://docs.gitlab.com/ee/api/README.html#status-codes).
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
1fd44207eb
README: Add a run section to demonstrate how to run a github lister 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
935b9cd24f
swh.lister.core: Make gitlab lister a paging lister instance 
Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
db36c499fe
swh.lister.gitlab: Do not store information we cannot have 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00