reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
708 commits 1 branch 0 tags 16 MiB
Commit graph

539 commits

Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
3ca566776f
swh.lister.gitlab: Make rate limit check optional 
Samples:
- https://0xacab.org/api/v4/projects/
- https://framagit.org/api/v4/projects/
- https://salsa.debian.org/api/v4/projects/
- https://gitlab.com/api/v4/projects/
- https://gitlab.freedesktop.org/api/v4/projects/
- https://gitlab.gnome.org/api/v4/projects/
- https://gitlab.inria.fr/api/v4/projects/

Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
79cd00737f
swh.lister.gitlab: Remove TODO about the 403 response code 
Multiple issues wish for the api to converge on 429 but nothing is
clear nor documented yet:
- https://gitlab.com/gitlab-com/infrastructure/issues/348
- https://gitlab.com/gitlab-org/gitlab-ce/issues/41309
- https://gitlab.com/gitlab-org/gitlab-ce/issues/46522

The only response code mentioned in the documentation is
403 (https://docs.gitlab.com/ee/api/README.html#status-codes).
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
935b9cd24f
swh.lister.core: Make gitlab lister a paging lister instance 
Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
db36c499fe
swh.lister.gitlab: Do not store information we cannot have 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00
Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id) 
And not 'project' id
 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister 
Related T989
 2018-07-03 12:23:11 +02:00
Antoine R. Dumont (@ardumont)
c350434127
swh.lister.core.lister_transports: Fix typo and pep8 violations 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting 2018-07-03 12:17:46 +02:00
Stefano Zacchiroli
4f2361ab0e fix typo in docstrings/comments (tnx codespell) 2018-03-12 12:02:30 +01:00
Nicolas Dandrimont
16e40151dd core.tasks: use proper TaskType import 2018-01-19 14:24:17 +01:00
Nicolas Dandrimont
77585de20c Force utf-8 encoding for tests 2017-10-30 17:05:18 +01:00
Nicolas Dandrimont
e477a46c60 Add missing __init__.py files 
Helps with tests autodetection
 2017-10-30 16:38:27 +01:00
Nicolas Dandrimont
6a7b0f802e swh.lister.debian.tasks: add task_queue attribute 2017-10-30 15:08:37 +01:00
Nicolas Dandrimont
458a9e6733 Add a new DebianListerTask 2017-10-30 14:19:43 +01:00
Nicolas Dandrimont
2857dcf926 Cleanup packaging 2017-10-12 17:16:49 +02:00
Nicolas Dandrimont
c6e455ce9b debian.lister: use get_packages() method on the snapshot to list packages 
Allows to list the packages outside of lister context (e.g. for loader tests)
 2017-10-10 16:37:15 +02:00
Nicolas Dandrimont
d2a71ac980 debian.lister: make file size an integer rather than a string 2017-10-10 16:36:58 +02:00
Nicolas Dandrimont
3efd0e33bf lister_transports: load version by using vcversioner's export module 2017-10-10 16:36:38 +02:00
Sushant Sushant
83ebb95705 Add a lister for Debian-like package archives 
This work is based on Sushant's internship and D229.
 2017-10-04 12:43:09 +02:00
Nicolas Dandrimont
6c8e16aae9 lister_transports: allow overriding the parameters to requests 2017-09-11 15:23:17 +02:00
Nicolas Dandrimont
52b5c140c1 lister_base: use the softwareheritage-scheduler-dev db by default 2017-09-11 15:21:56 +02:00
Nicolas Dandrimont
7476f5d533 lister_base: make the default filter return all entries 2017-09-11 15:21:56 +02:00
Stefano Zacchiroli
8a519af8f5 docstring: drop useless heading ":py" domain in crossrefs 2017-09-06 20:27:34 +02:00
Stefano Zacchiroli
e3c59f8fa9 sanitize docstrings for sphinx 2017-09-05 17:15:58 +02:00
Nicolas Dandrimont
5a08bb7278 lister_trasnports: send a custom http user agent 2017-09-05 15:21:55 +02:00
Nicolas Dandrimont
5eca777c75 lister_transports: filter known http response codes 2017-09-05 15:21:17 +02:00
Nicolas Dandrimont
de01552ac5 indexing_lister: actually filter objects by start and end of range 2017-09-05 14:51:35 +02:00
Nicolas Dandrimont
cf3220d1fb github.models: handle the fork argument 2017-09-05 11:38:27 +02:00
Nicolas Dandrimont
4b56b6037c github.models: add fork information to repos 2017-09-04 19:40:45 +02:00
Nicolas Dandrimont
f6f077b789 github.tasks: the github api is rooted at api.github.com 2017-09-04 17:42:20 +02:00
Nicolas Dandrimont
04c22dd56e lister_base: when storing responses for debugging, use a timestamp as filename 2017-09-04 17:41:54 +02:00
Nicolas Dandrimont
3fe5619bcc lister_base: use timezone-aware datetimes everywhere 2017-09-04 17:41:36 +02:00
Nicolas Dandrimont
a85f55dbf9 lister_transports: add headers to debug data 2017-09-04 17:41:06 +02:00
Nicolas Dandrimont
eb47600461 lister_transports: requests credentials are a tuple (username, password) 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
7c6ac9df0d lister_base: cache all responses 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
3c7cfe7d78 lister_base: prepare_folders takes its arguments flattened 2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
b6bdab70c6 lister_base: create_tasks returns a list of tasks 2017-06-28 16:44:52 +02:00
Nicolas Dandrimont
c677ad6067 indexing_lister: use task_ids to disable tasks 2017-06-28 16:44:27 +02:00
Nicolas Dandrimont
75ba12e395 remove useless __init__.py file 2017-06-12 18:22:04 +02:00
Nicolas Dandrimont
af60301f3a tasks: update to new swh.scheduler API 2017-06-12 15:40:28 +02:00