reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
173 commits 1 branch 0 tags 16 MiB
Commit graph

173 commits

This branch
This branch
All branches
Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
a51c36194e
swh.lister.gitlab: Add full gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
7954e03627
swh.lister: Document swh.lister.tasks's intent 
And remove uneeded indexing name from the RangeListerTask
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
ba146376d6
swh.lister: Add tests around the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
e1a460caa5
swh.lister.gitlab: Improve docstring 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
a527977b6c
README.md: Add a section to demo how to run the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
3ca566776f
swh.lister.gitlab: Make rate limit check optional 
Samples:
- https://0xacab.org/api/v4/projects/
- https://framagit.org/api/v4/projects/
- https://salsa.debian.org/api/v4/projects/
- https://gitlab.com/api/v4/projects/
- https://gitlab.freedesktop.org/api/v4/projects/
- https://gitlab.gnome.org/api/v4/projects/
- https://gitlab.inria.fr/api/v4/projects/

Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
79cd00737f
swh.lister.gitlab: Remove TODO about the 403 response code 
Multiple issues wish for the api to converge on 429 but nothing is
clear nor documented yet:
- https://gitlab.com/gitlab-com/infrastructure/issues/348
- https://gitlab.com/gitlab-org/gitlab-ce/issues/41309
- https://gitlab.com/gitlab-org/gitlab-ce/issues/46522

The only response code mentioned in the documentation is
403 (https://docs.gitlab.com/ee/api/README.html#status-codes).
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
1fd44207eb
README: Add a run section to demonstrate how to run a github lister 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
935b9cd24f
swh.lister.core: Make gitlab lister a paging lister instance 
Related T989
 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
db36c499fe
swh.lister.gitlab: Do not store information we cannot have 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
d9d582f7eb
swh.lister.gitlab: Change the default sort order using id 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
b84db26bee
swh.lister.gitlab: Make the lister's task instance parametric 
Related T989
 2018-07-11 11:26:18 +02:00
Antoine R. Dumont (@ardumont)
3760708897
swh.lister.gitlab: Use one configuration for the gitlab lister 
Related T989
 2018-07-11 11:19:06 +02:00
Antoine R. Dumont (@ardumont)
8ad70b3d60
swh.lister.gitlab: Make the lister instance parametric 
This impacts:
- the credentials reading which needs to be indexed per instance
- the models since a new instance column needs to referenced and
  indexed

Related T989
 2018-07-11 11:16:42 +02:00
Antoine R. Dumont (@ardumont)
9c5963f015
swh.lister.gitlab: Respect camel case convention 
Related T989
 2018-07-11 09:46:20 +02:00
Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url 2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id) 
And not 'project' id
 2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister 
Related T989
 2018-07-03 12:23:11 +02:00
Antoine R. Dumont (@ardumont)
c350434127
swh.lister.core.lister_transports: Fix typo and pep8 violations 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
1540c989b5
README: Use the right default configuration sample 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
27f3eaf9e9
README: Fix and document how to boostrap github lister model 2018-06-27 09:52:55 +02:00
Antoine R. Dumont (@ardumont)
53c12d54c3
etc/crontab: Remove dead code 
This is now running through the swh scheduler
 2018-06-26 15:48:35 +02:00
Antoine R. Dumont (@ardumont)
d868ea69a1
TODO: Remove no longer relevant file 2018-06-26 15:45:23 +02:00
Antoine R. Dumont (@ardumont)
050b927f5d
README.md: Update documentation to current state 
Related T989
 2018-06-26 15:45:22 +02:00
Antoine R. Dumont (@ardumont)
d9a25865a2
docs/tutorials: Format sentences 2018-06-26 10:48:12 +02:00
Antoine R. Dumont (@ardumont)
4b351321d8
docs/tutorials: Remove blank spaces 2018-06-26 10:48:03 +02:00
Antoine R. Dumont (@ardumont)
1f0f169a85
README.md: Rename main readme 2018-06-26 10:47:37 +02:00
Stefano Zacchiroli
4f2361ab0e fix typo in docstrings/comments (tnx codespell) 2018-03-12 12:02:30 +01:00
Nicolas Dandrimont
16e40151dd core.tasks: use proper TaskType import 2018-01-19 14:24:17 +01:00
Stefano Zacchiroli
b8ad57fc05 docs: integrate swh-lister tutorial 
originally appeared on the Software Heritage blog at
https://www.softwareheritage.org/2017/03/24/list-the-content-of-your-favorite-forge-in-just-a-few-steps/
by Avi Kelman

Closes T762
 2017-11-02 17:53:30 +01:00
Stefano Zacchiroli
fe4725d383 docs: add absolute anchor to documentation index 2017-11-02 11:09:07 +01:00
Nicolas Dandrimont
9b58ecbd6d Make setup.py use nose for tests 2017-10-30 17:11:25 +01:00
Nicolas Dandrimont
082f415952 swh.storage is the requirement with the schemata stuff 2017-10-30 17:06:02 +01:00
Nicolas Dandrimont
77585de20c Force utf-8 encoding for tests 2017-10-30 17:05:18 +01:00
Nicolas Dandrimont
2922b68570 Clean up dependencies to enable tests on build 2017-10-30 17:04:49 +01:00
Nicolas Dandrimont
e477a46c60 Add missing __init__.py files 
Helps with tests autodetection
 2017-10-30 16:38:27 +01:00
Nicolas Dandrimont
6a7b0f802e swh.lister.debian.tasks: add task_queue attribute 2017-10-30 15:08:37 +01:00
Nicolas Dandrimont
458a9e6733 Add a new DebianListerTask 2017-10-30 14:19:43 +01:00
Nicolas Dandrimont
6b2863f125 force SQLAlchemy >= 1.0 for package builds 2017-10-13 12:12:36 +02:00
Nicolas Dandrimont
2857dcf926 Cleanup packaging 2017-10-12 17:16:49 +02:00
Nicolas Dandrimont
751b64bc45 Remove the swh/loader/_version.py file on clean 2017-10-11 17:56:08 +02:00
Nicolas Dandrimont
a0d35c4a25 Update debian/rules for tests during build 2017-10-11 17:46:46 +02:00
Nicolas Dandrimont
3aaa2d1ed8 Update debian/control for new dependencies 2017-10-11 17:44:02 +02:00
Nicolas Dandrimont
c6e455ce9b debian.lister: use get_packages() method on the snapshot to list packages 
Allows to list the packages outside of lister context (e.g. for loader tests)
 2017-10-10 16:37:15 +02:00
Nicolas Dandrimont
d2a71ac980 debian.lister: make file size an integer rather than a string 2017-10-10 16:36:58 +02:00