reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
626 commits 1 branch 0 tags 16 MiB
Commit graph

36 commits

Author SHA1 Message Date
Boris Baldassari
04c0a50706 tuleap: initialise lister. 
tuleap-lister: fix args in test_task.

tuleap-lister: Add rate-limiting test + fix debug and typo.

tuleap-lister: code review: fix mocker + tests/setup_cli.

tuleap-lister: code review: fix relister > lister.

tuleap-lister: code review: fix test_task kwargs.

tuleap-lister: code review: Remove authentication useless lines + fix typos.

tuleap-lister: code review: improve results_simplified for svn repos.

tuleap-lister: code review: add name to CONTRIBUTORS file.

tuleap-lister: code review: Update tutorial for misc files to edit.

tuleap-lister: code review: Update copyright to 2021 exactly.

tuleap-lister: code review: Update py files perms -X.

tuleap-lister: code review: minimise json files.

tuleap-lister: code review: fix chmod on json files.

tuleap-lister: code review: fix var names + add tests.

tuleap-lister: code review: fix useless indirection.

tuleap-lister: code review: Add empty repo test, minor typo fixes.
 2021-05-26 11:09:12 +02:00
Antoine Lambert
e72c15e97a docs: Update listers execution instructions 
Remove outdated part about listers database and use swh CLI in
README for executing a lister instead of raw Python code.
 2021-02-05 14:51:24 +01:00
David Douard
a97613a056 Update the list of provided listers in the README file 2020-08-25 18:33:01 +02:00
Archit Agrawal
5727f15cf3 swh.lister.packagist 
Implement a packagist lister to list the
names and metadata url of all the
packages.

Closes 1776
 2019-07-19 19:59:30 +05:30
Archit Agrawal
0bf24469b7 swh.lister.cgit: Remove repo page visit step 
Remove the need to visit every page and extract the
origin url by introducing a parameter url_prefix.
The origin url is in format <prefix>/<repo_name> where
The prefix is same for all the repos for a particular
cgit instance.
 2019-06-28 20:02:07 +05:30
Archit Agrawal
7e3c79bb1d swh.lister.cgit: Add pagination support 
Some cgit instance have a pagination. Modifiy
lister to find all the pages and list all the repos
from all the pages.
 2019-06-28 19:27:25 +05:30
Archit Agrawal
b972a2a88d swh.lister.cgit 
Implemented a lister to list the repos for a given CGit instance.

Closes T1659
 2019-06-28 19:27:25 +05:30
Nancy-Chauhan
83b3a75f11 README: Add missing triple back quote 2019-06-19 11:38:19 +05:30
Archit Agrawal
a9a37a85bf swh.lister.cran 
Add a lister to list all the CRAN packages .
It uses the build-in API in R language to list the packages
and get their metadata. 

Closes T1709
 2019-06-11 21:26:31 +05:30
Archit Agrawal
7c6245e663 swh.lister.gnu: Add function to check for file extension. 
Added a function which will derive the extension from filename
and check if the fie extension match the type of file that is to be
archived.
 2019-06-11 15:12:53 +05:30
Archit Agrawal
709ba8a6e5 swh.lister.gnu: Add functionality to list all the tarballs for a package. 
As discussed in T1389 to ingest all packages using base loader, it need
a list of all the tarballs for a pakage.
Hence modifified lister to recursively list all the tarballs for a
package with their last updated time.
 2019-06-08 21:56:00 +05:30
Archit Agrawal
151f6cd223 swh.lister.gnu 
Implement first pass of gnu lister to list all the
packages present in https://ftp.gnu.org/
Add GNU lister in README and cli.py

Closes T1722
 2019-06-08 21:56:00 +05:30
archit
fedfd73c8e swh.lister.phabricator 
Add a lister of all hosted repositories on a Phabricator instance
Closes T808
 2019-05-15 19:54:33 +05:30
Antoine Lambert
7d192a2f1b README.md: Fix outdated instructions and improve formatting 2019-05-14 10:57:17 +02:00
Archit Agrawal
4b27f9d9c4 Updated toplevel function names in README 2019-03-24 10:54:12 +05:30
Archit Agrawal
26232db926 Removed Extra blank space 2019-03-20 00:54:42 +05:30
Archit Agrawal
fa91132364 Removed extra space from the README 2019-03-20 00:42:18 +05:30
Archit Agrawal
5acb1fefc1 Updated README.md for listers 2019-03-19 15:57:29 +05:30
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review 2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
bd4cf81c12
README: Reference pypi lister 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api 
Based solely on pypi's deprecated xmlrpc api [1].  No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://pypi.python.org/simple/

Related T422
 2018-08-01 10:25:21 +02:00
Antoine R. Dumont (@ardumont)
e62fea6def
README.md: Update to demo the lister debian run 2018-07-27 11:28:44 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
2dfa8bdb26
README.md: Add gitlab to the lister implementations referenced 2018-07-18 14:25:07 +02:00
Antoine R. Dumont (@ardumont)
f7eb3e529e
README.md: Simplify the readme to use the default configuration 
Related T1138
 2018-07-18 12:22:47 +02:00
Antoine R. Dumont (@ardumont)
7fadb1f609
README.md: Align github sample run 2018-07-17 15:48:49 +02:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
a527977b6c
README.md: Add a section to demo how to run the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
1fd44207eb
README: Add a run section to demonstrate how to run a github lister 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
1540c989b5
README: Use the right default configuration sample 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
27f3eaf9e9
README: Fix and document how to boostrap github lister model 2018-06-27 09:52:55 +02:00
Antoine R. Dumont (@ardumont)
050b927f5d
README.md: Update documentation to current state 
Related T989
 2018-06-26 15:45:22 +02:00
Antoine R. Dumont (@ardumont)
1f0f169a85
README.md: Rename main readme 2018-06-26 10:47:37 +02:00
Renamed from README (Browse further)