reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
708 commits 1 branch 0 tags 16 MiB
Commit graph

38 commits

Author SHA1 Message Date
KShivendu
d34a6232a6 gogs: Introduce Gogs lister 2022-08-03 16:22:06 +05:30
Boris Baldassari
8991c625ea lister: Add new maven lister 
The Maven lister retrieves the maven central indexes, exports them in a
convenient text format, and parse them to identify all src archives and
pom files in the maven repository. Then the pom files are downloaded and
analysed to find and yield any scm reference.

Note: This is a new version of the maven lister diff D6133 which takes
into account the initial round of reviews.

Related to T1724
 2021-11-29 17:33:13 +01:00
Boris Baldassari
04c0a50706 tuleap: initialise lister. 
tuleap-lister: fix args in test_task.

tuleap-lister: Add rate-limiting test + fix debug and typo.

tuleap-lister: code review: fix mocker + tests/setup_cli.

tuleap-lister: code review: fix relister > lister.

tuleap-lister: code review: fix test_task kwargs.

tuleap-lister: code review: Remove authentication useless lines + fix typos.

tuleap-lister: code review: improve results_simplified for svn repos.

tuleap-lister: code review: add name to CONTRIBUTORS file.

tuleap-lister: code review: Update tutorial for misc files to edit.

tuleap-lister: code review: Update copyright to 2021 exactly.

tuleap-lister: code review: Update py files perms -X.

tuleap-lister: code review: minimise json files.

tuleap-lister: code review: fix chmod on json files.

tuleap-lister: code review: fix var names + add tests.

tuleap-lister: code review: fix useless indirection.

tuleap-lister: code review: Add empty repo test, minor typo fixes.
 2021-05-26 11:09:12 +02:00
Antoine Lambert
e72c15e97a docs: Update listers execution instructions 
Remove outdated part about listers database and use swh CLI in
README for executing a lister instead of raw Python code.
 2021-02-05 14:51:24 +01:00
David Douard
a97613a056 Update the list of provided listers in the README file 2020-08-25 18:33:01 +02:00
Archit Agrawal
5727f15cf3 swh.lister.packagist 
Implement a packagist lister to list the
names and metadata url of all the
packages.

Closes 1776
 2019-07-19 19:59:30 +05:30
Archit Agrawal
0bf24469b7 swh.lister.cgit: Remove repo page visit step 
Remove the need to visit every page and extract the
origin url by introducing a parameter url_prefix.
The origin url is in format <prefix>/<repo_name> where
The prefix is same for all the repos for a particular
cgit instance.
 2019-06-28 20:02:07 +05:30
Archit Agrawal
7e3c79bb1d swh.lister.cgit: Add pagination support 
Some cgit instance have a pagination. Modifiy
lister to find all the pages and list all the repos
from all the pages.
 2019-06-28 19:27:25 +05:30
Archit Agrawal
b972a2a88d swh.lister.cgit 
Implemented a lister to list the repos for a given CGit instance.

Closes T1659
 2019-06-28 19:27:25 +05:30
Nancy-Chauhan
83b3a75f11 README: Add missing triple back quote 2019-06-19 11:38:19 +05:30
Archit Agrawal
a9a37a85bf swh.lister.cran 
Add a lister to list all the CRAN packages .
It uses the build-in API in R language to list the packages
and get their metadata. 

Closes T1709
 2019-06-11 21:26:31 +05:30
Archit Agrawal
7c6245e663 swh.lister.gnu: Add function to check for file extension. 
Added a function which will derive the extension from filename
and check if the fie extension match the type of file that is to be
archived.
 2019-06-11 15:12:53 +05:30
Archit Agrawal
709ba8a6e5 swh.lister.gnu: Add functionality to list all the tarballs for a package. 
As discussed in T1389 to ingest all packages using base loader, it need
a list of all the tarballs for a pakage.
Hence modifified lister to recursively list all the tarballs for a
package with their last updated time.
 2019-06-08 21:56:00 +05:30
Archit Agrawal
151f6cd223 swh.lister.gnu 
Implement first pass of gnu lister to list all the
packages present in https://ftp.gnu.org/
Add GNU lister in README and cli.py

Closes T1722
 2019-06-08 21:56:00 +05:30
archit
fedfd73c8e swh.lister.phabricator 
Add a lister of all hosted repositories on a Phabricator instance
Closes T808
 2019-05-15 19:54:33 +05:30
Antoine Lambert
7d192a2f1b README.md: Fix outdated instructions and improve formatting 2019-05-14 10:57:17 +02:00
Archit Agrawal
4b27f9d9c4 Updated toplevel function names in README 2019-03-24 10:54:12 +05:30
Archit Agrawal
26232db926 Removed Extra blank space 2019-03-20 00:54:42 +05:30
Archit Agrawal
fa91132364 Removed extra space from the README 2019-03-20 00:42:18 +05:30
Archit Agrawal
5acb1fefc1 Updated README.md for listers 2019-03-19 15:57:29 +05:30
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review 2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
bd4cf81c12
README: Reference pypi lister 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api 
Based solely on pypi's deprecated xmlrpc api [1].  No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://pypi.python.org/simple/

Related T422
 2018-08-01 10:25:21 +02:00
Antoine R. Dumont (@ardumont)
e62fea6def
README.md: Update to demo the lister debian run 2018-07-27 11:28:44 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
2dfa8bdb26
README.md: Add gitlab to the lister implementations referenced 2018-07-18 14:25:07 +02:00
Antoine R. Dumont (@ardumont)
f7eb3e529e
README.md: Simplify the readme to use the default configuration 
Related T1138
 2018-07-18 12:22:47 +02:00
Antoine R. Dumont (@ardumont)
7fadb1f609
README.md: Align github sample run 2018-07-17 15:48:49 +02:00
Antoine R. Dumont (@ardumont)
d88f1b60c9
core/lister: Make the tasks take an explicit lister_args argument 
Avoid eating *all* arbitrary arguments and passing them along to the
new_lister method.
 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
a527977b6c
README.md: Add a section to demo how to run the gitlab lister 
Related T989
 2018-07-11 15:56:32 +02:00
Antoine R. Dumont (@ardumont)
1fd44207eb
README: Add a run section to demonstrate how to run a github lister 2018-07-11 11:26:19 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
1540c989b5
README: Use the right default configuration sample 2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
27f3eaf9e9
README: Fix and document how to boostrap github lister model 2018-06-27 09:52:55 +02:00
Antoine R. Dumont (@ardumont)
050b927f5d
README.md: Update documentation to current state 
Related T989
 2018-06-26 15:45:22 +02:00
Antoine R. Dumont (@ardumont)
1f0f169a85
README.md: Rename main readme 2018-06-26 10:47:37 +02:00
Renamed from README (Browse further)