reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
349 commits 1 branch 0 tags 16 MiB
Commit graph

27 commits

Author SHA1 Message Date
Archit Agrawal
5727f15cf3 swh.lister.packagist 
Implement a packagist lister to list the
names and metadata url of all the
packages.

Closes 1776
 2019-07-19 19:59:30 +05:30
Archit Agrawal
0bf24469b7 swh.lister.cgit: Remove repo page visit step 
Remove the need to visit every page and extract the
origin url by introducing a parameter url_prefix.
The origin url is in format <prefix>/<repo_name> where
The prefix is same for all the repos for a particular
cgit instance.
 2019-06-28 20:02:07 +05:30
Archit Agrawal
b972a2a88d swh.lister.cgit 
Implemented a lister to list the repos for a given CGit instance.

Closes T1659
 2019-06-28 19:27:25 +05:30
Archit Agrawal
a9a37a85bf swh.lister.cran 
Add a lister to list all the CRAN packages .
It uses the build-in API in R language to list the packages
and get their metadata. 

Closes T1709
 2019-06-11 21:26:31 +05:30
Archit Agrawal
151f6cd223 swh.lister.gnu 
Implement first pass of gnu lister to list all the
packages present in https://ftp.gnu.org/
Add GNU lister in README and cli.py

Closes T1722
 2019-06-08 21:56:00 +05:30
David Douard
d6169c7141 cli: register the 'lister' cli subcommand 
also add a cli group named 'lister' for the sake of consistency with
other swh packages and rename the command as 'db-init', like:

  swh lister db-init LISTER [...]
 2019-05-22 13:36:57 +02:00
archit
fedfd73c8e swh.lister.phabricator 
Add a lister of all hosted repositories on a Phabricator instance
Closes T808
 2019-05-15 19:54:33 +05:30
Antoine Lambert
977d2459c3 Remove references to no more used lister_db_url conf entry 2019-05-13 15:18:56 +02:00
Antoine Lambert
8ffd8dadef cli: Fix initialization for all listers 
Prior to this commit, initializing all listers was failing after the
debian lister processing because of global insert_minimum_data init

Related T1629
 2019-04-10 11:33:55 +02:00
Antoine R. Dumont (@ardumont)
262c297a5e
lister.cli: Fix spelling typo 2019-02-14 10:47:28 +01:00
David Douard
0720c8e12e Kill (useless) --create-tables and --with-data cli command options 2019-02-06 15:38:11 +01:00
David Douard
4f2580a97c Make the --lister option of the cli tool a variadic argument 
and add a 'all' possibel value for it, so that one can initialize all the
database for all listers at once.
 2019-02-06 10:19:26 +01:00
David Douard
72658ff720 cli: fix debian lister so it also uses config overrides 2018-12-20 15:01:18 +01:00
Antoine Lambert
ffe4ac9a3c swh.lister.npm: Add an incremental npm lister 
This new lister enables to get only new or updated npm packages
since the last listing operation.

Related T1378
Closes T1398
 2018-12-03 17:58:27 +01:00
Antoine Lambert
605a67f51d swh.lister.npm : Add a lister of all available packages in the npm registry 
Related T1378
Closes T1380
 2018-11-26 11:04:13 +01:00
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api 
Based solely on pypi's deprecated xmlrpc api [1].  No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://pypi.python.org/simple/

Related T422
 2018-08-01 10:25:21 +02:00
Antoine R. Dumont (@ardumont)
b6b588dbbb
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:13:13 +02:00
Antoine R. Dumont (@ardumont)
2c69b586bc
Revert "lister.cli: Insert optional flag to permit post insert data" 
This reverts commit e61512afc7.

This commit contains one section not supposed to be there yet (pypi is
not ready)
 2018-07-27 11:12:01 +02:00
Antoine R. Dumont (@ardumont)
e61512afc7
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:06:22 +02:00
Antoine R. Dumont (@ardumont)
726d45b182
swh.lister.cli: Factorize supported listers 2018-07-27 10:21:38 +02:00
Antoine R. Dumont (@ardumont)
6c54b64a8f
swh.lister.cli: Add debian lister to the list of supported listers 2018-07-27 10:19:48 +02:00
Antoine R. Dumont (@ardumont)
63cff5b337
lister.cli: Fix broken imports 2018-07-17 15:48:48 +02:00
Antoine R. Dumont (@ardumont)
4c4aa0ead2
swh.lister: Make LISTER_NAME a class attribute 
swh.lister.gitlab: make the 'instance' a constructor parameter
 2018-07-11 17:43:41 +02:00
Antoine R. Dumont (@ardumont)
581028cfc5
swh.lister.cli: Fix cli docstring 2018-07-11 15:56:33 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli 2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions 2018-07-03 15:49:52 +02:00