swh.lister.cgit: Remove repo page visit step

Remove the need to visit every page and extract the
origin url by introducing a parameter url_prefix.
The origin url is in format <prefix>/<repo_name> where
The prefix is same for all the repos for a particular
cgit instance.
This commit is contained in:
Archit Agrawal 2019-06-28 18:15:23 +05:30
parent 7e3c79bb1d
commit 0bf24469b7
9 changed files with 215 additions and 223 deletions

View file

@ -214,7 +214,11 @@ import logging
from swh.lister.cgit.tasks import cgit_lister
logging.basicConfig(level=logging.DEBUG)
cgit_lister(base_url='http://git.savannah.gnu.org/cgit/')
# simple cgit instance
cgit_lister(url='https://git.kernel.org/')
# cgit instance whose listed repositories differ from the base url
cgit_lister(url='https://cgit.kde.org/',
url_prefix='https://anongit.kde.org/')
```
Licensing