reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
785 commits 1 branch 0 tags 16 MiB
Commit graph

615 commits

Author SHA1 Message Date
David Douard
4fc1968f1f Rename the bitbucket and github listers to remove the 'tld' part 
so that we can easily manage its configuration (especially in the docker
environment) by referring to this lister as only 'bitbucket' everywhere
(ie. python package name and config file names).
 2019-01-14 12:07:57 +01:00
David Douard
f46f3e2015 Remove explicit setting of the task base class 
since it's now the default base class in swh-scheduler (>= 0.0.39)
 2019-01-10 09:55:17 +01:00
David Douard
6030cb6315 Use pytest's conftest from swh-scheduler 2019-01-10 09:50:05 +01:00
David Douard
fb9265bb03 Generate the gitlab's instance name from the api_baseurl by default 
using the host of the given url.

This allows to create a lister task by simply specify the API base url
and prevent 'inconsistent by default' behavior, eg. with:

  swh-scheduler task add swh-lister-gitlab-full \
      api_baseurl=https://0xacab.org/api/v4

the created task does not use 'gitlab' as instance name (but '0xacab.org'
here).

It's still possible to explicitely specify the instance name if needed.
 2019-01-10 09:49:26 +01:00
David Douard
7db281aa38 Fix gitlab task: pass per_page lister arg paremeter to the lister constructor 2019-01-08 10:35:33 +01:00
David Douard
65f3b9edc8 Add tests for pypi tasks 2019-01-08 10:35:33 +01:00
David Douard
f63b8326c5 Add tests for npm tasks 2019-01-08 10:35:33 +01:00
David Douard
264e9ea574 Add tests for gitlab tasks 2019-01-08 10:35:33 +01:00
David Douard
33ec762bd4 Add tests for debian tasks 2019-01-08 10:35:33 +01:00
David Douard
f375df5892 Add tests for bitbucket tasks 2019-01-08 10:35:33 +01:00
David Douard
b7139619fd Add tests for github tasks 
in order to be able to run unit tests using celery pytest fixtures, we
use a dedicated swh_app fixture that ensure the "main" celery app
is the test app (otherwise subtasks won't work).
 2019-01-08 10:35:33 +01:00
David Douard
dd104759ae Small refactoring in indexing_lister 
- use a generator instead of a while loop
- declare a local logger and use it instead of the root logger.
 2019-01-08 10:35:33 +01:00
David Douard
0583b0e685 Add a 'ping' task for every lister. 2019-01-08 10:35:33 +01:00
David Douard
2d1f0643ff Heavy refactor of the task system 
Get rid of the class based task definition in favor of decorator-based
task declarations.

Doing so, we can get rid of core/tasks.py

Task names are explicitely set to keep compatibility with task
definitions in schedulers' database.

This also add debug statements at the beginning and end of each lister
task.
 2019-01-08 10:33:32 +01:00
David Douard
94c1eaf402 Fix: TaskType has been removed from scheduler 0.0.38 2018-12-20 15:06:19 +01:00
David Douard
72658ff720 cli: fix debian lister so it also uses config overrides 2018-12-20 15:01:18 +01:00
David Douard
5ff8093c5d Simplify listers Model constructor 
the default implementation of SQLAlchemy's declarative API should
work just fine.
 2018-12-12 18:27:11 +01:00
David Douard
532fe90df5 Kill the XMLRPC transport 
it looks unused.
 2018-12-12 18:25:25 +01:00
Antoine Lambert
ffe4ac9a3c swh.lister.npm: Add an incremental npm lister 
This new lister enables to get only new or updated npm packages
since the last listing operation.

Related T1378
Closes T1398
 2018-12-03 17:58:27 +01:00
Antoine Lambert
605a67f51d swh.lister.npm : Add a lister of all available packages in the npm registry 
Related T1378
Closes T1380
 2018-11-26 11:04:13 +01:00
David Douard
849b909a52 Fix rst syntax in docstrings 2018-11-09 11:20:31 +01:00
David Douard
cd20f4d223 Prevent deprecation warnings (logging.warn -> logging.warning) 2018-10-29 10:11:00 +01:00
David Douard
53840ac44b Prevent a flake8 error 2018-10-25 17:42:59 +02:00
Nicolas Dandrimont
b3e815bef8 Rename test methods to test_ to allow collection by pytest 
Part of T1261
 2018-10-15 10:11:07 +02:00
Antoine R. Dumont (@ardumont)
8f5b10b3a3
gitlab.lister: Break asap when problem exists during fetch info 
Prior to this, when wrong url injection happens, no information is
fetched because we have bad input in the first place (404, 400).  This
makes the debugging session quite hard.
 2018-10-08 13:55:33 +02:00
Antoine R. Dumont (@ardumont)
ae514a84c4
gitlab.lister: Do not expect gitlab instances to have credentials 
Only the main gitlab.com instance supports a rate limit (thus the
credentials need), others do not (as per summer 2018).
 2018-10-08 13:55:33 +02:00
Stefano Zacchiroli
0417c0c61a gitlab/models.py: drop unused import 2018-10-01 15:53:04 +02:00
Antoine R. Dumont (@ardumont)
5b20eff7d3
pypi.lister: Use https://pypi.org/project/<name>/ uri as project_url 
The previous url is correct and redirects to this new one.

Related T422
 2018-09-17 16:01:17 +02:00
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review 2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
8c5329ce7c
core.simple_lister: Move import instruction to top 2018-09-06 12:18:56 +02:00
Antoine R. Dumont (@ardumont)
3bb12649ec
pypi.lister: Shuffle the package list 2018-09-06 12:18:37 +02:00
Antoine R. Dumont (@ardumont)
86cfac9277
pypi.lister: Improve parameter names 2018-09-06 09:56:48 +02:00
Antoine R. Dumont (@ardumont)
267ce9b463
swh.lister.pypi: Adapt task creation dict according to latest dev 2018-08-01 14:52:35 +02:00
Antoine R. Dumont (@ardumont)
34eddb1fd7
core/simple_lister: Use bigger batch of data 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
3a65fbb4c8
swh.lister.pypi: Use pypi's legacy html based api to list packages 
The xmlrpc is marked as deprecated [1]. Even if it's not now, the
legacy api is not marked as deprecated. So moving towards this one
sounds more reasonable [2].

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api

Related T422
 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api 
Based solely on pypi's deprecated xmlrpc api [1].  No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://pypi.python.org/simple/

Related T422
 2018-08-01 10:25:21 +02:00
Antoine R. Dumont (@ardumont)
8b2ee221ac
core.lister_base: Batch create origins (storage) & tasks (scheduler) 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
b272a36237
core.lister_base: Refactor task creation 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
3e3b441646
core.lister_transports: Do not use bare except 2018-07-27 17:24:47 +02:00
Antoine R. Dumont (@ardumont)
b6b588dbbb
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:13:13 +02:00
Antoine R. Dumont (@ardumont)
2c69b586bc
Revert "lister.cli: Insert optional flag to permit post insert data" 
This reverts commit e61512afc7.

This commit contains one section not supposed to be there yet (pypi is
not ready)
 2018-07-27 11:12:01 +02:00
Antoine R. Dumont (@ardumont)
e61512afc7
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:06:22 +02:00
Antoine R. Dumont (@ardumont)
726d45b182
swh.lister.cli: Factorize supported listers 2018-07-27 10:21:38 +02:00
Antoine R. Dumont (@ardumont)
6c54b64a8f
swh.lister.cli: Add debian lister to the list of supported listers 2018-07-27 10:19:48 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
ff3afe391c
lister/core: Fix missing use case about no response from api server 
UnboundLocalError could happen otherwise
 2018-07-20 13:33:36 +02:00
Antoine R. Dumont (@ardumont)
bbef4bdeae
swh.lister.gitlab.tasks: Use gitlab as instance name for gitlab.com 2018-07-19 11:28:51 +02:00
Antoine R. Dumont (@ardumont)
30e14677e7
core/lister_base: Remove unused import 2018-07-18 16:05:16 +02:00
Antoine R. Dumont (@ardumont)
91f05745ac
core/lister: Make the listers' scheduler configuration adaptable 
Related T1138
 2018-07-18 12:19:21 +02:00