reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
915 commits 1 branch 0 tags 16 MiB
Commit graph

915 commits

This branch
This branch
All branches
Author SHA1 Message Date
David Douard
065d3a64fc Fix SWHIndexingLister.db_partition_indices(); ensure partition size is not zero 
when this lister is called for the first time, db_num_entries() may return
a null value, so the min() will also be zero, making the range() call crash.
 2019-01-15 15:32:06 +01:00
David Douard
028ceca90d Fix bitbucker lister: request_uri expect 'identifier' to be a date 
which it won't be when the lister is called for the first time (since
the run() method will be be called with min_bond=None in this case).
 2019-01-15 15:31:59 +01:00
David Douard
4fc1968f1f Rename the bitbucket and github listers to remove the 'tld' part 
so that we can easily manage its configuration (especially in the docker
environment) by referring to this lister as only 'bitbucket' everywhere
(ie. python package name and config file names).
 2019-01-14 12:07:57 +01:00
David Douard
f46f3e2015 Remove explicit setting of the task base class 
since it's now the default base class in swh-scheduler (>= 0.0.39)
 2019-01-10 09:55:17 +01:00
David Douard
6030cb6315 Use pytest's conftest from swh-scheduler 2019-01-10 09:50:05 +01:00
David Douard
fb9265bb03 Generate the gitlab's instance name from the api_baseurl by default 
using the host of the given url.

This allows to create a lister task by simply specify the API base url
and prevent 'inconsistent by default' behavior, eg. with:

  swh-scheduler task add swh-lister-gitlab-full \
      api_baseurl=https://0xacab.org/api/v4

the created task does not use 'gitlab' as instance name (but '0xacab.org'
here).

It's still possible to explicitely specify the instance name if needed.
 2019-01-10 09:49:26 +01:00
David Douard
7db281aa38 Fix gitlab task: pass per_page lister arg paremeter to the lister constructor 2019-01-08 10:35:33 +01:00
David Douard
65f3b9edc8 Add tests for pypi tasks 2019-01-08 10:35:33 +01:00
David Douard
f63b8326c5 Add tests for npm tasks 2019-01-08 10:35:33 +01:00
David Douard
264e9ea574 Add tests for gitlab tasks 2019-01-08 10:35:33 +01:00
David Douard
33ec762bd4 Add tests for debian tasks 2019-01-08 10:35:33 +01:00
David Douard
f375df5892 Add tests for bitbucket tasks 2019-01-08 10:35:33 +01:00
David Douard
b7139619fd Add tests for github tasks 
in order to be able to run unit tests using celery pytest fixtures, we
use a dedicated swh_app fixture that ensure the "main" celery app
is the test app (otherwise subtasks won't work).
 2019-01-08 10:35:33 +01:00
David Douard
dd104759ae Small refactoring in indexing_lister 
- use a generator instead of a while loop
- declare a local logger and use it instead of the root logger.
 2019-01-08 10:35:33 +01:00
David Douard
0583b0e685 Add a 'ping' task for every lister. 2019-01-08 10:35:33 +01:00
David Douard
2d1f0643ff Heavy refactor of the task system 
Get rid of the class based task definition in favor of decorator-based
task declarations.

Doing so, we can get rid of core/tasks.py

Task names are explicitely set to keep compatibility with task
definitions in schedulers' database.

This also add debug statements at the beginning and end of each lister
task.
 2019-01-08 10:33:32 +01:00
David Douard
94c1eaf402 Fix: TaskType has been removed from scheduler 0.0.38 2018-12-20 15:06:19 +01:00
David Douard
72658ff720 cli: fix debian lister so it also uses config overrides 2018-12-20 15:01:18 +01:00
David Douard
5ff8093c5d Simplify listers Model constructor 
the default implementation of SQLAlchemy's declarative API should
work just fine.
 2018-12-12 18:27:11 +01:00
David Douard
532fe90df5 Kill the XMLRPC transport 
it looks unused.
 2018-12-12 18:25:25 +01:00
Antoine Lambert
ffe4ac9a3c swh.lister.npm: Add an incremental npm lister 
This new lister enables to get only new or updated npm packages
since the last listing operation.

Related T1378
Closes T1398
 2018-12-03 17:58:27 +01:00
Antoine Lambert
605a67f51d swh.lister.npm : Add a lister of all available packages in the npm registry 
Related T1378
Closes T1380
 2018-11-26 11:04:13 +01:00
David Douard
da4eb6cd34 doc: update index to match new swh-doc format 
related to T1330
 2018-11-23 13:50:51 +01:00
David Douard
2fa3908283 docs: prevent sphinx from complaining about tutorial.rst not being included 2018-11-09 11:21:08 +01:00
David Douard
849b909a52 Fix rst syntax in docstrings 2018-11-09 11:20:31 +01:00
David Douard
9da0bd26eb setup: kill remaining nose whims to be around 2018-10-30 12:17:25 +01:00
David Douard
428b3aca57 Add a tox file 2018-10-29 10:11:11 +01:00
David Douard
cd20f4d223 Prevent deprecation warnings (logging.warn -> logging.warning) 2018-10-29 10:11:00 +01:00
David Douard
53840ac44b Prevent a flake8 error 2018-10-25 17:42:59 +02:00
David Douard
788efd31aa Fix a typo in setup.py 2018-10-23 19:09:24 +02:00
Nicolas Dandrimont
e0bbadfa61 Don't run pytest in the docs directory 2018-10-23 16:48:14 +02:00
Nicolas Dandrimont
b3e815bef8 Rename test methods to test_ to allow collection by pytest 
Part of T1261
 2018-10-15 10:11:07 +02:00
Antoine R. Dumont (@ardumont)
8f5b10b3a3
gitlab.lister: Break asap when problem exists during fetch info 
Prior to this, when wrong url injection happens, no information is
fetched because we have bad input in the first place (404, 400).  This
makes the debugging session quite hard.
 2018-10-08 13:55:33 +02:00
Antoine R. Dumont (@ardumont)
ae514a84c4
gitlab.lister: Do not expect gitlab instances to have credentials 
Only the main gitlab.com instance supports a rate limit (thus the
credentials need), others do not (as per summer 2018).
 2018-10-08 13:55:33 +02:00
David Douard
f72f2bc6ac add build/ to gitignore 2018-10-08 11:27:54 +02:00
David Douard
9a205b1938 setup: prepare for pypi upload 
related to T1242
 2018-10-08 11:27:54 +02:00
Antoine Pietri
f65a3672bb Add requirements-test.txt 2018-10-05 12:23:15 +02:00
Stefano Zacchiroli
3d65b9f223 docs: add title and brief module description 2018-10-01 15:53:13 +02:00
Stefano Zacchiroli
0417c0c61a gitlab/models.py: drop unused import 2018-10-01 15:53:04 +02:00
Stefano Zacchiroli
26d855c796 use /usr/bin/env python3 as shebang, to be nice to virtualenv 2018-09-19 17:17:51 +02:00
Antoine R. Dumont (@ardumont)
5b20eff7d3
pypi.lister: Use https://pypi.org/project/<name>/ uri as project_url 
The previous url is correct and redirects to this new one.

Related T422
 2018-09-17 16:01:17 +02:00
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review 2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
8c5329ce7c
core.simple_lister: Move import instruction to top 2018-09-06 12:18:56 +02:00
Antoine R. Dumont (@ardumont)
3bb12649ec
pypi.lister: Shuffle the package list 2018-09-06 12:18:37 +02:00
Antoine R. Dumont (@ardumont)
86cfac9277
pypi.lister: Improve parameter names 2018-09-06 09:56:48 +02:00
Antoine R. Dumont (@ardumont)
267ce9b463
swh.lister.pypi: Adapt task creation dict according to latest dev 2018-08-01 14:52:35 +02:00
Antoine R. Dumont (@ardumont)
bd4cf81c12
README: Reference pypi lister 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
34eddb1fd7
core/simple_lister: Use bigger batch of data 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
3a65fbb4c8
swh.lister.pypi: Use pypi's legacy html based api to list packages 
The xmlrpc is marked as deprecated [1]. Even if it's not now, the
legacy api is not marked as deprecated. So moving towards this one
sounds more reasonable [2].

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api

Related T422
 2018-08-01 10:25:28 +02:00