reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
750 commits 1 branch 0 tags 16 MiB
Commit graph

750 commits

This branch
This branch
All branches
Author SHA1 Message Date
David Douard
2d1f0643ff Heavy refactor of the task system 
Get rid of the class based task definition in favor of decorator-based
task declarations.

Doing so, we can get rid of core/tasks.py

Task names are explicitely set to keep compatibility with task
definitions in schedulers' database.

This also add debug statements at the beginning and end of each lister
task.
 2019-01-08 10:33:32 +01:00
David Douard
94c1eaf402 Fix: TaskType has been removed from scheduler 0.0.38 2018-12-20 15:06:19 +01:00
David Douard
72658ff720 cli: fix debian lister so it also uses config overrides 2018-12-20 15:01:18 +01:00
David Douard
5ff8093c5d Simplify listers Model constructor 
the default implementation of SQLAlchemy's declarative API should
work just fine.
 2018-12-12 18:27:11 +01:00
David Douard
532fe90df5 Kill the XMLRPC transport 
it looks unused.
 2018-12-12 18:25:25 +01:00
Antoine Lambert
ffe4ac9a3c swh.lister.npm: Add an incremental npm lister 
This new lister enables to get only new or updated npm packages
since the last listing operation.

Related T1378
Closes T1398
 2018-12-03 17:58:27 +01:00
Antoine Lambert
605a67f51d swh.lister.npm : Add a lister of all available packages in the npm registry 
Related T1378
Closes T1380
 2018-11-26 11:04:13 +01:00
David Douard
da4eb6cd34 doc: update index to match new swh-doc format 
related to T1330
 2018-11-23 13:50:51 +01:00
David Douard
2fa3908283 docs: prevent sphinx from complaining about tutorial.rst not being included 2018-11-09 11:21:08 +01:00
David Douard
849b909a52 Fix rst syntax in docstrings 2018-11-09 11:20:31 +01:00
David Douard
9da0bd26eb setup: kill remaining nose whims to be around 2018-10-30 12:17:25 +01:00
David Douard
428b3aca57 Add a tox file 2018-10-29 10:11:11 +01:00
David Douard
cd20f4d223 Prevent deprecation warnings (logging.warn -> logging.warning) 2018-10-29 10:11:00 +01:00
David Douard
53840ac44b Prevent a flake8 error 2018-10-25 17:42:59 +02:00
David Douard
788efd31aa Fix a typo in setup.py 2018-10-23 19:09:24 +02:00
Nicolas Dandrimont
e0bbadfa61 Don't run pytest in the docs directory 2018-10-23 16:48:14 +02:00
Nicolas Dandrimont
b3e815bef8 Rename test methods to test_ to allow collection by pytest 
Part of T1261
 2018-10-15 10:11:07 +02:00
Antoine R. Dumont (@ardumont)
8f5b10b3a3
gitlab.lister: Break asap when problem exists during fetch info 
Prior to this, when wrong url injection happens, no information is
fetched because we have bad input in the first place (404, 400).  This
makes the debugging session quite hard.
 2018-10-08 13:55:33 +02:00
Antoine R. Dumont (@ardumont)
ae514a84c4
gitlab.lister: Do not expect gitlab instances to have credentials 
Only the main gitlab.com instance supports a rate limit (thus the
credentials need), others do not (as per summer 2018).
 2018-10-08 13:55:33 +02:00
David Douard
f72f2bc6ac add build/ to gitignore 2018-10-08 11:27:54 +02:00
David Douard
9a205b1938 setup: prepare for pypi upload 
related to T1242
 2018-10-08 11:27:54 +02:00
Antoine Pietri
f65a3672bb Add requirements-test.txt 2018-10-05 12:23:15 +02:00
Stefano Zacchiroli
3d65b9f223 docs: add title and brief module description 2018-10-01 15:53:13 +02:00
Stefano Zacchiroli
0417c0c61a gitlab/models.py: drop unused import 2018-10-01 15:53:04 +02:00
Stefano Zacchiroli
26d855c796 use /usr/bin/env python3 as shebang, to be nice to virtualenv 2018-09-19 17:17:51 +02:00
Antoine R. Dumont (@ardumont)
5b20eff7d3
pypi.lister: Use https://pypi.org/project/<name>/ uri as project_url 
The previous url is correct and redirects to this new one.

Related T422
 2018-09-17 16:01:17 +02:00
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI 
Related T422
 2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review 2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
8c5329ce7c
core.simple_lister: Move import instruction to top 2018-09-06 12:18:56 +02:00
Antoine R. Dumont (@ardumont)
3bb12649ec
pypi.lister: Shuffle the package list 2018-09-06 12:18:37 +02:00
Antoine R. Dumont (@ardumont)
86cfac9277
pypi.lister: Improve parameter names 2018-09-06 09:56:48 +02:00
Antoine R. Dumont (@ardumont)
267ce9b463
swh.lister.pypi: Adapt task creation dict according to latest dev 2018-08-01 14:52:35 +02:00
Antoine R. Dumont (@ardumont)
bd4cf81c12
README: Reference pypi lister 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
34eddb1fd7
core/simple_lister: Use bigger batch of data 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
3a65fbb4c8
swh.lister.pypi: Use pypi's legacy html based api to list packages 
The xmlrpc is marked as deprecated [1]. Even if it's not now, the
legacy api is not marked as deprecated. So moving towards this one
sounds more reasonable [2].

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api

Related T422
 2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api 
Based solely on pypi's deprecated xmlrpc api [1].  No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])

[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods

[2] https://pypi.python.org/simple/

Related T422
 2018-08-01 10:25:21 +02:00
Antoine R. Dumont (@ardumont)
94913e2bf3
debian/rules: Fix double export instruction typo 2018-08-01 10:21:11 +02:00
Antoine R. Dumont (@ardumont)
8b2ee221ac
core.lister_base: Batch create origins (storage) & tasks (scheduler) 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
b272a36237
core.lister_base: Refactor task creation 2018-07-27 17:24:49 +02:00
Antoine R. Dumont (@ardumont)
3e3b441646
core.lister_transports: Do not use bare except 2018-07-27 17:24:47 +02:00
Antoine R. Dumont (@ardumont)
e62fea6def
README.md: Update to demo the lister debian run 2018-07-27 11:28:44 +02:00
Antoine R. Dumont (@ardumont)
b6b588dbbb
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:13:13 +02:00
Antoine R. Dumont (@ardumont)
2c69b586bc
Revert "lister.cli: Insert optional flag to permit post insert data" 
This reverts commit e61512afc7.

This commit contains one section not supposed to be there yet (pypi is
not ready)
 2018-07-27 11:12:01 +02:00
Antoine R. Dumont (@ardumont)
e61512afc7
lister.cli: Insert optional flag to permit post insert data 
That's needed for example for having the minimum necessary to make the
debian lister run.
 2018-07-27 11:06:22 +02:00
Antoine R. Dumont (@ardumont)
726d45b182
swh.lister.cli: Factorize supported listers 2018-07-27 10:21:38 +02:00
Antoine R. Dumont (@ardumont)
6c54b64a8f
swh.lister.cli: Add debian lister to the list of supported listers 2018-07-27 10:19:48 +02:00
Antoine R. Dumont (@ardumont)
364786a2da
lister/gitlab: Allow to define the per page elements to read 2018-07-20 13:41:25 +02:00
Antoine R. Dumont (@ardumont)
ff3afe391c
lister/core: Fix missing use case about no response from api server 
UnboundLocalError could happen otherwise
 2018-07-20 13:33:36 +02:00
Antoine R. Dumont (@ardumont)
bbef4bdeae
swh.lister.gitlab.tasks: Use gitlab as instance name for gitlab.com 2018-07-19 11:28:51 +02:00
Antoine R. Dumont (@ardumont)
30e14677e7
core/lister_base: Remove unused import 2018-07-18 16:05:16 +02:00