David Douard
f670de298f
Remove debug logging from tasks' code
...
since this is now handled by the SWHTask itself.
2019-01-17 13:58:29 +01:00
David Douard
f46f3e2015
Remove explicit setting of the task base class
...
since it's now the default base class in swh-scheduler (>= 0.0.39)
2019-01-10 09:55:17 +01:00
David Douard
65f3b9edc8
Add tests for pypi tasks
2019-01-08 10:35:33 +01:00
David Douard
0583b0e685
Add a 'ping' task for every lister.
2019-01-08 10:35:33 +01:00
David Douard
2d1f0643ff
Heavy refactor of the task system
...
Get rid of the class based task definition in favor of decorator-based
task declarations.
Doing so, we can get rid of core/tasks.py
Task names are explicitely set to keep compatibility with task
definitions in schedulers' database.
This also add debug statements at the beginning and end of each lister
task.
2019-01-08 10:33:32 +01:00
Antoine R. Dumont (@ardumont)
5b20eff7d3
pypi.lister: Use https://pypi.org/project/ <name>/ uri as project_url
...
The previous url is correct and redirects to this new one.
Related T422
2018-09-17 16:01:17 +02:00
Antoine R. Dumont (@ardumont)
ed64d24634
pypi.lister: Normalize pypi name to PyPI
...
Related T422
2018-09-14 13:24:48 +02:00
Antoine R. Dumont (@ardumont)
cba22b7d19
doc: Fix typos according to review
2018-09-06 14:58:03 +02:00
Antoine R. Dumont (@ardumont)
3bb12649ec
pypi.lister: Shuffle the package list
2018-09-06 12:18:37 +02:00
Antoine R. Dumont (@ardumont)
86cfac9277
pypi.lister: Improve parameter names
2018-09-06 09:56:48 +02:00
Antoine R. Dumont (@ardumont)
267ce9b463
swh.lister.pypi: Adapt task creation dict according to latest dev
2018-08-01 14:52:35 +02:00
Antoine R. Dumont (@ardumont)
3a65fbb4c8
swh.lister.pypi: Use pypi's legacy html based api to list packages
...
The xmlrpc is marked as deprecated [1]. Even if it's not now, the
legacy api is not marked as deprecated. So moving towards this one
sounds more reasonable [2].
[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods
[2] https://warehouse.readthedocs.io/api-reference/legacy/#simple-project-api
Related T422
2018-08-01 10:25:28 +02:00
Antoine R. Dumont (@ardumont)
6ff3b90859
swh.lister.pypi: Add a pypi lister implementation using xmlprc api
...
Based solely on pypi's deprecated xmlrpc api [1]. No other way of listing
pypi.org is referenced (except for parsing an html page through a
legacy api [2])
[1] https://warehouse.readthedocs.io/api-reference/xml-rpc/#pypi-s-xml-rpc-methods
[2] https://pypi.python.org/simple/
Related T422
2018-08-01 10:25:21 +02:00
