reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
736 commits 1 branch 0 tags 16 MiB
Commit graph

736 commits

This branch
This branch
All branches
Author SHA1 Message Date
Antoine Lambert
b743c36496 phabricator: Add test for new lister implementation 
Also remove no longer used JSON files.
 2021-01-18 12:11:26 +01:00
Antoine Lambert
d691c04eb8 phabricator: Allow to pass forge base URL as lister parameter 2021-01-18 12:11:26 +01:00
Antoine Lambert
d1fbccd988 lister: Add utility decorator to ease HTTP requests rate limit handling 
Add swh.lister.utils.throttling_retry decorator enabling to retry a
function that performs an HTTP request who can return a 429 status code.

The implementation is based on the tenacity module and it is assumed
that the requests library is used when querying an URL.

The default wait strategy is based on exponential backoff.

The default max number of attempts is set to 5, HTTPError exception
will then be reraised.

All tenacity.retry parameters can also be overridden in client code.
 2021-01-18 11:28:51 +01:00
Antoine Lambert
c782275296 phabricator/tasks: Fix task function return type 
Previously, the following error was raised when the task has finished
its execution: "Object of type ListerStats is not JSON serializable".

So ensure ListerStats object gets converted to dict before returning it.

Also add missing test for task function.
 2021-01-11 17:59:24 +01:00
Antoine Lambert
b48f71ff93 phabricator/tasks: Allow to pass api_token as optional parameter 
This is useful when one wants to test the lister in docker environment.
 2021-01-11 17:53:11 +01:00
Nicolas Dandrimont
9944295729 Implement phabricator lister using the new pattern class 2021-01-11 11:00:29 +01:00
Nicolas Dandrimont
734901747b Implement a base pattern for listers with no state storage 2021-01-11 11:00:29 +01:00
Nicolas Dandrimont
b63aa83b41 Reimplement the GitHub lister using the new pattern class 
This replaces the test data with some manually generated answers, which allows
us to test a few more cases for instantiating the lister.

This also expands test coverage to test behavior on rate-limited requests.
 2021-01-11 11:00:29 +01:00
Nicolas Dandrimont
f1eabc5283 Add a helper to instantiate a new-style lister from a config file 
This helper will be used in the task entry points.
 2021-01-11 11:00:29 +01:00
Nicolas Dandrimont
525fc0102d Hook up listers implemented with the new pattern to the CLI 
We stop depending on the ListerBase implementation. The main hoop we're jumping
through is the config override mechanism in swh.lister.get_lister, as it's
really specifc to the ListerBase `override_config` argument, which is dropped in
pattern.Lister (in favor of explicit arguments at lister instantiation).

We implement a small shim in swh.lister.pattern.Lister to give
backwards-compatibility for the new pattern to get_lister.

This generic configuration override mechanism will probably be completely
removed when the configuration mechanism is reworked. We'll see.
 2021-01-11 11:00:29 +01:00
Nicolas Dandrimont
9e083c1eea Introduce a simpler base pattern for lister implementations. 
This new pattern uses the lister support features introduced in swh.scheduler to
replace the database management done in previous iterations of the listers.
 2021-01-11 11:00:29 +01:00
David Douard
b1811f9f8a Add a cli section to the doc 2021-01-05 11:53:22 +01:00
Antoine R. Dumont (@ardumont)
d2f4781669
requirements: Rework dependencies 
Without the following, and the new swh.scheduler bump, some dependencies were
no longer resolved properly.

Related to T2746
 2020-11-23 15:39:51 +01:00
Antoine R. Dumont (@ardumont)
2e9bb5388b
requirements-test.txt: Explicit swh.core[db] as test requirement 
instead of pytest-postgresql which is a transitive dependency from swh.core[db]

Related to T2746
 2020-11-20 17:26:25 +01:00
Antoine R. Dumont (@ardumont)
884585034b
lister.phabricator.tests: Add task-type related to lister 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
243f3573bc
lister.npm.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
36bc51cec5
lister.gnu.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
20a91482ca
lister.gitlab.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
978fbbe029
lister.github.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
def0eb5060
lister.gitea.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
47ccce3817
lister.cran.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
Antoine R. Dumont (@ardumont)
5d4b38999d
lister.cgit.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
Antoine R. Dumont (@ardumont)
c5af0efbf4
lister.bitbucket.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
f09b9f8cfa
lister.launchpad.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b90ffa4bdd
tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b35dff1266
Create listing task with a default of 3 if unspecified 
This will allow to retry task if they do not specify it.
 2020-10-30 09:10:27 +01:00
Antoine R. Dumont (@ardumont)
e2a861c801
debian.tests: Fix test 
The scheduler fixture introduced truncates tables in between tests. The debian
tests unfortunately share state and it broke when that changed. This fixes the
test by avoiding the truncation of the scheduler db table "task".

Ideally those tests need to be reworked to avoid sharing state between tests.

[1] https://jenkins.softwareheritage.org/job/DLS/job/tests/1043
 2020-10-30 09:09:56 +01:00
Antoine R. Dumont (@ardumont)
a19cb5fb51
lister.pytest_plugin: Simplify fixture setup 2020-10-21 17:46:52 +02:00
Antoine R. Dumont (@ardumont)
d7d38090f5
lister.config: Adapt scheduler configuration structure 2020-10-19 09:42:16 +02:00
Antoine R. Dumont (@ardumont)
30ad6200a2
Drop mock_get_scheduler which creates indirection for no good reason 
This is no longer useful, as removing it and tests are still ok.
 2020-10-16 14:32:46 +02:00
Antoine R. Dumont (@ardumont)
2ff92ef406
lister_base: Drop leftover mixin SWHConfig which is no longer used 
Related to D4178
 2020-10-07 14:00:57 +02:00
Antoine R. Dumont (@ardumont)
56f08b73f6
lister*: Migrate away from SWHConfig mixin 
Related to T1532 T1410 D3965
 2020-10-05 13:30:38 +02:00
Stefano Zacchiroli
4d6b99188e tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops 2020-10-02 16:23:33 +02:00
Nicolas Dandrimont
5e4bb28398 Run isort after the CLI import changes 2020-09-25 14:19:21 +02:00
David Douard
d10f78d80c Adapt cli declaration entrypoint to swh.core 0.3 2020-09-23 17:42:00 +02:00
Antoine Lambert
22f7181294 python: Reorder imports with isort 
Related to T2610
 2020-09-17 17:48:27 +02:00
Antoine Lambert
d24846a957 pre-commit: Add isort hook and configuration 
Related to T2610
 2020-09-17 17:48:25 +02:00
Antoine Lambert
cef98c9b9b pre-commit: Update flake8 hook configuration 
flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8
 2020-09-17 13:55:01 +02:00
David Douard
a00f151462 cli: speedup the swh cli command startup time 
by moving import statements in functions.

Related to T2575.
 2020-09-10 17:15:38 +02:00
Antoine R. Dumont (@ardumont)
d1ce9b0912
tox/pytest: Activate doctest-module 
Related to D3899#96289
 2020-09-10 11:47:47 +02:00
Antoine R. Dumont (@ardumont)
31efda62e7
gitea.lister: Fix uid to be unique across instance 
The gitea lister can be run on multiple instances which could use the same id.
So listing another gitea instance, the current code would fail to insert data
for such case.

This commit fixes that behavior by prefixing the uid with the instance name.

Related to T2577
 2020-09-10 11:21:50 +02:00
Antoine R. Dumont (@ardumont)
e3c856b5ee
utils.split_range: Split into not overlapping ranges 
Existing listers use the `is_within_bound` [1] method from the base lister.
This method uses inclusive boundaries in all cases.

As some "range" task listers [2] [3] are using `split_range` function to create
"overlapping" ranges, this can cause concurrent insert issues down the line [4].

This commit adapts the function `split_range` to make the generated ranges no
longer overlap.

[1]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199

[2]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41

[3]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41

Related to T2577
 2020-09-10 11:01:44 +02:00
Antoine R. Dumont (@ardumont)
725c1fe4ad
test_utils: Migrate to pytest 2020-09-09 18:48:07 +02:00
Antoine R. Dumont (@ardumont)
66a61f3dd2
gitea.tasks: Fix parameter name from 'sort' to 'order' 
This fixes [1]

[1] https://sentry.softwareheritage.org/share/issue/b0119b56f24347bcb58ac28c68685c62/
 2020-09-09 12:10:23 +02:00
Antoine R. Dumont (@ardumont)
7d44bc2e75
launchpad.tasks: Update copyright headers 2020-09-08 14:42:37 +02:00
Vincent SELLIER
74ca3d0b87
Launchpad: rename task name to match conventions 
Related to T2358
 2020-09-08 14:21:47 +02:00
Antoine R. Dumont (@ardumont)
5a5b7ef70b
tests: Separate lister instantiations 
Prior to this commit, all listers were instantiated at the same time even if
only one was needed. This commit separates those instantiations.

The only drawback to this is the db model initialization which now happens at
each lister instantiation. This can be dealt with if needed at another time
though.
 2020-09-02 12:49:00 +02:00
Antoine R. Dumont (@ardumont)
92422dcf75
pytest_plugin: Instantiate only lister with no particular setup 
This should fix the remaining blocking problems in the jenkins build failure
report [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/78/consoleFull
 2020-09-02 12:25:15 +02:00
Antoine R. Dumont (@ardumont)
9437a643ad
pytest: Define plugin and declare it in the root conftest 
Then drop all unneeded and indirect imports
 2020-09-02 12:25:15 +02:00
Antoine R. Dumont (@ardumont)
e99d3464e4
test_cli: Exclude launchpad lister from the check 
This should fix the build [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/77/console
 2020-09-01 15:55:24 +02:00