reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
619 commits 1 branch 0 tags 16 MiB
Commit graph

619 commits

This branch
This branch
All branches
Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
20a91482ca
lister.gitlab.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
978fbbe029
lister.github.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
def0eb5060
lister.gitea.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
Antoine R. Dumont (@ardumont)
47ccce3817
lister.cran.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
Antoine R. Dumont (@ardumont)
5d4b38999d
lister.cgit.tests: Clarify lister configuration 2020-10-30 13:30:14 +01:00
Antoine R. Dumont (@ardumont)
c5af0efbf4
lister.bitbucket.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
f09b9f8cfa
lister.launchpad.tests: Clarify lister configuration 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b90ffa4bdd
tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
Antoine R. Dumont (@ardumont)
b35dff1266
Create listing task with a default of 3 if unspecified 
This will allow to retry task if they do not specify it.
 2020-10-30 09:10:27 +01:00
Antoine R. Dumont (@ardumont)
e2a861c801
debian.tests: Fix test 
The scheduler fixture introduced truncates tables in between tests. The debian
tests unfortunately share state and it broke when that changed. This fixes the
test by avoiding the truncation of the scheduler db table "task".

Ideally those tests need to be reworked to avoid sharing state between tests.

[1] https://jenkins.softwareheritage.org/job/DLS/job/tests/1043
 2020-10-30 09:09:56 +01:00
Antoine R. Dumont (@ardumont)
a19cb5fb51
lister.pytest_plugin: Simplify fixture setup 2020-10-21 17:46:52 +02:00
Antoine R. Dumont (@ardumont)
d7d38090f5
lister.config: Adapt scheduler configuration structure 2020-10-19 09:42:16 +02:00
Antoine R. Dumont (@ardumont)
30ad6200a2
Drop mock_get_scheduler which creates indirection for no good reason 
This is no longer useful, as removing it and tests are still ok.
 2020-10-16 14:32:46 +02:00
Antoine R. Dumont (@ardumont)
2ff92ef406
lister_base: Drop leftover mixin SWHConfig which is no longer used 
Related to D4178
 2020-10-07 14:00:57 +02:00
Antoine R. Dumont (@ardumont)
56f08b73f6
lister*: Migrate away from SWHConfig mixin 
Related to T1532 T1410 D3965
 2020-10-05 13:30:38 +02:00
Stefano Zacchiroli
4d6b99188e tox.ini: pin black to the pre-commit version (19.10b0) to avoid flip-flops 2020-10-02 16:23:33 +02:00
Nicolas Dandrimont
5e4bb28398 Run isort after the CLI import changes 2020-09-25 14:19:21 +02:00
David Douard
d10f78d80c Adapt cli declaration entrypoint to swh.core 0.3 2020-09-23 17:42:00 +02:00
Antoine Lambert
22f7181294 python: Reorder imports with isort 
Related to T2610
 2020-09-17 17:48:27 +02:00
Antoine Lambert
d24846a957 pre-commit: Add isort hook and configuration 
Related to T2610
 2020-09-17 17:48:25 +02:00
Antoine Lambert
cef98c9b9b pre-commit: Update flake8 hook configuration 
flake8 hook has been removed from https://github.com/pre-commit/pre-commit-hooks
so now use the one from https://gitlab.com/pycqa/flake8
 2020-09-17 13:55:01 +02:00
David Douard
a00f151462 cli: speedup the swh cli command startup time 
by moving import statements in functions.

Related to T2575.
 2020-09-10 17:15:38 +02:00
Antoine R. Dumont (@ardumont)
d1ce9b0912
tox/pytest: Activate doctest-module 
Related to D3899#96289
 2020-09-10 11:47:47 +02:00
Antoine R. Dumont (@ardumont)
31efda62e7
gitea.lister: Fix uid to be unique across instance 
The gitea lister can be run on multiple instances which could use the same id.
So listing another gitea instance, the current code would fail to insert data
for such case.

This commit fixes that behavior by prefixing the uid with the instance name.

Related to T2577
 2020-09-10 11:21:50 +02:00
Antoine R. Dumont (@ardumont)
e3c856b5ee
utils.split_range: Split into not overlapping ranges 
Existing listers use the `is_within_bound` [1] method from the base lister.
This method uses inclusive boundaries in all cases.

As some "range" task listers [2] [3] are using `split_range` function to create
"overlapping" ranges, this can cause concurrent insert issues down the line [4].

This commit adapts the function `split_range` to make the generated ranges no
longer overlap.

[1]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199

[2]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41

[3]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41

Related to T2577
 2020-09-10 11:01:44 +02:00
Antoine R. Dumont (@ardumont)
725c1fe4ad
test_utils: Migrate to pytest 2020-09-09 18:48:07 +02:00
Antoine R. Dumont (@ardumont)
66a61f3dd2
gitea.tasks: Fix parameter name from 'sort' to 'order' 
This fixes [1]

[1] https://sentry.softwareheritage.org/share/issue/b0119b56f24347bcb58ac28c68685c62/
 2020-09-09 12:10:23 +02:00
Antoine R. Dumont (@ardumont)
7d44bc2e75
launchpad.tasks: Update copyright headers 2020-09-08 14:42:37 +02:00
Vincent SELLIER
74ca3d0b87
Launchpad: rename task name to match conventions 
Related to T2358
 2020-09-08 14:21:47 +02:00
Antoine R. Dumont (@ardumont)
5a5b7ef70b
tests: Separate lister instantiations 
Prior to this commit, all listers were instantiated at the same time even if
only one was needed. This commit separates those instantiations.

The only drawback to this is the db model initialization which now happens at
each lister instantiation. This can be dealt with if needed at another time
though.
 2020-09-02 12:49:00 +02:00
Antoine R. Dumont (@ardumont)
92422dcf75
pytest_plugin: Instantiate only lister with no particular setup 
This should fix the remaining blocking problems in the jenkins build failure
report [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/78/consoleFull
 2020-09-02 12:25:15 +02:00
Antoine R. Dumont (@ardumont)
9437a643ad
pytest: Define plugin and declare it in the root conftest 
Then drop all unneeded and indirect imports
 2020-09-02 12:25:15 +02:00
Antoine R. Dumont (@ardumont)
e99d3464e4
test_cli: Exclude launchpad lister from the check 
This should fix the build [1]

[1] https://jenkins.softwareheritage.org/view/Debian%20packages/job/debian/job/packages/job/DLS/job/gbp-buildpackage/77/console
 2020-09-01 15:55:24 +02:00
Antoine R. Dumont (@ardumont)
250160ad75
launchpad: Add missing copyright headers 2020-09-01 15:53:26 +02:00
David Douard
a97613a056 Update the list of provided listers in the README file 2020-08-25 18:33:01 +02:00
David Douard
7ef7425d55 Keep plugins sorted in setup.py 2020-08-25 18:29:36 +02:00
Valentin Lorentz
dca617f32f Tell pytest not to recurse in dotdirs. 
pytest wastes a lot of time in .hypothesis and .git; this commit excludes them.
 2020-08-25 10:38:09 +02:00
Nicolas Dandrimont
211f4610df Move get_scheduler monkeypatching into an explicit pytest fixture 
This allows us to actually run the lister instantiation code instead of relying
on the underlying structure of the lister object. In turn, this allows future
listers to use the scheduler right in their __init__.
 2020-07-16 12:14:04 +02:00
Nicolas Dandrimont
d0c1df65f1 Only include relevant data files in MANIFEST.in 
The previous include would pull all .mypy_cache directories too, which are quite
large as they include the SQLAlchemy stubs.
 2020-07-16 12:14:04 +02:00
Nicolas Dandrimont
5f1fbbe8a4 Make sure LC_ALL is UTF-8 for pytest-postgresql to work 2020-07-15 16:17:07 +02:00
Nicolas Dandrimont
c9963d4302 Use the new names for the swh.scheduler test fixtures 2020-07-09 17:06:50 +02:00
Nicolas Dandrimont
014c446d05 Switch over to setuptools-scm 2020-06-25 12:12:47 +02:00
Léni Gauffier
1408517c08 Added GiteaLister 
Summary: Lister implementation for Gitea, works for (T2313). For now because of https://github.com/go-gitea/gitea/issues/9165 it would require setting its param limit to 50.

Reviewers: #reviewers, ardumont

Reviewed By: #reviewers, ardumont

Subscribers: ardumont

Differential Revision: https://forge.softwareheritage.org/D3107
 2020-06-10 17:04:28 +02:00
Stefano Zacchiroli
566294749e setup.py: add documentation link 2020-04-29 18:31:45 +02:00
Antoine R. Dumont (@ardumont)
d2daed02af
setup: Update the minimum required runtime python3 version 
Related to T2367
 2020-04-20 17:29:15 +02:00
Léni Gauffier
58ef08b083 Added LaunchpadLister 
Summary:
Related to T1734

From abandonned D2799

Reviewers: ardumont

Reviewed By: ardumont

Differential Revision: https://forge.softwareheritage.org/D2974
 2020-04-12 01:00:12 +02:00
David Douard
29325c1d94 Add a pyproject.toml file to target py37 for black 2020-04-08 22:16:53 +02:00
David Douard
93a4d8b784 Enable black 
- blackify all the python files,
- enable black in pre-commit,
- add a black tox environment.
 2020-04-08 16:31:22 +02:00
Antoine R. Dumont (@ardumont)
1ae75166c7
CONTRIBUTORS: Add Yann Gautier 2020-04-03 17:46:06 +02:00
Gautier Pugnonblanc Yann
e5fea84c55 review corrections 2020-02-20 09:13:49 +01:00