Antoine Lambert
4245c5046f
Remove no longer used models field in dict returned by register
2021-02-02 16:33:52 +01:00
tenma
6cd31769c1
tests: Remove no longer used conftest files
...
All the fixtures declared in them are not used anymore in the
tests of the listers ported to the new Lister API.
2021-01-26 17:09:04 +01:00
Antoine Lambert
ea8ecee541
tests: Fix errors after swh-scheduler API update
...
The PaginatedListedOriginList model has been updated in
rDSCHb93aa5be2c2d5dc2130e1027698f3e1255052d8d and the origins
field has been renamed to results.
2021-01-25 17:11:54 +01:00
tenma
b6a69b2ed9
gitea.lister: improve handling of credentials
2021-01-25 15:54:06 +01:00
tenma
c220d7d299
gitea.tests: split and make them more thorough
2021-01-25 15:54:06 +01:00
tenma
c780ad4b44
Reimplement Gitea lister using new Lister API
...
The lister is stateless and has full listing capability.
It can request the Gitea API using HTTP token authentication.
Rate-limiting was not encountered but is handled generically.
Added support for getting repo last update date through API.
2021-01-25 15:54:06 +01:00
Antoine R. Dumont (@ardumont)
def0eb5060
lister.gitea.tests: Clarify lister configuration
2020-10-30 13:30:15 +01:00
Antoine Lambert
22f7181294
python: Reorder imports with isort
...
Related to T2610
2020-09-17 17:48:27 +02:00
Antoine R. Dumont (@ardumont)
31efda62e7
gitea.lister: Fix uid to be unique across instance
...
The gitea lister can be run on multiple instances which could use the same id.
So listing another gitea instance, the current code would fail to insert data
for such case.
This commit fixes that behavior by prefixing the uid with the instance name.
Related to T2577
2020-09-10 11:21:50 +02:00
Antoine R. Dumont (@ardumont)
e3c856b5ee
utils.split_range: Split into not overlapping ranges
...
Existing listers use the `is_within_bound` [1] method from the base lister.
This method uses inclusive boundaries in all cases.
As some "range" task listers [2] [3] are using `split_range` function to create
"overlapping" ranges, this can cause concurrent insert issues down the line [4].
This commit adapts the function `split_range` to make the generated ranges no
longer overlap.
[1]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199
[2]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41
[3]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41
Related to T2577
2020-09-10 11:01:44 +02:00
Antoine R. Dumont (@ardumont)
66a61f3dd2
gitea.tasks: Fix parameter name from 'sort' to 'order'
...
This fixes [1]
[1] https://sentry.softwareheritage.org/share/issue/b0119b56f24347bcb58ac28c68685c62/
2020-09-09 12:10:23 +02:00
Antoine R. Dumont (@ardumont)
5a5b7ef70b
tests: Separate lister instantiations
...
Prior to this commit, all listers were instantiated at the same time even if
only one was needed. This commit separates those instantiations.
The only drawback to this is the db model initialization which now happens at
each lister instantiation. This can be dealt with if needed at another time
though.
2020-09-02 12:49:00 +02:00
Antoine R. Dumont (@ardumont)
9437a643ad
pytest: Define plugin and declare it in the root conftest
...
Then drop all unneeded and indirect imports
2020-09-02 12:25:15 +02:00
Nicolas Dandrimont
c9963d4302
Use the new names for the swh.scheduler test fixtures
2020-07-09 17:06:50 +02:00
Léni Gauffier
1408517c08
Added GiteaLister
...
Summary: Lister implementation for Gitea, works for (T2313). For now because of https://github.com/go-gitea/gitea/issues/9165 it would require setting its param limit to 50.
Reviewers: #reviewers, ardumont
Reviewed By: #reviewers, ardumont
Subscribers: ardumont
Differential Revision: https://forge.softwareheritage.org/D3107
2020-06-10 17:04:28 +02:00
