reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
949 commits 1 branch 0 tags 16 MiB
Commit graph

949 commits

This branch
This branch
All branches
Author SHA1 Message Date
David Douard
3ddfd00e90 Fix typos (and trailing ws) reported by codespell 2019-11-21 14:11:18 +01:00
David Douard
dfd8569c58 Add a pre-commit config file 2019-11-21 14:10:42 +01:00
Antoine R. Dumont (@ardumont)
1757b0112b
cran/gnu: Rename task_type to load-archive-files 2019-11-21 13:44:20 +01:00
Antoine R. Dumont (@ardumont)
1cf7c8e86b
lister.tests: Add missing task_type for package listers 
The scheduler module no longer initializes itself those task_type.
 2019-11-21 13:44:20 +01:00
Nicolas Dandrimont
5191d291da Migrate tox.ini to extras = xxx instead of deps = .[testing] 2019-11-21 13:36:33 +01:00
Nicolas Dandrimont
0459a0bb28 Merge tox environments 2019-11-21 13:33:42 +01:00
Nicolas Dandrimont
f9859f3025 Include all requirements in MANIFEST.in 2019-11-20 19:56:38 +01:00
Antoine R. Dumont (@ardumont)
484377cc13
lister.cli: Remove task type register cli 
It's now defined in swh.scheduler
 2019-11-18 10:41:46 +01:00
Antoine R. Dumont (@ardumont)
8d02458686
simple_lister: Flush to db more frequently 2019-11-15 11:48:09 +01:00
Antoine R. Dumont (@ardumont)
bf030c0f00
gnu.lister: Use url as primary key 
Otherwise, we are failing unicity constraint.

Related to T2070
 2019-11-15 11:24:21 +01:00
Antoine R. Dumont (@ardumont)
daa9a270fb
gnu.lister.tests: Add missing assertion 2019-11-15 10:49:13 +01:00
Antoine R. Dumont (@ardumont)
191043fff9
gnu.lister: Add missing retries_left parameter 2019-11-15 10:47:23 +01:00
Antoine R. Dumont (@ardumont)
d251201251
debian.models: Migrate tests from storage to debian lister model 
Related bb5d405
 2019-11-14 10:28:15 +01:00
Nicolas Dandrimont
b2e5ce32a9 Fix bogus NotImplementedError on Area.index_uris 2019-11-13 13:51:46 +01:00
Nicolas Dandrimont
773cd337f1 indexing lister: Avoid generating empty or duplicate ranges when partitioning 2019-11-12 17:54:51 +01:00
Nicolas Dandrimont
2c5528ef59 indexing lister: Force bounds of integer ranges to be integers 2019-11-12 17:52:13 +01:00
Nicolas Dandrimont
1e7a905a13 Support zero-based indexes in indexing_lister 2019-11-12 17:50:32 +01:00
Nicolas Dandrimont
e23960edc6 Add tests for IndexingLister.db_partition_indices function 2019-11-12 17:48:43 +01:00
Valentin Lorentz
8550c5607f Register lister tasks even if they do not derive from SWHTask. 
This happens if swh.scheduler.celery_backend.config isn't imported
before the CLI runs.
 2019-11-12 16:06:23 +01:00
Antoine R. Dumont (@ardumont)
ea7a08d05d
lister.debian: Actually use the db_engine passed to the hook function 2019-11-08 10:51:33 +01:00
Antoine R. Dumont (@ardumont)
eebbc859fc
lister.cli: Clarify configuration loading step 2019-11-08 10:50:51 +01:00
Antoine R. Dumont (@ardumont)
e8a67a7650
swh.lister: Remove completely references to swh.storage.schemata 
Related to 56d7cff
 2019-11-06 15:46:04 +01:00
Antoine R. Dumont (@ardumont)
81a31f3c06
tests: Bump dependency on latest swh-core 
This also modifies the test dataset to filename with url decoded filename.
As this is what the latest pytest plugin requires.
 2019-11-06 15:01:05 +01:00
Antoine R. Dumont (@ardumont)
e0dbca759c
lister.debian: Move run method parameters to constructor 2019-11-05 17:44:45 +01:00
Antoine R. Dumont (@ardumont)
b745c5a735
lister.debian: Default to run a listing on debian distribution 
That fixes the `swh lister run --lister debian` cli entrypoint.
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
a60e0bbc41
lister.debian: Fix task creation 
By adding a `retries_left`
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
f872792407
debian.lister: Send origin url as load-debian task parameter 
Instead of the old origin dict. That's what the debian loaders (old and new)
expect.
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
89b409d30f
gnu.lister: Update docstrings and fix type annotations 2019-11-04 11:06:16 +01:00
Antoine R. Dumont (@ardumont)
c6372eea7e
gnu.lister: Unify timestamp formats to isoformat date in model 
Related T2023
 2019-11-04 10:08:01 +01:00
Antoine R. Dumont (@ardumont)
93e7afda3b
gnu.lister: Format timestamp to isoformat string for the tar loader 
Related T2023
 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
9fd648987e
gnu.tree: Move helper utilities function to the bottom of the file 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
821f3f1cc2
lister.gnu: Move version parsing logic to the lister 
Related D2145 proposition
 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
7c247c8a4a
debian/lister: Use url parameter name instead of origin 
within the scheduled task.

Related D2135
 2019-11-04 10:00:55 +01:00
Antoine R. Dumont (@ardumont)
56d7cff6e1
debian/model: Install lister model within the lister repository 
This is no longer shared between the new debian loader and the lister.

The swh.storage.schemata module is still part of the swh.storage module though.
As this is still a dependency for the current swh.loader.debian production
loader. This will be cleaned up later.

Related D2135
 2019-11-04 10:00:54 +01:00
Stefano Zacchiroli
44bc4462e7 CRAN lister: fix compute_package_url interpolation 2019-10-28 15:50:23 +01:00
Stefano Zacchiroli
6159faa2f5 mypy: add typing annotations for novel lister abstractions 2019-10-28 15:35:21 +01:00
Stefano Zacchiroli
7dfd811e16 CRAN lister: make shelling out decoding compatible with Python 3.5 2019-10-28 15:35:21 +01:00
Stefano Zacchiroli
974f80f966 typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
Nicolas Dandrimont
78105940ff Stop binding tasks to a specific instance of the celery app 
The celery.shared_task decorator allows late-binding of tasks to any celery app,
which is well suited for our "task plugin" architecture.
 2019-10-18 18:02:25 +02:00
Antoine R. Dumont (@ardumont)
a64ae9641d
debian.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-15 12:21:24 +02:00
Antoine R. Dumont (@ardumont)
960868badb
pypi.tests: Remove trailing _ in test method name 2019-10-15 12:19:10 +02:00
Antoine R. Dumont (@ardumont)
b73c657ea7
npm.lister: Align docstrings 2019-10-15 11:19:48 +02:00
Antoine R. Dumont (@ardumont)
b4867ccda9
npm.tests: Add an integration test on listing with pagination 
Related T2032
 2019-10-15 10:49:29 +02:00
Antoine R. Dumont (@ardumont)
a8cde12d72
tests: Update pytest_plugin according to latest version change 2019-10-14 18:20:15 +02:00
Antoine R. Dumont (@ardumont)
9a1314c656
tox.ini: Install ipdb dependency for development environment 2019-10-14 14:47:57 +02:00
Antoine R. Dumont (@ardumont)
fcd8521622
tests/conftest: Log the db url used by tests 2019-10-14 14:47:56 +02:00
Antoine R. Dumont (@ardumont)
6b1c3d1fee
lister.core.db_utils: Remove dead code 2019-10-12 03:40:59 +02:00
Antoine R. Dumont (@ardumont)
f92ac83646
bitbucket.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:39:47 +02:00
Antoine R. Dumont (@ardumont)
0b8b1419e1
github.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:28:39 +02:00
Antoine R. Dumont (@ardumont)
1889875f67
gitlab.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:11:31 +02:00