reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
430 commits 1 branch 0 tags 16 MiB
Commit graph

303 commits

Author SHA1 Message Date
Antoine R. Dumont (@ardumont)
ea7a08d05d
lister.debian: Actually use the db_engine passed to the hook function 2019-11-08 10:51:33 +01:00
Antoine R. Dumont (@ardumont)
eebbc859fc
lister.cli: Clarify configuration loading step 2019-11-08 10:50:51 +01:00
Antoine R. Dumont (@ardumont)
e8a67a7650
swh.lister: Remove completely references to swh.storage.schemata 
Related to 56d7cff
 2019-11-06 15:46:04 +01:00
Antoine R. Dumont (@ardumont)
81a31f3c06
tests: Bump dependency on latest swh-core 
This also modifies the test dataset to filename with url decoded filename.
As this is what the latest pytest plugin requires.
 2019-11-06 15:01:05 +01:00
Antoine R. Dumont (@ardumont)
e0dbca759c
lister.debian: Move run method parameters to constructor 2019-11-05 17:44:45 +01:00
Antoine R. Dumont (@ardumont)
b745c5a735
lister.debian: Default to run a listing on debian distribution 
That fixes the `swh lister run --lister debian` cli entrypoint.
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
a60e0bbc41
lister.debian: Fix task creation 
By adding a `retries_left`
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
f872792407
debian.lister: Send origin url as load-debian task parameter 
Instead of the old origin dict. That's what the debian loaders (old and new)
expect.
 2019-11-05 10:35:51 +01:00
Antoine R. Dumont (@ardumont)
89b409d30f
gnu.lister: Update docstrings and fix type annotations 2019-11-04 11:06:16 +01:00
Antoine R. Dumont (@ardumont)
c6372eea7e
gnu.lister: Unify timestamp formats to isoformat date in model 
Related T2023
 2019-11-04 10:08:01 +01:00
Antoine R. Dumont (@ardumont)
93e7afda3b
gnu.lister: Format timestamp to isoformat string for the tar loader 
Related T2023
 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
9fd648987e
gnu.tree: Move helper utilities function to the bottom of the file 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
821f3f1cc2
lister.gnu: Move version parsing logic to the lister 
Related D2145 proposition
 2019-11-04 10:08:00 +01:00
Antoine R. Dumont (@ardumont)
7c247c8a4a
debian/lister: Use url parameter name instead of origin 
within the scheduled task.

Related D2135
 2019-11-04 10:00:55 +01:00
Antoine R. Dumont (@ardumont)
56d7cff6e1
debian/model: Install lister model within the lister repository 
This is no longer shared between the new debian loader and the lister.

The swh.storage.schemata module is still part of the swh.storage module though.
As this is still a dependency for the current swh.loader.debian production
loader. This will be cleaned up later.

Related D2135
 2019-11-04 10:00:54 +01:00
Stefano Zacchiroli
44bc4462e7 CRAN lister: fix compute_package_url interpolation 2019-10-28 15:50:23 +01:00
Stefano Zacchiroli
6159faa2f5 mypy: add typing annotations for novel lister abstractions 2019-10-28 15:35:21 +01:00
Stefano Zacchiroli
7dfd811e16 CRAN lister: make shelling out decoding compatible with Python 3.5 2019-10-28 15:35:21 +01:00
Stefano Zacchiroli
974f80f966 typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
Nicolas Dandrimont
78105940ff Stop binding tasks to a specific instance of the celery app 
The celery.shared_task decorator allows late-binding of tasks to any celery app,
which is well suited for our "task plugin" architecture.
 2019-10-18 18:02:25 +02:00
Antoine R. Dumont (@ardumont)
a64ae9641d
debian.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-15 12:21:24 +02:00
Antoine R. Dumont (@ardumont)
960868badb
pypi.tests: Remove trailing _ in test method name 2019-10-15 12:19:10 +02:00
Antoine R. Dumont (@ardumont)
b73c657ea7
npm.lister: Align docstrings 2019-10-15 11:19:48 +02:00
Antoine R. Dumont (@ardumont)
b4867ccda9
npm.tests: Add an integration test on listing with pagination 
Related T2032
 2019-10-15 10:49:29 +02:00
Antoine R. Dumont (@ardumont)
a8cde12d72
tests: Update pytest_plugin according to latest version change 2019-10-14 18:20:15 +02:00
Antoine R. Dumont (@ardumont)
fcd8521622
tests/conftest: Log the db url used by tests 2019-10-14 14:47:56 +02:00
Antoine R. Dumont (@ardumont)
6b1c3d1fee
lister.core.db_utils: Remove dead code 2019-10-12 03:40:59 +02:00
Antoine R. Dumont (@ardumont)
f92ac83646
bitbucket.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:39:47 +02:00
Antoine R. Dumont (@ardumont)
0b8b1419e1
github.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:28:39 +02:00
Antoine R. Dumont (@ardumont)
1889875f67
gitlab.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-12 03:11:31 +02:00
Antoine R. Dumont (@ardumont)
903b644c63
phabricator.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-11 15:30:39 +02:00
Antoine R. Dumont (@ardumont)
f3bf9ae50f
packagist.lister: Add integration test which checks scheduled tasks 
Related T2032
 2019-10-11 14:52:56 +02:00
Antoine R. Dumont (@ardumont)
678b7ea5bd
npm.lister: Add integration test which checks the scheduled tasks 
Related T2032
 2019-10-11 14:07:40 +02:00
Antoine R. Dumont (@ardumont)
599af25ad6
pypi.lister: Add integration test which checks the scheduled tasks 
Related T2032
 2019-10-11 13:24:55 +02:00
Antoine R. Dumont (@ardumont)
8d50e0d941
cran.lister: Fix cran lister and add proper integration test 
Which checks the cran lister tasks written in the scheduler.

Related d30d574dbe
Related 5ea9d5ed39

Related T2032
 2019-10-11 13:19:22 +02:00
Antoine R. Dumont (@ardumont)
ef2c1847e4
gnu.lister: Move tests datadir into its dedicated folder 
Relatd D2076#inline-13551
 2019-10-10 11:50:11 +02:00
Antoine R. Dumont (@ardumont)
0f0b840178
gnu.tests: Checks lister output from scheduler 
This also adds an swh-listers fixture which allows to retrieve a test ready
lister from its name (e.g gnu). Those listers have access to a scheduler
fixture so we can check the listing output from the scheduler instance.
 2019-10-09 18:23:51 +02:00
Antoine R. Dumont (@ardumont)
394658e53b
cgit.tests: Check the tasks from the scheduler 2019-10-09 17:57:57 +02:00
Antoine R. Dumont (@ardumont)
04ca318680
simple_lister: Extract common behavior in base class 2019-10-09 17:35:12 +02:00
Antoine R. Dumont (@ardumont)
61ce38a0b0
core.models: Fix typo 2019-10-09 17:35:12 +02:00
Antoine R. Dumont (@ardumont)
3ce6c5c6ef
lister/gnu: Modify gnu lister's loading task creation 
Loader Task signature for the loader gnu is now:
- args:
  - package
  - package urls

- kwargs:
  tarballs: List of Dict with keys archive (unchanged), 'time' (was 'date'),
      length (new)
 2019-10-04 11:05:58 +02:00
Antoine R. Dumont (@ardumont)
00bb6c7bbf
lister/gnu: Remove unneeded get_file method 2019-10-04 11:02:17 +02:00
Antoine R. Dumont (@ardumont)
322c9dc7e2
lister/gnu: Modify default policy to oneshot 2019-10-04 11:01:49 +02:00
Antoine R. Dumont (@ardumont)
d30d574dbe
cran.lister: Refactor and fix cran lister 
Prior to this commit, the code was actually duplicated with an old version
which would not work.

Related D1492#41287
 2019-10-02 11:06:59 +02:00
Antoine Lambert
04d8fdf8df github/lister: Prevent erroneous scheduler tasks disabling 
Closes T2014
 2019-09-19 14:30:30 +02:00
Antoine Lambert
7572228f7c listers: Ensure run can be called without bounds arguments 
Closes T2001
 2019-09-17 15:09:04 +02:00
Antoine Lambert
4c8d7baf75 phabricator/lister: Prevent erroneous scheduler tasks disabling 
Previously, the Phabricator lister was disabling some loading tasks while it was not
supposed to. More precisely, due to an invalid index provided to a database query,
the latest created scheduler task was disabled each time a new page of results was
provided to the lister by the Phabricator API. Moreover, database queries were not
filtered according to the Phabricator instance resulting in possible disabling of
scheduler tasks from other instances.

Closes T2000
 2019-09-16 20:05:48 +02:00
Antoine Lambert
e83902c2a3 phabricator/lister: Fix get_next_target_from_response return type 
Without that fix, errors are raised when one wants to list Phabricator repositories
in a specific index range. The issue is due to a comparison between a string and
an integer. So convert next extracted repository index to integer to match the
corresponding model type.

Closes T1997
 2019-09-16 13:36:46 +02:00
Antoine Lambert
1ebe762ea6 phabricator/lister: Do not override max_index when bootstrapping 
Turns out all newly listed repositories were filtered out because of that.
Consequently, no entries in the listers database and no scheduler loading
tasks were created when listing a Phabricator instance.

Closes T1999
 2019-09-16 13:34:22 +02:00
Antoine Lambert
7c8f4dc9a8 packagist/lister: Fix typos in docstring 2019-09-12 20:46:42 +02:00