swh-lister/swh/lister
Antoine Lambert a8871bd492 save_bulk: Speedup listing process with multi-threading
Check multiple origins in parallel using the concurrent.futures
module to greatly speedup the whole listing process.

Related to #4709
2024-10-29 11:18:33 +01:00
..
arch Use beautifulsoup4 CSS selectors to simplify code and type checking 2024-04-16 11:22:51 +02:00
aur python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
bioconductor Use beautifulsoup4 CSS selectors to simplify code and type checking 2024-04-16 11:22:51 +02:00
bitbucket tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
bower Harmonize listers parameters and add test to check mandatory ones 2023-09-06 11:55:34 +02:00
cgit Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
conda Harmonize listers parameters and add test to check mandatory ones 2023-09-06 11:55:34 +02:00
cpan python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
cran Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
crates crates: Remove crates metadata as loader argument 2024-08-27 12:28:05 +02:00
debian Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
dlang Remove spurious space 2023-09-21 09:18:54 +02:00
elm Elm stateful lister 2024-01-09 14:05:56 +01:00
gitea gogs, gitea: Fix task execution to pass along extra kwargs 2022-12-14 16:09:56 +01:00
github github: Ensure range listers do not override shared lister state 2024-10-28 15:37:02 +00:00
gitiles tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
gitlab tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
gitweb tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
gnu Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
gogs gitea, gogs: Ensure query parameters are not duplicated in API URLs 2024-06-05 15:27:58 +02:00
golang Make qa tools happy again 2024-08-27 17:40:30 +02:00
hackage pattern: Ensure termination date is set at the end of listing process 2024-10-24 12:33:40 +02:00
hex python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
julia Stateful Julia lister 2023-12-18 16:02:22 +01:00
launchpad Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
maven tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
nixguix Move tarball validation functions from nixguix to utils 2024-09-02 11:29:47 +02:00
npm tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
nuget Use beautifulsoup4 CSS selectors to simplify code and type checking 2024-04-16 11:22:51 +02:00
opam opam: Fix 'opam init' error when relisting an opam instance 2023-06-29 17:49:21 +02:00
packagist Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
pagure pagure/tasks: Add missing docstring for list_pagure task function 2023-06-23 14:29:17 +02:00
phabricator tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
pubdev python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
puppet python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
pypi Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
rpm python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
rubygems Use beautifulsoup4 CSS selectors to simplify code and type checking 2024-04-16 11:22:51 +02:00
save_bulk save_bulk: Speedup listing process with multi-threading 2024-10-29 11:18:33 +01:00
sourceforge sourceforge: Also skip ConnectionError when fetching project info 2024-09-05 14:52:56 +02:00
stagit tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
tests Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
tuleap tests: Fix mocking of sleep calls with tenacity 8.4.2 2024-06-28 18:15:36 +02:00
__init__.py Replace all remaining occurrences of the 'local' cls by 'postgresql' 2024-10-28 14:35:29 +01:00
cli.py cli: Print lister stats at the end of the run command 2023-11-07 19:00:53 +01:00
pattern.py pattern: Ensure termination date is set at the end of listing process 2024-10-24 12:33:40 +02:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
utils.py Move tarball validation functions from nixguix to utils 2024-09-02 11:29:47 +02:00