reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swh-lister/swh/lister
History
Antoine R. Dumont (@ardumont) 0ad37740d9
pattern: Make lister flush regularly origins to scheduler 
As origins is a generator, the previous behavior would try to consume the overall
generator to send the records.

This groups and sends batch of 100 origins to the scheduler for writing.

Related to T3003
2021-01-28 16:52:03 +01:00
..
bitbucket tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
cgit Make stateless lister constructors compatible with credentials 2021-01-28 14:42:56 +01:00
core Drop no longer swh.lister.core.{indexing,page_by_page}_lister 2021-01-27 15:42:57 +01:00
cran Make stateless lister constructors compatible with credentials 2021-01-28 14:42:56 +01:00
debian debian: Reimplement lister using new Lister API 2021-01-26 17:20:22 +01:00
gitea tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
github tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
gitlab gitlab: Deal with missing or trailing / in url input 2021-01-28 10:46:58 +01:00
gnu lister.gnu.tests: Clarify lister configuration 2020-10-30 13:30:15 +01:00
launchpad launchpad: Reimplement lister using new Lister API 2021-01-28 15:22:40 +01:00
npm tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
packagist python: Reorder imports with isort 2020-09-17 17:48:27 +02:00
phabricator tests: Remove no longer used conftest files 2021-01-26 17:09:04 +01:00
pypi Make stateless lister constructors compatible with credentials 2021-01-28 14:42:56 +01:00
tests launchpad: Reimplement lister using new Lister API 2021-01-28 15:22:40 +01:00
__init__.py Hook up listers implemented with the new pattern to the CLI 2021-01-11 11:00:29 +01:00
cli.py cli: Make cli work with new lister 2021-01-28 09:12:56 +01:00
pattern.py pattern: Make lister flush regularly origins to scheduler 2021-01-28 16:52:03 +01:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
pytest_plugin.py tests: Reduce db initialization fixtures to a minimum 2020-10-30 13:24:38 +01:00
utils.py lister: Add utility decorator to ease HTTP requests rate limit handling 2021-01-18 11:28:51 +01:00