reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

utils.split_range: Split into not overlapping ranges

Browse source 
Existing listers use the `is_within_bound` [1] method from the base lister.
This method uses inclusive boundaries in all cases.

As some "range" task listers [2] [3] are using `split_range` function to create
"overlapping" ranges, this can cause concurrent insert issues down the line [4].

This commit adapts the function `split_range` to make the generated ranges no
longer overlap.

[1]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/core/lister_base.py$194-199

[2]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/tasks.py$37-41

[3]
https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitea/tasks.py$36-41

Related to T2577
This commit is contained in:
Antoine R. Dumont (@ardumont) 2020-09-09 18:50:46 +02:00
parent 725c1fe4ad
commit e3c856b5ee
No known key found for this signature in database
GPG key ID: 52E2E9840D10C3B8
4 changed files with 54 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

21
swh/lister/utils.py
View file

@ -1,13 +1,28 @@
# Copyright (C) 2018 the Software Heritage developers
# Copyright (C) 2018-2020 the Software Heritage developers
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from typing import Iterator, Tuple
def split_range(total_pages, nb_pages):
def split_range(total_pages: int, nb_pages: int) -> Iterator[Tuple[int, int]]:
"""Split `total_pages` into mostly `nb_pages` ranges. In some cases, the last range can
have one more element.
>>> split_range(19, 10)
[(0, 9), (10, 19)]
>>> split_range(20, 3)
[(0, 2), (3, 5), (6, 8), (9, 11), (12, 14), (15, 17), (18, 20)]
>>> split_range(21, 3)
[(0, 2), (3, 5), (6, 8), (9, 11), (12, 14), (15, 17), (18, 21)]
"""
prev_index = None
for index in range(0, total_pages, nb_pages):
if index is not None and prev_index is not None:
yield prev_index, index
yield prev_index, index - 1
prev_index = index
if index != total_pages: