reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

pattern: Ensure accurate origin counts returned by run method

Browse source 
Previously, the run method was returning the total count of ListedOrigin
objects sent to scheduler database.

However, some listers can send multiple ListedOrigin objects for a given
origin URL during the listing process, for instance when an origin is
contained in multiple pages (e.g. gogs listing) or when the listing
is gathering multiple versions of an origin spread across multiple
pages (e.g. maven listing).

This changes ensures an accurate count of listed origins by maintaining
a set of origin URLs associated to the sent ListedOrigin objects.
This commit is contained in:
Antoine Lambert 2022-09-29 11:14:08 +02:00
parent 3928fc9ee9
commit 8d85b2e4e8
4 changed files with 32 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

3
swh/lister/arch/tests/test_lister.py
View file

@ -2,6 +2,7 @@
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
from swh.lister.arch.lister import ArchLister
expected_origins = [
@ -1371,7 +1372,7 @@ def test_arch_lister(datadir, requests_mock_datadir, swh_scheduler):
res = lister.run()
assert res.pages == 9
assert res.origins == 12
assert res.origins == 11
scheduler_origins = swh_scheduler.get_listed_origins(lister.lister_obj.id).results