reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
swh-lister/swh/lister
History
Antoine Lambert 8d85b2e4e8 pattern: Ensure accurate origin counts returned by run method 
Previously, the run method was returning the total count of ListedOrigin
objects sent to scheduler database.

However, some listers can send multiple ListedOrigin objects for a given
origin URL during the listing process, for instance when an origin is
contained in multiple pages (e.g. gogs listing) or when the listing
is gathering multiple versions of an origin spread across multiple
pages (e.g. maven listing).

This changes ensures an accurate count of listed origins by maintaining
a set of origin URLs associated to the sent ListedOrigin objects.
2022-09-29 11:14:08 +02:00
..
arch pattern: Ensure accurate origin counts returned by run method 2022-09-29 11:14:08 +02:00
aur Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
bitbucket Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
bower Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
cgit cgit/tests: Rename readme.md to readme 2022-09-26 13:22:10 +02:00
conda pattern: Ensure accurate origin counts returned by run method 2022-09-29 11:14:08 +02:00
cpan Cpan: List Perl module origins from cpan.org 2022-09-27 14:29:33 +02:00
cran python: Reformat code with black 22.3.0 2022-04-08 15:15:09 +02:00
crates tests: Simplify origin comparison and improve pytest diff on failure 2022-08-24 17:21:24 +02:00
debian Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
gitea Use generic HTTP retry policy by default and rename dedicated decorator 2022-09-26 10:48:40 +02:00
github Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
gitlab Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
gnu python: Reformat code with black 22.3.0 2022-04-08 15:15:09 +02:00
gogs Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
golang Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
hackage Hackage: List origins from hackage.haskell.org, The Haskell Package Repository 2022-09-27 14:22:03 +02:00
launchpad Use generic HTTP retry policy by default and rename dedicated decorator 2022-09-26 10:48:40 +02:00
maven Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
npm Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
nuget Nuget: Lister for NuGet the package manager for .NET 2022-09-27 14:56:36 +02:00
opam python: Reformat code with black 22.3.0 2022-04-08 15:15:09 +02:00
packagist Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
phabricator Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
pubdev Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
puppet Puppet: Lister for Puppet modules 2022-09-27 14:44:13 +02:00
pypi Use generic HTTP retry policy by default and rename dedicated decorator 2022-09-26 10:48:40 +02:00
sourceforge Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
tests pattern: Ensure accurate origin counts returned by run method 2022-09-29 11:14:08 +02:00
tuleap Refactor and deduplicate HTTP requests code in listers 2022-09-26 10:48:40 +02:00
__init__.py Update value of User-Agent HTTP request header used by listers 2022-09-26 10:48:40 +02:00
cli.py python: Reformat code with black 22.3.0 2022-04-08 15:15:09 +02:00
pattern.py pattern: Ensure accurate origin counts returned by run method 2022-09-29 11:14:08 +02:00
py.typed typing: minimal changes to make a no-op mypy run pass 2019-10-28 15:35:21 +01:00
utils.py Use generic HTTP retry policy by default and rename dedicated decorator 2022-09-26 10:48:40 +02:00