reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

bitbucket: Fix request to get next page of buggy page

Browse source 
The bitbucket Web API to list repositories has buggy pages that
needs to be skipped to continue the listing.

Previously the request to get the next page when a buggy page
is detected was missing the after query parameter so the request
was always returning the second page of repositories listing
endpoint.

Also refine buggy page detection by considering all HTTP status
code >= 500.
This commit is contained in:
Antoine Lambert 2025-01-22 11:49:28 +01:00
parent fc98bc1035
commit 3440881086
No known key found for this signature in database
GPG key ID: D026E5C2F802586D
2 changed files with 6 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

6
swh/lister/bitbucket/lister.py
View file

@ -124,16 +124,18 @@ class BitbucketLister(Lister[BitbucketListerState, List[Dict[str, Any]]]):
body = self.http_request(self.url, params=self.url_params).json()
yield body["values"]
except HTTPError as e:
if e.response is not None and e.response.status_code == 500:
if e.response is not None and e.response.status_code >= 500:
logger.warning(
"URL %s is buggy (error 500), skip it and get next page.",
"URL %s is buggy (error %s), skip it and get next page.",
e.response.url,
e.response.status_code,
)
body = self.http_request(
self.url,
params={
"pagelen": self.url_params["pagelen"],
"fields": "next",
"after": last_repo_cdate,
},
).json()

3
swh/lister/bitbucket/tests/test_lister.py
View file

@ -190,7 +190,8 @@ def test_bitbucket_lister_buggy_page(
BitbucketLister.API_URL,
[
{"json": bb_api_repositories_page1, "status_code": 200},
*[{"json": None, "status_code": 500}] * MAX_NUMBER_ATTEMPTS,
*[{"json": None, "status_code": 500}] * (MAX_NUMBER_ATTEMPTS - 1),
{"json": None, "status_code": 504},
{"json": {"next": bb_api_repositories_page1["next"]}, "status_code": 200},
{"json": bb_api_repositories_page2, "status_code": 200},
],