lister: Make sure lister that requires github tokens can use it

Deploying the nixguix lister, I realized that even though the credentials configuration
is properly set for all listers, the listers actually requiring github origin
canonicalization do not have access to the github credentials. It's lost during the
constructor to only focus on the lister's credentials. Which currently translates to
listers being rate-limited.

This commit fixes it by pushing the self.github_session instantiation in the constructor
when the lister explicitely requires the github session. Hence lifting the rate limit
for maven, packagist, nixguix, and github listers.

Related to infra/sysadm-environment#4655
This commit is contained in:
Antoine R. Dumont (@ardumont) 2022-10-26 17:23:40 +02:00
parent 81688ca17e
commit 92d494261f
No known key found for this signature in database
GPG key ID: 52E2E9840D10C3B8
6 changed files with 20 additions and 28 deletions

View file

@ -11,7 +11,6 @@ from typing import Any, Dict, Iterator, List, Optional
import iso8601
import requests
from swh.core.github.utils import GitHubSession
from swh.scheduler.interface import SchedulerInterface
from swh.scheduler.model import ListedOrigin
@ -60,14 +59,11 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
url=self.PACKAGIST_PACKAGES_LIST_URL,
instance="packagist",
credentials=credentials,
with_github_session=True,
)
self.session.headers.update({"Accept": "application/json"})
self.listing_date = datetime.now().astimezone(tz=timezone.utc)
self.github_session = GitHubSession(
credentials=self.credentials,
user_agent=str(self.session.headers["User-Agent"]),
)
def state_from_dict(self, d: Dict[str, Any]) -> PackagistListerState:
last_listing_date = d.get("last_listing_date")
@ -152,6 +148,7 @@ class PackagistLister(Lister[PackagistListerState, PackagistPageType]):
if visit_type == "git":
# Non-github urls will be returned as is, github ones will be canonical
# ones
assert self.github_session is not None
origin_url = (
self.github_session.get_canonical_url(origin_url) or origin_url
)