lister: Make sure lister that requires github tokens can use it

Deploying the nixguix lister, I realized that even though the credentials configuration
is properly set for all listers, the listers actually requiring github origin
canonicalization do not have access to the github credentials. It's lost during the
constructor to only focus on the lister's credentials. Which currently translates to
listers being rate-limited.

This commit fixes it by pushing the self.github_session instantiation in the constructor
when the lister explicitely requires the github session. Hence lifting the rate limit
for maven, packagist, nixguix, and github listers.

Related to infra/sysadm-environment#4655
This commit is contained in:
Antoine R. Dumont (@ardumont) 2022-10-26 17:23:40 +02:00
parent 81688ca17e
commit 92d494261f
No known key found for this signature in database
GPG key ID: 52E2E9840D10C3B8
6 changed files with 20 additions and 28 deletions

View file

@ -29,7 +29,6 @@ from urllib.parse import parse_qsl, urlparse
import requests
from requests.exceptions import ConnectionError, InvalidSchema, SSLError
from swh.core.github.utils import GitHubSession
from swh.core.tarball import MIMETYPE_TO_ARCHIVE_FORMAT
from swh.lister import TARBALL_EXTENSIONS
from swh.lister.pattern import CredentialsType, StatelessLister
@ -331,6 +330,7 @@ class NixGuixLister(StatelessLister[PageResult]):
url=url.rstrip("/"),
instance=instance,
credentials=credentials,
with_github_session=canonicalize,
)
# either full fqdn NixOS/nixpkgs or guix repository urls
# maybe add an assert on those specific urls?
@ -338,16 +338,6 @@ class NixGuixLister(StatelessLister[PageResult]):
self.extensions_to_ignore = DEFAULT_EXTENSIONS_TO_IGNORE + extensions_to_ignore
self.session = requests.Session()
# for testing purposes, we may want to skip this step (e.g. docker run and rate
# limit)
self.github_session = (
GitHubSession(
credentials=self.credentials,
user_agent=str(self.session.headers["User-Agent"]),
)
if canonicalize
else None
)
def build_artifact(
self, artifact_url: str, artifact_type: str, artifact_ref: Optional[str] = None