lister: Make sure lister that requires github tokens can use it

Deploying the nixguix lister, I realized that even though the credentials configuration
is properly set for all listers, the listers actually requiring github origin
canonicalization do not have access to the github credentials. It's lost during the
constructor to only focus on the lister's credentials. Which currently translates to
listers being rate-limited.

This commit fixes it by pushing the self.github_session instantiation in the constructor
when the lister explicitely requires the github session. Hence lifting the rate limit
for maven, packagist, nixguix, and github listers.

Related to infra/sysadm-environment#4655
This commit is contained in:
Antoine R. Dumont (@ardumont) 2022-10-26 17:23:40 +02:00
parent 81688ca17e
commit 92d494261f
No known key found for this signature in database
GPG key ID: 52E2E9840D10C3B8
6 changed files with 20 additions and 28 deletions

View file

@ -14,7 +14,6 @@ from bs4 import BeautifulSoup
import lxml
import requests
from swh.core.github.utils import GitHubSession
from swh.scheduler.interface import SchedulerInterface
from swh.scheduler.model import ListedOrigin
@ -88,15 +87,12 @@ class MavenLister(Lister[MavenListerState, RepoPage]):
credentials=credentials,
url=url,
instance=instance,
with_github_session=True,
)
self.session.headers.update({"Accept": "application/json"})
self.jar_origins: Dict[str, ListedOrigin] = {}
self.github_session = GitHubSession(
credentials=self.credentials,
user_agent=str(self.session.headers["User-Agent"]),
)
def state_from_dict(self, d: Dict[str, Any]) -> MavenListerState:
return MavenListerState(**d)
@ -294,6 +290,7 @@ class MavenLister(Lister[MavenListerState, RepoPage]):
return None
if url and visit_type == "git":
assert self.github_session is not None
# Non-github urls will be returned as is, github ones will be canonical ones
url = self.github_session.get_canonical_url(url)