lister: Make sure lister that requires github tokens can use it
Deploying the nixguix lister, I realized that even though the credentials configuration is properly set for all listers, the listers actually requiring github origin canonicalization do not have access to the github credentials. It's lost during the constructor to only focus on the lister's credentials. Which currently translates to listers being rate-limited. This commit fixes it by pushing the self.github_session instantiation in the constructor when the lister explicitely requires the github session. Hence lifting the rate limit for maven, packagist, nixguix, and github listers. Related to infra/sysadm-environment#4655
This commit is contained in:
parent
81688ca17e
commit
92d494261f
6 changed files with 20 additions and 28 deletions
|
@ -14,7 +14,6 @@ from bs4 import BeautifulSoup
|
|||
import lxml
|
||||
import requests
|
||||
|
||||
from swh.core.github.utils import GitHubSession
|
||||
from swh.scheduler.interface import SchedulerInterface
|
||||
from swh.scheduler.model import ListedOrigin
|
||||
|
||||
|
@ -88,15 +87,12 @@ class MavenLister(Lister[MavenListerState, RepoPage]):
|
|||
credentials=credentials,
|
||||
url=url,
|
||||
instance=instance,
|
||||
with_github_session=True,
|
||||
)
|
||||
|
||||
self.session.headers.update({"Accept": "application/json"})
|
||||
|
||||
self.jar_origins: Dict[str, ListedOrigin] = {}
|
||||
self.github_session = GitHubSession(
|
||||
credentials=self.credentials,
|
||||
user_agent=str(self.session.headers["User-Agent"]),
|
||||
)
|
||||
|
||||
def state_from_dict(self, d: Dict[str, Any]) -> MavenListerState:
|
||||
return MavenListerState(**d)
|
||||
|
@ -294,6 +290,7 @@ class MavenLister(Lister[MavenListerState, RepoPage]):
|
|||
return None
|
||||
|
||||
if url and visit_type == "git":
|
||||
assert self.github_session is not None
|
||||
# Non-github urls will be returned as is, github ones will be canonical ones
|
||||
url = self.github_session.get_canonical_url(url)
|
||||
|
||||
|
|
Loading…
Add table
Add a link
Reference in a new issue