reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

crates: Bump csv field size limit

Browse source 
A size limit of 1000000 was not enough to properly process
all CSV crates data so bump to a higher value.
This commit is contained in:
Antoine Lambert 2024-08-21 13:46:03 +02:00
parent f7abfafffe
commit b2ece7ca63
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
swh/lister/crates/lister.py
View file

@ -131,7 +131,7 @@ class CratesLister(Lister[CratesListerState, CratesListerPage]):
members_to_extract.append(member)
tf.extractall(members=members_to_extract, path=db_dump_path)
csv.field_size_limit(1000000)
csv.field_size_limit(10000000)
(crates_csv_path,) = list(db_dump_path.glob("*/data/crates.csv"))
(versions_csv_path,) = list(db_dump_path.glob("*/data/versions.csv"))