reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Commit graph

  • 11073e0dae
         add tests master reonokiy 2025-05-24 02:46:32 +08:00
  • 72d0bf21bc
         fix metadata key error reonokiy 2025-05-24 02:36:22 +08:00
  • d5d56e3a16
         add fdroid lister reonokiy 2025-05-18 16:37:02 +08:00
  • 213a4a152f
         crates: Bump chunk size when downloading database dump Antoine Lambert 2025-04-15 12:15:03 +02:00
  • ceb1b6450e gnu: Fix KeyError exception due to missing field in JSON data Antoine Lambert 2025-03-03 14:31:21 +01:00
  • 41c13438b4 Use swh-scheduler[pytest] instead of swh-scheduler[testing] Nicolas Dandrimont 2025-03-28 19:12:10 +01:00
  • 08fda328be Migration to psycopg3 Pierre-Yves David 2024-12-23 16:44:29 +01:00
  • 61cfd77da1
         debian: Fix error since python-debian 1.0 release Antoine Lambert 2025-03-13 11:30:50 +01:00
  • f2f9c7d19e Migrate from deprecated pkg_resources package to importlib.metadata Antoine Lambert 2025-02-25 20:28:52 +01:00
  • 6b4f84a384
         packagist: Fix mypy error after typing added to grouper Antoine Lambert 2025-02-25 10:54:12 +01:00
  • bde37867d8 docs: Fix broken external links Antoine Lambert 2025-02-06 14:33:25 +01:00
  • 3771a411ae
         tests: Remove no longer needed pytest custom marker named db Antoine Lambert 2025-02-17 16:29:09 +01:00
  • db00f23ec0
         save_bulk/test_lister: Fix flake8 warnings Antoine Lambert 2025-02-14 13:33:32 +01:00
  • 4b3a12fe76
         maven, sourceforge: Fix mypy errors Antoine Lambert 2025-02-14 13:32:26 +01:00
  • edef3b850c
         Apply swh-py-template v0.3.3 with copier Antoine Lambert 2025-02-14 13:32:03 +01:00
  • a3d66736a4
         maven: Update test that is now failing since beautifulsoup4 4.13 Antoine Lambert 2025-02-10 14:26:42 +01:00
  • 4d14e8928b Remove no longer used sql directory Antoine Lambert 2025-01-22 11:29:47 +01:00
  • 3440881086
         bitbucket: Fix request to get next page of buggy page Antoine Lambert 2025-01-22 11:49:28 +01:00
  • fc98bc1035 cli: Replace scheduler temporary backend by memory one Antoine Lambert 2024-12-11 12:02:46 +01:00
  • 63ca5b50a0 sourceforge: Catch correct ConnectionError exception Antoine Lambert 2024-11-07 14:25:29 +01:00
  • a8871bd492 save_bulk: Speedup listing process with multi-threading Antoine Lambert 2024-10-25 14:06:08 +02:00
  • 88a715d0c1 github: Ensure range listers do not override shared lister state Antoine Lambert 2024-10-25 16:48:22 +02:00
  • cccb8c21ff Replace all remaining occurrences of the 'local' cls by 'postgresql' David Douard 2024-10-28 14:30:11 +01:00
  • eadb704494 pattern: Ensure termination date is set at the end of listing process Antoine Lambert 2024-10-24 12:33:20 +02:00
  • 99f64ddbff save_bulk: Ensure high priority scheduling for first visits of origins Antoine Lambert 2024-10-09 19:06:30 +02:00
  • 0e1093e308 pattern: Add first_visits_queue_prefix parameter to Lister constructor Antoine Lambert 2024-10-09 19:00:57 +02:00
  • 7609ebf7e1 pattern: Store termination date to scheduler database at end of listing Antoine Lambert 2024-10-09 18:32:23 +02:00
  • 927aebbd0b sourceforge: Also skip ConnectionError when fetching project info Antoine Lambert 2024-09-03 12:04:12 +02:00
  • af24960bc2 Add save-bulk lister to check origins prior their insertion in database Antoine Lambert 2024-05-22 17:42:56 +02:00
  • 6618cf341c Move tarball validation functions from nixguix to utils Antoine Lambert 2024-05-23 14:17:46 +02:00
  • c0dc8edb05 Make qa tools happy again David Douard 2024-08-27 17:40:30 +02:00
  • c6baacbcd7 Apply swh-py-template v0.2.3 David Douard 2024-08-27 16:25:53 +02:00
  • 5003e6588f crates: Remove crates metadata as loader argument Antoine Lambert 2024-08-23 11:46:24 +02:00
  • 42e76ee62e crates: Speedup listing by processing crates in batch Antoine Lambert 2024-08-22 16:38:44 +02:00
  • c6aa490fc1 crates: Record lister state only if all crates were processed Antoine Lambert 2024-08-21 16:12:37 +02:00
  • aafaebd5de crates: Use looseversion.LooseVersion2 to parse crate versions Antoine Lambert 2024-08-21 15:17:28 +02:00
  • b2ece7ca63 crates: Bump csv field size limit Antoine Lambert 2024-08-21 13:46:03 +02:00
  • f7abfafffe GitHub: record whether the origin is a fork Nicolas Dandrimont 2024-07-17 18:26:57 +02:00
  • a7607abcf9 tests: Fix mocking of sleep calls with tenacity 8.4.2 Antoine Lambert 2024-06-27 11:40:58 +02:00
  • 323e277482 gitea, gogs: Ensure query parameters are not duplicated in API URLs Antoine Lambert 2024-06-05 15:14:07 +02:00
  • aaae1a6b0b launchpad, npm: Port code to updated swh-scheduler API Antoine Lambert 2024-05-03 13:52:02 +02:00
  • e51b808d72 nixguix: Ensure to not use a redirection URL as an origin URL Antoine Lambert 2024-04-24 14:25:39 +02:00
  • 41407e0eff Use beautifulsoup4 CSS selectors to simplify code and type checking Antoine Lambert 2024-04-15 16:58:46 +02:00
  • e6a35c55b0 Apply swh-py-template v0.2.0 David Douard 2024-03-29 13:55:23 +01:00
  • fdeb086f77 nixguix: Handle creation of svn-export visit types on svn sub-trees Antoine Lambert 2024-03-12 17:53:15 +01:00
  • b083b4f1f9 pytest: Fix tests execution with pytest 8.1 Antoine Lambert 2024-03-13 10:58:03 +01:00
  • 329cb2e44a requirements-test: Add missing swh-scheduler[testing] dependency Antoine Lambert 2024-03-13 10:56:47 +01:00
  • 32be94a89b tox: Bump mypy to 1.8.0 Antoine Lambert 2024-02-05 16:14:17 +01:00
  • 65e51e2925 nixguix: Update heuristic checking if URL targets a tarball file Antoine Lambert 2024-01-18 14:33:39 +01:00
  • ed8de05eea Remove the outdated list of swh.lister submodules from the readme David Douard 2024-01-17 18:05:58 +01:00
  • d70dd84939 Fix the listing of listers Jérémy Bobbio (Lunar) 2023-12-21 17:37:50 +01:00
  • 82ee095128 Elm stateful lister Franck Bret 2023-12-21 09:54:29 +01:00
  • 4b1f49ac22 Adapt and rebase Franck Bret 2023-12-18 12:02:46 +01:00
  • 3a1beae36e Elm Lister Franck Bret 2023-07-24 09:34:04 +02:00
  • f814e1179d nixguix: Exploit new submodule info in sources.json from Guix Antoine Lambert 2024-01-08 15:56:16 +01:00
  • 99bbd9d68f Stateful Julia lister Franck Bret 2023-10-13 10:49:26 +02:00
  • 053f0a93d5 Add latest blackify to git-blame-ignore-revs David Douard 2023-12-05 14:04:51 +01:00
  • 714fccc3c7 python: Fix black formatting after bump to 23.1.0 in pre-commit David Douard 2023-12-05 10:33:07 +01:00
  • ac52cfed21 Apply swh-py-template 0.1.6 David Douard 2023-12-03 17:38:55 +01:00
  • e4c707d807 pytest.ini: Ensure '--import-mode importlib' option is always used Antoine Lambert 2023-12-01 14:42:34 +01:00
  • c2402f405f Migrate to copier-based swh-py-template David Douard 2023-11-24 18:15:13 +01:00
  • 553884fa56 docs: include the README file in the main index page David Douard 2023-11-16 13:44:04 +01:00
  • a9b2980f14 Fix pygment language declaration in the README file David Douard 2023-11-15 17:29:21 +01:00
  • 4bcf4a4147 swh-core's github extra isn't needed anymore Nicolas Dandrimont 2023-11-14 15:58:15 +01:00
  • 4aee4da784 cran: Use pyreadr instead of rpy2 to read a RDS file from Python Antoine Lambert 2023-08-17 16:52:39 +02:00
  • 42d8e24d7e
         arch/lister: Drop artifact size approximation from the listing Antoine Lambert 2023-06-22 14:41:07 +02:00
  • 2eb3223496 cli: Print lister stats at the end of the run command Antoine Lambert 2023-06-22 14:41:07 +02:00
  • 7092e4e4ac cli: Use temporary scheduler as fallback when no configuration detected Antoine Lambert 2023-06-22 14:36:13 +02:00
  • 7344d264e7 Ensure HTTPError.response is not None Jérémy Bobbio (Lunar) 2023-10-18 10:14:55 +02:00
  • 968ddef295 Improve registry repository management Franck Bret 2023-10-12 14:31:48 +02:00
  • 360fa753ef Remove useless triple single quote from bash script Franck Bret 2023-10-09 15:15:21 +02:00
  • 7f97c2da67 Use a temp directory instead of /tmp Franck Bret 2023-09-21 11:30:42 +02:00
  • 1984037fe1 Replace obsolete comment, improve docstring Franck Bret 2023-09-21 11:07:22 +02:00
  • 3e414c5397 url and instance are now mandatory (related #501) Franck Bret 2023-09-21 10:56:50 +02:00
  • f8cfa05f3f Add Julia Lister for listing Julia Packages Franck Bret 2023-07-20 17:18:26 +02:00
  • 7b932f46b5 gitweb: Add optional base_git_url parameter to lister Antoine Lambert 2023-09-27 14:56:47 +02:00
  • 59a979642f gitweb: Ensure to strip any prefix before git clone URL Antoine Lambert 2023-09-26 14:18:36 +02:00
  • 88611642fc Introduce bioconductor lister Kumar Shivendu 2023-09-28 12:54:37 +00:00
  • a04975571c gitweb: Remove invalid use of str.rstrip Antoine Lambert 2023-09-26 13:42:37 +02:00
  • aa7b3fa7d8 rpm: Add config for listing EPEL source packages Antoine Lambert 2023-09-08 15:58:09 +02:00
  • bf806f2c7b Remove spurious space Franck Bret 2023-09-21 09:18:54 +02:00
  • ebba50882f Revert "Remove spurious space" Franck Bret 2023-09-21 07:14:44 +00:00
  • c9e2339af9 Remove spurious space Franck Bret 2023-09-20 17:01:35 +02:00
  • fa09df8ba8 Merge branch 'dlang' of gitlab.softwareheritage.org:franckbret/swh-lister into dlang Franck Bret 2023-09-20 16:51:25 +02:00
  • cc686268ba url and instance are now mandatory Franck Bret 2023-09-20 16:49:52 +02:00
  • 5b4d15090f Remove fedora entry as it as been replaced with rpm Franck Bret 2023-09-20 16:48:06 +02:00
  • eb48db902e Merge branch 'dlang' of gitlab.softwareheritage.org:franckbret/swh-lister into dlang Franck Bret 2023-09-19 16:53:48 +02:00
  • 2793ef9aad D lang lister Franck Bret 2023-07-18 10:33:30 +02:00
  • 1c964cccd3 maven/README: Fix links Valentin Lorentz 2023-09-14 13:41:29 +02:00
  • 6e7bc49ec7 Harmonize listers parameters and add test to check mandatory ones Antoine Lambert 2023-09-06 11:55:28 +02:00
  • 5f717e311d
         rpm: Adapt lister constructor to accept the credentials parameter Antoine R. Dumont (@ardumont) 2023-09-05 17:24:32 +02:00
  • a02fdbb4c8
         lister.github.utils: Drop no longer used module Antoine R. Dumont (@ardumont) 2023-08-22 11:15:04 +02:00
  • 91e4e33dd5 cran: Improve listing of R packages Antoine Lambert 2023-08-17 16:52:39 +02:00
  • 3a0e8b9995 requirements.txt: Sort packages by name Antoine Lambert 2023-08-17 10:45:09 +02:00
  • 95714f6f37 rpm: Turn fedora lister into a generic Red Hat based distribution one Antoine Lambert 2023-08-16 13:25:23 +00:00
  • fcfb7004db
         pypi: Allow passing configuration arguments to task Antoine R. Dumont (@ardumont) 2023-08-04 15:34:02 +02:00
  • 928d592e10
         sourceforge: Allow passing configuration arguments to task Antoine R. Dumont (@ardumont) 2023-08-04 15:18:22 +02:00
  • b02144b4f9
         packagist: Yield pages of origins to regularly record origins Antoine R. Dumont (@ardumont) 2023-08-03 16:12:13 +02:00
  • 15a4c4cdb4
         packagist: Skip package if unable to parse the last update date Antoine R. Dumont (@ardumont) 2023-08-03 15:44:30 +02:00
  • d4f3e91466
         packagist: Allow batch size records configuration in constructor Antoine R. Dumont (@ardumont) 2023-08-03 11:01:01 +02:00