reonokiy/swh-lister
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
958 commits 1 branch 0 tags 16 MiB
Commit graph

38 commits

Author SHA1 Message Date
Antoine Lambert
bde37867d8 docs: Fix broken external links 
Those were spotted thanks to the sphinx linkcheck builder
 2025-02-20 10:15:54 +00:00
David Douard
714fccc3c7 python: Fix black formatting after bump to 23.1.0 in pre-commit 2023-12-05 10:33:07 +01:00
David Douard
ac52cfed21 Apply swh-py-template 0.1.6 2023-12-03 17:54:52 +01:00
David Douard
c2402f405f Migrate to copier-based swh-py-template 2023-11-29 17:23:28 +01:00
David Douard
553884fa56 docs: include the README file in the main index page 
Convert README from markdown to ReST to make it embeddable in
docs/index.rst
 2023-11-16 16:25:56 +01:00
Antoine Lambert
4f57e84450 Use http_retry decorator from swh.core.retry module 
The http_retry decorator has been moved to swh-core package in order
to ease its reuse across swh packages.
 2023-04-13 14:19:57 +02:00
Antoine Lambert
4cec345028 docs: Include module indices only when building standalone package doc 
In order to remove warnings about /apidoc/*.rst files being included
multiple times in toc when building full swh documentation, prefer to
include module indices only when building standalone package documentation.

Also include them the proper sphinx way.

Related to T4496
 2022-12-19 13:46:51 +01:00
Antoine Lambert
db6ce12e9e Refactor and deduplicate HTTP requests code in listers 
Numerous listers were using the same page_request method or equivalent
in their implementation so prefer to deduplicate that code by adding
an http_request method in base lister class: swh.lister.pattern.Lister.

That method simply wraps a call to requests.Session.request and logs
some useful info for debugging and error reporting, also an HTTPError
will be raised if a request ends up with an error.

All listers using that new method now benefit of requests retry when
an HTTP error occurs thanks to the use of the http_retry decorator.
 2022-09-26 10:48:40 +02:00
Antoine Lambert
9c55acd286 Use generic HTTP retry policy by default and rename dedicated decorator 
Instead of retrying HTTP requests only for 429 status code by default,
prefer to use the generic retry policy enabling to also retry for status
codes >= 500 but also on ConnectionError exceptions.

Rename throttling_retry decorator to http_retry to reflect this change.
 2022-09-26 10:48:40 +02:00
Antoine Lambert
d38e05cff7 python: Reformat code with black 22.3.0 
Related to T3922
 2022-04-08 15:15:09 +02:00
Valentin Lorentz
f6ca1dc3dc docs: Fix ReST syntax 2022-01-24 16:57:38 +01:00
Valentin Lorentz
771c406710 Fix ReST syntax 2022-01-21 11:04:51 +01:00
Antoine R. Dumont (@ardumont)
40e227efac
docs: Fix sphinx warning 
Related to D6967
 2022-01-19 10:24:26 +01:00
Antoine R. Dumont (@ardumont)
04dc628091
docs: Explain task type registering to complete the save forge doc 
Related to T3629
 2021-10-08 16:07:41 +02:00
Antoine R. Dumont (@ardumont)
1a9c08c93f
docs: Add a save forge documentation 
This does not yet enter into the registration of a new lister.

Related to T3629
 2021-10-08 16:07:40 +02:00
Boris Baldassari
04c0a50706 tuleap: initialise lister. 
tuleap-lister: fix args in test_task.

tuleap-lister: Add rate-limiting test + fix debug and typo.

tuleap-lister: code review: fix mocker + tests/setup_cli.

tuleap-lister: code review: fix relister > lister.

tuleap-lister: code review: fix test_task kwargs.

tuleap-lister: code review: Remove authentication useless lines + fix typos.

tuleap-lister: code review: improve results_simplified for svn repos.

tuleap-lister: code review: add name to CONTRIBUTORS file.

tuleap-lister: code review: Update tutorial for misc files to edit.

tuleap-lister: code review: Update copyright to 2021 exactly.

tuleap-lister: code review: Update py files perms -X.

tuleap-lister: code review: minimise json files.

tuleap-lister: code review: fix chmod on json files.

tuleap-lister: code review: fix var names + add tests.

tuleap-lister: code review: fix useless indirection.

tuleap-lister: code review: Add empty repo test, minor typo fixes.
 2021-05-26 11:09:12 +02:00
Valentin Lorentz
40e1916510 Fix various Sphinx warnings 2021-04-13 21:56:08 +02:00
Valentin Lorentz
465506a0ce Remove old lister tutorial. 
Sphinx complains because it's an orphan document.
 2021-04-13 18:31:31 +02:00
Hezekiah Maina
7124627400 Added the right location on linter.yml file 2021-04-04 23:22:46 +03:00
Valentin Lorentz
df73073a67 docs: Fix title syntax 2021-03-19 09:58:38 +01:00
tenma
2e17729e97
docs: Add new "howto write a lister tutorial" with unified lister api 
This adds a new tutorial which details how to currently write the new listers (both
incremental or stateless). This proposes a python template file to start a new lister.

At last, this renames the previous tutorial into tutorial-2017.

Related to T3073
 2021-02-26 16:05:18 +01:00
Antoine Lambert
e72c15e97a docs: Update listers execution instructions 
Remove outdated part about listers database and use swh CLI in
README for executing a lister instead of raw Python code.
 2021-02-05 14:51:24 +01:00
David Douard
b1811f9f8a Add a cli section to the doc 2021-01-05 11:53:22 +01:00
Antoine Lambert
99fcd2b3f5 docs: Fix sphinx warnings 
Related to T2188
 2020-01-17 16:15:11 +01:00
David Douard
3ddfd00e90 Fix typos (and trailing ws) reported by codespell 2019-11-21 14:11:18 +01:00
Antoine R. Dumont (@ardumont)
481b30c540
docs: Fix toc 
Related T1984
 2019-09-06 12:31:06 +02:00
Antoine R. Dumont (@ardumont)
09f3605a7e
docs: Remove spurious blank spaces 2019-08-29 09:57:59 +02:00
Antoine R. Dumont (@ardumont)
b3463ecddc
Drop SWH prefix in classes everywhere 
It's redundant with the swh modules in itself.
 2019-06-20 19:08:46 +02:00
Archit Agrawal
8d1b5d2d2d Add new page "run_a new_lister" 
Add new page in lister tutorial which guides
through the process of running a new lister in
docker.
 2019-06-20 14:01:13 +05:30
Archit Agrawal
b2c6ddc35b tutorial: Add testing and how to run section 
Add testing section for lister. Also added how to run a
new lister section which elaborates about the steps
required to run the now lister in docker
 2019-06-20 14:01:13 +05:30
David Douard
da4eb6cd34 doc: update index to match new swh-doc format 
related to T1330
 2018-11-23 13:50:51 +01:00
David Douard
2fa3908283 docs: prevent sphinx from complaining about tutorial.rst not being included 2018-11-09 11:21:08 +01:00
Stefano Zacchiroli
3d65b9f223 docs: add title and brief module description 2018-10-01 15:53:13 +02:00
Antoine R. Dumont (@ardumont)
d9a25865a2
docs/tutorials: Format sentences 2018-06-26 10:48:12 +02:00
Antoine R. Dumont (@ardumont)
4b351321d8
docs/tutorials: Remove blank spaces 2018-06-26 10:48:03 +02:00
Stefano Zacchiroli
b8ad57fc05 docs: integrate swh-lister tutorial 
originally appeared on the Software Heritage blog at
https://www.softwareheritage.org/2017/03/24/list-the-content-of-your-favorite-forge-in-just-a-few-steps/
by Avi Kelman

Closes T762
 2017-11-02 17:53:30 +01:00
Stefano Zacchiroli
fe4725d383 docs: add absolute anchor to documentation index 2017-11-02 11:09:07 +01:00
Stefano Zacchiroli
67a4c62a64 docs/: add sphinx apidoc generation skeleton 
change cherry picked from python module template
commit 71b117ba0cf9f1251b1cac26d0994df03a4c787d
 2017-08-30 12:25:37 +02:00