Antoine R. Dumont (@ardumont)
f8055a34af
swh.lister.gitlab: Add notes around current limitations
2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
0e517b6143
swh.lister.gitlab: Use http_url_to_repo key to build the origin url
2018-07-11 09:45:54 +02:00
Antoine R. Dumont (@ardumont)
7a0e3a2379
gitlab.lister: Override filtering as it's not possible as is
2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
1335a5b25f
swh.lister.gitlab: Use gitlab's index identifier (page id)
...
And not 'project' id
2018-07-11 09:45:53 +02:00
Antoine R. Dumont (@ardumont)
3e62bc867e
swh.lister.cli: Simplify cli
2018-07-11 09:45:51 +02:00
Antoine R. Dumont (@ardumont)
afcd6997c4
swh.lister.cli: Add a basic cli to deal with create/drop db actions
2018-07-03 15:49:52 +02:00
Antoine R. Dumont (@ardumont)
0f9ba7bd9b
swh.lister.gitlab: Bootstrap gitlab lister
...
Related T989
2018-07-03 12:23:11 +02:00
Antoine R. Dumont (@ardumont)
c350434127
swh.lister.core.lister_transports: Fix typo and pep8 violations
2018-07-03 12:17:46 +02:00
Antoine R. Dumont (@ardumont)
f4fe1b058b
swh.lister.*: Formatting
2018-07-03 12:17:46 +02:00
Stefano Zacchiroli
4f2361ab0e
fix typo in docstrings/comments (tnx codespell)
2018-03-12 12:02:30 +01:00
Nicolas Dandrimont
16e40151dd
core.tasks: use proper TaskType import
2018-01-19 14:24:17 +01:00
Nicolas Dandrimont
77585de20c
Force utf-8 encoding for tests
2017-10-30 17:05:18 +01:00
Nicolas Dandrimont
e477a46c60
Add missing __init__.py files
...
Helps with tests autodetection
2017-10-30 16:38:27 +01:00
Nicolas Dandrimont
6a7b0f802e
swh.lister.debian.tasks: add task_queue attribute
2017-10-30 15:08:37 +01:00
Nicolas Dandrimont
458a9e6733
Add a new DebianListerTask
2017-10-30 14:19:43 +01:00
Nicolas Dandrimont
2857dcf926
Cleanup packaging
2017-10-12 17:16:49 +02:00
Nicolas Dandrimont
c6e455ce9b
debian.lister: use get_packages() method on the snapshot to list packages
...
Allows to list the packages outside of lister context (e.g. for loader tests)
2017-10-10 16:37:15 +02:00
Nicolas Dandrimont
d2a71ac980
debian.lister: make file size an integer rather than a string
2017-10-10 16:36:58 +02:00
Nicolas Dandrimont
3efd0e33bf
lister_transports: load version by using vcversioner's export module
2017-10-10 16:36:38 +02:00
Sushant Sushant
83ebb95705
Add a lister for Debian-like package archives
...
This work is based on Sushant's internship and D229.
2017-10-04 12:43:09 +02:00
Nicolas Dandrimont
6c8e16aae9
lister_transports: allow overriding the parameters to requests
2017-09-11 15:23:17 +02:00
Nicolas Dandrimont
52b5c140c1
lister_base: use the softwareheritage-scheduler-dev db by default
2017-09-11 15:21:56 +02:00
Nicolas Dandrimont
7476f5d533
lister_base: make the default filter return all entries
2017-09-11 15:21:56 +02:00
Stefano Zacchiroli
8a519af8f5
docstring: drop useless heading ":py" domain in crossrefs
2017-09-06 20:27:34 +02:00
Stefano Zacchiroli
e3c59f8fa9
sanitize docstrings for sphinx
2017-09-05 17:15:58 +02:00
Nicolas Dandrimont
5a08bb7278
lister_trasnports: send a custom http user agent
2017-09-05 15:21:55 +02:00
Nicolas Dandrimont
5eca777c75
lister_transports: filter known http response codes
2017-09-05 15:21:17 +02:00
Nicolas Dandrimont
de01552ac5
indexing_lister: actually filter objects by start and end of range
2017-09-05 14:51:35 +02:00
Nicolas Dandrimont
cf3220d1fb
github.models: handle the fork argument
2017-09-05 11:38:27 +02:00
Nicolas Dandrimont
4b56b6037c
github.models: add fork information to repos
2017-09-04 19:40:45 +02:00
Nicolas Dandrimont
f6f077b789
github.tasks: the github api is rooted at api.github.com
2017-09-04 17:42:20 +02:00
Nicolas Dandrimont
04c22dd56e
lister_base: when storing responses for debugging, use a timestamp as filename
2017-09-04 17:41:54 +02:00
Nicolas Dandrimont
3fe5619bcc
lister_base: use timezone-aware datetimes everywhere
2017-09-04 17:41:36 +02:00
Nicolas Dandrimont
a85f55dbf9
lister_transports: add headers to debug data
2017-09-04 17:41:06 +02:00
Nicolas Dandrimont
eb47600461
lister_transports: requests credentials are a tuple (username, password)
2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
7c6ac9df0d
lister_base: cache all responses
2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
3c7cfe7d78
lister_base: prepare_folders takes its arguments flattened
2017-09-04 17:11:00 +02:00
Nicolas Dandrimont
b6bdab70c6
lister_base: create_tasks returns a list of tasks
2017-06-28 16:44:52 +02:00
Nicolas Dandrimont
c677ad6067
indexing_lister: use task_ids to disable tasks
2017-06-28 16:44:27 +02:00
Nicolas Dandrimont
75ba12e395
remove useless __init__.py file
2017-06-12 18:22:04 +02:00
Nicolas Dandrimont
af60301f3a
tasks: update to new swh.scheduler API
2017-06-12 15:40:28 +02:00
Avi Kelman (fiendish)
68d77fd43f
Refactor lister code
...
Streamline production of new listers by aggressively moving core
functionality into progressively inherited (A->B->C) base classes
with the transport layer abstracted.
This should make common individual forge listers straightforward to
produce with minimal customization. Github and Bitbucket listers
can be used as examples of the indexing type.
2017-03-06 12:35:49 +01:00
Antoine Pietri
a6e43f2777
config: use 5002 as the default storage port
2017-02-21 17:23:58 +01:00
Antoine R. Dumont (@ardumont)
b217f55cfe
Update storage configuration reading
...
Related T613
2016-12-15 19:07:02 +01:00
Nicolas Dandrimont
d47905b0a1
tasks: add tasks for incremental and full updates
2016-10-20 17:19:39 +02:00
Nicolas Dandrimont
6fd0184229
lister: update Copyright
2016-10-20 16:59:28 +02:00
Nicolas Dandrimont
7fa507e6ff
lister: disable tasks for deleted repositories
...
When operating on a range of repositories, and we notice that a
repository has disappeared, we disable the task associated with that
repository.
2016-10-20 16:28:19 +02:00
Nicolas Dandrimont
a1a6228e05
lister: retrieve old task and origin id if a full_name has been recycled
...
If a repo changed hands, it is possible that a full_name is recycled. In
that case, we reuse the task_id and origin_id from the old repository
instead of recreating them.
2016-10-20 16:26:53 +02:00
Nicolas Dandrimont
ab25599cca
lister: create origins and tasks for repositories
2016-10-20 16:26:27 +02:00
Nicolas Dandrimont
1eb5d5cf79
lister: move "next" next_id computation earlier
2016-10-20 16:23:05 +02:00
