about summary refs log tree commit homepage
path: root/Documentation/public-inbox-clone.pod
DateCommit message (Collapse)
2024-02-14doc: fix formatting for CLI switch aliases
`=item' elements in Pod need to be surrounded by empty lines. It's an unfortunate waste of vertical space, but Pod is still better than *roff and usually available out-of-the-box.
2023-08-16doc: clone: add a note about git 2.41+ and fetch.hideRefs
It's unusable for large mirrors, otherwise.
2023-03-18clone: support --purge to delete remotely-deleted repos
This lets us clean up disk space when repos are removed on the remote side.
2023-03-18doc: clone: note the default value of --remote-manifest=
It may not be immediately obvious to users unfamiliar with grokmirror.
2023-03-14doc: clone: fix typo in --remote-manifest= description
Reported-by: Kyle Meyer <kyle@kyleam.com> Link: https://public-inbox.org/meta/87v8j4ql8k.fsf@kyleam.com/
2023-03-13doc: clone: document --remote-manifest= option
2023-03-07doc: update public-inbox-clone examples and help
Basically, public-inbox-clone has become grok-pull without config files nor absolute paths.
2023-02-27doc: update clone+fetch with 2.0+ switches
Because old versions will exist for a long time and our latest documentation is visible on the web, we must document when a switch appears to avoid confusing users of old versions.
2023-01-06clone: implement --exit-code
Since public-inbox-clone is now useful for incremental updates with manifest, --exit-code belongs here, too.
2023-01-06clone: document --project-list and --post-update-hook
I forgot to document these when I implemented them :x
2022-11-28lei_mirror: --manifest= affects destination, too
This probably makes the most sense, if a user wants to use an alternate path to read from, it's likely they want to write it there, too.
2022-11-28lei_mirror: respect `./' and `../' prefixes for CLI args
Users may wish to keep objstore and manifest files at a higher level to prevent direct access via HTTP(S), so those relative paths probably make sense.
2022-11-28clone: support --keep-going/-k like make(1)
This can be useful for intermittent network errors, and the required code changes makes it less dependent on global state.
2022-11-28clone|fetch: support passing --prune(-tags) to `git fetch'
We need to be able to get rid of removed branches and tags on the remote. --prune-tags is implied for non-objstore repos, and incompatible with objstore repos.
2022-11-28clone: support loading manifest.js.gz from destination
This will allow us to quickly check fingerprints against remotes with a single HTTP(S) request, saving us numerous `git show-refs' invocations.
2022-11-28clone: require `--objstore=' for default location
Allowing just `--objstore' without `=' was confusing, since it could eat one of the required parameters (URL or DESTINATION).
2022-11-28clone: flesh out --objstore behavior and document
We can support absolute paths to avoid surprising behaviors, but relative paths are preferred since the goal is to be accessible over the "dumb" HTTP git transport (the dumb transport is uses less memory and CPU on the server).
2022-11-28clone: support --inbox-version
This is part of `lei add-external --mirror', and it makes sense to have for development and testing. We'll also add a fallback in case somebody tries --inbox-version and fails due to a newer remote instances of public-inbox.
2022-11-28clone: support --inbox-config option
This allows avoiding 404s when trying _/text/config/raw on code repositories.
2022-11-28clone: support --dry-run / -n flag
It still makes HTTP(S) requests to retrieve the manifest or scrape HTML, but doesn't make permanent changes to the FS (aside from modifying {acm}time of ${TMPDIR-/tmp}).
2022-11-28clone: support --include and --exclude with multi-clone
These will be handy when someone is interested in a subset of inboxes on a large hosting site.
2021-11-04doc: add more 3rd-party refs, use Debian manpages for xapian
curl, torsocks, and gitglossary manpages are all newly referenced, so make sure they're linkified properly in HTML. We'll be using Debian's manpages as an ad-free, Tor-accessible host for manpages as a fallback since hosting manpages for all 3rd-party projects we reference doesn't scale.
2021-11-03doc: -clone|lei add-external: add bit about the Makefile
It's pretty useful, I think.
2021-09-24clone|--mirror: support --epoch=RANGE for partial clones
Partial (v2) clones should be useful addition for users wanting to conserve storage while having fast access to recent messages. Continuing work started in 876e74283ff3 (fetch: ignore non-writable epoch dirs, 2021-09-17), this creates bare, read-only epoch git repos. These git repos have the remotes pre-configured, but does not fetch any objects. The goal is to allow users to set the writable bit on a previously-skipped epoch and start fetching it. Shell completion support may not be necessary given how short the epoch ranges are, here. Cc: Luis Chamberlain <mcgrof@kernel.org> Link: https://public-inbox.org/meta/20210917002204.GA13112@dcvr/T/#u
2021-09-12new public-inbox-{clone,fetch} commands
Setting up and maintaining git-only mirrors of v2 inboxes is complex since multiple commands are required to clone and fetch into epochs. Unlike grokmirror, these commands do not require any configuration. Instead, they rely on existing git config files and work like "git clone --mirror" and "git fetch", respectively. Like grokmirror, they use manifest.js.gz, but only on a per-inbox basis so users won't have to clone every inbox of a large instance nor edit config files to include/exclude inboxes they're interested in.