| Date | Commit message (Collapse) |
|---|
|
We can't compare created_at times with lei if lei tests are
skipped due to Inline::C or Socket::MsgHdr unavailability.
Reported-by: Jörg Rödel <joro@8bytes.org>
Link: https://public-inbox.org/meta/YZebmAxlFJy4lqAw@8bytes.org/
|
|
Inbox->xdb does not exist, but this code path was apparently
never tested :x I noticed this on basic v2 inbox, but it could
happen with any v1/v2 inbox. Move ->num2docid into Search
so it's less awkward to use.
|
|
Oops, we shouldn't attempt to read a users' actual HOME when
running -index, since mine has a bunch of invalid entries in
there.
|
|
This allows IMAP mirrors to keep UIDVALIDITY synchronized (and
"LIST ACTIVE.TIMES" in NNTP). "lei add-external --mirror" will
automatically set it, as will the combination of
public-inbox-clone + public-inbox-index.
This avoids the need for extra endpoints or config entries,
at least...
|
|
Partial (v2) clones should be useful addition for users wanting
to conserve storage while having fast access to recent messages.
Continuing work started in 876e74283ff3 (fetch: ignore
non-writable epoch dirs, 2021-09-17), this creates bare,
read-only epoch git repos. These git repos have the remotes
pre-configured, but does not fetch any objects.
The goal is to allow users to set the writable bit on a
previously-skipped epoch and start fetching it.
Shell completion support may not be necessary given how short
the epoch ranges are, here.
Cc: Luis Chamberlain <mcgrof@kernel.org>
Link: https://public-inbox.org/meta/20210917002204.GA13112@dcvr/T/#u
|
|
The full pathname for "curl -o ..." was too noisy and confusing.
Reduce confusion by adding the ".tmp" suffix and relying on
"-C". We'll also avoid displaying "-C" in run_reap() and
rely on "--git-dir=" with "git fetch" to display progress for
users.
|
|
Since the beginning of time, I've been dropping Makefiles
in $INBOX_DIR (and above hiearchies) to organize groups
of commands.
make(1) is widely available in various flavors and a familiar
tool for our target audience. It is easy to run in the right
directory, typically has built-in shell completion, and doesn't
silently ignore errors by default like Bourne shell.
|
|
As noted in the new manpage entry, this is useful for avoiding
public-inbox-index invocations when there's nothing to update.
We use 127 to match "grok-pull", and also because it doesn't
conflict with any of the current curl(1) exit codes.
|
|
IMHO, this greatly improves code sharing and organization
between v2, extindex, and lei/store. Common git-related
logic for these is lightly-refactored and easier to reason
about.
The impetus for this big change was to ensure inboxes
created+managed by public-inbox-{clone,fetch} could have
alternates and configs setup properly without depending on
SQLite (via V2Writable). This change does that while
making old code shorter and better factored.
|
|
t/v2mirror.t and t/lei-mirror.t are now skipped when curl
is missing (instead of failing in appropriate places).
A bunch of which() checks are updated to use require_cmd
to avoid explicitly loading Spawn.
|
|
Timestamp comparisons only have 1 second granularity, which
isn't nearly enough for our test cases, and probably not for
real world use for "git send-email" bursts and fast SMTP
servers.
We'll continue to check modification times inside the manifest,
though, in case an extremely rare SHA-1 collision is found...
|
|
The v1 code path was totally half-baked after the change
to use manifest.js.gz :x
Fixes: ffb7fbda6869db4b ("fetch: use manifest.js.gz for v1")
|
|
Instead of generic "Unnamed repository" or "missing" messages,
show "mirror of $URL" since it seems like a better default when
creating a mirror.
|
|
Setting up and maintaining git-only mirrors of v2 inboxes is
complex since multiple commands are required to clone and fetch
into epochs.
Unlike grokmirror, these commands do not require any
configuration. Instead, they rely on existing git config files
and work like "git clone --mirror" and "git fetch",
respectively.
Like grokmirror, they use manifest.js.gz, but only on a
per-inbox basis so users won't have to clone every inbox of a
large instance nor edit config files to include/exclude inboxes
they're interested in.
|
|
If the mirror.done file doesn't exist for unlink, it's because
we already got another error, so don't confuse users by noting
an unlink error since the ENOENT is expected in the face of
other errors.
|
|
The current manifest.js.gz generation in WWW doesn't account for
PSGI mount prefixes (and grokmirror 1.x appears to work fine).
In other words, <https://yhbt.net/lore/lkml/manifest.js.gz>
currently has keys like "/lkml/git/0.git" and not
"/lore/lkml/git/0.git" where "/lore" is the PSGI mount prefix.
This works fine with the prefix accounted for in my grokmirror
(1.x) repos.conf like this:
site = https://yhbt.net/lore/
manifest = https://yhbt.net/lore/manifest.js.gz
Adding the PSGI mount prefix in manifest.js.gz is probably not
desirable since it would force the prefix into the locally
cloned path by grokmirror, and all the cloned directories
would have the remote PSGI mount prefix prepended to the
toplevel.
So, "lei add-external --mirror" needs to account for PSGI
mount prefixes by deducing the prefix based on available keys
in the manifest.js.gz hash table.
|
|
I just hit an unreproducible failure in t/lei-p2q.t and
lacked $lei_err information to diagnose it. Hopefully
this helps track down odd failures in the future.
|
|
v2 onions are insecure, deprecated and going away. v3 names are
unfortunately longer and more difficult to remember, but should
be more resistant to attack than v2 ones.
|
|
This makes it easier to manage test dependencies on systems
where optional stuff isn't installed. This fixes some lei tests
which didn't check for Plack before starting -httpd, and ensures
Parse::RecDescent is available for -imapd in case
Mail::IMAPClient stops using it.
|
|
While we were exiting with a error code, showing a successful
"# mirrored $URL" message is misleading and wrong. Don't show
success until everything is complete and the config is written.
|
|
git 2.11 and earlier could not handle git directories with
newlines in them, nor does libgit2 support them.
Followup-to: d87dd0e679587043 ("config: reject `\n' in `inboxdir'")
|
|
lei() and lei_ok() are superior since they offer prototype
checks and lei_ok() adds another check + description DRY-ness.
The $lei sub was only bound to a variable since it was in
t/lei.t and named subs don't work well with the key2sub()
wrapper.
|
|
DESTROY callbacks can clobber $?, so we must take care to
preserve it when exiting. We'll also try to make an effort to
ensure better DESTROY ordering and delete as much as possible
before x_it finishes.
We also need to load PublicInbox::Config when setting up
public inboxes.
|
|
IPv4 gets plenty of real-world coverage, and apparently there's
Debian buildd hosts which lack IPv4(*). So ensure everything
can work on IPv6 and not cause problems for odd setups.
(*) https://bugs.debian.org/979432
|
|
This will need some work to before it's generally applicable
to the rest of our code base.
|
|
This also updates lei_xsearch to follow the same pattern for
stopping curl(1) and tail(1) processes it spawns.
|
|
Option combinations which make no sense should fail
to prevent misunderstandings and avoid surprises.
|
|
This can be useful for users who want to clone and
mirror an existing public-inbox. This doesn't have
update support, yet, so users will need to run
"git fetch && public-inbox-index" for now.
|