Date | Commit message (Collapse) |
|
This is a transitionary interface which does NOT require an
event loop. It can be plugged into in current synchronous code
without major surgery.
It allows HTTP/1.1 pipelining-like functionality by taking
advantage of predictable and well-specified POSIX pipe semantics
by stuffing multiple git cat-file requests into the --batch pipe
With xt/git_async_cmp.t and GIANT_GIT_DIR=git.git, the async
interface is 10-25% faster than the synchronous interface since
it can keep the "git cat-file" process busier.
This is expected to improve performance on systems with slower
storage (but multiple cores).
|
|
run_die() doesn't require an $env arg, so there's no
point passing "undef" to it.
|
|
It's unnecessary code which I'm not sure we ever used. In
retrospect, completely clearing the environment doesn't make
sense for the processes we spawn. We don't need to clobber
individual environment variables in our code, either
(and if we did for tests, we can use 'local').
|
|
I haven't noticed this being a problem in practice, but
be consistent with the rest of the singleton stuff.
Since we always call Reset() at load time, only do
initialization in that sub and not at declaration.
|
|
Since we give users no indication or control of how "git gc"
runs, showing its progress is confusing.
|
|
Our attempt at using a self-pipe in signal handlers was
ineffective, since pure Perl code execution is deferred
and Perl doesn't use an internal self-pipe/eventfd. In
retrospect, I actually prefer the simplicity of Perl in
this regard...
We can use sigprocmask() from Perl, so we can introduce
signalfd(2) and EVFILT_SIGNAL support on Linux and *BSD-based
systems, respectively. These OS primitives allow us to avoid a
race where Perl checks for signals right before epoll_wait() or
kevent() puts the process to sleep.
The (few) systems nowadays without signalfd(2) or IO::KQueue
will now see wakeups every second to avoid missed signals.
|
|
Oops, IO::KQueue support was broken due to this missing
constant. Add a new ds-kqxs.t test case to ensure we
test the IO::KQueue path if IO::KQueue is available.
|
|
I'm not sure if TZ minute offsets aside from '00' or '30' exist,
but lets just deal with them properly when negative. Examples
taken from various inboxes on lore.kernel.org. These are mostly
message from spammers, but some are legitimate messages.
|
|
Since we're using Perl 5.10.1 and File::Temp 0.19+, we don't
need Xtmpdirs at all for cleaning up tempdirs on failure and
can just rely on the DESTROY handler provided by File::Temp.
|
|
While the master process has a self-pipe to avoid missing
signals, worker processes lack that aside from a pipe to
detect master death.
That pipe doesn't exist when there's no master process,
so it's possible DS::close never finishes because it
never woke up from epoll_wait. So create a pipe on
the worker_quit signal and force it into epoll/kevent
so it wakes up right away.
|
|
We need to block signals in workers during respawns
until they're ready to receive signals.
|
|
`$SIG{FOO} = "IGNORE"' will cause the daemon to miss signals
entirely. Instead, we can use sigprocmask to block signal
delivery until we have our signal handlers setup. This closes a
race where a PID file can be written for an init script and a
signal to be dropped via "IGNORE".
|
|
Perl 5.16.3 (and possibly older versions) fails with the
following errors (from CentOS7):
Use of ?PATTERN? without explicit operator is deprecated
Search pattern not terminated
|
|
This is distributed with Perl 5.10.1 and onwards, so it should
not be an installation burden for any users. I'm planning to
move away from tempdir() entirely and use File::Temp->newdir to
remove dependencies on END{} blocks.
|
|
This makes the subroutine behave more like which(1) command
and will make using spawn() in tests easier.
|
|
We need to bypass whatever Test::More does with END/DESTROY
handlers for use in lon-lived process. This doesn't affect
any of our normal code since we don't use END/DESTROY for
Xapcmd and its callers.
|
|
SearchIdx->new no longer accepts a GIT_DIR path as its argument
since commit 585314673236d664729fe3ab2d4fb229d1c0f2d5
("searchidx: require PublicInbox::Inbox (or InboxWritable) ref")
|
|
We've been using this in -edit, and will be using it in some
more scripts and tests to optimize for run_mode=2 with
run_script.
Keeping this in the *Writable modules since I don't see it being
useful for the WWW and NNTP read-only interfaces which use
PublicInbox::Inbox.
|
|
PublicInbox::Admin::config() just adds an extra layer of
indirection which we barely rely on. So get rid of this
global variable and make it easier to run tests in the
future without relying on global state.
|
|
IO::Compress::Gzip is a wrapper around Compress::Raw::Zlib,
anyways, and being able to easily detach buffers to return them
via ->getline is nice. This results in a 1-2% performance
improvement when fetching giant mboxes.
|
|
It'll make using Compress::Raw::Zlib easier, since we
can use that and import constants more easily.
|
|
We're gradually phasing mid_clean out (in favor of mids()).
|
|
InboxWritable caching the result of ->importer leads to a
circular references with returned (V2Writable|Import) object
holds onto the calling InboxWritable object.
With public-inbox-watch, this leads to a memory leak if a user
is reloading via SIGHUP after a message is imported (it would
only become noticeable with SIGHUPs after every message imported).
I would not expect anybody to to notice this in real-world
usage. I only noticed this since I was making -xcpdb suitable
for long-lived process use (e.g. "mod_perl style") and a flock
remained unreleased on v1 inboxes after resharding.
WatchMaildir (used by -watch) already handles caching of the
importer object itself, and all of our other real-world uses of
->importer are short-lived or designed for batch scripts, so
there's no need to cache the importer result internally.
|
|
Perl's "local" allows changes to %SIG (and %ENV) to be limited
to its enclosing block. This allows us to get rid of a global
variable and ad-hoc method for restoring signal handlers.
|
|
I sometimes post context-free documentation patches generated
with "-U0" to reduce size and bandwidth overhead when replacing
URLs or updating copyright notices. git-apply(1) needs the
--unidiff-zero switch to work properly with context-free
patches.
Given our search looks for blob OIDs, and we're never going
to be running the code we regenerate, "--unidiff-zero" ought
to be safe.
|
|
While I've never seen "git log" fail on its own, it could happen
one day and we should be prepared to abort indexing when it
happens.
Beef up tests for t/spawn.t to ensure close() behaves
on popen_rd the way we expect it to.
|
|
We don't want to define $SIG{__WARN__} in the worker to call an
existing non-default callback. Instead update ->{current_info}
the same way the V2Writable master process does.
I noticed this while reindexing with a large XAPIAN_FLUSH_THRESHOLD
and seeing the wrong epoch on my terminal from a shard because the
shard worker was spawned while reindexing a higher-numbered epoch.
|
|
While testing 216light.css changes, I managed to hit some cases
where dillo failed to render ' correctly, but I also can't
reproduce it reliably. Anyways, it's definitely a problem with
some old browsers and newer versions of highlight already work
around it, but Debian 10.x has 3.41, so use "'" to maximize
compatibility.
|
|
We need to detect "git apply" failures reliably when patches
fail. This is necessary for solving for blob 81c1164ae5 in
https://public-inbox.org/git/ when at least two messages can
solve for it (and one of them fails):
1. https://public-inbox.org/git/b9fb52b8-8168-6bf0-9a72-1e6c44a281a5@oracle.com/
2. https://public-inbox.org/git/56664222-6c29-09dc-ef78-7b380b113c4a@oracle.com/
|
|
When solving for blob 81c1164ae5 in https://public-inbox.org/git/,
at least two messages get indexed with the dfpost result for
that blob (after fixing MsgIter to decode all text/* parts):
1. https://public-inbox.org/git/b9fb52b8-8168-6bf0-9a72-1e6c44a281a5@oracle.com/
2. https://public-inbox.org/git/56664222-6c29-09dc-ef78-7b380b113c4a@oracle.com/
However, only the first message contains a usable patch. So
we must adjust SolverGit to account for multiple messages
hitting the same "dfpost:" search result and attempt
"git apply" on all results, not just the first.
In the future, changes to SearchIdx.pm may rid us of invalid
search results and speed up performance (at the expense of
developer/indexing time); but we need to account for old search
indices, first.
|
|
We want to index text/x-patch and text/x-diff, at least,
since "git format-patch" can generate a patch series as
attachments using --attach.
|
|
ISO-2202-JP and other non-UTF-8 messages need to be displayed
correctly.
Fixes: 7d82a8bc04ce ('handle "multipart/mixed" messages which are not multipart')
|
|
--reindex has gotten better over the years, and having parallel
Xapian DB directories would exceed all available disk space for
some users with giant inboxes.
|
|
Spell "Schwartzian" correctly, and clarify the location of
"modified" since we have multiple subs named "modified"
|
|
* learn:
doc: add public-inbox-learn(1) manpage
mda: support multiple List-ID matches
mda: prepare for multiple destinations
inboxwritable: add assert_usable_dir sub
mda: skip MIME parsing if spam
mda: hoist out mda_filter_adjust
filter/base: remove MAX_MID_SIZE constant
mda: hoist out List-ID handling and reuse in -learn
learn: hoist out remove_or_add subroutine
learn: GIT_COMMITTER_<NAME|EMAIL> may be "" or "0"
learn: update usage statement
learn: only map recipient list on "ham" or "rm"
learn: support multiple To/Cc headers
|
|
While it's not RFC2919-conformant, mail software can
theoretically set multiple List-ID headers. Deliver to all
inboxes which match a given List-ID since that's likely the
intended.
Cc: Eric W. Biederman <ebiederm@xmission.com>
Link: https://public-inbox.org/meta/87pniltscf.fsf@x220.int.ebiederm.org/
|
|
And use it for mda, since "0" could be a usable directory
if somebody insists on using relative paths...
|
|
We don't need it in the filter, here, since we have
one in the MDA package.
|
|
It's now possible to inject false-positive ham into an inbox
the same way -mda does via List-ID.
|
|
Since we index X-Alt-Message-ID (because we need to placate some
NNTP clients), we now display it as well, since that Message-ID
could be the X-Alt-Message-ID that the reader is actually
interested in.
|
|
Since we replace extra Message-ID headers with X-Alt-Message-ID
to placate NNTP clients, we should allow searching and indexing
on X-Alt-Message-ID just like we do with Message-ID.
|
|
And use it for the per-message permalink display.
|
|
"refer" is not the correct term, here; since that would mean
multiple messages have the current message in the "References:"
header, and that's a normal occurence.
Instead, we need to warn the reader that the given message
itself has multiple Message-IDs.
|
|
Browsers may underline '<' and '>' in links, which may be
confused with '≤' and '≥'. So have the Message-ID header
display follow what we do with In-Reply-To headers and move the
"<" and ">" outside of <a> in the HTML.
|
|
Mail headers can contain multiple headers of any type, so ensure
we don't hide any information we're getting in the per-message
permalink views.
This means it's possible to have multiple From, Date, To, Cc,
Subject, and In-Reply-To headers displayed.
The thread indices are a special case, I guess, since we run
out of space on the line if the headers too long and tools like
mutt only show the first one.
|
|
We can easily support searching on messages with
multiple From/To/Cc/Subject headers just like we
do with multiple Message-ID headers.
This matches the normal mutt pager display behavior.
|
|
* regen:
v2writable: use msgmap as multi_mid queue
v2writable: move git->cleanup to the correct place
v2writable: reindex handles 3-headered monsters
v2writable: improve "num_for" API and disambiguate
v2writable: set unindexed article number
|
|
I'm not sure they'll make a measurable difference or will
be worth the effort in the future given the prevalance
of HTTPS and giant socket buffers.
Using Inline::C for this may make more sense in the
future, too, especially if we want to be able to use
GnuTLS.
|
|
commit 476fc666c223f0fb ('reduce "PublicInbox::Hval->new_oneline" use')
was mis-titled, since it completely eliminated ->new_oneline use.
|
|
This was intended for solver, but it's unused since
commit 915cd090798069a4
("solver: switch patch application to use a callback")
|