| Date | Commit message (Collapse) |
|---|
|
Creating a scalar ref directly off substr() seemed to be causing
the underlying non-ref scalar to end up in Perl's scratchpad.
Assign the substr result to a local variable seems sufficient to
prevent multi-megabyte SVs from lingering indefinitely when a
read-only daemon serves rare, oversized blobs.
|
|
We need to use a non-greedy regexp to avoid capturing the
".git" suffix in the pathname before blindly appending our
own.
|
|
We want these long-lived processes to die naturally when their
parent dies. Hopefully this improves graceful shutdown for
-extindex because I'm interrupting a lot of reindexing...
|
|
While we try to invoke all pending callbacks to force error
handling, the current callback wasn't getting invoked on
invoked on async_abort if my_read/my_readline failed.
|
|
We make $req a reference upon retrying, but
"SCALAR(...)" in error messages isn't helpful, so
dereference the scalar ref.
|
|
This should help us catch BUG: errors (and then some) in
-extindex and other read-write code paths. Only read-only
daemons should warn on async callback failures, since those
aren't capable of causing data loss.
|
|
We need to abort both check-only and cat requests when
aborting, since we'll be aborting more aggressively
in in read-write paths.
|
|
Having git processes outlive DB handles is likely to hurt
from a fragmentation perspective if the DB handle needs to
be recreated immediately due to a git->cat_async callback.
So only unref DB handles when we're sure there's no live
git users left, otherwise check the inodes.
We'll also avoid needless localization checks in git->cleanup
and make the return value more obvious since the pid fields are
unconditionally deleted nowadays.
|
|
Long pathnames are difficult to read and distinguish in ps(1)
output. Deep paths can also slow down pathname resolution
when dealing with loose objects, so we put "cat-file --batch"
deeper into the directory tree.
Since v2 processes are in the form of $INBOXDIR/all.git, keep
the basename of $INBOXDIR in --git-dir= so it's easy to
distinguish between processes just by looking at ps(1).
While "git -C" also exists, it's only present in git 1.8.5+.
We also need to keep in mind the "directory" pointed to by
--git-dir= need not be a directory (nor a symlink pointing
to one).
This reduces pathname resolution overhead for v1 and v2 inbox
git processes, but unfortunately not for extindex since that
needs to store alternates as absolute paths.
|
|
Avoid relying on a giant cleanup hash and instead use the new
DS->add_uniq_timer API to amortize the pause times associated
with having to cleanup many inboxes. We can also use smaller
intervals for this, as well.
We now discard SQLite DB handles at cleanup. Each of these can
use several megabytes of memory, which adds up with
hundreds/thousands of inboxes. Since per-inbox access intervals
are unpredictable and opening an SQLite handle is relatively
inexpensive, release memory more aggressively to avoid the heap
having to hit swap.
|
|
Neither Inboxes nor ExtSearch objects were retrying correctly
when there are live git processes, but the inboxes were getting
rescanned for search or other reasons. Ensure the scan retries
eventually if there's live processes.
We also need to update the cleanup task to detect Xapian shard
count changes, since Xapian ->reopen is enough to detect any
other Xapian changes. Otherwise, we just issue an inexpensive
->reopen call and let Xapian check whether there's anything
worth reopening.
This also lets us eliminate the Devel::Peek dependency.
|
|
Check for unlinked mmap-ed files via /proc/$PID/maps every 60s
or so.
ExtSearch (extindex) is compatible-enough with Inbox objects to
be wired into the old per-inbox code, but the startup cost is
projected to be much higher down the line when there's >30K
inboxes, so we scan /proc/$PID/maps for deleted files before
unlinking. With old Inbox objects, it was (and is) simpler to
just kill processes w/o checking due to the low startup cost
(and non-portability of checking).
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20210921144754.gulkneuulzo27qbw@meerkat.local/
|
|
PIPE_BUF accounts for Linux being 4096 (and presumably other
OSes differing), while _POSIX_PIPE_BUF is the minimum 512
value.
|
|
When generating per-inbox manifests, we were forgetting to
cleanup per-epoch "git cat-file --batch" processes. Our
previous method of generating modified times was also stupidly
inefficient, so replace the pipeline with a single
"git for-each-ref" invocation.
|
|
git always quotes with leading zeros to ensure the octal
representation is 3 characters long. We enforce that to match
low ASCII characters (e.g. [x01-\x06]) that don't need the
range provided by 3 characters.
git_unquote now does a single pass so it won't get fooled by
decoded backslashes into parsing a digit as an octal character.
git_unquote is also capped to "\377" so we don't overflow a
byte.
|
|
Some cgit configs use trailing slashes in pathnames
which we preserve internally.
Before this change, trailing slashes in cgit config files
was causing ViewVCS (SolverGit) output to show up as "???"
for coderepos without cgitUrl configured.
|
|
By adding "+code" next to "mirror" at the top next to the search
box. Instead of showing "/path/to/$FOO", showing "$FOO.git"
makes it more obvious we're talking about a git repo, here,
instead of some random directory.
|
|
That logic is inlined directly into git_async_prefetch
in GitAsyncCat and I don't see it being useful outside
of a DS event loop.
|
|
This could lead to bad results when doing ls-tree -z
for v2 import in case there's multiple files. In any case,
the `local $/ = "\0"' in Import.pm is also eliminated to
reduce potential confusion and surprises.
|
|
This is necessary to avoid slowdowns with pathological cases
with many dates in the query, since each rev-parse invocation
takes ~5ms.
This is immeasurably slower with one open-ended range, but
already faster with any closed range featuring two dates which
require parsing via git.
|
|
Users are expected to be familiar with git's "approxidate"
functionality for parsing dates, so we'll expose that
in our UIs. Xapian itself has limited date parsing functionality
and I can't expect users to learn it.
This takes around 4-5ms on my aging workstation, so it'll
probably be made acceptable for the WWW UI, even.
libgit2 has a git__date_parse function which I expect to have
less overhead, but it's only for internal use at the moment.
|
|
Instead of forcing callers to set a variable to write into,
we'll just rely on wantarray.
|
|
Only saves us one line of code, but that's better than nothing.
|
|
If we're using ->qx, we're operating synchronously anyways,
so there's little point in relying on the event loop for
waitpid.
|
|
$git->qx and $git->popen now $env and $opt for redirects
like lower-level popen_rd. This may be beneficial in other
places.
|
|
Using "make update-copyrights" after setting GNULIB_PATH in my
config.mak
|
|
Objects with DESTROY callbacks get propagated to children, so we
must be careful to not invoke waitpid from children on their
sibling processes. Only parents (and their parents...) can reap
child processes.
|
|
This simplifies our code and provides a more consistent API for
error handling. PublicInbox::DS can be loaded nowadays on all
*BSDs and Linux distros easily without extra packages to
install.
The downside is possibly increased startup time, but it's
probably not as a big problem with lei being a daemon
(and -mda possibly following suite).
|
|
We can use the ternary operator to avoid an early return, here
|
|
Some of these changes may not be strictly necessary, but it
makes code easier to maintain and change. Hackers
using/modifying this code will no longer wonder if a particular
callsite needs to care about subclasses or not.
|
|
This makes GitAsyncCat more resilient to bugs in Gcf2 or even
git-cat-file itself. I noticed -imapd stuck on read(2) from
the Gcf2 pipe, so there may be a bug somewhere in Gcf2 or
PublicInbox::Git. This should make us more resilient to them
and hopefully help us notice and fix them.
|
|
We'll be using this for MiscIdx and pre-generating the necessary
JSON for manifest.js.gz, so make it easier to share code for
generating per-repo JSON entries for grokmirror.
|
|
->cat_async and ->check_async may trigger each other (in future
callers) while waiting, so we need a unified method to ensure
both complete. This doesn't affect current code, but allows us
to slightly simplify existing callers.
|
|
We need to loop the inflight check for nested callback
invocations to ensure we don't clog the pipe that feeds
`git cat-file'.
This bug was obscured by the fact that we're already
accounting for 64-char git OIDs with SHA-256 in the
pipe space calculation; perhaps we shouldn't do that.
|
|
For external indices, we'll need to support nested cat_async
invocations to deduplicate cross-posted messages.
Thus we need to ensure we do not clobber the {inflight*} queues
while stepping through and ensure {cat_rbuf} is stored before
invoking callbacks.
This fixes the ->cat_async-only case, but does not yet
account for the mix of ->check_async interspersed with
->cat_async calls, yet. More work will be needed on that
front at a later date.
|
|
It's currently not a problem as ->destroy doesn't
happen for no reason, we'll need to ensure future uses of
->destroy correctly discard the check_async buffer.
|
|
It seems easiest to have a singleton Gcf2Client client object
per daemon worker for all inboxes to use. This reduces overall
FD usage from pipes.
The `public-inbox-gcf2' command + manpage are gone and a `$^X'
one-liner is used, instead. This saves inodes for internal
commands and hopefully makes it easier to avoid mismatched
PERL5LIB include paths (as noticed during development :x).
We'll also make the existing cat-file process management
infrastructure more resilient to BOFHs on process killing
sprees (or in case our libgit2-based code fails on us).
(Rare) PublicInbox::WWW PSGI users NOT using public-inbox-httpd
won't automatically benefit from this change, and extra
configuration will be required (to be documented later).
|
|
This amortizes the cost of recreating PublicInbox::Gcf2 objects
when alternates change in v2 all.git.
|
|
Since we only get OIDs from trusted local data sources
(over.sqlite3), we can safely retry within the -gcf2 process
without worry about clients spamming us with requests for
invalid OIDs and triggering reopens.
|
|
This will help with eventual git SHA-256 transitions.
|
|
I'm seeing "read: Connection timed out" from in my syslog from
-httpd. The fail() calls in PublicInbox::Git seems to be the
only code path of ours which could trigger it...
ETIMEDOUT shouldn't happen on pipes, only sockets; and all of
our socket operations are non-blocking. So this could be
cgit-wwwhighlight-filter.lua, but that's connecting over
localhost, though on fairly loaded HW.
|
|
We can keep the git process more active by sending another
request to it while fetch_run_ops() is running. This
parallelization speeds up mutt's initial FETCH for headers by
around ~35%(!).
|
|
This allows v1 indexing to run while the `cat-file --batch-check'
process is waiting on high-latency storage.
|
|
parent.pm is leaner than base.pm, and Time::HiRes::stat is
more accurate, so take advantage of these Perl 5.10+-isms
since it's been over a year since we left 5.8 behind.
|
|
While this circular reference was carefully managed to not leak
memory; it was still triggering a warning at -imapd/-nntpd
shutdown due to the EPOLL_CTL_DEL op failing after the $Epoll FD
gets closed.
So remove the circular reference by providing a ref to `undef',
instead.
|
|
This matches the behavior of the existing synchronous ->cat_file
method. In fact, ->cat_file now becomes a small wrapper around
the ->cat_async method.
|
|
Trying to avoid a circular reference by relying on $ibx object
here makes no sense, since skipping GitCatAsync::close will
result in an FD leak, anyways. So keep GitAsyncCat contained to
git-only operations, since we'll be using it for Solver in the
distant feature.
|
|
This will make it easier to implement the retries on
alternates_changed() of the synchronous ->cat_file API.
|
|
We do this for the C10K-oriented HTTP/NNTP/IMAP processes, and
we may support thousands of git-cat-file processes in the
future.
|
|
This ought to improve overall performance with multiple clients.
Single client performance suffers a tiny bit due to extra
syscall overhead from epoll.
This also makes the existing async interface easier-to-use,
since calling cat_async_begin is no longer required.
|