Date | Commit message (Collapse) |
|
We can use "use" to get the namespace into the "BEGIN" phase of
the interpreter. While we're at it, use \&coderef syntax
explicitly instead of globbing everything.
|
|
This is distributed with Perl 5.10.1 and onwards, so it should
not be an installation burden for any users. I'm planning to
move away from tempdir() entirely and use File::Temp->newdir to
remove dependencies on END{} blocks.
|
|
We've been using this in -edit, and will be using it in some
more scripts and tests to optimize for run_mode=2 with
run_script.
Keeping this in the *Writable modules since I don't see it being
useful for the WWW and NNTP read-only interfaces which use
PublicInbox::Inbox.
|
|
Avoid 'Variable "%s" will not stay shared' warnings
when the contents of this script eval'ed into a sub.
|
|
Avoid 'Variable "%s" will not stay shared' warnings
when the contents of this script eval'ed into a sub.
|
|
Avoid 'Variable "%s" will not stay shared' warnings
when the contents of this script eval'ed into a sub.
We also need to rely on ->DESTROY instead of END{}
to unlink the lock file on sub exit.
|
|
Avoid 'Variable "%s" will not stay shared' warnings
when the contents of this script eval'ed into a sub.
|
|
PublicInbox::Admin::config() just adds an extra layer of
indirection which we barely rely on. So get rid of this
global variable and make it easier to run tests in the
future without relying on global state.
|
|
Instead of relying on END{} blocks, rely on ->DESTROY
so the temporary files go out-of-scope and system
resources get released, sooner.
|
|
Avoid 'Variable "%s" will not stay shared' warnings
when the contents of this script eval'ed into a sub.
|
|
We only need to parse the command-line once.
|
|
InboxWritable caching the result of ->importer leads to a
circular references with returned (V2Writable|Import) object
holds onto the calling InboxWritable object.
With public-inbox-watch, this leads to a memory leak if a user
is reloading via SIGHUP after a message is imported (it would
only become noticeable with SIGHUPs after every message imported).
I would not expect anybody to to notice this in real-world
usage. I only noticed this since I was making -xcpdb suitable
for long-lived process use (e.g. "mod_perl style") and a flock
remained unreleased on v1 inboxes after resharding.
WatchMaildir (used by -watch) already handles caching of the
importer object itself, and all of our other real-world uses of
->importer are short-lived or designed for batch scripts, so
there's no need to cache the importer result internally.
|
|
We need to check every print to a regular file for errors,
because storage devices inevitably fail.
|
|
exit($?) is never correct, since ($? >> 8) is needed to extract
the correct exit code, as other information (e.g. such as signal)
is encoded in $? in addition to the exit code.
|
|
While it's not RFC2919-conformant, mail software can
theoretically set multiple List-ID headers. Deliver to all
inboxes which match a given List-ID since that's likely the
intended.
Cc: Eric W. Biederman <ebiederm@xmission.com>
Link: https://public-inbox.org/meta/87pniltscf.fsf@x220.int.ebiederm.org/
|
|
Multiple List-ID headers will be supported in the next commit
|
|
And use it for mda, since "0" could be a usable directory
if somebody insists on using relative paths...
|
|
We don't want to waste cycles parsing the message for MIME bits
if it's spam.
|
|
It makes it easier to document the default -mda behavior is
stricter than normal, including "public-inbox-learn ham"
|
|
It's now possible to inject false-positive ham into an inbox
the same way -mda does via List-ID.
|
|
We'll be reusing it for List-ID processing in the next commit.
|
|
Users may be zeroes or blanks.
|
|
Use <foo|bar> since that seems to be the favored notation
for required command args (taking a hint from git(1) manpage).
While we're at it, remove the space after '<' for the redirect
to match git.git coding style.
|
|
It's assumed that "spam" can end up anywhere due to Bcc:, so we
need to scan every single inbox. However, "rm" is usually more
targeted and and "ham" obviously only belongs in some inboxes.
|
|
It's possible to specify these headers multiple times, and
PublicInbox::MDA->precheck takes that into account, so
-learn should, too.
|
|
* origin/inboxdir:
config: remove redundant inboxdir check
config: support "inboxdir" in addition to "mainrepo"
examples/grok-pull.post_update_hook: use "inbox_dir"
|
|
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
|
|
Since -mda now supports List-ID to better support mirroring of
existing mailing lists, it probably makes sense to support
disabling the precheck function to provide more accurate (though
potentially spammier) mirrors of lists
|
|
This also adds watchheader tests for -watch, which we never
had before :x
|
|
First, we use flock(2) to wait on parallel public-inbox-init(1)
invocations while we make multiple changes using git-config(1).
This flock allows -init processes to wait on each other if using
reasonable POSIX filesystems.
Then, we also need a git-config(1)-compatible lock to prevent
user-invoked git-config(1) processes from clobbering our
changes while we're holding the flock.
|
|
Since I intend to add support for --skip-artnum, disambiguating
the long option name makes sense. We'll support --skip
indefinitely for compatibility.
|
|
The httpd-supplied write callback is the leak culprit under Perl
5.16.3. undef-ing it immediately after use keeps a repeated
"git fetch" loop from monotonically increasing memory and FD use
on the Perl shipped with RHEL/CentOS 7.x.
Other endpoints tested showed no increase in memory use under
constant load with "ab -HAccept-Encoding:gzip -k", including the
async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via
SolverGit module.
|
|
Testing with perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7,
the Deflater middleware triggers a leak when used in conjunction
with our push-based responses from PublicInbox::Qspawn.
I could not find another solution to workaround the memory leak
in this case, and I could not find a specific leak fix in
the perl5180delta manpage[1] which looked like it would
solve our problem.
Attempting to workaround the issue proved futile. Using
internal Deflater-specific keys to prevent deflating in
GitHTTPBackend and Qspawn did not solve the problem:
$env->{"plack.skip-deflater"} = 1;
$env->{"psgix.no-compress"} = 1;
Nor did forcing an invalid encoding via "git fetch":
git -c http.extraheader=Accept-Encoding:gzap fetch
So this appears to be a problem with Plack::Util::response_cb
somewhere.
This does NOT appear to be a problem with ref() leaking as in
DS::next_tick[2], since I couldn't find where
Plack::Middleware::Deflater or Plack::Util::response_cb would be
calling ref() on a blessed reference to trigger a leak.
Also, oddly enough, the ref() use for backwards compatibility at
the top of PublicInbox::GitHTTPBackend::serve does NOT seem to
trigger a leak on 5.16.3 due to [2]:
# XXX compatibility... ugh, can we stop supporting this?
$git = PublicInbox::Git->new($git) unless ref($git);
[1] https://perldoc.perl.org/perl5180delta.html
[2] https://rt.perl.org/Public/Bug/Display.html?id=114340
|
|
|
|
This is only tested so far with my patches to Net::NNTP at:
https://rt.cpan.org/Ticket/Display.html?id=129967
Memory use in C10K situations is disappointing, but that's
the nature of compression.
gzip compression over HTTPS does have the advantage of not
keeping zlib streams open when clients are idle, at the
cost of worse compression.
|
|
It kinda, barely works, and I'm most happy I got it working
without any modifications to the main NNTP::event_step callback
thanks to the DS->write(CODE) support we inherited from
Danga::Socket.
|
|
The eval was unnecessary, and $0 can't be "--".
Tested with /bin/sh on FreeBSD 11.2
|
|
* origin/reshard:
xcpdb: support resharding v2 repos
xcpdb: use destination shard as progress prefix
xapcmd: preserve indexlevel based on the destination
v2writable: use a smaller default for Xapian partitions
|
|
v2 repos are sometimes created on machines where CPU
parallelization exceeds the capability of the storage devices.
In that case, users may reshard the Xapian DB to any smaller,
positive integer to avoid excessive overhead and contention when
bottlenecked by slow storage.
Resharding can also be used to increase shard count after
hardware upgrades.
|
|
We don't need to leave temporary files lying around.
|
|
mutt will set Content-Length, Lines, and Status headers
unconditionally, so we need to account for that before
doing header comparisons to avoid making expensive changes
when noop edits are made.
|
|
Fill in undef as "(unchanged)" when displaying commits
and prefix the epoch name.
|
|
This wrapper around V2Writable->replace provides a user-interface
for editing messages as single-message mboxes (or the raw text
via $EDITOR).
|
|
We'll be reusing the same logic for -edit
|
|
We'll be using this in -edit, and maybe other admin-oriented
tools for UI-consistency.
|
|
Editing and purging are similar operations involving history
rewrites, so there'll be common options and code between them.
|
|
While I don't expect git to suddenly start spewing non-ASCII
digits in places I'd expect ASCII, this would make things easier
for future hackers and reviewers.
|
|
In case some BOFH decides to randomly create directories
using non-ASCII digits all over the place.
|
|
And use it from Admin.
It's easy to tell what indexlevel=basic is from unconfigured
inboxes, but distinguishing between 'medium' and 'full' would
require stat()-ing position.* files which is fragile and
Xapian-implementation-dependent.
So use the metadata facility of Xapian and store it in the main
partition so Admin tools can deal better with unconfigured
inboxes copied using generic tools like cp(1) or rsync(1).
|
|
It's annoying for people using "git fetch && public-inbox-index"
as one user; and running -httpd/-nntpd as a different user
(where users see different config files).
|