about summary refs log tree commit homepage
path: root/script
DateCommit message (Collapse)
2020-01-01filter/base: export REJECT as a constant
And update callers to use it, as it makes the code a bit cleaner. Probably irrelvant, but it should be faster, too, as "perl -I lib -w -MO=Deparse $FILE" shows REJECT() calls are constant-folded.
2019-12-24remove "no warnings 'once'" in a few places
We can use "use" to get the namespace into the "BEGIN" phase of the interpreter. While we're at it, use \&coderef syntax explicitly instead of globbing everything.
2019-11-24check for File::Temp 0.19 for ->newdir method
This is distributed with Perl 5.10.1 and onwards, so it should not be an installation burden for any users. I'm planning to move away from tempdir() entirely and use File::Temp->newdir to remove dependencies on END{} blocks.
2019-11-16inboxwritable: add ->cleanup method
We've been using this in -edit, and will be using it in some more scripts and tests to optimize for run_mode=2 with run_script. Keeping this in the *Writable modules since I don't see it being useful for the WWW and NNTP read-only interfaces which use PublicInbox::Inbox.
2019-11-16learn: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16mda: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16init: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub. We also need to rely on ->DESTROY instead of END{} to unlink the lock file on sub exit.
2019-11-16index: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-16admin: get rid of singleton $CFG var
PublicInbox::Admin::config() just adds an extra layer of indirection which we barely rely on. So get rid of this global variable and make it easier to run tests in the future without relying on global state.
2019-11-16edit: use OO API of File::Temp to shorten lifetime
Instead of relying on END{} blocks, rely on ->DESTROY so the temporary files go out-of-scope and system resources get released, sooner.
2019-11-16edit: pass global variables into subs
Avoid 'Variable "%s" will not stay shared' warnings when the contents of this script eval'ed into a sub.
2019-11-14convert: remove duplicated GetOptions() call
We only need to parse the command-line once.
2019-11-14inboxwritable: drop {-importer} cyclic reference
InboxWritable caching the result of ->importer leads to a circular references with returned (V2Writable|Import) object holds onto the calling InboxWritable object. With public-inbox-watch, this leads to a memory leak if a user is reloading via SIGHUP after a message is imported (it would only become noticeable with SIGHUPs after every message imported). I would not expect anybody to to notice this in real-world usage. I only noticed this since I was making -xcpdb suitable for long-lived process use (e.g. "mod_perl style") and a flock remained unreleased on v1 inboxes after resharding. WatchMaildir (used by -watch) already handles caching of the importer object itself, and all of our other real-world uses of ->importer are short-lived or designed for batch scripts, so there's no need to cache the importer result internally.
2019-11-08edit: check for write errors writing "From_" line
We need to check every print to a regular file for errors, because storage devices inevitably fail.
2019-11-08edit: propagate correct editor exit code
exit($?) is never correct, since ($? >> 8) is needed to extract the correct exit code, as other information (e.g. such as signal) is encoded in $? in addition to the exit code.
2019-10-30mda: support multiple List-ID matches
While it's not RFC2919-conformant, mail software can theoretically set multiple List-ID headers. Deliver to all inboxes which match a given List-ID since that's likely the intended. Cc: Eric W. Biederman <ebiederm@xmission.com> Link: https://public-inbox.org/meta/87pniltscf.fsf@x220.int.ebiederm.org/
2019-10-30mda: prepare for multiple destinations
Multiple List-ID headers will be supported in the next commit
2019-10-30inboxwritable: add assert_usable_dir sub
And use it for mda, since "0" could be a usable directory if somebody insists on using relative paths...
2019-10-30mda: skip MIME parsing if spam
We don't want to waste cycles parsing the message for MIME bits if it's spam.
2019-10-30mda: hoist out mda_filter_adjust
It makes it easier to document the default -mda behavior is stricter than normal, including "public-inbox-learn ham"
2019-10-30mda: hoist out List-ID handling and reuse in -learn
It's now possible to inject false-positive ham into an inbox the same way -mda does via List-ID.
2019-10-30learn: hoist out remove_or_add subroutine
We'll be reusing it for List-ID processing in the next commit.
2019-10-30learn: GIT_COMMITTER_<NAME|EMAIL> may be "" or "0"
Users may be zeroes or blanks.
2019-10-30learn: update usage statement
Use <foo|bar> since that seems to be the favored notation for required command args (taking a hint from git(1) manpage). While we're at it, remove the space after '<' for the redirect to match git.git coding style.
2019-10-30learn: only map recipient list on "ham" or "rm"
It's assumed that "spam" can end up anywhere due to Bcc:, so we need to scan every single inbox. However, "rm" is usually more targeted and and "ham" obviously only belongs in some inboxes.
2019-10-30learn: support multiple To/Cc headers
It's possible to specify these headers multiple times, and PublicInbox::MDA->precheck takes that into account, so -learn should, too.
2019-10-17Merge remote-tracking branch 'origin/inboxdir'
* origin/inboxdir: config: remove redundant inboxdir check config: support "inboxdir" in addition to "mainrepo" examples/grok-pull.post_update_hook: use "inbox_dir"
2019-10-16config: support "inboxdir" in addition to "mainrepo"
"mainrepo" ws a bad name and artifact from the early days when I intended for there to be a "spamrepo" (now just the ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be especially confusing, since v2 needs at least two git repositories (epoch + all.git) to function and we shouldn't confuse users by having them point to a git repository for v2. Much of our documentation already references "INBOX_DIR" for command-line arguments, so use "inboxdir" as the git-config(1)-friendly variant for that. "mainrepo" remains supported indefinitely for compatibility. Users may need to revert to old versions, or may be referring to old documentation and must not be forced to change config files to account for this change. So if you're using "mainrepo" today, I do NOT recommend changing it right away because other bugs can lurk. Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
2019-10-16mda: support --no-precheck option
Since -mda now supports List-ID to better support mirroring of existing mailing lists, it probably makes sense to support disabling the precheck function to provide more accurate (though potentially spammier) mirrors of lists
2019-10-15mda, watch: wire up List-ID header support
This also adds watchheader tests for -watch, which we never had before :x
2019-10-05init: implement locking
First, we use flock(2) to wait on parallel public-inbox-init(1) invocations while we make multiple changes using git-config(1). This flock allows -init processes to wait on each other if using reasonable POSIX filesystems. Then, we also need a git-config(1)-compatible lock to prevent user-invoked git-config(1) processes from clobbering our changes while we're holding the flock.
2019-10-05init: favor --skip-epoch instead of --skip
Since I intend to add support for --skip-artnum, disambiguating the long option name makes sense. We'll support --skip indefinitely for compatibility.
2019-09-27qspawn: workaround Perl 5.16.3 leak, re-enable Deflater
The httpd-supplied write callback is the leak culprit under Perl 5.16.3. undef-ing it immediately after use keeps a repeated "git fetch" loop from monotonically increasing memory and FD use on the Perl shipped with RHEL/CentOS 7.x. Other endpoints tested showed no increase in memory use under constant load with "ab -HAccept-Encoding:gzip -k", including the async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via SolverGit module.
2019-09-26httpd: disable Deflater middleware by default on Perl <5.18
Testing with perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7, the Deflater middleware triggers a leak when used in conjunction with our push-based responses from PublicInbox::Qspawn. I could not find another solution to workaround the memory leak in this case, and I could not find a specific leak fix in the perl5180delta manpage[1] which looked like it would solve our problem. Attempting to workaround the issue proved futile. Using internal Deflater-specific keys to prevent deflating in GitHTTPBackend and Qspawn did not solve the problem: $env->{"plack.skip-deflater"} = 1; $env->{"psgix.no-compress"} = 1; Nor did forcing an invalid encoding via "git fetch": git -c http.extraheader=Accept-Encoding:gzap fetch So this appears to be a problem with Plack::Util::response_cb somewhere. This does NOT appear to be a problem with ref() leaking as in DS::next_tick[2], since I couldn't find where Plack::Middleware::Deflater or Plack::Util::response_cb would be calling ref() on a blessed reference to trigger a leak. Also, oddly enough, the ref() use for backwards compatibility at the top of PublicInbox::GitHTTPBackend::serve does NOT seem to trigger a leak on 5.16.3 due to [2]: # XXX compatibility... ugh, can we stop supporting this? $git = PublicInbox::Git->new($git) unless ref($git); [1] https://perldoc.perl.org/perl5180delta.html [2] https://rt.perl.org/Public/Bug/Display.html?id=114340
2019-09-09run update-copyrights from gnulib for 2019
2019-07-06nntp: support COMPRESS DEFLATE per RFC 8054
This is only tested so far with my patches to Net::NNTP at: https://rt.cpan.org/Ticket/Display.html?id=129967 Memory use in C10K situations is disappointing, but that's the nature of compression. gzip compression over HTTPS does have the advantage of not keeping zlib streams open when clients are idle, at the cost of worse compression.
2019-06-24nntp: NNTPS and NNTP+STARTTLS working
It kinda, barely works, and I'm most happy I got it working without any modifications to the main NNTP::event_step callback thanks to the DS->write(CODE) support we inherited from Danga::Socket.
2019-06-14edit: fix portability of editor invocation
The eval was unnecessary, and $0 can't be "--". Tested with /bin/sh on FreeBSD 11.2
2019-06-14Merge remote-tracking branch 'origin/reshard' into next
* origin/reshard: xcpdb: support resharding v2 repos xcpdb: use destination shard as progress prefix xapcmd: preserve indexlevel based on the destination v2writable: use a smaller default for Xapian partitions
2019-06-14xcpdb: support resharding v2 repos
v2 repos are sometimes created on machines where CPU parallelization exceeds the capability of the storage devices. In that case, users may reshard the Xapian DB to any smaller, positive integer to avoid excessive overhead and contention when bottlenecked by slow storage. Resharding can also be used to increase shard count after hardware upgrades.
2019-06-12edit: unlink temporary file when done
We don't need to leave temporary files lying around.
2019-06-10edit: drop unwanted headers before noop check
mutt will set Content-Length, Lines, and Status headers unconditionally, so we need to account for that before doing header comparisons to avoid making expensive changes when noop edits are made.
2019-06-10edit|purge: improve output on rewrites
Fill in undef as "(unchanged)" when displaying commits and prefix the epoch name.
2019-06-09edit: new tool to perform edits
This wrapper around V2Writable->replace provides a user-interface for editing messages as single-message mboxes (or the raw text via $EDITOR).
2019-06-09AdminEdit: move editability checks from -purge
We'll be reusing the same logic for -edit
2019-06-09admin: beef up resolve_inboxes to handle purge options
We'll be using this in -edit, and maybe other admin-oriented tools for UI-consistency.
2019-06-09purge: start moving common options to AdminEdit module
Editing and purging are similar operations involving history rewrites, so there'll be common options and code between them.
2019-06-05tighten up digit matches to ASCII for git output
While I don't expect git to suddenly start spewing non-ASCII digits in places I'd expect ASCII, this would make things easier for future hackers and reviewers.
2019-06-04require ASCII digits for local FS items
In case some BOFH decides to randomly create directories using non-ASCII digits all over the place.
2019-05-29searchidx: store indexlevel=medium as metadata
And use it from Admin. It's easy to tell what indexlevel=basic is from unconfigured inboxes, but distinguishing between 'medium' and 'full' would require stat()-ing position.* files which is fragile and Xapian-implementation-dependent. So use the metadata facility of Xapian and store it in the main partition so Admin tools can deal better with unconfigured inboxes copied using generic tools like cp(1) or rsync(1).