Date | Commit message (Collapse) |
|
* regen:
v2writable: use msgmap as multi_mid queue
v2writable: move git->cleanup to the correct place
v2writable: reindex handles 3-headered monsters
v2writable: improve "num_for" API and disambiguate
v2writable: set unindexed article number
|
|
I'm not sure they'll make a measurable difference or will
be worth the effort in the future given the prevalance
of HTTPS and giant socket buffers.
Using Inline::C for this may make more sense in the
future, too, especially if we want to be able to use
GnuTLS.
|
|
commit 476fc666c223f0fb ('reduce "PublicInbox::Hval->new_oneline" use')
was mis-titled, since it completely eliminated ->new_oneline use.
|
|
This was intended for solver, but it's unused since
commit 915cd090798069a4
("solver: switch patch application to use a callback")
|
|
InboxWritable::maildir_path_load exists and we may support
it for use with standalone scripts.
|
|
I'm not sure if this will get used anywhere, but at least
call a function which exists in dead code.
|
|
This hasn't been used since commit 48b21cb662c1e17b7 in 2016:
("declare Inbox object for reusability")
|
|
This hasn't been used since commit 1b7e935ab1690e28
("searchidx: fix incremental index with indexlevel=basic on v1")
|
|
Instead of storing Message-IDs in the Msgmap object, we can
store the blob OID.
For initial indexing of mirrors, this lets us preserve
$sync->{regen} by storing the intended article number in
the queue.
On --reindex, the article number we store in Msgmap is ignored
but only used for ordering purposes.
This also allows us to avoid ENOMEM errors if somebody abuses
our system by reusing Message-IDs; but we now risk ENOSPC
instead (but systems tend to have more FS storage than RAM).
|
|
We need to stop the git process to avoid leaking FDs
to Xapian if we recurse ->index_sync on reindex.
|
|
And maybe 8-headered ones, too...
I noticed --reindex failing on the linux-renesas-soc mirror due
one 3-headed monster of a message having 3 sets of headers;
while another normal message had a Message-ID that matched one
of the 3 IDs of the 3-headed monster.
We still try to do the majority of indexing backwards, but we
defer indexing multi-Message-ID'd messages until the end to
ensure we get all the "good" messages in before we process the
multi-headered ones.
Link: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/
|
|
Make it obvious that we're not the Msgmap sub and return an
array because it's less awkward than providing a modifiable ref
to a function to write to.
|
|
We'll actually use the keys of this hash in future commits.
|
|
* origin/inboxdir:
config: remove redundant inboxdir check
config: support "inboxdir" in addition to "mainrepo"
examples/grok-pull.post_update_hook: use "inbox_dir"
|
|
While it is possible to host source code from the root of a URL
using git-http-backend(1), the lack of pathname in the URL can
also be confusing to users. So just add the path name of the
project into the URL itself so users can invoke "git clone"
with one command-line argument instead of two.
Of course, previously documented URLs continue to work as normal.
|
|
This was causing compatibility problems for old configs
when using public-inbox-nntpd.
|
|
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
|
|
* listid:
wwwtext: show listid config directive(s)
mda, watch: wire up List-ID header support
config: allow "0" as a valid mainrepo path
config: avoid unnecessary '||' use
config: simplify lookup* methods
config: we always have {-section_order}
Config.pm: Add support for mailing list information
|
|
Since public-inbox-index may be run against a large list of
(intended) inboxes from the command-line, it's helpful to show
which directory fails the resolution.
|
|
We want to share this piece for potential mirror-ers just
like watchheader.
|
|
This also adds watchheader tests for -watch, which we never
had before :x
|
|
It's probably wrong to use relative path names, but things are
all relative these days anyways with shared and networked FSes.
|
|
'//' is available in Perl 5.10+ which allows `0' and `""' (empty
string) to remain unclobbered.
We also don't need '||=' for initializing our internal caches.
|
|
This ensures we always process inboxes in section order and
reduces the amount of code we have to maintain for each lookup.
Avoiding the cost of inboxes object creation is not worth the
code overhead; and we can implement a config cache via Storable
easily for large configs and -mda users.
|
|
Rewrite a bunch of tests to use ordered input (emulating
"git config -l" output) so we can always walk sections in
the order they were given in the config file.
|
|
The world has turned since I first started following mailing lists and
to my surprise every mailing list that I am subscribed to properly
sets the "List-ID:" mailing list header. So instead of doing
something clever and flexible I am adding support for looking up
public inbox mailing lists by their mailing list name.
That makes the work needed for each email trivial and easy to understand.
- Parse the "List-ID:" header.
- Lookup in the configuration which mailbox is connected to that
"List-ID:"
- Deliver the mail to that mailbox.
To that end this change enhances PublicInbox to have an additional
mailbox configuration parameter "listid" that holds the mailing list
name.
A method is added to the PublicInbox config object called
lookup_list_id that given a mailing list name will return the
PublicInbox in the configuration that is configured to handle that
mailing list.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
[ew: avoid autovivification of $ibx->{listid} for t/config.t]
|
|
I don't trust the MIME type to not munge my email messages in horrible
ways upon occasion. Therefore allow for passing in the raw message
value instead of trusting the mime object to preserve it.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
[ew: use "//" from Perl 5.10+ for defined check]
|
|
This is a plugin for SpamAssassin that happens to be quite
useful in keeping spam off lists I mirror. Hopefully more
people can find it useful now that it has a manpage.
|
|
We rely on Inbox::mm nowadays.
|
|
We need to warn when hitting file or socket memory limits
or misconfigurations which clear O_NONBLOCK to make it easier
to diagnose configuration problems. We'll also warn on
some other errors in case bugs creep in...
|
|
While we avoid generating absolute URLs in most cases, our
"git clone" instructions and URL headers in mboxrd files
contain full URLs.
So do the same thing we do for WwwAtomStream and pre-generate
the full URL before Plack::App::URLMap changes $env->{PATH_INFO}
and $env->{SCRIPT_NAME} back to their original values.
Reported-by: edef <edef@edef.eu>
Link: https://public-inbox.org/meta/cover.0f97c47bb88db8b875be7497289d8fedd3b11991.1569296942.git-series.edef@edef.eu/
|
|
This allows us to deal with newlines in config values,
since git-config(1) acquired "-z" support in git v1.5.3.
I'm not sure if it's actually useful in our case, but
maybe some multi-line texts could be added. And newlines
in path names are super useful!
|
|
The httpd-supplied write callback is the leak culprit under Perl
5.16.3. undef-ing it immediately after use keeps a repeated
"git fetch" loop from monotonically increasing memory and FD use
on the Perl shipped with RHEL/CentOS 7.x.
Other endpoints tested showed no increase in memory use under
constant load with "ab -HAccept-Encoding:gzip -k", including the
async psgi_qx code path used by $INBOX_URL/$OBJECT_ID/s/ via
SolverGit module.
|
|
This returns a git-config(1)-compatible file to make it easier
to get started on mirroring an existing public-inbox. Omitting
the "raw" from the URL works, as well, but I'm not sure if
it's very useful.
|
|
We don't need to make unnecesary writes to the git config file
and wear out storage devices every time we run
"public-inbox-index"
|
|
qmail.org seems unavailable.
|
|
Quoting Amitai Schleier, who made this same change in ikiwiki[1],
where lots of the public-inbox highlight code comes from:
> As of 3.51, searchFile() is no longer provided in highlight's Perl
> bindings (at least on NetBSD and OS X, as built from pkgsrc). This
> leaves us falling through to getConfDir(), which has been gone
> rather longer.
>
> From highlight git, it appears searchFile() and getFiletypesConfPath()
> both originated in the 3.14 release. The latter is still available in
> 3.51, and returns the same result searchFile() used to. Switch to it.
So, this should still be compatible with the version of highlight.pm in
Debian, but add support for newer versions as well.
[1]: commit 4d06df9583e6c4145f8c6fc2fd51d7894c0b85ce
Cc: Amitai Schleier <schmonz-web-ikiwiki@schmonz.com>
|
|
The perl-5.16.3-294.el7_6 RPM package on RHEL/CentOS 7 is
affected by a memory leak in Perl when calling `ref' on
blessed references. This resulted in a very slow leak that
manifests more quickly with a nonstop "git fetch" loop.
Use Scalar::Util::blessed to work around the issue.
Tested overnight on a CentOS 7 VM.
cf. https://rt.perl.org/Public/Bug/Display.html?id=114340
|
|
We support "-env" to clear the environment with spawn(),
which causes test failures but no runtime failures
(since "-env" isn't used anywhere in our real code)
Reported-and-tested-by: Alyssa Ross <hi@alyssa.is>
|
|
It was never used, and will not be needed.
|
|
We were emitting the same "<id>mailto:name@domain</id>" tag
for every feed (but not per-feed entry). This could cause
feed readers to mistake the top (news.atom) feed for other
feeds (search results, or per-thread feeds).
This is technically a breaking change for people relying on
per-thread or per-query feeds, but the only alternative is
to remain broken for anybody trying to follow multiple feeds
off the same inbox.
|
|
We need to handle arbitrary integers and case-insensitive
variations of human words to match git-config(1) behavior,
since that's what users would expect given we use config
files parseable by git-config(1).
|
|
Only removing $http->{env} is needed to prevent circular
references. $env->{'psgix.io'} does not need to be deleted
since $env will no longer have any references to it when
->close returns.
|
|
And explain why we need to do that delete in a comment.
|
|
Naming $start_cb consistently helps avoid confusing new readers,
and some comments will help with understanding flow
|
|
All of these circular references are designed to clear
themselves, but these will make actual errors from Devel::Cycle
easier-to-spot.
The circular reference in the limiter {run_queue} is not a real
problem, but we can avoid storing the circular reference until
we actually need to spawn the child, reducing the size of the
Qspawn object while it's in the queue, slightly.
We also do not need to have redundant checks to spawn new
processes, we should only spawn new processes when they're
->start-ed or after waitpid reaps them.
|
|
Generic PSGI servers have $env->{'psgi.errors'}, too,
so ensure they can log errors.
|
|
We don't use the return value in real code since we do waitpid
asynchronously, now. So simplify our runtime code at the cost
of making our test slighly more complex.
|
|
We don't need to hold onto the subprocess environ and
redirects/options for popen_rd after spawning the child process.
I do not expect this to fix problem of leaking unlinked regular
file descriptors (which I still can't reproduce), and it
definitely does not fix the problem of leaking pipe descriptors
(which I also can't reproduce).
This will save an FD sooner on non-public-inbox-httpd servers
which give a non-FD $env->{'psgi.input'}, however
Regardless, it's good to free up memory resources in our own
process ASAP we're done using them.
|
|
EINTR should not happen when using non-blocking sockets like we
do in our daemons, but maybe some OSes allow it to happen and
edge-triggered notifications won't notify us again.
So always retry immediately on EINTR without relying on kqueue
or epoll to notify us, and log any other unrecoverable errors
which may happen while we're at it.
|