Date | Commit message (Collapse) |
|
Since the introduction of over.sqlite3, SearchMsg is not tied to
our search functionality in any way, so stop confusing ourselves
and future hackers by just calling it "PublicInbox::Smsg".
Add a missing "use" in ExtMsg while we're at it.
|
|
When indexing messages without Date: and/or Received: headers,
fall back to using timestamps originally recorded by git in the
commit object. This allows git mirrors to preserve the import
datestamp and timestamp of a message according to what was fed
into git, instead of blindly falling back to the current time.
|
|
Date::Parse falls back to using the local timezone when
it's missing from an email, so only test in a reasonable
TZ (UTC) for server software.
|
|
We reach into the WwwListing package directly to retrieve
that JSON encoder/decoder object, and we can't rely on `use'
since WwwListing loading may fail if Plack is missing.
|
|
We already lazy-load WwwListing for the CGI script, and
hiding another layer of lazy-loading makes things difficult
to do WWW->preload.
We want long-lived processes to do all long-lived allocations up
front to avoid fragmentation in the allocator, but we'll still
support short-lived processes by lazy-loading individual modules
in the PublicInbox::* namespace.
Mixing up allocation lifetimes (e.g. doing immortal allocations
while a large amount of space is taken by short-lived objects)
will cause fragmentation in any allocator which favors large
contiguous regions for performance reasons. This includes any
malloc implementation which relies on sbrk() for the primary
heap, including glibc malloc.
|
|
We need to favor "Transfer-Encoding: chunked" over the value of
the Content-Length header. We should also reject bogus,
duplicate and/or unreasonable values for both these, since they
can trigger unexpected behavior when combined with other HTTP
parsers in proxies such as varnish, nginx, haproxy, etc...
See RFC 7230 (and RFC 2616) for more details:
https://tools.ietf.org/html/rfc7230
https://www.rfc-editor.org/errata_search.php?rfc=7230
|
|
We will occasionally see legit messages with zero lines,
be sure we index that count for NNTP clients.
I'm not sure about bytes being zero (aside from purged
messages), but we should've dealt with that earlier up
the stack.
|
|
Some old emails don't have timezone offsets, since our
Date::Parse code path takes a liberal interpretation of dates,
fallback to using "+0000" as the timezone offset since it's
closer to the actual date of the message than whatever the
current date is.
Reported-by: Leah Neukirchen <leah@vuxu.org>
Link: https://public-inbox.org/meta/87h7zfemur.fsf@vuxu.org/
Fixes: ae80a3fdb53d7014 ("MsgTime.pm: Use strptime to compute the time zone")
|
|
Some strange "From:" lines will cause Email::Address::XS to
leave '<' (and presumably '>') in the address which
git-fast-import won't accept even if quoted. Workaround this
problem by deleting '<' and '>' the same way we delete them for
the ident name.
Reported-by: Leah Neukirchen <leah@vuxu.org>
Link: https://public-inbox.org/meta/87h7zfemur.fsf@vuxu.org/
|
|
Import::remove is a documented interface, and the return
value of the V2Writable work-alike should try to be compatible
with what Import implements.
|
|
Instead, we add CRLF conversion to the only remaining place
which needs it, ViewVCS. This save many redundant ops in in
many places.
The only other place where this mattered was in
View::add_text_body, but we already started doing CRLF
conversions when we added diff parsing and link generation for
ViewVCS. Otherwise, all other places we used this was for
header viewing and Email::MIME doesn't preserve CRLF in headers.
|
|
We need to escape ampersands (and some other characters for href
attributes), so introduce a `mid_href' sub to do just that.
'<', '>' and '"' were always escaped, so there's no risk of tag
or attribute injection, but creative Message-IDs could cause
confusion for some parsers and generate invalid URLs.
Start getting rid of the bloated, over-engineered OO Hval API
while we're at it, I only noticed this bug because I started
killing off Hval->new* callers.
|
|
A long overdue test for behavior established in 2016.
Fixes: 1b28cc7f00a866cb ("view: try assuming UTF-8 for bogus charsets")
|
|
SearchIdx always requires DBD::SQLite, so only require it
after we've passed `require_mods(qw(DBD::SQLite))'.
|
|
XML::Feed pulls in a lot of dependencies, some of which XS.
That makes testing with blead or any non-OS-supplied Perl
installations more time consuming and more difficult because
of the need to have development headers and libraries for
libexpat1 or libxml2.
Performance from libexpat1 or libxml2 for our small tests cases
isn't relevant, either, and the pure Perl XML::TreePP seems up
to the task. It's also available in CentOS 7.x, FreeBSD 11.x,
and Debian, at least.
|
|
The x32 ABI allows users to take advantage of the extra
registers on x86-64 without the bloat of 64-bit pointers and
longs.
This ought to be significant since Perl was designed when 32-bit
was prevalent; and the common structs for ops, hashes, scalars,
and arrays use longs (SSize_t/Size_t) for things which should
never need 64-bits when processing emails.
Debian's x32 port seems to work quite nicely under a chroot
on an amd64 Linux system. All tests pass under x32, now.
|
|
I didn't wait until September to do it, this year!
|
|
It can cause unpredictable behavior and also slow things down.
Followup-to: e4d3be19612b2082 ("t: localize the PI_CONFIG env")
|
|
No need to call ref() and do a string comparison. Add some
extra tests using the {ReadOnly} attribute in DBI.pm.
|
|
Instead of serving $INBOX_DIR/all.git/description, since
$INBOX_DIR/all.git/description is not described in the
default message when it's missing.
|
|
We want to match "GET" and "HEAD" exactly, not requests which
start with "GET" or end with "HEAD". This doesn't seem like
a real problem for public-inboxes which are actually public
data anyways.
|
|
The (currently undocumented) "--no-index" flag did not trigger
the V2Writable->done call necessary to make the import
successful.
Fixes: eea47b676127bcdb ("convert: preserve highwater mark from v1 msgmap")
|
|
This is to be consistent with the `nproc(1)' code path. It also
quiets down a warning from Admin when "-j $JOBS" is specified,
since the master process (which distributes work to shards and
handles OverIdx and Msgmap) is considered a job on its own.
|
|
This is derived from a real-world test case where I encounterd
multiple Message-IDs in a v1 inbox causing regen problems.
Fixes: eea47b676127bcdb ("convert: preserve highwater mark from v1 msgmap")
|
|
If we're reusing the msgmap from a v1 inbox, we also need to
ensure the highwater mark doesn't get doubled in the v1->v2
conversion by internally triggering the equivalent of
"--reindex" on a fresh v2 inbox.
This was needed to convert an indexed v1 inbox which featured
messages with multiple Message-IDs in it. Fresh, unindexed
clones of v1 inboxes would not have been affected by this.
|
|
Lets always have Content-Disposition for files intended
to be downloaded for consumption by non-browsers, such
as pigz, zcat, "git am".
This is also to be consistent with the non-gzipped mbox
$MESSAGE_ID/raw endpoint.
|
|
Apparently I fixed this bug a while back in commit
f94c3a195a25a31d0215cd175938008fca473378 but did
not write tests.
|
|
New epochs are the most likely to have loose objects. git won't
be able to take advantage of pack indices and needs to scan
every alternate for the loose object via open/openat syscalls.
Those syscalls will add up some day when we've got hundreds or
thousands of epochs.
|
|
I'm not sure when `for (<"quoted string/glob/*">)' became
supported, and maybe it was inadvertant, but it fails
with Perl 5.10.1. Just use the glob() function to be
explicit.
|
|
We don't need IO::File for this test, but IO::Handle
is needed for ->autoflush with Perl <5.14.
Note: I haven't tested highlight.pm under 5.10.1 since
it's a weird dependency which isn't easy to install w/o
distro support.
|
|
Perl 5.14+ gained the ability to autoload IO::File
(and IO::Handle) on missing methods, so relying on
this breaks under 5.10.1.
There's no reason to load IO::File or IO::Handle
when built-in perlops work fine and are even a hair
faster.
|
|
Socket::TCP_DEFER_ACCEPT() did not appear in the Socket module
distributed with Perl until 5.14, despite it being available
since Linux 2.4.
|
|
And some more into t/mid.t. PublicInbox::View::msg_html may
change internally, so lets rely on the stable PSGI interface
to test it, rather than a test which reaches deep into the
internals.
|
|
This test will be expanded, and we can take advantage of
run_script to simplify our internal API use.
|
|
We don't call from_attr anywhere outside of tests, so don't
bloat normal processes with it.
|
|
We need to escape wide characters when making attribute names from
filename-looking things in diffstats.
|
|
The class parameter is pointless, especially for an internal
sub which only has one external caller in a test. Add a sub
prototype while we're at it to get some compile time checking.
|
|
popen_rd accepts arbitrary redirects, so we can reuse its
code to setup the pipe end we want to read, saving each
caller a few lines of code compared to calling pipe+spawn.
|
|
Outside of tests, this is only relevant for non-PSGI use, which
may happen someday...
Fixes: cb1c874520153f5c ("inbox: use PublicInbox::Git::host_prefix_url for base_url")
|
|
Some users just want to run -mda, -watch, and/or -nntpd.
Let them run just those without forcing them to pull in a
bunch of dependencies.
|
|
Most spawn and popen_rd callers die on failure to spawn,
anyways, and some are missing checks entirely. This saves
us a bunch of verbose error-checking code in callers.
This also makes popen_rd more consistent, since it already
dies on pipe creation failures.
|
|
We haven't used it in SolverGit, yet, and I'll be reworking it
to work with ->cat_async, instead.
|
|
Better not to duplicate the same logic across different classes.
Also, our git wrapper class is a strange place for
host_prefix_url, but it needs to be usable for coderepos, so
it's there, for now...
|
|
There's a bunch of leftover "require" and "use" statements we no
longer need and can get rid of, along with some excessive
imports via "use".
IO::Handle usage isn't always obvious, so add comments
describing why a package loads it. Along the same lines,
document the tmpdir support as the reason we depend on
File::Temp 0.19, even though every Perl 5.10.1+ user has it.
While we're at it, favor "use" over "require", since it it gives
us extra compile-time checking.
|
|
Yes, there's actually other timezones!
|
|
Creating a hash and iterating through it just to run "git
config" is ugly and slow. Just write out the text file in a
human-friendly way since the git-config file format is stable
and won't break randomly.
|
|
It was no longer used outside of tests, so don't penalize
regular users with the extra function. Just inline it for
t/search.t.
|
|
PublicInbox::Search always loads DBD::SQLite, so we
can't blindly "use" it in t/xcpdb-reshard.t. We also
need to account for that in TestCommon.
|
|
Since the beginning of this project, we've implicitly supported
inboxes with multiple URLs by relying on the Host: header sent
by the client ($env->{HTTP_HOST}).
We now offer the option to explicitly configure multiple URLs for
every inbox along with the ability to do a best-effort match for
matching hostnames.
|
|
It's now possible to use WwwStatic as a standalone PSGI
app to serve static files and recreate the award-winning
web design of https://public-inbox.org/ :>
|