| Date | Commit message (Collapse) |
|---|
|
The goal of this is to eventually remove the $smsg->{mime} field
which is easy-to-misuse and cause memory explosions which
necessitated fixes like commit 7d02b9e64455831d
("view: stop storing all MIME objects on large threads").
|
|
gitweb does the same with $GIT_DIR/description and gitweb.owner.
Allowing UTF-8 description should not cause problems when used
in responses for to the NNTP "LIST NEWSGROUPS" request, either,
since RFC 3977 section 7.6.6 recommends the description be UTF-8
(but does not require it).
Link: https://public-inbox.org/meta/20200528151216.l7vmnmrs4ojw372g@sourcephile.fr/
|
|
PublicInbox::Config.pm::_fill() assumes that address is an array.
Therefore when handling an unset address use an array containing a
single string, instead of a single string.
Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
|
|
PublicInbox::Eml has enough functionality to replace the
Email::MIME-based PublicInbox::MIME.
|
|
It's unnecessary overhead for anything which does Email::MIME
parsing. It was never done for v2 indexing, even though v1->v2
conversions did NOT remove those From_ lines. There was never a
need to remote From_ lines the v1 SearchIdx paths, either.
Hitting a /$INBOX_URL/$MSGID/T/ endpoint with an 18 message
thread reveals a ~0.5% speed improvement. This will become
more apparent when we have a faster MIME parser.
|
|
-Git->new and -Limiter->new will never fail unless there's
an OOM, so using `eval' is incorrect.
|
|
It's probably common to have inboxes initially setup without
these files properly configured, so don't memoize at that stage.
|
|
We want to be able to preload that, as well as to access it
in WwwText for a config comment in the config example.
|
|
Since the introduction of over.sqlite3, SearchMsg is not tied to
our search functionality in any way, so stop confusing ourselves
and future hackers by just calling it "PublicInbox::Smsg".
Add a missing "use" in ExtMsg while we're at it.
|
|
I didn't wait until September to do it, this year!
|
|
It's an old function which only gets called by inboxes w/o
SQLite indices.
|
|
We can use "//=" from Perl 5.10 to simplify the logic for these
methods. The use of chomp() in ->cloneurl was also unnecessary
since split(/\s+/s,...) already removes newlines.
|
|
This allows us to simplify version checking by avoiding
"//" or "||" operators sprinkled around.
|
|
And add a note to remind ourselves to use List::Util::uniq
when it becomes common.
|
|
Inbox.pm accessing the $in_loop variable directly raises
warnings when Inbox is loaded without DS.
|
|
Better not to duplicate the same logic across different classes.
Also, our git wrapper class is a strange place for
host_prefix_url, but it needs to be usable for coderepos, so
it's there, for now...
|
|
Since the beginning of this project, we've implicitly supported
inboxes with multiple URLs by relying on the Host: header sent
by the client ($env->{HTTP_HOST}).
We now offer the option to explicitly configure multiple URLs for
every inbox along with the ability to do a best-effort match for
matching hostnames.
|
|
We need to use $PublicInbox::DS::in_loop instead of ::running().
The latter is not valid for systems with signalfd or kqueue and
is now gone, completely.
Not needing periodic cleanups at all to deal with unlinked pack
indices will be a tougher task...
|
|
EvCleanup only existed since Danga::Socket was a separate
component, and cleanup code belongs with the event loop.
|
|
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
|
|
|
|
Try to remain consistent with our own documentation regarding
v2 git "epochs", first.
|
|
No sense in supporting multiple methods of initialization
for an internal class.
|
|
In case some BOFH decides to randomly create directories
using non-ASCII digits all over the place.
|
|
Don't waste more cycles than necessary if somebody decides to
put non-ASCII digits in their ~/.public-inbox/config
|
|
A constant stream of traffic to either httpd/nntpd would mean
git-cat-file processes never expire. Things can go bad after a
full repack, as a full repack will unlink old pack indices and
git-cat-file does not currently detect unlinked files.
We could do something complicated by recursively stat-ing
objects/pack of every git directory and alternate;
but that's probably not worth the trouble compared to
occasionally restarting the cat-file process.
So simplify the code and let httpd/nntpd expire them
periodically, since spawning a "git-cat-file --batch" process
isn't too expensive. We already spawn for every request which
hits git-http-backend, cgit, and git-apply.
In the future, we may optionally support the Git::Raw module
to avoid IPC; but we must remain careful to not leave lingering
FDs open to unlinked files after repack.
|
|
-index documentation avoid redundant v1 information and refers
readers to apropriate v1/v2 manpages. Search::Xapian can also
be optional, now, as only the PSGI search interface uses it.
Favor "INBOX_DIR" where appropriate, since "REPO_DIR" can be
confused for code repos which we also support.
XAPIAN_FLUSH_THRESHOLD is documented for all relevant
bulk commands.
|
|
* origin/xap-optional:
admin: improve warnings and errors for missing modules
searchidx: do not create empty Xapian partitions for basic
lazy load Xapian and make it optional for v2
www: use Inbox->over where appropriate
nntp: use Inbox->over directly
inbox: add ->over method to ease access
|
|
It's only useful for a corner case in long-running daemons when
an admin decides to compact or vacuum a Xapian or SQLite DB.
As a result, other scripts should run slightly faster. For
instance, this saves about 80ms (2.710s => 2.630s) in t/mda.t
on my remote workstation.
While we're at it, make sure EvCleanup is properly require'd
in Daemon.pm and HTTP.pm and document our use of Devel::Peek.
|
|
We no longer need it since Inbox->recent hits the overview
DB instead of Xapian
|
|
More tests work without Search::Xapian, now.
Usability issues still need to be fixed
|
|
We don't need to rely on Xapian search functionality for the
majority of the WWW code, even. subject_normalized is moved to
SearchMsg, where it (probably) makes more sense, anyways.
|
|
One small step towards making installing Xapian optional for v2
and providing more WWW and NNTP functionality without it.
|
|
We will still return a 404 by default to '/' for compatibility
with users of Plack::App::Cascade or similar. Inboxes are
sorted by modification times to help users detect activity
(similar to the /$INBOX/ topic view).
New configuration options:
* publicinbox.wwwlisting - configure the listing type
* publicinbox.<name>.hide - hide a particular inbox from the listing
See changes to public-inbox-config.pod for full descriptions
of the new options.
Requested-by: Leah Neukirchen <leah@vuxu.org>
https://public-inbox.org/meta/871sdfzy80.fsf@gmail.com/
|
|
For inboxes with SQLite enabled (all v2, and probably most v1);
we can use the overview DB to get the timestamp of the latest
message. It's faster than scanning git branches for commit times,
but not always the same.
|
|
This will make it easier to make command-line tools
from SolverGit.
|
|
Otherwise, long-running but idle git processes may keep unlinked
packs around indefinitely and waste disk space.
|
|
{mapping} overhead is now down to ~1.3M at the end of
a giant thread from hell.
|
|
We can't run cleanup stuff without Danga::Socket.
|
|
Searching across different inboxes is expensive without
SQLite (or Xapian) installed, so avoid doing expensive tree
lookups in git. Since SQLite is required for Xapian
support anyways, we won't need to check Xapian, either.
Sites without SQLite installed will simply 404 if somebody
requests a message which isn't in the current inbox.
|
|
Favor simpler internal APIs this time around, this cuts
a fair amount of code out and takes another step towards
removing Xapian as a dependency for v2 repos.
|
|
OFFSET in SQLite gets painful to deal with. Instead,
rely on timestamps (from Received:) for pagination.
This also sets us up for more precise Date searching
in case we want it.
|
|
This ought to provide better performance and scalability
which is less dependent on inbox size. Xapian does not
seem optimized for some queries used by the WWW homepage,
Atom feeds, XOVER and NEWNEWS NNTP commands.
This can actually make Xapian optional for NNTP usage,
and allow more functionality to work without Xapian
installed.
Indexing performance was extremely bad at first, but
DBI::Profile helped me optimize away problematic queries.
|
|
This is a smaller improvement than the landing /$INBOX/ page
because full message bodies are shown; but still saves around
100ms for my system with LKML.
|
|
This saves over 400ms on my system with the full LKML
with over 2.8 million messages.
|
|
Back in the day, we compressed long Message-IDs to SHA-1
hexdigests for the URL. This now redirects to a 301 in
the hopes we can remove these checks some day to reduce
overhead.
|
|
Too many similar functions doing the same basic thing was
redundant and misleading, especially since Message-ID is
no longer treated as a truly unique identifier.
For displaying threads in the HTML, this makes it clear
that we favor the primary Message-ID mapped to an NNTP
article number if a message cannot be found.
|
|
The Message-ID mapped to an NNTP article number is stronger,
so we will favor that for attachment lookups.
|
|
We do not need to care about ghosts at multiple call sites; they
cannot have a {blob} field and we've stored the blob field in
Xapian since SCHEMA_VERSION=13.
|
|
This will require multiple client invocations, but should reduce
load on the server and make it easier for readers to only clone
the latest data.
Unfortunately, supporting a cloneurl file for externally-hosted
repos will be more difficult as we cannot easily know if the
clones use v1 or v2 repositories, or how many git partitions
they have.
|