| Date | Commit message (Collapse) |
|---|
|
It works around some bugs in older Email::MIME which we'll
find useful.
|
|
Probably unnecessary, but set binmode for consistency across
platforms.
|
|
Leaking these pipes to child processes wasn't harmful, but
made determining relationships and dataflow between processes
more confusing.
|
|
Relying more on Xapian requires retrying reopens in more
places to ensure it does not fall down and show errors to
the user.
|
|
Fortunately, Xapian multiple database support makes things
easier but we still need to handle the skeleton DB separately.
|
|
This should give us an idea of how much a problem deduplication
will be.
|
|
We want to reduce the time in the main V2Writable process
spends writing to the pipe, as the main process itself is
the primary source of contention.
While we're at it, always flush after writing to ensure
the child sees it at once. (Grr... Perl doesn't use writev)
|
|
Instead of relying on the git object_id hash to partition,
round-robin to these partitions based on the NNTP article
number. This reduces the partition pipes as a source of
contention when two (or more) sequential messages end up
going to the same partition.
|
|
The parallelization requires splitting Msgmap, text+term
indexing, and thread-linking out into separate processes.
git-fast-import is fast, so we don't bother parallelizing it.
Msgmap (SQLite) and thread-linking (Xapian) must be serialized
because they rely on monotonically increasing numbers (NNTP
article number and internal thread_id, respectively).
We handle msgmap in the main process which drives fast-import.
When the article number is retrieved/generated, we write the
entire message to per-partition subprocesses via pipes for
expensive text+term indexing.
When these per-partition subprocesses are done with the
expensive text+term indexing, they write SearchMsg (small data)
to a shared pipe (inherited from the main V2Writable process)
back to the threader, which runs its own subprocess.
The number of text+term Xapian partitions is chosen at import
and can be made equal to the number of cores in a machine.
V2Writable --> Import -> git-fast-import
\-> SearchIdxThread -> Msgmap (synchronous)
\-> SearchIdxPart[n] -> SearchIdx[*]
\-> SearchIdxThread -> SearchIdx ("threader", a subprocess)
[* ] each subprocess writes to threader
|
|
It is less confusing without the clobber assignment; and
PublicInbox::MIME exists to workaround bugs in older
Email::MIME (which is in Debian 9 (stretch))
|
|
This will let us quickly test between v2 and v1 inboxes.
|
|
This is too slow, currently. Working with only 2017 LKML
archives:
git-only: ~1 minute
git + SQLite: ~12 minutes
git+Xapian+SQlite: ~45 minutes
So yes, it looks like we'll need to parallelize Xapian indexing,
at least.
|
|
Since we'll be adding new repositories to the `alternates' file
in git, we must restart the `git cat-file --batch' process as
git currently does not detect changes to the alternates file
in long-running cat-file processes.
Don't bother with the `--batch-check' process since we won't be
using it with v2.
|
|
Wrap the old Import package to enable creating new repos based
on size thresholds. This is better than relying on time-based
rotation as LKML traffic seems to be increasing.
|
|
It won't be in v2
|
|
In general, they are, but there's no way for or general purpose
mail server to enforce that. This is a step in allowing us
to handle more corner cases which existing lists throw at us.
|
|
This likely has no real world implications, though, as we
fall back to Msgmap lookups anyways.
Broken since commit 7eeadcb62729b0efbcb53cd9b7b181897c92cf9a
("search: remove unnecessary abstractions and functionality")
|
|
Despite email not existing until 1971; "Jan 1, 1970 00:00:00"
seems like a common default timestamp for some test emails
to use as a Date: header.
|
|
There's a lot of crap in archives and git-fast-import
accepts empty names and email addresses for authors
just fine.
|
|
Big lists are orders of magnitude more efficient with v2.
|
|
For LKML, it appears we need an even more liberal parser than
RFC2822 date parser in git. I have not validated Date::Parse
parses dates correctly, but this at least prevents
git-fast-import(1) from choking.
|
|
There's a lot of weird characters which show up in LKML archives
which we did not support before. Furthermore, allow spaces
before the '>' in the From: line as at least some non-spam
poster used it.
|
|
I decided not to copy the notmuch implementation regarding
serialization of integers to Xapian metadata.
|
|
This will allow easier-compatibility with v2 code which will
introduce content_id as the unique identifier.
The old "XMID" becomes "XM" as a free text searchable term.
"Q" becomes "XMID" as a boolean prefix.
There's no user-visible changes in this, but there needs to
be a schema version bump later on...
(more changes planned which can affect v1)
|
|
Wrap "get-mark" and "checkpoint" commands for git-fast-import
while documenting/cementing parts of the API.
|
|
This can be useful for getting baseline of performance
of just Email::MIME and Date: header parsing. We'll need
to do some Date: header parsing for LKML since there are
some wonky date formats which causes the git RFC822 parser
to choke.
|
|
Oops, I guess this code was never called and may not be
needed. But for now, import it so it can run properly.
|
|
|
|
Check for this before doing the Xapian-based v2 importer.
|
|
Call order will need to change a bit since this is going to be
tied to Xapian
|
|
We'll reuse this class in v2, but won't be utilizing
per-git-repository ssoma.lock files.
Meanwhile, stop treating ::Inbox objects as an afterthought
and allow importing name and email into them.
|
|
For machines which have never seen ssoma, they don't need the
index so stop creating it.
|
|
The mboxes I got from cregit have two spaces after the email
address, while the "git format-patch" output I'm used to dealing
with only has one space.
It's still a "strict" match in that it checks for something
resembling a timestamp, but it relaxes the number of spaces
between the email address and date.
|
|
Hostnames can contain '-' and this allows public-inbox-watch(1)
to work on machines which generate Maildir files with '-' in
them.
|
|
I'll be working as a contractor for The Linux Foundation on v2
in an effort to support LKML and associated lists.
|
|
Might as well, this release is mostly to serve as a checkpoint
for the start of new development on v2 stuff mentioned in the
TODO.
|
|
|
|
This can be useful for tarball distributions which lack full git
history.
|
|
Using update-copyrights from gnulib
While we're at it, use the SPDX identifier for AGPL-3.0+ to
ease mechanical processing.
|
|
Sometimes, it can be desirable to jump directly to the "nested"
view when viewing a thread skeleton. This makes it possible.
While we're at it, shorten some of the text to ensure it still
fits in 80 columns.
|
|
We leave the mailto: link out when obfuscating address, so
do not stuff the "</pre>" closing tag into it. Instead,
keep the closing tag in the same context as the opening one,
making it easier to keep track of.
|
|
Namely, we do not want to obfuscate the mail address of the
site itself.
|
|
This makes the wording less confusing when showing archives
for lists where the convention is reply-to-list.
I still hate reply-to-list, but it's still better than no
archives or list at all.
|
|
This can allow streaming parsers (SAX) to work a little more
efficiently as they can handle/discard all the metadata before
the big content.
|
|
I still hate that CSS is over-used, but colors are useful
and perhaps using them for highlighting won't be too bad;
but user-supplied colors will ALWAYS be supported.
|
|
Inspired by interest in LKML archival:
https://public-inbox.org/meta/d5546b24-5840-4ae9-d25b-5e3e737ed73b@linuxfoundation.org
|
|
Obfuscating username portions of the email address leads
to having subsequent parts of the address not being obfuscated;
which could mean we show someone else's email entirely.
In other words, obfuscating "john.doe@example.com" becomes
might mean "doe@example.com" is picked up by scanners.
In other news, email address obfuscation is still a horrible
usability issue and only exists to appease misguided people.
|
|
It is usually pointless to replace a single word with a '"' character.
|
|
This should prevent crawlers (including most robots.txt ignoring
ones) from burning our CPU time without severely compromising
usability for humans.
|
|
Some search results are gigantic, and search engines are
unlikely to be able to handle gzipped mboxes anyways.
|