| Date | Commit message (Collapse) |
|---|
|
Dikshunarees R gude!
|
|
Using `undef EXPR' like a function call actually frees the heap
memory associated with the scalar, whereas `$sv = undef' or
`$sv = ""' will hold the buffer around until $sv goes out
of scope.
The `sv_set_undef' documentation in the perlapi(1) manpage
explicitly states this:
The perl equivalent is "$sv = undef;". Note that it doesn't
free any string buffer, unlike "undef $sv".
And I've confirmed by reading Dump() output from Devel::Peek.
We'll also inline the old index_body sub in SearchIdx.pm to make
the scope of the scalar more obvious.
This change saves several hundred kB RSS on both -index and
-httpd when hitting large emails with thousands of lines.
|
|
These seem mostly harmless since Perl will just truncate the
match and start a new one on a newline boundary in our case.
The only downside is we'd end up with redundant <span> tags in
HTML.
Limiting the number of line matched ourselves with `{1,$NUM}'
doesn't seem prudent since lines vary in length, so we continue
to defer the job of limiting matches to the Perl regexp engine.
I've noticed this warning in practice on 100K+ line patches to
locale data.
|
|
We do not need run mid_clean() since mid_mime() uses mids()
to extract the msgid from inside the angle brackets.
|
|
For sharded v2 repositories with few-enough messages, it is
possible for shard[0] to go unused and never trigger the
->commit_txn_lazy to set the indexlevel field in Xapian
metadata.
So set it immediately at initialization and avoid this case.
While we're at it, avoid triggering needless pwrite syscalls
from ->set_metadata by checking with ->get_metadata, first.
|
|
We can finally get rid of the awkward, ad-hoc use of V2Writable,
SearchIdx, and OverIdx args for passing {cotime} and {autime}
between classes.
We'll still use those git time fields internally within
V2Writable and SearchIdx for (re)indexing, but that's not
worth avoiding as a fallback.
|
|
We can pass blessed PublicInbox::Smsg objects to internal
indexing APIs instead of having long parameter lists in some
places. The end goal is to avoid parsing redundant information
each step of the way and hopefully make things more
understandable.
|
|
Favor `$smsg->{mid}' instead of `$mid0' to reduce parameters
down-the-line, but favor passing the Email::MIME::Header object
around instead of relying on the bloat-prone `$smsg->{mime}'
and calling ->header_obj on it.
|
|
No need to pass extra parameters to this method, since
smsg has universal meanings for {blob} and {mid}.
|
|
Since the introduction of over.sqlite3, SearchMsg is not tied to
our search functionality in any way, so stop confusing ourselves
and future hackers by just calling it "PublicInbox::Smsg".
Add a missing "use" in ExtMsg while we're at it.
|
|
When indexing messages without Date: and/or Received: headers,
fall back to using timestamps originally recorded by git in the
commit object. This allows git mirrors to preserve the import
datestamp and timestamp of a message according to what was fed
into git, instead of blindly falling back to the current time.
|
|
I didn't wait until September to do it, this year!
|
|
Some people use "--{src,dst}-prefix=", try to deal with those
since git-apply can handle them when called by solver.
|
|
We already capture filenames on the lines beginning
with "---" and "+++", so it's redundant work to capture
filenames from "diff --git ..." lines.
|
|
We don't confuse human readers with the Xapian schema version.
We also want to make it obvious this is the version of the inbox
we're indexing, these are Search or SearchIdx objects, not Inbox
objects.
|
|
This allows us to simplify version checking by avoiding
"//" or "||" operators sprinkled around.
|
|
Most spawn and popen_rd callers die on failure to spawn,
anyways, and some are missing checks entirely. This saves
us a bunch of verbose error-checking code in callers.
This also makes popen_rd more consistent, since it already
dies on pipe creation failures.
|
|
There's a bunch of leftover "require" and "use" statements we no
longer need and can get rid of, along with some excessive
imports via "use".
IO::Handle usage isn't always obvious, so add comments
describing why a package loads it. Along the same lines,
document the tmpdir support as the reason we depend on
File::Temp 0.19, even though every Perl 5.10.1+ user has it.
While we're at it, favor "use" over "require", since it it gives
us extra compile-time checking.
|
|
It shouldn't be possible for v1 inboxes to have multiple matches
for a given Message-ID, so the sub would only get called once,
but strange things could happen in 2112 :>
|
|
Use the parameter names from the Search::Xapian::TermGenerator
manpage for our local variables instead of confusing names...
|
|
We now use the same regexp View::add_text_body uses.
|
|
Procedural function calls allow prototype checking, and
our add_message prototype was totally wrong to begin with.
Convert most of the "$self->index_*" calls to "index_*($self"
While we're at it, use "//=" to avoid some "unless" statements.
|
|
This ought to save some memory, but it's probably lost in the
noise given the cost of indexing. Regardless it still reduces
the indentation level and makes future changes easier to read.
|
|
As discovered by solver bug hunting, "git apply" also handles
the case where blank lines w/o leading space are treated as diff
context, apparently because GNU diff once did it:
https://public-inbox.org/git/b507b465f7831612b9d9fc643e3e5218b64e5bfa/s/
|
|
Xapian upstream is slowly phasing out the XS-based Search::Xapian
in favor of the SWIG-generated "Xapian" package. While Debian and
both FreeBSD have Search::Xapian, OpenBSD only includes the "Xapian"
binding.
More information about the status of the "Xapian" Perl module here:
https://trac.xapian.org/ticket/523
|
|
The "++" is not yet available in the SWIG-based "Xapian.pm" Perl
bindings, so use "++" where it's supported in both the XS
(Search::Xapian) and SWIG-based Xapian binding.
|
|
git_unquote works in-place, and we sometimes see strange
filenames, or badly munged diffs with terminal escape
characters (for colorization) end up in emails.
|
|
While I've never seen "git log" fail on its own, it could happen
one day and we should be prepared to abort indexing when it
happens.
Beef up tests for t/spawn.t to ensure close() behaves
on popen_rd the way we expect it to.
|
|
Since we replace extra Message-ID headers with X-Alt-Message-ID
to placate NNTP clients, we should allow searching and indexing
on X-Alt-Message-ID just like we do with Message-ID.
|
|
"mainrepo" ws a bad name and artifact from the early days when I
intended for there to be a "spamrepo" (now just the
ENV{PI_EMERGENCY} Maildir). With v2, "mainrepo" can be
especially confusing, since v2 needs at least two git
repositories (epoch + all.git) to function and we shouldn't
confuse users by having them point to a git repository for v2.
Much of our documentation already references "INBOX_DIR" for
command-line arguments, so use "inboxdir" as the
git-config(1)-friendly variant for that.
"mainrepo" remains supported indefinitely for compatibility.
Users may need to revert to old versions, or may be referring
to old documentation and must not be forced to change config
files to account for this change.
So if you're using "mainrepo" today, I do NOT recommend changing
it right away because other bugs can lurk.
Link: https://public-inbox.org/meta/874l0ice8v.fsf@alyssa.is/
|
|
|
|
Now that the code matches Xapian terminology, ensure
our comments match, too.
|
|
Another step towards keeping our internal data structures
consistent with Xapian naming.
|
|
PublicInbox::Inbox objects have minimal dependencies, so
drop code to support old tests which existed before the
PublicInbox::Inbox object came into existence.
|
|
Make it easier to detect if a partition is corrupt.
|
|
And use it from Admin.
It's easy to tell what indexlevel=basic is from unconfigured
inboxes, but distinguishing between 'medium' and 'full' would
require stat()-ing position.* files which is fragile and
Xapian-implementation-dependent.
So use the metadata facility of Xapian and store it in the main
partition so Admin tools can deal better with unconfigured
inboxes copied using generic tools like cp(1) or rsync(1).
|
|
We can't pass an empty string to `git merge-base --is-ancestor'
AFAIK, this did NOT present issues in the current test suite.
|
|
Emit information about reindexing git revision ranges when used
with xcpdb. Additionally, distinguish Xapian copy output from
v2 git epoch counting by increasing directory context info.
For now, v1 batches batches are emitted. v2 indexing is still
missing progress reporting for batches, as the data structures
for reindexing would benefit from a refactoring, first.
This does not currently affect the use of public-inbox-index,
but may in the future.
|
|
Copying an entire Xapian DB takes a long time, so update our
reindexing code to support partial reindexing, snapshot the
pre-copydatabase git revisions, perform the lengthy copy,
and do a partial reindex when the copy + renames are done.
|
|
* origin/xap-optional:
admin: improve warnings and errors for missing modules
searchidx: do not create empty Xapian partitions for basic
lazy load Xapian and make it optional for v2
www: use Inbox->over where appropriate
nntp: use Inbox->over directly
inbox: add ->over method to ease access
|
|
We only compress in OverIdx, now; since we no longer do overview
stuff in Xapian (and Xapian compresses document data, anyways).
|
|
No point in leaving a mess of empty directories when Xapian
doesn't load.
|
|
More tests work without Search::Xapian, now.
Usability issues still need to be fixed
|
|
We were reindexing the full history every invocation of -index
when Xapian was not used because we were incorrectly relying on
'last_commit' metadata stored in Xapian.
Rewrite the indexing logic to be less confusing while we're
at it, since we rely on `git merge-base --is-ancestor' nowadays.
Furthermore, we need to handle message removals from the
overview index correctly when Xapian is not in use.
Co-authored-by: Eric W. Biederman <ebiederm@xmission.com>
|
|
We'll be using it outside of searchidx...
|
|
* commit 'mem':
view: more culling for search threads
over: cull unneeded fields for get_thread
searchmsg: remove unused fields for PSGI in Xapian results
searchview: drop unused {seen} hashref
searchmsg: remove Xapian::Document field
searchmsg: get rid of termlist scanning for mid
httpd: remove psgix.harakiri reference
|
|
This allows v1 tests to continue working on git 1.8.0 for
now. This allows git 2.1.4 packaged with Debian 8 ("jessie")
to run old tests, at least.
I suppose it's safe to drop Debian 7 ("wheezy") due to our
dependency on git 1.8.0 for "merge-base --is-ancestor".
Writing V2 repositories requires git 2.6 for "get-mark"
support, so mask out tests for older gits.
|
|
We don't need to be carrying this around with the many SearchMsg
objects we have. This saves about 20K from a large SearchView
"&x=t" response.
|
|
The new t/*filter_rubylang.t tests call -index immediately
after -init, which causes confusing messages to show up to
the end user.
Check the validity of the ref before calling "git-log".
|
|
I've found two examples on https://lore.kernel.org/lkml/
where the messages declared themselves to be "multipart/mixed"
but were actually plain text:
<87llgalspt.fsf@free.fr>
<200308111450.h7BEoOu20077@mail.osdl.org>
With the mboxrd downloaded, mutt is able to view them without
difficulty.
Note: this change would require reindexing of Xapian to pick up
the changes. But it's only two ancient messages, the first was
resent by the original sender and the second is too old to be
relevant.
|