Date | Commit message (Collapse) |
|
The ->mset method always returns a Xapian mset nowadays, so
naming a parameter {mset} is too confusing. As it does with
MiscSearch, setting the {relevance} parameter to -1 now sorts by
ascending docid order. -2 is now supported for descending
docid order, too, since it may be useful for lei users.
|
|
SearchView will set it to `undef', others will set the 'mset'
option (for the ->mset method :P) to 2 which causes {relevance}
to be ignored.
And the 'mset' option is poorly named now that the message
is named ->mset...
|
|
As with ExtSearch, MiscSearch lacks a janky cleanup timer of
PublicInbox::Inbox objects, leading to info about
inboxes/newsgroups going stale. Fortunately, we don't use
MiscSearch very heavily, yet.
In the future, we may be able to detect new inboxes without
having to SIGHUP or restart daemons using MiscSearch.
|
|
Since ExtSearch lacks the janky cleanup timer of
PublicInbox::Inbox objects, its search results get stale.
Reopen the Xapian DB on every ->search call for now, as
reducing reopen calls doesn't seem worth the complexity.
The Xapian::Database::reopen operation itself takes only ~50us
on my old workstation with 3 shards totaling <200GB. Other
parts of Xapian dominates the search time, so the reopen seems
inconsequential with single-digit shard counts.
|
|
If "--all" is specified to index all inboxes, implicitly choose
the configured [extindex "all"] external index since "--all" is
incompatible with specifying inbox directories on the
command-line.
|
|
This makes diagnosing --watch problems easier when there's
50K inboxes by avoiding the lengthy scan (which is the reason
--watch exists in the first place).
|
|
With --watch, the output may be redirected to a pipe or socket
which Perl may decide to buffer. Ensure Perl doesn't buffer
these outputs since they can provide real-time status updates
in response to signals or FS activity.
|
|
We need to clobber the SIGUSR1 resync queue on SIGHUP to
invalidate old inbox objects. Furthermore, the lengthy
initial scan needs to ignore signals intended for the
event loop to avoid unexpected behavior. Finally, add
some progress output to inform users on the terminal
to inform users' of progress.
|
|
This reuses existing InboxIdle infrastructure to update external
indices based on per-inbox updates. This is an alternative to
auto-updating external indices via the -index command and also
works with existing uses of -mda and public-inbox-watch.
Using inotify (or EVFILT_VNODE) allows watching thousands of
inboxes without having to scan every single one at every
invocation.
This is especially beneficial in cases where an external index
is not writable to the users writing to per-inbox indices.
|
|
Encode::MIME::Header::_decode_octets did not correctly default
to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit
0c541dc5633a341cf44b818014b58e7f8be532e9). Provide the default
again to work with older Perls.
Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
(cherry picked from commit 46bd595f57cc3d425754b0d0770c125616e74448)
|
|
While git 1.8.5 learned --get-urlmatch, git did not learn to
match URLs against wildcards until 2.26. So only depend on
1.8.5 for this test since 2.26 is too new.
Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
(cherry picked from commit 382a2bdd54cfb6c28a935c2b9fe4f1b1c2f469c4)
|
|
We don't have to replace a bunch of existing watches
with identical new ones. On Linux with Linux::Inotify2
installed, this avoids a storm of inotify_add_watch(2)
and inotify_rm_watch(2) syscalls on SIGHUP with -imapd
and "-extindex --watch"
(cherry picked from commit 10bf54305da8422d9ece6b809996092c1c4b1786)
Note: this seems to fix missed wakeups with many watches,
so it's in the stable branch.
Link: https://public-inbox.org/meta/20201226201115.GA30142@dcvr/
|
|
It may not be obvious to users a ENOSPC error is from hitting
a (tunable) kernel-imposed limit on inotify watches, and not
some storage device running out of space. Give them a hint
here to reduce our own support burden.
(cherry picked from commit 5e05c2eb58a450849f1826f3d02ed62b814b6617)
|
|
->on_inbox_unlock callbacks could clobber $_, and this seems to
fix a problem with -extindex --watch failing to index some
inboxes after SIGHUP reload.
(cherry picked from commit b5e960f50289434025f5904c8c1311e4c8a02b82)
|
|
The Debian package linter (lintian) emits the following warning:
W: bad-whatis-entry
N:
N: A manual page should start with a NAME section, which lists the
N: program name and a brief description. The NAME section is used to
N: generate a database that can be queried by commands like apropos and
N: whatis. You are seeing this tag because lexgrog was unable to parse
N: the NAME section.
N:
N: Manual pages for multiple programs, functions, or files should list
N: each separated by a comma and a space, followed by \- and a common
N: description.
N:
N: Listed items may not contain any spaces. A manual page for a two-level
N: command such as fs listacl must look like fs_listacl so the list is
N: read correctly.
N:
N: Refer to the lexgrog(1) manual page, the groff_man(7) manual page, and
N: the groff_mdoc(7) manual page for details.
N:
N: Severity: warning
N:
N: Check: documentation/manual
N:
N: Renamed from: manpage-has-bad-whatis-entry
N:
for public-inbox-v1-format and public-inbox-v2-format.
Adapt the descriptions to make lexgrog and so lintian happy.
(cherry picked from commit 1350f5ab09f72c75ac2cd6c88f6a2b9e198fef55)
|
|
/$INBOX/manifest.js.gz should not attempt to match every inbox
in the domain (or every inbox); that is for /manifest.js.gz
(without a /$INBOX prefix).
Fixes: f303b4add8ea1883 ("wwwlisting: avoid hogging event loop")
(cherry picked from commit 3ce4c38119f13d419bb865a0aa9b66feff339308)
|
|
On Debian (at least) perl documentation is supposed to be installed in
section 3pm. With the build system hardcoding this to 3 instead this
results in a warning by the Debian package linter:
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Git.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::Import.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section usr/share/man/man3/PublicInbox::SaPlugin::ListMirror.3.gz:74 3 != 3pm
W: public-inbox: wrong-manual-section ... use --no-tag-display-limit to see all (or pipe to a file/program)
So use "$(MAN3EXT)" instead of "3".
(cherry picked from commit cd12cf22c4d161b7aec819f45628b983f0bd675d)
|
|
Plack::Test::ExternalServer doesn't depend on
Plack::Middleware::ReverseProxy, so we need to account for
some warnings in stderr if P::M::RP is missing.
(cherry picked from commit d26c2837f479b41182946a6540aad95d34b2b594)
|
|
We don't actually need Net::Server::Daemonize to support
the --daemonize flag, since the daemonize() sub provided
by N::S::D doesn't exactly do the things we want.
(cherry picked from commit ab11caeb8ae114087636748df4c956d0e625b237)
|
|
(cherry picked from commit 4df1904582c4d7bec64e05d1566cd48c8c2e888e)
|
|
We must use the result of link_refs() since it can trigger
merge_threads() and invalidate $old_tid. In case
merge_threads() isn't triggered, link_refs() will return
$old_tid anyways.
When rethreading and allocating new {tid}, we also must update
the row where the now-expired {tid} came from to ensure only the
new {tid} is seen when reindexing subsequent messages in
history. Otherwise, every subsequently reindexed+rethreaded
message could end up getting a new {tid}.
Reported-by: Kyle Meyer <kyle@kyleam.com>
Link: https://public-inbox.org/meta/87360nlc44.fsf@kyleam.com/
(cherry picked from commit 9356ec0cc5afc95a8fd398ddf898942ef0acdb74)
|
|
This prevents `<img src=' tags from being used to deep-link
image attachments from HTML outside of the current host and
reduces potential for abuse.
Some browsers (e.g. Firefox) favor content detection and will
display images irrespective of the Content-Type header being
"application/octet-stream", and "Content-Disposition: attachment"
doesn't stop them, either.
Tested with dillo and Firefox.
Reported-by: Leah Neukirchen <leah@vuxu.org>
(cherry picked from commit 46cbc5a7a4ba917bd7154be3b6e6898420ff85d3)
|
|
Recent (2020) versions of Email::MIME (and/or dependencies)
have different behavior than historical versions which seem
to be less DWIM and perhaps technically more correct. We'll
retain historical behavior for now, since it doesn't seem to
cause real problems and DWIM-ness is often required to make
sense of historical mail.
Tested on a FreeBSD 11.4 VM with the following packages:
p5-Email-MIME-1.949
p5-Email-MIME-ContentType-1.024_1
p5-Email-MIME-Encodings-1.315_2
(cherry picked from commit 5c4874309c4dd7aa8b9acd6bb97bd512360ab08a)
|
|
Perhaps some NNTP clients would be unhappy with the old value
"y". So use a bit more bandwidth+space to use the server-name
and historical "!not-for-mail" tail-entry to better conform to
a published RFC.
Reported-by: Andrey Melnikov <temnota.am@gmail.com>
(cherry picked from commit 47559005ee678592808c351daa1c9f2682beb248)
|
|
...instead of spaces. This is specified in RFC 5536 3.1.4.
Include references to RFC 1036, 5536 and 5537 in our docs while
we're at it.
Reported-by: Andrey Melnikov <temnota.am@gmail.com>
Link: https://public-inbox.org/meta/CA+PODjpUN5Q4gBFQhAzUNuMasVEdmp9f=8Uo0Ej0mFumdSwi4w@mail.gmail.com/
(cherry picked from commit 44227c2624e4f954943d632cd5335396351373be)
|
|
Apparently they happen (triggered by my -imapd instance), so
bail out by closing the underlying socket rather than stopping
the event loop and daemon process.
(cherry picked from commit c51c22c349529d9c377160abcc7961a6ca7b7d5c)
|
|
When dupe-finder was switched from ->search->{over_ro} to ->over, the
database handle was dropped. Restore it because a spot downstream
uses it.
Fixes: 73e3a6ed6e95adc6 (use more idiomatic internal API for ->over access)
(cherry picked from commit 0438ee2b1a5fbeb587ee62aa3e267ec09d281bd9)
|
|
We don't want a List-Id value being confused with a Xapian
term prefix, here.
Followup-to: 8b06cda3a3af3f0e ("mda: match List-Id insensitively")
(cherry picked from commit 447d049b70c6723a0de8e28aa7db1a4130394c4b)
|
|
Oops :x
(cherry picked from commit 28dec3a76687617a3a697058964dc0bd1351eb6b)
|
|
This ought to save a few cycles if a client disconnects while
in the middle of a (UID) FETCH. This avoids:
Can't call method "git" on an undefined value at .../PublicInbox/IMAP.pm
errors in stderr.
(cherry picked from commit 34880bc83077eac5739deca69f66df7685965064)
|
|
It's `d' for deletes, not `a'.
(cherry picked from commit 9dd93e4399c67abf74df6832181d442f81e0aa33)
|
|
This follows -watch commit b70473ab8296d31ebb600adb4fa8fe0ac5935ca8
to match List-Id headers case-insensitively.
Reported-by: Konstantin Ryabitsev <konstantin@linuxfoundation.org>
Link: https://public-inbox.org/meta/20200921180152.uyqluod7qxbwqubo@chatter.i7.local/
(cherry picked from commit 8b06cda3a3af3f0e88b599a01f03edb7fec4f2c4)
|
|
(cherry picked from commit dc93c36eb62d36e649b9500b7f190687a3fbcffd)
|
|
I should've dropped "PENDING" notes before the 1.6 release;
they're dropped now, and a note is added to remind my future
self to drop them before 1.7.
(cherry picked from commit 3b5d3d1910f1db526a488142c01f42db5255ac72)
|
|
Our code doesn't support multi-values for these, and having
unexpected arrays leads to unexpected results (e.g. showing
stuff like "ARRAY(0xDEADBEEFADD12E55)" in user interfaces). So
warn and only use the last value (matching git-config(1)
behavior without `--get-all').
(cherry picked from commit c8fe4673056571ed251c3a20cb0bbec586c60bd6)
|
|
We host our own -imapd manpage, and we started using a few more
git commands (fast-import for ages). We'll also need to link to
manpages.debian.org and live with long URLs for a few
non-standard manpages in software we reference.
(cherry picked from commit 31065006a3654968c0c4bbfe2d7399a3b326ce18)
|
|
It's another read-only daemon, and it may see more usage than
-nntpd as more users have IMAP support than NNTP.
(cherry picked from commit 1fcb5694c2fdd4200820f777f27f9f55cd51b6ae)
|
|
Oops :x
(cherry picked from commit a7cd8c6a2bfa1ebfe2edeb0d7a93de17fd334683)
|
|
Encode::MIME::Header::_decode_octets did not correctly default
to Encode::FB_DEFAULT until Encode 2.93 (perl5.git commit
0c541dc5633a341cf44b818014b58e7f8be532e9). Provide the default
again to work with older Perls.
Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
|
|
While git 1.8.5 learned --get-urlmatch, git did not learn to
match URLs against wildcards until 2.26. So only depend on
1.8.5 for this test since 2.26 is too new.
Reported-by: Ali Alnubani <alialnu@nvidia.com>
Link: https://public-inbox.org/meta/DM6PR12MB49106F8E3BD697B63B943A22DADB0@DM6PR12MB4910.namprd12.prod.outlook.com/
Tested-by: Ali Alnubani <alialnu@nvidia.com>
|
|
As with CORE::die and $SIG{__DIE__}, it turns out CORE::warn is
safe to use inside $SIG{__WARN__} handlers without triggering
infinite recursion. So fall back to reusing CORE::warn instead
of creating a new sub.
|
|
->on_inbox_unlock callbacks could clobber $_, and this seems to
fix a problem with -extindex --watch failing to index some
inboxes after SIGHUP reload.
|
|
We don't have to replace a bunch of existing watches
with identical new ones. On Linux with Linux::Inotify2
installed, this avoids a storm of inotify_add_watch(2)
and inotify_rm_watch(2) syscalls on SIGHUP with -imapd
and "-extindex --watch"
|
|
It may not be obvious to users a ENOSPC error is from hitting
a (tunable) kernel-imposed limit on inotify watches, and not
some storage device running out of space. Give them a hint
here to reduce our own support burden.
|
|
extindex users will likely want to use indexlevel=basic for
per-inbox indices, however extindex itself doesn't support basic
index level (yet?). Let's ensure we don't trip up extindex
users who specify "-L basic" on the -index command-line.
|
|
We only rely on git-rev-parse to resolve symbolic names ("HEAD")
to a SHA-* git commit ID. We'll assume any git commit IDs we
get from SQLite DBs are valid and let "git-log" fail if it
isn't.
|
|
Negation in flag names are confusing, but trying to deviate from
the DB_NO_SYNC name used by Xapian is also confusing.
|
|
We'll count the number of log changes (regardless of index or
unindex) and only attach inboxes to ExtSearchIdx objects when
they get new work. We'll also reduce lock bouncing and only
update external indices after all per-inbox indexing is done.
This also updates existing v2 indexing/unindexing callers
to be more consistent and ensures unindex log entries update
per-inbox last commit information.
|
|
Most distros ship with low RLIMIT_NOFILE limits and surprises
may lurk for admins who configure many inboxes. Keep FD usage
under control to avoid EMFILE errors at inopportune times during
reindex.
From what I can tell, this is the only place where extindex can
have unpredictable FD growth when there's thousands of inboxes,
and it's in an extremely rare code path.
|
|
This will make attach_inbox faster for no-op calls. It also
helps us avoid races in case msgmap or over.sqlite3 gets
unlinked while -extindex is running.
|