| Date | Commit message (Collapse) |
|---|
|
Hopefully more folks can download and run public-inbox,
nowadays.
|
|
Just having "limiter" in the prefix may confuse
it with something else. Use the full prefix to
avoid this confusion.
|
|
We want to encourage users to serve repositories. So enable
bitmaps by default so performance suffers less with smart HTTP.
|
|
We'll keep supporting "publicinboxlearn" indefinitely,
but "publicinboxwatch" is probably more appropriate
at the moment.
Noticed while writing documentation.
|
|
This will be important as we will have more of them.
|
|
This will allow reasonable titles to be generated for
manpages.
|
|
Since this is bundled with the source, we might as well use
internal APIs to avoid having duplicate code (and bugs :P)
|
|
Not everybody can run Tor, hopefully more can use Tor2web
even if it compromises their privacy. This should help
make system more resilient for users unable to use Tor.
|
|
We want the pod2man(1) executable for handling certain
options. Also, use the correct year while we're at it :P
|
|
This makes us closer to git.git style (though I'm not quite sure
why we do this...)
|
|
We use perlpod nowadays since it's Perl, like our code base.
|
|
Centralization sucks, so we mirror everything.
|
|
The internal help text links to the Xapian query parser
documentation anyways, but also provides information
on which prefixes exist.
|
|
Begin documenting some basic help functionality.
I may tweak the anchor names of the various HTML endpoints
to be more consistent with each other (old ones will be
supported for a short while), so I'm not documenting
those, for now.
This may become part of a builtin key-value store for
basic texts, but this probably shouldn't become a wiki
engine, either.
|
|
We're not to-the-letter about percent-encoding, but
we should allow all the characters. This is mainly
so we can effectively use the link to some Wikipedia
pages with parentheses in them:
https://en.wikipedia.org/wiki/Atom_(standard)
https://en.wikipedia.org/wiki/Git_(software)
|
|
For some reason, Alpine will set X-UNKNOWN for valid UTF-8.
Since we favor UTF-8 HTML anyways, try forcing Email::MIME to
handle text/plain as UTF-8 which might show up better.
At least this change renders
<alpine.DEB.2.20.1608131214070.4924@virtualbox>
properly by showing "•" (•) instead of
"⠢" (•)
Reported-by: Thomas Ferris Nicolaisen <tfnico@gmail.com>
|
|
Alpine seems to set charset=X-UNKNOWN for valid UTF-8 text,
which causes Email::MIME::body_str to fail as X-UNKNOWN
is not a valid encoding. So, blindly display the body
as plain-text but warn users about possibly mangled text.
Reported-by: Thomas Ferris Nicolaisen <tfnico@gmail.com>
|
|
There is no point in using an array to join on an
empty string (my original intention was probably to
join on "\n").
This is only preparation for the next change to show
a warning to in the attachment link.
|
|
This is similar to mairix in that it uses a "d:" prefix; but
only takes YYYYMMDD, for now. Using custom date/time parsers
via Perl will be much more work:
nntp://news.gmane.org/20151005222157.GE5880@survex.com
Anyhow, this ought to be more human-friendly than searching by
Unix timestamps, but it requires reindexing to take advantage of.
|
|
The Unix timestamp isn't meaningful for users searching,
we will start indexing the YYYYMMDD date stamp which may
use StringValueRangeProcessor, instead.
|
|
Also, at least add one of the Tor mirrors (the rest will
be discoverable through the mirrors themselves).
|
|
Not sure why or how I missed this before; but the common address
parsing routine we have should be more correct.
Add a test to ensure excessively quoted names don't make it
through, either.
|
|
Plenty more to do!
|
|
Ensure we usually strip one level of '<>' from Message-IDs,
since our internal SQLite, Xapian, and SHA-1 storage all
assume that.
Realistically, we screw up if somebody has '<<' or '>>',
but those are screwed up mail clients and we can deal with
it another time. Currently, this means some messages with
'>>' in References or Message-Id are not handled correctly,
yet, but we match the behavior of Mail::Thread in keeping
the extra '>'.
|
|
Based on reading RFC 3986, it seems '@', ':', '!', '$', '&',
"'", '; '(', ')', '*', '+', ',', ';', '=' are all allowed
in path-absolute where we have the Message-ID.
In any case, it seems '@' is fairly common in path components
nowadays and too common in Message-IDs.
|
|
I've seen 0x1b (\e) in at least one message and some other
possibly non-printable chars. In any case, make sure they're
valid XML with us-ascii encoding as far as xmlstarlet(1) thinks
so.
|
|
Apparently there are some really screwed up In-Reply-To
fields out there.
|
|
We can't blindly assume a ghost even exists in the DB, as the
rules can change internally for some corner-case Message-IDs.
|
|
|
|
Because buggy mail clients exist and generate invalid
In-Reply-To headers we cannot handle across the board...
|
|
SQLite might index quickly, so we hold the lock used by Xapian
for the duration. This probably needs to be reworked entirely,
actually.
|
|
I needed to use this to resurrect some messages missing
from my initial downloads from gmane...
|
|
Some browsers do not give any indication of the HTTP error
code on errors, so show the error text to the user like we
do in the top-level WWW module.
|
|
gmane is down at the moment, so lower that in priority
(hopefully it will be brought back up, again). Wikipedia also
lists a few more project-specific list providers, so include
those as well: https://en.wikipedia.org/wiki/Message-ID
|
|
We need to pass the Inbox object to SearchIdx to get altid
mappings properly for incremental imports.
TODO: use the Inbox object in more places where it makes sense
to do so.
|
|
Improve the discoverability of NNTP endpoints for users
who still know what NNTP is.
==> ~/.public-inbox/config <==
; aliases for the locally-run nntpd can be specified in
; the "publicinbox" section:
[publicinbox]
nntpserver = nntp://ou63pmih66umazou.onion/
nntpserver = news.public-inbox.org
; NNTPS is not supported natively, yet,
; but one can use haproxy or similar
; nntpserver = nntps://news.public-inbox.invalid/
; mirrors for specific inboxes may be specified either as full
; NNTP (or NNTPS) URLs, or with the server name only if the
; newsgroup name is specfied for a local NNTP server
[publicinbox "git"]
...
newsgroup = inbox.a.b.c
nntpmirror = nntp://czquwvybam4bgbro.onion/
nntpmirror = hjrcffqmbrq6wope.onion
; there may be a mirror on a different server with a
; different name:
nntpmirror = nntp://news.example.com/differently.named.group
; (And I really need to write manpages for all this...)
|
|
This can be useful for adding new lists, as restarting is
expensive (but still non-lossy).
|
|
Oops. We will inevitably need to support multiple altids for a
public-inbox one day.
|
|
For some existing mailing list archives, messages are identified
by serial number (such as NNTP article numbers in gmane). Those
links may become inaccessible (as is the current case for
gmane), so ensure users can still search based on old serial
numbers.
Now, I run the following periodically to get article numbers
from gmane (while news.gmane.org remains):
NNTPSERVER=news.gmane.org
export NNTPSERVER
GROUP=gmane.comp.version-control.git
perl -I lib scripts/xhdr-num2mid $GROUP --msgmap=/path/to/gmane.sqlite3
(I might integrate this further with public-inbox-* scripts one day).
My ~/.public-inbox/config as an added "altid" snippet which now
looks like this:
[publicinbox "git"]
address = git@vger.kernel.org
mainrepo = /path/to/git.vger.git
newsgroup = inbox.comp.version-control.git
; relative pathnames expand to $mainrepo/public-inbox/$file
altid = serial:gmane:file=gmane.sqlite3
And run "public-inbox-index --reindex /path/to/git.vger.git"
periodically.
This ought to allow searching for "gmane:12345" to work for
Xapian-enabled instances.
Disclaimer: while public-inbox supports NNTP and stable article
serial numbers, use of those for public links is discouraged
since it encourages centralization.
|
|
It is not unheard of for users to attempt finding messages by
entering Message-IDs into the "Search" box instead of using the
existing URL structure. So make it possible for them.
Fwiw, I've definitely encountered users who enter entire URLs
into generic search engines.
|
|
Oops, we must unescape each key=value pair in a QUERY_STRING
individually; otherwise we cannot interpret '&' or ';' in
query parameter values.
|
|
We must ensure cat-file process is launched before Xapian
grabs lock, too. Our use of "git cat-file --batch" has
the same problem as "git log" did, (which was fixed in
commit 3713c727cda431a0dc2865a7878c13ecf9f21851)
"searchidx: release Xapian FDs before spawning git log"
|
|
This will allow us to release and re-acquire Xapian locks
due to the lack of FD_CLOEXEC on some FDs.
|
|
We can cheaply keep the object around nowadays since it
spawns expensive processes only on an as-needed basis.
|
|
We do not need to pass the PublicInbox::Git object to
various callbacks.
|
|
This is necessary to delimit messages when viewed without
threading.
|
|
At least for public-inbox-httpd, this allows us to avoid having
a client monopolize one event loop tick of the server for too
long. It hurts throughput for the /all.mbox.gz endpoint, but I
doubt anybody cares and the latency improvement for other
clients would be appreciated.
We already do the same fairness thing for HTML pages.
|
|
When using <ul><li>..., we already setup <pre> tags
in thread_index_entry, so having an extra </pre> tag
causes validation errors.
Fixes: 6ef9b216156c ("view: use <hr> to delineate in /$MID/T/ view")
|
|
This warrants further investigation, but it appears we cannot
release Xapian reliably after forking "git log" due to the
lack of a close-on-exec flag on the Xapian flintlock FD
|
|
The sacrifice in vertical space might be worth it to improve
ease-of-reading, as it's unreasonable to expect an entire
message thread to be able to fit into a single window.
https://public-inbox.org/git/20160805093544.scvl4yshkfg2l26p@sigill.intra.peff.net/
Cc: Jeff King <peff@peff.net>
|