about summary refs log tree commit homepage
path: root/lib
DateCommit message (Collapse)
2017-06-26mda: set List-ID correctly according to RFC2919
Oops, due to an old mistake , List-ID was set incorrectly in the MDA. This could cause some breakage w.r.t. mail filters.
2017-06-23linkify: handle URLs in parenthesized statements
Sometimes, URLs exist at the end of parethesized statements, and we shouldn't unnecessarily capture that. (example: https://public-inbox.org/ruby-core/20170623032722.GA8124@dcvr/)
2017-06-23allow admins to configure non-obfuscated addresses/domains
We will also treat all known list addresses as non-obfuscated. By setting publicinbox.noObfuscate in ~/.public-inbox/config, this will allow users to disable address obfuscation on a per-domain or per-address basis.
2017-06-23config: assume lists have multiple addresses
This should simplify the rest of our code for handling the do-not-obfuscate list.
2017-06-23view: add newline before mailto: instructions in reply
This is necessary to retain consistent spacing around bullet points. Fixes: 666844ae42b5b17f ("reply: handle address obfuscation :<")
2017-06-23mbox: show application/mbox for obfuscated inboxes
Sigh, yet another place to handle obfuscation for misguided people who expect it. Maybe this will do something to prevent spammers from getting addresses, while still allowing the "curl $URL | git am" use case to work.
2017-06-23reply: handle address obfuscation :<
We can show users a lightly-obfuscated Bourne shell command for invoking "git send-email" for address obfuscation. However, I'm not sure if the mailto: arg will work effectively since URL encoding is probably too well-known to be effective.
2017-06-23searchidx: fallback to lookup on pre-set article numbers
Yet another hiccup from reusing pre-set article numbers on various ruby-lang.org mailing lists. This was causing messages to not appear to NNTP readers which use XOVER.
2017-06-23msgmap: ignore duplicates instead of dying
This prevents public-inbox-watch from dying when reloading (and thus rescanning) already-imported directories.
2017-06-23watchmaildir: deal with rejected (100) messages
The RubyLang filter is strict about what messages it rejects, so the spam learning path will not auto-train or remove messages missing X-Mail-Count headers.
2017-06-22filter/rubylang: reuse altid entry from inbox object
This allows users to DRY up their config a bit and avoid specifying altid twice when reusing the NNTP-centric msgmap for [ruby-*:\d+] serial numbers. My current work-in-progress ~/.public-inbox/config entry for the ruby-core list is: ------8<------- [publicinbox "ruby-core"] address = ruby-core@ruby-lang.org url = //public-inbox.org/ruby-core mainrepo = /path/to/ruby-core.git newsgroup = inbox.comp.lang.ruby.core watchheader = List-Id:<ruby-core.ruby-lang.org> altid = serial:ruby-core:file=msgmap.sqlite3 watch = maildir:/path/to/Maildir/.INBOX.ruby filter = PublicInbox::Filter::RubyLang
2017-06-22msgmap: mid_insert ignores duplicates instead of die-ing
This will allow smoother imports as occasional Message-ID duplicates happen and the best we can do is ignore the second one.
2017-06-22add filter for RubyLang lists
Unfortunately, it appears we have to reject this and instead add support filtering at View time(*), due to DKIM signatures in messages from ruby-lang.org. (*) which may not be worth it
2017-06-20import: fix encoding issues from weird "raw" emails
This seems to allow weirdly-encoded "raw" emails in blade.nagaokaut.ac.jp/ruby/ruby-core/* to be handled without difficulties.
2017-06-16view: implement optional address obfuscation
This is lightly-tested and seems to work. I'm still hesitant to support this, but the alternative of receiving death threats for displaying unobfuscated addresses seems to be not worth it.
2017-06-15reply: support Reply-To
Reply-To is common and probably should've been supported, since day one, but we won't omit other addresses, either.
2017-06-15replyto parameter support
This allows us to support centralized mailing lists (which suck, but better than no mailing list at all).
2017-06-15view: split out reply logic into its own module
We'll be adding more reply options for centralized mailing lists. So split out the logic so it's easy-to-find. Organizing code is hard :<
2017-06-15searchidx: remove messages correctly from Xapian index
This fixes a bug introduced in commit 7eeadcb62729b0efbcb53cd9b7b181897c92cf9a ("search: remove unnecessary abstractions and functionality")
2017-06-14search: allow searching within mail diffs
This can be tied into a repository browser to browse in-flight topics on a mailing list.
2017-06-14searchidx: switch to accounting by message bytes
Xapian memory usage is tied to the size of the indexed text, so take the raw message size into account when deciding when to flush Xapian data. More importantly, we now flush Xapian before we have it buffer beyond our maximum; and we do it unconditionally to prevent even high priority processes from OOM-ing.
2017-06-14search: remove unnecessary abstractions and functionality
This simplifies the code a bit and reduces the translation overhead for looking directly at data from tools shipped with Xapian. While we're at it, fix thread-all.t :)
2017-06-07filter/subjecttag: account for missing Subject: header
This is a high indicator of spam (but out-of-scope for this particular module) but sometimes it is not, and people legitimately forget to set a Subject: header at all.
2017-05-25import: reset :raw mode for commit title (subject)
This was necessary for the presence of the 0xa0 byte(*) in the Subject: of the message at: http://blade.nagaokaut.ac.jp/ruby/ruby-core/3220 (*) That is 0xa0, not 0x0a ("\n"), so I wonder if the nibbles got swapped somehow.
2017-05-23searchview: retry queries if uri_unescape-able
It is possible to have double-escaped queries when copy and pasting into browsers, so try to help users work around this common error by automatically retrying after unescaping once. Of course, we must inform the user when doing this results in success, in case they really meant to search for a double-escaped term which resulted in nothing. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> https://public-inbox.org/meta/CACBZZX5Gnow08r=0A1J_kt3a=zpGyMfvsqu8nAN7kacNnDm+dg@mail.gmail.com/
2017-05-23www: do not mangle characters from search queries
Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> https://public-inbox.org/meta/CACBZZX5Gnow08r=0A1J_kt3a=zpGyMfvsqu8nAN7kacNnDm+dg@mail.gmail.com/
2017-05-09www: avoid undefined warnings for query string parsing
Sometimes bots generate malformed queries with sequential "&" and ";" characters.
2017-05-09watchmaildir: show $@ in warning message
It should be helpful to know what error happened.
2017-05-09searchidx: use cached local $@ copy
umask should never fail and set $@, but use the cached local to be more explicit just in case.
2017-05-07searchidx: fix ghost root vivification
Due to the asynchronous nature of SMTP, it is possible for the root message of a thread (with no References/In-Reply-To) to arrive last in a series. We must preserve the thread_id of the ghost message in this case, as we do when vivifiying non-root ghosts. Otherwise, this causes threads to be broken when the root arrives last.
2017-04-11search: fix help message for searching within quotes
I'm not sure if people use either and it's not in mairix (where we base our abbreviations off of). Lets go with the shorter prefix since it's easier-to-type.
2017-04-04watchmaildir: do not reject lowercase flags on Maildir files
Dovecot uses 'a'..'z' (lowercase) to designate keywords in Maildir flags. This was preventing certain messages from being marked as spam. https://wiki2.dovecot.org/MailboxFormat/Maildir
2017-03-24searchview: show full (&x=t) messages in ascending chronlogical order
When displaying search results with full messages, it makes more sense to show them in ascending chronological order when going by date. Reverse chronological order makes more sense for search results which only show the subject.
2017-03-24searchview: add "t" id to link to thread overview
At least for the thread view (&x=t); this will make it easy to link to the overview.
2017-03-22extmsg: use updated mail-archive.com URL
Apparently mid.mail-archive.com does not support HTTPS, and the HTTP version redirects to the search query, anyways.
2017-03-14view: escape HTML description name
Otherwise funky filenames can cause HTML injection vulnerabilities (hope you have JavaScript disabled!)
2017-02-14www: do not unescape PATH_INFO twice
PSGI specs already require PATH_INFO to be unescaped; so our tests were wrong, too.
2017-02-11handle repeated References and In-Reply-To headers
It seems possible for git-send-email(1) to generate repeated repeated instances of References and In-Reply-To headers, as evidenced in: https://public-inbox.org/git/20161111124541.8216-17-vascomalmeida@sapo.pt/raw This causes a mismatch between how our search indexer threads and how our HTML view handles threading. In the future, View.pm will use the smsg-parsed {references} field and avoid redoing Email::MIME header parsing. We will still need to figure out a way to deal with messages with repeated Message-IDs, at some point, too.
2017-02-09config: do not slurp lines into memory
There's no need to hold everything in memory, here, since apparently "foreach" will read everything at once in array context (for some reason, I thought Perl5 was smart enough to avoid creating a temporary array, here...)
2017-02-06search: schema version bump for empty References/In-Reply-To
We cannot distinguish between legitimate ghosts and mis-threaded messages before commit 83425ef12e4b65cdcecd11ddcb38175d4a91d5a0 ("searchidx: deal with empty In-Reply-To and References headers") so we must rebuild the index in parallel to fix it.
2017-02-06Revert "searchidx: reindex clobbers old thread IDs"
Oops, that's broken, too. I guess the only way to reindex after fixing the thread detection is to start from scratch. This reverts commit 5d91adedf5f33ef1cb87df2a86306ddf370b4f8d.
2017-02-06searchidx: reindex clobbers old thread IDs
We cannot always reuse thread IDs since our threading logic may change as bugs are fixed.
2017-02-06searchidx: deal with empty In-Reply-To and References headers
In some messages, these headers exist, but have empty values. Do not let empty values throw off our search indexer to tie threads together, as it can make non-sensical threads grouped to a Message-Id of "" (empty string). See <https://public-inbox.org/git/11340844841342-git-send-email-mailing-lists.git@rawuncut.elitemail.org/raw> for an example of such a message. Thanks-to: Johannes Schindelin <Johannes.Schindelin@gmx.de> <https://public-inbox.org/git/alpine.DEB.2.20.1702041206130.3496@virtualbox/>
2017-02-06searchview: increase limit for displaying search results
We are in no danger of excessive buffering or OOM-ing, the main page for every inbox already loads 200 results; and thread page views even load 1000! Increase this to 200 for now.
2017-02-06searchview: clarify numeric summary at bottom
Xapian can only give estimated results when a result limit is given to it, so make clear it is an estimate to avoid showing non-sensical ranges when no results are returned.
2017-01-26add filter for Subject: tags
Some mailing lists add annoying tags into the Subject line which discourages readers from doing proper mail organization on the client side. They also waste precious screen space and attention span. Remove them from our archives to reduce clutter.
2017-01-26watchmaildir: allow arguments for filters
We'll want to allow some degree of configuration for various mailing lists.
2017-01-19watchmaildir: limit live importer processes
We don't want to be triggering OOM or swapping on weaker systems when we have dozens of inboxes as potential targets.
2017-01-18mime: avoid SUPER usage in Email::MIME subclass
We must call Email::Simple methods directly in our monkey patch for Email::MIME to call the intended method. Using SUPER in our subclass would instead hit a different, unintended method in Email::MIME. Reported-by: Junio C Hamano <gitster@pobox.com> <xmqq4m0wb43w.fsf@gitster.mtv.corp.google.com>
2017-01-11inbox: reinstate periodic cleanup of Xapian and SQLite objects
We may need to do this even more aggressively, since the Xapian database does not always give the latest results. This time, we'll do it without relying on weak references, and instead check refcounts.