about summary refs log tree commit homepage
DateCommit message (Collapse)
2017-07-02doc: add ruby-dev mirror to the list of hosted mirrors
It seems Xapian is not prepared for Japanese, unfortunately. https://public-inbox.org/meta/20170702213657.GA5312@dcvr/
2017-06-30doc: add a list of hosted archives for external projects
This will hopefully increase visibility of some archives.
2017-06-29view: cull redundant phrases in subjects
There is no need to show the same phrases over and over again in thread skeletons, it adds to visual noise and makes things more difficult to read.
2017-06-29scripts/import_maildir: rewrite to use Import
This will be much faster and invoking -mda for every message.
2017-06-29hval: only perform one substitution when obfuscating
Only one substitution character is necessary when obfuscating email addresses.
2017-06-26msgmap: reduce constant usage
It is needless bloat and doesn't seem to help with readability, in retrospect, either.
2017-06-26watch: avoid potential race condition while quitting
We must not trigger future activity when initializing a -watch shutdown.
2017-06-26tests: deal with the removal of '.' from @INC in newer Perl
Oops, this is needed for Perl 5.22 (tested 5.24.1) since '.' was removed due to security problems. Fwiw, I consider this change to Perl an overreaction and do not agree with it.
2017-06-26watch: commit changes to fast-import sooner
We should make changes visible sooner, even during lengthy scans.
2017-06-26watch: use "self-inotify-tempfile trick" for quit
This should be more reliable and safer as it'll ensure existing fast-import instances are shut down properly.
2017-06-26watch: improve fairness during full rescans
We need to ensure new messages are being processed fairly during full rescans, so have the ->scan subroutine yield and reschedule itself. Additionally, having a long-running task inside the signal handler is dangerous and subject to reentrancy bugs. Due to the limitations of the Filesys::Notify::Simple interface, we cannot rely on multiplexing I/O interfaces (select, IO::Poll, Danga::Socket, etc...) for this. Forking a separate process was considered, but it is more expensive for a mostly-idle process. So, we use a variant of the "self-pipe trick" via inotify (or whatever Filesys::Notify::Simple gives us). Instead of writing to our own pipe, we write to a file in our own temporary directory watched by Filesys::Notify::Simple to trigger events in signal handlers.
2017-06-26spamc: retry on EINTR
Signals can fire on us at any time if we're using blocking sysread.
2017-06-26watch: ensure HUP causes the scanner to be reloaded
Otherwise the old watcher may run indefinitely
2017-06-26mda: set List-ID correctly according to RFC2919
Oops, due to an old mistake , List-ID was set incorrectly in the MDA. This could cause some breakage w.r.t. mail filters.
2017-06-23linkify: handle URLs in parenthesized statements
Sometimes, URLs exist at the end of parethesized statements, and we shouldn't unnecessarily capture that. (example: https://public-inbox.org/ruby-core/20170623032722.GA8124@dcvr/)
2017-06-23allow admins to configure non-obfuscated addresses/domains
We will also treat all known list addresses as non-obfuscated. By setting publicinbox.noObfuscate in ~/.public-inbox/config, this will allow users to disable address obfuscation on a per-domain or per-address basis.
2017-06-23config: assume lists have multiple addresses
This should simplify the rest of our code for handling the do-not-obfuscate list.
2017-06-23view: add newline before mailto: instructions in reply
This is necessary to retain consistent spacing around bullet points. Fixes: 666844ae42b5b17f ("reply: handle address obfuscation :<")
2017-06-23mbox: show application/mbox for obfuscated inboxes
Sigh, yet another place to handle obfuscation for misguided people who expect it. Maybe this will do something to prevent spammers from getting addresses, while still allowing the "curl $URL | git am" use case to work.
2017-06-23reply: handle address obfuscation :<
We can show users a lightly-obfuscated Bourne shell command for invoking "git send-email" for address obfuscation. However, I'm not sure if the mailto: arg will work effectively since URL encoding is probably too well-known to be effective.
2017-06-23searchidx: fallback to lookup on pre-set article numbers
Yet another hiccup from reusing pre-set article numbers on various ruby-lang.org mailing lists. This was causing messages to not appear to NNTP readers which use XOVER.
2017-06-23msgmap: ignore duplicates instead of dying
This prevents public-inbox-watch from dying when reloading (and thus rescanning) already-imported directories.
2017-06-23watchmaildir: deal with rejected (100) messages
The RubyLang filter is strict about what messages it rejects, so the spam learning path will not auto-train or remove messages missing X-Mail-Count headers.
2017-06-22test for PublicInbox::Filter::RubyLang
This will make it easier to prevent breakage in the future.
2017-06-22filter/rubylang: reuse altid entry from inbox object
This allows users to DRY up their config a bit and avoid specifying altid twice when reusing the NNTP-centric msgmap for [ruby-*:\d+] serial numbers. My current work-in-progress ~/.public-inbox/config entry for the ruby-core list is: ------8<------- [publicinbox "ruby-core"] address = ruby-core@ruby-lang.org url = //public-inbox.org/ruby-core mainrepo = /path/to/ruby-core.git newsgroup = inbox.comp.lang.ruby.core watchheader = List-Id:<ruby-core.ruby-lang.org> altid = serial:ruby-core:file=msgmap.sqlite3 watch = maildir:/path/to/Maildir/.INBOX.ruby filter = PublicInbox::Filter::RubyLang
2017-06-22msgmap: mid_insert ignores duplicates instead of die-ing
This will allow smoother imports as occasional Message-ID duplicates happen and the best we can do is ignore the second one.
2017-06-22add filter for RubyLang lists
Unfortunately, it appears we have to reject this and instead add support filtering at View time(*), due to DKIM signatures in messages from ruby-lang.org. (*) which may not be worth it
2017-06-20import: fix encoding issues from weird "raw" emails
This seems to allow weirdly-encoded "raw" emails in blade.nagaokaut.ac.jp/ruby/ruby-core/* to be handled without difficulties.
2017-06-16view: implement optional address obfuscation
This is lightly-tested and seems to work. I'm still hesitant to support this, but the alternative of receiving death threats for displaying unobfuscated addresses seems to be not worth it.
2017-06-15reply: support Reply-To
Reply-To is common and probably should've been supported, since day one, but we won't omit other addresses, either.
2017-06-15replyto parameter support
This allows us to support centralized mailing lists (which suck, but better than no mailing list at all).
2017-06-15view: split out reply logic into its own module
We'll be adding more reply options for centralized mailing lists. So split out the logic so it's easy-to-find. Organizing code is hard :<
2017-06-15searchidx: remove messages correctly from Xapian index
This fixes a bug introduced in commit 7eeadcb62729b0efbcb53cd9b7b181897c92cf9a ("search: remove unnecessary abstractions and functionality")
2017-06-14search: allow searching within mail diffs
This can be tied into a repository browser to browse in-flight topics on a mailing list.
2017-06-14searchidx: switch to accounting by message bytes
Xapian memory usage is tied to the size of the indexed text, so take the raw message size into account when deciding when to flush Xapian data. More importantly, we now flush Xapian before we have it buffer beyond our maximum; and we do it unconditionally to prevent even high priority processes from OOM-ing.
2017-06-14search: remove unnecessary abstractions and functionality
This simplifies the code a bit and reduces the translation overhead for looking directly at data from tools shipped with Xapian. While we're at it, fix thread-all.t :)
2017-06-07filter/subjecttag: account for missing Subject: header
This is a high indicator of spam (but out-of-scope for this particular module) but sometimes it is not, and people legitimately forget to set a Subject: header at all.
2017-05-25import: reset :raw mode for commit title (subject)
This was necessary for the presence of the 0xa0 byte(*) in the Subject: of the message at: http://blade.nagaokaut.ac.jp/ruby/ruby-core/3220 (*) That is 0xa0, not 0x0a ("\n"), so I wonder if the nibbles got swapped somehow.
2017-05-23searchview: retry queries if uri_unescape-able
It is possible to have double-escaped queries when copy and pasting into browsers, so try to help users work around this common error by automatically retrying after unescaping once. Of course, we must inform the user when doing this results in success, in case they really meant to search for a double-escaped term which resulted in nothing. Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> https://public-inbox.org/meta/CACBZZX5Gnow08r=0A1J_kt3a=zpGyMfvsqu8nAN7kacNnDm+dg@mail.gmail.com/
2017-05-23www: do not mangle characters from search queries
Reported-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> https://public-inbox.org/meta/CACBZZX5Gnow08r=0A1J_kt3a=zpGyMfvsqu8nAN7kacNnDm+dg@mail.gmail.com/
2017-05-09www: avoid undefined warnings for query string parsing
Sometimes bots generate malformed queries with sequential "&" and ";" characters.
2017-05-09watchmaildir: show $@ in warning message
It should be helpful to know what error happened.
2017-05-09searchidx: use cached local $@ copy
umask should never fail and set $@, but use the cached local to be more explicit just in case.
2017-05-07spamassassin: update example ~/.spamassassin/user_prefs file
This is closer to what I run on the public-inbox.org servers.
2017-05-07searchidx: fix ghost root vivification
Due to the asynchronous nature of SMTP, it is possible for the root message of a thread (with no References/In-Reply-To) to arrive last in a series. We must preserve the thread_id of the ghost message in this case, as we do when vivifiying non-root ghosts. Otherwise, this causes threads to be broken when the root arrives last.
2017-04-11search: fix help message for searching within quotes
I'm not sure if people use either and it's not in mairix (where we base our abbreviations off of). Lets go with the shorter prefix since it's easier-to-type.
2017-04-05learn: scan all inboxes when learning spam
This matches the behavior of the -watch daemon since 6d534038285ddd760709ba76ea007f9108200097 ("watch: watchspam affects all configured inboxes")
2017-04-04watchmaildir: do not reject lowercase flags on Maildir files
Dovecot uses 'a'..'z' (lowercase) to designate keywords in Maildir flags. This was preventing certain messages from being marked as spam. https://wiki2.dovecot.org/MailboxFormat/Maildir
2017-03-24searchview: show full (&x=t) messages in ascending chronlogical order
When displaying search results with full messages, it makes more sense to show them in ascending chronological order when going by date. Reverse chronological order makes more sense for search results which only show the subject.
2017-03-24searchview: add "t" id to link to thread overview
At least for the thread view (&x=t); this will make it easy to link to the overview.