From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id F06E61F4B4; Mon, 28 Dec 2020 21:41:46 +0000 (UTC) Date: Mon, 28 Dec 2020 21:41:46 +0000 From: Eric Wong To: meta@public-inbox.org Subject: thoughts improving duplicate message handling... Message-ID: <20201228214146.GB17600@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline List-Id: Spinning off of https://public-inbox.org/meta/20201228213139.GA17600@dcvr/ > (*) The downside to this approach is IMAP UIDs (NNTP article > numbers) get changed, but I think I can workaround that. The > workaround I'm thinking of involves capturing exact blob OIDs > during the unindex phase to create an OID => UID mapping. > reindex would reuse the OID => UID mapping to keep the same > IMAP UID. It could be loosened to use ContentHash, or > whatever combination of Message-ID/From/Date/etc, too. I've noticed quite a few messages with identical Message-IDs, especially with /all/ (*) a) There are occasionally resent revisions of patches with the same Message-ID b) More often, a cross-posted message has different trailers depending on which list it was posted to. (And the PublicInbox::Filter::* API doesn't apply at indexing, only at git injection time) I'm thinking of two new features: 1) the WWW and lei UIs could have a "diff mode" that clearly shows differences between rendered texts This could be expensive with malicious inputs, but the existing "limiter" infrastructure used for serving packs should help, here. 2) support PublicInbox::Filter at index-time (requires reindexing to take effect, will cause new IMAP UIDs+NNTP article numbers to show up, possibly en-mass) (*) https://public-inbox.org/meta/20201126194543.GA30337@dcvr/