Re: [v2] introduction of content_id

[Konstantin Ryabitsev <konstantin@linuxfoundation.org>]

[-- Attachment #1.1: Type: text/plain, Size: 1966 bytes --]

On 02/09/18 13:17, Eric Wong wrote:
> In addition to the git object_id (blob SHA-1) and Message-Id
> header; it seems necessary to introduce an in-between identifier
> for deduplicating which isn't as loose as Message-Id or as
> strict as object_id: content_id
> 
> I think a hash of the following raw headers + raw body will
> suffice:
> 
> 	Subject, From, Date, Message-Id, References, To, Cc,
> 	In-Reply-To, MIME-Version, Content-Type,
> 	Content-Disposition, Content-Transfer-Encoding

That's similar to what ARC/DKIM do. E.g. in my mailbox the message has
the following headers sealed:

    h=archived-at:list-post:list-owner:list-subscribe:list-unsubscribe
         :list-help:list-archive:precedence:list-id:content-disposition
         :mime-version:message-id:subject:to:from:date
         :arc-authentication-results;

If we trim the arc- and list-specific headers, that's:

precedence
content-disposition
mime-version
message-id
subject
to
from
date

I'm not sure we should care about content-transfer-encoding, because
that can be mangled by intermediate MTAs (at least that used to happen
all the time in the past -- not sure if it's still the case).

> List-Id, X-Mailing-List should be left out so different
> readers/lists can share spam removals in cross posts.

Note, that mailing lists that modify the Subject header (e.g. to add
[mailinglist] identifier) will also be impacted similarly.

> (*) I noticed the first Received: header (last hop) is missing
>     from the cregit sources; but the first remaining Received:
>     header also includes the identity of the recipient in more
>     recent mails...

I specifically sanitized all Received: headers that didn't say "by
vger.kernel.org" because these are donated by individual users and I
didn't want to expose their potentially private info.

Best,
-- 
Konstantin Ryabitsev
Director, IT Infrastructure Security
The Linux Foundation


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 228 bytes --]