From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS6315 166.70.0.0/16 X-Spam-Status: No, score=-3.5 required=3.0 tests=AWL,BAYES_00, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE, SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from out02.mta.xmission.com (out02.mta.xmission.com [166.70.13.232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id 3C79C1F4BD; Wed, 9 Oct 2019 08:17:44 +0000 (UTC) Received: from in02.mta.xmission.com ([166.70.13.52]) by out02.mta.xmission.com with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.87) (envelope-from ) id 1iI7AF-0002wP-Cw; Wed, 09 Oct 2019 02:17:43 -0600 Received: from ip68-227-160-95.om.om.cox.net ([68.227.160.95] helo=x220.xmission.com) by in02.mta.xmission.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.87) (envelope-from ) id 1iI7AE-00061Y-Mt; Wed, 09 Oct 2019 02:17:43 -0600 From: ebiederm@xmission.com (Eric W. Biederman) To: Eric Wong Cc: meta@public-inbox.org References: <87imp05hlm.fsf@alyssa.is> <20191008001050.rwd7bh7cek7qrydi@dcvr> <87wodfctwd.fsf@x220.int.ebiederm.org> <20191008221108.3wsso25kviiwd7ek@dcvr> <87wodec1um.fsf@x220.int.ebiederm.org> <20191008224104.GA24142@dcvr> <87h84ibb9m.fsf@x220.int.ebiederm.org> <87wode9vxw.fsf_-_@x220.int.ebiederm.org> Date: Wed, 09 Oct 2019 03:16:58 -0500 In-Reply-To: <87wode9vxw.fsf_-_@x220.int.ebiederm.org> (Eric W. Biederman's message of "Wed, 09 Oct 2019 03:15:07 -0500") Message-ID: <87muea9vut.fsf_-_@x220.int.ebiederm.org> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-XM-SPF: eid=1iI7AE-00061Y-Mt;;;mid=<87muea9vut.fsf_-_@x220.int.ebiederm.org>;;;hst=in02.mta.xmission.com;;;ip=68.227.160.95;;;frm=ebiederm@xmission.com;;;spf=neutral X-XM-AID: U2FsdGVkX1+yCcehaqmnkmaET80diQKG/xJgHglTgbY= X-SA-Exim-Connect-IP: 68.227.160.95 X-SA-Exim-Mail-From: ebiederm@xmission.com Subject: [PATCH 1/4] PublicInbox::Import Smuggle a raw message into add X-SA-Exim-Version: 4.2.1 (built Thu, 05 May 2016 13:38:54 -0600) X-SA-Exim-Scanned: Yes (on in02.mta.xmission.com) List-Id: Date: Tue, 15 Jan 2019 16:36:42 -0600 I don't trust the MIME type to not munge my email messages in horrible ways upon occasion. Therefore allow for passing in the raw message value instead of trusting the mime object to preserve it. Signed-off-by: "Eric W. Biederman" --- As we discussed last time I was working to merge my imap import script. lib/PublicInbox/Import.pm | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm index 137b2b7800c4..b17c9d5c9fa8 100644 --- a/lib/PublicInbox/Import.pm +++ b/lib/PublicInbox/Import.pm @@ -394,16 +394,17 @@ sub add { } my $blob = $self->{mark}++; - my $str = $mime->as_string; - my $n = length($str); + my $raw_email = $mime->{-public_inbox_raw}; + $raw_email ||= $mime->as_string; + my $n = length($raw_email); $self->{bytes_added} += $n; print $w "blob\nmark :$blob\ndata ", $n, "\n" or wfail; - print $w $str, "\n" or wfail; + print $w $raw_email, "\n" or wfail; # v2: we need this for Xapian if ($self->{want_object_info}) { my $oid = $self->get_mark(":$blob"); - $self->{last_object} = [ $oid, $n, \$str ]; + $self->{last_object} = [ $oid, $n, \$raw_email ]; } my $ref = $self->{ref}; my $commit = $self->{mark}++; -- 2.23.0