about summary refs log tree commit homepage
path: root/lib/PublicInbox/Import.pm
diff options
context:
space:
mode:
authorEric Wong <e@80x24.org>2021-05-04 09:49:12 +0000
committerEric Wong <e@80x24.org>2021-05-04 23:08:01 +0000
commit40f3f2a2c805fc37c7ed35a60948856bd962b493 (patch)
tree0fe4d70c7a50a0122f92d48b832ada31a3b5a0bf /lib/PublicInbox/Import.pm
parent4481b372ba150c669b2fefe2d6ec5dccb5da1d40 (diff)
downloadpublic-inbox-40f3f2a2c805fc37c7ed35a60948856bd962b493.tar.gz
Since completely purging blobs from git is slow, users may wish
to index messages in Maildirs (and eventually other local
storage) without storing data in git.

Much code from LeiImport and LeiInput is reused, and a new dummy
FakeImport class supplies a non-storing $im->add and minimize
changes to LeiStore.

The tricky part of this command is to support "lei import"
after a message has gone through "lei index".  Relying on
$smsg->{bytes} == 0 (as we do for external-only vmd storage)
does not work here, since it would break searching for "z:"
byte-ranges when not using externals.

This eventually required PublicInbox::Import::add to use a
SharedKV to keep track of imported blobs and prevent
duplication.
Diffstat (limited to 'lib/PublicInbox/Import.pm')
-rw-r--r--lib/PublicInbox/Import.pm26
1 files changed, 13 insertions, 13 deletions
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index 3adf9dec..362cdc47 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -413,19 +413,19 @@ sub add {
                 $smsg->{blob} = $self->get_mark(":$blob");
                 $smsg->set_bytes($raw_email, $n);
                 if (my $oidx = delete $smsg->{-oidx}) { # used by LeiStore
-                        my @docids = $oidx->blob_exists($smsg->{blob});
-                        my @vivify_xvmd;
-                        for my $id (@docids) {
-                                if (my $cur = $oidx->get_art($id)) {
-                                        # already imported if bytes > 0
-                                        return if $cur->{bytes} > 0;
-                                        push @vivify_xvmd, $id;
-                                } else {
-                                        warn "W: $smsg->{blob} ",
-                                                "#$id gone (bug?)\n";
-                                }
-                        }
-                        $smsg->{-vivify_xvmd} = \@vivify_xvmd;
+                        my $eidx_git = delete $smsg->{-eidx_git};
+
+                        # we need this sharedkv to dedupe blobs added in the
+                        # same fast-import transaction
+                        my $u = $self->{uniq_skv} //= do {
+                                require PublicInbox::SharedKV;
+                                my $x = PublicInbox::SharedKV->new;
+                                $x->dbh;
+                                $x;
+                        };
+                        return if !$u->set_maybe(pack('H*', $smsg->{blob}), 1);
+                        return if (!$oidx->vivify_xvmd($smsg) &&
+                                        $eidx_git->check($smsg->{blob}));
                 }
         }
         my $ref = $self->{ref};