From: "Eric Wong (Contractor, The Linux Foundation)" <e@80x24.org>
To: meta@public-inbox.org
Subject: [WIP 11/17] import: APIs to support v2 use
Date: Thu, 15 Feb 2018 11:08:34 +0000 [thread overview]
Message-ID: <20180215110840.30413-12-e@80x24.org> (raw)
In-Reply-To: <20180215110840.30413-1-e@80x24.org>
Wrap "get-mark" and "checkpoint" commands for git-fast-import
while documenting/cementing parts of the API.
---
lib/PublicInbox/Import.pm | 28 ++++++++++++++++++++++------
t/import.t | 4 +++-
2 files changed, 25 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index b8e9dd0..811e355 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -138,9 +138,27 @@ sub check_remove_v1 {
(undef, $cur);
}
+# used for v2 (maybe)
+sub checkpoint {
+ my ($self) = @_;
+ return unless $self->{pid};
+ print { $self->{out} } "checkpoint\n" or wfail;
+ undef;
+}
+
+# used for v2
+sub get_mark {
+ my ($self, $mark) = @_;
+ die "not active\n" unless $self->{pid};
+ my ($r, $w) = $self->gfi_start;
+ print $w "get-mark $mark\n" or wfail;
+ defined(my $oid = <$r>) or die "get-mark failed, need git 2.6.0+\n";
+ $oid;
+}
+
# returns undef on non-existent
-# ('MISMATCH', msg) on mismatch
-# (:MARK, msg) on success
+# ('MISMATCH', Email::MIME) on mismatch
+# (:MARK, Email::MIME) on success
#
# For v2 inboxes, the content_id is returned instead of the msg
# v2 callers should check with Xapian before calling this as
@@ -189,6 +207,7 @@ sub remove {
}
# returns undef on duplicate
+# returns the :MARK of the most recent commit
sub add {
my ($self, $mime, $check_cb) = @_; # mime = Email::MIME
@@ -234,10 +253,7 @@ sub add {
# v2: we need this for Xapian
if ($self->{want_object_id}) {
- print $w "get-mark :$blob\n" or wfail;
- defined(my $object_id = <$r>) or
- die "get-mark failed, need git 2.6.0+\n";
- chomp($self->{last_object_id} = $object_id);
+ chomp($self->{last_object_id} = $self->get_mark(":$blob"));
}
my $ref = $self->{ref};
diff --git a/t/import.t b/t/import.t
index 92c82b9..ca59772 100644
--- a/t/import.t
+++ b/t/import.t
@@ -87,6 +87,8 @@ isnt($msg->header('Subject'), $mime->header('Subject'), 'subject mismatch');
$mime->header_set('Message-Id', '<failcheck@example.com>');
is($im->add($mime, sub { undef }), undef, 'check callback fails');
is($im->remove($mime), undef, 'message not added, so not removed');
-
+is(undef, $im->checkpoint, 'checkpoint works before ->done');
$im->done;
+is(undef, $im->checkpoint, 'checkpoint works after ->done');
+$im->checkpoint;
done_testing();
--
EW
next prev parent reply other threads:[~2018-02-15 11:08 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-09 20:51 [v2] one file to rule them all? Eric Wong
2018-02-15 10:55 ` Eric Wong
2018-02-15 11:08 ` [WIP 0/17] initial v2 work based on one-file tree Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 01/17] AUTHORS: add The Linux Foundation Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 02/17] watch_maildir: allow '-' in mail filename Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 03/17] scripts/import_vger_from_mbox: relax From_ line match slightly Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 04/17] import: stop writing legacy ssoma.index by default Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 05/17] import: begin supporting this without ssoma.lock Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 06/17] import: initial handling for v2 Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 07/17] t/import: test for last_object_id insertion Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 08/17] content_id: add test case Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 09/17] searchmsg: add mid_mime import for _extract_mid Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 10/17] scripts/import_vger_from_mbox: support --dry-run option Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` Eric Wong (Contractor, The Linux Foundation) [this message]
2018-02-15 11:08 ` [WIP 12/17] search: free up 'Q' prefix for a real unique identifier Eric Wong (Contractor, The Linux Foundation)
2018-02-22 21:08 ` Eric Wong
2018-02-15 11:08 ` [WIP 13/17] searchidx: fix comment around next_thread_id Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 14/17] address: extract more characters from email addresses Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 15/17] import: pass "raw" dates to git-fast-import(1) Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 16/17] scripts/import_vger_from_mbox: use v2 layout for import Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08 ` [WIP 17/17] import: quiet down warnings from bogus From: lines Eric Wong (Contractor, The Linux Foundation)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180215110840.30413-12-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).