user/dev discussion of public-inbox itself
 help / color / Atom feed
From: "Eric Wong (Contractor, The Linux Foundation)" <e@80x24.org>
To: meta@public-inbox.org
Subject: [WIP 11/17] import: APIs to support v2 use
Date: Thu, 15 Feb 2018 11:08:34 +0000
Message-ID: <20180215110840.30413-12-e@80x24.org> (raw)
In-Reply-To: <20180215110840.30413-1-e@80x24.org>

Wrap "get-mark" and "checkpoint" commands for git-fast-import
while documenting/cementing parts of the API.
---
 lib/PublicInbox/Import.pm | 28 ++++++++++++++++++++++------
 t/import.t                |  4 +++-
 2 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index b8e9dd0..811e355 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -138,9 +138,27 @@ sub check_remove_v1 {
 	(undef, $cur);
 }
 
+# used for v2 (maybe)
+sub checkpoint {
+	my ($self) = @_;
+	return unless $self->{pid};
+	print { $self->{out} } "checkpoint\n" or wfail;
+	undef;
+}
+
+# used for v2
+sub get_mark {
+	my ($self, $mark) = @_;
+	die "not active\n" unless $self->{pid};
+	my ($r, $w) = $self->gfi_start;
+	print $w "get-mark $mark\n" or wfail;
+	defined(my $oid = <$r>) or die "get-mark failed, need git 2.6.0+\n";
+	$oid;
+}
+
 # returns undef on non-existent
-# ('MISMATCH', msg) on mismatch
-# (:MARK, msg) on success
+# ('MISMATCH', Email::MIME) on mismatch
+# (:MARK, Email::MIME) on success
 #
 # For v2 inboxes, the content_id is returned instead of the msg
 # v2 callers should check with Xapian before calling this as
@@ -189,6 +207,7 @@ sub remove {
 }
 
 # returns undef on duplicate
+# returns the :MARK of the most recent commit
 sub add {
 	my ($self, $mime, $check_cb) = @_; # mime = Email::MIME
 
@@ -234,10 +253,7 @@ sub add {
 
 	# v2: we need this for Xapian
 	if ($self->{want_object_id}) {
-		print $w "get-mark :$blob\n" or wfail;
-		defined(my $object_id = <$r>) or
-				die "get-mark failed, need git 2.6.0+\n";
-		chomp($self->{last_object_id} = $object_id);
+		chomp($self->{last_object_id} = $self->get_mark(":$blob"));
 	}
 
 	my $ref = $self->{ref};
diff --git a/t/import.t b/t/import.t
index 92c82b9..ca59772 100644
--- a/t/import.t
+++ b/t/import.t
@@ -87,6 +87,8 @@ isnt($msg->header('Subject'), $mime->header('Subject'), 'subject mismatch');
 $mime->header_set('Message-Id', '<failcheck@example.com>');
 is($im->add($mime, sub { undef }), undef, 'check callback fails');
 is($im->remove($mime), undef, 'message not added, so not removed');
-
+is(undef, $im->checkpoint, 'checkpoint works before ->done');
 $im->done;
+is(undef, $im->checkpoint, 'checkpoint works after ->done');
+$im->checkpoint;
 done_testing();
-- 
EW


  parent reply index

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-02-09 20:51 [v2] one file to rule them all? Eric Wong
2018-02-15 10:55 ` Eric Wong
2018-02-15 11:08   ` [WIP 0/17] initial v2 work based on one-file tree Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 01/17] AUTHORS: add The Linux Foundation Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 02/17] watch_maildir: allow '-' in mail filename Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 03/17] scripts/import_vger_from_mbox: relax From_ line match slightly Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 04/17] import: stop writing legacy ssoma.index by default Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 05/17] import: begin supporting this without ssoma.lock Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 06/17] import: initial handling for v2 Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 07/17] t/import: test for last_object_id insertion Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 08/17] content_id: add test case Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 09/17] searchmsg: add mid_mime import for _extract_mid Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 10/17] scripts/import_vger_from_mbox: support --dry-run option Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` Eric Wong (Contractor, The Linux Foundation) [this message]
2018-02-15 11:08     ` [WIP 12/17] search: free up 'Q' prefix for a real unique identifier Eric Wong (Contractor, The Linux Foundation)
2018-02-22 21:08       ` Eric Wong
2018-02-15 11:08     ` [WIP 13/17] searchidx: fix comment around next_thread_id Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 14/17] address: extract more characters from email addresses Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 15/17] import: pass "raw" dates to git-fast-import(1) Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 16/17] scripts/import_vger_from_mbox: use v2 layout for import Eric Wong (Contractor, The Linux Foundation)
2018-02-15 11:08     ` [WIP 17/17] import: quiet down warnings from bogus From: lines Eric Wong (Contractor, The Linux Foundation)

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://public-inbox.org/README

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180215110840.30413-12-e@80x24.org \
    --to=e@80x24.org \
    --cc=meta@public-inbox.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox