From: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>
To: Jonathan Nieder <jrnieder@gmail.com>
Cc: Florian Achleitner <florian.achleitner.2.6.31@gmail.com>,
git@vger.kernel.org, David Michael Barr <davidbarr@google.com>,
Sverre Rabbelier <srabbelier@gmail.com>,
Jeff King <peff@peff.net>, Johannes Sixt <j.sixt@viscovery.net>
Subject: Re: [RFC 1/4 v2] Implement a basic remote helper for svn in C.
Date: Sun, 12 Aug 2012 12:06:59 +0200 [thread overview]
Message-ID: <1636924.tANzCnKezB@flobuntu> (raw)
In-Reply-To: <20120801194247.GE24357@copier>
Hi,
back to the pipe-topic.
On Wednesday 01 August 2012 12:42:48 Jonathan Nieder wrote:
> Hi again,
>
> Florian Achleitner wrote:
> > When the first line arrives at the remote-helper, it starts importing one
> > line at a time, leaving the remaining lines in the pipe.
> > For importing it requires the data from fast-import, which would be mixed
> > with import lines or queued at the end of them.
>
> Oh, good catch.
>
> The way it's supposed to work is that in a bidi-import, the remote
> helper reads in the entire list of refs to be imported and only once
> the newline indicating that that list is over arrives starts writing
> its fast-import stream. We could make this more obvious by not
> spawning fast-import until immediately before writing that newline.
>
> This needs to be clearly documented in the git-remote-helpers(1) page
> if the bidi-import command is introduced.
>
> If a remote helper writes commands for fast-import before that newline
> comes, that is a bug in the remote helper, plain and simple. It might
> be fun to diagnose this problem:
This would require all existing remote helpers that use 'import' to be ported
to the new concept, right? Probably there is no other..
>
> static void pipe_drained_or_die(int fd, const char *msg)
> {
> char buf[1];
> int flags = fcntl(fd, F_GETFL);
> if (flags < 0)
> die_errno("cannot get pipe flags");
> if (fcntl(fd, F_SETFL, flags | O_NONBLOCK))
> die_errno("cannot set up non-blocking pipe read");
> if (read(fd, buf, 1) > 0)
> die("%s", msg);
> if (fcntl(fd, F_SETFL, flags))
> die_errno("cannot restore pipe flags");
> }
> ...
>
> for (i = 0; i < nr_heads; i++) {
> write "import %s\n", to_fetch[i]->name;
> }
>
> if (getenv("GIT_REMOTE_HELPERS_SLOW_SANITY_CHECK"))
> sleep(1);
>
> pipe_drained_or_die("unexpected output from remote helper before
> fast-import launch");
>
> if (get_importer(transport, &fastimport))
> die("couldn't run fast-import");
> write_constant(data->helper->in, "\n");
I still don't believe that sharing the input pipe of the remote helper is
worth the hazzle.
It still requires an additional pipe to be setup, the one from fast-import to
the remote-helper, sharing one FD at the remote helper.
It still requires more than just stdin, stdout, stderr.
I would suggest to use a fifo. It can be openend independently, after forking
and on windows they have named pipes with similar semantics, so I think this
could be easily ported.
I would suggest the following changes:
- add a capability to the remote helper 'bidi-import', or 'bidi-pipe'. This
signals that the remote helper requires data from fast-import.
- add a command 'bidi-import', or 'bidi-pipe' that is tells the remote helper
which filename the fifo is at, so that it can open it and read it when it
handles 'import' commands.
- transport-helper.c creates the fifo on demand, i.e. on seeing the capability,
in the gitdir or in /tmp.
- fast-import gets the name of the fifo as a command-line argument. The
alternative would be to add a command, but that's not allowed, because it
changes the stream semantics.
Another alternative would be to use the existing --cat-pipe-fd argument. But
that requires to open the fifo before execing fast-import and makes us
dependent on the posix model of forking and inheriting file descriptors, while
opening a fifo in fast-import would not.
next prev parent reply other threads:[~2012-08-12 10:07 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-06-04 17:20 [RFC 0/4] Florian Achleitner
2012-06-04 17:20 ` [RFC 1/4] Implement a basic remote helper vor svn in C Florian Achleitner
2012-06-04 17:20 ` [RFC 2/4] Integrate remote-svn into svn-fe/Makefile Florian Achleitner
2012-06-04 17:20 ` [RFC 3/4] Add svndump_init_fd to allow reading dumps from arbitrary FDs Florian Achleitner
2012-06-04 17:20 ` [RFC 4/4] Add cat-blob report pipe from fast-import to remote-helper Florian Achleitner
2012-06-05 1:33 ` David Michael Barr
2012-06-05 6:56 ` Jeff King
2012-06-05 7:07 ` David Michael Barr
2012-06-05 8:14 ` Jeff King
2012-06-05 22:16 ` Florian Achleitner
2012-06-06 13:43 ` Jeff King
2012-06-06 21:04 ` Florian Achleitner
2012-06-05 8:51 ` Johannes Sixt
2012-06-05 9:07 ` Jeff King
2012-06-05 22:17 ` Florian Achleitner
2012-06-05 9:09 ` Johannes Sixt
2012-06-05 1:21 ` [RFC 3/4] Add svndump_init_fd to allow reading dumps from arbitrary FDs David Michael Barr
2012-06-29 7:49 ` [RFC 0/4 v2] Florian Achleitner
2012-06-29 7:54 ` [RFC 1/4 v2] Implement a basic remote helper for svn in C Florian Achleitner
2012-07-02 11:07 ` Jonathan Nieder
2012-07-06 0:30 ` Jonathan Nieder
2012-07-06 10:39 ` Florian Achleitner
2012-07-26 8:31 ` Florian Achleitner
2012-07-26 9:08 ` Jonathan Nieder
2012-07-26 16:16 ` Florian Achleitner
2012-07-28 7:00 ` Jonathan Nieder
2012-07-30 8:12 ` Florian Achleitner
2012-07-30 8:29 ` Jonathan Nieder
2012-07-30 13:55 ` Florian Achleitner
2012-07-30 16:55 ` Jonathan Nieder
2012-07-31 19:31 ` Florian Achleitner
2012-07-31 22:43 ` Jonathan Nieder
2012-08-01 8:25 ` Florian Achleitner
2012-08-01 19:42 ` Jonathan Nieder
2012-08-12 10:06 ` Florian Achleitner [this message]
2012-08-12 16:12 ` Jonathan Nieder
2012-08-12 19:39 ` Florian Achleitner
2012-08-12 20:10 ` Jonathan Nieder
2012-08-12 19:36 ` Jonathan Nieder
2012-07-26 17:29 ` Junio C Hamano
2012-07-30 8:12 ` Florian Achleitner
2012-07-26 9:45 ` Steven Michalske
[not found] ` <358E6F1E-8BAD-4F82-B270-0233AB86EF66@gmail.com>
2012-07-26 11:40 ` Jonathan Nieder
2012-07-26 14:28 ` Florian Achleitner
2012-07-26 14:54 ` Jonathan Nieder
2012-07-27 7:23 ` Florian Achleitner
2012-07-28 6:54 ` Jonathan Nieder
2012-06-29 7:58 ` [RFC 2/4 v2] Integrate remote-svn into svn-fe/Makefile Florian Achleitner
2012-06-29 7:59 ` [RFC 3/4 v2] Add svndump_init_fd to allow reading dumps from arbitrary FDs Florian Achleitner
2012-06-29 8:00 ` [RFC 4/4 v2] Add cat-blob report fifo from fast-import to remote-helper Florian Achleitner
2012-07-21 12:45 ` [RFC 4/4 v3] " Florian Achleitner
2012-07-21 14:48 ` Jonathan Nieder
2012-07-21 15:24 ` Florian Achleitner
2012-07-21 15:44 ` Jonathan Nieder
2012-07-22 21:03 ` Florian Achleitner
2012-07-22 21:24 ` Jonathan Nieder
2012-07-21 15:58 ` Jonathan Nieder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1636924.tANzCnKezB@flobuntu \
--to=florian.achleitner.2.6.31@gmail.com \
--cc=davidbarr@google.com \
--cc=git@vger.kernel.org \
--cc=j.sixt@viscovery.net \
--cc=jrnieder@gmail.com \
--cc=peff@peff.net \
--cc=srabbelier@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).