git+http:// proof-of-concept (not using CONNECT)

git+http:// proof-of-concept (not using CONNECT)

[Eric Wong]

I've been toying around with an HTTP server for various things over the
past few months and a few weeks ago I came up with the idea of tunneling
arbitrary stream protocols over HTTP with "Transfer-Encoding: chunked"
bodies on both ends.  This allowed me to tunnel the git:// protocol
among other things over something that may conform to RFC 2616.

To facilitate this, curl needed only one minor modification (now in
their CVS repository) to make stdin with "-T-" non-blocking:

  http://cool.haxx.se/cvs.cgi/curl/src/main.c.diff?r1=1.520&r2=1.521

Since git clone already supports proxy commands via GIT_PROXY_COMMAND, I
just created a one line shell script to use as GIT_PROXY_COMMAND.  No
modifications were needed to git itself on either end:

cat > /path/to/script <<\EOF
#!/bin/sh
exec curl --no-buffer -sSfT- http://$1:$2/
EOF

chmod +x /path/to/script
GIT_PROXY_COMMAND=/path/to/script
export GIT_PROXY_COMMAND

# Then, to test it on a small project on my server:
git clone git://git.bogomips.org:8080/pcu

# Of course, a larger project like git should work, too:
git clone git://git.bogomips.org:8080/mirrors/git

Hopefully it doesn't blow up or die on you, this is only lightly
tested.  Let me know if you manage to break it permanently.


The origin server I'm running for this is the latest release of
Unicorn[2], which supports sending a chunked HTTP response as it is
receiving a chunked request.  Unicorn just dechunks the request and
pipes it to git-daemon.  When git-daemon writes to stdout, Unicorn just
grabs the output and chunks it (via Rack[3] middleware) for the client.


This doesn't work in the face of most HTTP-aware proxies[1], so it
probably doesn't help those who have trouble accessing git:// servers in
the first place...  However, this could potentially be useful in places
where a proxy providing CONNECT is not available.


[1] - I run nginx on port 80 on bogomips.org and nginx (attempts to)
fully buffer all requests before proxying it to the backend, so it
definitely won't fly here.  HTTP proxies are perfectly alright with
taking chunked requests/responses and remove chunking on them (or
vice versa).

[2] - http://unicorn.bogomips.org/
[3] - http://rack.rubyforge.org/

-- 
Eric Wong

Re: git+http:// proof-of-concept (not using CONNECT)

[Constantine Plotnikov]

On Thu, Jul 2, 2009 at 12:54 PM, Eric Wong<normalperson@yhbt.net> wrote:
> This doesn't work in the face of most HTTP-aware proxies[1], so it
> probably doesn't help those who have trouble accessing git:// servers in
> the first place...  However, this could potentially be useful in places
> where a proxy providing CONNECT is not available.
>
The current lore is that to work with http proxies two separate
streams are needed,
one for input and one for outputs.

This technique is described at
http://xmpp.org/extensions/xep-0124.html#technique
and other places.

Regards,
Constantine

Re: git+http:// proof-of-concept (not using CONNECT)

[Eric Wong]

Constantine Plotnikov <constantine.plotnikov@gmail.com> wrote:
> On Thu, Jul 2, 2009 at 12:54 PM, Eric Wong<normalperson@yhbt.net> wrote:
> > This doesn't work in the face of most HTTP-aware proxies[1], so it
> > probably doesn't help those who have trouble accessing git:// servers in
> > the first place...  However, this could potentially be useful in places
> > where a proxy providing CONNECT is not available.
> >
> The current lore is that to work with http proxies two separate
> streams are needed,
> one for input and one for outputs.
> 
> This technique is described at
> http://xmpp.org/extensions/xep-0124.html#technique
> and other places.

Actually, the problem with my approach here is that any HTTP proxies are
free to rewrite chunked requests/responses to non-chunked ones (for
compression, ssl, or even performance reasons).  That and my approach
requires both the client and server to be simutaneously sending and
receiving responses in full-duplex channel which makes it impossible to
work without chunking.  IOW, there's no chance any HTTP proxy that
dechunks or queues/coalesces chunked requests/responses can work with
my approach.

The big upside is that only one HTTP request is needed for a full clone,
making this the most efficient way of using git over HTTP so far :)

-- 
Eric Wong

Re: git+http:// proof-of-concept (not using CONNECT)

[Tony Finch]

On Fri, 3 Jul 2009, Eric Wong wrote:
>
> That and my approach requires both the client and server to be
> simutaneously sending and receiving responses in full-duplex channel
> which makes it impossible to work without chunking.  IOW, there's no
> chance any HTTP proxy that dechunks or queues/coalesces chunked
> requests/responses can work with my approach.

Many HTTP servers will not be able to support it either because HTTP is a
half-duplex protocol (modulo request pipelining).

Tony.
-- 
f.anthony.n.finch  <dot@dotat.at>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

Re: git+http:// proof-of-concept (not using CONNECT)

[Eric Wong]

Tony Finch <dot@dotat.at> wrote:
> On Fri, 3 Jul 2009, Eric Wong wrote:
> >
> > That and my approach requires both the client and server to be
> > simutaneously sending and receiving responses in full-duplex channel
> > which makes it impossible to work without chunking.  IOW, there's no
> > chance any HTTP proxy that dechunks or queues/coalesces chunked
> > requests/responses can work with my approach.
> 
> Many HTTP servers will not be able to support it either because HTTP is a
> half-duplex protocol (modulo request pipelining).

Heh, as far as I know Unicorn is the only one :)

-- 
Eric Wong

Re: git+http:// proof-of-concept (not using CONNECT)

[Douglas Campos]

Any news about this approach? I've heard some noise about a CGI
implementation....

-- 
Douglas Campos
qmx.me

Re: git+http:// proof-of-concept (not using CONNECT)

[Eric Wong]

Douglas Campos <douglas@theros.info> wrote:
> Any news about this approach? I've heard some noise about a CGI
> implementation....

Hi Douglass,

I've been busy with other things.  This approach is probably
too limited to get around HTTP-aware proxies to be useful for
non-git:// users anyways.

It also turns out the curl patch that made it into 7.19.6 was still
unsuitable for bidirectional tunneling, looks like I'll have to port
curl over to the curl_multi_* interface to get it working properly
instead...

-- 
Eric Wong