slow process of post-receive script on a remote (samba) share

slow process of post-receive script on a remote (samba) share

[Tamas Csabina]

Dear list,

I am using Git bash from version 1.8.3.msysgit.0, on a Windows 7x64 PC.
I have an issue with executing git push if I have a post-receive
script configured.
The content of the script is not really important, as if I have a
script that contains only commented out lines (around 70 lines), my
git push command is delayed with around 5 seconds.


I`ve tested the script on another PC and it is working fine. No delay
at all. So there are some issues on my PC regarding how git processes
remote scripts.

I took a wireshark trace with 2 scenarios on my PC:

 1. just execute `cat <path_to_the_script>\post-receive` command in the git bash
 2. did a `real` git push

Results of the wireshark traces shows:

 1. Read AndX Request, FID: 0x228f, 1024 bytes at offset 0 (1024 bytes
at time, always)
 2. Read AndX Request, FID: 0x21c9, 1 byte at offset 0 (1 byte, always)

Conclusion:
git push command reads the post-receive script in 1 byte chunks, which
dramatically slows down the execution process.


If more information is required about my setup or configuration, I
will provided it happily, but I think this 1 byte read is the main
reason for the issue.

Has anyone seen something similar? Or have any clues what is going on?


Thanks,
tamas

ps: this is my 3rd (or 4th) attempt to post on the mailing list, using
a different email account this time. If any of the previous attempt
also arrived, then sorry for the double posts.

Re: slow process of post-receive script on a remote (samba) share

[Thomas Rast]

Tamas Csabina <tcsabina@gmail.com> writes:

> I am using Git bash from version 1.8.3.msysgit.0, on a Windows 7x64 PC.
> I have an issue with executing git push if I have a post-receive
> script configured.
> The content of the script is not really important, as if I have a
> script that contains only commented out lines (around 70 lines), my
> git push command is delayed with around 5 seconds.
>
>
> I`ve tested the script on another PC and it is working fine. No delay
> at all. So there are some issues on my PC regarding how git processes
> remote scripts.
>
> I took a wireshark trace with 2 scenarios on my PC:
>
>  1. just execute `cat <path_to_the_script>\post-receive` command in the git bash
>  2. did a `real` git push
>
> Results of the wireshark traces shows:
>
>  1. Read AndX Request, FID: 0x228f, 1024 bytes at offset 0 (1024 bytes
> at time, always)
>  2. Read AndX Request, FID: 0x21c9, 1 byte at offset 0 (1 byte, always)
>
> Conclusion:
> git push command reads the post-receive script in 1 byte chunks, which
> dramatically slows down the execution process.

git doesn't read the script; the interpreter does.  In the case of a
script, the interpreter is specified in the #! line at the top; in the
case of a binary executable, it is specified within the executable (and
for linux, is usually /lib/ld-linux.so.2).

Exactly the same should happen if you run the hook manually, so you can
try that to debug it.

Note also that Weird Things(tm) relating to SIGPIPE may happen if you
don't read your input.  Even if you are only fooling around for testing,
a post-receive hook must consume its input, e.g., by discarding it with
'cat >/dev/null'.

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: slow process of post-receive script on a remote (samba) share

[Tamas Csabina]

Hi Thomas,

Thanks for the reply.
The script is a bash script, just to mention.

Meanwhile I`ve figured it out that the sluggish post-receive execution
was due to a (mis)-configuration in the samba share where the remote
repository is hosted. These are:
oplocks = No
level2 oplocks = No

Removing these from the share`s section in the smb.conf solved the
issue, and the push process is taking up around 4 seconds, which I
think is reliable.


Now, do I have to worry about allowing oplocks on the remote
repository from the git point of view? Thinking about concurrent push
operations from different developers?


Tamas


On 13 June 2013 14:19, Thomas Rast <trast@inf.ethz.ch> wrote:
> Tamas Csabina <tcsabina@gmail.com> writes:
>
>> I am using Git bash from version 1.8.3.msysgit.0, on a Windows 7x64 PC.
>> I have an issue with executing git push if I have a post-receive
>> script configured.
>> The content of the script is not really important, as if I have a
>> script that contains only commented out lines (around 70 lines), my
>> git push command is delayed with around 5 seconds.
>>
>>
>> I`ve tested the script on another PC and it is working fine. No delay
>> at all. So there are some issues on my PC regarding how git processes
>> remote scripts.
>>
>> I took a wireshark trace with 2 scenarios on my PC:
>>
>>  1. just execute `cat <path_to_the_script>\post-receive` command in the git bash
>>  2. did a `real` git push
>>
>> Results of the wireshark traces shows:
>>
>>  1. Read AndX Request, FID: 0x228f, 1024 bytes at offset 0 (1024 bytes
>> at time, always)
>>  2. Read AndX Request, FID: 0x21c9, 1 byte at offset 0 (1 byte, always)
>>
>> Conclusion:
>> git push command reads the post-receive script in 1 byte chunks, which
>> dramatically slows down the execution process.
>
> git doesn't read the script; the interpreter does.  In the case of a
> script, the interpreter is specified in the #! line at the top; in the
> case of a binary executable, it is specified within the executable (and
> for linux, is usually /lib/ld-linux.so.2).
>
> Exactly the same should happen if you run the hook manually, so you can
> try that to debug it.
>
> Note also that Weird Things(tm) relating to SIGPIPE may happen if you
> don't read your input.  Even if you are only fooling around for testing,
> a post-receive hook must consume its input, e.g., by discarding it with
> 'cat >/dev/null'.
>
> --
> Thomas Rast
> trast@{inf,student}.ethz.ch

Re: slow process of post-receive script on a remote (samba) share

[Thomas Rast]

Tamas Csabina <tcsabina@gmail.com> writes:

> Meanwhile I`ve figured it out that the sluggish post-receive execution
> was due to a (mis)-configuration in the samba share where the remote
> repository is hosted. These are:
> oplocks = No
> level2 oplocks = No
[...]
> Now, do I have to worry about allowing oplocks on the remote
> repository from the git point of view? Thinking about concurrent push
> operations from different developers?

>From a brief glance at the relevant docs [1], it would seem that oplocks
are actually just an implementation detail for safe (in the context of
parallel access) client caching.  So they should be fully transparent to
any application usage.  However, the docs do mention that you may be in
trouble if the connection to the server times out.

That being said, some FSes see more usage and thus have been tested more
in this context, and git does tend to show some pretty weird issues on
broken network FSes (one such case: Lustre[2]).

In addition, there are some known races w.r.t. the handling of refs, and
of pruning, if you run git-gc while concurrent pushes are going on.
Jeff King and Michael Haggerty are currently working on getting them
fixed, see e.g. [3].  To see these, you'll have to hit the repo much
harder than a small team can.

So it *should* work, at least if you disable gc.auto and run git-gc
manually at some safe time.  But I wouldn't be surprised if there are
bugs lurking in the context of Windows usage on a Samba-hosted repo,
which sounds like a very rare combination.

And in any case, don't take my word for it; if your life or company
depends on this, you'll need to do your own testing to ensure that it
holds up.


Oh, and why do it that way?  You would most likely get much better
performance out of it if you hosted the repo over ssh (e.g. with
gitolite[4]) or a smart-http server, since the expensive operations (and
they are *expensive*) would be completely local to the server.  The
tradeoff there is that it also shifts a lot of CPU work to the server,
but if you can afford it, you should see a great speedup especially when
fetching large amounts of data (e.g. at cloen time).



[1]  http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html#id2615667

[2]  http://thread.gmane.org/gmane.comp.version-control.git/212109

[3]  http://thread.gmane.org/gmane.comp.version-control.git/223299

[4]  http://gitolite.com/gitolite/

-- 
Thomas Rast
trast@{inf,student}.ethz.ch

Re: slow process of post-receive script on a remote (samba) share

[Tamas Csabina]

Thanks for the follow up ideas.

We also decided to access the repos through ssh. With that, there is
no delay at all.


Regards,
Tamas



On 13 June 2013 16:29, Thomas Rast <trast@inf.ethz.ch> wrote:
> Tamas Csabina <tcsabina@gmail.com> writes:
>
>> Meanwhile I`ve figured it out that the sluggish post-receive execution
>> was due to a (mis)-configuration in the samba share where the remote
>> repository is hosted. These are:
>> oplocks = No
>> level2 oplocks = No
> [...]
>> Now, do I have to worry about allowing oplocks on the remote
>> repository from the git point of view? Thinking about concurrent push
>> operations from different developers?
>
> From a brief glance at the relevant docs [1], it would seem that oplocks
> are actually just an implementation detail for safe (in the context of
> parallel access) client caching.  So they should be fully transparent to
> any application usage.  However, the docs do mention that you may be in
> trouble if the connection to the server times out.
>
> That being said, some FSes see more usage and thus have been tested more
> in this context, and git does tend to show some pretty weird issues on
> broken network FSes (one such case: Lustre[2]).
>
> In addition, there are some known races w.r.t. the handling of refs, and
> of pruning, if you run git-gc while concurrent pushes are going on.
> Jeff King and Michael Haggerty are currently working on getting them
> fixed, see e.g. [3].  To see these, you'll have to hit the repo much
> harder than a small team can.
>
> So it *should* work, at least if you disable gc.auto and run git-gc
> manually at some safe time.  But I wouldn't be surprised if there are
> bugs lurking in the context of Windows usage on a Samba-hosted repo,
> which sounds like a very rare combination.
>
> And in any case, don't take my word for it; if your life or company
> depends on this, you'll need to do your own testing to ensure that it
> holds up.
>
>
> Oh, and why do it that way?  You would most likely get much better
> performance out of it if you hosted the repo over ssh (e.g. with
> gitolite[4]) or a smart-http server, since the expensive operations (and
> they are *expensive*) would be completely local to the server.  The
> tradeoff there is that it also shifts a lot of CPU work to the server,
> but if you can afford it, you should see a great speedup especially when
> fetching large amounts of data (e.g. at cloen time).
>
>
>
> [1]  http://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/locking.html#id2615667
>
> [2]  http://thread.gmane.org/gmane.comp.version-control.git/212109
>
> [3]  http://thread.gmane.org/gmane.comp.version-control.git/223299
>
> [4]  http://gitolite.com/gitolite/
>
> --
> Thomas Rast
> trast@{inf,student}.ethz.ch