From: Jason Vas Dias <jason.vas.dias@gmail.com>
To: Jeff King <peff@peff.net>, Eric Wong <normalperson@yhbt.net>
Cc: git@vger.kernel.org
Subject: Re: clone hang prevention / timeout?
Date: Thu, 14 Apr 2016 19:32:24 +0100 [thread overview]
Message-ID: <CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com> (raw)
In-Reply-To: <20160413222900.GA10011@sigill.intra.peff.net>
Thanks very much Eric & Jeff for your reply .
Personally, I would recommend setting the SO_RECVTIMEO for GIT server
sockets to a fixed default (eg. 5mins) , settable by a
'--receive-timeout' argument or configuration parameter .
The problem I was trying to overcome was cloning all the repositories under
https://anongit.freedesktop.org/xorg/* .
About 4 git clones would succeed in succession, but then typically the 5th
would hang in read() forever - I left one such hung 'git clone' for nearly an
hour and it had not progressed or timed out . I tried inserting a delay of
up to 30 seconds between clones, but this did not help.
Maybe freedesktop.org's GIT server is too overloaded and they have
to resort to disabling 1 out of 5 GIT successive clone operations from
same connection or something.
Here is my solution, in case anyone else needs it :
<quote><pre>
eips=()
counts=()
declare -i failed=0;
{ echo "$BASHPID" >/tmp/git.pid;
GIT_TRACE=2 exec git clone
${proto}://${user}anongit.freedesktop.org/${repo}$name; }&
while [ ! -f /tmp/git.pid ]; do sleep 1; done
git_pid="$(cat /tmp/git.pid)";
while [ -d /proc/$git_pid ]; do
IFS=$'\n';
declare -a kids=($(ps --ppid $git_pid -o 'pid=,eip='));
unset IFS;
declare -i n_kids=${#kids[@]} kid_n;
for ((kid_n=0; kid_n < n_kids; kid_n+=1)); do
declare -a ke=(${kids[kid_n]});
kid=${ke[0]}
eip=${ke[1]}
if [ ! -v 'eips['$kid']' ]; then
eips[$kid]="$eip";
elif [ "${eips[$kid]}" = "$eip" ]; then
if [ x = x"${counts[$kid]}" ]; then
counts[$kid]=1;
else
counts[$kid]=$((${counts[$kid]}+1));
if (( ${counts[$kid]} >= 30 )); then
echo 'child process '$kid' of git main process
'$git_pid' appears to be stuck - killing it.';
kill -TERM $kid;
((failed=1));
fi
fi
else
eips[$kid]="$eip";
counts[$kid]='';
fi
done ;
sleep 1;
done
wait
</quote></pre>
This is part of a script that reads a list of the Xorg projects,
sets $repo to top level subdirectory, and $name to the project name,
and initiates the GIT clone .
It deems any GIT _CHILD_ process (eg. git-index-pack) that have not
changed their instruction pointer register (EIP) for 30 seconds to be
"hung" .
There is logic at the end to retry all the failed clones.
It does work, but is far from pretty .
It sure would be nice if GIT had a timeout mechanism !
Thanks & Regards,
Jason
On 13/04/2016, Jeff King <peff@peff.net> wrote:
> On Mon, Apr 11, 2016 at 10:49:19PM +0100, Jason Vas Dias wrote:
>
>> Is there any option I can specify to get the clone to timeout, or do I
>> manually
>> have to strace the git process and send it a signal after a hang is
>> detected?
>
> Oh, one other thing you might consider, it something like "timeout" from
> GNU coreutils, which puts a hard cap on the length of time a process can
> run.
>
> It's totally unaware of the state of the process, though, so if you
> really do have a clone which takes an hour, it might very well kill it
> at 99% complete. It has no mechanism for "gee, this process looks like
> it hasn't done anything for 5 minutes".
>
> I don't know offhand of a general tool for that.
>
> -Peff
>
next prev parent reply other threads:[~2016-04-14 18:32 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-04-11 21:49 clone hang prevention / timeout? Jason Vas Dias
2016-04-12 8:01 ` Eric Wong
2016-04-13 22:24 ` Jeff King
2016-04-13 22:29 ` Jeff King
2016-04-14 18:32 ` Jason Vas Dias [this message]
2016-04-30 9:04 ` Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALyZvKzzB9yv-aavw2RH7D7fK=vG88-j9W0LAmD3P6=CkSJCuw@mail.gmail.com' \
--to=jason.vas.dias@gmail.com \
--cc=git@vger.kernel.org \
--cc=normalperson@yhbt.net \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).