* git-remote-fd problem
@ 2014-09-10 13:29 Jiri Sevcik
2014-09-10 14:47 ` Ilari Liusvaara
0 siblings, 1 reply; 4+ messages in thread
From: Jiri Sevcik @ 2014-09-10 13:29 UTC (permalink / raw)
To: git
[-- Attachment #1: Type: text/plain, Size: 828 bytes --]
Hi!
I have problem with using git-remote-fd function. I create two local
pipes for communication for locally running process git pull. Then I
start git-upload-pack on remote side (this process is started in
different part of my code and it works correctly). Communication runs
successfully and when remote side finish, I want to close local pipes
to finish local process. I call close function to all of these pipes,
but unfortunatelly local process doesnt finish (in this moment
transfered data aresaved in /.git/object/pack folder in .pack files)
and hanging on. But if I kill child of this local process, downloaded
files are unpacked and files are created sucessfully. I think this way
is not so correct. Can someone help me and tell me what am I doing
wrong? Thank You. Code is in an attachement (its part of big system).
[-- Attachment #2: git_remote.py --]
[-- Type: text/x-python, Size: 2644 bytes --]
#create local pipes for cimmunication
local_r, local_w = os.pipe()
remote_r, remote_w = os.pipe()
#start local git process
client_process = subprocess.Popen("/usr/bin/git pull fd::{0},{1}".format(remote_r, local_w,), shell=True)
#start git process on remote side
remoteGit = proc.runDaemon("sudo git-upload-pack /tmp/testGit")
#set
epoll = select.epoll()
epoll.register(local_r, select.EPOLLIN)
#register from remote side
epoll.register(proc.fd, select.EPOLLIN)
while True:
events = epoll.poll(1)
for fd, event in events:
if fd == local_r:
#event on local process
if event & select.EPOLLIN:
rd = os.read(local_r, 5000)
if rd:
#write data to remote side
remoteGit.writeToChannel(rd)
else:
proc.writeError("Local socket write error")
client_process.kill()
return 1
else:
proc.writeError("Local socket error")
client_process.kill()
return 1
else:
if event & select.EPOLLIN:
#get all data from remote side
data = remoteGit.getAll()
remoteGit.stderrWrite()
if not data:
#remote side finished - close local pipes - local process wouldnt finish
os.close(local_r)
os.close(local_w)
os.close(remote_r)
os.close(remote_w)
#get local process pid and kill his child to finish git-remote-fd
gitPid = psutil.Process(client_process.pid).children()[0].pid
os.kill(gitPid, signal.SIGTERM)
return 0
want = len(data)
writed = 0
offset = 0
while(writed != want):
#write data from remote side to local process
wr = os.write(remote_w, data[offset:])
if(wr < 0):
proc.writeError("Local socket write error")
return 1
writed += wr
offset += wr
else:
proc.writeError("Remote socket error")
client_process.kill()
return -1
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: git-remote-fd problem
2014-09-10 13:29 git-remote-fd problem Jiri Sevcik
@ 2014-09-10 14:47 ` Ilari Liusvaara
[not found] ` <CAOP4-92U+oFJJw38LWNmTqPtKMT=MDq0Ay9FiaGV5je77aUNpg@mail.gmail.com>
0 siblings, 1 reply; 4+ messages in thread
From: Ilari Liusvaara @ 2014-09-10 14:47 UTC (permalink / raw)
To: Jiri Sevcik; +Cc: git
On Wed, Sep 10, 2014 at 03:29:00PM +0200, Jiri Sevcik wrote:
> Hi!
> I have problem with using git-remote-fd function. I create two local
> pipes for communication for locally running process git pull. Then I
> start git-upload-pack on remote side (this process is started in
> different part of my code and it works correctly). Communication runs
> successfully and when remote side finish, I want to close local pipes
> to finish local process. I call close function to all of these pipes,
> but unfortunatelly local process doesnt finish (in this moment
> transfered data aresaved in /.git/object/pack folder in .pack files)
> and hanging on. But if I kill child of this local process, downloaded
> files are unpacked and files are created sucessfully. I think this way
> is not so correct. Can someone help me and tell me what am I doing
> wrong? Thank You. Code is in an attachement (its part of big system).
The remote-fd expects the transport to pass half-closes. So you can't
close all at once.
Let there be pipes W and R and transport connection C.
- W-read should be closed after being passed to remote-fd.
- R-write should be closed after being passed to remote-fd.
- Upon receiving "no more data" from C, close W-write.
- Upon receiving EOF from R-read, close it and signal "no more data"
to C.
If you have server end, the same applies, but with remote-fd replaced
by upload-pack/upload-archive/receive-pack.
-Ilari
^ permalink raw reply [flat|nested] 4+ messages in thread
* Fwd: git-remote-fd problem
[not found] ` <CAOP4-92U+oFJJw38LWNmTqPtKMT=MDq0Ay9FiaGV5je77aUNpg@mail.gmail.com>
@ 2014-12-29 9:47 ` Jiri Sevcik
2014-12-29 19:36 ` Ilari Liusvaara
0 siblings, 1 reply; 4+ messages in thread
From: Jiri Sevcik @ 2014-12-29 9:47 UTC (permalink / raw)
To: Ilari Liusvaara; +Cc: git
[-- Attachment #1: Type: text/plain, Size: 569 bytes --]
> The remote-fd expects the transport to pass half-closes. So you can't
> close all at once.
>
> Let there be pipes W and R and transport connection C.
>
> - W-read should be closed after being passed to remote-fd.
> - R-write should be closed after being passed to remote-fd.
> - Upon receiving "no more data" from C, close W-write.
> - Upon receiving EOF from R-read, close it and signal "no more data"
> to C.
Hi, I followed your advices, correctly close pipes but git clone still
doesnt finish and hanging on.
Code is in an attachement (its part of big system).
[-- Attachment #2: git_test_improved.py --]
[-- Type: text/x-python, Size: 2018 bytes --]
#create pipes
w_pipe = os.pipe()
r_pipe = os.pipe()
client_process = subprocess.Popen("/usr/bin/git clone fd::{0},{1} /tmp/gittest".format(r_pipe[0], w_pipe[1]), shell=True)
#closing pipes
os.close(r_pipe[0])
os.close(w_pipe[1])
epoll = select.epoll()
epoll.register(w_pipe[0], select.EPOLLIN)
epoll.register(proc.fd, select.EPOLLIN)
remoteGit = proc.runDaemon("git-upload-pack /tmp/testgit")
while True:
events = epoll.poll(1)
for fd, event in events:
if fd == w_pipe[0]:
if event & select.EPOLLIN:
rd = os.read(w_pipe[0], 10000)
if rd:
#write data to remove git server
remoteGit.writeToChannel(rd)
else:
proc.writeError("Local socket write error")
return 1
else:
proc.writeError("Local socket error")
return 1
elif fd == proc.fd:
if event & select.EPOLLIN:
#read data from remote git server
data = remoteGit.getAll()
remoteGit.stderrWrite()
if not data:
#remote server send EOF, close local pipe
#but git clone is still running
os.close(r_pipe[1])
return 0
want = len(data)
writed = 0
offset = 0
while(writed != want):
#write data from remote git server to local pipe
wr = os.write(r_pipe[1], data[offset:])
if(wr < 0):
return 1
writed += wr
offset += wr
else:
return -1
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: Fwd: git-remote-fd problem
2014-12-29 9:47 ` Fwd: " Jiri Sevcik
@ 2014-12-29 19:36 ` Ilari Liusvaara
0 siblings, 0 replies; 4+ messages in thread
From: Ilari Liusvaara @ 2014-12-29 19:36 UTC (permalink / raw)
To: Jiri Sevcik; +Cc: git
On Mon, Dec 29, 2014 at 10:47:58AM +0100, Jiri Sevcik wrote:
> > The remote-fd expects the transport to pass half-closes. So you can't
> > close all at once.
> >
> > Let there be pipes W and R and transport connection C.
> >
> > - W-read should be closed after being passed to remote-fd.
> > - R-write should be closed after being passed to remote-fd.
> > - Upon receiving "no more data" from C, close W-write.
> > - Upon receiving EOF from R-read, close it and signal "no more data"
> > to C.
>
> Hi, I followed your advices, correctly close pipes but git clone still
> doesnt finish and hanging on.
> Code is in an attachement (its part of big system).
Few ideas:
- Check that git clone (and its subprocesses) don't inherit
w_pipe[0] (/proc/<pid>/fd on Linux might be handy). If they do, that
prevents this program from closing the pipe.
- Setting environment variable GIT_TRANSLOOP_DEBUG to 1 might make
git spew lots of messages to stderr about reads, writes and closes.
>
> #create pipes
> w_pipe = os.pipe()
> r_pipe = os.pipe()
>
> client_process = subprocess.Popen("/usr/bin/git clone fd::{0},{1} /tmp/gittest".format(r_pipe[0], w_pipe[1]), shell=True)
> #closing pipes
> os.close(r_pipe[0])
> os.close(w_pipe[1])
>
> epoll = select.epoll()
> epoll.register(w_pipe[0], select.EPOLLIN)
> epoll.register(proc.fd, select.EPOLLIN)
>
> remoteGit = proc.runDaemon("git-upload-pack /tmp/testgit")
>
> while True:
> events = epoll.poll(1)
>
> for fd, event in events:
> if fd == w_pipe[0]:
> if event & select.EPOLLIN:
> rd = os.read(w_pipe[0], 10000)
> if rd:
> #write data to remove git server
> remoteGit.writeToChannel(rd)
> else:
> proc.writeError("Local socket write error")
> return 1
> else:
> proc.writeError("Local socket error")
> return 1
>
> elif fd == proc.fd:
> if event & select.EPOLLIN:
> #read data from remote git server
> data = remoteGit.getAll()
> remoteGit.stderrWrite()
>
> if not data:
> #remote server send EOF, close local pipe
> #but git clone is still running
> os.close(r_pipe[1])
> return 0
>
> want = len(data)
>
> writed = 0
> offset = 0
>
> while(writed != want):
> #write data from remote git server to local pipe
> wr = os.write(r_pipe[1], data[offset:])
>
> if(wr < 0):
> return 1
>
> writed += wr
> offset += wr
>
> else:
> return -1
-Ilari
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2014-12-29 19:43 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-09-10 13:29 git-remote-fd problem Jiri Sevcik
2014-09-10 14:47 ` Ilari Liusvaara
[not found] ` <CAOP4-92U+oFJJw38LWNmTqPtKMT=MDq0Ay9FiaGV5je77aUNpg@mail.gmail.com>
2014-12-29 9:47 ` Fwd: " Jiri Sevcik
2014-12-29 19:36 ` Ilari Liusvaara
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).