Re: [RFC \ WISH] Add -o option to git-rev-list

[Linus Torvalds <torvalds@osdl.org>]

On Sun, 10 Dec 2006, Marco Costalba wrote:
> 
> I consistently have len == 65536 bytes until the last fread() where
> it's less.

That's because fread() will block until it gets all data. Did you actually 
ever try this with a uncached tree and did you compare what you got with a 
plain "read()". 

On older kernels, I guarantee that you get 4kB at a time for reads, even 
for a blocking pipe. Because we have bigger pipe buffers these days, it 
_may_ return 64kB at a time every time, but only if the writer is much 
faster than the reader.

Based on the fact that you say that "read()" was ten times slower than 
fread(), I very much suspect you got 4kB at a time, and then slept 100ms 
each time or something.

Anyway, you should seriously also check the case when "git rev-list" is 
slow because you have cold caches and unpacked objects. You can't wait for 
it to synchronously write a thousand lines or so - that could take 
seconds.

> How can open in nonblocking mode with popen() ?
> 
> FILE *popen(const char *command, const char *type);

something like

	fcntl(fileno(file), F_SETFL, O_NONBLOCK);

will do it, and then your loop should look something like

	for (;;) {
		int count = read(fileno(file), buf, BUFSIZE);
		if (!count) {
			/* All done, no more to read */
			return 0;
		}
		if (count < 0) {
			if (errno == EAGAIN)
				break;
			if (errno == EINTR)
				continue;
			/* Anything else is fatal - report error */
			return -1;
		}
		... handle 'count' new bytes here ...
	}
	.. this is the EAGAIN case ..
	either set polling on the file descriptor, or use a
	timer here yo get back to this loop eventually.

looks about right.