Re: [RFC \ WISH] Add -o option to git-rev-list

[Linus Torvalds <torvalds@osdl.org>]

On Sun, 10 Dec 2006, Marco Costalba wrote:
> 
> - memory use: the next natural step with files is, instead of loading
> the file content in memory and *keep it there*, we could load one
> chunk at a time, index the chunk and discard. At the end we keep in
> memory only indexing info to quickly get to the data when needed, but
> the big part of data stay on the file.

Well, that's still just going slower than swapping.

The reason loading things into memory is nice is that:
 - in the common case, you don't need to do anything else.
 - if  the machine is low on memory, it can page things out just about as 
   easily as you could write things to a file anyway.

So don't worry too much about low-memory situations. Yes, there are cases 
where it's better to keep things in files and simply not have a big 
working set AT ALL, but if you keep something in a file and the file data 
is still part of the working set (ie you read it several times, but at the 
beginning and the end), that really isn't any better than having it in 
memory.

So the time to try to optimize memory usage is really only for "streaming 
behaviour" - where you need to touch something only once. Then the best 
option is to actually use a pipe and re-use the memory, but if you have 
file data, you can use things like fadvise(DONTNEED).

> - This is probably my ignorance, but experimenting with popen() I
> found I could not know *when* git-rev-list ends because both feof()
> and ferror() give 0 after a fread() with git-rev-list already defunct.

I suspect you had a bug somewhere. It could be a bug in stdio, but I doubt 
it.

You do realize that the correct way to check "feof()" is only _after_ 
fread() returns 0? 

Stdio ferror/feof is horrible to use corrrectly, and few people get it 
right.  Mostly because it's such a crap interface thanks to being "easy" 
to use and thus hiding all the error handling details on purpose.