Re: Looking for help to understand external filter driver code

[Lars Schneider <larsxschneider@gmail.com>]

On 19 Jul 2016, at 23:33, Junio C Hamano <gitster@pobox.com> wrote:

> Lars Schneider <larsxschneider@gmail.com> writes:
> 
>> Git writes --> 4 byte filename length
>> Git writes --> filename string
> 
> Why limit to 32GB?  Perhaps NUL termination is more appropriate
> here?
OK, I will use NUL termination for the filename. 
You're also right about the limit - I will use 8 byte to encode the 
content length.


>> Git writes --> 4 byte content length
>> Git writes --> content string
>> Git reads <-- 4 byte filtered content length
>> Git reads <-- filtered content
> 
> Do you really need to force the sender to know the length in
> advance?  Together with the sequential nature of the above exchange,
> i.e. the filter is forbidden from producing even a single byte of
> its output before reading everything Git feeds it, you are making it
> impossible to use filters that perform streaming conversion.
That is correct. However, for my particular use case streaming
conversion wouldn't be useful anyways:
https://github.com/github/git-lfs/pull/1382


> Of course, with the "sequential" thing, you do not have to worry
> about deadlocking hence no need for poll/select, but I am not sure
> that is a good thing.
Thanks for the confirmation. I consider to exchange a "filter protocol 
version" right after the filter process has started. That way someone 
could add a more evolved "filter driver protocol" later on that supports 
streaming and the external filter could pick whatever protocol is most
appropriate (and supported). Could that be an acceptable compromise
to get a serious review of the "sequential" thing?

Thanks,
Lars