Re: [PATCH v2] convert: add "status=delayed" to filter process protocol

[Jeff King <peff@peff.net>]

On Mon, Feb 27, 2017 at 11:32:47AM +0100, Lars Schneider wrote:

> I completely agree - I need to change that. However, the goal of the v2
> iteration was to get the "convert" interface in an acceptable state.
> That's what I intended to say in the patch comment section:
> 
>     "Please ignore all changes behind async_convert_to_working_tree() and 
>      async_filter_finish() for now as I plan to change the implementation 
>      as soon as the interface is in an acceptable state."

Ah, sorry, I missed that. I would think the underlying approach would
influence the interface to some degree. But as long as the interface
is sufficiently abstract, I think it gives you enough flexibility.

> > From Git's side, the loop is something like:
> > 
> >  while (delayed_items > 0) {
> > 	/* issue a wait, and get back the status/index pair */
> > 	status = send_wait(&index);
> > 	delayed_items--;
> > 
> > 	/*
> > 	 * use "index" to find the right item in our list of files;
> > 	 * the format can be opaque to the filter, so we could index
> > 	 * it however we like. But probably numeric indices in an array
> > 	 * are the simplest.
> > 	 */
> > 	assert(index > 0 && index < nr_items);
> > 	item[index].status = status;
> > 	if (status == SUCCESS)
> > 		read_content(&item[index]);
> >  }
> > 
> > and the filter side just attaches the "index" string to whatever its
> > internal queue structure is, and feeds it back verbatim when processing
> > that item finishes.
> 
> That could work! I had something like that in mind:
> 
> I teach Git a new command "list_completed" or similar. The filter
> blocks this call until at least one item is ready for Git. 
> Then the filter responds with a list of paths that identify the
> "ready items". Then Git asks for these ready items just with the
> path and not with any content. Could that work? Wouldn't the path
> be "unique" to identify a blob per filter run?

I think that could work, though I think there are few minor downsides
compared to what I wrote above:

  - if you respond with "these items are ready", and then make Git ask
    for each again, it's an extra round-trip for each set of ready
    items. You could just say "an item is ready; here it is" in a single
    response. For a local pipe the latency is probably negligible,
    though.

  - using paths as the index would probably work, but it means Git has
    to use the path to find the "struct checkout_entry" again. Which
    might mean a hashmap (though if you have them all in a sorted list,
    I guess you could also do a binary search).

  - Using an explicit index communicates to the filter not only what the
    index is, but also that Git is prepared to accept a delayed response
    for the item. For backwards compatibility, the filter would probably
    advertise "I have the 'delayed' capability", and then Git could
    choose to use it or not on a per-item basis. Realistically it would
    not change from item to item, but rather operation to operation. So
    that means we can easily convert the call-sites in Git to the async
    approach incrementally. As each one is converted, it turns on the
    flag that causes the filter code to send the "index" tag.

-Peff