Re: another packed-refs race

[Jeff King <peff@peff.net>]

On Mon, May 06, 2013 at 02:12:29PM +0200, Michael Haggerty wrote:

> > I think that would be correct (modulo that step 1 cannot happen for
> > enumeration). But we would like to avoid loading all loose refs if we
> > can. Especially on a cold cache, it can be quite slow, and you may not
> > even care about those refs for the current operation (I do not recall
> > the exact original motivation for the lazy loading, but it was something
> > along those lines).
> 
> Lazy loading was first inspired by the observation that effectively
> every git invocation was loading *all* loose references to do an
> iteration over refs/replace/ (which I've never even used!)  This was
> absolutely killing the performance of filter-branch, which creates a lot
> of loose references and invokes git many times--even though the cache
> was warm.

Yeah, obviously we want to avoid that. I _think_ we can even keep the
lazy loading, as long as keep the ordering as:

  1. Load a chunk of loose refs (whatever we need for the current
     iteration request).

  2. Double-check that our packed-refs cache is up to date, and reload
     if necessary.

  3. Return the results to the caller.

It would perhaps increase latency to getting results to the caller
(since otherwise we can start feeding answers to the caller as we walk
the tree), but should not increase the total amount of work (just the
extra stat() of packed-refs, once per for_each_ref, which is not much).

I'll see if I can cook up a patch.

-Peff