Re: [PATCH 00/30] [RFC] extensions.refFormat and packed-refs v2 file format

[Elijah Newren <newren@gmail.com>]

On Wed, Nov 16, 2022 at 6:45 AM Derrick Stolee <derrickstolee@github.com> wrote:
>
> On 11/14/22 9:47 PM, Elijah Newren wrote:
> > On Sun, Nov 13, 2022 at 4:07 PM Derrick Stolee <derrickstolee@github.com> wrote:
> >>
> >> On 11/11/22 6:28 PM, Elijah Newren wrote:
> >>> On Mon, Nov 7, 2022 at 11:01 AM Derrick Stolee via GitGitGadget
> >>> <gitgitgadget@gmail.com> wrote:
[...]
> >>>>  * (Secondary) Allow using a packed ref format for all refs, dropping loose
> >>>>    refs and creating a clear way to snapshot all refs at a given point in
> >>>>    time.
[...]
>
> The reason is in the goal "creating a clear way to snapshot all refs
> at a given point in time". This is a server-side benefit with no
> visible benefit to users, immediately.

Yes, sorry, I just missed it.  I didn't understand it and wrongly
assumed it was continuing to talk about the implementation details
rather than the benefit details.  My bad.

Thanks for patiently correcting me.

> The D/F conflicts and case-sensitive parts that could fall from that
> are not included in my goals. Part of that is because we would need a
> new reflog format to complete that part. Let's take things one step
> at a time and handle reflogs after we have ref update performance
> handled.

Ah, right, I can see how reflog would affect both of those problems
now that you highlight it, but it hadn't occurred to me before.

> >> The biggest benefit on the server side is actually for consistency
> >> checks. Using a stacked packed-refs (especially with a tip file
> >> that describes all of the layers) allows an atomic way to take a
> >> snapshot of the refs and run a checksum operation on their values.
> >> With loose refs, concurrent updates can modify the checksum during
> >> its computation. This is a super niche reason for this, but it's
> >> nice that the performance-only focus also ends up with a design
> >> that satisfies this goal.
> >
> > Ah...so this is the reason for your secondary goal?  Re-reading it
> > looks like you did state this, I just missed it without the longer
> > explanation.
> >
> > Anyway, it might be worth calling out in your cover letter that there
> > are (at least) three benefits to this secondary goal of yours -- the
> > one you list here, plus the two I list above.
>
> I suppose I assumed that the D/F and case conflicts were a "known"
> benefit and a huge motivation of the reftable work.

Yes, and I thought you had just found a simpler solution to those
problems that might not provide all the benefits of reftable (e.g.
performance with huge numbers of refs) but did solve those particular
problems.  I've only looked at reftable from the surface from a
distance, and I was unaware previously that reflog also affected these
two problems (though it seems obvious in hindsight).  And I do
remember you calling out that you weren't changing the reflog format
in your cover letter, but I didn't understand the ramifications of
that statement at the time.

> Instead of trying
> to solve all of the ref problems at once, I wanted to focus on the
> subset that I knew could be solved with a simpler solution, leaving
> the full solution to later steps. It would help to be explicit about
> how this direction helps solve this problem while also being clear
> about how it does not solve it completely.

It certainly would have helped me.  :-)

Thanks for explaining all these details.