Evaluation of ref-api branch status

Evaluation of ref-api branch status

[Michael Haggerty]

Junio,

Now that 1.7.8 is out, I wanted to figure out the status of the
remaining ref-api changes that are in flight, including the differences
between between my tree and yours.

I understand that "next" will soon be re-rolled.  Will the re-roll be
based on the current "pu", or will you start from scratch?


AFAIK there are three groups of ref-api patch series in flight, some of
which have gone through multiple iterations:

ref-api-B == mh/ref-api-2
    [PATCH v3 00/14] Tidying up references code

ref-api-C == mh/ref-api-3
    [PATCH v2 00/12] Use refs API more consistently

ref-api-D == mh/ref-api-take-2
    [PATCH 00/28] Store references hierarchically in cache

On 2011-10-28 I sent re-rolled versions of ref-api-B and ref-api-C to
the list followed by the first version of ref-api-D.  You committed
these three patch series to your repository as a single branch,
"mh/ref-api-take-2".  The mh/ref-api-take-2 branch thus includes the
logical equivalent of all three of the patch series listed above.

Additionally, on 2011-11-15 I sent a fix to patch 26/28 of ref-api-D:

    [PATCH] Fix "is_refname_available(): query only possibly-conflicting
 references"

I don't see this patch in your tree at all.


Meanwhile, you added the following fix on top of master, mh/ref-api-2,
and ref-api-3:

    refs: loosen over-strict "format" check

[Aside: I don't understand why you created three independent commits
rather than create an ur-commit on $(git merge-base master mh/ref-api-2
mh/ref-api-3) then merging it to each of the branches.]


Both mh/ref-api-2 (including your fix) and mh/ref-api-3 (including your
fix) have been merged into "next".  You have never merged
mh/ref-api-take-2 anywhere.  On the mh/ref-api-take-2 branch:

* ce766d41fa corresponds to mh/ref-api-2 (without your fix)
* 633ebc45c0 corresponds to mh/ref-api-3 (without your fix)

Therefore, your 633ebc45c0..mh/ref-api-take-2 corresponds to my newest
"ref-api-D" patch series.  It does not include my fix from 2011-11-15.
Nor does it include any form of your fix "refs: loosen over-strict
"format" check".


What is the difference between the beginning of mh/ref-api-take-2 as
compared to mh/ref-api-2 + mh/ref-api-3?  Equivalently, what is the
difference between the last versions of ref-api-B and ref-api-C vs.
earlier versions of those patch series?  The differences are relatively
minor:

* A whitespace problem in the earlier version of the series was fixed.
This whitespace problem was introduced in caa80697 and has gotten into
"pu" (in refs.c starting at line 1093).

* resolve_gitlink_packed_ref() was simplified a bit and got a docstring.

* A function do_for_each_ref_in_arrays() was extracted.

* The series were rebased.  The changes made to rebase them are roughly
equivalent to the work that you did in merges bea03b2455 and 773b817986.
 (BTW, your merges look correct to me.)

* It does *not* include your fix "refs: loosen over-strict "format" check".

SUMMARY:

The patches from mh/ref-api-take-2 through 633ebc45c0 plus your fix are
slightly preferable to the patches in mh/ref-api-2 and mh/ref-api-3, in
the ways listed in the bullet-points above.  If you are rewinding "next"
anyway, I suggest that you take the former instead of the latter.  If
you prefer to stick with the latter, let me know and I will submit the
remaining differences as patches to be applied on top of mh/ref-api-3.


I hope that this summary is helpful to you.  (It certainly helped me
figure out where things stand.)

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: Evaluation of ref-api branch status

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> Now that 1.7.8 is out, I wanted to figure out the status of the
> remaining ref-api changes that are in flight, including the differences
> between between my tree and yours.

I understand that the ultimate and largest objective of these series is to
make sure that the performance does not suffer from very many negative
lookups on the refs/replace hierarchy (which almost always is empty in a
sane repository), and I do want to see that happen. I also think it is
good that the series tries to make sure that the various codepaths we
create new refs do not let the user accidentally create refs with bad
names and/or contents.

When we see a questionable ref that is _already_ in the respository,
however, we may warn but it is wrong to declare the repository to be
broken and refuse to work. That would make it cumbersome or impossible to
even _fix_ such breakage. Regressions of that kind wer in the earlier part
of the series already in 1.7.8 and it was rather unpleasant having to
hotfix them. As our test suite does not deliberately create these
questionable refs and/or try to use them (and I personally do not in my
regular workflow either), I am afraid that it is rather hard to realize
what kind of refnames are what we intended to forbid even from earlier
days but did not _bother_ to check and enforce the rule against so far,
but are forbidden by the updated code, until we unleash the new logic to
the end users with various third party tools that created them [*1*]. I
would want to see us get this part right and solid, and include it in
1.7.9.

It would be nice if we can include also the bits to read hierarchies
lazily, but as they come on top of your B & C, it may end up in 1.7.10 and
I do not mind it at all. Reference resolution is one of the central things
in the user experience, and we cannot afford to ship anything that is
half-done-and-mostly-works over slow-but-correct.

Alternatively you _could_ swap the order of your B & C and D and try to
have your D early while giving B & C more time to cook, but I suspect the
order you chose would be better in the longer term.

> I understand that "next" will soon be re-rolled.  Will the re-roll be
> based on the current "pu", or will you start from scratch?

As to the two quickfix patches that are on two remaining topics from you
in my tree, I did them merely as a short-term band-aid. I was expecting,
after 1.7.8 when we eject the topics out of 'next', that they will be
rebased on top of 'master' that already contains a proper fix before these
topics started touching the same codepath.

My reading of your summary suggests that it would be easiest to drop the
three mh/ref-api* topics from my tree, especially the 'refs: loosen
over-strict "format" check' band-aid patches, and re-queue a re-roll from
you.

Thanks.

[Footnote]

*1* Trying to be lenient when reading and being strict when writing as the
general principle would be the safe and sane way forward, and making sure
that we warn *loudly* when we think we are merely being lenient (i.e. we
think we found a bad ref with questionable name and/or contents) would be
a good way to catch our mistakes early.  E.g. ".git/config does not record
a correct object name" glitch was caught only because you added the
warning so while it was painful to hotfix that late in the cycle, the
warning was worth it.

Re: Evaluation of ref-api branch status

[Michael Haggerty]

On 12/05/2011 07:26 PM, Junio C Hamano wrote:
> Michael Haggerty <mhagger@alum.mit.edu> writes:
>> Now that 1.7.8 is out, I wanted to figure out the status of the
>> remaining ref-api changes that are in flight, including the differences
>> between between my tree and yours.
> 
> I understand that the ultimate and largest objective of these series is to
> make sure that the performance does not suffer from very many negative
> lookups on the refs/replace hierarchy (which almost always is empty in a
> sane repository), and I do want to see that happen. I also think it is
> good that the series tries to make sure that the various codepaths we
> create new refs do not let the user accidentally create refs with bad
> names and/or contents.
> 
> When we see a questionable ref that is _already_ in the respository,
> however, we may warn but it is wrong to declare the repository to be
> broken and refuse to work. [...]  I
> would want to see us get this part right and solid, and include it in
> 1.7.9.
> [...]
> 
> Alternatively you _could_ swap the order of your B & C and D and try to
> have your D early while giving B & C more time to cook, but I suspect the
> order you chose would be better in the longer term.

I will evaluate whether this is feasible.

>> I understand that "next" will soon be re-rolled.  Will the re-roll be
>> based on the current "pu", or will you start from scratch?
> 
> As to the two quickfix patches that are on two remaining topics from you
> in my tree, I did them merely as a short-term band-aid. I was expecting,
> after 1.7.8 when we eject the topics out of 'next', that they will be
> rebased on top of 'master' that already contains a proper fix before these
> topics started touching the same codepath.
> 
> My reading of your summary suggests that it would be easiest to drop the
> three mh/ref-api* topics from my tree, especially the 'refs: loosen
> over-strict "format" check' band-aid patches, and re-queue a re-roll from
> you.

OK, then, I will try to re-roll the series on top of master, and build
the equivalent of your quick-fix into the logical point in the series.
I will also check whether it is feasible to push more of the refname
sanity checks to later in the series and make them more informational
where old data is concerned.  How much time to I have to work on this
while still leaving enough time to comfortably integrate it into 1.7.9?

Michael

-- 
Michael Haggerty
mhagger@alum.mit.edu
http://softwareswirl.blogspot.com/

Re: Evaluation of ref-api branch status

[Junio C Hamano]

Michael Haggerty <mhagger@alum.mit.edu> writes:

> On 12/05/2011 07:26 PM, Junio C Hamano wrote:
> ...
>> My reading of your summary suggests that it would be easiest to drop the
>> three mh/ref-api* topics from my tree, especially the 'refs: loosen
>> over-strict "format" check' band-aid patches, and re-queue a re-roll from
>> you.
>
> OK, then, I will try to re-roll the series on top of master, and build
> the equivalent of your quick-fix into the logical point in the series.

These quick-fixes were necessary _only_ because the queued topics predate
the real fix already in 1.7.8 (and I generally refuse to criss cross merge
from master to topics), so I expect you won't need their equivalents if
the topic is rebased on top of 1.7.8

> ... How much time to I have to work on this
> while still leaving enough time to comfortably integrate it into 1.7.9?

I am hoping we can have rc0 very early next year [*1*].


[Reference]

*1* https://www.google.com/calendar/embed?src=jfgbl2mrlipp4pb6ieih0qr3so%40group.calendar.google.com&ctz=America/Los_Angeles