Re: [PATCH v3] doc-diff: don't `cd_to_toplevel`

[Junio C Hamano <gitster@pobox.com>]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> In particular, the tests t2024 and t5552 are broken for
> ma/doc-diff-usage-fix on Windows. The reason seems to be that those are
> already broken for the base commit that Junio picked:
> jk/diff-rendered-docs (actually, not the tip of it, but the commit fixed
> by Martin's patch).
>
> Of course, I understand why Junio picks base commits that are far, far in
> the past (and have regressions that current `master` does not have), it
> makes sense from the point of view where the fixes should be as close to
> the commits they fix.  The downside is that we cannot automate regression
> testing more than we do now, with e.g. myself acting as curator of test
> failures.

I think by "regressions that current master does not have", you
actually meant "unrelated known breakages we have fixed since then".

If you use topic-branch workflow, it is unavoidable and expected, as
each topic has its focus and the topic to fix doc-diff are not about
making checkout test or fetch negotiator test to pass on all
platforms.

I am wondering if the automated testing can be made more useful by
limiting the scope of testing if it is run on individual topic.  For
four primary integration branches, we do want to ensure all tests
keep passing (or at least no tests that passed in the previous round
start failing), but it would match the topic-branch workflow better
if the automated testing allowed an individual topic to rely on
unrelated known breakages to be dealt with by other topics and newer
integration branches.

For a topic like doc-diff that is primarily meant for developers and
documenters, it does not matter much, but for an old but important
bug, forking the topic to fix it at a point close to the origin is
crucial---that is what would allow people to merge the fix to an
older maintenance track, without cherry-picking.  It is especially
true when the bug being fixed is more severe than unrelated
breakages that have been fixed since then.

Possible approaches I considered and rejected are:

 - Perhaps find updated tests in the topic and run only those?  We
   can complain a topic that is meant as a "fix" to something if it
   does not have test while trying to find such tests ;-)  But this
   does not work if a "fix" has unintended consequences and breaks
   existing tests that do not have apparent relation known to the
   author of the patch.

 - Perhaps find the fork point, run tests to find known breakages
   and exclude them?  This would simply be not practical, as it
   doubles the number of tests run, for individual topic branches
   because there are an order of magnitude more of them than the
   primary integration branches.

Possibly, if we limit the fork point to tagged releases, the latter
approach might turn out to be doable.  For ma/doc-diff-usage-fix,
the base commit ad51743007 was v2.20.0-rc0~232, so we could round up
and pick v2.20.0 as the base instead (the purpose of picking an older
point is to allow merging to older maintenance track, and it is
unlikely that people would keep using v2.20.0-rc0 as the base of
their forked distribution).  Then if the automated testing keeps
record of known breakages (whether they are later fixed or still
broken) for these tagged releases, testing a new topic forked from
one of them and deciding not to worry about breakages of tests that
are known to be broken (or perhaps deciding to skip these tests in
the first place) may become feasible.

I dunno.  Limiting fork points to tagged releases for topics that
are meant for the maintenance tracks may have other benefits by
restricting the shape of the resulting dag, so I am willing to try
it as an experiment, whether it would help your workflow or not.