Re: [PATCH] Make some commit hashes in tests reproducible

[Junio C Hamano <gitster@pobox.com>]

Jeff King <peff@peff.net> writes:

> That's using the same start point as test_tick, though really it could
> be anything. I've intentionally _not_ called test_tick at the beginning
> of each script, because that would throw off all of the scripts that do
> use it by one tick (whereas the first test_tick will overwrite these
> values).

Yup, I think that is a sensible approach.

> Trying to devil's advocate against this line of reasoning:
>
>   - using the current timestamp introduces more randomness into the test
>     suite, which could uncover problems. I'm somewhat skeptical, as the
>     usual outcome I see here is that we realize a test's expected output
>     is simply racy, and we remove the raciness by using test_tick

True.

>   - using the current timestamp could alert us to problems that occur
>     only as the clock ticks forward (e.g., if we had a Y2021 bug, we'd
>     notice when the clock rolled forward).

True.

>   - some tests may rely on having a "recent" timestamp in commits (e.g.,
>     when looking at relative date handling). I think all of the
>     relative-time tests already use a specific date, though, because
>     otherwise we have too many problems with raciness.

True.

Another thing we could do is under DEVELOPER=YesPlease we can set
timestamps you just added here to some random time.

The ones that do care about reproducibility guarantee by using
test_tick would not be affected, and those that were happy with the
current time should be happy with such a random timestamp.  Or we
could just drop what this patch does only under DEVELOPER=YesPlease
which amounts to be the same as setting random time.

> Note that the patch above does seem to cause two tests to fail. One of
> them I _suspect_ is a raciness problem (order of commits output changes,
> which implies the original was expecting the time to increment between
> two commits without running test_tick).

Ahh, ok, "git commit && git commit" without these environment
variable set and exported may or may not get the same timestamp from
the wallclock during the testing.  With your patch, they are
guaranteed to get the same timestamp.  On the surface, it sounds
like a valid approach to smoke out any tests that rely on the
passage of time, but realistically the machines are fast enough that
it won't be a rare occasion for these two "git commit" invocations
to be done within a single second, so I am not sure how much we
would be gaining.  We _are_ getting more reproducible test
environment, though, which may be a plus.