Re: [PATCH v2 9/9] gpg-interface t: extend the existing GPG tests with GPGSM

[Jeff King <peff@peff.net>]

On Wed, Jul 11, 2018 at 12:38:24PM +0200, Henning Schild wrote:

> > Can we save a dummy generated key and just import it? That's what we
> > do for the regular gpg case.
> 
> I will look into storing a binary and leaving notes how it was
> generated, just like regular gpg does. The reason i did not do that in
> the first place is that x509 certs have a validity and we introduce
> time into the picture. But i will see if i can generate epoch->infinity
> to get the host clock or just the future out of the picture.

That would be preferable. But even if we can just get something like a
10-year expiration, that may be enough. Somebody dealing with failing
tests and regenerating keys in ten years is probably not the end of the
world.

It could hurt people with drastically incorrect system clocks, but I
suspect there are other tests with similar problems (especially if your
clock is in the past).

> > We're going to have a lot of duplicated tests here. That's a
> > maintenance burden when one of them needs fixes later. And when new
> > tests are added, we won't automatically get them tested under each
> > format.
> > 
> > Can we move the battery of tests into a function that takes a few
> > parameters (prereq name, branch to look at, etc) and then call it for
> > both the gpg/gpgsm cases?
> 
> I guess this is part of the earlier "allow GPGSM without GPG" and i
> can ignore it if we agree that this is not needed?

I think it's orthogonal. Even if GPGSM requires GPG, you'd still want to
make sure that whatever exercise we give to the GPG code is also
exercised using GPGSM.

I will note, though, that _some_ tests are not really exercising
gpg-specific bits, but more how we react to it (e.g., how we format %G
placeholders). And it's probably OK for those to just be run once.

So in an ideal world, the test script would probably look something
like:

  # this function holds tests which exercise the interactions
  # with the gpg binary itself
  type_specific_tests() {
	prereq=$1
	branch=$2

	test_expect_success $prereq "test whatever ($prereq)" '
		some test using $branch here
	'
  }

  test_expect_success 'setup' '
	set up both openpgp and x509 branches here
  '

  type_specific_tests GPG openpgp
  type_specific_tests GPGSM x509

  # and now tests that generically care about getting _some_ signature
  # result (e.g., the way we format signature info)

  # and then probably a few tests specific to how the config is handled,
  # like your new gpg.format coverage

But in practice, the type-specific bits are often muddled together with
the type-independent ones (e.g., we happen to test the parsing of a
failed signature from gpg by checking how %G? is formatted or similar).

So it may be simplest to just run most of the tests twice, once with gpg
and once with gpgsm. I kind of wonder if all of t7510 could just be
bumped into a function. Or even into a sourced file and run from two
different scripts. See the way that t8001 and t8002 use
annotate-tests.sh for an example.

-Peff