git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
* reversion in GIT_COMMON_DIR refs path
@ 2017-05-16 17:10 Joey Hess
  2017-05-16 17:50 ` Ævar Arnfjörð Bjarmason
  2017-05-19 14:37 ` reversion in GIT_COMMON_DIR refs path Joey Hess
  0 siblings, 2 replies; 14+ messages in thread
From: Joey Hess @ 2017-05-16 17:10 UTC (permalink / raw)
  To: Git Mailing List; +Cc: Nguyễn Thái Ngọc Duy, Peter Simons

Bisecting this test suite failure
https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
are both set. While before refs were looked for in GIT_COMMON_DIR,
now they're not.

Test case:

#!/bin/sh
set -e
set -x
rm -rf testdir
git init testdir
cd testdir
echo 1 > foo
git add foo
git commit -m add
mkdir dummy
mkdir dummy/overlay
cp .git/index .git/HEAD dummy/overlay
#cp .git/refs .git/packed-refs dummy/overlay -a
cd dummy
export GIT_COMMON_DIR=`pwd`/../.git
export GIT_DIR=`pwd`/overlay
git rev-parse --git-path refs/heads/master
git show refs/heads/master

This script succeeds with git 2.11.0, but with 2.13.0, it fails:

fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.

It seems to be failing to look up refs in GIT_COMMON_DIR.
Note that uncommenting the commented out line in the script, to copy the refs
into GIT_DIR, makes it succeed.

git rev-parse --git-path refs/heads/master shows the GIT_COMMON_DIR/refs path
still (as gitrepository-layout documents). So this reversion made
different parts of git disagreeing about the refs path.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: reversion in GIT_COMMON_DIR refs path
  2017-05-16 17:10 reversion in GIT_COMMON_DIR refs path Joey Hess
@ 2017-05-16 17:50 ` Ævar Arnfjörð Bjarmason
  2017-05-16 17:59   ` Joey Hess
  2017-05-19 14:37 ` reversion in GIT_COMMON_DIR refs path Joey Hess
  1 sibling, 1 reply; 14+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2017-05-16 17:50 UTC (permalink / raw)
  To: Joey Hess
  Cc: Git Mailing List, Nguyễn Thái Ngọc Duy,
	Peter Simons

On Tue, May 16, 2017 at 7:10 PM, Joey Hess <id@joeyh.name> wrote:
> Bisecting this test suite failure
> https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
> I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.

That links's broken for me. Looking at your wiki it looks like you
mean: https://git-annex.branchable.com/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/

I have no idea what this bug is about, but side-question: It looks
like this is git-annex's own test suite that's failing with 2.13.0, is
that right?

It would be very nice to have a test in git itself to test with
git-annex. I.e. some optional test that just pulls down the latest
git-annex release & runs its tests against the git we're building.

Thanks for annex b.t.w., I use it a lot.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: reversion in GIT_COMMON_DIR refs path
  2017-05-16 17:50 ` Ævar Arnfjörð Bjarmason
@ 2017-05-16 17:59   ` Joey Hess
  2017-05-16 20:37     ` [PATCH] tests: add an optional test to test git-annex Ævar Arnfjörð Bjarmason
  0 siblings, 1 reply; 14+ messages in thread
From: Joey Hess @ 2017-05-16 17:59 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: Git Mailing List

[-- Attachment #1: Type: text/plain, Size: 1071 bytes --]

Ævar Arnfjörð Bjarmason wrote:
> On Tue, May 16, 2017 at 7:10 PM, Joey Hess <id@joeyh.name> wrote:
> > Bisecting this test suite failure
> > https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
> > I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.
> 
> That links's broken for me. Looking at your wiki it looks like you
> mean: https://git-annex.branchable.com/bugs/git-annex_in_nixpkgs_fails_with_git-2.13.0/

Thanks for correcting that

> I have no idea what this bug is about, but side-question: It looks
> like this is git-annex's own test suite that's failing with 2.13.0, is
> that right?

Yes indeed.

> It would be very nice to have a test in git itself to test with
> git-annex. I.e. some optional test that just pulls down the latest
> git-annex release & runs its tests against the git we're building.
> 
> Thanks for annex b.t.w., I use it a lot.

If the git devs are ok with this, I certianly would be happy if such
tests were run, at least occasionally, on the git side!

-- 
see shy jo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH] tests: add an optional test to test git-annex
  2017-05-16 17:59   ` Joey Hess
@ 2017-05-16 20:37     ` Ævar Arnfjörð Bjarmason
  2017-05-16 22:10       ` Joey Hess
  2017-05-17  2:45       ` Junio C Hamano
  0 siblings, 2 replies; 14+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2017-05-16 20:37 UTC (permalink / raw)
  To: git; +Cc: Junio C Hamano, Joey Hess, Ævar Arnfjörð Bjarmason

Add an optional test to test git-annex. It's guarded by a new
EXTERNAL_TESTS environment variable. Running this test takes me 10
minutes.

As reported by Joey Hess in "reversion in GIT_COMMON_DIR refs path"[1]
commit f57f37e2e1 ("files-backend: remove the use of git_path()",
2017-03-26) first released as part of the 2.13.0 broke git-annex's
test suite.

This could have been spotted by us before the release by optionally
running the git-annex test suite as part of git itself. This optional
test does that. It currently fails due to the reported regression,
but, passes on the 2.12.0 release.

The git-annex revision to test can be specified with the
GIT_TEST_GIT_ANNEX_REVISION environment variable. Joey has expressed
interest in testing development versions of git against git-annex[2],
and can now test the latest revision with:

    EXTERNAL_TESTS=1 GIT_TEST_GIT_ANNEX_REVISION='@{u}' ./t9950-git-annex.sh

By default the test finds the latest git-annex release tag and tests
that, since the primary purpose is to test regressions in git which
cause git-annex to fail, not regressions in git-annex itself.

The t9* test namespace is currently full as documented in t/README. In
lie of an empty t9X for "external tools" this change claims t995* for
that purpose.

1. <20170516171028.5eagqr2sw5a2i77d@kitenet.net>
   (https://public-inbox.org/git/20170516175906.hdwn4x5md7dj7fo3@kitenet.net/T/)
2. http://git-annex.branchable.com/devblog/day_459__git_bug/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---

On Tue, May 16, 2017 at 7:59 PM, Joey Hess <id@joeyh.name> wrote:
> Ævar Arnfjörð Bjarmason wrote:
>> On Tue, May 16, 2017 at 7:10 PM, Joey Hess <id@joeyh.name> wrote:
>> I have no idea what this bug is about, but side-question: It looks
>> like this is git-annex's own test suite that's failing with 2.13.0, is
>> that right?
>
> Yes indeed.
>
>> It would be very nice to have a test in git itself to test with
>> git-annex. I.e. some optional test that just pulls down the latest
>> git-annex release & runs its tests against the git we're building.
>>
>> Thanks for annex b.t.w., I use it a lot.
>
> If the git devs are ok with this, I certianly would be happy if such
> tests were run, at least occasionally, on the git side!

I for one would run this test occasionally, and perhaps we could even
run it as part of Travis eventually (although there would be a *lot*
of Haskell deps, on my box "apt build-dep git-annex" brought in 1/2 GB
of packages).

As noted in the commit message, once this is part of git.git you can
easily set an environment variable to test the bleeding edge of git
against any arbitrary git-annex version.

 t/t9950-git-annex.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 52 insertions(+)
 create mode 100755 t/t9950-git-annex.sh

diff --git a/t/t9950-git-annex.sh b/t/t9950-git-annex.sh
new file mode 100755
index 0000000000..2cbc1f4be3
--- /dev/null
+++ b/t/t9950-git-annex.sh
@@ -0,0 +1,52 @@
+#!/bin/sh
+
+test_description='the git-annex test suite'
+. ./test-lib.sh
+
+if test -z "$EXTERNAL_TESTS"
+then
+	skip_all='skipping tests of external tools. EXTERNAL_TESTS not defined'
+	test_done
+fi
+
+if test -n "$NO_CURL"
+then
+	skip_all='skipping test, git built without http support'
+	test_done
+fi
+
+test_expect_success 'clone git-annex' '
+	git clone https://git.joeyh.name/git/git-annex.git
+'
+
+if test -n "$GIT_TEST_GIT_ANNEX_REVISION"
+then
+	test_expect_success "plan to test git-annex $GIT_TEST_GIT_ANNEX_REVISION" "
+		echo '$GIT_TEST_GIT_ANNEX_REVISION' >revision-to-test
+	"
+else
+	test_expect_success "plan to test git-annex's latest release tag" '
+		git -C git-annex tag --sort=version:refname -l "[0-9]*.[0-9]*" |
+			tail -n 1 >revision-to-test
+	'
+fi
+
+test_expect_success 'checkout $(cat revision-to-test) for testing' '
+	git -C git-annex checkout $(cat revision-to-test)
+'
+
+test_expect_success 'build git-annex (if this fails, you are likely missing its Haskell dependencies' '
+	(
+		cd git-annex &&
+		make
+	)
+'
+
+test_expect_success 'test git-annex' '
+	(
+		cd git-annex &&
+		make test
+	)
+'
+
+test_done
-- 
2.13.0.303.g4ebf302169


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-16 20:37     ` [PATCH] tests: add an optional test to test git-annex Ævar Arnfjörð Bjarmason
@ 2017-05-16 22:10       ` Joey Hess
  2017-05-17  2:45       ` Junio C Hamano
  1 sibling, 0 replies; 14+ messages in thread
From: Joey Hess @ 2017-05-16 22:10 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: git

[-- Attachment #1: Type: text/plain, Size: 391 bytes --]

Nice work.

Note that you can export BUILDER=stack and git-annex will build with a
known good dependency stack, which can be more reliable/cross platform
than using apt to install its build dependencies. That needs
https://docs.haskellstack.org/ installed. Also it currently needs
GIT_TEST_GIT_ANNEX_REVISION=master since I improved git-annex's
Makefile slightly.

-- 
see shy jo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-16 20:37     ` [PATCH] tests: add an optional test to test git-annex Ævar Arnfjörð Bjarmason
  2017-05-16 22:10       ` Joey Hess
@ 2017-05-17  2:45       ` Junio C Hamano
  2017-05-17  6:47         ` Ævar Arnfjörð Bjarmason
  1 sibling, 1 reply; 14+ messages in thread
From: Junio C Hamano @ 2017-05-17  2:45 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: git, Joey Hess

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Add an optional test to test git-annex. It's guarded by a new
> EXTERNAL_TESTS environment variable. Running this test takes me 10
> minutes.

Well, it is one thing to place git-annex under CI to make sure its
latest and greatest works together well with our latest and greatest
(and it may be something we want to see happen), but driving its
tests from our testsuite sounds like a tail wagging the dog, at
least to me.

I do not mind at all to place the simple reproduction recipe Joey
posted as a new test in our test suite, though.  That kind of test
that catches changes to externally visible behaviour surely belongs
to our test suite.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17  2:45       ` Junio C Hamano
@ 2017-05-17  6:47         ` Ævar Arnfjörð Bjarmason
  2017-05-17 10:19           ` Junio C Hamano
  0 siblings, 1 reply; 14+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2017-05-17  6:47 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Git Mailing List, Joey Hess

On Wed, May 17, 2017 at 4:45 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> Add an optional test to test git-annex. It's guarded by a new
>> EXTERNAL_TESTS environment variable. Running this test takes me 10
>> minutes.

[Re-arranged your mail because it worked better with my reply]

> I do not mind at all to place the simple reproduction recipe Joey
> posted as a new test in our test suite, though.  That kind of test
> that catches changes to externally visible behaviour surely belongs
> to our test suite.

This is not a replacement for having an isolated test for the issue
Joey noted. We should have a separate patch for that, but I did not
have time/interest in writing that up. This change is orthagonal to
that.

> Well, it is one thing to place git-annex under CI to make sure its
> latest and greatest works together well with our latest and greatest
> (and it may be something we want to see happen), but driving its
> tests from our testsuite sounds like a tail wagging the dog, at
> least to me.

To me this is just a question of:

* Is it the case that git-annex tests for a lot of edge cases we don't
test for: Yes, probably. As evidenced by them spotting this
regression, and not us.

* We can (and should) add a test for the specific breakage we caused
in 2.13.0, but that's no replacement for other things annex may be
covering & we may be missing which'll catch future breakages.

* It's a pretty established practice to test a library (git) along
with its consumers (e.g. annex) before a major release.

* This allows us to do that at minimal cost. I think it makes sense to
add this and integration tests for other similar utilities if they're
similarly easy to integrate.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17  6:47         ` Ævar Arnfjörð Bjarmason
@ 2017-05-17 10:19           ` Junio C Hamano
  2017-05-17 19:33             ` Ævar Arnfjörð Bjarmason
  2017-05-17 23:56             ` Brandon Williams
  0 siblings, 2 replies; 14+ messages in thread
From: Junio C Hamano @ 2017-05-17 10:19 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: Git Mailing List, Joey Hess

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>> Well, it is one thing to place git-annex under CI to make sure its
>> latest and greatest works together well with our latest and greatest
>> (and it may be something we want to see happen), but driving its
>> tests from our testsuite sounds like a tail wagging the dog, at
>> least to me.
>
> To me this is just a question of:
>
> * Is it the case that git-annex tests for a lot of edge cases we don't
> test for: Yes, probably. As evidenced by them spotting this
> regression, and not us.

And I'd encourage them to keep doing so.

> * We can (and should) add a test for the specific breakage we caused
> in 2.13.0, but that's no replacement for other things annex may be
> covering & we may be missing which'll catch future breakages.
>
> * It's a pretty established practice to test a library (git) along
> with its consumers (e.g. annex) before a major release.

I am not so sure about the division of labor.  What you are
advocating would work _ONLY_ if we test with a perfect & bug-free
version of the consumers.  If they are also a moving target, then
I do not think it is worth it.  After all, we are *not* in the
business of testing these consumers.

Unless I misunderstood you and you were saying that we freeze a
version, or a set of versions, of customer that is/are known to pass
their own tests, and test the combination of that frozen version of
the customer with our daily development.  If that is the case, then
I would agree that we are using their test to test us, not them.
But I somehow didn't get that impression, hence my reaction.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17 10:19           ` Junio C Hamano
@ 2017-05-17 19:33             ` Ævar Arnfjörð Bjarmason
  2017-05-17 23:03               ` Stefan Beller
  2017-05-17 23:56             ` Brandon Williams
  1 sibling, 1 reply; 14+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2017-05-17 19:33 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Git Mailing List, Joey Hess, Stefan Beller

On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>>> Well, it is one thing to place git-annex under CI to make sure its
>>> latest and greatest works together well with our latest and greatest
>>> (and it may be something we want to see happen), but driving its
>>> tests from our testsuite sounds like a tail wagging the dog, at
>>> least to me.
>>
>> To me this is just a question of:
>>
>> * Is it the case that git-annex tests for a lot of edge cases we don't
>> test for: Yes, probably. As evidenced by them spotting this
>> regression, and not us.
>
> And I'd encourage them to keep doing so.

The point of this patch is that we can do this more systematically and
reliably, not have people discover this sort of thing after a major
release.

I.e. we can be pro-active about this instead of waiting for bug
reports to roll in.

>> * We can (and should) add a test for the specific breakage we caused
>> in 2.13.0, but that's no replacement for other things annex may be
>> covering & we may be missing which'll catch future breakages.
>>
>> * It's a pretty established practice to test a library (git) along
>> with its consumers (e.g. annex) before a major release.
>
> I am not so sure about the division of labor.  What you are
> advocating would work _ONLY_ if we test with a perfect & bug-free
> version of the consumers.  If they are also a moving target, then
> I do not think it is worth it.  After all, we are *not* in the
> business of testing these consumers.
>
> Unless I misunderstood you and you were saying that we freeze a
> version, or a set of versions, of customer that is/are known to pass
> their own tests, and test the combination of that frozen version of
> the customer with our daily development.  If that is the case, then
> I would agree that we are using their test to test us, not them.
> But I somehow didn't get that impression, hence my reaction.

The test I'm adding tests the release version of git-annex, so I think
in practice we don't have to worry about random changes of theirs
producing false positives for us.

The utility of this test is that sometime close to release someone
(e.g. me) can run it, if it fails let's see if it fails on the last
release version of ours, if so it's probably upstream breakage, or
like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.

It'll never trip some random tester up since you need to explicitly
opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
objections. This incurs no burden on either devs, packagers or users,
and would have demonstrably detected an issue we'd rather have wanted
to know about pre-release than post-release as is the case now.

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17 19:33             ` Ævar Arnfjörð Bjarmason
@ 2017-05-17 23:03               ` Stefan Beller
  2017-05-17 23:59                 ` Brandon Williams
  0 siblings, 1 reply; 14+ messages in thread
From: Stefan Beller @ 2017-05-17 23:03 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason
  Cc: Junio C Hamano, Git Mailing List, Joey Hess

On Wed, May 17, 2017 at 12:33 PM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>>
>>>> Well, it is one thing to place git-annex under CI to make sure its
>>>> latest and greatest works together well with our latest and greatest
>>>> (and it may be something we want to see happen), but driving its
>>>> tests from our testsuite sounds like a tail wagging the dog, at
>>>> least to me.
>>>
>>> To me this is just a question of:
>>>
>>> * Is it the case that git-annex tests for a lot of edge cases we don't
>>> test for: Yes, probably. As evidenced by them spotting this
>>> regression, and not us.
>>
>> And I'd encourage them to keep doing so.
>
> The point of this patch is that we can do this more systematically and
> reliably, not have people discover this sort of thing after a major
> release.
>
> I.e. we can be pro-active about this instead of waiting for bug
> reports to roll in.

We can be pro-active without this patch, too. ;)
I guess this makes it just easier for someone out of the Git community to
be proactive in searching for defects.

>
> The utility of this test is that sometime close to release someone
> (e.g. me) can run it, if it fails let's see if it fails on the last
> release version of ours, if so it's probably upstream breakage, or
> like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.
>
> It'll never trip some random tester up since you need to explicitly
> opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
> objections. This incurs no burden on either devs, packagers or users,
> and would have demonstrably detected an issue we'd rather have wanted
> to know about pre-release than post-release as is the case now.

I think this patch could spark a discussion what we expect from our test suite.
Is it a contract that we promise, as in "Here are some blessed workflows,
our users should see and imitate"?

This patch sort of feels like (a) we let others dictate the contract to us
and (b) we choose the easy way out as we do not write enough tests on
our own, so we force downstream to help us out.

I guess it is appropriate to compare this level of testing to other external
tests, such as cppcheck, travis, coverity. For some of them (travis) we
carry some code with us (.travis), not so for others (coverity would benefit
from [1] that I carry outside of Junios tree). Admittedly these external tools
are all focused on testing, not on building on top of Git, which brings
in other expectations from these tools.

[1] https://github.com/stefanbeller/git/commit/0781fbafb9dd2a995ba62a9af5f7581e3cf05359

---
As mentioned in the other thread[2], the idea was floated to have this test or
the downstream project as a submodule, and if you are interested in the test,
then you would obtain the submodule and run the tests (in there?)
That would have the benefit of less 'uninteresting' glue logic for those who
are not interested in these tests as gitlinks may be easier to ignore
for a developer
than code. (Less data to fetch, git-grep also doesn't yield results
for the uninteresting
parts as these submodules would be uninitialized).
(The more I talk about this "downstream tests in submodules"
idea, the less convinced I am)

[2] https://public-inbox.org/git/20170517113824.31700-1-avarab@gmail.com/

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17 10:19           ` Junio C Hamano
  2017-05-17 19:33             ` Ævar Arnfjörð Bjarmason
@ 2017-05-17 23:56             ` Brandon Williams
  1 sibling, 0 replies; 14+ messages in thread
From: Brandon Williams @ 2017-05-17 23:56 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: Ævar Arnfjörð Bjarmason, Git Mailing List,
	Joey Hess

On 05/17, Junio C Hamano wrote:
> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> 
> >> Well, it is one thing to place git-annex under CI to make sure its
> >> latest and greatest works together well with our latest and greatest
> >> (and it may be something we want to see happen), but driving its
> >> tests from our testsuite sounds like a tail wagging the dog, at
> >> least to me.
> >
> > To me this is just a question of:
> >
> > * Is it the case that git-annex tests for a lot of edge cases we don't
> > test for: Yes, probably. As evidenced by them spotting this
> > regression, and not us.
> 
> And I'd encourage them to keep doing so.
> 
> > * We can (and should) add a test for the specific breakage we caused
> > in 2.13.0, but that's no replacement for other things annex may be
> > covering & we may be missing which'll catch future breakages.
> >
> > * It's a pretty established practice to test a library (git) along
> > with its consumers (e.g. annex) before a major release.
> 
> I am not so sure about the division of labor.  What you are
> advocating would work _ONLY_ if we test with a perfect & bug-free
> version of the consumers.  If they are also a moving target, then
> I do not think it is worth it.  After all, we are *not* in the
> business of testing these consumers.

I agree with this. It makes no sense to test consumers of git, its
downstream's job to do that.  Though I do think that its perfectly
reasonable to test that our API works as advertised such that consumer's
can rely on git.

> 
> Unless I misunderstood you and you were saying that we freeze a
> version, or a set of versions, of customer that is/are known to pass
> their own tests, and test the combination of that frozen version of
> the customer with our daily development.  If that is the case, then
> I would agree that we are using their test to test us, not them.
> But I somehow didn't get that impression, hence my reaction.

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH] tests: add an optional test to test git-annex
  2017-05-17 23:03               ` Stefan Beller
@ 2017-05-17 23:59                 ` Brandon Williams
  0 siblings, 0 replies; 14+ messages in thread
From: Brandon Williams @ 2017-05-17 23:59 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Ævar Arnfjörð Bjarmason, Junio C Hamano,
	Git Mailing List, Joey Hess

On 05/17, Stefan Beller wrote:
> On Wed, May 17, 2017 at 12:33 PM, Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
> > On Wed, May 17, 2017 at 12:19 PM, Junio C Hamano <gitster@pobox.com> wrote:
> >> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
> >>
> >>>> Well, it is one thing to place git-annex under CI to make sure its
> >>>> latest and greatest works together well with our latest and greatest
> >>>> (and it may be something we want to see happen), but driving its
> >>>> tests from our testsuite sounds like a tail wagging the dog, at
> >>>> least to me.
> >>>
> >>> To me this is just a question of:
> >>>
> >>> * Is it the case that git-annex tests for a lot of edge cases we don't
> >>> test for: Yes, probably. As evidenced by them spotting this
> >>> regression, and not us.
> >>
> >> And I'd encourage them to keep doing so.
> >
> > The point of this patch is that we can do this more systematically and
> > reliably, not have people discover this sort of thing after a major
> > release.
> >
> > I.e. we can be pro-active about this instead of waiting for bug
> > reports to roll in.
> 
> We can be pro-active without this patch, too. ;)
> I guess this makes it just easier for someone out of the Git community to
> be proactive in searching for defects.
> 
> >
> > The utility of this test is that sometime close to release someone
> > (e.g. me) can run it, if it fails let's see if it fails on the last
> > release version of ours, if so it's probably upstream breakage, or
> > like with the 2.13.0 release if it's OK on 2.12.0 it's our bug.
> >
> > It'll never trip some random tester up since you need to explicitly
> > opt-in via EXTERNAL_TESTS=1, so honestly I'm a bit puzzled by these
> > objections. This incurs no burden on either devs, packagers or users,
> > and would have demonstrably detected an issue we'd rather have wanted
> > to know about pre-release than post-release as is the case now.
> 
> I think this patch could spark a discussion what we expect from our test suite.
> Is it a contract that we promise, as in "Here are some blessed workflows,
> our users should see and imitate"?
> 
> This patch sort of feels like (a) we let others dictate the contract to us
> and (b) we choose the easy way out as we do not write enough tests on
> our own, so we force downstream to help us out.
> 
> I guess it is appropriate to compare this level of testing to other external
> tests, such as cppcheck, travis, coverity. For some of them (travis) we
> carry some code with us (.travis), not so for others (coverity would benefit
> from [1] that I carry outside of Junios tree). Admittedly these external tools
> are all focused on testing, not on building on top of Git, which brings
> in other expectations from these tools.
> 
> [1] https://github.com/stefanbeller/git/commit/0781fbafb9dd2a995ba62a9af5f7581e3cf05359
> 
> ---
> As mentioned in the other thread[2], the idea was floated to have this test or
> the downstream project as a submodule, and if you are interested in the test,
> then you would obtain the submodule and run the tests (in there?)

Despite working on improving submodules, I'm still very skeptical of
adding submodules to git.git, at least at this point in time.  I just
feel like the submodule experience still has a lot of work before we
begin using it...Though My feeling may be completely unfounded.

> That would have the benefit of less 'uninteresting' glue logic for those who
> are not interested in these tests as gitlinks may be easier to ignore
> for a developer
> than code. (Less data to fetch, git-grep also doesn't yield results
> for the uninteresting
> parts as these submodules would be uninitialized).
> (The more I talk about this "downstream tests in submodules"
> idea, the less convinced I am)
> 
> [2] https://public-inbox.org/git/20170517113824.31700-1-avarab@gmail.com/

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: reversion in GIT_COMMON_DIR refs path
  2017-05-16 17:10 reversion in GIT_COMMON_DIR refs path Joey Hess
  2017-05-16 17:50 ` Ævar Arnfjörð Bjarmason
@ 2017-05-19 14:37 ` Joey Hess
  2017-05-22 11:11   ` Duy Nguyen
  1 sibling, 1 reply; 14+ messages in thread
From: Joey Hess @ 2017-05-19 14:37 UTC (permalink / raw)
  To: Git Mailing List; +Cc: Nguyễn Thái Ngọc Duy, Peter Simons

[-- Attachment #1: Type: text/plain, Size: 908 bytes --]

Joey Hess wrote:
> Bisecting this test suite failure
> https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
> I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.
> 
> It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
> are both set. While before refs were looked for in GIT_COMMON_DIR,
> now they're not.

In case there's any doubt about whether this is a reversion or an
intentional change, see gitrepository-layout(5):

       refs
           References are stored in subdirectories of this directory. The git
           prune command knows to preserve objects reachable from refs found
           in this directory and its subdirectories. This directory is ignored
           if $GIT_COMMON_DIR is set and "$GIT_COMMON_DIR/refs" will be used
           instead.

So the documented behavior is broken.

-- 
see shy jo

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: reversion in GIT_COMMON_DIR refs path
  2017-05-19 14:37 ` reversion in GIT_COMMON_DIR refs path Joey Hess
@ 2017-05-22 11:11   ` Duy Nguyen
  0 siblings, 0 replies; 14+ messages in thread
From: Duy Nguyen @ 2017-05-22 11:11 UTC (permalink / raw)
  To: Joey Hess; +Cc: Git Mailing List, Peter Simons

On Fri, May 19, 2017 at 9:37 PM, Joey Hess <id@joeyh.name> wrote:
> Joey Hess wrote:
>> Bisecting this test suite failure
>> https://git-annex.branchable.com/git-annex_in_nixpkgs_fails_with_git-2.13.0/
>> I landed on commit f57f37e2e1bf11ab4cdfd221ad47e961ba9353a0 to git.
>>
>> It seems that changed resolving refs paths when GIT_DIR and GIT_COMMON_DIR
>> are both set. While before refs were looked for in GIT_COMMON_DIR,
>> now they're not.
>
> In case there's any doubt about whether this is a reversion or an
> intentional change, see gitrepository-layout(5):
>
>        refs
>            References are stored in subdirectories of this directory. The git
>            prune command knows to preserve objects reachable from refs found
>            in this directory and its subdirectories. This directory is ignored
>            if $GIT_COMMON_DIR is set and "$GIT_COMMON_DIR/refs" will be used
>            instead.
>
> So the documented behavior is broken.

It's a gray area. When I wrote that I think I forgot about
per-worktree refs (refs/bisect/*) so "This directory is ignored" is
not completely true. The final line (probably won't help you much) is
"per-repo refs must be read from $GIT_COMMON_DIR/refs, per-worktree
from $GIT_DIR". The fact that we looked per-repo (like master) in
$GIT_DIR is probably an unwanted side effect.
-- 
Duy

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2017-05-22 11:12 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-16 17:10 reversion in GIT_COMMON_DIR refs path Joey Hess
2017-05-16 17:50 ` Ævar Arnfjörð Bjarmason
2017-05-16 17:59   ` Joey Hess
2017-05-16 20:37     ` [PATCH] tests: add an optional test to test git-annex Ævar Arnfjörð Bjarmason
2017-05-16 22:10       ` Joey Hess
2017-05-17  2:45       ` Junio C Hamano
2017-05-17  6:47         ` Ævar Arnfjörð Bjarmason
2017-05-17 10:19           ` Junio C Hamano
2017-05-17 19:33             ` Ævar Arnfjörð Bjarmason
2017-05-17 23:03               ` Stefan Beller
2017-05-17 23:59                 ` Brandon Williams
2017-05-17 23:56             ` Brandon Williams
2017-05-19 14:37 ` reversion in GIT_COMMON_DIR refs path Joey Hess
2017-05-22 11:11   ` Duy Nguyen

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).