Bug: duplicate sections in .git/config after remote removal

Bug: duplicate sections in .git/config after remote removal

[Jason Frey]

While the impact of this bug is minimal, and git itself is not
affected, it can affect external tools that want to read the
.git/config file, expecting unique section names.

To reproduce:

Given the following example .git/config file (I am leaving out the
[core] section for brevity):

    [remote "origin"]
        url = git@github.com:Fryguy/example.git
        fetch = +refs/heads/*:refs/remotes/origin/*
    [branch "master"]
        remote = origin
        merge = refs/heads/master

Running `git remote rm origin` will result in the following contents:

    [branch "master"]

Running `git remote add origin git@github.com:Fryguy/example.git` will
result in the following contents:

    [branch "master"]
    [remote "origin"]
        url = git@github.com:Fryguy/example.git
        fetch = +refs/heads/*:refs/remotes/origin/*

And finally, running `git fetch origin; git branch -u origin/master`
will result in the following contents:

    [branch "master"]
    [remote "origin"]
        url = git@github.com:Fryguy/example.git
        fetch = +refs/heads/*:refs/remotes/origin/*
    [branch "master"]
        remote = origin
        merge = refs/heads/master

at which point you can see the duplicate sections (even though one is
empty).  Also note that if you do the steps again, you will be left
with 3 sections, 2 of which are empty.  This process can be repeated
over and over.

Thanks,
Jason

Re: Bug: duplicate sections in .git/config after remote removal

[Stefan Beller]

On Tue, Mar 27, 2018 at 1:41 PM Jason Frey <jfrey@redhat.com> wrote:

> at which point you can see the duplicate sections (even though one is
> empty).  Also note that if you do the steps again, you will be left
> with 3 sections, 2 of which are empty.  This process can be repeated
> over and over.

I agree that this is an issue for the user, and there were some attempts
to fix it in the past. (feel free to dig them up in the archive at
https://public-inbox.org/git)

IIRC the problem is (a) with the loose file format (What if the user put
a valuable comment just after or before the '[branch "master"]' line?)
as well as (b) the way the parser/writer works (single pass, line by line)

(b) specifically made it a "huge effort, but little return" bug,
so nobody got around for a proper fix.

Thanks,
Stefan

Re: Bug: duplicate sections in .git/config after remote removal

[Ævar Arnfjörð Bjarmason]

On Tue, Mar 27 2018, Jason Frey wrote:

> While the impact of this bug is minimal, and git itself is not
> affected, it can affect external tools that want to read the
> .git/config file, expecting unique section names.
>
> To reproduce:
>
> Given the following example .git/config file (I am leaving out the
> [core] section for brevity):
>
>     [remote "origin"]
>         url = git@github.com:Fryguy/example.git
>         fetch = +refs/heads/*:refs/remotes/origin/*
>     [branch "master"]
>         remote = origin
>         merge = refs/heads/master
>
> Running `git remote rm origin` will result in the following contents:
>
>     [branch "master"]
>
> Running `git remote add origin git@github.com:Fryguy/example.git` will
> result in the following contents:
>
>     [branch "master"]
>     [remote "origin"]
>         url = git@github.com:Fryguy/example.git
>         fetch = +refs/heads/*:refs/remotes/origin/*
>
> And finally, running `git fetch origin; git branch -u origin/master`
> will result in the following contents:
>
>     [branch "master"]
>     [remote "origin"]
>         url = git@github.com:Fryguy/example.git
>         fetch = +refs/heads/*:refs/remotes/origin/*
>     [branch "master"]
>         remote = origin
>         merge = refs/heads/master
>
> at which point you can see the duplicate sections (even though one is
> empty).  Also note that if you do the steps again, you will be left
> with 3 sections, 2 of which are empty.  This process can be repeated
> over and over.

This can be annoying and result in some very verbose config files when
we automatically edit them, e.g.:

    (rm -v /tmp/test.ini; for i in {1..3}; do git config -f /tmp/test.ini foo.bar 0 && git config -f /tmp/test.ini --unset foo.bar; done; cat /tmp/test.ini)
    removed '/tmp/test.ini'
    [foo]
    [foo]
    [foo]

But it's not so clear that it should be called a bug, yes we could be a
bit smarter and not add obvious crap like the example above (duplicate
sections at the end), but it gets less obvious in more complex cases,
see my c8b2cec09e ("branch: add test for -m renaming multiple config
sections", 2017-06-18) for one such example.

Git has a config format that's hybrid human/machine editable. Consider a
case like:

    [gc]
    ;; Here's all the gc config we set up to avoid the great outage of 2015
    autoDetach = false
    ;; Our aliases
    [alias]
    st = status

Now, if I run `git config gc.auto 0` is it better if we end up with:

    [gc]
    ;; Here's all the gc config we set up to avoid the great outage of 2015
    autoDetach = false
    auto = 0
    ;; Our aliases
    [alias]
    st = status

Or something that makes it more clear that a machine added something at
the end:

    [gc]
    ;; Here's all the gc config we set up to avoid the great outage of 2015
    autoDetach = false
    ;; Our aliases
    [alias]
    st = status
    [gc]
    auto = 0

Most importantly though, regardless of what we decide to do when we
machine-edit the file, it's also human-editable, and being able to
repeat sections is part of our config format that you're simply going to
have to deal with.

The external tool (presumably some generic *.ini parser) you're trying
to point at git's config is broken for that purpose if it doesn't handle
duplicate sections. You're probably better off trying to parse `git
config --list --null` than trying to make it work.

I don't think we'd ever want to get rid of this feature, it's *very*
useful. Both for config via the include macro, and for people to
manually paste some config they want to try out to the end of their
config, without having to manually edit it to incorporate it into their
already existing sections.

Re: Bug: duplicate sections in .git/config after remote removal

[Philip Oakley]

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
> On Tue, Mar 27 2018, Jason Frey wrote:
>
>> While the impact of this bug is minimal, and git itself is not
>> affected, it can affect external tools that want to read the
>> .git/config file, expecting unique section names.
>>
>> To reproduce:
>>
>> Given the following example .git/config file (I am leaving out the
>> [core] section for brevity):
>>
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> Running `git remote rm origin` will result in the following contents:
>>
>>     [branch "master"]
>>
>> Running `git remote add origin git@github.com:Fryguy/example.git` will
>> result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>
>> And finally, running `git fetch origin; git branch -u origin/master`
>> will result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> at which point you can see the duplicate sections (even though one is
>> empty).  Also note that if you do the steps again, you will be left
>> with 3 sections, 2 of which are empty.  This process can be repeated
>> over and over.
>
> This can be annoying and result in some very verbose config files when
> we automatically edit them, e.g.:
>
>    (rm -v /tmp/test.ini; for i in {1..3}; do git config -f /tmp/test.ini 
> foo.bar 0 && git config -f /tmp/test.ini --unset foo.bar; done; cat 
> /tmp/test.ini)
>    removed '/tmp/test.ini'
>    [foo]
>    [foo]
>    [foo]
>
> But it's not so clear that it should be called a bug, yes we could be a
> bit smarter and not add obvious crap like the example above (duplicate
> sections at the end), but it gets less obvious in more complex cases,
> see my c8b2cec09e ("branch: add test for -m renaming multiple config
> sections", 2017-06-18) for one such example.
>
> Git has a config format that's hybrid human/machine editable. Consider a
> case like:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>
> Now, if I run `git config gc.auto 0` is it better if we end up with:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    auto = 0
>    ;; Our aliases
>    [alias]
>    st = status
>
> Or something that makes it more clear that a machine added something at
> the end:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>    [gc]
>    auto = 0
>
> Most importantly though, regardless of what we decide to do when we
> machine-edit the file, it's also human-editable, and being able to
> repeat sections is part of our config format that you're simply going to
> have to deal with.

One option may be to create  a simple 'lint' style checker that simply 
hiughlights and suggests options so the user can decide for themselves what 
they need to do. This would help span the gap between hard format and the 
soft format capabiulities of machine readable ini files, the Git config 
reader and being human readable.

Thus duplicate sections would be noted, likewise the presence of comments 
immediately preceding a section header, or terminating a section (with or 
without spacing?), etc.Such a config_lint could reside in the contrib as a 
supprt tool, and may in the long term be a guide to a common format. 
However, as noted, it would be more of a long term aspiration..


>
> The external tool (presumably some generic *.ini parser) you're trying
> to point at git's config is broken for that purpose if it doesn't handle
> duplicate sections. You're probably better off trying to parse `git
> config --list --null` than trying to make it work.
>
> I don't think we'd ever want to get rid of this feature, it's *very*
> useful. Both for config via the include macro, and for people to
> manually paste some config they want to try out to the end of their
> config, without having to manually edit it to incorporate it into their
> already existing sections.
>

--
Philip 

Re: Bug: duplicate sections in .git/config after remote removal

[Philip Oakley]

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
> On Tue, Mar 27 2018, Jason Frey wrote:
>
>> While the impact of this bug is minimal, and git itself is not
>> affected, it can affect external tools that want to read the
>> .git/config file, expecting unique section names.
>>
>> To reproduce:
>>
>> Given the following example .git/config file (I am leaving out the
>> [core] section for brevity):
>>
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> Running `git remote rm origin` will result in the following contents:
>>
>>     [branch "master"]
>>
>> Running `git remote add origin git@github.com:Fryguy/example.git` will
>> result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>
>> And finally, running `git fetch origin; git branch -u origin/master`
>> will result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> at which point you can see the duplicate sections (even though one is
>> empty).  Also note that if you do the steps again, you will be left
>> with 3 sections, 2 of which are empty.  This process can be repeated
>> over and over.
>
> This can be annoying and result in some very verbose config files when
> we automatically edit them, e.g.:
>
>    (rm -v /tmp/test.ini; for i in {1..3}; do git config -f /tmp/test.ini 
> foo.bar 0 && git config -f /tmp/test.ini --unset foo.bar; done; cat 
> /tmp/test.ini)
>    removed '/tmp/test.ini'
>    [foo]
>    [foo]
>    [foo]
>
> But it's not so clear that it should be called a bug, yes we could be a
> bit smarter and not add obvious crap like the example above (duplicate
> sections at the end), but it gets less obvious in more complex cases,
> see my c8b2cec09e ("branch: add test for -m renaming multiple config
> sections", 2017-06-18) for one such example.
>
> Git has a config format that's hybrid human/machine editable. Consider a
> case like:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>
> Now, if I run `git config gc.auto 0` is it better if we end up with:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    auto = 0
>    ;; Our aliases
>    [alias]
>    st = status
>
> Or something that makes it more clear that a machine added something at
> the end:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>    [gc]
>    auto = 0
>
> Most importantly though, regardless of what we decide to do when we
> machine-edit the file, it's also human-editable, and being able to
> repeat sections is part of our config format that you're simply going to
> have to deal with.

One option may be to create  a simple 'lint' style checker that simply 
hiughlights and suggests options so the user can decide for themselves what 
they need to do. This would help span the gap between hard format and the 
soft format capabiulities of machine readable ini files, the Git config 
reader and being human readable.

Thus duplicate sections would be noted, likewise the presence of comments 
immediately preceding a section header, or terminating a section (with or 
without spacing?), etc.Such a config_lint could reside in the contrib as a 
supprt tool, and may in the long term be a guide to a common format. 
However, as noted, it would be more of a long term aspiration..


>
> The external tool (presumably some generic *.ini parser) you're trying
> to point at git's config is broken for that purpose if it doesn't handle
> duplicate sections. You're probably better off trying to parse `git
> config --list --null` than trying to make it work.
>
> I don't think we'd ever want to get rid of this feature, it's *very*
> useful. Both for config via the include macro, and for people to
> manually paste some config they want to try out to the end of their
> config, without having to manually edit it to incorporate it into their
> already existing sections.
>

--
Philip 

Re: Bug: duplicate sections in .git/config after remote removal

[Philip Oakley]

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
> On Tue, Mar 27 2018, Jason Frey wrote:
>
>> While the impact of this bug is minimal, and git itself is not
>> affected, it can affect external tools that want to read the
>> .git/config file, expecting unique section names.
>>
>> To reproduce:
>>
>> Given the following example .git/config file (I am leaving out the
>> [core] section for brevity):
>>
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> Running `git remote rm origin` will result in the following contents:
>>
>>     [branch "master"]
>>
>> Running `git remote add origin git@github.com:Fryguy/example.git` will
>> result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>
>> And finally, running `git fetch origin; git branch -u origin/master`
>> will result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> at which point you can see the duplicate sections (even though one is
>> empty).  Also note that if you do the steps again, you will be left
>> with 3 sections, 2 of which are empty.  This process can be repeated
>> over and over.
>
> This can be annoying and result in some very verbose config files when
> we automatically edit them, e.g.:
>
>    (rm -v /tmp/test.ini; for i in {1..3}; do git config -f /tmp/test.ini 
> foo.bar 0 && git config -f /tmp/test.ini --unset foo.bar; done; cat 
> /tmp/test.ini)
>    removed '/tmp/test.ini'
>    [foo]
>    [foo]
>    [foo]
>
> But it's not so clear that it should be called a bug, yes we could be a
> bit smarter and not add obvious crap like the example above (duplicate
> sections at the end), but it gets less obvious in more complex cases,
> see my c8b2cec09e ("branch: add test for -m renaming multiple config
> sections", 2017-06-18) for one such example.
>
> Git has a config format that's hybrid human/machine editable. Consider a
> case like:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>
> Now, if I run `git config gc.auto 0` is it better if we end up with:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    auto = 0
>    ;; Our aliases
>    [alias]
>    st = status
>
> Or something that makes it more clear that a machine added something at
> the end:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>    [gc]
>    auto = 0
>
> Most importantly though, regardless of what we decide to do when we
> machine-edit the file, it's also human-editable, and being able to
> repeat sections is part of our config format that you're simply going to
> have to deal with.

One option may be to create  a simple 'lint' style checker that simply 
hiughlights and suggests options so the user can decide for themselves what 
they need to do. This would help span the gap between hard format and the 
soft format capabiulities of machine readable ini files, the Git config 
reader and being human readable.

Thus duplicate sections would be noted, likewise the presence of comments 
immediately preceding a section header, or terminating a section (with or 
without spacing?), etc.Such a config_lint could reside in the contrib as a 
supprt tool, and may in the long term be a guide to a common format. 
However, as noted, it would be more of a long term aspiration..


>
> The external tool (presumably some generic *.ini parser) you're trying
> to point at git's config is broken for that purpose if it doesn't handle
> duplicate sections. You're probably better off trying to parse `git
> config --list --null` than trying to make it work.
>
> I don't think we'd ever want to get rid of this feature, it's *very*
> useful. Both for config via the include macro, and for people to
> manually paste some config they want to try out to the end of their
> config, without having to manually edit it to incorporate it into their
> already existing sections.
>

--
Philip 

Re: Bug: duplicate sections in .git/config after remote removal

[Philip Oakley]

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
> On Tue, Mar 27 2018, Jason Frey wrote:
>
>> While the impact of this bug is minimal, and git itself is not
>> affected, it can affect external tools that want to read the
>> .git/config file, expecting unique section names.
>>
>> To reproduce:
>>
>> Given the following example .git/config file (I am leaving out the
>> [core] section for brevity):
>>
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> Running `git remote rm origin` will result in the following contents:
>>
>>     [branch "master"]
>>
>> Running `git remote add origin git@github.com:Fryguy/example.git` will
>> result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>
>> And finally, running `git fetch origin; git branch -u origin/master`
>> will result in the following contents:
>>
>>     [branch "master"]
>>     [remote "origin"]
>>         url = git@github.com:Fryguy/example.git
>>         fetch = +refs/heads/*:refs/remotes/origin/*
>>     [branch "master"]
>>         remote = origin
>>         merge = refs/heads/master
>>
>> at which point you can see the duplicate sections (even though one is
>> empty).  Also note that if you do the steps again, you will be left
>> with 3 sections, 2 of which are empty.  This process can be repeated
>> over and over.
>
> This can be annoying and result in some very verbose config files when
> we automatically edit them, e.g.:
>
>    (rm -v /tmp/test.ini; for i in {1..3}; do git config -f /tmp/test.ini
> foo.bar 0 && git config -f /tmp/test.ini --unset foo.bar; done; cat
> /tmp/test.ini)
>    removed '/tmp/test.ini'
>    [foo]
>    [foo]
>    [foo]
>
> But it's not so clear that it should be called a bug, yes we could be a
> bit smarter and not add obvious crap like the example above (duplicate
> sections at the end), but it gets less obvious in more complex cases,
> see my c8b2cec09e ("branch: add test for -m renaming multiple config
> sections", 2017-06-18) for one such example.
>
> Git has a config format that's hybrid human/machine editable. Consider a
> case like:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>
> Now, if I run `git config gc.auto 0` is it better if we end up with:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    auto = 0
>    ;; Our aliases
>    [alias]
>    st = status
>
> Or something that makes it more clear that a machine added something at
> the end:
>
>    [gc]
>    ;; Here's all the gc config we set up to avoid the great outage of 2015
>    autoDetach = false
>    ;; Our aliases
>    [alias]
>    st = status
>    [gc]
>    auto = 0
>
> Most importantly though, regardless of what we decide to do when we
> machine-edit the file, it's also human-editable, and being able to
> repeat sections is part of our config format that you're simply going to
> have to deal with.

One option may be to create  a simple 'lint' style checker that simply
hiughlights and suggests options so the user can decide for themselves what
they need to do. This would help span the gap between hard format and the
soft format capabiulities of machine readable ini files, the Git config
reader and being human readable.

Thus duplicate sections would be noted, likewise the presence of comments
immediately preceding a section header, or terminating a section (with or
without spacing?), etc.Such a config_lint could reside in the contrib as a
supprt tool, and may in the long term be a guide to a common format.
However, as noted, it would be more of a long term aspiration..


>
> The external tool (presumably some generic *.ini parser) you're trying
> to point at git's config is broken for that purpose if it doesn't handle
> duplicate sections. You're probably better off trying to parse `git
> config --list --null` than trying to make it work.
>
> I don't think we'd ever want to get rid of this feature, it's *very*
> useful. Both for config via the include macro, and for people to
> manually paste some config they want to try out to the end of their
> config, without having to manually edit it to incorporate it into their
> already existing sections.
>

--
Philip

Re: Bug: duplicate sections in .git/config after remote removal

[Johannes Schindelin]

Hi Stefan & Jason,

On Tue, 27 Mar 2018, Stefan Beller wrote:

> On Tue, Mar 27, 2018 at 1:41 PM Jason Frey <jfrey@redhat.com> wrote:
> 
> > at which point you can see the duplicate sections (even though one is
> > empty).  Also note that if you do the steps again, you will be left
> > with 3 sections, 2 of which are empty.  This process can be repeated
> > over and over.
> 
> I agree that this is an issue for the user, and there were some attempts
> to fix it in the past. (feel free to dig them up in the archive at
> https://public-inbox.org/git)

Note: as far as I remember, the attempted fixes were exclusively trying to
remove the empty section. But this report suggests that we could instead
*keep* empty sections, but then reuse them when a new value is added.

> IIRC the problem is (a) with the loose file format (What if the user put
> a valuable comment just after or before the '[branch "master"]' line?)
> as well as (b) the way the parser/writer works (single pass, line by line)
> 
> (b) specifically made it a "huge effort, but little return" bug,
> so nobody got around for a proper fix.

Yes, (a) makes removing an empty section something less of a desirable
thing. Unless there are no comments before and after the section, of
course, and yes, (b) is a real thing.

On a positive note: I just finished work on a set of patches addressing
this:
https://github.com/git/git/compare/master...dscho:empty-config-section (I
plan on submitting this tomorrow)

Ciao,
Dscho