is this data corruption?

is this data corruption?

[Samuel Wales]

i am not subscribed, but am of the impression that's ok.  please copy
me directly.


tldr: git diff is showing differences that do not exist in the files themselves.

i have nothing staged, nothing fancy like stashing, etc.  this is a
repo of mostly emacs org mode files.  mostly ascii text.

git status and these commands show nothing unusual:

    git fsck --strict --no-dangling
    git gc --prune="0 days"


the problem that seems like data corruption is that a few lines appear
twice as - and once as +.  but in the current version of the files,
those lines exist only once.  here are the lines.  there are 2 -
versions and one + version:

+***************** REF bigpart is a partition
+biglike and homelike are distracting nonsense i think except
+to describe inferior filesets.  anomalous subset of home
+might be called homelike or so.


emacs magit shows the same problem.  however, it shows a slightly
different diff.  i did a meta-diff on git diff vs. magit, and there
are about 800 + real-content lines that magit shows but git diff does
not.  i do not know what this means.  wc -l is like

  62540 aaa.diff
  62965 bbb--magit.txt

idk why a diff would be different with only + lines being different?


in summary, what is wrong with my repo, if anything, and what can i do
about it?  nothing on the web for git corruption seems to say much,
other than pull from github or whatever.  this is my own repo, the
original repo, so i cannot do that.  org annex has an uncorrupt tool
of some kind, but it did not seem relevant.  i do have rsnapshot
[basically rsync] backups of the repo and the most significant files
and dirs, but i do not know what one does to use that to repair any
issues.  i won't get into why, but changes were made over months.

is there a protocol for this?

would git fsck have balked?

thank you!


p.s.

i have no reason to believe this is related, but git diff has
intermingled emacs org mode entries.  but i don't have to talk about
it in org terms; in generic text terms, it has intermingled parts of
different paragraphs.  as a user, i'd prefer that completely unrelated
paragraphs not be mingled, regardless of minimality.  if possible.

with respect to the intermingling only, unless this is related to the
possible corruption, i will presume the diff is correct, in that a
patch from it would produce the same result as a patch that does not
intermingle.  i believe this intermingling is because diff does not
understand org, or paragraphs for that matter.  in org, an entry
starts with "^[*]+ " and ends at the beginning of another entry or at
eof.  they consist in my case mostly of ascii text paragraphs.  just
as with paragraphs, if you move an entry, you don't expect it to be
mingled with a different one in the diff.

i have been told that this cannot be fixed by merely telling a
slightly improved differ that stuff between stars is worth preserving,
but that a parser, not merely a couple of regexps, is needed to reduce
this intermingling.  i have also been told that difftastic uses
tree-sitter, which might get such a syntax for emacs org mode.  and so
maybe at some point git diff can use that.  idk.

idk if any of this is related but i include it for completeness.

also, please don't laugh, but i am using git version 2.11.0.  i will
upgrade pending various library and os stuff but my main concern is
not for git, but for possible corruption in the repo and what is
possible to do, at least given rsnapshot, to fix it.

RE: is this data corruption?

[rsbecker]

On December 30, 2022 7:18 PM, Samuel Wales wrote:
>i am not subscribed, but am of the impression that's ok.  please copy me directly.
>
>
>tldr: git diff is showing differences that do not exist in the files themselves.
>
>i have nothing staged, nothing fancy like stashing, etc.  this is a repo of mostly
>emacs org mode files.  mostly ascii text.
>
>git status and these commands show nothing unusual:
>
>    git fsck --strict --no-dangling
>    git gc --prune="0 days"
>
>
>the problem that seems like data corruption is that a few lines appear twice as -
>and once as +.  but in the current version of the files, those lines exist only once.
>here are the lines.  there are 2 - versions and one + version:
>
>+***************** REF bigpart is a partition biglike and homelike are
>+distracting nonsense i think except to describe inferior filesets.
>+anomalous subset of home might be called homelike or so.
>
>
>emacs magit shows the same problem.  however, it shows a slightly different diff.
>i did a meta-diff on git diff vs. magit, and there are about 800 + real-content lines
>that magit shows but git diff does not.  i do not know what this means.  wc -l is like
>
>  62540 aaa.diff
>  62965 bbb--magit.txt
>
>idk why a diff would be different with only + lines being different?
>
>
>in summary, what is wrong with my repo, if anything, and what can i do about it?
>nothing on the web for git corruption seems to say much, other than pull from
>github or whatever.  this is my own repo, the original repo, so i cannot do that.  org
>annex has an uncorrupt tool of some kind, but it did not seem relevant.  i do have
>rsnapshot [basically rsync] backups of the repo and the most significant files and
>dirs, but i do not know what one does to use that to repair any issues.  i won't get
>into why, but changes were made over months.
>
>is there a protocol for this?
>
>would git fsck have balked?
>
>thank you!
>
>
>p.s.
>
>i have no reason to believe this is related, but git diff has intermingled emacs org
>mode entries.  but i don't have to talk about it in org terms; in generic text terms, it
>has intermingled parts of different paragraphs.  as a user, i'd prefer that
>completely unrelated paragraphs not be mingled, regardless of minimality.  if
>possible.
>
>with respect to the intermingling only, unless this is related to the possible
>corruption, i will presume the diff is correct, in that a patch from it would produce
>the same result as a patch that does not intermingle.  i believe this intermingling is
>because diff does not understand org, or paragraphs for that matter.  in org, an
>entry starts with "^[*]+ " and ends at the beginning of another entry or at eof.
>they consist in my case mostly of ascii text paragraphs.  just as with paragraphs, if
>you move an entry, you don't expect it to be mingled with a different one in the
>diff.
>
>i have been told that this cannot be fixed by merely telling a slightly improved
>differ that stuff between stars is worth preserving, but that a parser, not merely a
>couple of regexps, is needed to reduce this intermingling.  i have also been told
>that difftastic uses tree-sitter, which might get such a syntax for emacs org mode.
>and so maybe at some point git diff can use that.  idk.
>
>idk if any of this is related but i include it for completeness.
>
>also, please don't laugh, but i am using git version 2.11.0.  i will upgrade pending
>various library and os stuff but my main concern is not for git, but for possible
>corruption in the repo and what is possible to do, at least given rsnapshot, to fix it.

I cannot account for your emacs issues, but have a question. Have you done any git add operations? Git diff is subject to what specifically is staged. So you might be comparing your file with partially staged content that could account for partial diffs. See if diff --cached makes a difference. Also try different algorithms, like --patience or --diff-algorithm=histogram.

--Randall

Re: is this data corruption?

[Samuel Wales]

more below.

On 12/30/22, rsbecker@nexbridge.com <rsbecker@nexbridge.com> wrote:
> On December 30, 2022 7:18 PM, Samuel Wales wrote:
>>i am not subscribed, but am of the impression that's ok.  please copy me
>> directly.

...

> I cannot account for your emacs issues, but have a question. Have you done
> any git add operations? Git diff is subject to what specifically is staged.

idk what this means.  there is nothing in the staging area at this
time, if that is relevant.

> So you might be comparing your file with partially staged content that could
> account for partial diffs. See if diff --cached makes a difference. Also try

diff --cached produces nothing.  0 bytes.

> different algorithms, like --patience or --diff-algorithm=histogram.

i tried both with git 2.20.  they produce different output from
regular git diff.  i looked for the paragraph i mentioned.  in regular
git diff, the problem is the same as i described, with - - +.  in both
patience and histogram, it is -.

it occurs to me that, although unlikely, i might have in principle had
a duplicate copy of those lines in A, and deleted one, and moved the
other, when i created the current version, B. thus, i /think/ both
regular and patience/histogram could be /in principle/ correct on that
one point.  to confirm, i will check rsnapshot using grep -c to count
the matches in all versions of the original files.  unless i report
back, the number will be 1 in all of them, i.e. git diff and magit
status buffer are both producing an incorrect diff.

however, even if this is user error, i.e., i deleted a duplicate and
moved it, the patience/histogram - result seems still incorrect:

 The next few commands work with both grub> and grub rescue>.
-***************** REF bigpart is a partition
-biglike and homelike are distracting nonsense i think except
-to describe inferior filesets.  anomalous subset of home
-might be called homelike or so.

 1) The first command you should run invokes the pager, for

you can see here that the context lines are from a different org
entry.  it is intermingling a live - line with a context line that is
not adjacent to it.  that seems corrupt unless i, even more
improbably, moved an entry from inside another entry.

>
> --Randall



>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: is this data corruption?

[Samuel Wales]

as expected, there is only one copy of bigpart is a partition in all
of my rsnapshot files and all of my emacs backup files.

therefore, the diff in git and magit are incorrect.  it was not a
question of me removing a duplicate.

for future reference, is this mailing list correct for this question
or should i use the git help list which i discovered just now?


On 12/30/22, Samuel Wales <samologist@gmail.com> wrote:
> more below.
>
> On 12/30/22, rsbecker@nexbridge.com <rsbecker@nexbridge.com> wrote:
>> On December 30, 2022 7:18 PM, Samuel Wales wrote:
>>>i am not subscribed, but am of the impression that's ok.  please copy me
>>> directly.
>
> ...
>
>> I cannot account for your emacs issues, but have a question. Have you
>> done
>> any git add operations? Git diff is subject to what specifically is
>> staged.
>
> idk what this means.  there is nothing in the staging area at this
> time, if that is relevant.
>
>> So you might be comparing your file with partially staged content that
>> could
>> account for partial diffs. See if diff --cached makes a difference. Also
>> try
>
> diff --cached produces nothing.  0 bytes.
>
>> different algorithms, like --patience or --diff-algorithm=histogram.
>
> i tried both with git 2.20.  they produce different output from
> regular git diff.  i looked for the paragraph i mentioned.  in regular
> git diff, the problem is the same as i described, with - - +.  in both
> patience and histogram, it is -.
>
> it occurs to me that, although unlikely, i might have in principle had
> a duplicate copy of those lines in A, and deleted one, and moved the
> other, when i created the current version, B. thus, i /think/ both
> regular and patience/histogram could be /in principle/ correct on that
> one point.  to confirm, i will check rsnapshot using grep -c to count
> the matches in all versions of the original files.  unless i report
> back, the number will be 1 in all of them, i.e. git diff and magit
> status buffer are both producing an incorrect diff.
>
> however, even if this is user error, i.e., i deleted a duplicate and
> moved it, the patience/histogram - result seems still incorrect:
>
>  The next few commands work with both grub> and grub rescue>.
> -***************** REF bigpart is a partition
> -biglike and homelike are distracting nonsense i think except
> -to describe inferior filesets.  anomalous subset of home
> -might be called homelike or so.
>
>  1) The first command you should run invokes the pager, for
>
> you can see here that the context lines are from a different org
> entry.  it is intermingling a live - line with a context line that is
> not adjacent to it.  that seems corrupt unless i, even more
> improbably, moved an entry from inside another entry.
>
>>
>> --Randall
>
>
>
>>
>>
>
>
> --
> The Kafka Pandemic
>
> A blog about science, health, human rights, and misopathy:
> https://thekafkapandemic.blogspot.com
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: is this data corruption?

[Samuel Wales]

r> Have you done any git add operations?

i presume this is staging.  i do staging in magit all the time.  but
right now the staging area is empty and so is diff --cached.

On 12/30/22, rsbecker@nexbridge.com <rsbecker@nexbridge.com> wrote:
> On December 30, 2022 7:18 PM, Samuel Wales wrote:
>>i am not subscribed, but am of the impression that's ok.  please copy me
>> directly.
>>
>>
>>tldr: git diff is showing differences that do not exist in the files
>> themselves.
>>
>>i have nothing staged, nothing fancy like stashing, etc.  this is a repo of
>> mostly
>>emacs org mode files.  mostly ascii text.
>>
>>git status and these commands show nothing unusual:
>>
>>    git fsck --strict --no-dangling
>>    git gc --prune="0 days"
>>
>>
>>the problem that seems like data corruption is that a few lines appear
>> twice as -
>>and once as +.  but in the current version of the files, those lines exist
>> only once.
>>here are the lines.  there are 2 - versions and one + version:
>>
>>+***************** REF bigpart is a partition biglike and homelike are
>>+distracting nonsense i think except to describe inferior filesets.
>>+anomalous subset of home might be called homelike or so.
>>
>>
>>emacs magit shows the same problem.  however, it shows a slightly different
>> diff.
>>i did a meta-diff on git diff vs. magit, and there are about 800 +
>> real-content lines
>>that magit shows but git diff does not.  i do not know what this means.  wc
>> -l is like
>>
>>  62540 aaa.diff
>>  62965 bbb--magit.txt
>>
>>idk why a diff would be different with only + lines being different?
>>
>>
>>in summary, what is wrong with my repo, if anything, and what can i do
>> about it?
>>nothing on the web for git corruption seems to say much, other than pull
>> from
>>github or whatever.  this is my own repo, the original repo, so i cannot do
>> that.  org
>>annex has an uncorrupt tool of some kind, but it did not seem relevant.  i
>> do have
>>rsnapshot [basically rsync] backups of the repo and the most significant
>> files and
>>dirs, but i do not know what one does to use that to repair any issues.  i
>> won't get
>>into why, but changes were made over months.
>>
>>is there a protocol for this?
>>
>>would git fsck have balked?
>>
>>thank you!
>>
>>
>>p.s.
>>
>>i have no reason to believe this is related, but git diff has intermingled
>> emacs org
>>mode entries.  but i don't have to talk about it in org terms; in generic
>> text terms, it
>>has intermingled parts of different paragraphs.  as a user, i'd prefer
>> that
>>completely unrelated paragraphs not be mingled, regardless of minimality.
>> if
>>possible.
>>
>>with respect to the intermingling only, unless this is related to the
>> possible
>>corruption, i will presume the diff is correct, in that a patch from it
>> would produce
>>the same result as a patch that does not intermingle.  i believe this
>> intermingling is
>>because diff does not understand org, or paragraphs for that matter.  in
>> org, an
>>entry starts with "^[*]+ " and ends at the beginning of another entry or at
>> eof.
>>they consist in my case mostly of ascii text paragraphs.  just as with
>> paragraphs, if
>>you move an entry, you don't expect it to be mingled with a different one
>> in the
>>diff.
>>
>>i have been told that this cannot be fixed by merely telling a slightly
>> improved
>>differ that stuff between stars is worth preserving, but that a parser, not
>> merely a
>>couple of regexps, is needed to reduce this intermingling.  i have also
>> been told
>>that difftastic uses tree-sitter, which might get such a syntax for emacs
>> org mode.
>>and so maybe at some point git diff can use that.  idk.
>>
>>idk if any of this is related but i include it for completeness.
>>
>>also, please don't laugh, but i am using git version 2.11.0.  i will
>> upgrade pending
>>various library and os stuff but my main concern is not for git, but for
>> possible
>>corruption in the repo and what is possible to do, at least given
>> rsnapshot, to fix it.
>
> I cannot account for your emacs issues, but have a question. Have you done
> any git add operations? Git diff is subject to what specifically is staged.
> So you might be comparing your file with partially staged content that could
> account for partial diffs. See if diff --cached makes a difference. Also try
> different algorithms, like --patience or --diff-algorithm=histogram.
>
> --Randall
>
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: is this data corruption?

[Samuel Wales]

p.p.s.  git 2.20 has the same problem.

On 12/30/22, Samuel Wales <samologist@gmail.com> wrote:
> i am not subscribed, but am of the impression that's ok.  please copy
> me directly.
>
>
> tldr: git diff is showing differences that do not exist in the files
> themselves.
>
> i have nothing staged, nothing fancy like stashing, etc.  this is a
> repo of mostly emacs org mode files.  mostly ascii text.
>
> git status and these commands show nothing unusual:
>
>     git fsck --strict --no-dangling
>     git gc --prune="0 days"
>
>
> the problem that seems like data corruption is that a few lines appear
> twice as - and once as +.  but in the current version of the files,
> those lines exist only once.  here are the lines.  there are 2 -
> versions and one + version:
>
> +***************** REF bigpart is a partition
> +biglike and homelike are distracting nonsense i think except
> +to describe inferior filesets.  anomalous subset of home
> +might be called homelike or so.
>
>
> emacs magit shows the same problem.  however, it shows a slightly
> different diff.  i did a meta-diff on git diff vs. magit, and there
> are about 800 + real-content lines that magit shows but git diff does
> not.  i do not know what this means.  wc -l is like
>
>   62540 aaa.diff
>   62965 bbb--magit.txt
>
> idk why a diff would be different with only + lines being different?
>
>
> in summary, what is wrong with my repo, if anything, and what can i do
> about it?  nothing on the web for git corruption seems to say much,
> other than pull from github or whatever.  this is my own repo, the
> original repo, so i cannot do that.  org annex has an uncorrupt tool
> of some kind, but it did not seem relevant.  i do have rsnapshot
> [basically rsync] backups of the repo and the most significant files
> and dirs, but i do not know what one does to use that to repair any
> issues.  i won't get into why, but changes were made over months.
>
> is there a protocol for this?
>
> would git fsck have balked?
>
> thank you!
>
>
> p.s.
>
> i have no reason to believe this is related, but git diff has
> intermingled emacs org mode entries.  but i don't have to talk about
> it in org terms; in generic text terms, it has intermingled parts of
> different paragraphs.  as a user, i'd prefer that completely unrelated
> paragraphs not be mingled, regardless of minimality.  if possible.
>
> with respect to the intermingling only, unless this is related to the
> possible corruption, i will presume the diff is correct, in that a
> patch from it would produce the same result as a patch that does not
> intermingle.  i believe this intermingling is because diff does not
> understand org, or paragraphs for that matter.  in org, an entry
> starts with "^[*]+ " and ends at the beginning of another entry or at
> eof.  they consist in my case mostly of ascii text paragraphs.  just
> as with paragraphs, if you move an entry, you don't expect it to be
> mingled with a different one in the diff.
>
> i have been told that this cannot be fixed by merely telling a
> slightly improved differ that stuff between stars is worth preserving,
> but that a parser, not merely a couple of regexps, is needed to reduce
> this intermingling.  i have also been told that difftastic uses
> tree-sitter, which might get such a syntax for emacs org mode.  and so
> maybe at some point git diff can use that.  idk.
>
> idk if any of this is related but i include it for completeness.
>
> also, please don't laugh, but i am using git version 2.11.0.  i will
> upgrade pending various library and os stuff but my main concern is
> not for git, but for possible corruption in the repo and what is
> possible to do, at least given rsnapshot, to fix it.
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: is this data corruption?

[Junio C Hamano]

Samuel Wales <samologist@gmail.com> writes:

> the problem that seems like data corruption is that a few lines appear
> twice as - and once as +.  but in the current version of the files,
> those lines exist only once.  here are the lines.  there are 2 -
> versions and one + version:
>
> +***************** REF bigpart is a partition
> +biglike and homelike are distracting nonsense i think except
> +to describe inferior filesets.  anomalous subset of home
> +might be called homelike or so.

Sorry, but I do not see any "-" versions in the above.  Simple and
minimum reproduction recipe would be useful for the list to help.

https://www.chiark.greenend.org.uk/~sgtatham/bugs.html

Thanks.

Re: is this data corruption?

[Samuel Wales]

hi,

thanks for reading/thinking.

> Sorry, but I do not see any "-" versions in the above.  Simple and
minimum reproduction recipe would be useful for the list to help.

the - versions are identical.  as i wrote, there are 2 - versions in
addition.  i didn't want to make a pointlessly long email.

it would be great if i could provide a reproduction recipe.  i don't
know how i couold do that in this case.  for example, when i try to
pare down the repo and make private things obscured, i run into the
proble that when i do staging, things change around, thus erasing that
particular bug.  and other stuff like my limited knowledge of git.

i'm afraid i am stuck with hoping for other types of solutions like
"i've seen this before" or "that's not possuble unless..."or something
a priori like that.  but if you have an idea of how to create a
minimal repro in less than a week, let me know.


On 12/31/22, Junio C Hamano <gitster@pobox.com> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> the problem that seems like data corruption is that a few lines appear
>> twice as - and once as +.  but in the current version of the files,
>> those lines exist only once.  here are the lines.  there are 2 -
>> versions and one + version:
>>
>> +***************** REF bigpart is a partition
>> +biglike and homelike are distracting nonsense i think except
>> +to describe inferior filesets.  anomalous subset of home
>> +might be called homelike or so.
>
> Sorry, but I do not see any "-" versions in the above.  Simple and
> minimum reproduction recipe would be useful for the list to help.
>
> https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>
> Thanks.
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com

Re: is this data corruption?

[Samuel Wales]

of course i am still hoping, if possible,for generic adice on
repairing a repo, detecting a bug, whether it is a bug, what looks
curiously strange, maybe some more questions like the previous one
about what does --cached say, what does --histogram and --patience do,
etc., any of this whether or not a repro is possible.  will repro if
possible just seems impossible.

On 12/31/22, Samuel Wales <samologist@gmail.com> wrote:
> hi,
>
> thanks for reading/thinking.
>
>> Sorry, but I do not see any "-" versions in the above.  Simple and
> minimum reproduction recipe would be useful for the list to help.
>
> the - versions are identical.  as i wrote, there are 2 - versions in
> addition.  i didn't want to make a pointlessly long email.
>
> it would be great if i could provide a reproduction recipe.  i don't
> know how i couold do that in this case.  for example, when i try to
> pare down the repo and make private things obscured, i run into the
> proble that when i do staging, things change around, thus erasing that
> particular bug.  and other stuff like my limited knowledge of git.
>
> i'm afraid i am stuck with hoping for other types of solutions like
> "i've seen this before" or "that's not possuble unless..."or something
> a priori like that.  but if you have an idea of how to create a
> minimal repro in less than a week, let me know.
>
>
> On 12/31/22, Junio C Hamano <gitster@pobox.com> wrote:
>> Samuel Wales <samologist@gmail.com> writes:
>>
>>> the problem that seems like data corruption is that a few lines appear
>>> twice as - and once as +.  but in the current version of the files,
>>> those lines exist only once.  here are the lines.  there are 2 -
>>> versions and one + version:
>>>
>>> +***************** REF bigpart is a partition
>>> +biglike and homelike are distracting nonsense i think except
>>> +to describe inferior filesets.  anomalous subset of home
>>> +might be called homelike or so.
>>
>> Sorry, but I do not see any "-" versions in the above.  Simple and
>> minimum reproduction recipe would be useful for the list to help.
>>
>> https://www.chiark.greenend.org.uk/~sgtatham/bugs.html
>>
>> Thanks.
>>
>
>
> --
> The Kafka Pandemic
>
> A blog about science, health, human rights, and misopathy:
> https://thekafkapandemic.blogspot.com
>


-- 
The Kafka Pandemic

A blog about science, health, human rights, and misopathy:
https://thekafkapandemic.blogspot.com