Help understanding unexpected diff output

Help understanding unexpected diff output

[Ulrich Windl]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

Hi!

I hope we all agree that diff should make our lives easier, so here is something where I think it does not apply:

I inspected some changes using "git diff", and it was hard to me to find the actual differences (see git-diff-png):
I had to look a long time at those olcAccess lines to find any difference, so finally I pasted both blocks into separate Emacs files a and b  (via PuTTY) and used "ediff-files" (see diff-emacs.png).
Within there, the was only a part of one line changed!

I don't understand this. I had even used "git diff -w" to ignore space changes (which aren't expected anyway).

Any ideas what might be wrong and how to get better diff output?

(git-2.26.2-27.36.1.x86_64 of SLES12 being used)

Regards,
Ulrich






[-- Attachment #2: diff-git.PNG --]
[-- Type: image/png, Size: 70291 bytes --]

[-- Attachment #3: diff-emacs.PNG --]
[-- Type: image/png, Size: 27142 bytes --]

Re: Help understanding unexpected diff output

[Junio C Hamano]

"Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de> writes:

> I inspected some changes using "git diff", and it was hard to me
> to find the actual differences (see git-diff-png): I had to look a
> long time at those olcAccess lines to find any difference, so
> finally I pasted both blocks into separate Emacs files a and b
> (via PuTTY) and used "ediff-files" (see diff-emacs.png).  Within
> there, the was only a part of one line changed!

With redactions and making them into two images, you did a good job
killing interested parties' willingness to help, as they made using
tools almost impossible.

So here is only based on eyeballing the images, and I probably have
missed what you wanted readers to find, but don't blame me ;-)

It seems that the file got 6 lines removed, that begins with
"olcAccess: {$n}" (0 <= n <= 6), at around 8th line of the file, and
then very similar 6 lines that are not identical to what got removed
at the end of the file.  The ediff you showed tells me that the 6
lines removed from very early part of the file and 6 lines added to
the end of the file are similar but not identical.  These two 6-line
blocks differ only by the 5th line (i.e. "olcAccess: {4}") having
",pwdGraceUseTime,pwdChangedTime" in the newer version, but not in
the older version.

Am I reading what you sent us correctly so far?

Now, if this change to the "olcAccess: {4}" line in the new file
weren't there, i.e. if the 6-line block that disappeared from the
early part of the file and the 6-line block that was added near the
end of the file were identical (in such a case, your ediff would
show no changes, I presume), then what would your desired output
from "git diff"?

I would imagine that the answer would still be a very similar looking

    --- a/0/...
    +++ b/0/...
    @@ -5,12 +5,6 @@
     olcDatabase: {1}hdb
     olcDbDirectory: /var/lib/ldap
     olcSuffix: dc=... 
    -olcAccess: {0} ...
    -olcAccess: {1} ...
    -olcAccess: {2} ...
    -olcAccess: {3} ...
    -olcAccess: {4} ...
    -olcAccess: {5} ...
    olcLimits: {0}dn.exact=....
     olcRootDn: cn=Admin,dc=...
     olcRootPW: {SSHA}yZ...
    @@ -52,6 +46,12 @@
     entryUUID: ...
     creatorsName: cn=config
     createTimestamp: ...
    -entryCSN: ...
    +olcAccess: {0} ...
    +olcAccess: {1} ...
    +olcAccess: {2} ...
    +olcAccess: {3} ...
    +olcAccess: {4} ...
    +olcAccess: {5} ...
    +entryCSN: ...
     modifiersName: cn=config
    -modifyTimestamp: ...
    +modifyTimestamp: ...

with identical "olcAccess: {$n}" lines removed from earlier part the
file and added to the end of the file.

So I am not sure what you are asking.  It would not be correct to show
something like

    --- a/0/...
    +++ b/0/...
    @@ -52,6 +46,6 @@
     entryUUID: ...
     creatorsName: cn=config
     createTimestamp: ...
    -entryCSN: ...
    +entryCSN: ...
     modifiersName: cn=config
    -modifyTimestamp: ...
    +modifyTimestamp: ...

and pretend that the original "olcAccess: {$n}" lines did not get
moved from the earlier part of the file to the end of the file.

Perhaps you may be looking for "git diff --color-moved"?  I dunno.

Antw: [EXT] Re: Help understanding unexpected diff output

[Ulrich Windl]

[-- Attachment #1: Type: text/plain, Size: 11213 bytes --]

>>> Junio C Hamano <gitster@pobox.com> schrieb am 13.07.2020 um 17:12 in
Nachricht
<xmqqsgdvo3ec.fsf@gitster.c.googlers.com>:
> "Ulrich Windl" <Ulrich.Windl@rz.uni‑regensburg.de> writes:
> 
>> I inspected some changes using "git diff", and it was hard to me
>> to find the actual differences (see git‑diff‑png): I had to look a
>> long time at those olcAccess lines to find any difference, so
>> finally I pasted both blocks into separate Emacs files a and b
>> (via PuTTY) and used "ediff‑files" (see diff‑emacs.png).  Within
>> there, the was only a part of one line changed!
> 
> With redactions and making them into two images, you did a good job
> killing interested parties' willingness to help, as they made using
> tools almost impossible.

Junio,

we had some discussions in the past, and you should know that I'm willing to
provide information if needed.
Part of the problem is that I can't (to my knowledge) copy and paste colored
text into plain text emails. Thus screenshots.
My intention was to bring up the issue, making it (more or less obvious) that
the differences are much harder to see in git output than in Emacs output.
The redaction was due to a password that could easily be cracked if presented
and some resource name I wanted to avoid to get a clearance to publish first.
It still could be a "stupid me", but can't we talk like adults?

> 
> So here is only based on eyeballing the images, and I probably have
> missed what you wanted readers to find, but don't blame me ;‑)
> 
> It seems that the file got 6 lines removed, that begins with
> "olcAccess: {$n}" (0 <= n <= 6), at around 8th line of the file, and
> then very similar 6 lines that are not identical to what got removed
> at the end of the file.  The ediff you showed tells me that the 6
> lines removed from very early part of the file and 6 lines added to
> the end of the file are similar but not identical.  These two 6‑line
> blocks differ only by the 5th line (i.e. "olcAccess: {4}") having
> ",pwdGraceUseTime,pwdChangedTime" in the newer version, but not in
> the older version.
> 
> Am I reading what you sent us correctly so far?

Yes, if you look at the highlighted part (difference in cyan, "chink" in
yellow) in the Emacs screenshot, it should point out the "real" differences,
while git output made it harder to see.

> 
> Now, if this change to the "olcAccess: {4}" line in the new file
> weren't there, i.e. if the 6‑line block that disappeared from the
> early part of the file and the 6‑line block that was added near the
> end of the file were identical (in such a case, your ediff would
> show no changes, I presume), then what would your desired output
> from "git diff"?
> 
> I would imagine that the answer would still be a very similar looking
> 
>     ‑‑‑ a/0/...
>     +++ b/0/...
>     @@ ‑5,12 +5,6 @@
>      olcDatabase: {1}hdb
>      olcDbDirectory: /var/lib/ldap
>      olcSuffix: dc=... 
>     ‑olcAccess: {0} ...
>     ‑olcAccess: {1} ...
>     ‑olcAccess: {2} ...
>     ‑olcAccess: {3} ...
>     ‑olcAccess: {4} ...
>     ‑olcAccess: {5} ...
>     olcLimits: {0}dn.exact=....
>      olcRootDn: cn=Admin,dc=...
>      olcRootPW: {SSHA}yZ...
>     @@ ‑52,6 +46,12 @@
>      entryUUID: ...
>      creatorsName: cn=config
>      createTimestamp: ...
>     ‑entryCSN: ...
>     +olcAccess: {0} ...
>     +olcAccess: {1} ...
>     +olcAccess: {2} ...
>     +olcAccess: {3} ...
>     +olcAccess: {4} ...
>     +olcAccess: {5} ...
>     +entryCSN: ...
>      modifiersName: cn=config
>     ‑modifyTimestamp: ...
>     +modifyTimestamp: ...
> 
> with identical "olcAccess: {$n}" lines removed from earlier part the
> file and added to the end of the file.
> 
> So I am not sure what you are asking.  It would not be correct to show
> something like

According to my eyes (just using Emacs to make sure) those olcAccess lines
"numbered" 0 to 3, and 5 should be "context lines" as they are not changed.
Only line numbered {4} has a difference, and as the order 0 to 5 is unchanged I
don't see why the diff-block is that large. I see that some lines that are not
changed were reorderd, but that should not affect the olcAccess block. Maybe I
just need different diff options; I don't know.


> 
>     ‑‑‑ a/0/...
>     +++ b/0/...
>     @@ ‑52,6 +46,6 @@
>      entryUUID: ...
>      creatorsName: cn=config
>      createTimestamp: ...
>     ‑entryCSN: ...
>     +entryCSN: ...
>      modifiersName: cn=config
>     ‑modifyTimestamp: ...
>     +modifyTimestamp: ...
> 
> and pretend that the original "olcAccess: {$n}" lines did not get
> moved from the earlier part of the file to the end of the file.
> 
> Perhaps you may be looking for "git diff ‑‑color‑moved"?  I dunno.

I kind of understand: The 5-line block of olcAccess was moved within a larger
block, preventing diff from seeing the small change within.
I'm attaching the "git diff --color-moved" output as another screenshot (this
time with very wide terminal to make one line a line). Still I redacted
("blued-out") some information I may not publish, but believe me that the parts
"blued out" are identical.

In addition I made a test-case that includes only the first 18 columns (cut
-c1-18) of the file, both before (b0) and after (b1) the commit. That should be
good enough for the reordering effect. A plain "diff -u shows":
# diff -u b0 b1
--- b0  2020-07-14 10:43:05.000000000 +0200
+++ b1  2020-07-14 10:41:37.000000000 +0200
@@ -5,12 +5,6 @@
 olcDatabase: {1}hd
 olcDbDirectory: /v
 olcSuffix: dc=sap,
-olcAccess: {0}to *
-olcAccess: {1}to d
-olcAccess: {2}to a
-olcAccess: {3}to a
-olcAccess: {4}to a
-olcAccess: {5}to *
 olcLimits: {0}dn.e
 olcRootDN: cn=Admi
 olcRootPW: {SSHA}y
@@ -52,6 +46,12 @@
 entryUUID: db3ffe4
 creatorsName: cn=c
 createTimestamp: 2
-entryCSN: 20170706
+olcAccess: {0}to *
+olcAccess: {1}to d
+olcAccess: {2}to a
+olcAccess: {3}to a
+olcAccess: {4}to a
+olcAccess: {5}to *
+entryCSN: 20200114
 modifiersName: cn=
 modifyTimestamp: 2

So it seems the actual problem was that the block had moved, and I did not
notice that when looking at the diffs. An "sdiff" illustrates:
 # sdiff b0 b1

dn: olcDatabase={1                                              dn:
olcDatabase={1
objectClass: olcDa                                              objectClass:
olcDa
objectClass: olcHd                                              objectClass:
olcHd
olcDatabase: {1}hd                                              olcDatabase:
{1}hd
olcDbDirectory: /v                                             
olcDbDirectory: /v
olcSuffix: dc=sap,                                              olcSuffix:
dc=sap,
olcAccess: {0}to *                                            <
olcAccess: {1}to d                                            <
olcAccess: {2}to a                                            <
olcAccess: {3}to a                                            <
olcAccess: {4}to a                                            <
olcAccess: {5}to *                                            <
olcLimits: {0}dn.e                                              olcLimits:
{0}dn.e
olcRootDN: cn=Admi                                              olcRootDN:
cn=Admi
olcRootPW: {SSHA}y                                              olcRootPW:
{SSHA}y
olcSecurity: ssf=1                                              olcSecurity:
ssf=1
olcSyncrepl: {0}ri                                              olcSyncrepl:
{0}ri
olcSyncrepl: {1}ri                                              olcSyncrepl:
{1}ri
olcSyncrepl: {2}ri                                              olcSyncrepl:
{2}ri
olcUpdateRef: ldap                                              olcUpdateRef:
ldap
olcUpdateRef: ldap                                              olcUpdateRef:
ldap
olcUpdateRef: ldap                                              olcUpdateRef:
ldap
olcMirrorMode: TRU                                              olcMirrorMode:
TRU
olcDbCacheSize: 10                                             
olcDbCacheSize: 10
olcDbCheckpoint: 1                                             
olcDbCheckpoint: 1
olcDbConfig: {0}se                                              olcDbConfig:
{0}se
olcDbConfig: {1}se                                              olcDbConfig:
{1}se
olcDbConfig: {2}se                                              olcDbConfig:
{2}se
olcDbConfig: {3}se                                              olcDbConfig:
{3}se
olcDbConfig: {4}se                                              olcDbConfig:
{4}se
olcDbConfig: {5}se                                              olcDbConfig:
{5}se
olcDbIDLcacheSize:                                             
olcDbIDLcacheSize:
olcDbIndex: object                                              olcDbIndex:
object
olcDbIndex: uidNum                                              olcDbIndex:
uidNum
olcDbIndex: gidNum                                              olcDbIndex:
gidNum
olcDbIndex: member                                              olcDbIndex:
member
olcDbIndex: member                                              olcDbIndex:
member
olcDbIndex: cn eq,                                              olcDbIndex: cn
eq,
olcDbIndex: uid eq                                              olcDbIndex:
uid eq
olcDbIndex: sn eq,                                              olcDbIndex: sn
eq,
olcDbIndex: givenN                                              olcDbIndex:
givenN
olcDbIndex: entryU                                              olcDbIndex:
entryU
olcDbIndex: entryC                                              olcDbIndex:
entryC
olcDbIndex: ipServ                                              olcDbIndex:
ipServ
olcDbIndex: ipServ                                              olcDbIndex:
ipServ
olcDbIndex: roleOc                                              olcDbIndex:
roleOc
olcDbIndex: mail e                                              olcDbIndex:
mail e
olcDbIndex: displa                                              olcDbIndex:
displa
olcDbIndex: modify                                              olcDbIndex:
modify
structuralObjectCl                                             
structuralObjectCl
entryUUID: db3ffe4                                              entryUUID:
db3ffe4
creatorsName: cn=c                                              creatorsName:
cn=c
createTimestamp: 2                                             
createTimestamp: 2
entryCSN: 20170706                                            | olcAccess:
{0}to *
                                                              > olcAccess:
{1}to d
                                                              > olcAccess:
{2}to a
                                                              > olcAccess:
{3}to a
                                                              > olcAccess:
{4}to a
                                                              > olcAccess:
{5}to *
                                                              > entryCSN:
20200114
modifiersName: cn=                                              modifiersName:
cn=
modifyTimestamp: 2                                             
modifyTimestamp: 2

Regards,
Ulrich





[-- Attachment #2: diff-git-moved.PNG --]
[-- Type: image/png, Size: 68991 bytes --]

[-- Attachment #3: b0 --]
[-- Type: application/octet-stream, Size: 1065 bytes --]


dn: olcDatabase={1
objectClass: olcDa
objectClass: olcHd
olcDatabase: {1}hd
olcDbDirectory: /v
olcSuffix: dc=sap,
olcAccess: {0}to *
olcAccess: {1}to d
olcAccess: {2}to a
olcAccess: {3}to a
olcAccess: {4}to a
olcAccess: {5}to *
olcLimits: {0}dn.e
olcRootDN: cn=Admi
olcRootPW: {SSHA}y
olcSecurity: ssf=1
olcSyncrepl: {0}ri
olcSyncrepl: {1}ri
olcSyncrepl: {2}ri
olcUpdateRef: ldap
olcUpdateRef: ldap
olcUpdateRef: ldap
olcMirrorMode: TRU
olcDbCacheSize: 10
olcDbCheckpoint: 1
olcDbConfig: {0}se
olcDbConfig: {1}se
olcDbConfig: {2}se
olcDbConfig: {3}se
olcDbConfig: {4}se
olcDbConfig: {5}se
olcDbIDLcacheSize:
olcDbIndex: object
olcDbIndex: uidNum
olcDbIndex: gidNum
olcDbIndex: member
olcDbIndex: member
olcDbIndex: cn eq,
olcDbIndex: uid eq
olcDbIndex: sn eq,
olcDbIndex: givenN
olcDbIndex: entryU
olcDbIndex: entryC
olcDbIndex: ipServ
olcDbIndex: ipServ
olcDbIndex: roleOc
olcDbIndex: mail e
olcDbIndex: displa
olcDbIndex: modify
structuralObjectCl
entryUUID: db3ffe4
creatorsName: cn=c
createTimestamp: 2
entryCSN: 20170706
modifiersName: cn=
modifyTimestamp: 2

[-- Attachment #4: b1 --]
[-- Type: application/octet-stream, Size: 1065 bytes --]


dn: olcDatabase={1
objectClass: olcDa
objectClass: olcHd
olcDatabase: {1}hd
olcDbDirectory: /v
olcSuffix: dc=sap,
olcLimits: {0}dn.e
olcRootDN: cn=Admi
olcRootPW: {SSHA}y
olcSecurity: ssf=1
olcSyncrepl: {0}ri
olcSyncrepl: {1}ri
olcSyncrepl: {2}ri
olcUpdateRef: ldap
olcUpdateRef: ldap
olcUpdateRef: ldap
olcMirrorMode: TRU
olcDbCacheSize: 10
olcDbCheckpoint: 1
olcDbConfig: {0}se
olcDbConfig: {1}se
olcDbConfig: {2}se
olcDbConfig: {3}se
olcDbConfig: {4}se
olcDbConfig: {5}se
olcDbIDLcacheSize:
olcDbIndex: object
olcDbIndex: uidNum
olcDbIndex: gidNum
olcDbIndex: member
olcDbIndex: member
olcDbIndex: cn eq,
olcDbIndex: uid eq
olcDbIndex: sn eq,
olcDbIndex: givenN
olcDbIndex: entryU
olcDbIndex: entryC
olcDbIndex: ipServ
olcDbIndex: ipServ
olcDbIndex: roleOc
olcDbIndex: mail e
olcDbIndex: displa
olcDbIndex: modify
structuralObjectCl
entryUUID: db3ffe4
creatorsName: cn=c
createTimestamp: 2
olcAccess: {0}to *
olcAccess: {1}to d
olcAccess: {2}to a
olcAccess: {3}to a
olcAccess: {4}to a
olcAccess: {5}to *
entryCSN: 20200114
modifiersName: cn=
modifyTimestamp: 2

Re: Antw: [EXT] Re: Help understanding unexpected diff output

[Junio C Hamano]

"Ulrich Windl" <Ulrich.Windl@rz.uni-regensburg.de> writes:

>> I would imagine that the answer would still be a very similar looking
>> 
>>     ‑‑‑ a/0/...
>>     +++ b/0/...
>>     @@ ‑5,12 +5,6 @@
>>      olcDatabase: {1}hdb
>>      olcDbDirectory: /var/lib/ldap
>>      olcSuffix: dc=... 
>>     ‑olcAccess: {0} ...
>>     ‑olcAccess: {1} ...
>>     ‑olcAccess: {2} ...
>>     ‑olcAccess: {3} ...
>>     ‑olcAccess: {4} ...
>>     ‑olcAccess: {5} ...
>>     olcLimits: {0}dn.exact=....
>>      olcRootDn: cn=Admin,dc=...
>>      olcRootPW: {SSHA}yZ...
>>     @@ ‑52,6 +46,12 @@
>>      entryUUID: ...
>>      creatorsName: cn=config
>>      createTimestamp: ...
>>     ‑entryCSN: ...
>>     +olcAccess: {0} ...
>>     +olcAccess: {1} ...
>>     +olcAccess: {2} ...
>>     +olcAccess: {3} ...
>>     +olcAccess: {4} ...
>>     +olcAccess: {5} ...
>>     +entryCSN: ...
>>      modifiersName: cn=config
>>     ‑modifyTimestamp: ...
>>     +modifyTimestamp: ...
>> 
>> with identical "olcAccess: {$n}" lines removed from earlier part the
>> file and added to the end of the file.
>> 
>> So I am not sure what you are asking.  It would not be correct to show
>> something like
>
> According to my eyes (just using Emacs to make sure) those olcAccess lines
> "numbered" 0 to 3, and 5 should be "context lines" as they are not changed.

And the "slightly different" example I asked you about has {4} in
both blocks identical, you are arguing that these 6 lines all should
appear as context?

It certainly is possible but then the patch would look quite
different.  As "olcAccess: {0}" line in the preimage (i.e. a/)
appears at line 8, but that same line appears in the postimage
(i.e. b/) at line 49, such a patch that match these two olcAccess
blocks as unchanged MUST delete many lines that come after
"olcAccess: {5}" in the preimage (i.e. starting at line 14 with
olcLimits, line ending at entryCSN: at line 55 must be removed,
because they used to appear immediately after "olcAccess: {5}" in
the preimage a/, but in the postimage b/, none of tme appear after
the "olcAccess: {5}" that you claim to be common and unchanged (in
the postimage, instead you have only four lines that has"entryCSN:",
"modiferName", etc. before the end of the file).

Of course, these lines in the line range 14-55 actually are the ones
that did not change, as we can see above, so if you insist that you
must keep the 6-line "olcAccess" block as common and unchanged,
because your desired patch is deleting them from the pre-image after
"olcAccess: {5}" line, your desired patch must be adding them back
to the postimage somewhere (namely, before "olcAccess: {0}" line).

Such a patch is also a valid one in that it expresses the difference
between a/ and b/ in terms of a sequence of "delete these lines from
here" and "insert these lines to here", but it would be far less
interesting than what we see above.  Instead of "we deleted 6 lines
near the beginning of the file" plus "then added 6 lines near the
end--ah, these 6 lines by the way are identical", you would instead
say "we deleted the block of lines 14-55 that appear immediately
after olcAccess:{5}" plus "we inserted the identical block of lines
immediately before olcAccess{0}".  And you'd complain in the
opposite way: "These lines are identical but appear in different
locations in the preimage and the postimage -- why aren't they shown
as context lines?"

Re: Antw: [EXT] Re: Help understanding unexpected diff output

[Philip Oakley]

Hi Ulrich,
(only just catching up with intriguing queries)

On 14/07/2020 09:52, Ulrich Windl wrote:
> some lines that are not
> changed were reorderd

It may be me, but I think your comment above is at the core of the
unexpected diff output.

If text is moved, then normally it will (obviously?) require something
within the diff report to say where it was removed from, and then where
it was inserted. It seems like your expectation in your use-case was
that the moved line changes wouldn't be shown.

I don't believe there is an `--ignore-moved-lines` option, but if
there's a good rationale / use-case for it, then maybe it's something to
look at coding.

--
Philip