[PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[Rob Browning]

Add a space to the blank line after the version header in the example
gpgsig commit header.

Signed-off-by: Rob Browning <rlb@defaultvalue.org>
---

 While investigating this (while parsing commit objects) I also
 noticed a commit in a repo that had a blank continuation line (" \n")
 after the "END PGP SIGNATURE" line.

 Presuming that's valid, I suppose we could consider detailing any
 additional specifications, i.e. what kind of trailing content a
 parser should expect/ignore, etc.

 Documentation/technical/signature-format.txt | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/technical/signature-format.txt b/Documentation/technical/signature-format.txt
index 2c9406a56a..6acc0b1247 100644
--- a/Documentation/technical/signature-format.txt
+++ b/Documentation/technical/signature-format.txt
@@ -78,7 +78,7 @@ author A U Thor <author@example.com> 1465981137 +0000
 committer C O Mitter <committer@example.com> 1465981137 +0000
 gpgsig -----BEGIN PGP SIGNATURE-----
  Version: GnuPG v1
-
+ 
  iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
  HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
  DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
-- 
2.30.2

Re: [PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[Jeff King]

On Sat, Oct 09, 2021 at 11:33:38AM -0500, Rob Browning wrote:

> Add a space to the blank line after the version header in the example
> gpgsig commit header.

Thanks, this is a good catch. The space is important for the header
continuation.

> diff --git a/Documentation/technical/signature-format.txt b/Documentation/technical/signature-format.txt
> index 2c9406a56a..6acc0b1247 100644
> --- a/Documentation/technical/signature-format.txt
> +++ b/Documentation/technical/signature-format.txt
> @@ -78,7 +78,7 @@ author A U Thor <author@example.com> 1465981137 +0000
>  committer C O Mitter <committer@example.com> 1465981137 +0000
>  gpgsig -----BEGIN PGP SIGNATURE-----
>   Version: GnuPG v1
> -
> + 
>   iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
>   HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
>   DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA

The patch is quite subtle to read, of course. :) But more importantly,
it is subtle for somebody reading the documentation to notice. Perhaps
it's worth calling it out explicitly? E.g., squashing in something like:

diff --git a/Documentation/technical/signature-format.txt b/Documentation/technical/signature-format.txt
index 6acc0b1247..f418431050 100644
--- a/Documentation/technical/signature-format.txt
+++ b/Documentation/technical/signature-format.txt
@@ -68,7 +68,9 @@ signed tag message body
 - created by: `git commit -S`
 - payload: commit object
 - embedding: header entry `gpgsig`
-  (content is preceded by a space)
+  (content is preceded by a space; note that this includes the
+   "empty" line between the GnuPG header and signature, which
+   consists of a single space).
 - example: commit with subject `signed commit`
 
 ----

>  While investigating this (while parsing commit objects) I also
>  noticed a commit in a repo that had a blank continuation line (" \n")
>  after the "END PGP SIGNATURE" line.
> 
>  Presuming that's valid, I suppose we could consider detailing any
>  additional specifications, i.e. what kind of trailing content a
>  parser should expect/ignore, etc.

It is valid. It looks like GitHub's signed-merges may do this. From our
perspective, we are not really defining what is valid content or not. We
take everything in the gpgsig header (including multiple lines connected
by space-continuation), and feed it to gpg. So if gpg is happy with it,
we are happy with it.

Thinking with a security hat on, it's possible this could lead to
confusion (say, if I include multiple signatures but some tools check
one and some the other). But I'd be hesitant to start adding
restrictions without even a plausible attack scenario.

-Peff

Re: [PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> On Sat, Oct 09, 2021 at 11:33:38AM -0500, Rob Browning wrote:
>
>> Add a space to the blank line after the version header in the example
>> gpgsig commit header.
>
> Thanks, this is a good catch. The space is important for the header
> continuation.
>
>> diff --git a/Documentation/technical/signature-format.txt b/Documentation/technical/signature-format.txt
>> index 2c9406a56a..6acc0b1247 100644
>> --- a/Documentation/technical/signature-format.txt
>> +++ b/Documentation/technical/signature-format.txt
>> @@ -78,7 +78,7 @@ author A U Thor <author@example.com> 1465981137 +0000
>>  committer C O Mitter <committer@example.com> 1465981137 +0000
>>  gpgsig -----BEGIN PGP SIGNATURE-----
>>   Version: GnuPG v1
>> -
>> + 
>>   iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
>>   HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
>>   DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
>
> The patch is quite subtle to read, of course. :) But more importantly,
> it is subtle for somebody reading the documentation to notice. Perhaps
> it's worth calling it out explicitly? E.g., squashing in something like:
> ...
> -  (content is preceded by a space)
> +  (content is preceded by a space; note that this includes the
> +   "empty" line between the GnuPG header and signature, which
> +   consists of a single space).
>  - example: commit with subject `signed commit`

To those who are reading on paper or on terminal, the difference
will not be even seen.  It only can be _found_ if you are in an
editor or a pager and explicitly look for a trailing whitespace (or
told your tool to highlight such for you).

I wonder if we can have some typesetting convention for this part of
the documentation.  Perhaps something like

    In the following example, the end of line that ends with a
    whitespace letter is highlighted with a "$" sign; if you are
    trying to recreate these example by hand, do not cut and paste
    them---they are there primarily to highlight extra whitespace at
    the end of some lines.

before a displayed material like this:

  committer C O Mitter <committer@example.com> 1465981137 +0000
  gpgsig -----BEGIN PGP SIGNATURE-----
   Version: GnuPG v1
   $
   iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
   HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
   DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA

perhaps?

I am reasonably sure that running "git am" on Rob's patch will by
default end up as a no-op patch, because auto-correcting whitespace
breakage is the default setting I use, and a change like the one
suggested above would help avoid such a problem, too.

Thanks.

Re: [PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[Jeff King]

On Mon, Oct 11, 2021 at 01:04:48PM -0700, Junio C Hamano wrote:

> > The patch is quite subtle to read, of course. :) But more importantly,
> > it is subtle for somebody reading the documentation to notice. Perhaps
> > it's worth calling it out explicitly? E.g., squashing in something like:
> > ...
> > -  (content is preceded by a space)
> > +  (content is preceded by a space; note that this includes the
> > +   "empty" line between the GnuPG header and signature, which
> > +   consists of a single space).
> >  - example: commit with subject `signed commit`
> 
> To those who are reading on paper or on terminal, the difference
> will not be even seen.  It only can be _found_ if you are in an
> editor or a pager and explicitly look for a trailing whitespace (or
> told your tool to highlight such for you).

Yeah, I was hoping that calling it out explicitly would help to serve
that purpose, even if they can't see it. But it's still pretty subtle.

I did consider annotating it more directly in the example, but I was
worried it would end up being syntactically confusing, because we don't
have a well-known convention for marking "end of line". But the example
you showed:

> I wonder if we can have some typesetting convention for this part of
> the documentation.  Perhaps something like
> 
>     In the following example, the end of line that ends with a
>     whitespace letter is highlighted with a "$" sign; if you are
>     trying to recreate these example by hand, do not cut and paste
>     them---they are there primarily to highlight extra whitespace at
>     the end of some lines.
> 
> before a displayed material like this:
> 
>   committer C O Mitter <committer@example.com> 1465981137 +0000
>   gpgsig -----BEGIN PGP SIGNATURE-----
>    Version: GnuPG v1
>    $
>    iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
>    HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
>    DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
> 
> perhaps?

...coupled with the explanation you gave is not too bad. I had thought
to maybe do something like:

   gpgsig -----BEGIN PGP SIGNATURE-----
   _Version: GnuPG v1
   _
   _iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
   _HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
   _DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA

but that is pretty ugly. I like yours much better.

-Peff

Re: [PATCH 1/1] signature-format.txt: add space to fix gpgsig continuation line

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>>     In the following example, the end of line that ends with a
>>     whitespace letter is highlighted with a "$" sign; if you are
>>     trying to recreate these example by hand, do not cut and paste
>>     them---they are there primarily to highlight extra whitespace at
>>     the end of some lines.
>> 
>> before a displayed material like this:
>> 
>>   committer C O Mitter <committer@example.com> 1465981137 +0000
>>   gpgsig -----BEGIN PGP SIGNATURE-----
>>    Version: GnuPG v1
>>    $
>>    iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
>>    HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
>>    DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
>> 
>> perhaps?
>
> ...coupled with the explanation you gave is not too bad. I had thought
> to maybe do something like:
>
>    gpgsig -----BEGIN PGP SIGNATURE-----
>    _Version: GnuPG v1
>    _
>    _iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
>    _HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
>    _DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
>
> but that is pretty ugly. I like yours much better.

Yes, I did consider a replacement for SP and rejected for the same
reason as you did.  We could use the same "we only show the 'here is
a SP' sign when the presence of the SP cannot be seen" convention to
make it less distracting, but I think '$' taken from "cat -e"
probably is easier to see and understand for those who are the
target audience of anything in Documentation/technical/ hierarchy.

Thanks.

[PATCH] signature-format.txt: explain and illustrate multi-line headers

[Junio C Hamano]

A signature attached to a signed commit, and the contents of the
commit that merged a signed tag, are both recorded as a value of an
object header field as a multi-line value, and are subject to the
formatting convention for multi-line values in the headers, with a
leading SP signaling that the rest of the line is a continuation of
the previous line.  Most notably, an empty line in such a multi-line
value would result in a line with a sole SP on it.

Examples in the signature-format technical documentation include a
few of these cases but we did not show these otherwise invisible SPs
in the example.  These trailing spaces cannot be seen on display or
on paper, and forces the readers to look for them in their editors
or pagers, even if we added them to the document.

Extend the overview section to explain the multi-line value
formatting and highlight these otherwise invisible SPs by inventing
the "a dollar-sign at the end of line that appears after SP merely
signals that there is a SP there, and the dollar-sign itself does
not appear in the real file" notation, inspired by "cat -e" output,
to help readers to learn exactly where such "a single SP that is
originally an empty line" appears in the examples.

Reported-by: Rob Browning <rlb@defaultvalue.org>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---

    It turns out that the document has another block of text that
    needed the same treatment, so here is an attempt to follow
    through with the "$"-notation approach.

 Documentation/technical/signature-format.txt | 25 ++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/Documentation/technical/signature-format.txt b/Documentation/technical/signature-format.txt
index 2c9406a56a..45448fba24 100644
--- a/Documentation/technical/signature-format.txt
+++ b/Documentation/technical/signature-format.txt
@@ -13,9 +13,25 @@ Signatures always begin with `-----BEGIN PGP SIGNATURE-----`
 and end with `-----END PGP SIGNATURE-----`, unless gpg is told to
 produce RFC1991 signatures which use `MESSAGE` instead of `SIGNATURE`.
 
+Signatures sometimes appear as a part of the normal payload
+(e.g. a signed tag has the signature block appended after the payload
+that the signature applies to), and sometimes appear in the value of
+an object header (e.g. a merge commit that merged a signed tag would
+have the entire tag contents on its "mergetag" header).  In the case
+of the latter, the usual multi-line formatting rule for object
+headers applies.  I.e. the second and subsequent lines are prefixed
+with a SP to signal that the line is continued from the previous
+line.
+
+This is even true for an originally empty line.  In the following
+examples, the end of line that ends with a whitespace letter is
+highlighted with a `$` sign; if you are trying to recreate these
+example by hand, do not cut and paste them---they are there
+primarily to highlight extra whitespace at the end of some lines.
+
 The signed payload and the way the signature is embedded depends
 on the type of the object resp. transaction.
 
 == Tag signatures
 
 - created by: `git tag -s`
@@ -78,7 +95,7 @@ author A U Thor <author@example.com> 1465981137 +0000
 committer C O Mitter <committer@example.com> 1465981137 +0000
 gpgsig -----BEGIN PGP SIGNATURE-----
  Version: GnuPG v1
-
+ $
  iQEcBAABAgAGBQJXYRjRAAoJEGEJLoW3InGJ3IwIAIY4SA6GxY3BjL60YyvsJPh/
  HRCJwH+w7wt3Yc/9/bW2F+gF72kdHOOs2jfv+OZhq0q4OAN6fvVSczISY/82LpS7
  DVdMQj2/YcHDT4xrDNBnXnviDO9G7am/9OE77kEbXrp7QPxvhjkicHNwy2rEflAA
@@ -128,13 +145,13 @@ mergetag object 04b871796dc0420f8e7561a895b52484b701d51a
  type commit
  tag signedtag
  tagger C O Mitter <committer@example.com> 1465981006 +0000
-
+ $
  signed tag
-
+ $
  signed tag message body
  -----BEGIN PGP SIGNATURE-----
  Version: GnuPG v1
-
+ $
  iQEcBAABAgAGBQJXYRhOAAoJEGEJLoW3InGJklkIAIcnhL7RwEb/+QeX9enkXhxn
  rxfdqrvWd1K80sl2TOt8Bg/NYwrUBw/RWJ+sg/hhHp4WtvE1HDGHlkEz3y11Lkuh
  8tSxS3qKTxXUGozyPGuE90sJfExhZlW4knIQ1wt/yWqM+33E9pN4hzPqLwyrdods
-- 
2.33.1-816-gd427230396

Re: [PATCH] signature-format.txt: explain and illustrate multi-line headers

[Rob Browning]

Junio C Hamano <gitster@pobox.com> writes:

> A signature attached to a signed commit, and the contents of the
> commit that merged a signed tag, are both recorded as a value of an
> object header field as a multi-line value, and are subject to the
> formatting convention for multi-line values in the headers, with a
> leading SP signaling that the rest of the line is a continuation of
> the previous line.  Most notably, an empty line in such a multi-line
> value would result in a line with a sole SP on it.

One question I had was whether git's requirement was strictly a space,
or if it was following the rfc-822 convention where (if I remember
correctly) a tab is equivalent, i.e. the LWSP production in the grammar.

https://datatracker.ietf.org/doc/html/rfc822#section-3.2

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: [PATCH] signature-format.txt: explain and illustrate multi-line headers

[Junio C Hamano]

Rob Browning <rlb@defaultvalue.org> writes:

> One question I had was whether git's requirement was strictly a space,
> or if it was following the rfc-822 convention where (if I remember
> correctly) a tab is equivalent, i.e. the LWSP production in the grammar.

We use a single SP when writing and we accept a single SP when
reading.  See commit.c::add_extra_header() for the writing side, and
commit.c::read_commit_extra_header_lines() for the reading side.

Unlike in RFC-822 style e-mail headers, 

 * keywords in our object headers are not followed by a colon;

 * the value carried on our object header is not a "logically a
   single line of characters".  

 * our headers do not go through their "unfolding" (i.e. removal of
   CRLF eol markers to form a single long line, while preserving the
   WSP that immediately followed the CRLF).  We instead remove the
   SP that signalled the line as a continuation of the previous line
   to keep the original line structure.

With so little similarity, there is no reason for us to mimick their
"folding" rule.

We limit to the SP and not LWP for another reason.  Because the
exact byte sequence in the object (including the header part of
"commit" and "tag" objects) determines the name of the object, it
avoids ambiguity to make sure we do not allow unnecessary
"flexibility" in the way the same thing can be expressed.  If the
same signature is attached to a pair of otherwise identical commits
in their headers, one with SP signaling continued lines, the other
using HT (or random permutations of choice between SP/HT---making
2^N variants for a N line signature block), we would needlessly
create many variants of the "same" commit, which is not ideal.

Re: [PATCH] signature-format.txt: explain and illustrate multi-line headers

[Rob Browning]

Junio C Hamano <gitster@pobox.com> writes:

> With so little similarity, there is no reason for us to mimick their
> "folding" rule.

Agreed.  I'd just lazily guessed it might be 822 (was also thinking of
Debian package headers), but certainly shouldn't have glossed over the
missing colon (for example) -- might be worth making sure the rules
described are covered in the technical docs, if they're not already, and
then perhaps refer to them in the section we're adjusting.

> We limit to the SP and not LWP for another reason.  Because the
> exact byte sequence in the object (including the header part of
> "commit" and "tag" objects) determines the name of the object

Ahh, right, of course.

> we would needlessly create many variants of the "same" commit, which
> is not ideal.

Indeed.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: [PATCH] signature-format.txt: explain and illustrate multi-line headers

[Junio C Hamano]

Rob Browning <rlb@defaultvalue.org> writes:

> Agreed.  I'd just lazily guessed it might be 822 (was also thinking of
> ...
> Ahh, right, of course.
> ...
> Indeed.
>
> Thanks

So, back to the original discussion; would the replacement
documentation update be satisfactory?

Re: [PATCH] signature-format.txt: explain and illustrate multi-line headers

[Rob Browning]

Junio C Hamano <gitster@pobox.com> writes:

> Rob Browning <rlb@defaultvalue.org> writes:
>
>> Agreed.  I'd just lazily guessed it might be 822 (was also thinking of
>> ...
>> Ahh, right, of course.
>> ...
>> Indeed.
>>
>> Thanks
>
> So, back to the original discussion; would the replacement
> documentation update be satisfactory?

Certainly, and thanks.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4