git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* pkt-line and LF terminated lines of data
@ 2021-04-28 22:22 dwh
  2021-04-29  0:12 ` Ævar Arnfjörð Bjarmason
  2021-04-29  3:16 ` Junio C Hamano
  0 siblings, 2 replies; 4+ messages in thread
From: dwh @ 2021-04-28 22:22 UTC (permalink / raw)
  To: git

I was just reading the Documentation/technical/protocol-common.txt
description of the pkt-line format. One detail that is left out is how a
receiver of pkt-line encoded data determines if a line is binary data or
contains non-binary data.

The documentation says:

> A non-binary line SHOULD BE terminated by an LF, which if present MUST
> be included in the total length. Receivers MUST treat pkt-lines with
> non-binary data the same whether or not they contain the trailing LF
> (stripping the LF if present, and not complaining when it is missing).

It seems like a pkt-line with binary data could easily end with 0x0a
(LF) and a receiver would strip it off even though that is a legitimate
byte in the binary stream. I don't think receivers should be trying to
determine if the pkt-line is binary or non-binary and never strip off
any 0x0a bytes at the end of a pkt-line.

The client code that relies on the pkt-line receiver is where the logic
should reside that figures out what to do with strings that end with LF.
The pkt-line receiver just parses the pkg-line length, reads the correct
number of bytes and passes them along for further processing.

What am I missing? What should be added to this documentation that gives
more detail on when/why/how a pkt-line would be determined to be
non-binary and the LF stripping would occur?

Cheers!
Dave

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: pkt-line and LF terminated lines of data
  2021-04-28 22:22 pkt-line and LF terminated lines of data dwh
@ 2021-04-29  0:12 ` Ævar Arnfjörð Bjarmason
  2021-04-29  2:08   ` brian m. carlson
  2021-04-29  3:16 ` Junio C Hamano
  1 sibling, 1 reply; 4+ messages in thread
From: Ævar Arnfjörð Bjarmason @ 2021-04-29  0:12 UTC (permalink / raw)
  To: dwh; +Cc: git


On Wed, Apr 28 2021, dwh@linuxprogrammer.org wrote:

> I was just reading the Documentation/technical/protocol-common.txt
> description of the pkt-line format. One detail that is left out is how a
> receiver of pkt-line encoded data determines if a line is binary data or
> contains non-binary data.

They don't. The "is it binary" is a client convention / awareness of the
consumed payload.

> The documentation says:
>
>> A non-binary line SHOULD BE terminated by an LF, which if present MUST
>> be included in the total length. Receivers MUST treat pkt-lines with
>> non-binary data the same whether or not they contain the trailing LF
>> (stripping the LF if present, and not complaining when it is missing).
>
> It seems like a pkt-line with binary data could easily end with 0x0a
> (LF) and a receiver would strip it off even though that is a legitimate
> byte in the binary stream. I don't think receivers should be trying to
> determine if the pkt-line is binary or non-binary and never strip off
> any 0x0a bytes at the end of a pkt-line.
>
> The client code that relies on the pkt-line receiver is where the logic
> should reside that figures out what to do with strings that end with LF.
> The pkt-line receiver just parses the pkg-line length, reads the correct
> number of bytes and passes them along for further processing.
>
> What am I missing? What should be added to this documentation that gives
> more detail on when/why/how a pkt-line would be determined to be
> non-binary and the LF stripping would occur?

My reading of the quoted documentation is that it already matches what
you're suggesting it should say.

I.e. it doesn't anything about how binary data is handled, what it does
say is that if a client knows that a line is non-binary it should be
treating "foo" and "foo\n" the same, i.e. being lenient and interpret
both as "foo".

See PACKET_READ_CHOMP_NEWLINE in pkt-line.c, and the entirety of the
commit that introduced the blurb you're quoting: 1c9b659d983
(pack-protocol: clarify LF-handling in PKT-LINE(), 2015-09-03).

If you tweak the pkt-line.c code to do PACKET_READ_CHOMP_NEWLINE
unconditionally you'll get a lot of test failures, digging into those is
a good starting point to see how the binary v.s. non-binary cases are
handled.

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: pkt-line and LF terminated lines of data
  2021-04-29  0:12 ` Ævar Arnfjörð Bjarmason
@ 2021-04-29  2:08   ` brian m. carlson
  0 siblings, 0 replies; 4+ messages in thread
From: brian m. carlson @ 2021-04-29  2:08 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: dwh, git

[-- Attachment #1: Type: text/plain, Size: 1120 bytes --]

On 2021-04-29 at 00:12:00, Ævar Arnfjörð Bjarmason wrote:
> 
> On Wed, Apr 28 2021, dwh@linuxprogrammer.org wrote:
> 
> > I was just reading the Documentation/technical/protocol-common.txt
> > description of the pkt-line format. One detail that is left out is how a
> > receiver of pkt-line encoded data determines if a line is binary data or
> > contains non-binary data.
> 
> They don't. The "is it binary" is a client convention / awareness of the
> consumed payload.

As a general rule of thumb, if the pkt-line is a pack or object, it's
binary, and otherwise (a ref advertisement, a negotiation, etc.) it's
text.  Almost any case where GIT_TRACE_PACKET prints the output is going
to be text, and almost any case where it does not is going to be binary.

Note that whether something is text or binary is specifically an
attribute of what part of the protocol is being parsed, not what bytes
it contains.  For example, a ref advertisement with a ref containing the
bytes FE and FF is text, despite those not appearing in UTF-8.
-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: pkt-line and LF terminated lines of data
  2021-04-28 22:22 pkt-line and LF terminated lines of data dwh
  2021-04-29  0:12 ` Ævar Arnfjörð Bjarmason
@ 2021-04-29  3:16 ` Junio C Hamano
  1 sibling, 0 replies; 4+ messages in thread
From: Junio C Hamano @ 2021-04-29  3:16 UTC (permalink / raw)
  To: dwh; +Cc: git

dwh@linuxprogrammer.org writes:

> What am I missing? What should be added to this documentation that gives
> more detail on when/why/how a pkt-line would be determined to be
> non-binary and the LF stripping would occur?

There is (unfortunately) none.  What payload each packet carries is
determined by the protocol, so a man-in-the-middle who is observing
a protocol exchange without any knowledge of the protocol that runs
on top of the pkt-line layer by definition would not be able to tell
which packet is binary (as it is unaware of what exchange is made).


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2021-04-29  3:16 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-04-28 22:22 pkt-line and LF terminated lines of data dwh
2021-04-29  0:12 ` Ævar Arnfjörð Bjarmason
2021-04-29  2:08   ` brian m. carlson
2021-04-29  3:16 ` Junio C Hamano

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).