[PATCH 0/3] hash-object doc: small fixes

[PATCH 0/3] hash-object doc: small fixes

[Ævar Arnfjörð Bjarmason]

Small doc fixes. Maybe trivial enough to land in 2.22, but there's no
rush.

A pair of #leftoverbits I noticed is that we've implemented the
"--stdin-paths" option via unquote_c_style() from day one, so our
current docs lie (and still do with this series) about wanting
\n-delimited files, you can't hash a file called '"foo"' as you'd
expect, you need to pass '"\"foo\""'.

I wonder if we should document this at this point, or just change it
and add a "-z" option. None of our tests fail if I remove this
unquote_c_style() codepath, and it's never been documented, but
someone in the wild may have organically depended on it.

Ævar Arnfjörð Bjarmason (3):
  hash-object doc: stop mentioning git-cvsimport
  hash-object doc: elaborate on -w and --literally promises
  hash-object doc: point to ls-files and rev-parse

 Documentation/git-hash-object.txt | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

-- 
2.21.0.1020.gf2820cf01a

[PATCH 1/3] hash-object doc: stop mentioning git-cvsimport

[Ævar Arnfjörð Bjarmason]

Remove a reference to git-cvsimport in the intro. As can be seen from
the history of this command[1] it was originally intended for use with
git-cvsimport, but how it uses it (and that it uses it at all) is
irrelevant trivia at this point.

1. See 7672db20c2 ("[PATCH] Expose object ID computation functions.",
   2005-07-08) and 8b8840e046 ("[PATCH] cvsgit fixes: spaces in
   filenames and CVS server dialog woes", 2005-08-15).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/git-hash-object.txt | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt
index 814e74406a..df9e2c58bd 100644
--- a/Documentation/git-hash-object.txt
+++ b/Documentation/git-hash-object.txt
@@ -18,9 +18,7 @@ Computes the object ID value for an object with specified type
 with the contents of the named file (which can be outside of the
 work tree), and optionally writes the resulting object into the
 object database.  Reports its object ID to its standard output.
-This is used by 'git cvsimport' to update the index
-without modifying files in the work tree.  When <type> is not
-specified, it defaults to "blob".
+When <type> is not specified, it defaults to "blob".
 
 OPTIONS
 -------
-- 
2.21.0.1020.gf2820cf01a

[PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Ævar Arnfjörð Bjarmason]

Clarify the hash-object docs to explicitly note that the --literally
option guarantees that a loose object will be written, but that a
normal -w ("write") invocation doesn't.

At first I thought talking about "loose object" in the docs was a
mistake in 83115ac4a8 ("git-hash-object.txt: document --literally
option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object:
add --literally option", 2014-09-11) this was intended all along.

As seen in 4dd1fbc7b1 ("Bigfile: teach "git add" to send a large file
straight to a pack", 2011-05-08) we'll otherwise not guarantee that we
write loose objects, so let's explicitly note that, using vague enough
language to leave the door open to any arbitrary future storage
format, not just loose objects and packs.

While I'm at it remove the mention of "--stdin" from the "--literally"
documentation. This wasn't correct, the "--literally" option combines
with any other option (e.g. "--stdin-paths"), not just "--stdin".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/git-hash-object.txt | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt
index df9e2c58bd..100630d021 100644
--- a/Documentation/git-hash-object.txt
+++ b/Documentation/git-hash-object.txt
@@ -28,6 +28,12 @@ OPTIONS
 
 -w::
 	Actually write the object into the object database.
++
+No guarantees are made as to how the object is added to the database
+(e.g. whether as a loose object, or streamed to a pack), except in the
+case of the `--literally` option. How it's written may depend on
+heuristics such as the `core.bigFileThreshold` configuration variable
+(see linkgit:git-config[1]).
 
 --stdin::
 	Read the object from standard input instead of from a file.
@@ -53,10 +59,11 @@ OPTIONS
 	is always implied, unless the `--path` option is given.
 
 --literally::
-	Allow `--stdin` to hash any garbage into a loose object which might not
+	Allow for hashing arbitrary data which might not
 	otherwise pass standard object parsing or git-fsck checks. Useful for
 	stress-testing Git itself or reproducing characteristics of corrupt or
-	bogus objects encountered in the wild.
+	bogus objects encountered in the wild. When writing objects guarantees
+	that the written object will be a loose object, for ease of debugging.
 
 GIT
 ---
-- 
2.21.0.1020.gf2820cf01a

[PATCH 3/3] hash-object doc: point to ls-files and rev-parse

[Ævar Arnfjörð Bjarmason]

Amend the intro to note that it's better to ask the index about files
already tracked by it.

I've seen uses of this in the wild where the use-case was finding
object IDs for files found in a freshly cloned repo, i.e. something
like "git ls-files | git hash-object --stdin-paths".

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/git-hash-object.txt | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/Documentation/git-hash-object.txt b/Documentation/git-hash-object.txt
index 100630d021..077d83ec65 100644
--- a/Documentation/git-hash-object.txt
+++ b/Documentation/git-hash-object.txt
@@ -20,6 +20,13 @@ work tree), and optionally writes the resulting object into the
 object database.  Reports its object ID to its standard output.
 When <type> is not specified, it defaults to "blob".
 
+This command won't take the shortcut of seeing if the path(s) to be
+hashed are present in the index, in which case their already computed
+object ID could be retrieved by linkgit:git-ls-files[1] via the
+`--stage` option, or by linkgit:git-rev-parse[1]. Use those instead of
+e.g. `--stdin-paths` if the intent is to say find the blob object IDs
+of checked-out files.
+
 OPTIONS
 -------
 
-- 
2.21.0.1020.gf2820cf01a

Re: [PATCH 1/3] hash-object doc: stop mentioning git-cvsimport

[Jeff King]

On Mon, May 20, 2019 at 11:53:10PM +0200, Ævar Arnfjörð Bjarmason wrote:

> Remove a reference to git-cvsimport in the intro. As can be seen from
> the history of this command[1] it was originally intended for use with
> git-cvsimport, but how it uses it (and that it uses it at all) is
> irrelevant trivia at this point.

Yeah, this is definitely obscuring the main point for most readers. The
patch looks good.

-Peff

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Jeff King]

On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote:

> Clarify the hash-object docs to explicitly note that the --literally
> option guarantees that a loose object will be written, but that a
> normal -w ("write") invocation doesn't.

I had to double-check here: you mean that _when_ we are writing an
object, "--literally" would always write loose, right?

> At first I thought talking about "loose object" in the docs was a
> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally
> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object:
> add --literally option", 2014-09-11) this was intended all along.

Hmm. After reading both of those, I do think it's mostly an
implementation detail. I would not be at all surprised to find that the
test suite relies on this (e.g., cleaning up with rm
.git/objects/ab/cd1234). But I suspect we also rely on that for the
non-literal case, too. ;)

So I am on the fence. In some sense it doesn't hurt to document the
behavior, but I'm not sure I would want to lock us in to any particular
behavior, even for --literally. The intent of the option (as I recall)
really is just "let us write whatever trash we want as an object,
ignoring all quality checks".

>  --literally::
> -	Allow `--stdin` to hash any garbage into a loose object which might not
> +	Allow for hashing arbitrary data which might not
>  	otherwise pass standard object parsing or git-fsck checks. Useful for
>  	stress-testing Git itself or reproducing characteristics of corrupt or
> -	bogus objects encountered in the wild.
> +	bogus objects encountered in the wild. When writing objects guarantees
> +	that the written object will be a loose object, for ease of debugging.

I had to read this last sentence a few times to parse it. Maybe a comma
before guarantees would help? Or even:

  When writing objects, this option guarantees that the written object
  will be a loose object, for ease of debugging.

-Peff

Re: [PATCH 3/3] hash-object doc: point to ls-files and rev-parse

[Jeff King]

On Mon, May 20, 2019 at 11:53:12PM +0200, Ævar Arnfjörð Bjarmason wrote:

> Amend the intro to note that it's better to ask the index about files
> already tracked by it.
> 
> I've seen uses of this in the wild where the use-case was finding
> object IDs for files found in a freshly cloned repo, i.e. something
> like "git ls-files | git hash-object --stdin-paths".

This example left me scratching my head, because it seems like such a
bad way to accomplish the goal that I had trouble figuring out what the
goal was in the first place.

Of course I am pretty familiar with Git, so my mindset is not
necessarily the same as the average reader of the manpage. But the
suggested documentation really seems to me to be warning about a
convoluted use case that most people would not come up with in the first
place.

And adding it in the introductory text means that people are more likely
to read and be confused by it, when they are really just looking to find
details on more mundane usage.

Or put another way: I could come up with a hundred bad ways to script
Git, but we are probably better off enumerating the _good_ ways to do
so, rather than listing the bad ways and leaving the user to figure out
what's left.

So I'm mildly negative on this patch.

-Peff

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Jakub Narebski]

Jeff King <peff@peff.net> writes:
> On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> Clarify the hash-object docs to explicitly note that the --literally
>> option guarantees that a loose object will be written, but that a
>> normal -w ("write") invocation doesn't.
>
> I had to double-check here: you mean that _when_ we are writing an
> object, "--literally" would always write loose, right?
>
>> At first I thought talking about "loose object" in the docs was a
>> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally
>> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object:
>> add --literally option", 2014-09-11) this was intended all along.
>
> Hmm. After reading both of those, I do think it's mostly an
> implementation detail. I would not be at all surprised to find that the
> test suite relies on this (e.g., cleaning up with rm
> .git/objects/ab/cd1234). But I suspect we also rely on that for the
> non-literal case, too. ;)
>
> So I am on the fence. In some sense it doesn't hurt to document the
> behavior, but I'm not sure I would want to lock us in to any particular
> behavior, even for --literally. The intent of the option (as I recall)
> really is just "let us write whatever trash we want as an object,
> ignoring all quality checks".

I thik that this implemetation detail of `--literally` is here to stay;
how would you otherwise fix the issue if garbage object makes Git crash?

However, I would prefer to have options state _intent_; if there is
legitimate need for a tool that creates loose objects, it would be
better to have separate `--loose` option to `git hash-object` (which
would imply `-w`, otherwise it doesn't have sense).

>>  --literally::
>> -	Allow `--stdin` to hash any garbage into a loose object which might not
>> +	Allow for hashing arbitrary data which might not
>>  	otherwise pass standard object parsing or git-fsck checks. Useful for
>>  	stress-testing Git itself or reproducing characteristics of corrupt or
>> -	bogus objects encountered in the wild.
>> +	bogus objects encountered in the wild. When writing objects guarantees
>> +	that the written object will be a loose object, for ease of debugging.
>
> I had to read this last sentence a few times to parse it. Maybe a comma
> before guarantees would help? Or even:
>
>   When writing objects, this option guarantees that the written object
>   will be a loose object, for ease of debugging.

I agree that this reads better.

Regards,
--
Jakub Narębski

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Ævar Arnfjörð Bjarmason]

On Fri, May 24 2019, Jakub Narebski wrote:

> Jeff King <peff@peff.net> writes:
>> On Mon, May 20, 2019 at 11:53:11PM +0200, Ævar Arnfjörð Bjarmason wrote:
>>
>>> Clarify the hash-object docs to explicitly note that the --literally
>>> option guarantees that a loose object will be written, but that a
>>> normal -w ("write") invocation doesn't.
>>
>> I had to double-check here: you mean that _when_ we are writing an
>> object, "--literally" would always write loose, right?
>>
>>> At first I thought talking about "loose object" in the docs was a
>>> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally
>>> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object:
>>> add --literally option", 2014-09-11) this was intended all along.
>>
>> Hmm. After reading both of those, I do think it's mostly an
>> implementation detail. I would not be at all surprised to find that the
>> test suite relies on this (e.g., cleaning up with rm
>> .git/objects/ab/cd1234). But I suspect we also rely on that for the
>> non-literal case, too. ;)
>>
>> So I am on the fence. In some sense it doesn't hurt to document the
>> behavior, but I'm not sure I would want to lock us in to any particular
>> behavior, even for --literally. The intent of the option (as I recall)
>> really is just "let us write whatever trash we want as an object,
>> ignoring all quality checks".
>
> I thik that this implemetation detail of `--literally` is here to stay;
> how would you otherwise fix the issue if garbage object makes Git crash?
>
> However, I would prefer to have options state _intent_; if there is
> legitimate need for a tool that creates loose objects, it would be
> better to have separate `--loose` option to `git hash-object` (which
> would imply `-w`, otherwise it doesn't have sense).

I wonder if we can just remove this option and replace it with a
GIT_TEST_* env variable, or even a test-tool helper. I can't see why
anyone other than our own test suite wants this, and that's why it was
added. So why document it & expose it in a plumbing tool?

>>>  --literally::
>>> -	Allow `--stdin` to hash any garbage into a loose object which might not
>>> +	Allow for hashing arbitrary data which might not
>>>  	otherwise pass standard object parsing or git-fsck checks. Useful for
>>>  	stress-testing Git itself or reproducing characteristics of corrupt or
>>> -	bogus objects encountered in the wild.
>>> +	bogus objects encountered in the wild. When writing objects guarantees
>>> +	that the written object will be a loose object, for ease of debugging.
>>
>> I had to read this last sentence a few times to parse it. Maybe a comma
>> before guarantees would help? Or even:
>>
>>   When writing objects, this option guarantees that the written object
>>   will be a loose object, for ease of debugging.
>
> I agree that this reads better.
>
> Regards,

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Jeff King]

On Fri, May 24, 2019 at 12:12:10PM +0200, Ævar Arnfjörð Bjarmason wrote:

> > I thik that this implemetation detail of `--literally` is here to stay;
> > how would you otherwise fix the issue if garbage object makes Git crash?
> >
> > However, I would prefer to have options state _intent_; if there is
> > legitimate need for a tool that creates loose objects, it would be
> > better to have separate `--loose` option to `git hash-object` (which
> > would imply `-w`, otherwise it doesn't have sense).
> 
> I wonder if we can just remove this option and replace it with a
> GIT_TEST_* env variable, or even a test-tool helper. I can't see why
> anyone other than our own test suite wants this, and that's why it was
> added. So why document it & expose it in a plumbing tool?

I can think of a few reasons you might want it in the general toolbox:

  - you could be recreating a known-buggy history and want to overcome
    consistency checks (e.g., imagine a tool that imports or modifies
    git history, and needs to recreate objects literally, warts and
    all). I think we don't do a lot of quality checks in hash-object
    now, but we have discussed it (and --literally would be the obvious
    escape hatch).

  - folks like security researchers who are poking at Git and want to
    see how it reacts to various almost-correct inputs. They _could_ use
    the test-tool helper, but being able to demonstrate bugs with the
    standard toolbox is helpful for communication.

  - likewise for general non-security debugging ("it behaves in this
    funny way if I violate this constraint...")

So I actually think it's nice to have it as part of the plumbing, as
long as it's buried and documented sufficiently that unsuspecting users
do not stumble onto it.

And at any rate, now that it has been in the wild for a while, we risk
breaking somebody who has come to depend on it.

-Peff

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> Clarify the hash-object docs to explicitly note that the --literally
> option guarantees that a loose object will be written, but that a
> normal -w ("write") invocation doesn't.
>
> At first I thought talking about "loose object" in the docs was a
> mistake in 83115ac4a8 ("git-hash-object.txt: document --literally
> option", 2015-05-04), but as is clear from 5ba9a93b39 ("hash-object:
> add --literally option", 2014-09-11) this was intended all along.

I have to admit that this "loose only" was the doing of my
defeatism.  IOW, I was utterly pessimistic that I would be able to
add more types (and more importantly, unbounded number of random
types) of objects in the packstream.

So, "loose object" limitation is a practical one for those of us who
cannot think of a reasonable way to cram arbitrary number of random
new types into just 3 bits of the "type" bitfield, and not inherent
to the "hash-object --literally" command.

So I am very happy to see the first hunk of this patch, but I doubt
there is much value in the last sentence the second hunk adds.

Thanks.

Re: [PATCH 2/3] hash-object doc: elaborate on -w and --literally promises

[Junio C Hamano]

Jakub Narebski <jnareb@gmail.com> writes:

> I thik that this implemetation detail of `--literally` is here to stay;
> how would you otherwise fix the issue if garbage object makes Git crash?

By repacking, presumably ;-).

More importantly, there needs a way to extend "enum object_type" to
allow unbounded number of arbitrary (garbage) types before we can
allow --literally to record such a garbage type in a pack stream.

So I'd expect the implementation detail would stay for a long time.
But there is nothing that says `--literally` inherently must write
loose.  It is plausible that a new implementation writes objects of
known/valid types to a pack stream, while unknown/garbage types to
loose objects.

> However, I would prefer to have options state _intent_; if there is
> legitimate need for a tool that creates loose objects, it would be
> better to have separate `--loose` option to `git hash-object` (which
> would imply `-w`, otherwise it doesn't have sense).

Yes, I very much agree with that.