From: Derrick Stolee <stolee@gmail.com>
To: Jakub Narebski <jnareb@gmail.com>,
Derrick Stolee <dstolee@microsoft.com>
Cc: "git@vger.kernel.org" <git@vger.kernel.org>,
"gitster@pobox.com" <gitster@pobox.com>,
"avarab@gmail.com" <avarab@gmail.com>,
"marten.agren@gmail.com" <marten.agren@gmail.com>,
"peff@peff.net" <peff@peff.net>
Subject: Re: [PATCH v3 16/20] commit-graph: verify contents match checksum
Date: Mon, 4 Jun 2018 07:55:36 -0400 [thread overview]
Message-ID: <8b7695b6-049a-de92-b4ee-70a73e5cf583@gmail.com> (raw)
In-Reply-To: <86muwdtcc7.fsf@gmail.com>
On 6/2/2018 11:52 AM, Jakub Narebski wrote:
> Derrick Stolee <dstolee@microsoft.com> writes:
>
>> The commit-graph file ends with a SHA1 hash of the previous contents. If
>> a commit-graph file has errors but the checksum hash is correct, then we
>> know that the problem is a bug in Git and not simply file corruption
>> after-the-fact.
>>
>> Compute the checksum right away so it is the first error that appears,
>> and make the message translatable since this error can be "corrected" by
>> a user by simply deleting the file and recomputing. The rest of the
>> errors are useful only to developers.
> Should we then provide --quiet / --verbose options, so that ordinary
> user is not flooded with error messages meant for power users and Git
> developers, then?
>
>> Be sure to continue checking the rest of the file data if the checksum
>> is wrong. This is important for our tests, as we break the checksum as
>> we modify bytes of the commit-graph file.
> Well, we could have used sha1sum program, or test-sha1 helper to fix the
> checksum after corrupting the commit-graph file...
>
>> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>> ---
>> commit-graph.c | 16 ++++++++++++++--
>> t/t5318-commit-graph.sh | 6 ++++++
>> 2 files changed, 20 insertions(+), 2 deletions(-)
>>
>> diff --git a/commit-graph.c b/commit-graph.c
>> index d2b291aca2..a33600c584 100644
>> --- a/commit-graph.c
>> +++ b/commit-graph.c
>> @@ -841,6 +841,7 @@ void write_commit_graph(const char *obj_dir,
>> oids.nr = 0;
>> }
>>
>> +#define VERIFY_COMMIT_GRAPH_ERROR_HASH 2
>> static int verify_commit_graph_error;
>>
>> static void graph_report(const char *fmt, ...)
>> @@ -860,7 +861,9 @@ static void graph_report(const char *fmt, ...)
>> int verify_commit_graph(struct commit_graph *g)
>> {
>> uint32_t i, cur_fanout_pos = 0;
>> - struct object_id prev_oid, cur_oid;
>> + struct object_id prev_oid, cur_oid, checksum;
>> + struct hashfile *f;
>> + int devnull;
>>
>> if (!g) {
>> graph_report("no commit-graph file loaded");
>> @@ -879,6 +882,15 @@ int verify_commit_graph(struct commit_graph *g)
>> if (verify_commit_graph_error)
>> return verify_commit_graph_error;
>>
>> + devnull = open("/dev/null", O_WRONLY);
>> + f = hashfd(devnull, NULL);
>> + hashwrite(f, g->data, g->data_len - g->hash_len);
>> + finalize_hashfile(f, checksum.hash, CSUM_CLOSE);
>> + if (hashcmp(checksum.hash, g->data + g->data_len - g->hash_len)) {
>> + graph_report(_("the commit-graph file has incorrect checksum and is likely corrupt"));
>> + verify_commit_graph_error = VERIFY_COMMIT_GRAPH_ERROR_HASH;
>> + }
> Is it the best way of calculating the SHA-1 checksum that out internal
> APIs provide? Is it how SHA-1 checksum is calculated and checked for
> packfiles?
This pattern is similar to hashfd_check() in csum-file.c, except we are
hashing the file data directly instead of re-creating it from scratch
(as is done for 'git index-pack --verify').
>
>> +
>> for (i = 0; i < g->num_commits; i++) {
>> struct commit *graph_commit;
>>
>> @@ -916,7 +928,7 @@ int verify_commit_graph(struct commit_graph *g)
>> cur_fanout_pos++;
>> }
>>
>> - if (verify_commit_graph_error)
>> + if (verify_commit_graph_error & ~VERIFY_COMMIT_GRAPH_ERROR_HASH)
>> return verify_commit_graph_error;
> So if we detected that checksum do not match, but we have not found an
> error, we say that it is all right?
This only prevents us from stopping early. We will still report an error.
>
>>
>> for (i = 0; i < g->num_commits; i++) {
>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
>> index 240aef6add..2680a2ebff 100755
>> --- a/t/t5318-commit-graph.sh
>> +++ b/t/t5318-commit-graph.sh
>> @@ -279,6 +279,7 @@ GRAPH_COMMIT_DATA_WIDTH=`expr $HASH_LEN + 16`
>> GRAPH_OCTOPUS_DATA_OFFSET=`expr $GRAPH_COMMIT_DATA_OFFSET + \
>> $GRAPH_COMMIT_DATA_WIDTH \* $NUM_COMMITS`
>> GRAPH_BYTE_OCTOPUS=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4`
>> +GRAPH_BYTE_FOOTER=`expr $GRAPH_OCTOPUS_DATA_OFFSET + 4 \* $NUM_OCTOPUS_EDGES`
>>
>> # usage: corrupt_graph_and_verify <position> <data> <string>
>> # Manipulates the commit-graph file at the position
>> @@ -388,4 +389,9 @@ test_expect_success 'detect incorrect parent for octopus merge' '
>> "invalid parent"
>> '
>>
>> +test_expect_success 'detect invalid checksum hash' '
>> + corrupt_graph_and_verify $GRAPH_BYTE_FOOTER "\00" \
>> + "incorrect checksum"
> This would not work under GETTEXT_POISON, as the message is marked as
> translatable, but corrupt_graph_and_verify uses 'grep' and not
> 'test_i18grep' from t/test-lib-functions.sh
I fixed this locally based on Szeder's comment.
>
>> +'
> If it is pure checksum corruption, wouldn't this fail because it is not
> a failure (exit code is 0)?
It is not zero, so the test passes.
next prev parent reply other threads:[~2018-06-04 11:55 UTC|newest]
Thread overview: 149+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-04-17 18:10 [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 01/12] fixup! commit-graph: always load commit-graph information Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 03/12] commit-graph: check file header information Derrick Stolee
2018-04-19 15:58 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 02/12] commit-graph: add 'check' subcommand Derrick Stolee
2018-04-19 13:24 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-04-19 17:21 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 06/12] commit: force commit to parse from object database Derrick Stolee
2018-04-20 12:13 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 05/12] commit-graph: check fanout and lookup table Derrick Stolee
2018-04-20 7:27 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-04-20 12:18 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-04-20 16:47 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-04-20 17:17 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 09/12] fsck: check commit-graph Derrick Stolee
2018-04-20 16:59 ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-04-20 17:34 ` Jakub Narebski
2018-04-20 18:33 ` Ævar Arnfjörð Bjarmason
2018-04-17 18:10 ` [RFC PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-04-20 19:10 ` Jakub Narebski
2018-04-17 18:50 ` [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-10 17:34 ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Derrick Stolee
2018-05-10 17:34 ` [PATCH 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-10 18:15 ` Martin Ågren
2018-05-10 17:34 ` [PATCH 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-10 18:21 ` Martin Ågren
2018-05-10 17:34 ` [PATCH 03/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 04/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-10 18:29 ` Martin Ågren
2018-05-11 15:17 ` Derrick Stolee
2018-05-10 17:34 ` [PATCH 05/12] commit: force commit to parse from object database Derrick Stolee
2018-05-10 17:34 ` [PATCH 06/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 07/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-10 17:34 ` [PATCH 08/12] fsck: verify commit-graph Derrick Stolee
2018-05-10 17:34 ` [PATCH 09/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-10 17:34 ` [PATCH 10/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-10 17:34 ` [PATCH 11/12] fetch: compute commit-graph by default Derrick Stolee
2018-05-10 17:34 ` [PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-05-10 19:05 ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Martin Ågren
2018-05-10 19:22 ` Stefan Beller
2018-05-11 17:23 ` Derrick Stolee
2018-05-11 17:30 ` Martin Ågren
2018-05-10 19:17 ` Ævar Arnfjörð Bjarmason
2018-05-11 17:23 ` Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 00/12] Integrate commit-graph into fsck and gc Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-12 13:31 ` Martin Ågren
2018-05-14 13:27 ` Derrick Stolee
2018-05-20 12:10 ` Jakub Narebski
2018-05-11 21:15 ` [PATCH v2 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-12 13:35 ` Martin Ågren
2018-05-14 13:31 ` Derrick Stolee
2018-05-20 20:00 ` Jakub Narebski
2018-05-11 21:15 ` [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption Derrick Stolee
2018-05-12 13:43 ` Martin Ågren
2018-05-21 18:53 ` Jakub Narebski
2018-05-24 16:28 ` Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-12 20:50 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 05/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-11 21:15 ` [PATCH v2 06/12] commit: force commit to parse from object database Derrick Stolee
2018-05-12 20:54 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-12 20:55 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-12 21:17 ` Martin Ågren
2018-05-14 13:44 ` Derrick Stolee
2018-05-15 21:12 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 09/12] fsck: verify commit-graph Derrick Stolee
2018-05-17 18:13 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-17 18:20 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-17 18:16 ` Martin Ågren
2018-05-11 21:15 ` [PATCH v2 12/12] commit-graph: update design document Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 01/20] commit-graph: UNLEAK before die() Derrick Stolee
2018-05-24 22:47 ` Stefan Beller
2018-05-25 0:08 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE Derrick Stolee
2018-05-26 18:46 ` Jakub Narebski
2018-05-26 20:30 ` brian m. carlson
2018-06-02 19:43 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 03/20] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-27 10:23 ` Jakub Narebski
2018-05-29 12:31 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 04/20] commit: force commit to parse from object database Derrick Stolee
2018-05-27 18:04 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 05/20] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-27 19:12 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 06/20] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-27 22:55 ` Jakub Narebski
2018-05-30 16:07 ` Derrick Stolee
2018-06-02 21:19 ` Jakub Narebski
2018-06-04 11:30 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 07/20] commit-graph: verify catches corrupt signature Derrick Stolee
2018-05-28 14:05 ` Jakub Narebski
2018-05-29 12:43 ` Derrick Stolee
2018-06-02 22:30 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 08/20] commit-graph: verify required chunks are present Derrick Stolee
2018-05-28 17:11 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup Derrick Stolee
2018-05-30 13:34 ` Jakub Narebski
2018-05-30 16:18 ` Derrick Stolee
2018-06-02 4:38 ` Duy Nguyen
2018-06-04 11:32 ` Derrick Stolee
2018-06-04 14:42 ` Duy Nguyen
2018-05-24 16:25 ` [PATCH v3 10/20] commit-graph: verify objects exist Derrick Stolee
2018-05-30 19:22 ` Jakub Narebski
2018-05-31 12:53 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 11/20] commit-graph: verify root tree OIDs Derrick Stolee
2018-05-30 22:24 ` Jakub Narebski
2018-05-31 13:16 ` Derrick Stolee
2018-06-02 22:50 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 12/20] commit-graph: verify parent list Derrick Stolee
2018-06-01 23:21 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 13/20] commit-graph: verify generation number Derrick Stolee
2018-06-02 12:23 ` Jakub Narebski
2018-06-04 11:47 ` Derrick Stolee
2018-05-24 16:25 ` [PATCH v3 14/20] commit-graph: verify commit date Derrick Stolee
2018-06-02 12:29 ` Jakub Narebski
2018-05-24 16:25 ` [PATCH v3 15/20] commit-graph: test for corrupted octopus edge Derrick Stolee
2018-06-02 12:39 ` Jakub Narebski
2018-06-04 13:08 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 16/20] commit-graph: verify contents match checksum Derrick Stolee
2018-05-30 12:35 ` SZEDER Gábor
2018-06-02 15:52 ` Jakub Narebski
2018-06-04 11:55 ` Derrick Stolee [this message]
2018-05-24 16:26 ` [PATCH v3 17/20] fsck: verify commit-graph Derrick Stolee
2018-06-02 16:17 ` Jakub Narebski
2018-06-04 11:59 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 18/20] commit-graph: add '--reachable' option Derrick Stolee
2018-06-02 17:34 ` Jakub Narebski
2018-06-04 12:44 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 19/20] gc: automatically write commit-graph files Derrick Stolee
2018-06-02 18:03 ` Jakub Narebski
2018-06-04 12:51 ` Derrick Stolee
2018-05-24 16:26 ` [PATCH v3 20/20] commit-graph: update design document Derrick Stolee
2018-06-02 18:27 ` Jakub Narebski
2018-05-24 21:15 ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Ævar Arnfjörð Bjarmason
2018-05-25 4:11 ` Junio C Hamano
2018-05-29 4:27 ` Junio C Hamano
2018-05-29 12:37 ` Derrick Stolee
2018-05-29 13:41 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=8b7695b6-049a-de92-b4ee-70a73e5cf583@gmail.com \
--to=stolee@gmail.com \
--cc=avarab@gmail.com \
--cc=dstolee@microsoft.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jnareb@gmail.com \
--cc=marten.agren@gmail.com \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).