git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Derrick Stolee <stolee@gmail.com>
To: Jakub Narebski <jnareb@gmail.com>,
	Derrick Stolee <dstolee@microsoft.com>
Cc: "git@vger.kernel.org" <git@vger.kernel.org>,
	"gitster@pobox.com" <gitster@pobox.com>,
	"avarab@gmail.com" <avarab@gmail.com>,
	"marten.agren@gmail.com" <marten.agren@gmail.com>,
	"peff@peff.net" <peff@peff.net>
Subject: Re: [PATCH v3 17/20] fsck: verify commit-graph
Date: Mon, 4 Jun 2018 07:59:15 -0400	[thread overview]
Message-ID: <c1fa14f3-f1cc-a91f-6192-e97e6437d0c4@gmail.com> (raw)
In-Reply-To: <867enhtb6n.fsf@gmail.com>

On 6/2/2018 12:17 PM, Jakub Narebski wrote:
> Derrick Stolee <dstolee@microsoft.com> writes:
>
>> If core.commitGraph is true, verify the contents of the commit-graph
>> during 'git fsck' using the 'git commit-graph verify' subcommand. Run
>> this check on all alternates, as well.
> All right, so we have one config variable to control the use of
> serialized commit-graph feaature.  Nice.
>
>> We use a new process for two reasons:
>>
>> 1. The subcommand decouples the details of loading and verifying a
>>     commit-graph file from the other fsck details.
> All right, I can agree with that.
>
> On the other hand using subcommand makes debugging harder, though not in
> this case (well separated functionality that can be easily called with a
> standalone command to be debugged).
>
>> 2. The commit-graph verification requires the commits to be loaded
>>     in a specific order to guarantee we parse from the commit-graph
>>     file for some objects and from the object database for others.
> I don't quite understand this.  Could you explain it in more detail?

We use `lookup_commit()` when verifying the commit-graph. If these 
commits were loaded earlier in the process and parsed directly from the 
object database, then we aren't comparing the commit-graph file contents 
against the ODB.

>
>> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
>> ---
>>   Documentation/git-fsck.txt |  3 +++
>>   builtin/fsck.c             | 21 +++++++++++++++++++++
>>   t/t5318-commit-graph.sh    |  8 ++++++++
>>   3 files changed, 32 insertions(+)
>>
>> diff --git a/Documentation/git-fsck.txt b/Documentation/git-fsck.txt
>> index b9f060e3b2..ab9a93fb9b 100644
>> --- a/Documentation/git-fsck.txt
>> +++ b/Documentation/git-fsck.txt
>> @@ -110,6 +110,9 @@ Any corrupt objects you will have to find in backups or other archives
>>   (i.e., you can just remove them and do an 'rsync' with some other site in
>>   the hopes that somebody else has the object you have corrupted).
>>   
>> +If core.commitGraph is true, the commit-graph file will also be inspected
> Shouldn't we use `core.commitGraph` here?
>
>> +using 'git commit-graph verify'. See linkgit:git-commit-graph[1].
>> +
>>   Extracted Diagnostics
>>   ---------------------
>>   
>> diff --git a/builtin/fsck.c b/builtin/fsck.c
>> index ef78c6c00c..a6d5045b77 100644
>> --- a/builtin/fsck.c
>> +++ b/builtin/fsck.c
>> @@ -16,6 +16,7 @@
>>   #include "streaming.h"
>>   #include "decorate.h"
>>   #include "packfile.h"
>> +#include "run-command.h"
>>   
>>   #define REACHABLE 0x0001
>>   #define SEEN      0x0002
>> @@ -45,6 +46,7 @@ static int name_objects;
>>   #define ERROR_REACHABLE 02
>>   #define ERROR_PACK 04
>>   #define ERROR_REFS 010
>> +#define ERROR_COMMIT_GRAPH 020
> Minor nitpick and a sidenote: I wonder if it wouldn't be better to
> either use hexadecimal constants, or use (1 << n) for all ERROR_*
> preprocesor constants.
>
>>   
>>   static const char *describe_object(struct object *obj)
>>   {
>> @@ -815,5 +817,24 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>>   	}
>>   
>>   	check_connectivity();
>> +
>> +	if (core_commit_graph) {
>> +		struct child_process commit_graph_verify = CHILD_PROCESS_INIT;
>> +		const char *verify_argv[] = { "commit-graph", "verify", NULL, NULL, NULL, NULL };
> I see that NULL at index 2 and 3 (at 3rd and 4th place) are here for
> "--object-dir" and <alternates-object-dir-path>, the last one is
> terminator for that case, but what is next to last NULL (at 5th place)
> for?
>
>> +		commit_graph_verify.argv = verify_argv;
>> +		commit_graph_verify.git_cmd = 1;
>> +
>> +		if (run_command(&commit_graph_verify))
>> +			errors_found |= ERROR_COMMIT_GRAPH;
>> +
>> +		prepare_alt_odb();
>> +		for (alt = alt_odb_list; alt; alt = alt->next) {
>> +			verify_argv[2] = "--object-dir";
>> +			verify_argv[3] = alt->path;
>> +			if (run_command(&commit_graph_verify))
>> +				errors_found |= ERROR_COMMIT_GRAPH;
>> +		}
>> +	}
> For performance reasons it may be better to start those 'git
> commit-graph verify' commands asynchronously earlier, so that they can
> run in parallel / concurrently wth other checks, and wait for them and
> get their error code at the end of git-fsck run.
>
> But that is probably better left for a separate commit.
>
>> +
>>   	return errors_found;
>>   }
>> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh
>> index 2680a2ebff..4941937163 100755
>> --- a/t/t5318-commit-graph.sh
>> +++ b/t/t5318-commit-graph.sh
>> @@ -394,4 +394,12 @@ test_expect_success 'detect invalid checksum hash' '
>>   		"incorrect checksum"
>>   '
>>   
>> +test_expect_success 'git fsck (checks commit-graph)' '
>> +	cd "$TRASH_DIRECTORY/full" &&
>> +	git fsck &&
>> +	corrupt_graph_and_verify $GRAPH_BYTE_FOOTER "\00" \
>> +		"incorrect checksum" &&
>> +	test_must_fail git fsck
>> +'
> All right; though the same caveats apply as with previous commit in
> series.  Perhaps it would be better to truncate commit-graph file, or
> corrupt it in some 'random' place.
>
>> +
>>   test_done
> Best,


  reply	other threads:[~2018-06-04 11:59 UTC|newest]

Thread overview: 149+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-04-17 18:10 [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 01/12] fixup! commit-graph: always load commit-graph information Derrick Stolee
2018-04-17 18:10 ` [RFC PATCH 02/12] commit-graph: add 'check' subcommand Derrick Stolee
2018-04-19 13:24   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 03/12] commit-graph: check file header information Derrick Stolee
2018-04-19 15:58   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-04-19 17:21   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 05/12] commit-graph: check fanout and lookup table Derrick Stolee
2018-04-20  7:27   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 06/12] commit: force commit to parse from object database Derrick Stolee
2018-04-20 12:13   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-04-20 12:18   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-04-20 16:47   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-04-20 17:17   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 09/12] fsck: check commit-graph Derrick Stolee
2018-04-20 16:59   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-04-20 19:10   ` Jakub Narebski
2018-04-17 18:10 ` [RFC PATCH 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-04-20 17:34   ` Jakub Narebski
2018-04-20 18:33     ` Ævar Arnfjörð Bjarmason
2018-04-17 18:50 ` [RFC PATCH 00/12] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-10 17:34 ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Derrick Stolee
2018-05-10 17:34   ` [PATCH 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-10 18:15     ` Martin Ågren
2018-05-10 17:34   ` [PATCH 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-10 18:21     ` Martin Ågren
2018-05-10 17:34   ` [PATCH 03/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-10 17:34   ` [PATCH 04/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-10 18:29     ` Martin Ågren
2018-05-11 15:17       ` Derrick Stolee
2018-05-10 17:34   ` [PATCH 05/12] commit: force commit to parse from object database Derrick Stolee
2018-05-10 17:34   ` [PATCH 06/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-10 17:34   ` [PATCH 07/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-10 17:34   ` [PATCH 08/12] fsck: verify commit-graph Derrick Stolee
2018-05-10 17:34   ` [PATCH 09/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-10 17:34   ` [PATCH 10/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-10 17:34   ` [PATCH 11/12] fetch: compute commit-graph by default Derrick Stolee
2018-05-10 17:34   ` [PATCH 12/12] commit-graph: update design document Derrick Stolee
2018-05-10 19:05   ` [PATCH 00/12] Integrate commit-graph into fsck, gc, and fetch Martin Ågren
2018-05-10 19:22     ` Stefan Beller
2018-05-11 17:23       ` Derrick Stolee
2018-05-11 17:30         ` Martin Ågren
2018-05-10 19:17   ` Ævar Arnfjörð Bjarmason
2018-05-11 17:23     ` Derrick Stolee
2018-05-11 21:15   ` [PATCH v2 00/12] Integrate commit-graph into fsck and gc Derrick Stolee
2018-05-11 21:15     ` [PATCH v2 01/12] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-12 13:31       ` Martin Ågren
2018-05-14 13:27         ` Derrick Stolee
2018-05-20 12:10       ` Jakub Narebski
2018-05-11 21:15     ` [PATCH v2 02/12] commit-graph: verify file header information Derrick Stolee
2018-05-12 13:35       ` Martin Ågren
2018-05-14 13:31         ` Derrick Stolee
2018-05-20 20:00       ` Jakub Narebski
2018-05-11 21:15     ` [PATCH v2 03/12] commit-graph: test that 'verify' finds corruption Derrick Stolee
2018-05-12 13:43       ` Martin Ågren
2018-05-21 18:53       ` Jakub Narebski
2018-05-24 16:28         ` Derrick Stolee
2018-05-11 21:15     ` [PATCH v2 04/12] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-12 20:50       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 05/12] commit-graph: verify fanout and lookup table Derrick Stolee
2018-05-11 21:15     ` [PATCH v2 06/12] commit: force commit to parse from object database Derrick Stolee
2018-05-12 20:54       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 07/12] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-12 20:55       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 08/12] commit-graph: verify commit contents against odb Derrick Stolee
2018-05-12 21:17       ` Martin Ågren
2018-05-14 13:44         ` Derrick Stolee
2018-05-15 21:12       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 09/12] fsck: verify commit-graph Derrick Stolee
2018-05-17 18:13       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 10/12] commit-graph: add '--reachable' option Derrick Stolee
2018-05-17 18:16       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 11/12] gc: automatically write commit-graph files Derrick Stolee
2018-05-17 18:20       ` Martin Ågren
2018-05-11 21:15     ` [PATCH v2 12/12] commit-graph: update design document Derrick Stolee
2018-05-24 16:25     ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 01/20] commit-graph: UNLEAK before die() Derrick Stolee
2018-05-24 22:47         ` Stefan Beller
2018-05-25  0:08           ` Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 02/20] commit-graph: fix GRAPH_MIN_SIZE Derrick Stolee
2018-05-26 18:46         ` Jakub Narebski
2018-05-26 20:30           ` brian m. carlson
2018-06-02 19:43             ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 03/20] commit-graph: parse commit from chosen graph Derrick Stolee
2018-05-27 10:23         ` Jakub Narebski
2018-05-29 12:31           ` Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 04/20] commit: force commit to parse from object database Derrick Stolee
2018-05-27 18:04         ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 05/20] commit-graph: load a root tree from specific graph Derrick Stolee
2018-05-27 19:12         ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 06/20] commit-graph: add 'verify' subcommand Derrick Stolee
2018-05-27 22:55         ` Jakub Narebski
2018-05-30 16:07           ` Derrick Stolee
2018-06-02 21:19             ` Jakub Narebski
2018-06-04 11:30               ` Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 07/20] commit-graph: verify catches corrupt signature Derrick Stolee
2018-05-28 14:05         ` Jakub Narebski
2018-05-29 12:43           ` Derrick Stolee
2018-06-02 22:30             ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 08/20] commit-graph: verify required chunks are present Derrick Stolee
2018-05-28 17:11         ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 09/20] commit-graph: verify corrupt OID fanout and lookup Derrick Stolee
2018-05-30 13:34         ` Jakub Narebski
2018-05-30 16:18           ` Derrick Stolee
2018-06-02  4:38         ` Duy Nguyen
2018-06-04 11:32           ` Derrick Stolee
2018-06-04 14:42             ` Duy Nguyen
2018-05-24 16:25       ` [PATCH v3 10/20] commit-graph: verify objects exist Derrick Stolee
2018-05-30 19:22         ` Jakub Narebski
2018-05-31 12:53           ` Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 11/20] commit-graph: verify root tree OIDs Derrick Stolee
2018-05-30 22:24         ` Jakub Narebski
2018-05-31 13:16           ` Derrick Stolee
2018-06-02 22:50             ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 12/20] commit-graph: verify parent list Derrick Stolee
2018-06-01 23:21         ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 13/20] commit-graph: verify generation number Derrick Stolee
2018-06-02 12:23         ` Jakub Narebski
2018-06-04 11:47           ` Derrick Stolee
2018-05-24 16:25       ` [PATCH v3 14/20] commit-graph: verify commit date Derrick Stolee
2018-06-02 12:29         ` Jakub Narebski
2018-05-24 16:25       ` [PATCH v3 15/20] commit-graph: test for corrupted octopus edge Derrick Stolee
2018-06-02 12:39         ` Jakub Narebski
2018-06-04 13:08           ` Derrick Stolee
2018-05-24 16:26       ` [PATCH v3 16/20] commit-graph: verify contents match checksum Derrick Stolee
2018-05-30 12:35         ` SZEDER Gábor
2018-06-02 15:52         ` Jakub Narebski
2018-06-04 11:55           ` Derrick Stolee
2018-05-24 16:26       ` [PATCH v3 17/20] fsck: verify commit-graph Derrick Stolee
2018-06-02 16:17         ` Jakub Narebski
2018-06-04 11:59           ` Derrick Stolee [this message]
2018-05-24 16:26       ` [PATCH v3 18/20] commit-graph: add '--reachable' option Derrick Stolee
2018-06-02 17:34         ` Jakub Narebski
2018-06-04 12:44           ` Derrick Stolee
2018-05-24 16:26       ` [PATCH v3 19/20] gc: automatically write commit-graph files Derrick Stolee
2018-06-02 18:03         ` Jakub Narebski
2018-06-04 12:51           ` Derrick Stolee
2018-05-24 16:26       ` [PATCH v3 20/20] commit-graph: update design document Derrick Stolee
2018-06-02 18:27         ` Jakub Narebski
2018-05-24 21:15       ` [PATCH v3 00/20] Integrate commit-graph into 'fsck' and 'gc' Ævar Arnfjörð Bjarmason
2018-05-25  4:11       ` Junio C Hamano
2018-05-29  4:27       ` Junio C Hamano
2018-05-29 12:37         ` Derrick Stolee
2018-05-29 13:41           ` Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c1fa14f3-f1cc-a91f-6192-e97e6437d0c4@gmail.com \
    --to=stolee@gmail.com \
    --cc=avarab@gmail.com \
    --cc=dstolee@microsoft.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=jnareb@gmail.com \
    --cc=marten.agren@gmail.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).