git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: "SZEDER Gábor" <szeder.dev@gmail.com>
Cc: "Junio C Hamano" <gitster@pobox.com>,
	"Derrick Stolee" <stolee@gmail.com>, "Jeff King" <peff@peff.net>,
	"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
	"Eric Sunshine" <sunshine@sunshineco.com>,
	git@vger.kernel.org
Subject: Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits
Date: Fri, 22 Mar 2019 15:28:34 +0100	[thread overview]
Message-ID: <87k1grc7al.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <20190322111829.GC22459@szeder.dev>


On Fri, Mar 22 2019, SZEDER Gábor wrote:

> On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote:
>>
>> On Fri, Mar 22 2019, SZEDER Gábor wrote:
>>
>> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all
>> > progress, 2019-01-19) was a bit overeager when it added progress
>> > percentages to the "Expanding reachable commits in commit graph" phase
>> > as well, because most of the time the number of commits that phase has
>> > to iterate over is not known in advance and grows significantly, and,
>> > consequently, we end up with nonsensical numbers:
>> >
>> >   $ git commit-graph write --reachable
>> >   Expanding reachable commits in commit graph: 138606% (824706/595), done.
>> >   [...]
>> >
>> >   $ git rev-parse v5.0 | git commit-graph write --stdin-commits
>> >   Expanding reachable commits in commit graph: 81264400% (812644/1), done.
>> >   [...]
>> >
>> > Therefore, don't show progress percentages in the "Expanding reachable
>> > commits in commit graph" phase.
>>
>> There's indeed a bug here as your examples show, but there *are* cases
>> where it's correct, as the commit message for my patch on "master" shows
>> there's cases where we correctly.
>>
>> So this "fixes" things by always removing the progress, why not instead
>> pass down the state to close_reachable() about what we're walking over,
>> so we can always show progress, or at least in some cases?
>
> The cases where it does display correct percentages are exceptional,
> and doesn't worth the effort to try to find out whether ther current
> operation happens to be such a case.

It's the "write" entry point without arguments that displays the correct
progress. So not exceptional, but yeah, it's not what we use on "gc".

In any case, the problem is that sometimes we've walked the full set of
commits already, and some other times we haven't.

So in cases where we have we can show progress, and as a TODO (I think
this came up in previous discussions), we could do better if we had a
approximate_commit_count().

In any case, the below fix seems correct to me, but I haven't poked it
much. It *does* suffer from a theoretical race with the progress bar
similar to d9b1b309cf ("commit-graph write: show progress for object
search", 2019-01-19), but I work around it in the same way:

diff --git a/commit-graph.c b/commit-graph.c
index 47e9be0a3a..0fab3d8b2b 100644
--- a/commit-graph.c
+++ b/commit-graph.c
@@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com
 	}
 }
 
-static void close_reachable(struct packed_oid_list *oids, int report_progress)
+static void close_reachable(struct packed_oid_list *oids, int report_progress,
+			    uint64_t oids_count_for_progress)
 {
 	int i;
 	struct commit *commit;
@@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)
 	 */
 	if (report_progress)
 		progress = start_delayed_progress(
-			_("Expanding reachable commits in commit graph"), oids->nr);
+			_("Expanding reachable commits in commit graph"),
+			oids_count_for_progress);
 	for (i = 0; i < oids->nr; i++) {
 		display_progress(progress, i + 1);
 		commit = lookup_commit(the_repository, &oids->list[i]);
@@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress)
 		if (commit && !parse_commit(commit))
 			add_missing_parents(oids, commit);
 	}
+	if (oids->nr < oids_count_for_progress)
+		display_progress(progress, oids_count_for_progress);
 	stop_progress(&progress);
 
 	if (report_progress)
@@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir,
 	uint64_t progress_cnt = 0;
 	struct strbuf progress_title = STRBUF_INIT;
 	unsigned long approx_nr_objects;
+	uint64_t oids_count_for_progress = 0;
 
 	if (!commit_graph_compatible(the_repository))
 		return;
@@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir,
 		if (oids.progress_done < approx_nr_objects)
 			display_progress(oids.progress, approx_nr_objects);
 		stop_progress(&oids.progress);
+		oids_count_for_progress = oids.nr;
 	}
 
-	close_reachable(&oids, report_progress);
+	close_reachable(&oids, report_progress, oids_count_for_progress);
 
 	if (report_progress)
 		progress = start_delayed_progress(


  reply	other threads:[~2019-03-22 14:28 UTC|newest]

Thread overview: 133+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-04 20:27 [PATCH 0/2] commit-graph: add progress output Ævar Arnfjörð Bjarmason
2018-09-04 20:27 ` [PATCH 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-09-04 21:16   ` Eric Sunshine
2018-09-04 22:07   ` Junio C Hamano
2018-09-05 11:58     ` Derrick Stolee
2018-09-05 12:07       ` Ævar Arnfjörð Bjarmason
2018-09-05 21:46       ` Junio C Hamano
2018-09-05 22:12         ` Derrick Stolee
2018-09-07 15:11       ` Ævar Arnfjörð Bjarmason
2018-09-07 15:23         ` Ævar Arnfjörð Bjarmason
2018-09-07 17:15           ` Jeff King
2018-09-07 17:25             ` Derrick Stolee
2018-09-05 12:06   ` Derrick Stolee
2018-09-07 12:40   ` Ævar Arnfjörð Bjarmason
2018-09-07 13:12     ` Derrick Stolee
2018-09-04 20:27 ` [PATCH 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason
2018-09-04 22:10   ` Junio C Hamano
2018-09-05 12:07 ` [PATCH 0/2] commit-graph: " Derrick Stolee
2018-09-07 18:29 ` [PATCH v2 " Ævar Arnfjörð Bjarmason
2018-09-11 20:26   ` Junio C Hamano
2018-09-07 18:29 ` [PATCH v2 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-09-21 20:01   ` Derrick Stolee
2018-09-21 21:43     ` Junio C Hamano
2018-09-21 21:57       ` Junio C Hamano
2018-09-07 18:29 ` [PATCH v2 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason
2018-09-16  6:55   ` Duy Nguyen
2018-09-17 15:33     ` [PATCH v3 0/2] commit-graph: " Ævar Arnfjörð Bjarmason
2018-09-17 15:33     ` [PATCH v3 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-10-10 20:37       ` SZEDER Gábor
2018-10-10 21:56         ` Ævar Arnfjörð Bjarmason
2018-10-10 22:19           ` SZEDER Gábor
2018-10-10 22:37             ` Ævar Arnfjörð Bjarmason
2018-10-11 17:52               ` Ævar Arnfjörð Bjarmason
2018-10-15 16:05                 ` SZEDER Gábor
2018-10-12  6:09         ` Junio C Hamano
2018-10-12 15:07           ` Ævar Arnfjörð Bjarmason
2018-10-12 15:12             ` Derrick Stolee
2018-10-15 16:54       ` SZEDER Gábor
2018-11-19 16:02         ` SZEDER Gábor
2018-11-19 20:23           ` [PATCH] commit-graph: split up close_reachable() " Ævar Arnfjörð Bjarmason
2018-11-19 20:38             ` Derrick Stolee
2018-11-19 22:57             ` SZEDER Gábor
2018-11-20 15:04               ` [PATCH 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason
2018-11-20 16:58                 ` SZEDER Gábor
2018-11-20 19:50                   ` [PATCH v2 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason
2018-11-20 23:38                     ` SZEDER Gábor
2018-11-20 19:50                   ` [PATCH v2 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason
2018-11-21  1:23                   ` SZEDER Gábor
2018-11-21  1:25                     ` [PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' SZEDER Gábor
2018-11-21  3:29                       ` Junio C Hamano
2018-11-21 11:32                         ` Derrick Stolee
2019-01-18 17:05                           ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization SZEDER Gábor
2019-01-18 17:05                             ` [PATCH v2 1/2] commit-graph: rename "large edges" to "extra edges" SZEDER Gábor
2019-01-18 17:05                             ` [PATCH v2 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor
2019-01-19  9:32                               ` Martin Ågren
2019-01-18 19:41                             ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization Junio C Hamano
2018-11-21  1:26                     ` [PATCH 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor
2018-11-21 11:33                       ` Derrick Stolee
2018-11-22 13:28                       ` [PATCH v3 00/10] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 0/9] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2019-01-19  1:26                             ` Junio C Hamano
2019-01-19 20:21                             ` [PATCH v6 00/10] " Ævar Arnfjörð Bjarmason
2019-01-22 18:30                               ` Derrick Stolee
2019-01-23 17:52                                 ` Junio C Hamano
2019-01-22 19:37                               ` Junio C Hamano
2019-01-19 20:21                             ` [PATCH v6 01/10] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 02/10] commit-graph: rename "large edges" to "extra edges" Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 03/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2019-01-23 17:51                               ` [PATCH v6.1 03/10] commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily SZEDER Gábor
2019-01-19 20:21                             ` [PATCH v6 04/10] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2019-03-22 10:28                               ` [PATCH] commit-graph: don't show progress percentages while expanding reachable commits SZEDER Gábor
2019-03-22 11:11                                 ` Ævar Arnfjörð Bjarmason
2019-03-22 11:18                                   ` SZEDER Gábor
2019-03-22 14:28                                     ` Ævar Arnfjörð Bjarmason [this message]
2019-03-22 14:36                                       ` Ævar Arnfjörð Bjarmason
2019-03-22 14:55                                       ` SZEDER Gábor
2019-03-22 15:11                                         ` Ævar Arnfjörð Bjarmason
2019-03-22 15:49                                           ` SZEDER Gábor
2019-03-22 16:52                                             ` SZEDER Gábor
2019-03-22 17:23                                             ` Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 1/9] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 2/9] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 3/9] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason
2019-01-18 17:16                             ` SZEDER Gábor
2019-01-16 13:29                           ` [PATCH v5 4/9] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 5/9] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 6/9] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 7/9] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 8/9] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 9/9] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2019-01-17 13:23                             ` [PATCH] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason
2019-01-17 15:09                               ` Derrick Stolee
2019-01-17 16:35                                 ` Derrick Stolee
2018-11-22 15:39                         ` [PATCH v4 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2018-11-22 18:59                         ` [PATCH v3 00/10] commit-graph write: progress output improvements Eric Sunshine
2018-11-22 13:28                       ` [PATCH v3 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason
2018-09-17 15:33     ` [PATCH v3 2/2] commit-graph verify: add " Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=87k1grc7al.fsf@evledraar.gmail.com \
    --to=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=stolee@gmail.com \
    --cc=sunshine@sunshineco.com \
    --cc=szeder.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).