From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com> To: "SZEDER Gábor" <szeder.dev@gmail.com> Cc: "Junio C Hamano" <gitster@pobox.com>, "Derrick Stolee" <stolee@gmail.com>, "Jeff King" <peff@peff.net>, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>, "Eric Sunshine" <sunshine@sunshineco.com>, git@vger.kernel.org Subject: Re: [PATCH] commit-graph: don't show progress percentages while expanding reachable commits Date: Fri, 22 Mar 2019 18:23:03 +0100 Message-ID: <87ef6ydds8.fsf@evledraar.gmail.com> (raw) In-Reply-To: <20190322154943.GF22459@szeder.dev> On Fri, Mar 22 2019, SZEDER Gábor wrote: > On Fri, Mar 22, 2019 at 04:11:24PM +0100, Ævar Arnfjörð Bjarmason wrote: >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote: >> >> > On Fri, Mar 22, 2019 at 03:28:34PM +0100, Ævar Arnfjörð Bjarmason wrote: >> >> >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote: >> >> >> >> > On Fri, Mar 22, 2019 at 12:11:26PM +0100, Ævar Arnfjörð Bjarmason wrote: >> >> >> >> >> >> On Fri, Mar 22 2019, SZEDER Gábor wrote: >> >> >> >> >> >> > Commit 49bbc57a57 (commit-graph write: emit a percentage for all >> >> >> > progress, 2019-01-19) was a bit overeager when it added progress >> >> >> > percentages to the "Expanding reachable commits in commit graph" phase >> >> >> > as well, because most of the time the number of commits that phase has >> >> >> > to iterate over is not known in advance and grows significantly, and, >> >> >> > consequently, we end up with nonsensical numbers: >> >> >> > >> >> >> > $ git commit-graph write --reachable >> >> >> > Expanding reachable commits in commit graph: 138606% (824706/595), done. >> >> >> > [...] >> >> >> > >> >> >> > $ git rev-parse v5.0 | git commit-graph write --stdin-commits >> >> >> > Expanding reachable commits in commit graph: 81264400% (812644/1), done. >> >> >> > [...] >> >> >> > >> >> >> > Therefore, don't show progress percentages in the "Expanding reachable >> >> >> > commits in commit graph" phase. >> >> >> >> >> >> There's indeed a bug here as your examples show, but there *are* cases >> >> >> where it's correct, as the commit message for my patch on "master" shows >> >> >> there's cases where we correctly. >> >> >> >> >> >> So this "fixes" things by always removing the progress, why not instead >> >> >> pass down the state to close_reachable() about what we're walking over, >> >> >> so we can always show progress, or at least in some cases? >> >> > >> >> > The cases where it does display correct percentages are exceptional, >> >> > and doesn't worth the effort to try to find out whether ther current >> >> > operation happens to be such a case. >> >> >> >> It's the "write" entry point without arguments that displays the correct >> >> progress. So not exceptional, but yeah, it's not what we use on "gc". >> > >> > Bit it displays the correct number only if all the reachable commits >> > are in packfiles, which is not necessarily the case (e.g. unpacked >> > small packs during 'git fetch'). >> >> No, argument-less "write" only considers packed commits. > > No, it considers packed commits as starting points, and then expands > to all reachable commits, that's what that loop in question is about. > > $ git init > Initialized empty Git repository in /tmp/test/.git/ > $ echo >file > $ git add file > $ git commit -q -m initial > $ echo 1 >file > $ git commit -q -m 1 file > $ git rev-parse HEAD | git pack-objects > .git/objects/pack/pack > Enumerating objects: 1, done. > Counting objects: 100% (1/1), done. > ece3ff72952af2b28e048fa5c58db88c44312876 > Writing objects: 100% (1/1), done. > Total 1 (delta 0), reused 0 (delta 0) > $ git commit-graph write > Computing commit graph generation numbers: 100% (2/2), done. Ah, indeed. I think in practice it'll be unlikely to happen in practice except on servers due to receive.unpackLimit, and then it won't be off by much. So I think it's best to do something like my WIP patch upthread + the "snap to 100%" at the end behavior. I'll try to clean that up / test it / submit that sometime soon. >> >> In any case, the problem is that sometimes we've walked the full set of >> >> commits already, and some other times we haven't. >> > >> > ... and that we can't really be sure whether we've walked the full set >> > of commits until after this loop. >> >> I'm fairly sure we can when we start with a full walk. See my >> explanation in <87imwbc6x8.fsf@evledraar.gmail.com>, but I may have >> missed something. >> >> >> So in cases where we have we can show progress, and as a TODO (I think >> >> this came up in previous discussions), we could do better if we had a >> >> approximate_commit_count(). >> >> >> >> In any case, the below fix seems correct to me, but I haven't poked it >> >> much. It *does* suffer from a theoretical race with the progress bar >> >> similar to d9b1b309cf ("commit-graph write: show progress for object >> >> search", 2019-01-19), but I work around it in the same way: >> >> >> >> diff --git a/commit-graph.c b/commit-graph.c >> >> index 47e9be0a3a..0fab3d8b2b 100644 >> >> --- a/commit-graph.c >> >> +++ b/commit-graph.c >> >> @@ -693,7 +693,8 @@ static void add_missing_parents(struct packed_oid_list *oids, struct commit *com >> >> } >> >> } >> >> >> >> -static void close_reachable(struct packed_oid_list *oids, int report_progress) >> >> +static void close_reachable(struct packed_oid_list *oids, int report_progress, >> >> + uint64_t oids_count_for_progress) >> >> { >> >> int i; >> >> struct commit *commit; >> >> @@ -717,7 +718,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress) >> >> */ >> >> if (report_progress) >> >> progress = start_delayed_progress( >> >> - _("Expanding reachable commits in commit graph"), oids->nr); >> >> + _("Expanding reachable commits in commit graph"), >> >> + oids_count_for_progress); >> >> for (i = 0; i < oids->nr; i++) { >> >> display_progress(progress, i + 1); >> >> commit = lookup_commit(the_repository, &oids->list[i]); >> >> @@ -725,6 +727,8 @@ static void close_reachable(struct packed_oid_list *oids, int report_progress) >> >> if (commit && !parse_commit(commit)) >> >> add_missing_parents(oids, commit); >> >> } >> >> + if (oids->nr < oids_count_for_progress) >> >> + display_progress(progress, oids_count_for_progress); >> >> stop_progress(&progress); >> >> >> >> if (report_progress) >> >> @@ -829,6 +833,7 @@ void write_commit_graph(const char *obj_dir, >> >> uint64_t progress_cnt = 0; >> >> struct strbuf progress_title = STRBUF_INIT; >> >> unsigned long approx_nr_objects; >> >> + uint64_t oids_count_for_progress = 0; >> >> >> >> if (!commit_graph_compatible(the_repository)) >> >> return; >> >> @@ -934,9 +939,10 @@ void write_commit_graph(const char *obj_dir, >> >> if (oids.progress_done < approx_nr_objects) >> >> display_progress(oids.progress, approx_nr_objects); >> >> stop_progress(&oids.progress); >> >> + oids_count_for_progress = oids.nr; >> >> } >> >> >> >> - close_reachable(&oids, report_progress); >> >> + close_reachable(&oids, report_progress, oids_count_for_progress); >> >> >> >> if (report_progress) >> >> progress = start_delayed_progress( >> >>
next prev parent reply other threads:[~2019-03-22 17:23 UTC|newest] Thread overview: 133+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-09-04 20:27 [PATCH 0/2] commit-graph: add progress output Ævar Arnfjörð Bjarmason 2018-09-04 20:27 ` [PATCH 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason 2018-09-04 21:16 ` Eric Sunshine 2018-09-04 22:07 ` Junio C Hamano 2018-09-05 11:58 ` Derrick Stolee 2018-09-05 12:07 ` Ævar Arnfjörð Bjarmason 2018-09-05 21:46 ` Junio C Hamano 2018-09-05 22:12 ` Derrick Stolee 2018-09-07 15:11 ` Ævar Arnfjörð Bjarmason 2018-09-07 15:23 ` Ævar Arnfjörð Bjarmason 2018-09-07 17:15 ` Jeff King 2018-09-07 17:25 ` Derrick Stolee 2018-09-05 12:06 ` Derrick Stolee 2018-09-07 12:40 ` Ævar Arnfjörð Bjarmason 2018-09-07 13:12 ` Derrick Stolee 2018-09-04 20:27 ` [PATCH 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason 2018-09-04 22:10 ` Junio C Hamano 2018-09-05 12:07 ` [PATCH 0/2] commit-graph: " Derrick Stolee 2018-09-07 18:29 ` [PATCH v2 " Ævar Arnfjörð Bjarmason 2018-09-11 20:26 ` Junio C Hamano 2018-09-07 18:29 ` [PATCH v2 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason 2018-09-21 20:01 ` Derrick Stolee 2018-09-21 21:43 ` Junio C Hamano 2018-09-21 21:57 ` Junio C Hamano 2018-09-07 18:29 ` [PATCH v2 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason 2018-09-16 6:55 ` Duy Nguyen 2018-09-17 15:33 ` [PATCH v3 0/2] commit-graph: " Ævar Arnfjörð Bjarmason 2018-09-17 15:33 ` [PATCH v3 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason 2018-10-10 20:37 ` SZEDER Gábor 2018-10-10 21:56 ` Ævar Arnfjörð Bjarmason 2018-10-10 22:19 ` SZEDER Gábor 2018-10-10 22:37 ` Ævar Arnfjörð Bjarmason 2018-10-11 17:52 ` Ævar Arnfjörð Bjarmason 2018-10-15 16:05 ` SZEDER Gábor 2018-10-12 6:09 ` Junio C Hamano 2018-10-12 15:07 ` Ævar Arnfjörð Bjarmason 2018-10-12 15:12 ` Derrick Stolee 2018-10-15 16:54 ` SZEDER Gábor 2018-11-19 16:02 ` SZEDER Gábor 2018-11-19 20:23 ` [PATCH] commit-graph: split up close_reachable() " Ævar Arnfjörð Bjarmason 2018-11-19 20:38 ` Derrick Stolee 2018-11-19 22:57 ` SZEDER Gábor 2018-11-20 15:04 ` [PATCH 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason 2018-11-20 16:58 ` SZEDER Gábor 2018-11-20 19:50 ` [PATCH v2 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason 2018-11-20 19:50 ` [PATCH v2 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason 2018-11-20 19:50 ` [PATCH v2 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason 2018-11-20 23:38 ` SZEDER Gábor 2018-11-20 19:50 ` [PATCH v2 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2018-11-20 19:50 ` [PATCH v2 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason 2018-11-20 19:50 ` [PATCH v2 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2018-11-20 19:50 ` [PATCH v2 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason 2018-11-21 1:23 ` SZEDER Gábor 2018-11-21 1:25 ` [PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' SZEDER Gábor 2018-11-21 3:29 ` Junio C Hamano 2018-11-21 11:32 ` Derrick Stolee 2019-01-18 17:05 ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization SZEDER Gábor 2019-01-18 17:05 ` [PATCH v2 1/2] commit-graph: rename "large edges" to "extra edges" SZEDER Gábor 2019-01-18 17:05 ` [PATCH v2 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor 2019-01-19 9:32 ` Martin Ågren 2019-01-18 19:41 ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization Junio C Hamano 2018-11-21 1:26 ` [PATCH 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor 2018-11-21 11:33 ` Derrick Stolee 2018-11-22 13:28 ` [PATCH v3 00/10] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 0/9] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason 2019-01-19 1:26 ` Junio C Hamano 2019-01-19 20:21 ` [PATCH v6 00/10] " Ævar Arnfjörð Bjarmason 2019-01-22 18:30 ` Derrick Stolee 2019-01-23 17:52 ` Junio C Hamano 2019-01-22 19:37 ` Junio C Hamano 2019-01-19 20:21 ` [PATCH v6 01/10] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 02/10] commit-graph: rename "large edges" to "extra edges" Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 03/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason 2019-01-23 17:51 ` [PATCH v6.1 03/10] commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily SZEDER Gábor 2019-01-19 20:21 ` [PATCH v6 04/10] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason 2019-01-19 20:21 ` [PATCH v6 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason 2019-03-22 10:28 ` [PATCH] commit-graph: don't show progress percentages while expanding reachable commits SZEDER Gábor 2019-03-22 11:11 ` Ævar Arnfjörð Bjarmason 2019-03-22 11:18 ` SZEDER Gábor 2019-03-22 14:28 ` Ævar Arnfjörð Bjarmason 2019-03-22 14:36 ` Ævar Arnfjörð Bjarmason 2019-03-22 14:55 ` SZEDER Gábor 2019-03-22 15:11 ` Ævar Arnfjörð Bjarmason 2019-03-22 15:49 ` SZEDER Gábor 2019-03-22 16:52 ` SZEDER Gábor 2019-03-22 17:23 ` Ævar Arnfjörð Bjarmason [this message] 2019-01-16 13:29 ` [PATCH v5 1/9] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 2/9] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 3/9] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason 2019-01-18 17:16 ` SZEDER Gábor 2019-01-16 13:29 ` [PATCH v5 4/9] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 5/9] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 6/9] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 7/9] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 8/9] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason 2019-01-16 13:29 ` [PATCH v5 9/9] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason 2019-01-17 13:23 ` [PATCH] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason 2019-01-17 15:09 ` Derrick Stolee 2019-01-17 16:35 ` Derrick Stolee 2018-11-22 15:39 ` [PATCH v4 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason 2018-11-22 15:39 ` [PATCH v4 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason 2018-11-22 18:59 ` [PATCH v3 00/10] commit-graph write: progress output improvements Eric Sunshine 2018-11-22 13:28 ` [PATCH v3 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason 2018-11-22 13:28 ` [PATCH v3 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason 2018-11-20 15:04 ` [PATCH 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason 2018-09-17 15:33 ` [PATCH v3 2/2] commit-graph verify: add " Ævar Arnfjörð Bjarmason
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=87ef6ydds8.fsf@evledraar.gmail.com \ --to=avarab@gmail.com \ --cc=git@vger.kernel.org \ --cc=gitster@pobox.com \ --cc=pclouds@gmail.com \ --cc=peff@peff.net \ --cc=stolee@gmail.com \ --cc=sunshine@sunshineco.com \ --cc=szeder.dev@gmail.com \ --subject='Re: [PATCH] commit-graph: don'\''t show progress percentages while expanding reachable commits' \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
git@vger.kernel.org list mirror (unofficial, one of many) This inbox may be cloned and mirrored by anyone: git clone --mirror https://public-inbox.org/git git clone --mirror http://ou63pmih66umazou.onion/git git clone --mirror http://czquwvybam4bgbro.onion/git git clone --mirror http://hjrcffqmbrq6wope.onion/git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V1 git git/ https://public-inbox.org/git \ git@vger.kernel.org public-inbox-index git Example config snippet for mirrors. Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.version-control.git nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git nntp://news.gmane.io/gmane.comp.version-control.git note: .onion URLs require Tor: https://www.torproject.org/ code repositories for project(s) associated with this inbox: https://80x24.org/mirrors/git.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git