git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
From: "René Scharfe" <l.s.r@web.de>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Cc: "SZEDER Gábor" <szeder.dev@gmail.com>, git@vger.kernel.org
Subject: Re: [PATCH 4/7] commit-graph: fix bogus counter in "Scanning merged commits" progress line
Date: Tue, 6 Jul 2021 18:02:32 +0200	[thread overview]
Message-ID: <0a03bae6-53cf-98e5-635a-6f61583f5dd1@web.de> (raw)
In-Reply-To: <87y2ak5noa.fsf@evledraar.gmail.com>

Am 06.07.21 um 01:28 schrieb Ævar Arnfjörð Bjarmason:
>
> On Sun, Jul 04 2021, René Scharfe wrote:
>
>> Am 26.06.21 um 23:38 schrieb Ævar Arnfjörð Bjarmason:
>>>
>>> On Sat, Jun 26 2021, René Scharfe wrote:
>>>
>>>> Am 26.06.21 um 16:11 schrieb Ævar Arnfjörð Bjarmason:
>>> The only one I know of is "Enumerating objects", which notably stalls at
>>> the start, and which I'm proposing changing the output of in:
>>> https://lore.kernel.org/git/patch-18.25-e21fc66623f-20210623T155626Z-avarab@gmail.com/
>>
>> That's annoying, but the first number I see there has five or six digits,
>> so it's not an example of the issue mentioned above for me.
>
> Because it stalls and shows nothing, but with my patch it'll show
> something while stalling, FWIW on linux.git from a cold cache it took
> 5-10s before showing anything.
>
>> Your patch shows ", stalled." while pack-objects starts up.  I'm not sure
>> this helps.  Perhaps there are cases when it gets stuck, but it's hard to
>> determine by the clock alone.  When I run gc, it just needs a few seconds
>> to prepare something and then starts visibly counting objects.  A more
>> fine-grained report of the preparation steps would help, but seeing
>> "stalled" would just scare me.
>
> Fair enough, I have other patches to have it show a spinner. Again, API
> v.s. UI. The idea is that we show something before we start the loop.

A spinner would be nicer, but I would be more interested to see what it is
actually spending all that time on.  A separate progress line might be
justified here.

>>> Also, because for these cases that you're focusing on where we count up
>>> to exactly 100% and we therefore expect N calls to display_progress()
>>> (igroning the rare but allowed duplicate calls with the same number,
>>> which most callers don't use). We could just have a convenience API of:
>>>
>>>     start_progress()
>>>     for i in (0..X-1):
>>>         progress_update() /* passing "i" not needed, we increment internally */
>>>         work()
>>>     stop_progress()
>>>
>>> Then we could even make showing 0/N or 1/N the first time configuable,
>>> but we could only do both if we use the API as I'm suggesting, not as
>>> you want to use it.
>>
>> A function that increments the progress number relatively can be used
>> with both models.  It's more useful for the count-iterations model,
>> though, as in the count-work model you can piggy-back on the loop
>> counter check:
>>
>> 	for (i = 0; display_progress(p, i), i < X; i++)
>> 		work();
>
> Aside from this whole progress API discussion I find sticking stuff like
> that in the for-loop body to be less readable.
>
> But no, that can't be used with both models, because it conflates the 0
> of the 1st iteration with 0 of doing prep work. I.e.:
>
>     p = start_progress();
>     display_progress(p, 0);
>     prep_work();
>     for (i = 0; i < 100; i++)
>         display_progress(p, i + 1);
>
> Which is implicitly how that "stalled" patch views the world, i.e. our
> count is -1 is at start_progress() (that's already the case in
> progress.c).
>
> If you set it to 0 you're not working on the 1st item yet, but
> explicitly doing setup.
>
> Then at n=1 you're starting work on the 1st item.

A distinct preparation phase feels like an extension to the progress
API.  A symmetric cleanup phase at the end may make sense as well then.

I assume that preparations would be done between the start_progress call
and the first display_progress (no matter what number it reports).  And
cleanup would be done between the last display_progress call and the
stop_progress call.

In the count-iterations model this might report the time taken fro the
first or last item as preparation or cleanup depending on the placement
of the display_progress call.  That shouldn't be much of a problem,
though, as the value of one work item is zero in that model.

> FWIW the "complicated" here was referring to dealing with break/continue.
>
> Yes I'll grant you that there's cases where the uglyness/oddity of that
> for-loop trick is going to be better than dealing with that, but there's
> also while loops doing progress, callbacks etc.

while loops can easily be converted to for loops, of course.

Callbacks are a different matter.  I think we should use them less in
general (they force different operations to use the same set of
parameters, which is worked around with context structs).  A function
to increment progress would help them because then they wouldn't need
to keep track of the item/iteration count themselves in a context
variable.

However, in some cases display_progress calls are rate-limited, e.g.
midx_display_sparse_progress does that for performance reasons.  I
wonder why, and whether this is a problem that needs to be addressed
for all callers.  We don't want the progress API to delay the actual
progress significantly!  Currently display_progress avoids updating
the progress counter; an increment function would need to write an
updated value at each call.

> Picking an API pattern that works with all of that makes sense, since
> the UI can render the count one way or the other.

Right.

>>> If we do run into those cases and want to convert them to show some
>>> intra-item progress we'd need to first migrate them over to suggested
>>> way of using the API if we picked yours first, with my suggested use we
>>> only need to add new API calls (display_throughput(), and similar future
>>> calls/implicit display).
>>
>> I don't see why.  The intra-item progress numbers need to be reported in
>> any case if they are to be shown somehow.  If the model is clear then we
>> can show unambiguous output.
>
> Because you want to show:
>
>     Files 1/3 (33%) Bytes 17kB/9GB (0%)
>
> Not:
>
>     Files 0/3 (33%) Bytes 17kB/9GB (0%)
>
> You're downloading the 1st file, not the 0th file, so the code is a
> for-loop (or equivalent) with a display_progress(p, i + 1) for that
> file, not display_progress(p, i).
>
> This is the main reason I prefer the API and UI of reporting "what item
> am I on?" v.s. "how many items are done?", because it's easy to add
> intra-item state to the former.

Both look confusing.  If I'd care enough about one of the files or each
of them that I'd like to know their individual progress then I'd
certainly would want to see their names instead of some random number.

And as you write above: The display part can easily add or subtract one
to convert the number between models.

> Sure, anyway, let's assume all those numbers are magically known and
> constant. The point was that as noted above you're downloading the 1st
> file, not the 0th file, and want to show throughput/ETA etc. for that
> file.

OK, but still some kind of indication would have to be given that the
Bytes relate to a particular File instead of being the total for this
activity.  Perhaps like this, but it's a bit cluttered:

   File 1 (Bytes 17kB/9GB, 0% done) of 3 (0% done in total)

René

  reply	other threads:[~2021-07-06 16:02 UTC|newest]

Thread overview: 117+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-06-20 20:02 [PATCH 0/7] progress: verify progress counters in the test suite SZEDER Gábor
2021-06-20 20:02 ` [PATCH 1/7] progress: introduce GIT_TEST_CHECK_PROGRESS to verify progress counters SZEDER Gábor
2021-06-21  7:09   ` Ævar Arnfjörð Bjarmason
2021-06-22 15:55   ` Taylor Blau
2021-06-20 20:02 ` [PATCH 2/7] progress: catch nested/overlapping progresses with GIT_TEST_CHECK_PROGRESS SZEDER Gábor
2021-06-22 16:00   ` Taylor Blau
2021-08-30 21:15     ` SZEDER Gábor
2021-06-20 20:02 ` [PATCH 3/7] progress: catch backwards counting " SZEDER Gábor
2021-06-20 20:03 ` [PATCH 4/7] commit-graph: fix bogus counter in "Scanning merged commits" progress line SZEDER Gábor
2021-06-20 22:13   ` Ævar Arnfjörð Bjarmason
2021-06-21 18:32     ` René Scharfe
2021-06-21 20:08       ` Ævar Arnfjörð Bjarmason
2021-06-26  8:27         ` René Scharfe
2021-06-26 14:11           ` Ævar Arnfjörð Bjarmason
2021-06-26 20:22             ` René Scharfe
2021-06-26 21:38               ` Ævar Arnfjörð Bjarmason
2021-07-04 12:15                 ` René Scharfe
2021-07-05 14:09                   ` Junio C Hamano
2021-07-05 23:28                   ` Ævar Arnfjörð Bjarmason
2021-07-06 16:02                     ` René Scharfe [this message]
2021-06-27 17:31               ` Felipe Contreras
2021-06-20 20:03 ` [PATCH 5/7] entry: show finer-grained counter in "Filtering content" " SZEDER Gábor
2021-06-20 20:03 ` [PATCH 6/7] [RFC] entry: don't show "Filtering content: ... done." line in case of errors SZEDER Gábor
2021-06-21 18:32   ` René Scharfe
2021-06-23  1:52     ` Taylor Blau
2021-08-30 21:17       ` SZEDER Gábor
2021-06-20 20:03 ` [PATCH 7/7] test-lib: enable GIT_TEST_CHECK_PROGRESS by default SZEDER Gábor
2021-06-21  0:59 ` [PATCH 0/7] progress: verify progress counters in the test suite Ævar Arnfjörð Bjarmason
2021-06-23  2:04   ` Taylor Blau
2021-06-23 17:48     ` [PATCH 00/25] progress.c: various fixes + SZEDER's RFC code Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 01/25] progress.c tests: fix breakage with COLUMNS != 80 Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 02/25] progress.c tests: make start/stop verbs on stdin Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 03/25] progress.c tests: test some invalid usage Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 04/25] progress.c tests: add a "signal" verb Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 05/25] progress.c: move signal handler functions lower Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 06/25] progress.c: call progress_interval() from progress_test_force_update() Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 07/25] progress.c: stop eagerly fflush(stderr) when not a terminal Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 08/25] progress.c: add temporary variable from progress struct Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 09/25] midx perf: add a perf test for multi-pack-index Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 10/25] progress.c: remove the "sparse" mode nano-optimization Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 11/25] pack-bitmap-write.c: add a missing stop_progress() Ævar Arnfjörð Bjarmason
2021-09-17  5:14         ` SZEDER Gábor
2021-09-17  5:56           ` Ævar Arnfjörð Bjarmason
2021-09-17 21:38             ` SZEDER Gábor
2021-06-23 17:48       ` [PATCH 12/25] progress.c: add & assert a "global_progress" variable Ævar Arnfjörð Bjarmason
2021-09-16 18:31         ` SZEDER Gábor
2021-06-23 17:48       ` [PATCH 13/25] progress.[ch]: move the "struct progress" to the header Ævar Arnfjörð Bjarmason
2021-09-16 19:42         ` SZEDER Gábor
2021-06-23 17:48       ` [PATCH 14/25] progress.[ch]: move test-only code away from "extern" variables Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 15/25] progress.c: pass "is done?" (again) to display() Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 16/25] progress.[ch]: convert "title" to "struct strbuf" Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 17/25] progress.c: refactor display() for less confusion, and fix bug Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 18/25] progress.c: emit progress on first signal, show "stalled" Ævar Arnfjörð Bjarmason
2021-09-16 18:37         ` SZEDER Gábor
2021-06-23 17:48       ` [PATCH 19/25] commit-graph: fix bogus counter in "Scanning merged commits" progress line Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 20/25] midx: don't provide a total for QSORT() progress Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 21/25] entry: show finer-grained counter in "Filtering content" progress line Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [PATCH 22/25] progress.c: add a stop_progress_early() function Ævar Arnfjörð Bjarmason
2021-06-24 10:35         ` Ævar Arnfjörð Bjarmason
2021-06-25  1:24         ` Andrei Rybak
2021-06-23 17:48       ` [PATCH 23/25] entry: deal with unexpected "Filtering content" total Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [RFC/PATCH 24/25] progress: assert last update in stop_progress() Ævar Arnfjörð Bjarmason
2021-06-23 17:48       ` [RFC/PATCH 25/25] progress: assert counting upwards in display() Ævar Arnfjörð Bjarmason
2021-06-23 17:59       ` [PATCH 00/25] progress.c: various fixes + SZEDER's RFC code Randall S. Becker
2021-06-23 20:01         ` Ævar Arnfjörð Bjarmason
2021-06-23 20:25           ` Randall S. Becker
2021-06-23 21:57 ` [PATCH 0/4] WIP/POC check isatty(2)-protected progress lines SZEDER Gábor
2021-06-23 21:57   ` [PATCH 1/4] WIP progress, isatty(2), hidden progress lnies for GIT_TEST_CHECK_PROGRESS SZEDER Gábor
2021-06-23 21:57   ` [PATCH 2/4] blame: fix progress total with line ranges SZEDER Gábor
2021-06-23 21:57   ` [PATCH 3/4] read-cache: avoid overlapping progress lines SZEDER Gábor
2021-06-23 21:57   ` [PATCH 4/4] preload-index: fix "Refreshing index" progress line SZEDER Gábor
2021-06-23 22:11   ` [PATCH 0/4] WIP/POC check isatty(2)-protected progress lines SZEDER Gábor
2021-06-24 10:43     ` Ævar Arnfjörð Bjarmason
2021-06-24 10:45   ` Ævar Arnfjörð Bjarmason
2021-07-22 12:20 ` [PATCH 0/3] progress.c API users: fix bogus counting Ævar Arnfjörð Bjarmason
2021-07-22 12:20   ` [PATCH 1/3] commit-graph: fix bogus counter in "Scanning merged commits" progress line Ævar Arnfjörð Bjarmason
2021-07-23 21:55     ` Junio C Hamano
2021-08-02 21:07     ` SZEDER Gábor
2021-07-22 12:20   ` [PATCH 2/3] midx: don't provide a total for QSORT() progress Ævar Arnfjörð Bjarmason
2021-07-23 21:56     ` Junio C Hamano
2021-08-05 15:07     ` Phillip Wood
2021-08-05 19:07       ` Ævar Arnfjörð Bjarmason
2021-07-22 12:20   ` [PATCH 3/3] entry: show finer-grained counter in "Filtering content" progress line Ævar Arnfjörð Bjarmason
2021-07-23 22:01     ` Junio C Hamano
2021-08-02 22:05       ` SZEDER Gábor
2021-08-02 21:48     ` SZEDER Gábor
2021-08-05 11:01   ` [PATCH v2 0/3] progress.c API users: fix bogus counting Ævar Arnfjörð Bjarmason
2021-08-05 11:01     ` [PATCH v2 1/3] commit-graph: fix bogus counter in "Scanning merged commits" progress line Ævar Arnfjörð Bjarmason
2021-08-05 11:01     ` [PATCH v2 2/3] midx: don't provide a total for QSORT() progress Ævar Arnfjörð Bjarmason
2021-08-05 11:01     ` [PATCH v2 3/3] entry: show finer-grained counter in "Filtering content" progress line Ævar Arnfjörð Bjarmason
2021-08-23 10:29     ` [PATCH v3 0/2] progress.c API users: fix bogus counting Ævar Arnfjörð Bjarmason
2021-08-23 10:29       ` [PATCH v3 1/2] commit-graph: fix bogus counter in "Scanning merged commits" progress line Ævar Arnfjörð Bjarmason
2021-08-23 10:29       ` [PATCH v3 2/2] entry: show finer-grained counter in "Filtering content" " Ævar Arnfjörð Bjarmason
2021-09-09  1:10       ` [PATCH v4 0/2] progress.c API users: fix bogus counting Ævar Arnfjörð Bjarmason
2021-09-09  1:10         ` [PATCH v4 1/2] commit-graph: fix bogus counter in "Scanning merged commits" progress line Ævar Arnfjörð Bjarmason
2021-09-09  1:10         ` [PATCH v4 2/2] entry: show finer-grained counter in "Filtering content" " Ævar Arnfjörð Bjarmason
2021-09-09 20:02         ` [PATCH v4 0/2] progress.c API users: fix bogus counting Junio C Hamano
2021-07-22 12:54 ` [PATCH 0/8] progress: assert "global_progress" + test fixes / cleanup Ævar Arnfjörð Bjarmason
2021-07-22 12:54   ` [PATCH 1/8] progress.c tests: make start/stop verbs on stdin Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 2/8] progress.c tests: test some invalid usage Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 3/8] progress.c: move signal handler functions lower Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 4/8] progress.c: call progress_interval() from progress_test_force_update() Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 6/8] progress.c: add temporary variable from progress struct Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 7/8] pack-bitmap-write.c: add a missing stop_progress() Ævar Arnfjörð Bjarmason
2021-07-22 12:55   ` [PATCH 8/8] progress.c: add & assert a "global_progress" variable Ævar Arnfjörð Bjarmason
2021-09-16 21:34     ` [PATCH 12/25] " Ævar Arnfjörð Bjarmason
2021-07-23 22:02   ` [PATCH 0/8] progress: assert "global_progress" + test fixes / cleanup Junio C Hamano
2021-09-20 23:09   ` [PATCH v2 " Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 1/8] progress.c tests: make start/stop verbs on stdin Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 2/8] progress.c tests: test some invalid usage Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 3/8] progress.c: move signal handler functions lower Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 4/8] progress.c: call progress_interval() from progress_test_force_update() Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 5/8] progress.c: stop eagerly fflush(stderr) when not a terminal Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 6/8] progress.c: add temporary variable from progress struct Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 7/8] pack-bitmap-write.c: add a missing stop_progress() Ævar Arnfjörð Bjarmason
2021-09-20 23:09     ` [PATCH v2 8/8] progress.c: add & assert a "global_progress" variable Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=0a03bae6-53cf-98e5-635a-6f61583f5dd1@web.de \
    --to=l.s.r@web.de \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=szeder.dev@gmail.com \
    --subject='Re: [PATCH 4/7] commit-graph: fix bogus counter in "Scanning merged commits" progress line' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Code repositories for project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).