git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Derrick Stolee <stolee@gmail.com>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>, git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>, "Jeff King" <peff@peff.net>,
	"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
	"SZEDER Gábor" <szeder.dev@gmail.com>,
	"Eric Sunshine" <sunshine@sunshineco.com>
Subject: Re: [PATCH] commit-graph write: use pack order when finding commits
Date: Thu, 17 Jan 2019 10:09:18 -0500	[thread overview]
Message-ID: <5b1146a1-9ec6-f30e-80e9-cb7cd4dc1a58@gmail.com> (raw)
In-Reply-To: <20190117132345.29791-1-avarab@gmail.com>

On 1/17/2019 8:23 AM, Ævar Arnfjörð Bjarmason wrote:
> Slightly optimize the "commit-graph write" step by using
> FOR_EACH_OBJECT_PACK_ORDER with for_each_object_in_pack(). See commit
> [1] and [2] for the facility and a similar optimization for "cat-file".
>
> On Linux it is around 5% slower to run:
>
>      echo 1 >/proc/sys/vm/drop_caches &&
>      cat .git/objects/pack/* >/dev/null &&
>      git cat-file --batch-all-objects --batch-check --unordered
>
> Than the same thing with the "cat" omitted. This is as expected, since
> we're iterating in pack order and the "cat" is extra work.
>
> Before this change the opposite was true of "commit-graph write". We
> were 6% faster if we first ran "cat" to efficiently populate the FS
> cache for our sole big pack on linux.git than if we had it populated
> via for_each_object_in_pack(). Now we're 3% faster without the "cat"
> instead.
>
> The tests were done on an unloaded Linux 3.10 system with 10 runs for
> each.
>
> 1. 736eb88fdc ("for_each_packed_object: support iterating in
>     pack-order", 2018-08-10)
>
> 2. 0750bb5b51 ("cat-file: support "unordered" output for
>     --batch-all-objects", 2018-08-10)
Thanks, Aevar! I plan to give this a test on the Windows repository, as 
this is the way we generate the commit-graph in VFS for Git.

I created a PR on microsoft/git [1] and am generating the installers now [2]

[1] https://github.com/Microsoft/git/pull/110

[2] https://dev.azure.com/gvfs/ci/_build/results?buildId=6114

The code itself looks good, I just need to double-check the performance 
numbers.

Thanks,
-Stolee

  reply	other threads:[~2019-01-17 15:09 UTC|newest]

Thread overview: 133+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-09-04 20:27 [PATCH 0/2] commit-graph: add progress output Ævar Arnfjörð Bjarmason
2018-09-04 20:27 ` [PATCH 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-09-04 21:16   ` Eric Sunshine
2018-09-04 22:07   ` Junio C Hamano
2018-09-05 11:58     ` Derrick Stolee
2018-09-05 12:07       ` Ævar Arnfjörð Bjarmason
2018-09-05 21:46       ` Junio C Hamano
2018-09-05 22:12         ` Derrick Stolee
2018-09-07 15:11       ` Ævar Arnfjörð Bjarmason
2018-09-07 15:23         ` Ævar Arnfjörð Bjarmason
2018-09-07 17:15           ` Jeff King
2018-09-07 17:25             ` Derrick Stolee
2018-09-05 12:06   ` Derrick Stolee
2018-09-07 12:40   ` Ævar Arnfjörð Bjarmason
2018-09-07 13:12     ` Derrick Stolee
2018-09-04 20:27 ` [PATCH 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason
2018-09-04 22:10   ` Junio C Hamano
2018-09-05 12:07 ` [PATCH 0/2] commit-graph: " Derrick Stolee
2018-09-07 18:29 ` [PATCH v2 " Ævar Arnfjörð Bjarmason
2018-09-11 20:26   ` Junio C Hamano
2018-09-07 18:29 ` [PATCH v2 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-09-21 20:01   ` Derrick Stolee
2018-09-21 21:43     ` Junio C Hamano
2018-09-21 21:57       ` Junio C Hamano
2018-09-07 18:29 ` [PATCH v2 2/2] commit-graph verify: " Ævar Arnfjörð Bjarmason
2018-09-16  6:55   ` Duy Nguyen
2018-09-17 15:33     ` [PATCH v3 0/2] commit-graph: " Ævar Arnfjörð Bjarmason
2018-09-17 15:33     ` [PATCH v3 1/2] commit-graph write: " Ævar Arnfjörð Bjarmason
2018-10-10 20:37       ` SZEDER Gábor
2018-10-10 21:56         ` Ævar Arnfjörð Bjarmason
2018-10-10 22:19           ` SZEDER Gábor
2018-10-10 22:37             ` Ævar Arnfjörð Bjarmason
2018-10-11 17:52               ` Ævar Arnfjörð Bjarmason
2018-10-15 16:05                 ` SZEDER Gábor
2018-10-12  6:09         ` Junio C Hamano
2018-10-12 15:07           ` Ævar Arnfjörð Bjarmason
2018-10-12 15:12             ` Derrick Stolee
2018-10-15 16:54       ` SZEDER Gábor
2018-11-19 16:02         ` SZEDER Gábor
2018-11-19 20:23           ` [PATCH] commit-graph: split up close_reachable() " Ævar Arnfjörð Bjarmason
2018-11-19 20:38             ` Derrick Stolee
2018-11-19 22:57             ` SZEDER Gábor
2018-11-20 15:04               ` [PATCH 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason
2018-11-20 16:58                 ` SZEDER Gábor
2018-11-20 19:50                   ` [PATCH v2 0/6] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 1/6] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 2/6] commit-graph write: add more " Ævar Arnfjörð Bjarmason
2018-11-20 23:38                     ` SZEDER Gábor
2018-11-20 19:50                   ` [PATCH v2 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-20 19:50                   ` [PATCH v2 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason
2018-11-21  1:23                   ` SZEDER Gábor
2018-11-21  1:25                     ` [PATCH 1/2] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' SZEDER Gábor
2018-11-21  3:29                       ` Junio C Hamano
2018-11-21 11:32                         ` Derrick Stolee
2019-01-18 17:05                           ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization SZEDER Gábor
2019-01-18 17:05                             ` [PATCH v2 1/2] commit-graph: rename "large edges" to "extra edges" SZEDER Gábor
2019-01-18 17:05                             ` [PATCH v2 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor
2019-01-19  9:32                               ` Martin Ågren
2019-01-18 19:41                             ` [PATCH v2 0/2] commit-graph: minor cleanup and optimization Junio C Hamano
2018-11-21  1:26                     ` [PATCH 2/2] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily SZEDER Gábor
2018-11-21 11:33                       ` Derrick Stolee
2018-11-22 13:28                       ` [PATCH v3 00/10] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 0/9] commit-graph write: progress output improvements Ævar Arnfjörð Bjarmason
2019-01-19  1:26                             ` Junio C Hamano
2019-01-19 20:21                             ` [PATCH v6 00/10] " Ævar Arnfjörð Bjarmason
2019-01-22 18:30                               ` Derrick Stolee
2019-01-23 17:52                                 ` Junio C Hamano
2019-01-22 19:37                               ` Junio C Hamano
2019-01-19 20:21                             ` [PATCH v6 01/10] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 02/10] commit-graph: rename "large edges" to "extra edges" Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 03/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2019-01-23 17:51                               ` [PATCH v6.1 03/10] commit-graph: don't call write_graph_chunk_extra_edges() unnecessarily SZEDER Gábor
2019-01-19 20:21                             ` [PATCH v6 04/10] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2019-01-19 20:21                             ` [PATCH v6 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2019-03-22 10:28                               ` [PATCH] commit-graph: don't show progress percentages while expanding reachable commits SZEDER Gábor
2019-03-22 11:11                                 ` Ævar Arnfjörð Bjarmason
2019-03-22 11:18                                   ` SZEDER Gábor
2019-03-22 14:28                                     ` Ævar Arnfjörð Bjarmason
2019-03-22 14:36                                       ` Ævar Arnfjörð Bjarmason
2019-03-22 14:55                                       ` SZEDER Gábor
2019-03-22 15:11                                         ` Ævar Arnfjörð Bjarmason
2019-03-22 15:49                                           ` SZEDER Gábor
2019-03-22 16:52                                             ` SZEDER Gábor
2019-03-22 17:23                                             ` Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 1/9] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 2/9] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 3/9] commit-graph write: add "Writing out" progress output Ævar Arnfjörð Bjarmason
2019-01-18 17:16                             ` SZEDER Gábor
2019-01-16 13:29                           ` [PATCH v5 4/9] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 5/9] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 6/9] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 7/9] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 8/9] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2019-01-16 13:29                           ` [PATCH v5 9/9] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2019-01-17 13:23                             ` [PATCH] commit-graph write: use pack order when finding commits Ævar Arnfjörð Bjarmason
2019-01-17 15:09                               ` Derrick Stolee [this message]
2019-01-17 16:35                                 ` Derrick Stolee
2018-11-22 15:39                         ` [PATCH v4 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2018-11-22 15:39                         ` [PATCH v4 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2018-11-22 18:59                         ` [PATCH v3 00/10] commit-graph write: progress output improvements Eric Sunshine
2018-11-22 13:28                       ` [PATCH v3 01/10] commit-graph: rename 'num_extra_edges' variable to 'num_large_edges' Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 02/10] commit-graph: don't call write_graph_chunk_large_edges() unnecessarily Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 03/10] commit-graph write: rephrase confusing progress output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 04/10] commit-graph write: add "Writing out" " Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 05/10] commit-graph write: more descriptive "writing out" output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 06/10] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 07/10] commit-graph write: add more descriptive progress output Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 08/10] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 09/10] commit-graph write: add itermediate progress Ævar Arnfjörð Bjarmason
2018-11-22 13:28                       ` [PATCH v3 10/10] commit-graph write: emit a percentage for all progress Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 3/6] commit-graph write: show progress for object search Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 4/6] commit-graph write: add more describing progress output Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 5/6] commit-graph write: remove empty line for readability Ævar Arnfjörð Bjarmason
2018-11-20 15:04               ` [PATCH 6/6] commit-graph write: add even more progress output Ævar Arnfjörð Bjarmason
2018-09-17 15:33     ` [PATCH v3 2/2] commit-graph verify: add " Ævar Arnfjörð Bjarmason

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5b1146a1-9ec6-f30e-80e9-cb7cd4dc1a58@gmail.com \
    --to=stolee@gmail.com \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=sunshine@sunshineco.com \
    --cc=szeder.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).