From: Jeff King <peff@peff.net>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Cc: "Michael Haggerty" <mhagger@alum.mit.edu>,
"Junio C Hamano" <gitster@pobox.com>,
"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
"David Turner" <novalis@novalis.org>,
"Git Mailing List" <git@vger.kernel.org>
Subject: Re: [PATCH 00/20] Separate `ref_cache` into a separate module
Date: Mon, 20 Mar 2017 14:30:23 -0400 [thread overview]
Message-ID: <20170320183022.mlv2y5g6pn2i7znq@sigill.intra.peff.net> (raw)
In-Reply-To: <CACBZZX7Kqo=w-c2LOH9a4xf6WpnhE5tsvNHa1O2-4dvYTKZxEA@mail.gmail.com>
On Mon, Mar 20, 2017 at 07:24:12PM +0100, Ævar Arnfjörð Bjarmason wrote:
> On Mon, Mar 20, 2017 at 5:33 PM, Michael Haggerty <mhagger@alum.mit.edu> wrote:
>
> > I've completed a draft of an epic 48-patch series implementing all of
> > the above points on my GitHub fork [1] as branch
> > `wip/mmap-packed-refs`. It dramatically speeds up performance and
> > reduces memory usage for some tasks in repositories with very many
> > packed references.
>
> Just curious, what sort of performance numbers on mainly what sort of
> operations? I'd expect the fixed cost of e.g. `git log -1 >/dev/null`
> to go down, which can take as long as a second on some repos with a
> huge amount of refs, but anything else?
Anything that reads refs (and that includes anything that touches an
object, because it looks at refs/replace) loads the entire packed-refs
file into memory. So I'd expect you're paying that "1 second" cost on a
lot of commands.
Even commands that traverse the entire ref namespace should improve, as
we'd spend a lot less time allocating and copying data into the
in-memory refs cache. Not to mention that the memory usage is reduced,
though that probably only really matters for pathological cases.
-Peff
next prev parent reply other threads:[~2017-03-20 18:30 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-20 16:33 [PATCH 00/20] Separate `ref_cache` into a separate module Michael Haggerty
2017-03-20 16:33 ` [PATCH 01/20] get_ref_dir(): don't call read_loose_refs() for "refs/bisect" Michael Haggerty
2017-03-20 16:33 ` [PATCH 02/20] refs_read_raw_ref(): new function Michael Haggerty
2017-03-20 16:33 ` [PATCH 03/20] refs_ref_iterator_begin(): " Michael Haggerty
2017-03-20 16:33 ` [PATCH 04/20] refs_verify_refname_available(): implement once for all backends Michael Haggerty
2017-03-20 17:42 ` Jeff King
2017-03-20 22:20 ` Michael Haggerty
2017-03-20 16:33 ` [PATCH 05/20] refs_verify_refname_available(): use function in more places Michael Haggerty
2017-03-20 16:33 ` [PATCH 06/20] Rename `add_ref()` to `add_ref_entry()` Michael Haggerty
2017-03-20 16:33 ` [PATCH 07/20] Rename `find_ref()` to `find_ref_entry()` Michael Haggerty
2017-03-20 16:33 ` [PATCH 08/20] Rename `remove_entry()` to `remove_entry_from_dir()` Michael Haggerty
2017-03-20 16:33 ` [PATCH 09/20] refs: split `ref_cache` code into separate files Michael Haggerty
2017-03-20 17:49 ` Jeff King
2017-03-20 19:47 ` Junio C Hamano
2017-03-20 20:35 ` Stefan Beller
2017-03-20 22:40 ` Junio C Hamano
2017-03-20 16:33 ` [PATCH 10/20] ref-cache: introduce a new type, ref_cache Michael Haggerty
2017-03-20 16:33 ` [PATCH 11/20] refs: record the ref_store in ref_cache, not ref_dir Michael Haggerty
2017-03-20 17:51 ` Jeff King
2017-03-20 22:39 ` Michael Haggerty
2017-03-20 16:33 ` [PATCH 12/20] ref-cache: use a callback function to fill the cache Michael Haggerty
2017-03-20 16:33 ` [PATCH 13/20] refs: handle "refs/bisect/" in `loose_fill_ref_dir()` Michael Haggerty
2017-03-20 16:33 ` [PATCH 14/20] do_for_each_entry_in_dir(): eliminate `offset` argument Michael Haggerty
2017-03-20 16:33 ` [PATCH 15/20] get_loose_ref_dir(): function renamed from get_loose_refs() Michael Haggerty
2017-03-20 16:33 ` [PATCH 16/20] get_loose_ref_cache(): new function Michael Haggerty
2017-03-20 16:33 ` [PATCH 17/20] cache_ref_iterator_begin(): make function smarter Michael Haggerty
2017-03-20 16:33 ` [PATCH 18/20] commit_packed_refs(): use reference iteration Michael Haggerty
2017-03-20 18:05 ` Jeff King
2017-03-22 8:42 ` Michael Haggerty
2017-03-22 13:06 ` Jeff King
2017-03-20 16:33 ` [PATCH 19/20] files_pack_refs(): " Michael Haggerty
2017-03-20 16:33 ` [PATCH 20/20] do_for_each_entry_in_dir(): delete function Michael Haggerty
2017-03-20 17:25 ` [PATCH 00/20] Separate `ref_cache` into a separate module Junio C Hamano
2017-03-20 18:12 ` Jeff King
2017-03-20 18:24 ` Ævar Arnfjörð Bjarmason
2017-03-20 18:30 ` Jeff King [this message]
2017-03-20 22:32 ` Junio C Hamano
2017-03-20 22:48 ` Michael Haggerty
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170320183022.mlv2y5g6pn2i7znq@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=mhagger@alum.mit.edu \
--cc=novalis@novalis.org \
--cc=pclouds@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).