From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: Derrick Stolee <stolee@gmail.com>
Cc: "Junio C Hamano" <gitster@pobox.com>,
"SZEDER Gábor" <szeder.dev@gmail.com>,
git@vger.kernel.org, "Jeff King" <peff@peff.net>,
"Stefan Beller" <sbeller@google.com>,
"Duy Nguyen" <pclouds@gmail.com>,
"Jonathan Tan" <jonathantanmy@google.com>
Subject: Re: [PATCH 0/4] Bloom filter experiment
Date: Tue, 16 Oct 2018 14:57:30 +0200 [thread overview]
Message-ID: <87zhvecamd.fsf@evledraar.gmail.com> (raw)
In-Reply-To: <8c66cbe3-6830-05cb-f3bb-be2e4902e8f5@gmail.com>
On Tue, Oct 16 2018, Derrick Stolee wrote:
> On 10/16/2018 12:45 AM, Junio C Hamano wrote:
>> Derrick Stolee <stolee@gmail.com> writes:
>>
>>> 2. The filters are sized according to the number of changes in each
>>> commit, with a minimum of one 64-bit word.
>>> ...
>>> 6. When we compute the Bloom filters, we don't store a filter for
>>> commits whose first-parent diff has more than 512 paths.
>> Just being curious but was 512 taken out of thin air or is there
>> some math behind it, e.g. to limit false positive rate down to
>> certain threshold? With a wide-enough bitset, you could store
>> arbitrary large number of paths with low enough false positive, I
>> guess, but is there a point where there is too many paths in the
>> change that gives us diminishing returns and not worth having a
>> filter in the first place?
> 512 is somewhat arbitrary, but having a maximum size is not.
>> In a normal source-code-control context, the set of paths modified
>> by any single commit ought to be a small subset of the entire paths,
>> and whole-tree changes ought to be fairly rare. In a project for
>> which that assumption does not hold, it might help to have a
>> negative bloom filter (i.e. "git log -- A" asks "does the commit
>> modify A?" and the filter would say "we know it does not, because we
>> threw all the paths that are not touched to the bloom filter"), but
>> I think that would optimize for a wrong case.
>
> A commit with many changed paths is very rare. The 512 I picked above
> is enough to cover 99% of commits in each of the repos I sampled when
> first investigating Bloom filters.
>
> When a Bloom filter response says "maybe yes" (in our case, "maybe not
> TREESAME"), then we need to verify that it is correct. In the extreme
> case that every path is changed, then the Bloom filter does nothing
> but add extra work.
>
> These extreme cases are also not unprecedented: in our Azure Repos
> codebase, we were using core.autocrlf to smudge CRLFs to LFs, but
> when it was time to dogfood VFS for Git, we needed to turn off the
> smudge filter. So, there is one commit that converts every LF to a
> CRLF in every text file. Storing a Bloom filter for those ~250,000
> entries would take ~256KB for essentially no value. By not storing a
> filter for this commit, we go immediately to the regular TREESAME
> check, which would happen for most pathspecs.
>
> This is all to say: having a maximum size is good. 512 is big enough
> to cover _most_ commits, but not so big that we may store _really_ big
> filters.
Makes sense. 512 is good enough to hardcode initially, but I couldn't
tell from briefly skimming the patches if it was possible to make this
size dynamic per-repo when the graph/filter is written.
I.e. we might later add some discovery step where we look at N number of
commits at random, until we're satisfied that we've come up with some
average/median number of total (recursive) tree entries & how many tend
to be changed per-commit.
I.e. I can imagine repositories (with some automated changes) where we
have 10k files and tend to change 1k per commit, or ones with 10k files
where we tend to change just 1-10 per commit, which would mean a
larger/smaller filter would be needed / would do.
next prev parent reply other threads:[~2018-10-16 12:57 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-10-03 13:23 We should add a "git gc --auto" after "git clone" due to commit graph Ævar Arnfjörð Bjarmason
2018-10-03 13:36 ` SZEDER Gábor
2018-10-03 13:42 ` Derrick Stolee
2018-10-03 14:18 ` Ævar Arnfjörð Bjarmason
2018-10-03 14:01 ` Ævar Arnfjörð Bjarmason
2018-10-03 14:17 ` SZEDER Gábor
2018-10-03 14:22 ` Ævar Arnfjörð Bjarmason
2018-10-03 14:53 ` SZEDER Gábor
2018-10-03 15:19 ` Ævar Arnfjörð Bjarmason
2018-10-03 16:59 ` SZEDER Gábor
2018-10-05 6:09 ` Junio C Hamano
2018-10-10 22:07 ` SZEDER Gábor
2018-10-10 23:01 ` Ævar Arnfjörð Bjarmason
2018-10-03 19:08 ` Stefan Beller
2018-10-03 19:21 ` Jeff King
2018-10-03 20:35 ` Ævar Arnfjörð Bjarmason
2018-10-03 17:47 ` Stefan Beller
2018-10-03 18:47 ` Ævar Arnfjörð Bjarmason
2018-10-03 18:51 ` Jeff King
2018-10-03 18:59 ` Derrick Stolee
2018-10-03 19:18 ` Jeff King
2018-10-08 16:41 ` SZEDER Gábor
2018-10-08 16:57 ` Derrick Stolee
2018-10-08 18:10 ` SZEDER Gábor
2018-10-08 18:29 ` Derrick Stolee
2018-10-09 3:08 ` Jeff King
2018-10-09 13:48 ` Bloom Filters (was Re: We should add a "git gc --auto" after "git clone" due to commit graph) Derrick Stolee
2018-10-09 18:45 ` Ævar Arnfjörð Bjarmason
2018-10-09 18:46 ` Jeff King
2018-10-09 19:03 ` Derrick Stolee
2018-10-09 21:14 ` Jeff King
2018-10-09 23:12 ` Bloom Filters Jeff King
2018-10-09 23:13 ` [PoC -- do not apply 1/3] initial tree-bitmap proof of concept Jeff King
2018-10-09 23:14 ` [PoC -- do not apply 2/3] test-tree-bitmap: add "dump" mode Jeff King
2018-10-10 0:48 ` Junio C Hamano
2018-10-11 3:13 ` Jeff King
2018-10-09 23:14 ` [PoC -- do not apply 3/3] test-tree-bitmap: replace ewah with custom rle encoding Jeff King
2018-10-10 0:58 ` Junio C Hamano
2018-10-11 3:20 ` Jeff King
2018-10-11 12:33 ` Bloom Filters Derrick Stolee
2018-10-11 13:43 ` Jeff King
2018-10-09 21:30 ` We should add a "git gc --auto" after "git clone" due to commit graph SZEDER Gábor
2018-10-09 19:34 ` [PATCH 0/4] Bloom filter experiment SZEDER Gábor
2018-10-09 19:34 ` [PATCH 1/4] Add a (very) barebones Bloom filter implementation SZEDER Gábor
2018-10-09 19:34 ` [PATCH 2/4] commit-graph: write a Bloom filter containing changed paths for each commit SZEDER Gábor
2018-10-09 21:06 ` Jeff King
2018-10-09 21:37 ` SZEDER Gábor
2018-10-09 19:34 ` [PATCH 3/4] revision.c: use the Bloom filter to speed up path-limited revision walks SZEDER Gábor
2018-10-09 19:34 ` [PATCH 4/4] revision.c: add GIT_TRACE_BLOOM_FILTER for a bit of statistics SZEDER Gábor
2018-10-09 19:47 ` [PATCH 0/4] Bloom filter experiment Derrick Stolee
2018-10-11 1:21 ` [PATCH 0/2] Per-commit filter proof of concept Jonathan Tan
2018-10-11 1:21 ` [PATCH 1/2] One filter per commit Jonathan Tan
2018-10-11 12:49 ` Derrick Stolee
2018-10-11 19:11 ` [PATCH] Per-commit and per-parent filters for 2 parents Jonathan Tan
2018-10-11 1:21 ` [PATCH 2/2] Only make bloom filter for first parent Jonathan Tan
2018-10-11 7:37 ` [PATCH 0/2] Per-commit filter proof of concept Ævar Arnfjörð Bjarmason
2018-10-15 14:39 ` [PATCH 0/4] Bloom filter experiment Derrick Stolee
2018-10-16 4:45 ` Junio C Hamano
2018-10-16 11:13 ` Derrick Stolee
2018-10-16 12:57 ` Ævar Arnfjörð Bjarmason [this message]
2018-10-16 13:03 ` Derrick Stolee
2018-10-18 2:00 ` Junio C Hamano
2018-10-16 23:41 ` Jonathan Tan
2018-10-08 23:02 ` We should add a "git gc --auto" after "git clone" due to commit graph Junio C Hamano
2018-10-03 14:32 ` Duy Nguyen
2018-10-03 16:45 ` Duy Nguyen
2018-10-04 21:42 ` [RFC PATCH] " Ævar Arnfjörð Bjarmason
2018-10-05 12:05 ` Derrick Stolee
2018-10-05 13:05 ` Ævar Arnfjörð Bjarmason
2018-10-05 13:45 ` Derrick Stolee
2018-10-05 14:04 ` Ævar Arnfjörð Bjarmason
2018-10-05 19:21 ` Jeff King
2018-10-05 19:41 ` Derrick Stolee
2018-10-05 19:47 ` Jeff King
2018-10-05 20:00 ` Derrick Stolee
2018-10-05 20:02 ` Jeff King
2018-10-05 20:01 ` Ævar Arnfjörð Bjarmason
2018-10-05 20:09 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87zhvecamd.fsf@evledraar.gmail.com \
--to=avarab@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jonathantanmy@google.com \
--cc=pclouds@gmail.com \
--cc=peff@peff.net \
--cc=sbeller@google.com \
--cc=stolee@gmail.com \
--cc=szeder.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).