From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
"Nguyen Thai Ngoc Duy" <pclouds@gmail.com>,
"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
"Ramsay Jones" <ramsay@ramsayjones.plus.com>,
"Jeff King" <peff@peff.net>,
"Christian Couder" <chriscool@tuxfamily.org>
Subject: [PATCH v5 00/22] Add configuration options for split-index
Date: Mon, 6 Mar 2017 10:41:41 +0100 [thread overview]
Message-ID: <20170306094203.28250-1-chriscool@tuxfamily.org> (raw)
Goal
~~~~
We want to make it possible to use the split-index feature
automatically by just setting a new "core.splitIndex" configuration
variable to true.
This can be valuable as split-index can help significantly speed up
`git rebase` especially along with the work to libify `git apply`
that has been merged to master
(see https://github.com/git/git/commit/81358dc238372793b1590efa149cc1581d1fbd98)
and is now in v2.11.
Design
~~~~~~
The design is similar as the previous work that introduced
"core.untrackedCache".
The new "core.splitIndex" configuration option can be either true,
false or undefined which is the default.
When it is true, the split index is created, if it does not already
exists, when the index is read. When it is false, the split index is
removed if it exists, when the index is read. Otherwise it is left as
is.
Along with this new configuration variable, the two following options
are also introduced:
- splitIndex.maxPercentChange
This is to avoid having too many changes accumulating in the split
index while in split index mode. The git-update-index
documentation says:
If split-index mode is already enabled and `--split-index` is
given again, all changes in $GIT_DIR/index are pushed back to
the shared index file.
but it is probably better to not expect the user to think about it
and to have a mechanism that pushes back all changes to the shared
index file automatically when some threshold is reached.
The default threshold is when the number of entries in the split
index file reaches 20% of the number of entries in the shared
index file. The new "splitIndex.maxPercentChange" config option
lets people tweak this value.
- splitIndex.sharedIndexExpire
To make sure that old sharedindex files are eventually removed
when a new one has been created, we "touch" the shared index file
every time a split index file using the shared index file is
either created or read from. Then we can delete shared indexes
with an mtime older than two weeks (by default), when we create a
new shared index file. The new "splitIndex.sharedIndexExpire"
config option lets people tweak this grace period.
This idea was suggested by Duy in:
https://public-inbox.org/git/CACsJy8BqMFASHf5kJgUh+bd7XG98CafNydE964VJyPXz-emEvA@mail.gmail.com/
and after some experiments, I agree that it is much simpler than
what I thought could be done during our discussion.
Junio also thinks that we have to do "time-based GC" in:
https://public-inbox.org/git/xmqqeg33ccjj.fsf@gitster.mtv.corp.google.com/
Note that this patch series doesn't address a leak when removing a
split-index, but Duy wrote that he has a patch to fix this leak:
https://public-inbox.org/git/CACsJy8AisF2ZVs7JdnVFp5wdskkbVQQQ=DBq5UzE1MOsCfBMtQ@mail.gmail.com/
Highlevel view of the patches in the series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are very few and very small differences between this patch
series and the v4 patch series sent a few weeks ago. The small
differences are all in patches 15/22 and 17/22. The other patches are
the same as in v4.
Except for patch 1/22 and 2/22, there are 3 big steps, one for each
new configuration variable introduced.
- Patch 1/22 marks a message for translation and can be applied
separately.
- Patch 2/22 improves the existing indentation style of t1700 by
using different here document style.
Step 1 is:
- Patches 3/22 to 6/22 introduce the functions that are reading
the "core.splitIndex" configuration variable and tweaking the
split index depending on its value.
- Patch 7/22 adds a few tests for the new feature.
- Patches 8/22 and 9/22 add some documentation for the new
feature.
Step 2 is:
- Patches 10/22 and 11/22 introduce the functions that are reading
the "splitIndex.maxPercentChange" configuration variable and
regenerating a new shared index file depending on its value.
- Patch 12/22 adds a few tests for the new feature.
- Patch 13/22 add some documentation for the new feature.
Step 3 is:
- Patches 14/22 to 17/22 introduce the functions that are reading
the "splitIndex.sharedIndexExpire" configuration variable and
expiring old shared index files depending on its value.
In patch 15/22 a warning message starts with "could" instead of
"Could" as suggested by Junio.
In patch 17/22, can_delete_shared_index() has been renamed
should_delete_shared_index() and error_errno() has been changed
to warning_errno() as suggested by Junio.
- Patch 18/22 adds a few tests for the new feature.
- Patches 19/22 and 20/22 update the mtime of the shared index
file when a split index based on the shared index file is read
from. 19/22 is a refactoring to make the actual change in 20/22
easier.
- Patches 21/22 and 22/22 add some documentation for the new
feature.
Links
~~~~~
This patch series is also available here:
https://github.com/chriscool/git/commits/config-split-index
The previous versions were:
RFC: https://github.com/chriscool/git/commits/config-split-index7
v1: https://github.com/chriscool/git/commits/config-split-index72
v2: https://github.com/chriscool/git/commits/config-split-index99
v3: https://github.com/chriscool/git/commits/config-split-index102
v4: https://github.com/chriscool/git/commits/config-split-index121
On the mailing list the related patch series and discussions were:
RFC: https://public-inbox.org/git/20160711172254.13439-1-chriscool@tuxfamily.org/
v1: https://public-inbox.org/git/20161023092648.12086-1-chriscool@tuxfamily.org/
v2: https://public-inbox.org/git/20161217145547.11748-1-chriscool@tuxfamily.org/
v3: https://public-inbox.org/git/20161226102222.17150-1-chriscool@tuxfamily.org/
v4: https://public-inbox.org/git/20170227180019.18666-1-chriscool@tuxfamily.org/
Christian Couder (22):
config: mark an error message up for translation
t1700: change here document style
config: add git_config_get_split_index()
split-index: add {add,remove}_split_index() functions
read-cache: add and then use tweak_split_index()
update-index: warn in case of split-index incoherency
t1700: add tests for core.splitIndex
Documentation/config: add information for core.splitIndex
Documentation/git-update-index: talk about core.splitIndex config var
config: add git_config_get_max_percent_split_change()
read-cache: regenerate shared index if necessary
t1700: add tests for splitIndex.maxPercentChange
Documentation/config: add splitIndex.maxPercentChange
sha1_file: make check_and_freshen_file() non static
read-cache: touch shared index files when used
config: add git_config_get_expiry() from gc.c
read-cache: unlink old sharedindex files
t1700: test shared index file expiration
read-cache: refactor read_index_from()
read-cache: use freshen_shared_index() in read_index_from()
Documentation/config: add splitIndex.sharedIndexExpire
Documentation/git-update-index: explain splitIndex.*
Documentation/config.txt | 29 ++++
Documentation/git-update-index.txt | 43 ++++-
builtin/gc.c | 18 +--
builtin/update-index.c | 25 +--
cache.h | 8 +
config.c | 42 ++++-
read-cache.c | 157 ++++++++++++++++--
sha1_file.c | 2 +-
split-index.c | 22 +++
split-index.h | 2 +
t/t1700-split-index.sh | 324 +++++++++++++++++++++++++++----------
11 files changed, 540 insertions(+), 132 deletions(-)
--
2.12.0.206.g74921e51d6.dirty
next reply other threads:[~2017-03-06 10:19 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-03-06 9:41 Christian Couder [this message]
2017-03-06 9:41 ` [PATCH v5 01/22] config: mark an error message up for translation Christian Couder
2017-03-06 9:41 ` [PATCH v5 02/22] t1700: change here document style Christian Couder
2017-03-06 9:41 ` [PATCH v5 03/22] config: add git_config_get_split_index() Christian Couder
2017-03-06 9:41 ` [PATCH v5 04/22] split-index: add {add,remove}_split_index() functions Christian Couder
2017-03-06 9:41 ` [PATCH v5 05/22] read-cache: add and then use tweak_split_index() Christian Couder
2017-03-06 9:41 ` [PATCH v5 06/22] update-index: warn in case of split-index incoherency Christian Couder
2017-03-06 9:41 ` [PATCH v5 07/22] t1700: add tests for core.splitIndex Christian Couder
2017-03-06 9:41 ` [PATCH v5 08/22] Documentation/config: add information " Christian Couder
2017-03-06 9:41 ` [PATCH v5 09/22] Documentation/git-update-index: talk about core.splitIndex config var Christian Couder
2017-03-06 9:41 ` [PATCH v5 10/22] config: add git_config_get_max_percent_split_change() Christian Couder
2017-03-06 9:41 ` [PATCH v5 11/22] read-cache: regenerate shared index if necessary Christian Couder
2017-03-06 9:41 ` [PATCH v5 12/22] t1700: add tests for splitIndex.maxPercentChange Christian Couder
2017-03-06 9:41 ` [PATCH v5 13/22] Documentation/config: add splitIndex.maxPercentChange Christian Couder
2017-03-06 9:41 ` [PATCH v5 14/22] sha1_file: make check_and_freshen_file() non static Christian Couder
2017-03-06 9:41 ` [PATCH v5 15/22] read-cache: touch shared index files when used Christian Couder
2017-03-06 9:41 ` [PATCH v5 16/22] config: add git_config_get_expiry() from gc.c Christian Couder
2017-03-06 9:41 ` [PATCH v5 17/22] read-cache: unlink old sharedindex files Christian Couder
2017-03-06 9:41 ` [PATCH v5 18/22] t1700: test shared index file expiration Christian Couder
2017-03-06 9:42 ` [PATCH v5 19/22] read-cache: refactor read_index_from() Christian Couder
2017-03-06 9:42 ` [PATCH v5 20/22] read-cache: use freshen_shared_index() in read_index_from() Christian Couder
2017-03-06 9:42 ` [PATCH v5 21/22] Documentation/config: add splitIndex.sharedIndexExpire Christian Couder
2017-03-06 9:42 ` [PATCH v5 22/22] Documentation/git-update-index: explain splitIndex.* Christian Couder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20170306094203.28250-1-chriscool@tuxfamily.org \
--to=christian.couder@gmail.com \
--cc=avarab@gmail.com \
--cc=chriscool@tuxfamily.org \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=pclouds@gmail.com \
--cc=peff@peff.net \
--cc=ramsay@ramsayjones.plus.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).