git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Brandon Williams <bmwill@google.com>
To: Thomas Gummerer <t.gummerer@gmail.com>
Cc: git@vger.kernel.org, "Lars Schneider" <larsxschneider@gmail.com>,
	"Jeff King" <peff@peff.net>, "Junio C Hamano" <gitster@pobox.com>,
	"Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>,
	"SZEDER Gábor" <szeder.dev@gmail.com>
Subject: Re: [PATCH v3 1/3] read-cache: fix reading the shared index for other repos
Date: Tue, 16 Jan 2018 13:42:39 -0800	[thread overview]
Message-ID: <20180116214239.GA3622@google.com> (raw)
In-Reply-To: <20180107223015.17720-2-t.gummerer@gmail.com>

On 01/07, Thomas Gummerer wrote:
> read_index_from() takes a path argument for the location of the index
> file.  For reading the shared index in split index mode however it just
> ignores that path argument, and reads it from the gitdir of the current
> repository.
> 
> This works as long as an index in the_repository is read.  Once that
> changes, such as when we read the index of a submodule, or of a
> different working tree than the current one, the gitdir of
> the_repository will no longer contain the appropriate shared index,
> and git will fail to read it.
> 
> For example t3007-ls-files-recurse-submodules.sh was broken with
> GIT_TEST_SPLIT_INDEX set in 188dce131f ("ls-files: use repository
> object", 2017-06-22), and t7814-grep-recurse-submodules.sh was also
> broken in a similar manner, probably by introducing struct repository
> there, although I didn't track down the exact commit for that.
> 
> be489d02d2 ("revision.c: --indexed-objects add objects from all
> worktrees", 2017-08-23) breaks with split index mode in a similar
> manner, not erroring out when it can't read the index, but instead
> carrying on with pruning, without taking the index of the worktree into
> account.
> 
> Fix this by passing an additional gitdir parameter to read_index_from,
> to indicate where it should look for and read the shared index from.
> 
> read_cache_from() defaults to using the gitdir of the_repository.  As it
> is mostly a convenience macro, having to pass get_git_dir() for every
> call seems overkill, and if necessary users can have more control by
> using read_index_from().

I'm not saying we need to change this again but I got to thinking about
what the root cause for this bug is and I think it's a design flaw in
how split index is implemented.  IIUC Split index is an index extension
that can be enabled to limit the size of the index file that is written
when making changes to the index.  It breaks the index into two pieces,
index (which contains only changes) and sharedindex.XXXXX (which
contains unchanged information) where 'XXXXX' is a value found in the
index file.  If we don't do anything fancy then these two files live
next to one another in a repository's git directory at $GIT_DIR/index
and $GIT_DIR/sharedindex.XXXXX.  This seems to work all well and fine
except that this isn't always the case and the read_index_from function
takes this into account by enabling a caller to specify a path to where
the index file is located.  We can do this by specifying the index file
we want to use by setting GIT_INDEX_FILE.

Being able to specify an index file via the environment is a feature
that has existed for a very long time (one that I personally think makes
things difficult because of things like additions like the sharedindex)
and I don't think was taken into account when introducing the split
index extension.  In this case if i were to specify a location of an
index file in my home directory '~/index' and be using the split index
feature then the corresponding sharedindex file would live in my
repository's git directory '~/project/.git/sharedindex.XXXXX'.  So the
sharedindex file is always located relative to the project's git
directory and not the index file itself, which is kind of confusing.
Maybe a better design would be to have the sharedindex file located
relative to the index file.

Anyway, some food for thought.

-- 
Brandon Williams

  parent reply	other threads:[~2018-01-16 21:42 UTC|newest]

Thread overview: 95+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2017-12-10 21:21 [PATCH 0/3] fixes for split index mode Thomas Gummerer
2017-12-10 21:22 ` [PATCH 1/3] repository: fix repo_read_index with submodules Thomas Gummerer
2017-12-11 18:54   ` Brandon Williams
2017-12-11 20:37     ` Thomas Gummerer
2017-12-10 21:22 ` [PATCH 2/3] prune: fix pruning with multiple worktrees and split index Thomas Gummerer
2017-12-11 19:09   ` Brandon Williams
2017-12-11 21:39     ` Thomas Gummerer
2017-12-10 21:22 ` [PATCH 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX Thomas Gummerer
2017-12-10 23:37   ` Eric Sunshine
2017-12-11 21:09   ` SZEDER Gábor
2017-12-11 21:42     ` Thomas Gummerer
2017-12-12 15:54       ` Lars Schneider
2017-12-12 19:15         ` Junio C Hamano
2017-12-12 20:15           ` Thomas Gummerer
2017-12-12 20:51             ` Junio C Hamano
2017-12-13 23:21               ` Thomas Gummerer
2017-12-13 17:21           ` Lars Schneider
2017-12-13 17:38             ` Junio C Hamano
2017-12-13 17:46               ` Lars Schneider
2017-12-13 23:28                 ` Thomas Gummerer
2017-12-17 22:51 ` [PATCH v2 0/3] fixes for split index mode Thomas Gummerer
2017-12-17 22:51   ` [PATCH v2 1/3] repository: fix repo_read_index with submodules Thomas Gummerer
2017-12-18 18:01     ` Brandon Williams
2017-12-18 23:05       ` Thomas Gummerer
2017-12-18 23:05         ` Brandon Williams
2017-12-17 22:51   ` [PATCH v2 2/3] prune: fix pruning with multiple worktrees and split index Thomas Gummerer
2017-12-18 18:19     ` Brandon Williams
2018-01-03 22:18       ` Thomas Gummerer
2018-01-04 19:12         ` Junio C Hamano
2017-12-17 22:51   ` [PATCH v2 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX Thomas Gummerer
2017-12-18 18:16     ` Lars Schneider
2018-01-04 20:13       ` Thomas Gummerer
2018-01-05 11:03         ` Lars Schneider
2018-01-07 20:02         ` Thomas Gummerer
2018-01-07 22:30   ` [PATCH v3 0/3] fixes for split index mode Thomas Gummerer
2018-01-07 22:30     ` [PATCH v3 1/3] read-cache: fix reading the shared index for other repos Thomas Gummerer
2018-01-08 10:41       ` Duy Nguyen
2018-01-08 22:41         ` Thomas Gummerer
2018-01-13 22:33           ` Thomas Gummerer
2018-01-08 23:38         ` Brandon Williams
2018-01-09  1:24           ` Duy Nguyen
2018-01-16 21:42       ` Brandon Williams [this message]
2018-01-17  0:16         ` Duy Nguyen
2018-01-17  0:32           ` Brandon Williams
2018-01-17 18:16           ` Jonathan Nieder
2018-01-18 10:19             ` Duy Nguyen
2018-01-19 21:57       ` Junio C Hamano
2018-01-20 11:58         ` Thomas Gummerer
2018-01-22  6:14           ` Junio C Hamano
2018-01-27 12:18             ` Thomas Gummerer
2018-02-07 22:41             ` Junio C Hamano
2018-01-07 22:30     ` [PATCH v3 2/3] split-index: don't write cache tree with null oid entries Thomas Gummerer
2018-01-07 22:30     ` [PATCH v3 3/3] travis: run tests with GIT_TEST_SPLIT_INDEX Thomas Gummerer
2018-01-13 22:37     ` [PATCH v3 4/3] read-cache: don't try to write index if we can't write shared index Thomas Gummerer
2018-01-14  9:36       ` Duy Nguyen
2018-01-14 10:18         ` [PATCH 1/3] read-cache.c: change type of "temp" in write_shared_index() Nguyễn Thái Ngọc Duy
2018-01-14 10:18           ` [PATCH 2/3] read-cache.c: move tempfile creation/cleanup out of write_shared_index Nguyễn Thái Ngọc Duy
2018-01-14 10:18           ` [PATCH 3/3] read-cache: don't write index twice if we can't write shared index Nguyễn Thái Ngọc Duy
2018-01-18 11:36             ` SZEDER Gábor
2018-01-18 12:47               ` Duy Nguyen
2018-01-18 13:29                 ` Jeff King
2018-01-18 13:36                   ` Duy Nguyen
2018-01-18 15:00                     ` Duy Nguyen
2018-01-18 21:37                       ` Jeff King
2018-01-18 22:32                         ` SZEDER Gábor
2018-01-19  0:30                           ` Duy Nguyen
2018-01-22 13:32                           ` [PATCH 0/5] Travis CI: don't run the test suite as root in the 32 bit Linux build SZEDER Gábor
2018-01-22 13:32                             ` [PATCH 1/5] travis-ci: use 'set -x' for the commands under 'su' " SZEDER Gábor
2018-01-22 13:32                             ` [PATCH 2/5] travis-ci: use 'set -e' in the 32 bit Linux build job SZEDER Gábor
2018-01-23 16:26                               ` Jeff King
2018-01-23 16:32                                 ` Jeff King
2018-01-24 12:12                                 ` SZEDER Gábor
2018-01-24 15:49                                   ` Jeff King
2018-01-22 13:32                             ` [PATCH 3/5] travis-ci: don't repeat the path of the cache directory SZEDER Gábor
2018-01-23 16:30                               ` Jeff King
2018-01-24 13:14                                 ` SZEDER Gábor
2018-01-22 13:32                             ` [PATCH 4/5] travis-ci: don't run the test suite as root in the 32 bit Linux build SZEDER Gábor
2018-01-23 16:43                               ` Jeff King
2018-01-24 13:45                                 ` SZEDER Gábor
2018-01-24 15:56                                   ` Jeff King
2018-01-24 18:01                                     ` Jeff King
2018-01-24 19:51                                       ` Jeff King
2018-01-22 13:32                             ` [PATCH 5/5] travis-ci: don't fail if user already exists on 32 bit Linux build job SZEDER Gábor
2018-01-23 16:46                               ` Jeff King
2018-01-24  0:32                                 ` Duy Nguyen
2018-01-24 19:39                                 ` SZEDER Gábor
2018-01-22 18:27                 ` [PATCH 3/3] read-cache: don't write index twice if we can't write shared index SZEDER Gábor
2018-01-22 19:46                   ` Eric Sunshine
2018-01-22 22:10                     ` SZEDER Gábor
2018-01-24  9:11                   ` Duy Nguyen
2018-01-26 22:44                   ` Lars Schneider
2018-01-14 14:29         ` [PATCH v3 4/3] read-cache: don't try to write index " Thomas Gummerer
2018-01-18 21:53     ` [PATCH v3 0/3] fixes for split index mode Thomas Gummerer
2018-01-19 18:34       ` Junio C Hamano
2018-01-19 21:11         ` Thomas Gummerer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20180116214239.GA3622@google.com \
    --to=bmwill@google.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=larsxschneider@gmail.com \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=szeder.dev@gmail.com \
    --cc=t.gummerer@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).