git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>
To: gitster@pobox.com
Cc: carenas@gmail.com, git@jeffhostetler.com, git@vger.kernel.org,
	newren@gmail.com, pawelparuzel95@gmail.com, pclouds@gmail.com,
	peff@peff.net, sandals@crustytoothpaste.net,
	szeder.dev@gmail.com, tboegi@web.de,
	Ramsay Jones <ramsay@ramsayjones.plus.com>
Subject: [PATCH] clone: fix colliding file detection on APFS
Date: Tue, 20 Nov 2018 17:28:53 +0100	[thread overview]
Message-ID: <20181120162853.22441-1-pclouds@gmail.com> (raw)
In-Reply-To: <xmqq36rwcwtu.fsf@gitster-ct.c.googlers.com>

Commit b878579ae7 (clone: report duplicate entries on case-insensitive
filesystems - 2018-08-17) adds a warning to user when cloning a repo
with case-sensitive file names on a case-insensitive file system. The
"find duplicate file" check was doing by comparing inode number (and
only fall back to fspathcmp() when inode is known to be unreliable
because fspathcmp() can't cover all case folding cases).

The inode check is very simple, and wrong. It compares between a
32-bit number (sd_ino) and potentially a 64-bit number (st_ino). When
an inode is larger than 2^32 (which seems to be the case for APFS), it
will be truncated and stored in sd_ino, but comparing with itself will
fail.

As a result, instead of showing a pair of files that have the same
name, we show just one file (marked before the beginning of the
loop). We fail to find the original one.

The fix could be just a simple type cast (*)

    dup->ce_stat_data.sd_ino == (unsigned int)st->st_ino

but this is no longer a reliable test, there are 4G possible inodes
that can match sd_ino because we only match the lower 32 bits instead
of full 64 bits.

There are two options to go. Either we ignore inode and go with
fspathcmp() on Apple platform. This means we can't do accurate inode
check on HFS anymore, or even on APFS when inode numbers are still
below 2^32.

Or we just to to reduce the odds of matching a wrong file by checking
more attributes, counting mostly on st_size because st_xtime is likely
the same. This patch goes with this direction, hoping that false
positive chances are too small to be seen in practice.

While at there, enable the test on Cygwin (verified working by Ramsay
Jones)

(*) this is also already done inside match_stat_data()

Reported-by: Carlo Arenas <carenas@gmail.com>
Helped-by: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 So I'm going with match_stat_data(). But I don't know, perhaps just
 ignoring inode (like Carlo's original patch) is safer/better?

 Tested on case-insensitive JFS on Linux. But I don't think it really
 matters because I'm not even sure if I could push inode above 2^32
 with this. Hacking JFS for this test sounds fun, but no time for that.

 entry.c          | 4 ++--
 t/t5601-clone.sh | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/entry.c b/entry.c
index 5d136c5d55..0a3c451f5f 100644
--- a/entry.c
+++ b/entry.c
@@ -404,7 +404,7 @@ static void mark_colliding_entries(const struct checkout *state,
 {
 	int i, trust_ino = check_stat;
 
-#if defined(GIT_WINDOWS_NATIVE)
+#if defined(GIT_WINDOWS_NATIVE) || defined(__CYGWIN__)
 	trust_ino = 0;
 #endif
 
@@ -419,7 +419,7 @@ static void mark_colliding_entries(const struct checkout *state,
 		if (dup->ce_flags & (CE_MATCHED | CE_VALID | CE_SKIP_WORKTREE))
 			continue;
 
-		if ((trust_ino && dup->ce_stat_data.sd_ino == st->st_ino) ||
+		if ((trust_ino && !match_stat_data(&dup->ce_stat_data, st)) ||
 		    (!trust_ino && !fspathcmp(ce->name, dup->name))) {
 			dup->ce_flags |= CE_MATCHED;
 			break;
diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh
index f1a49e94f5..c28d51bd59 100755
--- a/t/t5601-clone.sh
+++ b/t/t5601-clone.sh
@@ -628,7 +628,7 @@ test_expect_success 'clone on case-insensitive fs' '
 	)
 '
 
-test_expect_success !MINGW,!CYGWIN,CASE_INSENSITIVE_FS 'colliding file detection' '
+test_expect_success !MINGW,CASE_INSENSITIVE_FS 'colliding file detection' '
 	grep X icasefs/warning &&
 	grep x icasefs/warning &&
 	test_i18ngrep "the following paths have collided" icasefs/warning
-- 
2.19.1.1327.g328c130451.dirty


  reply	other threads:[~2018-11-20 16:29 UTC|newest]

Thread overview: 96+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-07-27  9:59 Git clone and case sensitivity Paweł Paruzel
2018-07-27 20:59 ` brian m. carlson
2018-07-28  4:36   ` Duy Nguyen
2018-07-28  4:45     ` Duy Nguyen
2018-07-28  4:48       ` Jeff King
2018-07-28  5:11         ` Duy Nguyen
2018-07-28  9:48           ` Simon Ruderich
2018-07-28  9:56           ` Jeff King
2018-07-28 18:05             ` brian m. carlson
2018-07-29  5:26             ` Duy Nguyen
2018-07-29  9:28               ` Jeff King
2018-07-30 15:27                 ` [PATCH/RFC] clone: report duplicate entries on case-insensitive filesystems Nguyễn Thái Ngọc Duy
2018-07-31 18:23                   ` Torsten Bögershausen
2018-08-01 15:25                     ` Duy Nguyen
2018-07-31 18:44                   ` Elijah Newren
2018-07-31 19:12                     ` Junio C Hamano
2018-07-31 19:29                       ` Jeff King
2018-07-31 20:12                         ` Junio C Hamano
2018-07-31 20:37                           ` Jeff King
2018-07-31 20:57                             ` Junio C Hamano
2018-08-01 21:20                               ` Junio C Hamano
2018-08-02 14:43                                 ` Duy Nguyen
2018-08-02 16:27                                   ` Junio C Hamano
2018-08-02 19:06                                     ` Jeff King
2018-08-02 21:14                                       ` Junio C Hamano
2018-08-02 21:28                                         ` Jeff King
2018-08-03 18:23                                           ` Jeff Hostetler
2018-08-03 18:49                                             ` Junio C Hamano
2018-08-03 18:53                                             ` Jeff King
2018-08-05 14:01                                               ` Jeff Hostetler
2018-08-03 14:28                                   ` Torsten Bögershausen
2018-08-01 15:21                     ` Duy Nguyen
2018-07-31 19:13                   ` Junio C Hamano
2018-08-01 15:16                     ` Duy Nguyen
2018-08-07 19:01                   ` [PATCH v2] " Nguyễn Thái Ngọc Duy
2018-08-07 19:31                     ` Junio C Hamano
2018-08-08 19:48                       ` Jeff Hostetler
2018-08-08 22:31                         ` Jeff King
2018-08-09  0:41                           ` Junio C Hamano
2018-08-09 14:23                             ` Jeff King
2018-08-09 21:14                               ` Jeff Hostetler
2018-08-09 21:34                                 ` Jeff King
2018-08-09 21:40                                 ` Elijah Newren
2018-08-09 21:44                                   ` Jeff King
2018-08-09 21:53                                     ` Elijah Newren
2018-08-09 21:59                                       ` Jeff King
2018-08-09 23:05                                         ` Elijah Newren
2018-08-09 22:07                                   ` Junio C Hamano
2018-08-10 15:36                     ` [PATCH v3 0/1] clone: warn on colidding entries on checkout Nguyễn Thái Ngọc Duy
2018-08-10 15:36                       ` [PATCH v3 1/1] clone: report duplicate entries on case-insensitive filesystems Nguyễn Thái Ngọc Duy
2018-08-10 16:42                         ` Junio C Hamano
2018-08-11 10:09                         ` SZEDER Gábor
2018-08-11 13:16                           ` Duy Nguyen
2018-08-13 16:55                             ` Junio C Hamano
2018-08-13 17:12                               ` Duy Nguyen
2018-08-10 16:12                       ` [PATCH v3 0/1] clone: warn on colidding entries on checkout Junio C Hamano
2018-08-12  9:07                       ` [PATCH v4] clone: report duplicate entries on case-insensitive filesystems Nguyễn Thái Ngọc Duy
2018-08-13 15:32                         ` Jeff Hostetler
2018-08-13 17:18                         ` Junio C Hamano
2018-08-15 19:08                         ` Torsten Bögershausen
2018-08-15 19:35                           ` Duy Nguyen
2018-08-16 15:56                             ` [PATCH] config.txt: clarify core.checkStat = minimal Nguyễn Thái Ngọc Duy
2018-08-16 17:01                               ` Junio C Hamano
2018-08-16 18:19                                 ` Duy Nguyen
2018-08-16 22:29                                   ` Junio C Hamano
2018-08-17 15:26                                   ` Junio C Hamano
2018-08-17 15:29                                     ` Duy Nguyen
2018-08-15 19:38                           ` [PATCH v4] clone: report duplicate entries on case-insensitive filesystems Junio C Hamano
2018-08-16 14:03                             ` Torsten Bögershausen
2018-08-16 15:42                               ` Duy Nguyen
2018-08-16 16:23                               ` Junio C Hamano
2018-08-17 16:16                         ` [PATCH v5] " Nguyễn Thái Ngọc Duy
2018-08-17 17:20                           ` Junio C Hamano
2018-08-17 18:00                             ` Duy Nguyen
2018-08-17 19:46                           ` Torsten Bögershausen
2018-11-19  8:20                           ` Carlo Marcelo Arenas Belón
2018-11-19 12:28                             ` Torsten Bögershausen
2018-11-19 17:14                               ` Carlo Arenas
2018-11-19 18:24                                 ` Duy Nguyen
2018-11-19 21:03                                   ` Duy Nguyen
2018-11-19 21:04                                     ` Duy Nguyen
2018-11-19 21:17                                     ` Duy Nguyen
2018-11-19 23:29                                     ` Ramsay Jones
2018-11-19 23:54                                       ` Ramsay Jones
2018-11-20  1:05                                         ` Carlo Arenas
2018-11-20  2:22                                     ` Junio C Hamano
2018-11-20 16:28                                       ` Nguyễn Thái Ngọc Duy [this message]
2018-11-20 19:20                                         ` [PATCH] clone: fix colliding file detection on APFS Ramsay Jones
2018-11-20 19:35                                         ` Carlo Arenas
2018-11-20 19:38                                           ` Duy Nguyen
2018-11-22 17:59                                         ` [PATCH v1 1/1] t5601-99: Enable colliding file detection for MINGW tboegi
2018-11-22 20:16                                           ` Carlo Marcelo Arenas Belón
2018-11-23 11:24                                             ` Johannes Schindelin
2018-11-19 17:21                               ` [PATCH v5] clone: report duplicate entries on case-insensitive filesystems Ramsay Jones
2018-11-19 19:39                                 ` Carlo Arenas
2018-07-31 19:39                 ` Git clone and case sensitivity Jeff Hostetler

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20181120162853.22441-1-pclouds@gmail.com \
    --to=pclouds@gmail.com \
    --cc=carenas@gmail.com \
    --cc=git@jeffhostetler.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=newren@gmail.com \
    --cc=pawelparuzel95@gmail.com \
    --cc=peff@peff.net \
    --cc=ramsay@ramsayjones.plus.com \
    --cc=sandals@crustytoothpaste.net \
    --cc=szeder.dev@gmail.com \
    --cc=tboegi@web.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).