From: Steffen Prohaska <prohaska@zib.de>
To: Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org, "Johannes Sixt" <j6t@kdbg.org>,
"John Keeping" <john@keeping.me.uk>,
"Jonathan Nieder" <jrnieder@gmail.com>,
"Kyle J. McKay" <mackyle@gmail.com>,
"Torsten Bögershausen" <tboegi@web.de>,
"Eric Sunshine" <sunshine@sunshineco.com>,
"Steffen Prohaska" <prohaska@zib.de>
Subject: [PATCH v4] compat: Fix read() of 2GB and more on Mac OS X
Date: Mon, 19 Aug 2013 17:41:19 +0200 [thread overview]
Message-ID: <1376926879-30846-1-git-send-email-prohaska@zib.de> (raw)
In-Reply-To: <1376900499-662-1-git-send-email-prohaska@zib.de>
Previously, filtering 2GB or more through an external filter (see test)
failed on Mac OS X 10.8.4 (12E55) for a 64-bit executable with:
error: read from external filter cat failed
error: cannot feed the input to external filter cat
error: cat died of signal 13
error: external filter cat failed 141
error: external filter cat failed
The reason was that read() immediately returns with EINVAL if nbyte >=
2GB. According to POSIX [1], if the value of nbyte passed to read() is
greater than SSIZE_MAX, the result is implementation-defined. The write
function has the same restriction [2]. Since OS X still supports
running 32-bit executables, the 32-bit limit (SSIZE_MAX = INT_MAX
= 2GB - 1) seems to be also imposed on 64-bit executables under certain
conditions. For write, the problem has been addressed in a earlier
commit [6c642a].
The problem for read() is addressed in a similar way by introducing
a wrapper function in compat that always reads less than 2GB. It is
very likely that the read() and write() wrappers are always used
together. To avoid introducing another option, NEEDS_CLIPPED_WRITE is
changed to NEEDS_CLIPPED_IO and used to activate both wrappers.
To avoid expanding the read compat macro in constructs like
'vtbl->read(...)', 'read' is renamed to 'readfn' in two cases. The
solution seems more robust than using '#undef read'.
Note that 'git add' exits with 0 even if it prints filtering errors to
stderr. The test, therefore, checks stderr. 'git add' should probably
be changed (sometime in another commit) to exit with nonzero if
filtering fails. The test could then be changed to use test_must_fail.
Thanks to the following people for their suggestions:
Johannes Sixt <j6t@kdbg.org>
John Keeping <john@keeping.me.uk>
Jonathan Nieder <jrnieder@gmail.com>
Kyle J. McKay <mackyle@gmail.com>
Torsten Bögershausen <tboegi@web.de>
Eric Sunshine <sunshine@sunshineco.com>
[1] http://pubs.opengroup.org/onlinepubs/009695399/functions/read.html
[2] http://pubs.opengroup.org/onlinepubs/009695399/functions/write.html
[6c642a] 6c642a878688adf46b226903858b53e2d31ac5c3
compate/clipped-write.c: large write(2) fails on Mac OS X/XNU
Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
Makefile | 10 +++++-----
builtin/var.c | 10 +++++-----
compat/{clipped-write.c => clipped-io.c} | 11 ++++++++++-
config.mak.uname | 2 +-
git-compat-util.h | 5 ++++-
streaming.c | 4 ++--
t/t0021-conversion.sh | 14 ++++++++++++++
7 files changed, 41 insertions(+), 15 deletions(-)
rename compat/{clipped-write.c => clipped-io.c} (53%)
diff --git a/Makefile b/Makefile
index 3588ca1..f54134d 100644
--- a/Makefile
+++ b/Makefile
@@ -69,8 +69,8 @@ all::
# Define NO_MSGFMT_EXTENDED_OPTIONS if your implementation of msgfmt
# doesn't support GNU extensions like --check and --statistics
#
-# Define NEEDS_CLIPPED_WRITE if your write(2) cannot write more than
-# INT_MAX bytes at once (e.g. MacOS X).
+# Define NEEDS_CLIPPED_IO if your read(2) and/or write(2) cannot handle more
+# than INT_MAX bytes at once (e.g. Mac OS X).
#
# Define HAVE_PATHS_H if you have paths.h and want to use the default PATH
# it specifies.
@@ -1493,9 +1493,9 @@ ifndef NO_MSGFMT_EXTENDED_OPTIONS
MSGFMT += --check --statistics
endif
-ifdef NEEDS_CLIPPED_WRITE
- BASIC_CFLAGS += -DNEEDS_CLIPPED_WRITE
- COMPAT_OBJS += compat/clipped-write.o
+ifdef NEEDS_CLIPPED_IO
+ BASIC_CFLAGS += -DNEEDS_CLIPPED_IO
+ COMPAT_OBJS += compat/clipped-io.o
endif
ifneq (,$(XDL_FAST_HASH))
diff --git a/builtin/var.c b/builtin/var.c
index aedbb53..06f8459 100644
--- a/builtin/var.c
+++ b/builtin/var.c
@@ -28,7 +28,7 @@ static const char *pager(int flag)
struct git_var {
const char *name;
- const char *(*read)(int);
+ const char *(*readfn)(int);
};
static struct git_var git_vars[] = {
{ "GIT_COMMITTER_IDENT", git_committer_info },
@@ -43,8 +43,8 @@ static void list_vars(void)
struct git_var *ptr;
const char *val;
- for (ptr = git_vars; ptr->read; ptr++)
- if ((val = ptr->read(0)))
+ for (ptr = git_vars; ptr->readfn; ptr++)
+ if ((val = ptr->readfn(0)))
printf("%s=%s\n", ptr->name, val);
}
@@ -53,9 +53,9 @@ static const char *read_var(const char *var)
struct git_var *ptr;
const char *val;
val = NULL;
- for (ptr = git_vars; ptr->read; ptr++) {
+ for (ptr = git_vars; ptr->readfn; ptr++) {
if (strcmp(var, ptr->name) == 0) {
- val = ptr->read(IDENT_STRICT);
+ val = ptr->readfn(IDENT_STRICT);
break;
}
}
diff --git a/compat/clipped-write.c b/compat/clipped-io.c
similarity index 53%
rename from compat/clipped-write.c
rename to compat/clipped-io.c
index b8f98ff..ec3232a 100644
--- a/compat/clipped-write.c
+++ b/compat/clipped-io.c
@@ -1,10 +1,19 @@
#include "../git-compat-util.h"
+#undef read
#undef write
/*
- * Version of write that will write at most INT_MAX bytes.
+ * Versions of read() and write() that limit nbyte to INT_MAX.
* Workaround a xnu bug on Mac OS X
*/
+
+ssize_t clipped_read(int fd, void *buf, size_t nbyte)
+{
+ if (nbyte > INT_MAX)
+ nbyte = INT_MAX;
+ return read(fd, buf, nbyte);
+}
+
ssize_t clipped_write(int fildes, const void *buf, size_t nbyte)
{
if (nbyte > INT_MAX)
diff --git a/config.mak.uname b/config.mak.uname
index b27f51d..fb39726 100644
--- a/config.mak.uname
+++ b/config.mak.uname
@@ -95,7 +95,7 @@ ifeq ($(uname_S),Darwin)
NO_MEMMEM = YesPlease
USE_ST_TIMESPEC = YesPlease
HAVE_DEV_TTY = YesPlease
- NEEDS_CLIPPED_WRITE = YesPlease
+ NEEDS_CLIPPED_IO = YesPlease
COMPAT_OBJS += compat/precompose_utf8.o
BASIC_CFLAGS += -DPRECOMPOSE_UNICODE
endif
diff --git a/git-compat-util.h b/git-compat-util.h
index 115cb1d..4a875cb 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -185,7 +185,10 @@ typedef unsigned long uintptr_t;
#define probe_utf8_pathname_composition(a,b)
#endif
-#ifdef NEEDS_CLIPPED_WRITE
+#ifdef NEEDS_CLIPPED_IO
+ssize_t clipped_read(int fd, void *buf, size_t nbyte);
+#define read(x,y,z) clipped_read((x),(y),(z))
+
ssize_t clipped_write(int fildes, const void *buf, size_t nbyte);
#define write(x,y,z) clipped_write((x),(y),(z))
#endif
diff --git a/streaming.c b/streaming.c
index debe904..2ac3047 100644
--- a/streaming.c
+++ b/streaming.c
@@ -20,7 +20,7 @@ typedef ssize_t (*read_istream_fn)(struct git_istream *, char *, size_t);
struct stream_vtbl {
close_istream_fn close;
- read_istream_fn read;
+ read_istream_fn readfn;
};
#define open_method_decl(name) \
@@ -101,7 +101,7 @@ int close_istream(struct git_istream *st)
ssize_t read_istream(struct git_istream *st, void *buf, size_t sz)
{
- return st->vtbl->read(st, buf, sz);
+ return st->vtbl->readfn(st, buf, sz);
}
static enum input_source istream_source(const unsigned char *sha1,
diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh
index e50f0f7..b92e6cb 100755
--- a/t/t0021-conversion.sh
+++ b/t/t0021-conversion.sh
@@ -190,4 +190,18 @@ test_expect_success 'required filter clean failure' '
test_must_fail git add test.fc
'
+test -n "$GIT_TEST_LONG" && test_set_prereq EXPENSIVE
+
+test_expect_success EXPENSIVE 'filter large file' '
+ git config filter.largefile.smudge cat &&
+ git config filter.largefile.clean cat &&
+ for i in $(test_seq 1 2048); do printf "%1048576d" 1; done >2GB &&
+ echo "2GB filter=largefile" >.gitattributes &&
+ git add 2GB 2>err &&
+ ! test -s err &&
+ rm -f 2GB &&
+ git checkout -- 2GB 2>err &&
+ ! test -s err
+'
+
test_done
--
1.8.4.rc3.5.g4f480ff
next prev parent reply other threads:[~2013-08-19 15:44 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-08-17 12:40 [PATCH] xread(): Fix read error when filtering >= 2GB on Mac OS X Steffen Prohaska
2013-08-17 15:27 ` John Keeping
2013-08-17 15:56 ` Torsten Bögershausen
2013-08-17 17:16 ` Johannes Sixt
2013-08-17 18:57 ` Jonathan Nieder
2013-08-17 20:25 ` Kyle J. McKay
2013-08-17 21:23 ` Jonathan Nieder
2013-08-19 6:38 ` [PATCH v2] compat: Fix read() of 2GB and more " Steffen Prohaska
2013-08-19 7:54 ` John Keeping
2013-08-19 8:20 ` Steffen Prohaska
2013-08-19 8:20 ` Johannes Sixt
2013-08-19 8:25 ` Stefan Beller
2013-08-19 8:40 ` Johannes Sixt
2013-08-19 8:28 ` Steffen Prohaska
2013-08-19 8:21 ` [PATCH v3] " Steffen Prohaska
2013-08-19 13:59 ` Eric Sunshine
2013-08-19 16:33 ` Junio C Hamano
2013-08-19 15:41 ` Steffen Prohaska [this message]
2013-08-19 16:04 ` [PATCH v4] " Linus Torvalds
2013-08-19 16:37 ` Steffen Prohaska
2013-08-19 17:24 ` Junio C Hamano
2013-08-19 17:16 ` Junio C Hamano
2013-08-19 17:28 ` Linus Torvalds
2013-08-19 21:56 ` Kyle J. McKay
2013-08-19 22:51 ` Linus Torvalds
2013-08-27 4:59 ` Junio C Hamano
2013-08-20 6:43 ` [PATCH v5 0/2] Fix IO of >=2GB on Mac OS X by limiting IO chunks Steffen Prohaska
2013-08-20 6:43 ` [PATCH v5 1/2] xread, xwrite: Limit size of IO, fixing IO of 2GB and more on Mac OS X Steffen Prohaska
2013-08-20 19:37 ` Junio C Hamano
2013-08-21 19:50 ` Torsten Bögershausen
2013-08-20 6:43 ` [PATCH v5 2/2] Revert "compate/clipped-write.c: large write(2) fails on Mac OS X/XNU" Steffen Prohaska
2013-08-21 13:46 ` [PATCH v6 0/2] Fix IO >= 2GB on Mac, fixed typo Steffen Prohaska
2013-08-21 13:46 ` [PATCH v5 1/2] xread, xwrite: Limit size of IO, fixing IO of 2GB and more on Mac OS X Steffen Prohaska
2013-08-21 13:46 ` [PATCH v5 2/2] Revert "compate/clipped-write.c: large write(2) fails on Mac OS X/XNU" Steffen Prohaska
2013-08-21 15:58 ` [PATCH v6 0/2] Fix IO >= 2GB on Mac, fixed typo Junio C Hamano
2013-08-19 8:27 ` [PATCH v2] compat: Fix read() of 2GB and more on Mac OS X Johannes Sixt
2013-08-19 14:41 ` Torsten Bögershausen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1376926879-30846-1-git-send-email-prohaska@zib.de \
--to=prohaska@zib.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=john@keeping.me.uk \
--cc=jrnieder@gmail.com \
--cc=mackyle@gmail.com \
--cc=sunshine@sunshineco.com \
--cc=tboegi@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).