From: Junio C Hamano <gitster@pobox.com>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Jeff King <peff@peff.net>, Git Mailing List <git@vger.kernel.org>,
Lars Schneider <larsxschneider@gmail.com>,
Eric Wong <e@80x24.org>,
Johannes Schindelin <johannes.schindelin@gmx.de>
Subject: Re: [PATCH v3 2/3] sha1_file: open window into packfiles with O_CLOEXEC
Date: Fri, 28 Oct 2016 06:32:28 -0700 [thread overview]
Message-ID: <xmqqh97w38gj.fsf@gitster.mtv.corp.google.com> (raw)
In-Reply-To: <xmqq60od42s0.fsf@gitster.mtv.corp.google.com> (Junio C. Hamano's message of "Thu, 27 Oct 2016 19:37:35 -0700")
Junio C Hamano <gitster@pobox.com> writes:
> Hmph. This may not fly well in practice, though.
We can flip the order around, and then it becomes simpler to later
drop atime support. The first step goes like this.
-- >8 --
From: Junio C Hamano <gitster@pobox.com>
Date: Fri, 28 Oct 2016 06:23:07 -0700
Subject: [PATCH] git_open(): untangle possible NOATIME and CLOEXEC interactions
The way we structured the fallback/retry mechanism for opening with
O_NOATIME and O_CLOEXEC meant that if we failed due to lack of
support to open the file with O_NOATIME option (i.e. EINVAL), we
would still try to drop O_CLOEXEC first and retry, and then drop
O_NOATIME. A platform on which O_NOATIME is defined in the header
without support from the kernel wouldn't have a chance to open with
O_CLOEXEC option due to this code structure.
Arguably, O_CLOEXEC is more important than O_NOATIME, as the latter
is mostly about performance, while the former can affect correctness.
Instead use O_CLOEXEC to open the file, and then use fcntl(2) to set
O_NOATIME on the resulting file descriptor. open(2) itself does not
cause atime to be updated according to Linus [*1*].
The helper to do the former can be usable in the codepath in
ce_compare_data() that was recently added to open a file descriptor
with O_CLOEXEC; use it while we are at it.
*1* <CA+55aFw83E+zOd+z5h-CA-3NhrLjVr-anL6pubrSWttYx3zu8g@mail.gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
cache.h | 1 +
read-cache.c | 9 +--------
sha1_file.c | 39 +++++++++++++++++++--------------------
3 files changed, 21 insertions(+), 28 deletions(-)
diff --git a/cache.h b/cache.h
index a902ca1f8e..6f1c21a352 100644
--- a/cache.h
+++ b/cache.h
@@ -1122,6 +1122,7 @@ extern int write_sha1_file(const void *buf, unsigned long len, const char *type,
extern int hash_sha1_file_literally(const void *buf, unsigned long len, const char *type, unsigned char *sha1, unsigned flags);
extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned char *);
extern int force_object_loose(const unsigned char *sha1, time_t mtime);
+extern int git_open_cloexec(const char *name, int flags);
extern int git_open(const char *name);
extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
diff --git a/read-cache.c b/read-cache.c
index db5d910642..c27d3e240b 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -156,14 +156,7 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)
static int ce_compare_data(const struct cache_entry *ce, struct stat *st)
{
int match = -1;
- static int cloexec = O_CLOEXEC;
- int fd = open(ce->name, O_RDONLY | cloexec);
-
- if ((cloexec & O_CLOEXEC) && fd < 0 && errno == EINVAL) {
- /* Try again w/o O_CLOEXEC: the kernel might not support it */
- cloexec &= ~O_CLOEXEC;
- fd = open(ce->name, O_RDONLY | cloexec);
- }
+ int fd = git_open_cloexec(ce->name, O_RDONLY);
if (fd >= 0) {
unsigned char sha1[20];
diff --git a/sha1_file.c b/sha1_file.c
index 09045df1dc..dfbf398183 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1559,31 +1559,30 @@ int check_sha1_signature(const unsigned char *sha1, void *map,
return hashcmp(sha1, real_sha1) ? -1 : 0;
}
-int git_open(const char *name)
+int git_open_cloexec(const char *name, int flags)
{
- static int sha1_file_open_flag = O_NOATIME | O_CLOEXEC;
-
- for (;;) {
- int fd;
-
- errno = 0;
- fd = open(name, O_RDONLY | sha1_file_open_flag);
- if (fd >= 0)
- return fd;
+ static int cloexec = O_CLOEXEC;
+ int fd = open(name, flags | cloexec);
+ if ((cloexec & O_CLOEXEC) && fd < 0 && errno == EINVAL) {
/* Try again w/o O_CLOEXEC: the kernel might not support it */
- if ((sha1_file_open_flag & O_CLOEXEC) && errno == EINVAL) {
- sha1_file_open_flag &= ~O_CLOEXEC;
- continue;
- }
+ cloexec &= ~O_CLOEXEC;
+ fd = open(name, flags | cloexec);
+ }
+ return fd;
+}
- /* Might the failure be due to O_NOATIME? */
- if (errno != ENOENT && (sha1_file_open_flag & O_NOATIME)) {
- sha1_file_open_flag &= ~O_NOATIME;
- continue;
- }
- return -1;
+int git_open(const char *name)
+{
+ static int noatime = O_NOATIME;
+ int fd = git_open_cloexec(name, O_RDONLY);
+
+ if (0 <= fd && (noatime & O_NOATIME)) {
+ int flags = fcntl(fd, F_GETFL);
+ if (fcntl(fd, F_SETFL, flags | noatime))
+ noatime = 0;
}
+ return fd;
}
static int stat_sha1_file(const unsigned char *sha1, struct stat *st)
--
2.10.1-791-g404733b9cf
next prev parent reply other threads:[~2016-10-28 13:32 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-24 18:02 [PATCH v2 0/2] Use CLOEXEC to avoid fd leaks larsxschneider
2016-10-24 18:02 ` [PATCH v2 1/2] sha1_file: open window into packfiles with CLOEXEC larsxschneider
2016-10-25 10:27 ` Johannes Schindelin
2016-10-25 16:58 ` Junio C Hamano
2016-10-24 18:03 ` [PATCH v2 2/2] read-cache: make sure file handles are not inherited by child processes larsxschneider
2016-10-24 18:39 ` Eric Wong
2016-10-24 19:53 ` Junio C Hamano
2016-10-25 10:33 ` Johannes Schindelin
2016-10-25 17:02 ` Junio C Hamano
2016-10-24 19:22 ` Johannes Sixt
2016-10-24 19:53 ` Lars Schneider
2016-10-25 21:39 ` Johannes Sixt
2016-10-24 18:23 ` [PATCH v2 0/2] Use CLOEXEC to avoid fd leaks Junio C Hamano
2016-10-25 11:27 ` Johannes Schindelin
2016-10-25 18:16 ` [PATCH v3 0/3] quick reroll of Lars's git_open() w/ O_CLOEXEC Junio C Hamano
2016-10-25 18:16 ` [PATCH v3 1/3] sha1_file: rename git_open_noatime() to git_open() Junio C Hamano
2016-10-25 18:16 ` [PATCH v3 2/3] sha1_file: open window into packfiles with O_CLOEXEC Junio C Hamano
2016-10-26 4:25 ` Jeff King
2016-10-26 16:23 ` Junio C Hamano
2016-10-26 16:47 ` Jeff King
2016-10-26 17:52 ` Junio C Hamano
2016-10-26 20:17 ` Jeff King
2016-10-26 21:15 ` Junio C Hamano
2016-10-27 10:24 ` Jeff King
2016-10-27 21:49 ` Junio C Hamano
2016-10-27 22:38 ` Linus Torvalds
2016-10-27 22:56 ` Junio C Hamano
2016-10-27 23:09 ` Linus Torvalds
2016-10-27 23:19 ` Linus Torvalds
2016-10-27 23:36 ` Junio C Hamano
2016-10-27 23:44 ` Linus Torvalds
2016-10-28 1:08 ` Junio C Hamano
2016-10-28 2:37 ` Junio C Hamano
2016-10-28 5:51 ` Eric Wong
2016-10-28 11:11 ` Johannes Schindelin
2016-10-28 16:13 ` Linus Torvalds
2016-10-28 16:48 ` Junio C Hamano
2016-10-28 17:38 ` Linus Torvalds
2016-10-28 17:47 ` Junio C Hamano
2016-10-29 1:26 ` Junio C Hamano
2016-10-29 8:25 ` Johannes Schindelin
2016-10-29 17:06 ` Linus Torvalds
2016-10-31 17:37 ` Junio C Hamano
2016-10-31 13:56 ` Jeff King
2016-10-31 17:55 ` Junio C Hamano
2016-10-31 18:05 ` Jeff King
2016-10-28 13:32 ` Junio C Hamano [this message]
2016-10-28 13:33 ` Junio C Hamano
2016-10-28 7:51 ` Jeff King
2016-10-25 18:16 ` [PATCH v3 3/3] read-cache: make sure file handles are not inherited by child processes Junio C Hamano
2016-10-25 21:33 ` Eric Wong
2016-10-25 22:54 ` Junio C Hamano
2016-10-25 21:48 ` [PATCH v3 0/3] quick reroll of Lars's git_open() w/ O_CLOEXEC Lars Schneider
2016-10-25 22:56 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqh97w38gj.fsf@gitster.mtv.corp.google.com \
--to=gitster@pobox.com \
--cc=e@80x24.org \
--cc=git@vger.kernel.org \
--cc=johannes.schindelin@gmx.de \
--cc=larsxschneider@gmail.com \
--cc=peff@peff.net \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).