* [PATCH] git_open_noatime: return with errno=0 on success @ 2015-07-08 12:38 Clemens Buchacher 2015-07-08 18:51 ` Eric Sunshine 0 siblings, 1 reply; 7+ messages in thread From: Clemens Buchacher @ 2015-07-08 12:38 UTC (permalink / raw) To: git; +Cc: Linus Torvalds, Martin Schröder In read_sha1_file_extended we die if read_object fails with a fatal error. We detect a fatal error if errno is non-zero and is not ENOENT. If the object could not be read because it does not exist, this is not considered a fatal error and we want to return NULL. Somewhere down the line, read_object calls git_open_noatime to open a pack index file, for example. We first try open with O_NOATIME. If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the second open succeeds, errno is however still set to EPERM from the first attemt. When we finally determine that the object does not exist, read_object returns NULL and read_sha1_file_extended dies with a fatal error: fatal: failed to read object <sha1>: Operation not permitted Fix this by resetting errno to zero before we call open again. Cc: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com> Helped-by: Martin Schröder <martin.h.schroeder@intel.com> --- sha1_file.c | 1 + 1 file changed, 1 insertion(+) diff --git a/sha1_file.c b/sha1_file.c index 77cd81d..62b7ad6 100644 --- a/sha1_file.c +++ b/sha1_file.c @@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name) static int sha1_file_open_flag = O_NOATIME; for (;;) { + errno = 0; int fd = open(name, O_RDONLY | sha1_file_open_flag); if (fd >= 0) return fd; -- 1.9.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] git_open_noatime: return with errno=0 on success 2015-07-08 12:38 [PATCH] git_open_noatime: return with errno=0 on success Clemens Buchacher @ 2015-07-08 18:51 ` Eric Sunshine 2015-08-04 8:24 ` Clemens Buchacher 0 siblings, 1 reply; 7+ messages in thread From: Eric Sunshine @ 2015-07-08 18:51 UTC (permalink / raw) To: Clemens Buchacher; +Cc: Git List, Linus Torvalds, Martin Schröder On Wed, Jul 8, 2015 at 8:38 AM, Clemens Buchacher <clemens.buchacher@intel.com> wrote: > In read_sha1_file_extended we die if read_object fails with a fatal > error. We detect a fatal error if errno is non-zero and is not > ENOENT. If the object could not be read because it does not exist, > this is not considered a fatal error and we want to return NULL. > > Somewhere down the line, read_object calls git_open_noatime to open > a pack index file, for example. We first try open with O_NOATIME. > If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the > second open succeeds, errno is however still set to EPERM from the > first attemt. When we finally determine that the object does not s/attemt/attempt/ > exist, read_object returns NULL and read_sha1_file_extended dies > with a fatal error: > > fatal: failed to read object <sha1>: Operation not permitted > > Fix this by resetting errno to zero before we call open again. > > Cc: Linus Torvalds <torvalds@linux-foundation.org> > Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com> > Helped-by: Martin Schröder <martin.h.schroeder@intel.com> ^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH] git_open_noatime: return with errno=0 on success 2015-07-08 18:51 ` Eric Sunshine @ 2015-08-04 8:24 ` Clemens Buchacher 2015-08-04 21:03 ` Junio C Hamano 0 siblings, 1 reply; 7+ messages in thread From: Clemens Buchacher @ 2015-08-04 8:24 UTC (permalink / raw) To: git; +Cc: Eric Sunshine, Linus Torvalds, Martin Schröder In read_sha1_file_extended we die if read_object fails with a fatal error. We detect a fatal error if errno is non-zero and is not ENOENT. If the object could not be read because it does not exist, this is not considered a fatal error and we want to return NULL. Somewhere down the line, read_object calls git_open_noatime to open a pack index file, for example. We first try open with O_NOATIME. If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the second open succeeds, errno is however still set to EPERM from the first attempt. When we finally determine that the object does not exist, read_object returns NULL and read_sha1_file_extended dies with a fatal error: fatal: failed to read object <sha1>: Operation not permitted Fix this by resetting errno to zero before we call open again. Cc: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com> --- This is a re-submission without changes except for a typo fix in the comments (thanks Eric). The original submission received no other comments, but I think it is a clear improvement and I hope it was just missed the first time. Best regards, Clemens sha1_file.c | 1 + 1 file changed, 1 insertion(+) diff --git a/sha1_file.c b/sha1_file.c index 77cd81d..62b7ad6 100644 --- a/sha1_file.c +++ b/sha1_file.c @@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name) static int sha1_file_open_flag = O_NOATIME; for (;;) { + errno = 0; int fd = open(name, O_RDONLY | sha1_file_open_flag); if (fd >= 0) return fd; -- 1.9.4 ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] git_open_noatime: return with errno=0 on success 2015-08-04 8:24 ` Clemens Buchacher @ 2015-08-04 21:03 ` Junio C Hamano 2015-08-05 8:59 ` Linus Torvalds 0 siblings, 1 reply; 7+ messages in thread From: Junio C Hamano @ 2015-08-04 21:03 UTC (permalink / raw) To: Clemens Buchacher Cc: git, Eric Sunshine, Linus Torvalds, Martin Schröder Clemens Buchacher <clemens.buchacher@intel.com> writes: > diff --git a/sha1_file.c b/sha1_file.c > index 77cd81d..62b7ad6 100644 > --- a/sha1_file.c > +++ b/sha1_file.c > @@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name) > static int sha1_file_open_flag = O_NOATIME; > > for (;;) { > + errno = 0; > int fd = open(name, O_RDONLY | sha1_file_open_flag); Please avoid decl-after-stmt, which this codebase does not accept. > if (fd >= 0) > return fd; More importantly, is this the right place to clear errno? I would agree it is a good idea to clear it after seeing the first open fail due to lack of O_NOATIME before trying open for the second time, iow, more like this? sha1_file.c | 1 + 1 file changed, 1 insertion(+) diff --git a/sha1_file.c b/sha1_file.c index 1cee438..bf2f229 100644 --- a/sha1_file.c +++ b/sha1_file.c @@ -1467,6 +1467,7 @@ int git_open_noatime(const char *name) /* Might the failure be due to O_NOATIME? */ if (errno != ENOENT && sha1_file_open_flag) { + errno = 0; sha1_file_open_flag = 0; continue; } ^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH] git_open_noatime: return with errno=0 on success 2015-08-04 21:03 ` Junio C Hamano @ 2015-08-05 8:59 ` Linus Torvalds 2015-08-05 14:36 ` Clemens Buchacher 0 siblings, 1 reply; 7+ messages in thread From: Linus Torvalds @ 2015-08-05 8:59 UTC (permalink / raw) To: Junio C Hamano Cc: Clemens Buchacher, Git Mailing List, Eric Sunshine, Martin Schröder On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote: > > I would agree it is a good idea to clear it after seeing the first > open fail due to lack of O_NOATIME before trying open for the second > time, iow, more like this? So I don't think this is _wrong_ per se, but I think the deeper issue is that somebody cares about 'errno' here in the first place. A stale 'errno' generally shouldn't matter, because we either (a) return success (and nobody should look at errno) or (b) return an error later, without setting errno for that _later_ error. and I think either of those two situations are the real bug, and this "clear stale errno" is just a workaround. But as mentioned, I don't think clearign errno is wrong, so I'm not objecting to the patch. I just suspect there's something else goign on too.. Linus ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] git_open_noatime: return with errno=0 on success 2015-08-05 8:59 ` Linus Torvalds @ 2015-08-05 14:36 ` Clemens Buchacher 2015-08-05 16:27 ` Junio C Hamano 0 siblings, 1 reply; 7+ messages in thread From: Clemens Buchacher @ 2015-08-05 14:36 UTC (permalink / raw) To: Linus Torvalds Cc: Junio C Hamano, Git Mailing List, Eric Sunshine, Martin Schröder On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote: > On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote: > > > > I would agree it is a good idea to clear it after seeing the first > > open fail due to lack of O_NOATIME before trying open for the second > > time, iow, more like this? Looks good to me. > So I don't think this is _wrong_ per se, but I think the deeper issue > is that somebody cares about 'errno' here in the first place. > > A stale 'errno' generally shouldn't matter, because we either > > (a) return success (and nobody should look at errno) > > or > > (b) return an error later, without setting errno for that _later_ error. > > and I think either of those two situations are the real bug, and this > "clear stale errno" is just a workaround. I agree. But I do not see how to get there easily. We are trying to read an object. We first try to read from a pack. We may encounter broken pack files, missing index files, unreadable files, but those errors are not necessarily fatal since we may still be able to read the object from the next pack file or from a sha1 file. If finally we do not find the object anywhere, in read_sha1_file_extended we try our best to die with an appropriate error message, for example by looking at errno, and otherwise we just return NULL. Most callers seem to die explicitly or they dereference the null pointer. I think we should instead output error messages closer to the source, like for example in map_sha1_file, but continue anyway. In particular we should immediately report failures due to EPERM or unexpected ENOENT. In the end we may return NULL without another message, but at least the user should have some hints about what went wrong along the way. ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH] git_open_noatime: return with errno=0 on success 2015-08-05 14:36 ` Clemens Buchacher @ 2015-08-05 16:27 ` Junio C Hamano 0 siblings, 0 replies; 7+ messages in thread From: Junio C Hamano @ 2015-08-05 16:27 UTC (permalink / raw) To: Clemens Buchacher Cc: Linus Torvalds, Git Mailing List, Eric Sunshine, Martin Schröder Clemens Buchacher <clemens.buchacher@intel.com> writes: > On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote: > ... >> A stale 'errno' generally shouldn't matter, because we either >> >> (a) return success (and nobody should look at errno) >> >> or >> >> (b) return an error later, without setting errno for that _later_ error. >> >> and I think either of those two situations are the real bug, and this >> "clear stale errno" is just a workaround. > > I agree. But I do not see how to get there easily. > > We are trying to read an object. We first try to read from a pack. We > may encounter broken pack files, missing index files, unreadable files, > but those errors are not necessarily fatal since we may still be able to > read the object from the next pack file or from a sha1 file. > > If finally we do not find the object anywhere, in > read_sha1_file_extended we try our best to die with an appropriate error > message, for example by looking at errno, and otherwise we just return > NULL. Most callers seem to die explicitly or they dereference the null > pointer. > > I think we should instead output error messages closer to the source, > like for example in map_sha1_file, but continue anyway. Hmm, if we find one data source unreadable but an alternative usable, do we really want that error message? What should it say? "error: cannot read from pack"? Such a message, unless we later give "info: but we managed to read it from elsewhere" and make sure these two messages are clearly associated with each other, would make things unnecessarily alarming, wouldn't it? Perhaps we should not rely so heavily on 'errno', but explicitly pass around error code (or enough information to formulate an intelligent message at the end) in the callchain instead. Then the earlier part can notice EPERM on a pack, for example, and return to the caller, and after consulting an alternate data source (e.g. loose object file), the caller can then choose to say "we managed to read the data, but FYI, you may want to check the permission bits of this pack", or choose to stay silent. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2015-08-05 16:27 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-07-08 12:38 [PATCH] git_open_noatime: return with errno=0 on success Clemens Buchacher 2015-07-08 18:51 ` Eric Sunshine 2015-08-04 8:24 ` Clemens Buchacher 2015-08-04 21:03 ` Junio C Hamano 2015-08-05 8:59 ` Linus Torvalds 2015-08-05 14:36 ` Clemens Buchacher 2015-08-05 16:27 ` Junio C Hamano
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).