[PATCH] git_open_noatime: return with errno=0 on success

[PATCH] git_open_noatime: return with errno=0 on success

[Clemens Buchacher]

In read_sha1_file_extended we die if read_object fails with a fatal
error. We detect a fatal error if errno is non-zero and is not
ENOENT. If the object could not be read because it does not exist,
this is not considered a fatal error and we want to return NULL.

Somewhere down the line, read_object calls git_open_noatime to open
a pack index file, for example. We first try open with O_NOATIME.
If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the
second open succeeds, errno is however still set to EPERM from the
first attemt. When we finally determine that the object does not
exist, read_object returns NULL and read_sha1_file_extended dies
with a fatal error:

    fatal: failed to read object <sha1>: Operation not permitted

Fix this by resetting errno to zero before we call open again.

Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
Helped-by: Martin Schröder <martin.h.schroeder@intel.com>
---
 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sha1_file.c b/sha1_file.c
index 77cd81d..62b7ad6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
 	static int sha1_file_open_flag = O_NOATIME;
 
 	for (;;) {
+		errno = 0;
 		int fd = open(name, O_RDONLY | sha1_file_open_flag);
 		if (fd >= 0)
 			return fd;
-- 
1.9.4

Re: [PATCH] git_open_noatime: return with errno=0 on success

[Eric Sunshine]

On Wed, Jul 8, 2015 at 8:38 AM, Clemens Buchacher
<clemens.buchacher@intel.com> wrote:
> In read_sha1_file_extended we die if read_object fails with a fatal
> error. We detect a fatal error if errno is non-zero and is not
> ENOENT. If the object could not be read because it does not exist,
> this is not considered a fatal error and we want to return NULL.
>
> Somewhere down the line, read_object calls git_open_noatime to open
> a pack index file, for example. We first try open with O_NOATIME.
> If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the
> second open succeeds, errno is however still set to EPERM from the
> first attemt. When we finally determine that the object does not

s/attemt/attempt/

> exist, read_object returns NULL and read_sha1_file_extended dies
> with a fatal error:
>
>     fatal: failed to read object <sha1>: Operation not permitted
>
> Fix this by resetting errno to zero before we call open again.
>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
> Helped-by: Martin Schröder <martin.h.schroeder@intel.com>

[PATCH] git_open_noatime: return with errno=0 on success

[Clemens Buchacher]

In read_sha1_file_extended we die if read_object fails with a fatal
error. We detect a fatal error if errno is non-zero and is not
ENOENT. If the object could not be read because it does not exist,
this is not considered a fatal error and we want to return NULL.

Somewhere down the line, read_object calls git_open_noatime to open
a pack index file, for example. We first try open with O_NOATIME.
If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the
second open succeeds, errno is however still set to EPERM from the
first attempt. When we finally determine that the object does not
exist, read_object returns NULL and read_sha1_file_extended dies
with a fatal error:

    fatal: failed to read object <sha1>: Operation not permitted

Fix this by resetting errno to zero before we call open again.

Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
---

This is a re-submission without changes except for a typo fix in the
comments (thanks Eric). The original submission received no other
comments, but I think it is a clear improvement and I hope it was just
missed the first time.

Best regards,
Clemens

 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sha1_file.c b/sha1_file.c
index 77cd81d..62b7ad6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
 	static int sha1_file_open_flag = O_NOATIME;
 
 	for (;;) {
+		errno = 0;
 		int fd = open(name, O_RDONLY | sha1_file_open_flag);
 		if (fd >= 0)
 			return fd;
-- 
1.9.4

Re: [PATCH] git_open_noatime: return with errno=0 on success

[Junio C Hamano]

Clemens Buchacher <clemens.buchacher@intel.com> writes:

> diff --git a/sha1_file.c b/sha1_file.c
> index 77cd81d..62b7ad6 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
>  	static int sha1_file_open_flag = O_NOATIME;
>  
>  	for (;;) {
> +		errno = 0;
>  		int fd = open(name, O_RDONLY | sha1_file_open_flag);

Please avoid decl-after-stmt, which this codebase does not accept.

>  		if (fd >= 0)
>  			return fd;

More importantly, is this the right place to clear errno?

I would agree it is a good idea to clear it after seeing the first
open fail due to lack of O_NOATIME before trying open for the second
time, iow, more like this?


 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/sha1_file.c b/sha1_file.c
index 1cee438..bf2f229 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1467,6 +1467,7 @@ int git_open_noatime(const char *name)
 
 		/* Might the failure be due to O_NOATIME? */
 		if (errno != ENOENT && sha1_file_open_flag) {
+			errno = 0;
 			sha1_file_open_flag = 0;
 			continue;
 		}

Re: [PATCH] git_open_noatime: return with errno=0 on success

[Linus Torvalds]

On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
> I would agree it is a good idea to clear it after seeing the first
> open fail due to lack of O_NOATIME before trying open for the second
> time, iow, more like this?

So I don't think this is _wrong_ per se, but I think the deeper issue
is that somebody cares about 'errno' here in the first place.

A stale 'errno' generally shouldn't matter, because we either

 (a) return success (and nobody should look at errno)

or

 (b) return an error later, without setting errno for that _later_ error.

and I think either of those two situations are the real bug, and this
"clear stale errno" is just a workaround.

But as mentioned, I don't think clearign errno is wrong, so I'm not
objecting to the patch. I just suspect there's something else goign on
too..

              Linus

Re: [PATCH] git_open_noatime: return with errno=0 on success

[Clemens Buchacher]

On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote:
> On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
> >
> > I would agree it is a good idea to clear it after seeing the first
> > open fail due to lack of O_NOATIME before trying open for the second
> > time, iow, more like this?

Looks good to me.

> So I don't think this is _wrong_ per se, but I think the deeper issue
> is that somebody cares about 'errno' here in the first place.
> 
> A stale 'errno' generally shouldn't matter, because we either
> 
>  (a) return success (and nobody should look at errno)
> 
> or
> 
>  (b) return an error later, without setting errno for that _later_ error.
> 
> and I think either of those two situations are the real bug, and this
> "clear stale errno" is just a workaround.

I agree. But I do not see how to get there easily.

We are trying to read an object. We first try to read from a pack. We
may encounter broken pack files, missing index files, unreadable files,
but those errors are not necessarily fatal since we may still be able to
read the object from the next pack file or from a sha1 file.

If finally we do not find the object anywhere, in
read_sha1_file_extended we try our best to die with an appropriate error
message, for example by looking at errno, and otherwise we just return
NULL. Most callers seem to die explicitly or they dereference the null
pointer.

I think we should instead output error messages closer to the source,
like for example in map_sha1_file, but continue anyway. In particular we
should immediately report failures due to EPERM or unexpected ENOENT. In
the end we may return NULL without another message, but at least the
user should have some hints about what went wrong along the way.

Re: [PATCH] git_open_noatime: return with errno=0 on success

[Junio C Hamano]

Clemens Buchacher <clemens.buchacher@intel.com> writes:

> On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote:
> ...
>> A stale 'errno' generally shouldn't matter, because we either
>> 
>>  (a) return success (and nobody should look at errno)
>> 
>> or
>> 
>>  (b) return an error later, without setting errno for that _later_ error.
>> 
>> and I think either of those two situations are the real bug, and this
>> "clear stale errno" is just a workaround.
>
> I agree. But I do not see how to get there easily.
>
> We are trying to read an object. We first try to read from a pack. We
> may encounter broken pack files, missing index files, unreadable files,
> but those errors are not necessarily fatal since we may still be able to
> read the object from the next pack file or from a sha1 file.
>
> If finally we do not find the object anywhere, in
> read_sha1_file_extended we try our best to die with an appropriate error
> message, for example by looking at errno, and otherwise we just return
> NULL. Most callers seem to die explicitly or they dereference the null
> pointer.
>
> I think we should instead output error messages closer to the source,
> like for example in map_sha1_file, but continue anyway.

Hmm, if we find one data source unreadable but an alternative
usable, do we really want that error message?  What should it say?
"error: cannot read from pack"?  Such a message, unless we later
give "info: but we managed to read it from elsewhere" and make sure
these two messages are clearly associated with each other, would
make things unnecessarily alarming, wouldn't it?

Perhaps we should not rely so heavily on 'errno', but explicitly
pass around error code (or enough information to formulate an
intelligent message at the end) in the callchain instead.

Then the earlier part can notice EPERM on a pack, for example, and
return to the caller, and after consulting an alternate data source
(e.g. loose object file), the caller can then choose to say "we
managed to read the data, but FYI, you may want to check the
permission bits of this pack", or choose to stay silent.