[PATCH] clone: Skip pack-*.keep files when cloning locally

[PATCH] clone: Skip pack-*.keep files when cloning locally

[Jens Lindstrom]

The pack-*.keep files are temporary, and serve no purpose in the
clone.  They would probably also never be deleted from the clone if
copied, since the process that created them only expects to have to
delete them from the original repository.

Worse, though, they are created with access bits 0600, so if the
user trying to clone the repository is different from the user that
caused the pack-*.keep file to be created, the clone will likely
fail due to not being allowed to read (and thus copy) the file in
the first place.

Signed-off-by: Jens Lindstrom <jl@opera.com>
---
 builtin/clone.c |    5 +++++
 1 file changed, 5 insertions(+)

diff --git a/builtin/clone.c b/builtin/clone.c
index 035ab64..0ec0ec9 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -342,6 +342,11 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,
 			continue;
 		}
 
+		/* Skip pack-*.keep files, they are temporary and not
+		   relevant to the clone, and might not be accessible. */
+		if (!strcmp(src->buf + strlen(src->buf) - 5, ".keep"))
+			continue;
+
 		/* Files that cannot be copied bit-for-bit... */
 		if (!strcmp(src->buf + src_baselen, "/info/alternates")) {
 			copy_alternates(src, dest, src_repo);
-- 
1.7.10.4

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Junio C Hamano]

Jens Lindstrom <jl@opera.com> writes:

Hmph.  I am of two minds here.

> The pack-*.keep files are temporary, and serve no purpose in the
> clone.

They are not temporary, actually.  A user can deliberatey create a
"keep" marker after packing with a good set of parameters, so that
the resulting pack will be kept, instead of letting a later repack
(with faster set of parameters) remove and replace it with less
optimal results.

> Worse, though, they are created with access bits 0600, so if the
> user trying to clone the repository is different from the user that
> caused the pack-*.keep file to be created, the clone will likely
> fail due to not being allowed to read (and thus copy) the file in
> the first place.

I am perfectly fine with a change that allows a local clone to skip
and not error out when such a "keep" marker cannot be copied, I do
not know if it is a good idea to unconditionally skip and not even
attempt to copy it.

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> I am perfectly fine with a change that allows a local clone to skip
> and not error out when such a "keep" marker cannot be copied, I do
> not know if it is a good idea to unconditionally skip and not even
> attempt to copy it.

That is, something like this, perhaps?

We could even create an empty file, as it is only the presense that
matters for ".keep" files, but I found it a bit too much special
casing to my taste.

 builtin/clone.c        |  9 ++++++++-
 t/t5701-clone-local.sh | 15 +++++++++++++++
 2 files changed, 23 insertions(+), 1 deletion(-)

diff --git a/builtin/clone.c b/builtin/clone.c
index 66bff57..4b7cd9b 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -356,8 +356,15 @@ static void copy_or_link_directory(struct strbuf *src, struct strbuf *dest,
 				die_errno(_("failed to create link '%s'"), dest->buf);
 			option_no_hardlinks = 1;
 		}
-		if (copy_file_with_time(dest->buf, src->buf, 0666))
+		if (copy_file_with_time(dest->buf, src->buf, 0666)) {
+			if (!strncmp(src->buf + src_baselen, "/pack/pack-", 11) &&
+			    !suffixcmp(src->buf, ".keep"))
+				goto skip;
+
 			die_errno(_("failed to copy file to '%s'"), dest->buf);
+		skip:
+			warning("skipping %s", src->buf);
+		}
 	}
 	closedir(dir);
 }
diff --git a/t/t5701-clone-local.sh b/t/t5701-clone-local.sh
index 7ff6e0e..bb5dddd 100755
--- a/t/t5701-clone-local.sh
+++ b/t/t5701-clone-local.sh
@@ -134,4 +134,19 @@ test_expect_success 'cloning a local path with --no-local does not hardlink' '
 	! repo_is_hardlinked force-nonlocal
 '
 
+test_expect_success SANITY 'clone --no-hardlinks with unreadable .keep' '
+	mkdir strictsrc &&
+	(
+		cd strictsrc &&
+		git init &&
+		git commit --allow-empty -m initial &&
+		git repack -a -d &&
+		packname=$(echo .git/objects/pack/pack-*.idx) &&
+		keepname=${packname%.idx}.keep &&
+		>"$keepname" &&
+		chmod a= "$keepname"
+	) &&
+	git clone --local --no-hardlinks strictsrc dst
+'
+
 test_done

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Jens Lindström]

On Fri, Jun 28, 2013 at 8:38 PM, Junio C Hamano <gitster@pobox.com> wrote:

>> The pack-*.keep files are temporary, and serve no purpose in the
>> clone.
>
> They are not temporary, actually. A user can deliberatey create a
> "keep" marker after packing with a good set of parameters, so that
> the resulting pack will be kept, instead of letting a later repack
> (with faster set of parameters) remove and replace it with less
> optimal results.

Ah, I see.  Was (obviously) not aware of that.  It would perhaps be a
good idea to be able to differentiate between such permanent keep
files and the temporary ones created by built-in commands.

Also, even if some keep files are permanent in the source repository,
is it always a good idea to copy them over to the clone?  This would
only happen for some types of clones, anyway.


On Fri, Jun 28, 2013 at 10:38 PM, Junio C Hamano <gitster@pobox.com> wrote:

> That is, something like this, perhaps?

Comments:

With this patch, it still fails with --local, when the link() call
fails.  This seems a bit odd, in particular in the cases where --local
is implied.  IOW, one would not expect that adding --local would
change behavior, but here adding it causes the operation to fail.

Also, since failing to link() once implicitly enables --no-hardlinks,
it would copy the rest of the repository without trying to use link(),
which might make the whole operation much more expensive.

Applying the exception for inaccessible .keep files for link() as well
would seem a better solution to me.

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Junio C Hamano]

Jens Lindström <jl@opera.com> writes:

> On Fri, Jun 28, 2013 at 8:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
>>> The pack-*.keep files are temporary, and serve no purpose in the
>>> clone.
>>
>> They are not temporary, actually. A user can deliberatey create a
>> "keep" marker after packing with a good set of parameters, so that
>> the resulting pack will be kept, instead of letting a later repack
>> (with faster set of parameters) remove and replace it with less
>> optimal results.
>
> Ah, I see.  Was (obviously) not aware of that.  It would perhaps be a
> good idea to be able to differentiate between such permanent keep
> files and the temporary ones created by built-in commands.

I am not sure if we should care that deeply about them in the first
place.  For these temporary ".keep" lockfiles to matter, you have to
be doing "clone --local --no-hardlinks", which is a "cp -R" that
bypasses the usual Git transport mechanism, while there is still
activity in the source repository (a fetch is slurping new objects
into a newly created pack with such a temporary ".keep" lockfile,
the refs may not have been updated yet).  The result is expected to
have inconsistencies even if ".keep" were readable in such a copy.

> Also, even if some keep files are permanent in the source repository,
> is it always a good idea to copy them over to the clone?

The local "cheap cp -R clone" are primarily used by people copying
their own private repository that is in a quiescent state, and in
that set-up, copying ".keep" by default has been a good idea.  Of
course, after copying to a new repository, if they want to repack
with different parameters, they need to unplug ".keep" files
manually, and it can be argued that the default could have been not
to copy and we could have forced those who want to reuse a tight
pack they created earlier to manually copy or create the ".keep"
files instead.
>
> On Fri, Jun 28, 2013 at 10:38 PM, Junio C Hamano <gitster@pobox.com> wrote:
>
>> That is, something like this, perhaps?
>
> Comments:
>
> With this patch, it still fails with --local, when the link() call
> fails.

Oh, I wasn't even attempting to look at the hardlink codepath in the
"something like this" illustration patch ;-).

Besides, I think you can make a hardlink to a file that you cannot
read.

If you apply only the parts of the patch that adds the test, without
applying either your patch or mine to builtin/clone.c, and remove
"--no-hardlinks" to force the hardlink codepath, what happens?  We
create an unreadable $keepname in strictsrc repository and I think
the local clone (which is not "cp -R" but something like "find"
piped to "cpio -pluadm") would make a "copy" of it just fine.

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Jens Lindström]

On Mon, Jul 1, 2013 at 6:20 PM, Junio C Hamano <gitster@pobox.com> wrote:
> I am not sure if we should care that deeply about them in the first
> place.

Fine by me; I don't really have a strong opinion on the matter.

> Besides, I think you can make a hardlink to a file that you cannot
> read.

Not always.  The Linux kernel can at least be configured not to allow
it.  It seems this is enabled by default in at least Debian.

This restriction had me a bit confused when I was testing variations
here; I expected all "access denied" failures to be because of .keep
files, but in fact creating hardlinks to other files (.idx and .pack)
failed too, even though they were readable.  This caused "git clone
--local" to fail, while "git clone" succeeded after falling back to
copying instead of linking.

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Junio C Hamano]

Jens Lindström <jl@opera.com> writes:

> On Mon, Jul 1, 2013 at 6:20 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> I am not sure if we should care that deeply about them in the first
>> place.
>
> Fine by me; I don't really have a strong opinion on the matter.
>
>> Besides, I think you can make a hardlink to a file that you cannot
>> read.
>
> Not always.  The Linux kernel can at least be configured not to allow
> it.  It seems this is enabled by default in at least Debian.

You learn a new thing every day, I guess.  I am on Debian, I do not
think I did any customization in that area, and I can hardlink just
fine.

> This restriction had me a bit confused when I was testing variations
> here; I expected all "access denied" failures to be because of .keep
> files, but in fact creating hardlinks to other files (.idx and .pack)
> failed too, even though they were readable.  

Is it possible that you are tripping cross-device link?  The reason
why we have "attempt to hardlink but fall back to copy" is exactly
because it is fairly common that people try local-cheap clone without
realizing the source and the destination may be on separate filesystems.

Re: [PATCH] clone: Skip pack-*.keep files when cloning locally

[Jens Lindström]

On Wed, Jul 3, 2013 at 7:26 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Jens Lindström <jl@opera.com> writes:
>
>> Not always.  The Linux kernel can at least be configured not to allow
>> it.  It seems this is enabled by default in at least Debian.
>
> You learn a new thing every day, I guess.  I am on Debian, I do not
> think I did any customization in that area, and I can hardlink just
> fine.

To configure it, write "1" (on) or "0" (off) to
/proc/sys/fs/protected_{hard,sym}links.  I can't remember (or imagine)
that I enabled it on any of my systems.  One of my systems is Debian
Squeeze with a 2.6.32 kernel, and it doesn't have those files, so I
guess it might have been added in some more recent kernel version.

>> This restriction had me a bit confused when I was testing variations
>> here; I expected all "access denied" failures to be because of .keep
>> files, but in fact creating hardlinks to other files (.idx and .pack)
>> failed too, even though they were readable.
>
> Is it possible that you are tripping cross-device link?  The reason
> why we have "attempt to hardlink but fall back to copy" is exactly
> because it is fairly common that people try local-cheap clone without
> realizing the source and the destination may be on separate filesystems.

No, I was certainly cloning within a single file system, and I can
confirm that a plain "ln src dest" command fails unless the user can
both read and write the source file.  So if the cloning user has
read-only access to the repository, copying will work and linking
won't (depending on the kernel,) in which case it of course is
excellent that git falls back to copying instead of linking.