inode problem when using git on a sshfs filesystem

inode problem when using git on a sshfs filesystem

[Yann Droneaud]

Hi,

For some days, my usage of git is not as seamless as before.

I'm using git along sshfs/fuse (don't blame me for that), and 
each time I try to rebase one of my branch, I have a conflict when applying
the third commit. Doing the same operation on a local filesystem works without any problem.

===== Part one: git =====

When I try to rebase one specific branch, git rebase failed when applying the third commit,
telling me about uncommited 

I've tried to do it from scratch, using git format-patch / git am
but git am also abort on the third patch with the error message:

error: <path>/<filename>: does not match index

So I've tried to diagnose the problem using :

 - git diff / git status : doesn't return anything.

 - git ls-tree HEAD -l <path>/<filename> : returns the correct mode and file size.

 - git log --raw --all --full-history -- <path>/<filename> : 
   returns valid information matching the one retrieved above.

 - git hash-object <path>/<filename> :
   gives the correct sha1 for the file, as recorded in the patch extracted using git format-patch 
   and reported by git ls-tree or git log (see before)

 - git diff-files : it shows a lot a file, all of them in same directory

   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename0>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename1>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename2>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename3>
   :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename>

BTW, there's no conflict when applying the patch manually with patch: the patch itself is fine
Using git apply --index also work, but only if it's applied alone:
apply each patches in series and git apply fails in the same third patch.

After diving into git source code and some debugging session, I've found
that the inode number recorded in the active_cache doesn't match the one
on the filesystem for <pach>/<filename> : that's why git apply --index refuse to apply the patch.

Then I tried to monitor stat() information for the file in <path> during
git operations.
1) After applying the first patch, files in <path> were affected different inode number
2) Using strace, I checked that git apply didn't make anything specials to thoses files.
The only thing strange git did, was trying to unlink(<path>), but this failed since the <path>
directory wasn't empty.

Note: the first patch remove, change and add some files in <path> directory, while 
the third patch changes another file in <path> directory

As a workaround: running git diff / git diff --cached / git status between each
git apply --index command seems to update the cache and allows me to apply all the patches
without problem. But it's not an easy path to follow when rebasing branches.

Surprisingly, when looking at strace output, it seems that git apply, once work done, is calling lstat() 
for all the files under <path>, and it sees the new inodes allocated to those files, but I don't know what 
it is doing with those information, if it's not stored in the index.

To conclude, it was a bit hard to diagnose from git point of view.

====== Part two: sshfs / fuse ======

At this time sshfs seems to be guilty of bad behavior, breaking somes POSIX rules.

So I tried the following testcase on another computer to reproduce the
problem outside of git.

Here the results:

$ mkdir dir
$ touch dir/a dir/b
$ stat -t dir/*
dir/a 0 0 81b4 500 500 15 3 1 0 0 1297882724 1297882724 1297882724 4096
dir/b 0 0 81b4 500 500 15 4 1 0 0 1297882726 1297882726 1297882726 4096
$ rmdir dir
rmdir: failed to remove `dir1': Operation not permitted
$ stat -t dir/*
dir/a 0 0 81b4 500 500 15 6 1 0 0 1297882724 1297882724 1297882724 4096
dir/b 0 0 81b4 500 500 15 7 1 0 0 1297882726 1297882726 1297882726 4096

One can see that inode 3 became inode 6 and inode 4 became inode 7 after the failed
unlink operation on dir. Which seems to be a bit uncommon for me.

Note: on a local filesystem, rmdir failed with message rmdir: failed to remove `dir1': Directory not empty

I try to add some debug support to fuse / sshfs in order to locate more precisely the problem:
(lines beginning by -/+ where added by me in libfuse, line beginning with --/++ in sshfs)

$ sshfs localhost:<export> <mount> -o sshfs_debug,debug,cache=no -d -f -s

unique: 22, opcode: FORGET (2), nodeid: 4, insize: 48, pid: 0
- forget 4
FORGET 4/1
DELETE: 4
+ forget 4
unique: 23, opcode: FORGET (2), nodeid: 3, insize: 48, pid: 0
- forget 3
FORGET 3/1
DELETE: 3
+ forget 3
unique: 24, opcode: RMDIR (11), nodeid: 1, insize: 44, pid: 9044
- rmdir 1 dir
rmdir /dir
-- rmdir(/dir)
[00020] RMDIR
  [00020]         STATUS       28bytes (0ms)
++ rmdir(/dir) = -1
   unique: 24, error: -1 (Operation not permitted), outsize: 16
+ rmdir 1 dir
unique: 25, opcode: FORGET (2), nodeid: 2, insize: 48, pid: 0
- forget 2
FORGET 2/1
DELETE: 2
+ forget 2

One can see that the reference to files under the directory are asked by
the kernel to be forgotten, even if the directory is not yet removed.

This seems a bit illogical since a directory with files under it can't
be removed (but FORGET could apply to file deleted but still referenced
by a process).

Note: if the file is opened, the inode associated to the file name
didn't change. Hopefully.

I've tried to reproduce the problem with other virtual filesystem like
shm / tmpfs / devtmpfs / ramfs : no problem.

I've also tried with NFS (local), and there's no problem too (the inode
numbers reported from NFS client side are the same than the server
side).

So it seems this a FUSE only problem, and I haven't found exactly why.

Regards.

-- 
Yann Droneaud



------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

Re: inode problem when using git on a sshfs filesystem

[Michael J Gruber]

Yann Droneaud venit, vidit, dixit 16.02.2011 23:04:
> Hi,
> 
> For some days, my usage of git is not as seamless as before.
> 
> I'm using git along sshfs/fuse (don't blame me for that), and 
> each time I try to rebase one of my branch, I have a conflict when applying
> the third commit. Doing the same operation on a local filesystem works without any problem.
> 
> ===== Part one: git =====
> 
> When I try to rebase one specific branch, git rebase failed when applying the third commit,
> telling me about uncommited 
> 
> I've tried to do it from scratch, using git format-patch / git am
> but git am also abort on the third patch with the error message:
> 
> error: <path>/<filename>: does not match index
> 
> So I've tried to diagnose the problem using :
> 
>  - git diff / git status : doesn't return anything.
> 
>  - git ls-tree HEAD -l <path>/<filename> : returns the correct mode and file size.
> 
>  - git log --raw --all --full-history -- <path>/<filename> : 
>    returns valid information matching the one retrieved above.
> 
>  - git hash-object <path>/<filename> :
>    gives the correct sha1 for the file, as recorded in the patch extracted using git format-patch 
>    and reported by git ls-tree or git log (see before)
> 
>  - git diff-files : it shows a lot a file, all of them in same directory
> 
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename0>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename1>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename2>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename3>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename>
> 
> BTW, there's no conflict when applying the patch manually with patch: the patch itself is fine
> Using git apply --index also work, but only if it's applied alone:
> apply each patches in series and git apply fails in the same third patch.
> 
> After diving into git source code and some debugging session, I've found
> that the inode number recorded in the active_cache doesn't match the one
> on the filesystem for <pach>/<filename> : that's why git apply --index refuse to apply the patch.
> 
> Then I tried to monitor stat() information for the file in <path> during
> git operations.
> 1) After applying the first patch, files in <path> were affected different inode number
> 2) Using strace, I checked that git apply didn't make anything specials to thoses files.
> The only thing strange git did, was trying to unlink(<path>), but this failed since the <path>
> directory wasn't empty.
> 
> Note: the first patch remove, change and add some files in <path> directory, while 
> the third patch changes another file in <path> directory
> 
> As a workaround: running git diff / git diff --cached / git status between each
> git apply --index command seems to update the cache and allows me to apply all the patches
> without problem. But it's not an easy path to follow when rebasing branches.
> 
> Surprisingly, when looking at strace output, it seems that git apply, once work done, is calling lstat() 
> for all the files under <path>, and it sees the new inodes allocated to those files, but I don't know what 
> it is doing with those information, if it's not stored in the index.
> 
> To conclude, it was a bit hard to diagnose from git point of view.
> 
> ====== Part two: sshfs / fuse ======
> 
> At this time sshfs seems to be guilty of bad behavior, breaking somes POSIX rules.
> 
> So I tried the following testcase on another computer to reproduce the
> problem outside of git.
> 
> Here the results:
> 
> $ mkdir dir
> $ touch dir/a dir/b
> $ stat -t dir/*
> dir/a 0 0 81b4 500 500 15 3 1 0 0 1297882724 1297882724 1297882724 4096
> dir/b 0 0 81b4 500 500 15 4 1 0 0 1297882726 1297882726 1297882726 4096
> $ rmdir dir
> rmdir: failed to remove `dir1': Operation not permitted
> $ stat -t dir/*
> dir/a 0 0 81b4 500 500 15 6 1 0 0 1297882724 1297882724 1297882724 4096
> dir/b 0 0 81b4 500 500 15 7 1 0 0 1297882726 1297882726 1297882726 4096
> 
> One can see that inode 3 became inode 6 and inode 4 became inode 7 after the failed
> unlink operation on dir. Which seems to be a bit uncommon for me.
> 
> Note: on a local filesystem, rmdir failed with message rmdir: failed to remove `dir1': Directory not empty
> 
> I try to add some debug support to fuse / sshfs in order to locate more precisely the problem:
> (lines beginning by -/+ where added by me in libfuse, line beginning with --/++ in sshfs)
> 
> $ sshfs localhost:<export> <mount> -o sshfs_debug,debug,cache=no -d -f -s
> 
> unique: 22, opcode: FORGET (2), nodeid: 4, insize: 48, pid: 0
> - forget 4
> FORGET 4/1
> DELETE: 4
> + forget 4
> unique: 23, opcode: FORGET (2), nodeid: 3, insize: 48, pid: 0
> - forget 3
> FORGET 3/1
> DELETE: 3
> + forget 3
> unique: 24, opcode: RMDIR (11), nodeid: 1, insize: 44, pid: 9044
> - rmdir 1 dir
> rmdir /dir
> -- rmdir(/dir)
> [00020] RMDIR
>   [00020]         STATUS       28bytes (0ms)
> ++ rmdir(/dir) = -1
>    unique: 24, error: -1 (Operation not permitted), outsize: 16
> + rmdir 1 dir
> unique: 25, opcode: FORGET (2), nodeid: 2, insize: 48, pid: 0
> - forget 2
> FORGET 2/1
> DELETE: 2
> + forget 2
> 
> One can see that the reference to files under the directory are asked by
> the kernel to be forgotten, even if the directory is not yet removed.
> 
> This seems a bit illogical since a directory with files under it can't
> be removed (but FORGET could apply to file deleted but still referenced
> by a process).
> 
> Note: if the file is opened, the inode associated to the file name
> didn't change. Hopefully.
> 
> I've tried to reproduce the problem with other virtual filesystem like
> shm / tmpfs / devtmpfs / ramfs : no problem.
> 
> I've also tried with NFS (local), and there's no problem too (the inode
> numbers reported from NFS client side are the same than the server
> side).
> 
> So it seems this a FUSE only problem, and I haven't found exactly why.
> 
> Regards.
> 

fuse-sshfs is a bit of a pita whenever you try to treat it like a "real"
file-system... OTOH, git makes assumptions on stability of device ids,
e.g. On such a (non)-fs, I'd suggest to try in this order of preference:

- Work from a local clone :) Pushing over ssh is fine.

- Have your workdir on a local file system, gitdir on sshfs.

- Try whether core.fsyncobjectfiles helps (shot in the dark).

- Try whether core.ignoreStat works for you. (This impacts your workflow!)

Michael

Re: [sshfs] inode problem when using git on a sshfs filesystem

[Miklos Szeredi]

On Wed, 16 Feb 2011, Yann Droneaud wrote:
> Hi,
> 
> For some days, my usage of git is not as seamless as before.
> 
> I'm using git along sshfs/fuse (don't blame me for that), and 
> each time I try to rebase one of my branch, I have a conflict when applying
> the third commit. Doing the same operation on a local filesystem works without any problem.

Yann, thanks for looking into this.

Your findings are not surprising: unlike NFS, sshfs doesn't provide
inode numbers and the fuse library also doesn't guarantee stable inode
numbers by default.

Fuse version 2.8.x has a "noforget" option that should provide stable
inode numbers, at the cost of unbounded memory use.  Could you please
try if this option fixes these issues?

Thanks,
Miklos

> 
> ===== Part one: git =====
> 
> When I try to rebase one specific branch, git rebase failed when applying the third commit,
> telling me about uncommited 
> 
> I've tried to do it from scratch, using git format-patch / git am
> but git am also abort on the third patch with the error message:
> 
> error: <path>/<filename>: does not match index
> 
> So I've tried to diagnose the problem using :
> 
>  - git diff / git status : doesn't return anything.
> 
>  - git ls-tree HEAD -l <path>/<filename> : returns the correct mode and file size.
> 
>  - git log --raw --all --full-history -- <path>/<filename> : 
>    returns valid information matching the one retrieved above.
> 
>  - git hash-object <path>/<filename> :
>    gives the correct sha1 for the file, as recorded in the patch extracted using git format-patch 
>    and reported by git ls-tree or git log (see before)
> 
>  - git diff-files : it shows a lot a file, all of them in same directory
> 
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename0>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename1>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename2>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename3>
>    :100644 100644 <sha1> 0000000000000000000000000000000000000000 M <path>/<filename>
> 
> BTW, there's no conflict when applying the patch manually with patch: the patch itself is fine
> Using git apply --index also work, but only if it's applied alone:
> apply each patches in series and git apply fails in the same third patch.
> 
> After diving into git source code and some debugging session, I've found
> that the inode number recorded in the active_cache doesn't match the one
> on the filesystem for <pach>/<filename> : that's why git apply --index refuse to apply the patch.
> 
> Then I tried to monitor stat() information for the file in <path> during
> git operations.
> 1) After applying the first patch, files in <path> were affected different inode number
> 2) Using strace, I checked that git apply didn't make anything specials to thoses files.
> The only thing strange git did, was trying to unlink(<path>), but this failed since the <path>
> directory wasn't empty.
> 
> Note: the first patch remove, change and add some files in <path> directory, while 
> the third patch changes another file in <path> directory
> 
> As a workaround: running git diff / git diff --cached / git status between each
> git apply --index command seems to update the cache and allows me to apply all the patches
> without problem. But it's not an easy path to follow when rebasing branches.
> 
> Surprisingly, when looking at strace output, it seems that git apply, once work done, is calling lstat() 
> for all the files under <path>, and it sees the new inodes allocated to those files, but I don't know what 
> it is doing with those information, if it's not stored in the index.
> 
> To conclude, it was a bit hard to diagnose from git point of view.
> 
> ====== Part two: sshfs / fuse ======
> 
> At this time sshfs seems to be guilty of bad behavior, breaking somes POSIX rules.
> 
> So I tried the following testcase on another computer to reproduce the
> problem outside of git.
> 
> Here the results:
> 
> $ mkdir dir
> $ touch dir/a dir/b
> $ stat -t dir/*
> dir/a 0 0 81b4 500 500 15 3 1 0 0 1297882724 1297882724 1297882724 4096
> dir/b 0 0 81b4 500 500 15 4 1 0 0 1297882726 1297882726 1297882726 4096
> $ rmdir dir
> rmdir: failed to remove `dir1': Operation not permitted
> $ stat -t dir/*
> dir/a 0 0 81b4 500 500 15 6 1 0 0 1297882724 1297882724 1297882724 4096
> dir/b 0 0 81b4 500 500 15 7 1 0 0 1297882726 1297882726 1297882726 4096
> 
> One can see that inode 3 became inode 6 and inode 4 became inode 7 after the failed
> unlink operation on dir. Which seems to be a bit uncommon for me.
> 
> Note: on a local filesystem, rmdir failed with message rmdir: failed to remove `dir1': Directory not empty
> 
> I try to add some debug support to fuse / sshfs in order to locate more precisely the problem:
> (lines beginning by -/+ where added by me in libfuse, line beginning with --/++ in sshfs)
> 
> $ sshfs localhost:<export> <mount> -o sshfs_debug,debug,cache=no -d -f -s
> 
> unique: 22, opcode: FORGET (2), nodeid: 4, insize: 48, pid: 0
> - forget 4
> FORGET 4/1
> DELETE: 4
> + forget 4
> unique: 23, opcode: FORGET (2), nodeid: 3, insize: 48, pid: 0
> - forget 3
> FORGET 3/1
> DELETE: 3
> + forget 3
> unique: 24, opcode: RMDIR (11), nodeid: 1, insize: 44, pid: 9044
> - rmdir 1 dir
> rmdir /dir
> -- rmdir(/dir)
> [00020] RMDIR
>   [00020]         STATUS       28bytes (0ms)
> ++ rmdir(/dir) = -1
>    unique: 24, error: -1 (Operation not permitted), outsize: 16
> + rmdir 1 dir
> unique: 25, opcode: FORGET (2), nodeid: 2, insize: 48, pid: 0
> - forget 2
> FORGET 2/1
> DELETE: 2
> + forget 2
> 
> One can see that the reference to files under the directory are asked by
> the kernel to be forgotten, even if the directory is not yet removed.
> 
> This seems a bit illogical since a directory with files under it can't
> be removed (but FORGET could apply to file deleted but still referenced
> by a process).
> 
> Note: if the file is opened, the inode associated to the file name
> didn't change. Hopefully.
> 
> I've tried to reproduce the problem with other virtual filesystem like
> shm / tmpfs / devtmpfs / ramfs : no problem.
> 
> I've also tried with NFS (local), and there's no problem too (the inode
> numbers reported from NFS client side are the same than the server
> side).
> 
> So it seems this a FUSE only problem, and I haven't found exactly why.
> 
> Regards.
> 
> -- 
> Yann Droneaud

------------------------------------------------------------------------------
The ultimate all-in-one performance toolkit: Intel(R) Parallel Studio XE:
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb

Re: [sshfs] inode problem when using git on a sshfs filesystem

[Yann Droneaud]

> On Wed, 16 Feb 2011, Yann Droneaud wrote:
>> Hi,
>>
>> For some days, my usage of git is not as seamless as before.
>>
>> I'm using git along sshfs/fuse (don't blame me for that), and
>> each time I try to rebase one of my branch, I have a conflict when
>> applying
>> the third commit. Doing the same operation on a local filesystem works
>> without any problem.
>
> Yann, thanks for looking into this.
>
> Your findings are not surprising: unlike NFS, sshfs doesn't provide
> inode numbers and the fuse library also doesn't guarantee stable inode
> numbers by default.
>

But why does it have such behavior when trying to rmdir() a non empty
directory ?

> Fuse version 2.8.x has a "noforget" option that should provide stable
> inode numbers, at the cost of unbounded memory use.  Could you please
> try if this option fixes these issues?
>

Yes, this option seems to fix the problem.

I will try it for a while to see if this is stable enough for a full day
of git working. (How can I check memory usage ?)

BTW, the [no]forget option did not appears in sshfs --help output.

Regards.

-- 
Yann Droneaud

Re: inode problem when using git on a sshfs filesystem

[Yann Droneaud]

> Hi,
>
> For some days, my usage of git is not as seamless as before.
>
> I'm using git along sshfs/fuse (don't blame me for that), and
> each time I try to rebase one of my branch, I have a conflict when
> applying
> the third commit. Doing the same operation on a local filesystem works
> without any problem.
>
> ===== Part one: git =====
>
> When I try to rebase one specific branch, git rebase failed when applying
> the third commit,
> telling me about uncommited
>

For the record, here's the exact message for a rebase:

$ git rebase origin
First, rewinding head to replay your work on top of it...
Applying: patch1
Applying: patch2
Applying: patch3
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
error: Your local changes to '<path>/<filename>' would be overwritten by
merge.  Aborting.
Please, commit your changes or stash them before you can merge.
Failed to merge in the changes.
Patch failed at 0003 patch3


> As a workaround: running git diff / git diff --cached / git status between
> each
> git apply --index command seems to update the cache and allows me to apply
> all the patches
> without problem. But it's not an easy path to follow when rebasing
> branches.
>

The exact command to update the index is "git diff".

Regards.

-- 
Yann Droneaud

Re: [sshfs] inode problem when using git on a sshfs filesystem

[Miklos Szeredi]

On Thu, 17 Feb 2011, Yann Droneaud wrote:
> > On Wed, 16 Feb 2011, Yann Droneaud wrote:
> >> Hi,
> >>
> >> For some days, my usage of git is not as seamless as before.
> >>
> >> I'm using git along sshfs/fuse (don't blame me for that), and
> >> each time I try to rebase one of my branch, I have a conflict when
> >> applying
> >> the third commit. Doing the same operation on a local filesystem works
> >> without any problem.
> >
> > Yann, thanks for looking into this.
> >
> > Your findings are not surprising: unlike NFS, sshfs doesn't provide
> > inode numbers and the fuse library also doesn't guarantee stable inode
> > numbers by default.
> >
> 
> But why does it have such behavior when trying to rmdir() a non empty
> directory ?

The VFS (part of the linux kernel that implements the generic
filesystem logic) clears the directory entry from the cache prior to
actually trying to remove the directory.  This has the effect that any
children of the directory are also cleared from the cache, hence the
behavior you see in rmdir.

> 
> > Fuse version 2.8.x has a "noforget" option that should provide stable
> > inode numbers, at the cost of unbounded memory use.  Could you please
> > try if this option fixes these issues?
> >
> 
> Yes, this option seems to fix the problem.
> 
> I will try it for a while to see if this is stable enough for a full day
> of git working. (How can I check memory usage ?)
> 
> BTW, the [no]forget option did not appears in sshfs --help output.

Oh, that's an oversight on my part.  Will fix.

Thanks,
Miklos

Re: [sshfs] inode problem when using git on a sshfs filesystem

[Yann Droneaud]

> On Thu, 17 Feb 2011, Yann Droneaud wrote:
>> > On Wed, 16 Feb 2011, Yann Droneaud wrote:
>> >> Hi,
>> >>
>> >> For some days, my usage of git is not as seamless as before.
>> >>
>> >> I'm using git along sshfs/fuse (don't blame me for that), and
>> >> each time I try to rebase one of my branch, I have a conflict when
>> >> applying
>> >> the third commit. Doing the same operation on a local filesystem
>> works
>> >> without any problem.
>> >
>> > Yann, thanks for looking into this.
>> >
>> > Your findings are not surprising: unlike NFS, sshfs doesn't provide
>> > inode numbers and the fuse library also doesn't guarantee stable inode
>> > numbers by default.
>> >
>>
>> But why does it have such behavior when trying to rmdir() a non empty
>> directory ?
>
> The VFS (part of the linux kernel that implements the generic
> filesystem logic) clears the directory entry from the cache prior to
> actually trying to remove the directory.  This has the effect that any
> children of the directory are also cleared from the cache, hence the
> behavior you see in rmdir.
>

I tried to check that behavor: if the VFS is dropping dentry before doing
a rmdir(), them lookup files under this directory should be slower than
before rmdir().

On a ext4 filesystem, directory with 338 sub directories and 1992 files, 
i've tried the following commands:

/* drop all */
# echo 3 > /proc/sys/vm/drop_caches
# time ls -alR directory > /dev/null
real	0m0.140s
user	0m0.000s
sys	0m0.080s

/* drop cache */
# echo 1 > /proc/sys/vm/drop_caches
# time ls -alR directory > /dev/null
real	0m0.119s
user	0m0.000s
sys	0m0.072s

/* drop dentry and inode */
# echo 2 > /proc/sys/vm/drop_caches
# time ls -alR directory > /dev/null
real	0m0.089s
user	0m0.016s
sys	0m0.040s

/* read from cache */
# time ls -alR directory > /dev/null
real	0m0.063s
user	0m0.004s
sys	0m0.036s

$ rmdir directory
rmdir: failed to remove `directory': Directory not empty

/* re read from cache */
$ time ls -alR directory > /dev/null
real	0m0.065s
user	0m0.012s
sys	0m0.036s

As you can see, after doing rmdir(), the time taken to walk trough the
directories is the same than before calling it, so the dentry seems not
flushed out of the cache after the rmdir().

I really prefer this behavior. The one you explained would be a real pain:
if one user call rmdir on / inside a loop, the full dentry cache will be
dropped each time: this would affect performance for the whole system.

But perhaps I doesn't understand how the VFS is working, which is the most
probable solution. Please correct me.

Regards.

-- 
Yann Droneaud

Re: [sshfs] inode problem when using git on a sshfs filesystem

[Miklos Szeredi]

On Thu, 17 Feb 2011, Yann Droneaud wrote:
> > The VFS (part of the linux kernel that implements the generic
> > filesystem logic) clears the directory entry from the cache prior to
> > actually trying to remove the directory.  This has the effect that any
> > children of the directory are also cleared from the cache, hence the
> > behavior you see in rmdir.
> >
> 
> I tried to check that behavor: if the VFS is dropping dentry before doing
> a rmdir(), them lookup files under this directory should be slower than
> before rmdir().
> 
> On a ext4 filesystem, directory with 338 sub directories and 1992 files, 
> i've tried the following commands:
> 
> /* drop all */
> # echo 3 > /proc/sys/vm/drop_caches
> # time ls -alR directory > /dev/null
> real	0m0.140s
> user	0m0.000s
> sys	0m0.080s
> 
> /* drop cache */
> # echo 1 > /proc/sys/vm/drop_caches
> # time ls -alR directory > /dev/null
> real	0m0.119s
> user	0m0.000s
> sys	0m0.072s
> 
> /* drop dentry and inode */
> # echo 2 > /proc/sys/vm/drop_caches
> # time ls -alR directory > /dev/null
> real	0m0.089s
> user	0m0.016s
> sys	0m0.040s
> 
> /* read from cache */
> # time ls -alR directory > /dev/null
> real	0m0.063s
> user	0m0.004s
> sys	0m0.036s
> 
> $ rmdir directory
> rmdir: failed to remove `directory': Directory not empty
> 
> /* re read from cache */
> $ time ls -alR directory > /dev/null
> real	0m0.065s
> user	0m0.012s
> sys	0m0.036s
> 
> As you can see, after doing rmdir(), the time taken to walk trough the
> directories is the same than before calling it, so the dentry seems not
> flushed out of the cache after the rmdir().

tucsk:~ # time find /usr -ls > /dev/null

real    0m1.922s
user    0m1.104s
sys     0m0.800s
tucsk:~ # rmdir /usr
rmdir: failed to remove `/usr': Directory not empty
tucsk:~ # time find /usr -ls > /dev/null

real    0m2.637s
user    0m1.172s
sys     0m1.448s
tucsk:~ # time find /usr -ls > /dev/null

real    0m1.943s
user    0m1.068s
sys     0m0.848s
tucsk:~ # 

As you can see it does get significantly slower after the rmdir.

> I really prefer this behavior. The one you explained would be a real pain:
> if one user call rmdir on / inside a loop, the full dentry cache will be
> dropped each time: this would affect performance for the whole system.

Yes.  But in practice this doesn't really matter, removing non-empty
directories happens rarely.

Thanks,
Miklos

Re: [fuse-devel] [sshfs] inode problem when using git on a sshfs filesystem

[Goswin von Brederlow]

"Yann Droneaud" <yann@droneaud.fr> writes:

>> On Wed, 16 Feb 2011, Yann Droneaud wrote:
>> Fuse version 2.8.x has a "noforget" option that should provide stable
>> inode numbers, at the cost of unbounded memory use.  Could you please
>> try if this option fixes these issues?
>>
>
> Yes, this option seems to fix the problem.
>
> I will try it for a while to see if this is stable enough for a full day
> of git working. (How can I check memory usage ?)

The memory usage comes from the inodes. Fuse will never forget an inode
(unless you delete the dile/dir) that was once visited (thereby giving
allways the same inode number). As long as you don't have a billion
files that generally doesn't matter.

MfG
        Goswin