RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the real path

[Eric DeCosta <edecosta@mathworks.com>]

> -----Original Message-----
> From: Eric DeCosta
> Sent: Tuesday, September 6, 2022 3:34 PM
> To: 'Jeff Hostetler' <git@jeffhostetler.com>; 'Eric DeCosta via GitGitGadget'
> <gitgitgadget@gmail.com>; 'git@vger.kernel.org' <git@vger.kernel.org>
> Subject: RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the
> real path
> 
> 
> 
> > -----Original Message-----
> > From: Eric DeCosta
> > Sent: Tuesday, September 6, 2022 3:02 PM
> > To: Jeff Hostetler <git@jeffhostetler.com>; Eric DeCosta via
> > GitGitGadget <gitgitgadget@gmail.com>; git@vger.kernel.org
> > Subject: RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths
> > to the real path
> >
> >
> >
> > > -----Original Message-----
> > > From: Jeff Hostetler <git@jeffhostetler.com>
> > > Sent: Tuesday, September 6, 2022 1:14 PM
> > > To: Eric DeCosta <edecosta@mathworks.com>; Eric DeCosta via
> > > GitGitGadget <gitgitgadget@gmail.com>; git@vger.kernel.org
> > > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event
> > > paths to the real path
> > >
> > >
> > >
> > > On 9/2/22 12:35 PM, Eric DeCosta wrote:
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Jeff Hostetler <git@jeffhostetler.com>
> > > >> Sent: Thursday, September 1, 2022 4:06 PM
> > > >> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>;
> > > >> git@vger.kernel.org
> > > >> Cc: Eric DeCosta <edecosta@mathworks.com>
> > > >> Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event
> > > >> paths to the real path
> > > >>
> > > >>
> > > >>
> > > >> On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote:
> > > >>> From: Eric DeCosta <edecosta@mathworks.com>
> > > >>>
> > > >>> Consider the following network working directory that is mounted
> > > >>> under
> > > >>> /System/Volumes/Data:
> > > >>>
> > > >>> /network/working/directory
> > > >>>
> > > >>> The git working directory path is:
> > > >>>
> > > >>> /System/Volumes/Data/network/working/directory
> > > >>>
> > > >>> The paths reported by FSEvents always start with /network.
> > > >>> fsmonitor expects paths to be under the working directory;
> > > >>> therefore it fails to match /network/... and ignores the change.
> > > >>
> > > >> I'm not sure I understand what's going on here.
> > > >> Are you saying that FSEvents reports network mounted directories
> > > >> with a path relative to the mount-point rather than an absolute path?
> > > >>
> > > >
> > > > Yes
> > > >
> > > >> In your example, is "/network/working/directory" a valid path on
> > > >> your system (that also happens to be the same directory as
> > > >> "/System/Volumes/...")
> > > >>
> > > >> That is, do you have some aliasing going on where both paths are
> > > >> valid -- like a pair of hard-linked directories?
> > > >> Or, is there something special about a network mount point?
> > > >>
> > > >>
> > > >> Did you have to "cd /System/Volumes/..." to get Git to have the
> > > >> absolute path be this? Or were you doing a "cd /network/..."?
> > > >> (Sorry to ask so many questions but I don't have a pair of
> > > >> systems to test any of this on right now.)
> > > >>
> > > >
> > > > "/network/working/directory" is mounted under
> > > > "/System/Volumes/Data/network/working/directory"
> > > >
> > > > There is also a symlink:
> > > >
> > > > "/network" -> "/System/Volumes/Data/network"
> > > >
> > > > No matter if I "cd /System/Volumes/Data/network/working/directory"
> > > > or "cd /network/working/directory" the paths reported by FSEvents
> > > > always start with "/network/working/directory"
> > > >
> > > > If I do a similar symlink with local directories, I do not get
> > > > this unexpected behavior.
> > > >
> > > > I need to remove the symlink and try it that way, but I need to
> > > > coordinate with the machine's owner.
> > >
> > >
> > > I think you have stumbled upon a recent MacOS feature called
> "firmlinks".
> > > I'm just reading up on it myself, so I shouldn't speculate here
> > > (yet), but maybe you too could do some reading on the subject.
> > >
> > > This makes me wonder if the symlink is historical. The real magic is
> > > in the firmlinks. For example, if I do:
> > >
> > > $ (cd / ; ls -l | grep Users)
> > > drwxr-xr-x 6 root admin 192 Apr 6 2020 Users
> > >
> > > $ (cd /Users ; df .)
> > > Filesystem 512-blocks Used Available Capacity iused ifree %iused
> > > Mounted on
> > > /dev/disk1s1 976490576 608954344 338850488 65% 4181323 4878271557
> 0%
> > > /System/Volumes/Data
> > >
> > > we can see that /Users is on /System/Volumes/Data (and there is a
> > > /System/Volumes/Data/Users directory with the same metadata), but it
> > > is not a symlink.
> > >
> > >
> > > See [1] for more info.
> > >
> > > [1]
> > > http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink-
> > > magic.html <https://protect-
> > >
> >
> us.mimecast.com/s/QN4qCYEZB2h1p7jWcG1Cbw?domain=swiftforensics.co
> > > m>
> > >
> > >
> >
> > Hmm, I don't see anything that suggests a firmlink is involved but
> > this is new to me so maybe I just don't see it. There is an automount
> > in addition to the symlink though.
> >
> > So, dispensing with the abstract "/network" path, and getting to the
> > details of my environment:
> >
> > % mount | grep /mathworks
> > map auto.sfs.sol2.mathworks on /System/Volumes/Data/mathworks
> (autofs,
> > automounted, nobrowse) triggered map auto.sfs.sol2.devel on
> > /System/Volumes/Data/mathworks/devel (autofs, automounted,
> > nobrowse) triggered map sfs.worldwide.US-Natick.sol2.devel.sbs on
> > /System/Volumes/Data/mathworks/devel/sbs (autofs, automounted,
> > nobrowse) triggered map sfs.worldwide.US-Natick.devel.sbs.29 on
> > /System/Volumes/Data/mathworks/devel/sbs/29 (autofs, automounted,
> > nobrowse)
> > batfs-sb29-nfs:/vmgr/sbs29/edecosta.Bbashprod1.j1928560.2 on
> >
> /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19
> > 28560.2 (nfs, nodev, automounted, noatime, nobrowse) ...
> >
> > % ls -l / | grep mathworks
> > lrwxr-xr-x   1 root  wheel    35 Aug 29 10:25 home@ ->
> > /System/Volumes/Data/mathworks/home
> > lrwxr-xr-x   1 root  wheel    30 Aug 29 10:25 mathworks@ ->
> > /System/Volumes/Data/mathworks
> >
> > My worktree is:
> >
> /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19
> > 28560.2
> > or, equivalently via the symlink just:
> > /mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2
> >
> > I have sudo'ers access now; so I can try mounting things manually,
> > mess about with symlinks and see what I get.
> >
> > > [...]
> > > >>> @@ -209,7 +209,12 @@ static void
> > > >> fsevent_callback(ConstFSEventStreamRef streamRef,
> > > >>> /*
> > > >>> * On Mac, we receive an array of absolute paths.
> > > >>> */
> > > >>> - path_k = paths[k];
> > > >>> + if (fsm_settings__get_allow_remote(the_repository) > 0) {
> > > >>> + strbuf_reset(&tmp); strbuf_realpath_forgiving(&tmp, paths[k],
> > > >>> + 0); path_k = tmp.buf; } else path_k = paths[k];
> > > >>
> > > >> This feels wrong.
> > > >>
> > > >> It is not that fsmonitor.allowRemote is true, but whether or not
> > > >> this particular file system *IS* remote, right?
> > > >
> > > > True. I suppose each path could be checked, or some bit could be
> > > > set if the working directory is remote, perhaps along the lines of
> > > > fsmonitor_ipc__get_path. Then the determination of the IPC path
> > > > and whether it is remote would be in one place. fsm-settings-*
> > > > would then just get that bit and check it against allowRemote.
> > > >
> > > > Thoughts?
> > >
> > > I'll do some digging here. There ought to be a way to tell if a
> > > component directory in a pathname has a firmlink peer. (But I'm
> > > traveling for GitMerge, so I might not have time to look at this
> > > until
> > > afterwards.)
> > >
> > > This would indicate that a "bi-directional wormhole" (their words)
> > > alias is available (and hopefully, a way to computer the other
> > > peer....)
> > >
> > > I'm thinking that the "/network/..." path in the FSEvents is because
> > > FSEventD is using a particular peer spelling (that we weren't
> > > expecting).
> > >
> > > If we can compute the spellings of the peers once when the daemon
> > > starts up (and maybe make a little dictionary), then we can do
> > > prefix tricks on the absolute path after the path_k = paths[k]; step
> > > in
> > > fsevent_callback() to extract a relative path rather than an
> > > absolute path.
> > >
> > > Then call fsmonitor_classify_path_relative() instead of _absolute().
> > >
> > > Watch out though, there are several places that do rel = path+k +
> > > ...state->path_worktree_watch.len...
> > > that would need to be adjusted.
> > >
> > > Hope this helps,
> > > Jeff
> > >
> > >
> > Thanks for the insights, I'll dig around and test things out some more.
> >
> > -Eric
> 
> This is informative:
> 
> https://developer.apple.com/forums/thread/120665
> 
> -Eric
> 
Conversation with one of our macOS admins:

Q: Is /mathworks a symbolic link or some sort of firmlink or synthetic firmlink?

A: It’s a “synthetic” link, per apple. Config is managed in /etc/synthetic.conf

The auto mounter will also automatically create it based on the contents of /etc/auto_master even without the synthetic.conf file. We went back and forth with apple for a while on the “right” way to do it and now I’m not sure what we settled on, off the top of my head

I don’t know if they’re the same under the hood, but it seemed to be able to create root level links even though you can’t with ln anymore

I suspect apple wants users to use the synthetic.conf file in all cases...
----

It appears that the automounter is creating "synthetic firmlinks", a la https://derflounder.wordpress.com/2020/01/18/creating-root-level-directories-and-symbolic-links-on-macos-catalina/

When I look in /etc/auto_master I see several entries that appear as symlinks in the output of "ls -l /" but in reality they are synthetic firmlinks.

-Eric