Tracking file metadata in git -- fix metastore or enhance git?

Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

Hi all,


as you know, it's not possible to track metadata like mtime etc via
git natively. metastore exists, but its binary storage format is
merge-unfriendly. The author knows and acknowledges this but lacks the
time to fix it.

We might try and fix up metastore to use ASCII with BASE64 for xattr
etc instead of the current binary format. metastore supports versioned
stores so it will handle this gracefully. Before going down this path,
I wanted to ask if anyone would be interested to implement this in git
directly. Obviously, an optional and native way is vastly preferable
to a third-party tool. Both for reasons of potential bit-rot and more
users. Yet, we lack the confidence to do this in git directly
ourselves.

If this is implemented in git, the type(s) of metadata to track should
be fine-grained and fully optional, of course.


Any and all feedback appreciated,
Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Thorsten Glaser]

Richard Hartmann dixit:

>git natively. metastore exists, but its binary storage format is

According to the documentation, metastore supports saving/restoring
POSIX Extended Attributes (however I got EOPNOTSUPP trying to setfacl
and setxattr, so I couldn’t verify the claim). It does *not* however
track the stuff visible using lsattr(1) on ext2fs.

Personally, I think this should definitively be tracked in git, not
in extra files (like “ EA DATA. SF” in OS/2) managed with external
tools (reminds me too much of things like umsdos and Rock Ridge as
well). Support needs not be part of core git, especially restoring
can be dangerous (consider SELinux), plus it’s very platform and
filesystem dependent, but seeing that people seem to want to use
git for keeping their /etc (I use RCS, myself, but volunteered my
C skill to change metastore if it needs be) there is definitively
a market for having it there, in a standardised manner.

bye,
//mirabilos
-- 
FWIW, I'm quite impressed with mksh interactively. I thought it was much
*much* more bare bones. But it turns out it beats the living hell out of
ksh93 in that respect. I'd even consider it for my daily use if I hadn't
wasted half my life on my zsh setup. :-) -- Frank Terbeck in #!/bin/mksh

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

On Thu, Apr 7, 2011 at 21:27, Thorsten Glaser <tg@mirbsd.de> wrote:

> but seeing that people seem to want to use
> git for keeping their /etc (I use RCS, myself, but volunteered my
> C skill to change metastore if it needs be) there is definitively
> a market for having it there, in a standardised manner.

etckeeper does exist and it's very useful.


Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Michael J Gruber]

Richard Hartmann venit, vidit, dixit 08.04.2011 02:29:
> On Thu, Apr 7, 2011 at 21:27, Thorsten Glaser <tg@mirbsd.de> wrote:
> 
>> but seeing that people seem to want to use
>> git for keeping their /etc (I use RCS, myself, but volunteered my
>> C skill to change metastore if it needs be) there is definitively
>> a market for having it there, in a standardised manner.
> 
> etckeeper does exist and it's very useful.

While I think there is some consensus not to have attribs (besides x) in
core git, contrib/ may be a good place.

Note that etckeeper and metastore predate our notes feature. By now, a
notes tree is a perfect place to store meta information. You can attach
notes to blobs perfectly (see textconv-cache). That may be an option for
a reimplementation, depending on how you want the versioning of the
files to be related with the versioning of the meta data.

Michael

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Jonathan Nieder]

Hi,

Michael J Gruber wrote:

> Note that etckeeper and metastore predate our notes feature. By now, a
> notes tree is a perfect place to store meta information. You can attach
> notes to blobs perfectly (see textconv-cache). That may be an option for
> a reimplementation, depending on how you want the versioning of the
> files to be related with the versioning of the meta data.

Wouldn't that mean the same content couldn't have one owner in one
commit and a different owner in another commit?

I think the most native-looking way to store metadata associated to
paths is .gitattributes.  It also has the nice feature of allowing a
single attribute to apply to multiple files.  But the tools for
looking them up (e.g., git check-attr) do not get much use yet so they
probably could use a lot of improvement.

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Thorsten Glaser]

Jonathan Nieder dixit:

>I think the most native-looking way to store metadata associated to
>paths is .gitattributes.  It also has the nice feature of allowing a
>single attribute to apply to multiple files.

Eh, no. Think of extended attributes like, say, NTFS Resource Forks.
They’re just different “lines” into the “plane” a file can be, if
you excuse the metapher. (All parallel, of course.) They are just
another facet of each file.

bye,
//mirabilos
-- 
22:20⎜<asarch> The crazy that persists in his craziness becomes a master
22:21⎜<asarch> And the distance between the craziness and geniality is
only measured by the success 18:35⎜<asarch> "Psychotics are consistently
inconsistent. The essence of sanity is to be inconsistently inconsistent

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Jonathan Nieder]

Thorsten Glaser wrote:
> Jonathan Nieder dixit:

>> I think the most native-looking way to store metadata associated to
>> paths is .gitattributes.  It also has the nice feature of allowing a
>> single attribute to apply to multiple files.
>
> Eh, no. Think of extended attributes like, say, NTFS Resource Forks.
> They’re just different “lines” into the “plane” a file can be, if
> you excuse the metapher. (All parallel, of course.)

Do you mean no, it doesn't have that feature? ;-)

Each git commit (try it with "git cat-file commit HEAD) looks like so:

	tree <tree name>
	parent <commit name for first parent>
	parent <commit name for second parent>
	...
	author <author identity and time of authorship>
	committer <committer identity and time committed>
	encoding <encoding of log message (optional)>

	<free-form change description>

Where could one sneak in some per-path metadata?

 - as new header fields after "encoder" (teaching git fsck, git commit
   --amend, and so on about it)?  That can work but it would slow down
   operations not interested in this metadata.  It is best not to have
   O(number of paths) header fields.

 - in the change description?  Yes, that can work, too, and it doesn't
   even require changing the commit format.

 - a new header field pointing to another object?  That is possible as
   a last resort.

Anyway, filenames and associated content are not what commits are
about; commits are just nodes in a revision graph, with trees representing
the tracked trees.

Okay, so what about the trees?

	<mode> SP <filename> NUL <object name>
	...

Where can we sneak something in?

 - use a currently invalid <mode>?  No, tracking metadata is probably
   not worth breaking old git clients.
 - use an invalid object name?  No (for the same reason).
 - use a special filename?  Then old git clients will treat the file
   as a regular file, so they still get access to the data.

So you see, using ordinary files (whether called .gitattributes or
foo.c.ntfs-resource-fork) to track this extra data makes a lot of
sense.

Now Michael mentioned an alternative, which is to store this
information in separate objects.  That way, you could push your
history without the extra metadata, you could edit the metadata
without changing the commit names of the history, separately
garbage-collect metadata you're not interested in, etc.  If that is
your goal, then "git notes" is exactly the right solution.

> They are just
> another facet of each file.

Sure, like the atime, the inode number, the uid of the user who wrote
them, and the model number of the disk used to store it.

Oh, you mean they're _relevant_ facets?  Yes, that's believable,
though I suspect that's only going to sometimes be the case.  So the
operator should say "yes, I'm interested in tracking this extra
information".  To summarize the above, some ways this could work
behind the scenes:

 * dotfiles with metadata;

 * a Makefile to install files with metadata (i.e., the "source"
   consists of plain files, while the "build product" has the
   specified metadata);

 * something else.  Hopefully the above explains the relevant
   constraints so you can surprise us.

Hope that helps.
Jonathan

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Thorsten Glaser]

Jonathan Nieder dixit:

>Each git commit (try it with "git cat-file commit HEAD) looks like so:
[…]
>So you see, using ordinary files […] makes a lot of
>sense.

True. I’d forgotten for a second that git is a stupid content
tracker and confused it with a VCS. My bad. Your reasoning is
of course sound. (I must admit I personally don’t have much
experience with git, other than losing a lot of data once.)

>Now Michael mentioned an alternative, which is to store this
>information in separate objects.  That way, you could push your

This might be an alternative. RichiH, what are the requirements?
I fear that the dotfile way will meet them better, considering
that they are versioned together, not separately, and what you
told me quickly about the idea.

bye,
//mirabilos
-- 
13:47⎜<tobiasu> if i were omnipotent, i would divide by zero
                all day long ;)
(thinking about http://lobacevski.tumblr.com/post/3260866481 by waga)

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

Ignoring any potentially flame-ish comments in the shared interest of
factual discussion and eventual consensus followed by progress; please
do the same :)


On Fri, Apr 8, 2011 at 21:58, Thorsten Glaser <tg@mirbsd.de> wrote:

> This might be an alternative. RichiH, what are the requirements?

The main use case from my/our POV is to track mtime etc to maintain
relative and absolute differences which can be extremely useful for
data even if it's not needed for code files.


> I fear that the dotfile way will meet them better, considering
> that they are versioned together, not separately, and what you
> told me quickly about the idea.

Points to consider:

* In most cases I can think of, it would be preferable to track
changes in metadata along with the actual files.

* It should be possible for other repositories to ignore this metadata.

* I am not sure if notes are suitable for this. Using notes.displayRef
to hide those notes is an option, but it would force everyone to set
this up locally. Else, it would be very spammy.

* .gitattributes is too important to fill with potentially thousands
of lines, imo. Using it to define what metadata should be stored would
make sense, though.

* would a .gitmetadata make sense for storage? Alternatively, a
.git/objects/??/*.metadata per object could make sense.


Thoughts?
Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Chris Webb]

Richard Hartmann <richih.mailinglist@gmail.com> writes:

> * In most cases I can think of, it would be preferable to track
> changes in metadata along with the actual files.
> 
> * It should be possible for other repositories to ignore this metadata.
> 
> * I am not sure if notes are suitable for this. Using notes.displayRef
> to hide those notes is an option, but it would force everyone to set
> this up locally. Else, it would be very spammy.
> 
> * .gitattributes is too important to fill with potentially thousands
> of lines, imo. Using it to define what metadata should be stored would
> make sense, though.
> 
> * would a .gitmetadata make sense for storage? Alternatively, a
> .git/objects/??/*.metadata per object could make sense.

One other possibility which occurs: would it be possible to store
permissions, attributes and so on in a header line (or header lines) in the
blob itself, which are stripped from the file and applied to the filesystem
by a smudge filter, and added back by a clean filter?

If it works sensibly, this is roughly equivalent to adding a .gitmetadata
file as you suggest above, except that the permissions and attribute data
are stored with each file, so the resulting diffs will be nicer and merges
should always be trivial without any special handling.

(Disclaimer: I've never tried using smudge filters; maybe they can't be used
in the way I describe!)

Cheers,

Chris.

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

On Sat, Apr 9, 2011 at 10:11, Chris Webb <chris@arachsys.com> wrote:

> (Disclaimer: I've never tried using smudge filters; maybe they can't be used
> in the way I describe!)

Smudge filters are on my half-read list of New Cool Stuff so I can't
really comment on this.

Another idea would be to simply tack all metadata into the same place
as the existing chmod stuff.

While one large text file would be nice in case one wanted to make
manual changes, attaching versioned, binary information to every
object directly is a lot more efficient. I think I am convinced the
latter would be the cleaner approach, now. With a clean layout, this
would not need to be a huge code drop supporting every metadata type
in existence; it could be extended as needed.

As noted above, .gitattributes could be used to manage what kind of
metadata should be stored.


One large question in my mind is if anyone who's familiar enough with
the codebase and has the time would be interested in actually
implementing this.


Thanks,
Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Jonathan Nieder]

Hi again.

Chris Webb wrote:

> (Disclaimer: I've never tried using smudge filters; maybe they can't be used
> in the way I describe!)

In the current form, they couldn't.  smudge/clean filters take their
input through stdin and write their output to stdout --- there is not
a chance to look at the on-disk directory entry to get metadata.

It might be possible to introduce a new %p substitution pointing to
the actual file the filter is working on, since with a few exceptions,
the file passed to a clean filter is already on-disk.  Exceptions:

 * "git blame --contents=- -- <path>" reads from stdin instead of <path>.

 * the low-level --renormalize facility (see git-merge(1)) and its
   callers feed a clean filter with output from a smudge filter before
   content hits the disk.

 * "git hash-object --path=<path> --stdin" reads from stdin instead of
   <path>.

I'd be worried about using a clean filter to store timestamps.
Treating a file as changed whenever mtime changes could be confusing.
Treating atime changes as content changes would be even stranger.

Richard Hartmann wrote:

> One large question in my mind is if anyone who's familiar enough with
> the codebase and has the time would be interested in actually
> implementing this.

I don't think this has to touch git core, except perhaps as mentioned
above.  Please feel free to cc me if working on hooks (pre-commit hook
or clean filter) to automatically track metadata and some mechanism to
restore it.  I'd be glad to give feedback and help in any other way I
can.

To be clear, I will not be driving this forward --- it's just not
something I've ever needed, so I'd trust others to do a better job of
taking good care of the actual use cases.

Ciao,
Jonathan

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Junio C Hamano]

Jonathan Nieder <jrnieder@gmail.com> writes:

> I'd be worried about using a clean filter to store timestamps.
> Treating a file as changed whenever mtime changes could be confusing.
> Treating atime changes as content changes would be even stranger.

That is exactly why git does not store metainfo.  Storing auxiliary
information on commit objects or tree objects to help build or deploy
procedures is fine, but that kind of information should be stored
somewhere that the normal git operations would not care about.

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

On Sun, Apr 10, 2011 at 03:03, Junio C Hamano <gitster@pobox.com> wrote:

> That is exactly why git does not store metainfo.  Storing auxiliary
> information on commit objects or tree objects to help build or deploy
> procedures is fine, but that kind of information should be stored
> somewhere that the normal git operations would not care about.

Keeping this information out of git's way during operations in which
it's not needed is obviously A Good Thing. Yet, this would only happen
when this information has been added to the repository on purpose.
Thus, I don't see this as a deal-breaker.


Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

FYI, Harley J Pig <harleypig@gmail.com> had this to say on the
vcs-home mailing list

> I've written a metastore clone for a project where we need to store a
> linux distribution in version control (legacy code).  I'm also using
> it for my personal vcs-home stuff.  It is a naive and bluntly
> straightforward way to do this, but it seems to be working.  You can
> find it at https://github.com/harleypig/gitperms
>
> I use git hooks and a central file to (re)store the metadata.  Maybe
> it can be of some use to someone else.


Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

Hi all,

as I don't want to let this go stale, here are the three options:

1) fix metastore
2) default to gitperms
3) extend git


I still think 3) would be best, but someone would need to step up to
do this. Is anyone up for this task? If not, we will have to resort to
1) or 2)


Thanks again,
Richard

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Jonathan Nieder]

Hi again,

Not sure if my thoughts will be useful here since you dropped me from
the cc list.  But anyway:

Richard Hartmann wrote:

> here are the three options:
>
> 1) fix metastore
> 2) default to gitperms
> 3) extend git
>
> I still think 3) would be best, but someone would need to step up to
> do this. Is anyone up for this task? If not, we will have to resort to
> 1) or 2)

The usual practice in git development is

 (1) people make scripts wrapping plumbing commands (see git(1)) that
     work well for themselves
 (2) they tell the git list about it and publish it
 (3) an idea emerges that this is suitable for inclusion, and it
     gets included

In particular, git's design is not so monolithic --- "extend git" can
mean "add a script" or "add a builtin" so it is not so involved as you
seem to think.  See also contrib/README for a place to stop on the
way.

Anyway, if you want something the just works, my suggestion is (4) use
the hook scripts from etckeeper.  Last time I looked into this they
worked better than metastore.

Hope that helps.
Jonathan

Re: Tracking file metadata in git -- fix metastore or enhance git?

[johnnyutahh]

Following up on 
http://git.661346.n2.nabble.com/Tracking-file-metadata-in-git-fix-metastore-or-enhance-git-td6251248.html
this discussion re: git file-metadata-management , posted this:

http://superuser.com/questions/367729/how-to-reuse-extend-etckeepers-metadata-engine-for-git-control-of-non-etc-file

Any further movement on this topic?



Jonathan Nieder-2 wrote
> 
> Hi again,
> 
> Not sure if my thoughts will be useful here since you dropped me from
> the cc list.  But anyway:
> 
> Richard Hartmann wrote:
> 
>> here are the three options:
>>
>> 1) fix metastore
>> 2) default to gitperms
>> 3) extend git
>>
>> I still think 3) would be best, but someone would need to step up to
>> do this. Is anyone up for this task? If not, we will have to resort to
>> 1) or 2)
> 
> The usual practice in git development is
> 
>  (1) people make scripts wrapping plumbing commands (see git(1)) that
>      work well for themselves
>  (2) they tell the git list about it and publish it
>  (3) an idea emerges that this is suitable for inclusion, and it
>      gets included
> 
> In particular, git's design is not so monolithic --- "extend git" can
> mean "add a script" or "add a builtin" so it is not so involved as you
> seem to think.  See also contrib/README for a place to stop on the
> way.
> 
> Anyway, if you want something the just works, my suggestion is (4) use
> the hook scripts from etckeeper.  Last time I looked into this they
> worked better than metastore.
> 
> Hope that helps.
> Jonathan
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@.kernel
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


--
View this message in context: http://git.661346.n2.nabble.com/Tracking-file-metadata-in-git-fix-metastore-or-enhance-git-tp6251248p7092383.html
Sent from the git mailing list archive at Nabble.com.

Re: Tracking file metadata in git -- fix metastore or enhance git?

[Richard Hartmann]

On Wed, Dec 14, 2011 at 05:54, johnnyutahh
<git.nabble.com@johnnyutahh.com> wrote:

> Any further movement on this topic?

Not really. ATM, our best bet is to agree on a language and just implement it.


Richard