Requirements for integrating a new git subcommand

Requirements for integrating a new git subcommand

[Eric S. Raymond]

I have completed work on git-weave (the tool I had called 'gitpacker' in some
previous postings).  I want to submit a patch that integrates it into git;
in hopes of smoothing the process I have some technical and procedural 
questions.  

First, however, let me present the git-weave documentation for design
review:

----------------------------------------------------------------------
git-weave(1)
============

NAME
----
git-weave - Weave a sequence of trees and log data into a repository

SYNOPSIS
--------
[verse]
'git-weave' [-v] [-m limit] [-q] indir outdir

DESCRIPTION
-----------
git-weave accepts an input directory containing a sequence of
subdirectories and a metadata file, and composes them into a 
git repository created under the specified output directory
(which must not exist).

If the input directory is identifiably a git repository, the weave
operation is reversed; tree states from each commit are unraveled into
the output directory with a log holding commit metadata
(committer/author/comment information and parent headers representing
links of the repository DAG) and tags.

This tool is primarily intended for importing and working with project
histories that have been preserved only as linear sequences of release
snapshots.  It may also be useful for surgery on linear repositories

While the weave operation can build a commit graph with any structure
desired, an important restriction of the inverse (unraveling)
operation is that it operates on *master branches only*. The unravel
operation discards non-master-branch content, emitting a warning 
to standard error when it has to do so.

Commits from the repository's master branch are unraveled into
directories named for integers from 1 increasing, but their order of
composition when re-woven is actually set by the sequence of entries
in the metadata file.  File trees may be inserted or removed without
hindering re-weaving provided the pointers in the log's parent fields
are fixed up properly.

METADATA FILE FORMAT
--------------------
The metadata file format will contain three kinds of stanzas: entries
for commits, entries for lightweight tags, and entries for annotated
tags.

A commit stanza has headers similar to those in a commit raw log:
commit, committer, author, and optionally parent headers.  The header
contents are not hash IDs, but arbitrary text cookies either declared
by a previous commit stanza or referencing one.  The following example
declares "8" to be a commit ID, and references a previous commit
identified as '7'.  Note that commit IDs are not required to be
numeric strings, though the unravel operation generates them that way.

------------
commit 8
parent 7
author Eric S. Raymond <esr@thyrsus.com> 1325026869 +0000
committer Eric S. Raymond <esr@thyrsus.com> 1325026869 +0000

Initial revision
.
------------

The text body of a commit comment or tag comment entry is delimited
from the headers by an empty line; the text body must always end with
"." on a line by itself; and text lines beginning with "."  will have
an additional "." prepended to them.

A commit stanza may also have a "directory" header.  If present, this 
sets the name of the subdirectory in which git-weave expects to
find the content tree for this commit.  For example

------------
commit 24
directory intercal-0.17
parent 23
author Eric S. Raymond <esr@thyrsus.com> 1325026489 +0000
committer Eric S. Raymond <esr@thyrsus.com> 1325026489 +0000

The state of the INTERCAL project at release 0.17.
.
------------

A label stanza declares a lightweight tag.  This example declares a
tag 'sample' pointing at the commit identified as 102.

------------
label sample
refers-to 102
------------

A tag stanza declares an annotated tag.  This one declares a tag named
'annotated1' pointing at the commit declared as 99.

------------
tag annotated1
refers-to 99
tagger Eric S. Raymond <esr@thyrsus.com> Sat Nov 17 03:16:26 2012 -0500

This is an example annotated tag.
.
------------

When you are composing commit and tag stanzas by hand, you can count
on any of the date formats normally acceptable to git to be
recognized.

If, when weaving, any committer or author or tagger line, the date is omitted,
git-weave will supply as a default the latest modification time of
any file in the corresponding tree.

If a committer or author or tagger line is omitted entirely, the
user's name and email address as retrieved by ''git-config'' will
be supplied as defaults, and the date will default as above.

Thus, the following variation on one of the previous examples 
is a valid stanza:

------------
commit 24
directory intercal-0.17
parent 23

The state of the INTERCAL project at release 0.17.
.
------------

OPTIONS
-------
-q::
	Be quiet.  Suppress the normal spinning-baton progress meter
	with timing information.

-m::
	Limit the number of commits or trees processed to a specified
	integer maximum. '0' means process all of them.

-v::
	Be verbose, enabling progress and command-execution messages
	This option will probably be of interest only to developers;
	consult the source code for details.

EXAMPLES
--------
* Weave a sequence of trees in the directory 'unraveled' 
into a git repository in the directory 'repo'. 
+
------------
$ rm -fr repo; git-weave unraveled repo
------------
+
The metadata is expected to be in 'unraveled/log'.  This mode of
operation is triggered when there is no file 'unraveled/.git', 

* Unravel a repository in the directory 'repo' into a sequence
of file trees and a metadata log in the directory 'unraveled'.
+
------------
$ rm -fr unraveled; git-weave repo unraveled
------------
+
This mode of operation is triggered when there is a 'repo/.git' file.

SEE ALSO
--------
linkgit:git-log[1]
linkgit:git-checkout[1]
linkgit:git-add[1]
linkgit:git-mktree[1]
linkgit:git-ls-tree[1]
linkgit:git-update-references[1]

GIT
---
Not yet part of the linkgit:git[1] suite
----------------------------------------------------------------------

Yes, there are scripts in contrib that do similar things.  git-weave
is an improvement is several ways:  (a) it is documented, (b) I am
shipping it with a functional test, (c) I am prepared to maintain it 
and am quite unlikely to drop out of sight :-), (d) it does both
the import operation *and its inverse*, and (e) it is rather more
powerful, including the ability to decorate the import with 
annotated tags.

Now *my* questions:

1. I have a round-trip test for the tool that I can very easily adapt
to speak TAP.  To function, the test will require a small linear
history to operate on in the form of an import-stream file (so the
result of round-tripping through a weave-unravel can be diffed against
the original).  Does the distribution include any test repos?  If
so, where can I find them?

2. I understand that a "git foo" command is typically implemented
as "git-foo" binary or script in /usr/lib/git-core.  What I don't
know is what the other interfacing requirements are.  Are they
documented anywhere?  In particular...

3. Is there any registration protocol other than simply installing 
the extension in the subcommand library?

4. How does "git help" work?  That is, how is a subcommand expected
to know when it is being called to export its help text?

5. I don't see any extensions written in Python.  Are there any special
requirements or exclusions for Python scripts?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Shawn Pearce]

On Wed, Nov 21, 2012 at 9:30 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> I have completed work on git-weave (the tool I had called 'gitpacker' in some
> previous postings).  I want to submit a patch that integrates it into git;
> in hopes of smoothing the process I have some technical and procedural
> questions.
...
> Now *my* questions:
>
> 1. I have a round-trip test for the tool that I can very easily adapt
> to speak TAP.  To function, the test will require a small linear
> history to operate on in the form of an import-stream file (so the
> result of round-tripping through a weave-unravel can be diffed against
> the original).  Does the distribution include any test repos?  If
> so, where can I find them?

No. We create the repositories from scratch using a series of
commands. If you look at the test library the environment is set in a
predictable way so the author, committer and timestamps are all set to
a single well known value, allowing Git to create a commit that is
reproducible on all platforms. A test_tick function is used in the
scripts to move the clock, allowing different times to be used. For an
example see t7502-commit.sh, or really any script in that directory.

> 2. I understand that a "git foo" command is typically implemented
> as "git-foo" binary or script in /usr/lib/git-core.  What I don't
> know is what the other interfacing requirements are.  Are they
> documented anywhere?  In particular...

Nope. "git foo" will invoke "git-foo" with GIT_DIR set in the
environment, so you know what repository to act against, and so does
any git command you recursively invoke. Other than that there really
aren't any interface requirements. Your program is executed with
whatever arguments the caller gave you. Its pretty simple UNIX stuff.
:-)

> 3. Is there any registration protocol other than simply installing
> the extension in the subcommand library?

Nope. Running "git whatever-this-is-i-have-no-idea" will try to
execute "git-whatever-this-is-i-have-no-idea" via $PATH, after adding
/usr/lib/git-core to the front of $PATH. Its pretty simple actually.
If your standard C library can find the program in $PATH its run, if
it can't find it, it dies.

> 4. How does "git help" work?  That is, how is a subcommand expected
> to know when it is being called to export its help text?

IIRC "git help foo" runs "man git-foo".

> 5. I don't see any extensions written in Python.  Are there any special
> requirements or exclusions for Python scripts?

Nope, it just has to be executable. We don't have any current Python
code. IIRC the last Python code was the implementation of
git-merge-recursive, which was ported to C many years ago. We avoid
Python because it is not on every platform where Git is installed. Yes
Python is very portable and can be installed in many places, but we
prefer not to make it a requirement.

Re: Requirements for integrating a new git subcommand

[Eric S. Raymond]

Shawn Pearce <spearce@spearce.org>:
> [Lots of helpful stuff ended by]
> > 4. How does "git help" work?  That is, how is a subcommand expected
> > to know when it is being called to export its help text?
> 
> IIRC "git help foo" runs "man git-foo".

OK, that makes sense.

> > 5. I don't see any extensions written in Python.  Are there any special
> > requirements or exclusions for Python scripts?
> 
> Nope, it just has to be executable. We don't have any current Python
> code. IIRC the last Python code was the implementation of
> git-merge-recursive, which was ported to C many years ago. We avoid
> Python because it is not on every platform where Git is installed. Yes
> Python is very portable and can be installed in many places, but we
> prefer not to make it a requirement.

I find that odd.  You avoid Python but use shellscripts?  *blink*

One would think shellscripts were a much more serious portability problem.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Michael J Gruber]

Eric S. Raymond venit, vidit, dixit 22.11.2012 23:11:
> Shawn Pearce <spearce@spearce.org>:
>> [Lots of helpful stuff ended by]
>>> 4. How does "git help" work?  That is, how is a subcommand expected
>>> to know when it is being called to export its help text?
>>
>> IIRC "git help foo" runs "man git-foo".
> 
> OK, that makes sense.
> 
>>> 5. I don't see any extensions written in Python.  Are there any special
>>> requirements or exclusions for Python scripts?
>>
>> Nope, it just has to be executable. We don't have any current Python
>> code. IIRC the last Python code was the implementation of
>> git-merge-recursive, which was ported to C many years ago. We avoid
>> Python because it is not on every platform where Git is installed. Yes
>> Python is very portable and can be installed in many places, but we
>> prefer not to make it a requirement.
> 
> I find that odd.  You avoid Python but use shellscripts?  *blink*
> 
> One would think shellscripts were a much more serious portability problem.

Different versions of python can be a mess to deal with, also, at least
with respect to standard modules being "standard" or not for a specific
version.

In any case, the point is that we try to avoid *additional*
dependencies. Shell and perl are given with the status quo.

That being said, we also have remote helpers in python. The testsuite
can run tests depending on the availability of python.

Regarding git-weave, I'm wondering (without having looked at the code)
how this relates to git-archiv and git-fast-import/export, i.e. how much
this leverages existing infrastructure rather than reinventing the
wheel. Do your "trees" correspond to a "git tree"?

Again, without having looked at the code, it seems to me that exploding
blob and tree objects might give you a structure not much unlike
weave's, and your instruction sheet resembles that of fast-import quite
a bit (plus date fill-in etc.).

One could even dream about implementing this as a remote helper instead...

Michael

Re: Requirements for integrating a new git subcommand

[Peter Krefting]

Eric S. Raymond:

> git-weave(1)

> Yes, there are scripts in contrib that do similar things.

I was just about to say that the import direction of this seems to 
fill the same need as contrib/fast-import/import-directories.perl that 
I submitted a few years back.

Your version seems only to be able to import a linear history, 
however, my tool does support creating merge commits (basically, the 
history I had to import was very messy and contained a lot of snapshot 
directories having been worked on in parallel).

> (b) I am shipping it with a functional test,

Hmm, indeed. I have been thinking of trying to wrap up the test suite 
I have locally into something that could work within the Git testing 
framework, but haven't found the time to or energy for so far.


Anyway, my sentiment is that if you can add support for merges in you 
weave tool, then I am very much for removing my old script from the 
repository.

-- 
\\// Peter - http://www.softwolves.pp.se/

Re: Requirements for integrating a new git subcommand

[Eric S. Raymond]

Peter Krefting <peter@softwolves.pp.se>:
> I was just about to say that the import direction of this seems to
> fill the same need as contrib/fast-import/import-directories.perl
> that I submitted a few years back.

Yours was the closest in functionality, yes.

> Your version seems only to be able to import a linear history,
> however, my tool does support creating merge commits (basically, the
> history I had to import was very messy and contained a lot of
> snapshot directories having been worked on in parallel).

git-weave can *weave* (import) a sequence with merge commits, as your
tool can.  What it can't do is unravel a nonlinear repo into a tree
sequence that will round-trip through the weave operation.  (Though I
thought of a way I might be able to fix that last night.)
 
> Anyway, my sentiment is that if you can add support for merges in
> you weave tool, then I am very much for removing my old script from
> the repository.

The support is there.  I will take this as direction to (a) add a 
test load demonstrating this, and (b) include the removal of
import-directories.perl in my integration patch.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Eric S. Raymond]

Michael J Gruber <git@drmicha.warpmail.net>:
> Regarding git-weave, I'm wondering (without having looked at the code)
> how this relates to git-archiv and git-fast-import/export, i.e. how much
> this leverages existing infrastructure rather than reinventing the
> wheel. Do your "trees" correspond to a "git tree"?

The unravel operation of git-weave is something like running
git-archive on every revision and saving the results in
sequentially-named directories, except that it also produces a
metadata file that allows the operation to be inverted.
So it is strictly more powerful.

The weave operation could be implemented using git fast-import, which
I am quite intimately familiar with from having written reposurgeon.
Functionally, the difference is that it would be a PITA to patch a
fast-import stream to insert or modify or remove revisions in the
middle, because the content of any given revision is in blobs that can
stretch arbitrarily far back from its commit and are shared with
other revisions.  With git-weave tree-sequences these operations
are easy and safe.

> Again, without having looked at the code, it seems to me that exploding
> blob and tree objects might give you a structure not much unlike
> weave's, and your instruction sheet resembles that of fast-import quite
> a bit (plus date fill-in etc.).

The weave log resembles an import stream, yes - that's because they
have to capture the same data ontology.  One major difference is that weave
logs are designed to be generated and edited by humans.
 
> One could even dream about implementing this as a remote helper instead...

What is a "remote helper" in this context?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Eric S. Raymond]

Shawn Pearce <spearce@spearce.org>:
> Nope, it just has to be executable. We don't have any current Python
> code. IIRC the last Python code was the implementation of
> git-merge-recursive, which was ported to C many years ago.

This turns out not to be quite true.  The tree currently includes 
two Python scripts, a Perforce importer and a test helper.

I'm in he process of writing up a document on command integration
based on your answers. I will submit it for incusion in the tree
shortly.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Christian Couder]

On Thu, Nov 22, 2012 at 11:11 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Shawn Pearce <spearce@spearce.org>:
>> [Lots of helpful stuff ended by]
>> > 4. How does "git help" work?  That is, how is a subcommand expected
>> > to know when it is being called to export its help text?
>>
>> IIRC "git help foo" runs "man git-foo".
>
> OK, that makes sense.

"git help" can also launch a browser to display the HTML version of
the man page.
It is looking for man pages and HTML docs in the paths given by "git
--man-path" and "git --html-path".

Cheers,
Christian.

Re: Requirements for integrating a new git subcommand

[Junio C Hamano]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> While the weave operation can build a commit graph with any structure
> desired, an important restriction of the inverse (unraveling)
> operation is that it operates on *master branches only*. The unravel
> operation discards non-master-branch content, emitting a warning 
> to standard error when it has to do so.

Imagine that I have a simple four-commit diamond:

    I---A
     \   \
      B---M

where Amy and Bob forked the initial commit made by Ian and created
one commit each, and their branches were merged into one 'master'
branch by a merge commit made by Mac.  How many state snapshots will
I see when I ask you to unravel this?  Three, or four?

If you are going to give me all four states, then I do not
understand why this needs to be limited to the master branch only.
Even if you start from a single commit at the tip of 'master', once
you hit a merge, you will need to follow all of two (or more) paths
to dig down to the root(s), so supporting to start digging from more
than one commit is not all that different.

If you are going to give me only three states, following the first
parent ancestry chain, then the description needs to state it more
clearly.  I am not saying first-parent-only history is useless, but
the user needs to know that merges are flattened in the unraveled
result and the resulting history becomes linear, following the first
parent ancestry chain of the original history (if that is what the
tool does) before deciding if this tool matches what she needs.

As to the procedural stuff, I think others have sufficiently
answered already.  If I may add something, a new stuff typically
start its life in contrib/ before it proves to be useful.

Re: Requirements for integrating a new git subcommand

[Eric S. Raymond]

Junio C Hamano <gitster@pobox.com>:
> "Eric S. Raymond" <esr@thyrsus.com> writes:
> 
> > While the weave operation can build a commit graph with any structure
> > desired, an important restriction of the inverse (unraveling)
> > operation is that it operates on *master branches only*. The unravel
> > operation discards non-master-branch content, emitting a warning 
> > to standard error when it has to do so.
> 
> Imagine that I have a simple four-commit diamond:
> 
>     I---A
>      \   \
>       B---M
> 
> where Amy and Bob forked the initial commit made by Ian and created
> one commit each, and their branches were merged into one 'master'
> branch by a merge commit made by Mac.  How many state snapshots will
> I see when I ask you to unravel this?  Three, or four?

You will see four tree states.  I have managed to remove the
master-branch-only restriction.

> As to the procedural stuff, I think others have sufficiently
> answered already.  If I may add something, a new stuff typically
> start its life in contrib/ before it proves to be useful.

Thank you, I have submitted a documentation patch which folds
in the on-list discussion.

As a separate point...are you requesting that I submit my integration
patch to drop git-weave in contrib?  If so, I will of course comply.

But I will point out that git-weave is not a half-thought out
experiment; it is fully documented and has a functional test.  The
case for its usefulness is bolstered by one previous contrib script,
which the author has agreed to retire in favor of git-weave.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Requirements for integrating a new git subcommand

[Peter Krefting]

Eric S. Raymond:

> and (b) include the removal of import-directories.perl in my 
> integration patch.

Yes, please.

-- 
\\// Peter - http://www.softwolves.pp.se/