[PATCH 5/5] archimport - better handling of temp dirs

[PATCH 5/5] archimport - better handling of temp dirs

[martin]

Switched from backwards hard-coded tmp directory creation to using
File::Temp::tempdir() to create the directory inside $TMP_PATH or
what the user has provided via the -t parameter.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>


---

 git-archimport.perl |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)

67d3cf04c6a2353835d52a4d1ae85c7fb91bad36
diff --git a/git-archimport.perl b/git-archimport.perl
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -23,7 +23,6 @@ See man (1) git-archimport for more deta
 
  - create tag objects instead of ref tags
  - audit shell-escaping of filenames
- - better handling of temp directories
  - hide our private tags somewhere smarter
  - find a way to make "cat *patches | patch" safe even when patchfiles are missing newlines  
 
@@ -37,7 +36,7 @@ use strict;
 use warnings;
 use Getopt::Std;
 use File::Spec;
-use File::Temp qw(tempfile);
+use File::Temp qw(tempfile tempdir);
 use File::Path qw(mkpath);
 use File::Basename qw(basename dirname);
 use String::ShellQuote;
@@ -72,9 +71,10 @@ usage if $opt_h;
 @ARGV >= 1 or usage();
 my @arch_roots = @ARGV;
 
-my $tmp = $opt_t;
-$tmp ||= '/tmp';
-$tmp .= '/git-archimport/';
+my ($tmpdir, $tmpdirname) = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
+my $tmp = $opt_t || 1;
+$tmp = tempdir('git-archimport-XXXXXX', TMPDIR => 1, CLEANUP => 1);
+$opt_v && print "+ Using $tmp as temporary directory\n";
 
 my @psets  = ();                # the collection
 my %psets  = ();                # the collection, by name

archimport documentation updates and tidyups

[martin]

This patch series covers the Documentation for the archimport 
script, plus the remaining TODOs that I consider important.

With these patches I consider archimport done, modulo bugs and 
interesting user requests.

 [PATCH 1/5] archimport documentation update
 [PATCH 2/5] archimport documentation tidyup
 [PATCH 3/5] archimport - update in-script doco, options tidyup
 [PATCH 4/5] archimport - use GIT_DIR instead of hardcoded ".git"
 [PATCH 5/5] archimport - better handling of temp dirs

[PATCH 3/5] archimport - update in-script doco, options tidyup

[martin]

Updated the usage/help message to match asciidoc documentation. The perldoc
documentation now includes the first paragraph from the asciidoc documentation
and points users to the manpage.

Updated TODO section.

Removed some redundant options from the getopt() invocation.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>


---

 git-archimport.perl |   33 ++++++++++++++++-----------------
 1 files changed, 16 insertions(+), 17 deletions(-)

8d625fb42f58a6b4b2af135efe76726e8f0579e0
diff --git a/git-archimport.perl b/git-archimport.perl
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -6,23 +6,27 @@
 # The basic idea is to walk the output of tla abrowse, 
 # fetch the changesets and apply them. 
 #
+
 =head1 Invocation
 
-    git-archimport -i <archive>/<branch> [<archive>/<branch>]
-    [ <archive>/<branch> ]
+    git-archimport [ -h ] [ -v ] [ -T ] [ -t tempdir ] <archive>/<branch> [ <archive>/<branch> ]
 
-    The script expects you to provide the key roots where it can start the
-    import from an 'initial import' or 'tag' type of Arch commit. It will
-    then follow all the branching and tagging within the provided roots.
+Imports a project from one or more Arch repositories. It will follow branches
+and repositories within the namespaces defined by the <archive/branch>
+parameters suppplied. If it cannot find the remote branch a merge comes from
+it will just import it as a regular commit. If it can find it, it will mark it 
+as a merge whenever possible.
 
-    It will die if it sees branches that have different roots. 
+See man (1) git-archimport for more details.
 
-=head2 TODO
+=head1 TODO
 
- - keep track of merged patches, and mark a git merge when it happens
- - smarter rules to parse the archive history "up" and "down"
- - be able to continue an import where we left off
+ - create tag objects instead of ref tags
  - audit shell-escaping of filenames
+ - better handling of temp directories
+ - use GIT_DIR instead of hardcoded ".git"
+ - hide our private tags somewhere smarter
+ - find a way to make "cat *patches | patch" safe even when patchfiles are missing newlines  
 
 =head1 Devel tricks
 
@@ -54,14 +58,13 @@ our($opt_h,$opt_v, $opt_T,
 sub usage() {
     print STDERR <<END;
 Usage: ${\basename $0}     # fetch/update GIT from Arch
-       [ -h ] [ -v ] [ -T ] 
-       [ -C GIT_repository ] [ -t tempdir ] 
+       [ -h ] [ -v ] [ -T ] [ -t tempdir ] 
        repository/arch-branch [ repository/arch-branch] ...
 END
     exit(1);
 }
 
-getopts("ThviC:t:") or usage();
+getopts("Thvt:") or usage();
 usage if $opt_h;
 
 @ARGV >= 1 or usage();
@@ -71,10 +74,6 @@ my $tmp = $opt_t;
 $tmp ||= '/tmp';
 $tmp .= '/git-archimport/';
 
-my $git_tree = $opt_C;
-$git_tree ||= ".";
-
-
 my @psets  = ();                # the collection
 my %psets  = ();                # the collection, by name
 

[PATCH 1/5] archimport documentation update

[martin]

Updated and expanded the command description, and added a reference of the
command line options.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>


---

 Documentation/git-archimport.txt |   63 +++++++++++++++++++++++++++++++-------
 1 files changed, 51 insertions(+), 12 deletions(-)

b3ee96f94c9ec4378d18a13074d1bbeebb0858b5
diff --git a/Documentation/git-archimport.txt b/Documentation/git-archimport.txt
--- a/Documentation/git-archimport.txt
+++ b/Documentation/git-archimport.txt
@@ -8,24 +8,63 @@ git-archimport - Import an arch reposito
 
 SYNOPSIS
 --------
-`git-archimport` [--option...] <args>
+`git-archimport` [ -h ] [ -v ] [ -T ] [ -t tempdir ] 
+                 <archive/branch> [ <archive/branch> ]
 
 DESCRIPTION
 -----------
-The script expects you to provide the key roots where it can start the
-import from an 'initial import' or 'tag' type of Arch commit. It will
-then follow all the branching and tagging within the provided roots.
-
-It will die if it sees branches that have different roots.
-
+Imports a project from one or more Arch repositories. It will follow branches
+and repositories within the namespaces defined by the <archive/branch>
+parameters suppplied. If it cannot find the remote branch a merge comes from
+it will just import it as a regular merge. If it can find it, it will perform
+a merge whenever possible. 
+
+The script expects you to provide the key roots where it can start the import 
+from an 'initial import' or 'tag' type of Arch commit. It will follow and import 
+new branches within the provided roots. 
+
+It expects to be dealing with one project only. If it sees 
+branches that have different roots, it will refuse to run. In that case, edit your
+<archive/branch> parameters to define clearly the scope of the import. 
+
+`git-archimport` uses `tla` extensively in the background to access the Arch repository.
+Make sure you have a recent version of `tla` available in the path. `tla` must
+know about the repositories you pass to `git-archimport`. 
+
+For the initial import `git-archimport` expects to find itself in an empty 
+directory. To follow the development of a project that uses Arch, rerun 
+`git-archimport` with the same parameters as the initial import to perform incremental imports.
+of a project using Arch.
+
+Patch merge data from Arch is used to mark merges in GIT as well. GIT 
+does not care much about tracking patches, and only considers a merge when a
+branch incorporates all the commits since the point they forked. The end result
+is that GIT will have a good idea of how far branches have diverged. So the 
+import process does lose some patch-trading metadata.
+
+Fortunately, when you try and merge branches imported from Arch, 
+GIT will find a good merge base, and it has a good chance of identifying 
+patches that have been traded out-of-sequence between the branches. 
 
 OPTIONS
 -------
---option::
-	Some option not yet documented.
 
-<args>...::
-	Some argument not yet documented.
+-h::
+	Display usage.
+
+-v::
+	Verbose output. 
+
+-T::
+	Many tags. Will create a tag for every commit, reflecting the commit 
+	name in the Arch repository.
+
+-t <tmpdir>::
+	Override the default tempdir.
+
+
+<archive/branch>::
+	Archive/branch identifier in a format that `tla log` understands. 
 
 
 Author
@@ -34,7 +73,7 @@ Written by Martin Langhoff <martin@catal
 
 Documentation
 --------------
-Documentation by Junio C Hamano and the git-list <git@vger.kernel.org>.
+Documentation by Junio C Hamano, Martin Langhoff and the git-list <git@vger.kernel.org>.
 
 GIT
 ---

[PATCH 2/5] archimport documentation tidyup

[martin]

New "merges" headline, clarified some parts that were not easy to understand.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>


---

 Documentation/git-archimport.txt |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

da341e8530b0d852c9b6910778df6b41cdee3e54
diff --git a/Documentation/git-archimport.txt b/Documentation/git-archimport.txt
--- a/Documentation/git-archimport.txt
+++ b/Documentation/git-archimport.txt
@@ -3,7 +3,7 @@ git-archimport(1)
 
 NAME
 ----
-git-archimport - Import an arch repository into git
+git-archimport - Import an Arch repository into GIT
 
 
 SYNOPSIS
@@ -16,8 +16,8 @@ DESCRIPTION
 Imports a project from one or more Arch repositories. It will follow branches
 and repositories within the namespaces defined by the <archive/branch>
 parameters suppplied. If it cannot find the remote branch a merge comes from
-it will just import it as a regular merge. If it can find it, it will perform
-a merge whenever possible. 
+it will just import it as a regular commit. If it can find it, it will mark it 
+as a merge whenever possible (see discussion below). 
 
 The script expects you to provide the key roots where it can start the import 
 from an 'initial import' or 'tag' type of Arch commit. It will follow and import 
@@ -34,8 +34,9 @@ know about the repositories you pass to 
 For the initial import `git-archimport` expects to find itself in an empty 
 directory. To follow the development of a project that uses Arch, rerun 
 `git-archimport` with the same parameters as the initial import to perform incremental imports.
-of a project using Arch.
 
+MERGES
+------
 Patch merge data from Arch is used to mark merges in GIT as well. GIT 
 does not care much about tracking patches, and only considers a merge when a
 branch incorporates all the commits since the point they forked. The end result

[PATCH 4/5] archimport - use GIT_DIR instead of hardcoded ".git"

[martin]

Use GIT_DIR from the environment instead of a hardcoded '.git' string.

Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>


---

 git-archimport.perl |   42 ++++++++++++++++++++++--------------------
 1 files changed, 22 insertions(+), 20 deletions(-)

600258cfefc38939f09f267090ba400d80ab14c3
diff --git a/git-archimport.perl b/git-archimport.perl
--- a/git-archimport.perl
+++ b/git-archimport.perl
@@ -24,7 +24,6 @@ See man (1) git-archimport for more deta
  - create tag objects instead of ref tags
  - audit shell-escaping of filenames
  - better handling of temp directories
- - use GIT_DIR instead of hardcoded ".git"
  - hide our private tags somewhere smarter
  - find a way to make "cat *patches | patch" safe even when patchfiles are missing newlines  
 
@@ -52,6 +51,9 @@ use IPC::Open2;
 $SIG{'PIPE'}="IGNORE";
 $ENV{'TZ'}="UTC";
 
+my $git_dir = $ENV{"GIT_DIR"} || ".git";
+$ENV{"GIT_DIR"} = $git_dir;
+
 our($opt_h,$opt_v, $opt_T,
     $opt_C,$opt_t);
 
@@ -179,7 +181,7 @@ foreach my $root (@arch_roots) {
 ##      and put an initial import
 ##      or a full tag
 my $import = 0;
-unless (-d '.git') { # initial import
+unless (-d $git_dir) { # initial import
     if ($psets[0]{type} eq 'i' || $psets[0]{type} eq 't') {
         print "Starting import from $psets[0]{id}\n";
 	`git-init-db`;
@@ -190,11 +192,11 @@ unless (-d '.git') { # initial import
     }
 } else {    # progressing an import
     # load the rptags
-    opendir(DIR, ".git/archimport/tags")
+    opendir(DIR, "$git_dir/archimport/tags")
 	|| die "can't opendir: $!";
     while (my $file = readdir(DIR)) {
 	# skip non-interesting-files
-	next unless -f ".git/archimport/tags/$file";
+	next unless -f "$git_dir/archimport/tags/$file";
 	next if     $file =~ m/--base-0$/; # don't care for base-0
 	my $sha = ptag($file);
 	chomp $sha;
@@ -238,7 +240,7 @@ foreach my $ps (@psets) {
     }
 
     unless ($import) { # skip for import
-        if ( -e ".git/refs/heads/$ps->{branch}") {
+        if ( -e "$git_dir/refs/heads/$ps->{branch}") {
             # we know about this branch
             `git checkout    $ps->{branch}`;
         } else {
@@ -291,7 +293,7 @@ foreach my $ps (@psets) {
     # imports don't give us good info
     # on added files. Shame on them
     if ($ps->{type} eq 'i' || $ps->{type} eq 't') { 
-        `find . -type f -print0 | grep -zv '^./.git' | xargs -0 -l100 git-update-index --add`;
+        `find . -type f -print0 | grep -zv '^./$git_dir' | xargs -0 -l100 git-update-index --add`;
         `git-ls-files --deleted -z | xargs --no-run-if-empty -0 -l100 git-update-index --remove`;
     }
 
@@ -355,8 +357,8 @@ foreach my $ps (@psets) {
     # Who's your daddy?
     #
     my @par;
-    if ( -e ".git/refs/heads/$ps->{branch}") {
-        if (open HEAD, "<.git/refs/heads/$ps->{branch}") {
+    if ( -e "$git_dir/refs/heads/$ps->{branch}") {
+        if (open HEAD, "<$git_dir/refs/heads/$ps->{branch}") {
             my $p = <HEAD>;
             close HEAD;
             chomp $p;
@@ -403,11 +405,11 @@ foreach my $ps (@psets) {
     #
     # Update the branch
     # 
-    open  HEAD, ">.git/refs/heads/$ps->{branch}";
+    open  HEAD, ">$git_dir/refs/heads/$ps->{branch}";
     print HEAD $commitid;
     close HEAD;
-    unlink ('.git/HEAD');
-    symlink("refs/heads/$ps->{branch}",".git/HEAD");
+    unlink ("$git_dir/HEAD");
+    symlink("refs/heads/$ps->{branch}","$git_dir/HEAD");
 
     # tag accordingly
     ptag($ps->{id}, $commitid); # private tag
@@ -436,7 +438,7 @@ sub apply_import {
 
     `tla get -s --no-pristine -A $ps->{repo} $ps->{id} $tmp/import`;
     die "Cannot get import: $!" if $?;    
-    `rsync -v --archive --delete --exclude '.git' --exclude '.arch-ids' --exclude '{arch}' $tmp/import/* ./`;
+    `rsync -v --archive --delete --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/import/* ./`;
     die "Cannot rsync import:$!" if $?;
     
     `rm -fr $tmp/import`;
@@ -482,7 +484,7 @@ sub apply_cset {
     }
 
     # bring in new files
-    `rsync --archive --exclude '.git' --exclude '.arch-ids' --exclude '{arch}' $tmp/changeset/new-files-archive/* ./`;
+    `rsync --archive --exclude '$git_dir' --exclude '.arch-ids' --exclude '{arch}' $tmp/changeset/new-files-archive/* ./`;
 
     # deleted files are hinted from the commitlog processing
 
@@ -577,7 +579,7 @@ sub tag {
     $tag = shell_quote($tag);
     
     if ($commit) {
-        open(C,">.git/refs/tags/$tag")
+        open(C,">$git_dir/refs/tags/$tag")
             or die "Cannot create tag $tag: $!\n";
         print C "$commit\n"
             or die "Cannot write tag $tag: $!\n";
@@ -585,7 +587,7 @@ sub tag {
             or die "Cannot write tag $tag: $!\n";
         print " * Created tag ' $tag' on '$commit'\n" if $opt_v;
     } else {                    # read
-        open(C,"<.git/refs/tags/$tag")
+        open(C,"<$git_dir/refs/tags/$tag")
             or die "Cannot read tag $tag: $!\n";
         $commit = <C>;
         chomp $commit;
@@ -603,12 +605,12 @@ sub ptag {
     $tag =~ s|/|--|g; 
     $tag = shell_quote($tag);
     
-    unless (-d '.git/archimport/tags') {
-        mkpath('.git/archimport/tags');
+    unless (-d "$git_dir/archimport/tags") {
+        mkpath("$git_dir/archimport/tags");
     }
 
     if ($commit) {              # write
-        open(C,">.git/archimport/tags/$tag")
+        open(C,">$git_dir/archimport/tags/$tag")
             or die "Cannot create tag $tag: $!\n";
         print C "$commit\n"
             or die "Cannot write tag $tag: $!\n";
@@ -618,10 +620,10 @@ sub ptag {
 	    unless $tag =~ m/--base-0$/;
     } else {                    # read
         # if the tag isn't there, return 0
-        unless ( -s ".git/archimport/tags/$tag") {
+        unless ( -s "$git_dir/archimport/tags/$tag") {
             return 0;
         }
-        open(C,"<.git/archimport/tags/$tag")
+        open(C,"<$git_dir/archimport/tags/$tag")
             or die "Cannot read tag $tag: $!\n";
         $commit = <C>;
         chomp $commit;

Re: [PATCH 1/5] archimport documentation update

[A Large Angry SCM]

martin@catalyst.net.nz wrote:
> Updated and expanded the command description, and added a reference of the
> command line options.
> 
> Signed-off-by: Martin Langhoff <martin@catalyst.net.nz>
> 
[...]

Thanks Martin!