[PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[Jeremie Nikaes]

Implement a gate between git and mediawiki, allowing git users to
push and pull objects from mediawiki just as one would do with a
classic git repository thanks to remote-helpers.

Currently supported commands are :
      git clone mediawiki::http://onewiki.com
      git pull

You need the following packages installed (available on common
repositories):
      libmediawiki-api-perl
      libdatetime-format-iso8601-perl

Use remote helpers in order to be as transparent as possible
to the git user.

Download Mediawiki revisions through the Mediawiki API and then
fast-import into git.

Mediawiki revisions and git commits are linked thanks to notes bound to
commits.

The import part is done on a refs/mediawiki/<remote> branch before
coming to refs/remote/origin/master (Huge thanks to Jonathan Nieder
for his help)

For now, the whole wiki is cloned, but it will be possible to clone only
some pages: the clone is based on a list of pages which is now all
pages.

Code clarified & improved with the help of Jeff King and Junio C Hamano.

We were not able to reproduce the empty timestamp bug noticed by Jeff
King, thus needing some further testing. A placeholder is still
implemented in case.

Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
---
 contrib/mw-to-git/git-remote-mediawiki     |  306 ++++++++++++++++++++++++++++
 contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
 2 files changed, 313 insertions(+), 0 deletions(-)
 create mode 100755 contrib/mw-to-git/git-remote-mediawiki
 create mode 100644 contrib/mw-to-git/git-remote-mediawiki.txt

diff --git a/contrib/mw-to-git/git-remote-mediawiki b/contrib/mw-to-git/git-remote-mediawiki
new file mode 100755
index 0000000..9db64c2
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki
@@ -0,0 +1,306 @@
+#! /usr/bin/perl
+
+use strict;
+use Switch;
+use MediaWiki::API;
+use Storable qw(freeze thaw);
+use DateTime::Format::ISO8601;
+use Encode qw(encode_utf8);
+
+# Mediawiki filenames can contain forward slashes. This variable decides by which pattern they should be replaced
+my $slash_replacement = "<slash>";
+
+my $remotename = $ARGV[0];
+my $url = $ARGV[1];
+
+print STDERR "$url\n";
+
+# commands parser
+my $loop = 1;
+my $entry;
+my @cmd;
+while ($loop) {
+	$| = 1; #flush STDOUT
+	$entry = <STDIN>;
+	print STDERR $entry;
+	chomp($entry);
+	@cmd = undef;
+	@cmd = split(/ /,$entry);
+	switch ($cmd[0]) {
+		case "capabilities" {
+			if ($cmd[1] eq "") {
+				mw_capabilities();
+			} else {
+			       $loop = 0;
+			}
+		}
+		case "list" {
+			if ($cmd[2] eq "") {
+				mw_list($cmd[1]);
+			} else {
+			       $loop = 0;
+			}
+		}
+		case "import" {
+			if ($cmd[1] ne "" && $cmd[2] eq "") {
+				mw_import($cmd[1]);
+			} 
+		}
+		case "option" {
+			if ($cmd[1] ne "" && $cmd[2] ne "" && $cmd[3] eq "") {
+				mw_option($cmd[1],$cmd[2]);
+			} 
+		}
+		case "push" {
+			#check the pattern <src>:<dst>
+			my @pushargs = split(/:/,$cmd[1]);
+			if ($pushargs[1] ne "" && $pushargs[2] eq "") {
+				mw_push($pushargs[0],$pushargs[1]);
+			} else {
+			       $loop = 0;
+			}
+		} else {
+			$loop = 0;
+		}
+	}
+	close(FILE);
+}
+
+########################## Functions ##############################
+
+sub get_last_local_revision {
+	# Get last commit sha1
+	my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
+
+	# Get note regarding that commit
+	chomp($commit_sha1);
+	my $note = `git log --notes=mediawiki 2>/dev/null | grep "mediawiki_revision: " | sed -n 1p`;
+	$note =~ s/^\s+//; #left trim of note 
+	my @note_info = split(/ /, $note);
+
+	my $lastrevision_number;
+	if (!($note_info[0] eq "mediawiki_revision:")) {
+		print STDERR "No previous mediawiki revision found";
+		$lastrevision_number = 0;
+	} else {
+		# Notes are formatted : mediawiki_revision: #number
+		$lastrevision_number = $note_info[1];
+		chomp($lastrevision_number);
+		print STDERR "Last local mediawiki revision found is $lastrevision_number ";
+	}
+	return $lastrevision_number;
+}
+
+sub get_last_remote_revision {
+	my $mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+
+	my $pages = $mediawiki->list({
+		action => 'query',
+		list => 'allpages',
+		aplimit => 500,
+	});
+
+	my $max_rev_num = 0;
+
+	foreach my $page (@$pages) {
+		my $id = $page->{pageid};
+
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			pageids => $id,
+		};
+
+		my $result = $mediawiki->api($query);
+		
+		my $lastrev = pop(@{$result->{query}->{pages}->{$id}->{revisions}});
+		
+		$max_rev_num = ($lastrev->{revid} > $max_rev_num ? $lastrev->{revid} : $max_rev_num);
+	}
+
+	print STDERR "Last remote revision found is $max_rev_num\n";
+	return $max_rev_num;
+}
+
+sub literal_data {
+	my ($content) = @_;
+	print "data ", bytes::length($content), "\n", $content;
+}
+
+sub mw_capabilities {
+	# Revisions are imported to the private namespace
+	# refs/mediawiki/$remotename/ by the helper and fetched into
+	# refs/remotes/$remotename later by fetch.
+	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
+	print STDOUT "import\n";
+	print STDOUT "list\n";
+	print STDOUT "option\n";
+	print STDOUT "push\n";
+	print STDOUT "\n";
+}
+
+sub mw_list {
+	# MediaWiki do not have branches, we consider one branch arbitrarily
+	# called master
+	print STDOUT "? refs/heads/master\n";
+	print STDOUT '@'."refs/heads/master HEAD\n";
+	print STDOUT "\n";
+
+}
+
+sub mw_option {
+	print STDERR "not yet implemented \n";
+	print STDOUT "unsupported\n";
+}
+
+sub mw_import {
+	my @wiki_name = split(/:\/\//,$url);
+	my $wiki_name = $wiki_name[1];
+
+	my $mediawiki = MediaWiki::API->new;
+	$mediawiki->{config}->{api_url} = "$url/api.php";
+
+	my $pages = $mediawiki->list({
+		action => 'query',
+		list => 'allpages',
+		aplimit => 500,
+	});
+	if ($pages == undef) {
+		print STDERR "fatal: '$url' does not appear to be a mediawiki\n";
+		exit;
+	}
+
+	my @revisions;
+	print STDERR "Searching revisions...\n";
+	my $fetch_from = get_last_local_revision() + 1;
+	if ($fetch_from == 1) {
+		print STDERR ", fetching from beginning\n";
+	} else {
+		print STDERR ", fetching from here\n";
+	}
+	my $n = 1;
+	my $last_timestamp = 0;
+	foreach my $page (@$pages) {
+		my $id = $page->{pageid};
+
+		print STDERR "$n/", scalar(@$pages), ": ". encode_utf8($page->{title})."\n";
+		$n++;
+
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'ids',
+			rvdir => 'newer',
+			rvstartid => $fetch_from,
+			rvlimit => 500,
+			pageids => $page->{pageid},
+		};
+
+		my $revnum = 0;
+		# Get 500 revisions at a time due to the mediawiki api limit
+		while (1) {
+			my $result = $mediawiki->api($query);
+
+			# Parse each of those 500 revisions
+			foreach my $revision (@{$result->{query}->{pages}->{$id}->{revisions}}) {
+				my $page_rev_ids;
+				$page_rev_ids->{pageid} = $page->{pageid};
+				$page_rev_ids->{revid} = $revision->{revid};
+				push (@revisions, $page_rev_ids);
+				$revnum++;
+			}
+			last unless $result->{'query-continue'};
+			$query->{rvstartid} = $result->{'query-continue'}->{revisions}->{rvstartid};
+		}
+		print STDERR "  Found ", $revnum, " revision(s).\n";
+	}
+
+	# Creation of the fast-import stream
+	print STDERR "Fetching & writing export data...\n";
+	binmode STDOUT, ':binary';
+	$n = 0;
+
+	foreach my $pagerevids (sort {$a->{revid} <=> $b->{revid}} @revisions) {
+		#fetch the content of the pages
+		my $query = {
+			action => 'query',
+			prop => 'revisions',
+			rvprop => 'content|timestamp|comment|user|ids',
+			revids => $pagerevids->{revid},
+		};
+
+		my $result = $mediawiki->api($query);
+
+		my $rev = pop(@{$result->{query}->{pages}->{$pagerevids->{pageid}}->{revisions}});
+
+		$n++;
+		my $user = $rev->{user} || 'Anonymous';
+		my $dt;
+
+		# This has to be verified, haven't been able to reproduce the scenario
+		if ($rev->{timestamp} != undef) {
+			$dt = DateTime::Format::ISO8601->parse_datetime($rev->{timestamp});
+		} else {
+			$dt = $last_timestamp + 1;
+		}
+		$last_timestamp = $dt;
+
+
+		my $comment = defined $rev->{comment} ? $rev->{comment} : '*Empty MediaWiki Message*';
+		my $title = encode_utf8($result->{query}->{pages}->{$pagerevids->{pageid}}->{title});
+		my $content = $rev->{'*'};
+		
+		$title =~ y/ /_/;
+		$title =~ s/\//$slash_replacement/g;
+
+		print STDERR "$n/", scalar(@revisions), ": Revision n°$pagerevids->{revid} of $title\n";
+
+		print "commit refs/mediawiki/$remotename/master\n";
+		print "mark :$n\n";
+		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
+		literal_data(encode_utf8($comment));
+		# If it's not a clone, needs to know where to start from
+		if ($fetch_from != 1 && $n == 1) {
+			print "from refs/mediawiki/$remotename/master^0\n";
+		}
+		print "M 644 inline $title.mw\n";
+		literal_data(encode_utf8($content));
+		print "\n\n";
+
+
+		# mediawiki revision number in the git note
+		if ($fetch_from == 1 && $n == 1) {
+			print "reset refs/notes/mediawiki\n";
+		}
+		print "commit refs/notes/mediawiki\n";
+		print "committer $user <$user\@$wiki_name> ", $dt->epoch, " +0000\n";
+		literal_data(encode_utf8("note added by git-mediawiki"));
+		if ($fetch_from != 1 && $n == 1) {
+			print "from refs/notes/mediawiki^0\n";
+		}
+		print "N inline :$n\n";
+		literal_data(encode_utf8("mediawiki_revision: " . $pagerevids->{revid}));
+		print "\n\n";
+	}
+
+	if ($fetch_from == 1) {
+		if ($n != 0) {
+			print "reset $_[0]\n"; #$_[0] contains refs/heads/master
+			print "from :$n\n";
+		} else {
+			print STDERR "You appear to have cloned an empty mediawiki\n";
+			#What do we have to do here ? If nothing is done, an error is thrown saying that
+			#HEAD is refering to unknown object 0000000000000000000
+		}
+	}
+
+}
+
+sub mw_push {
+
+	print STDERR "Not yet implemented";
+	print "\n";
+}
diff --git a/contrib/mw-to-git/git-remote-mediawiki.txt b/contrib/mw-to-git/git-remote-mediawiki.txt
new file mode 100644
index 0000000..4d211f5
--- /dev/null
+++ b/contrib/mw-to-git/git-remote-mediawiki.txt
@@ -0,0 +1,7 @@
+Git-Mediawiki is a project which aims the creation of a gate
+between git and mediawiki, allowing git users to push and pull
+objects from mediawiki just as one would do with a classic git
+repository thanks to remote-helpers.
+
+For more information, visit the wiki at
+https://github.com/Bibzball/Git-Mediawiki/wiki
-- 
1.7.4.1

Re: [PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[Junio C Hamano]

Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr> writes:

> Implement a gate between git and mediawiki, allowing git users to
> ...
> For now, the whole wiki is cloned, but it will be possible to clone only
> some pages: the clone is based on a list of pages which is now all
> pages.
>
> Code clarified & improved with the help of Jeff King and Junio C Hamano.
>
> We were not able to reproduce the empty timestamp bug noticed by Jeff
> King, thus needing some further testing. A placeholder is still
> implemented in case.
>
> Signed-off-by: Jérémie Nikaes <jeremie.nikaes@ensimag.imag.fr>
> Signed-off-by: Arnaud Lacurie <arnaud.lacurie@ensimag.imag.fr>
> Signed-off-by: Claire Fousse <claire.fousse@ensimag.imag.fr>
> Signed-off-by: David Amouyal <david.amouyal@ensimag.imag.fr>
> Signed-off-by: Matthieu Moy <matthieu.moy@grenoble-inp.fr>
> Signed-off-by: Sylvain Boulmé <sylvain.boulme@imag.fr>
> ---
>  contrib/mw-to-git/git-remote-mediawiki     |  306 ++++++++++++++++++++++++++++
>  contrib/mw-to-git/git-remote-mediawiki.txt |    7 +
>  2 files changed, 313 insertions(+), 0 deletions(-)

This seems to have grown a bit.

I won't repeat issues I pointed out in the earlier round but not updated
in this patch to save time.

> +########################## Functions ##############################
> +
> +sub get_last_local_revision {
> +	# Get last commit sha1
> +	my $commit_sha1 = `git rev-parse refs/mediawiki/$remotename/master 2>/dev/null`;
> +
> +	# Get note regarding that commit
> +	chomp($commit_sha1);
> +	my $note = `git log --notes=mediawiki 2>/dev/null | grep "mediawiki_revision: " | sed -n 1p`;

Two and half issues:

 - You are writing Perl no?  Don't call grep/sed from it.  Your
   environment is much richer and more flexible ;-).

 - You grab $commit_sha1 but never use it. Did you mean to throw it at the
   "git log" above?

 - Is there a reason you use "git log" to traverse the history all the way
   down to the root commit?  Wouldn't

	git notes --ref=mediawiki show $commit_sha1

   or even better yet, just doing

	git notes --ref=mediawiki show refs/mediawiki/$remotename/master

   without the first rev-parse sufficient?

   Are you protecting against the case where some commits in the history
   leading to mediawiki/$remotename/master may not have the "mediawiki"
   note, and falling back to the latest commit that has note? You may find
   such a commit, but that may be different from the commit at the tip of
   mediawiki/$remotename/master branch. Is that a correct thing to do?
   IOW, does _any_ previous version do for the purpose of this function?
   (This paragraph is not a rhetorical question).

> +sub mw_capabilities {
> +	# Revisions are imported to the private namespace
> +	# refs/mediawiki/$remotename/ by the helper and fetched into
> +	# refs/remotes/$remotename later by fetch.
> +	print STDOUT "refspec refs/heads/*:refs/mediawiki/$remotename/*\n";
> +	print STDOUT "import\n";
> +	print STDOUT "list\n";
> +	print STDOUT "option\n";
> +	print STDOUT "push\n";
> +	print STDOUT "\n";

There are many explicit references to STDOUT like this, and also many
unqualified "print" that spits out to the default which is STDOUT in the
codepath to feed fast-import. Is that intentional, or is it just coming
from difference in style of people who worked on different parts of the
code?

If there is no reason to use two styles, please pick one and stick to
it. If there _is_ reason, please document what are the criteria to choose
which one in each codepath.  Otherwise you would waste time of your
reviewers who have to wonder which one is correct in which codepath.

Re: [PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[Thomas Adam]

Hi,

On 6 June 2011 10:20, Jeremie Nikaes <jeremie.nikaes@ensimag.imag.fr> wrote:

I've got some points of my own for consideration.

> @@ -0,0 +1,306 @@
> +#! /usr/bin/perl
> +
> +use strict;

use warnings; ?

> +use Switch;

Ugh -- no.  This is terrible.  Look at this:

[~]% corelist Switch
Switch was first released with perl v5.7.3 and removed from v5.13.1

Since you do not specify a minimum perl version you might be alright,
but for those people on 5.14 -- they won't have this module, for good
reason.  You can, if you wanted use "given/when" as alternate
constructs to this.

> +use MediaWiki::API;
> +use Storable qw(freeze thaw);

This might have problems transcending storable formats made on a 32bit
machine, and then trying to unpack them again on 64bit.  Do you really
need the need for these storable items to be encoded as binary?

[...]

-- Thomas Adam

Re: [PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[Jérémie NIKAES]

> This seems to have grown a bit.
>
> I won't repeat issues I pointed out in the earlier round but not updated
> in this patch to save time.

Yes sorry, I actually commited a patch where the weird loop control
was not fixed yet. My mistake, will fix this asap.

About the print STDERR lines, what should we do about them ? They do
grow the patch up but if we remove them completely the script seems to
be frozen because of no text output for the user. Should we put them
in a verbose mode ?

> Two and half issues:
>
>  - You are writing Perl no?  Don't call grep/sed from it.  Your
>   environment is much richer and more flexible ;-).

Okay it's true that we discovered Perl as we started this project so
we were more comfortable with this kind of commands. Will work on
that.
>
>  - You grab $commit_sha1 but never use it. Did you mean to throw it at the
>   "git log" above?
>
>  - Is there a reason you use "git log" to traverse the history all the way
>   down to the root commit?  Wouldn't
>
>        git notes --ref=mediawiki show $commit_sha1
>
>   or even better yet, just doing
>
>        git notes --ref=mediawiki show refs/mediawiki/$remotename/master
>
>   without the first rev-parse sufficient?
>
>   Are you protecting against the case where some commits in the history
>   leading to mediawiki/$remotename/master may not have the "mediawiki"
>   note, and falling back to the latest commit that has note? You may find
>   such a commit, but that may be different from the commit at the tip of
>   mediawiki/$remotename/master branch. Is that a correct thing to do?
>   IOW, does _any_ previous version do for the purpose of this function?
>   (This paragraph is not a rhetorical question).

You are totally right. We actually were trying to find a command that
did this without having to parse the git log, git notes
--ref=mediawiki show refs/mediawiki/$remotename/master does this
perfectly.
In practice, a mediawiki/$remotename/master commit should ALWAYS have
a note attached to it, so we did not try to protect against this case
here.
Should we be concerned and do something about it ?

> There are many explicit references to STDOUT like this, and also many
> unqualified "print" that spits out to the default which is STDOUT in the
> codepath to feed fast-import. Is that intentional, or is it just coming
> from difference in style of people who worked on different parts of the
> code?
>
> If there is no reason to use two styles, please pick one and stick to
> it. If there _is_ reason, please document what are the criteria to choose
> which one in each codepath.  Otherwise you would waste time of your
> reviewers who have to wonder which one is correct in which codepath.

Yes you are right, that slipped too. Thanks for noticing this, fixing it.

As always, thanks for the feedback, working on improving this.

--
Jérémie Nikaes

Re: [PATCH] Add a remote helper to interact with mediawiki, pull & clone handled

[Jérémie NIKAES]

> use warnings; ?

Thanks, added this, showed some uses of undefined variables now fixed.
Will figure in the next patch.
>
>> +use Switch;
>
> Ugh -- no.  This is terrible.  Look at this:
>
> [~]% corelist Switch
> Switch was first released with perl v5.7.3 and removed from v5.13.1
>
> Since you do not specify a minimum perl version you might be alright,
> but for those people on 5.14 -- they won't have this module, for good
> reason.  You can, if you wanted use "given/when" as alternate
> constructs to this.

Yes, as I said to Junio Hamano in my previous mail, that's a mistake
from my part, we don't use switch anymore. Sorry about that

>
>> +use MediaWiki::API;
>> +use Storable qw(freeze thaw);
>
> This might have problems transcending storable formats made on a 32bit
> machine, and then trying to unpack them again on 64bit.  Do you really
> need the need for these storable items to be encoded as binary?

Hmmm that is a leftover use from the script of Jeff King that we used.
We don't use this anymore, so that should not be a problem.

Thanks for your advices !

--
Jérémie Nikaes