WIP: asciidoc replacement

WIP: asciidoc replacement

[Johannes Schindelin]

Hi,

I do not want to depend on more than necessary in msysGit, and therefore I 
started to write an asciidoc replacement.

So here it is: a perl script that does a good job on many .txt files in 
Documentation/, although for some it deviates from "make man"'s output, 
and for others it is outright broken.  It is meant to be run in 
Documentation/.

My intention is not to fix the script for all cases, but to make patches 
to Documentation/*.txt themselves, so that they are more consistent (and 
incidentally nicer to the script).

Now, I hear you already moan: "But Dscho, you know you suck at Perl!"

Yeah, I know, but maybe instead of bashing on me (pun intended), you may 
want to enlighten me with tips how to make it nicer to read.  (Yes, there 
are no comments; yes, I will gladly add them where appropriate; yes, html 
is just a stub.)

So here, without further ado, da script:

-- snip --
#!/usr/bin/perl

$conv = new man_page();
$conv->{manual} = 'Git Manual';
$conv->{git_version} = 'Git ' . `cat ../GIT-VERSION-FILE`;
$conv->{git_version} =~ s/GIT_VERSION = //;
$conv->{git_version} =~ s/-/\\-/;
$conv->{git_version} =~ s/\n//;
$conv->{date} = `date +%m/%d/%Y`;
$conv->{date} =~ s/\n//;

$par = '';
handle_file($ARGV[0]);
$conv->finish();

sub handle_text {
	if ($par =~ /^\. /s) {
		my @lines = split(/^\. /m, $par);
		shift @lines;
		$conv->enumeration(\@lines);
	} elsif ($par =~ /^\* /s) {
		my @lines = split(/^\* /m, $par);
		shift @lines;
		$conv->enumeration(\@lines, 'unnumbered');
	} elsif ($par =~ /^\[verse\]/) {
		$par =~ s/\[verse\] *\n?//;
		$conv->verse($par);
	} elsif ($par =~ /^(\t|  +)/s) {
		$par =~ s/^$1//mg;
		$par =~ s/^\+$//mg;
		$conv->indent($par);
	} elsif ($par =~ /^([^\n]*)::\n((\t|  +).*)$/s) {
		my ($first, $rest, $indent) = ($1, $2, $3);
		$rest =~ s/^\+$//mg;
		while ($rest =~ /^(.*?\n\n)--+\n(.*?\n)--+\n\n(.*)$/s) {
			my ($pre, $verb, $post) = ($1, $2, $3);

			$pre =~ s/^(\t|$indent)//mg;
			if ($first ne '') {
				$conv->begin_item($first, $pre);
				$first = '';
			} else {
				$conv->normal($pre);
			}

			$conv->verbatim($verb);
			$rest = $post;
		}
		$rest =~ s/^(\t|$indent)//mg;
		if ($first ne '') {
			$conv->begin_item($first, $rest);
		} else {
			$conv->normal($rest);
		}
		$conv->end_item();
	} elsif ($par =~ /^-+\n(.*\n)-+\n$/s) {
		$conv->verbatim($1);
	} else {
		$conv->normal($par);
	}
	$par = '';
}

sub handle_file {
	my $in;
	open($in, '<' . $_[0]);
	while (<$in>) {
		if (/^=+$/) {
			if ($par ne '' && length($_) >= length($par)) {
				$conv->header($par);
				$par = '';
				next;
			}
		} elsif (/^-+$/) {
			if ($par ne '' && length($_) >= length($par)) {
				$conv->section($par);
				$par = '';
				next;
			}
		} elsif (/^~+$/) {
			if ($par ne '' && length($_) >= length($par)) {
				$conv->subsection($par);
				$par = '';
				next;
			}
		} elsif (/^\[\[(.*)\]\]$/) {
			handle_text();
			$conv->anchor($1);
			next;
		} elsif (/^$/) {
			if ($par =~ /^-+\n.*[^-]\n$/s) {
				# fallthru; is verbatim, but needs more.
			} elsif ($par =~ /::\n$/s) {
				# is item, but needs more.
				next;
			} else {
				handle_text();
				next;
			}
		} elsif (/^include::(.*)\[\]$/) {
			handle_text();
			handle_file($1);
			next;
		}

		# convert "\--" to "--"
		s/\\--/--/g;
		# convert "\*" to "*"
		s/\\\*/*/g;

		# handle gitlink:
		s/gitlink:([^\[ ]*)\[(\d+)\]/sprintf "%s",
			$conv->get_link($1, $2)/ge;
		# handle link:
		s/link:([^\[ ]*)\[(.+)\]/sprintf "%s",
			$conv->get_link($1, $2, 'external')/ge;

		$par .= $_;
	}
	close($in);
	handle_text();
}

package man_page;

sub new {
	my ($class) = @_;
	my $self = {
		sep => '',
		links => [],
#		generator => 'Home grown git txt2man converter'
		generator => 'DocBook XSL Stylesheets v1.71.1 <http://docbook.sf.net/>'
	};
	bless $self, $class;
	return $self;
}

sub header {
	my ($self, $text) = @_;
	$text =~ s/-/\\-/g;

	if ($self->{preamble_shown} == undef) {
		$title = $text;
		$title =~ s/\(\d+\)$//;
		print '.\"     Title: ' . $title
			. '.\"    Author: ' . "\n"
			. '.\" Generator: ' . $self->{generator} . "\n"
			. '.\"      Date: ' . $self->{date} . "\n"
			. '.\"    Manual: ' . $self->{manual} . "\n"
			. '.\"    Source: ' . $self->{git_version} . "\n"
			. '.\"' . "\n";
	}

	$text =~ tr/a-z/A-Z/;
	my $suffix = "\"$self->{date}\" \"$self->{git_version}\""
		. " \"$self->{manual}\"";
	$text =~ s/^(.*)\((\d+)\)$/.TH "\1" "\2" $suffix/;
	print $text;

	if ($self->{preamble_shown} == undef) {
		print '.\" disable hyphenation' . "\n"
			. '.nh' . "\n"
			. '.\" disable justification (adjust text to left'
				. ' margin only)' . "\n"
			. '.ad l' . "\n";
		$self->{preamble_shown} = 1;
	}

	$self->{last_op} = 'header';
}

sub section {
	my ($self, $text) = @_;

	$text =~ tr/a-z/A-Z/;
	$text =~ s/^(.*)$/.SH "\1"/;

	print $text;

	$self->{last_op} = 'section';
}

sub subsection {
	my ($self, $text) = @_;

	$text =~ s/^(.*)$/.SS "\1"/;

	print $text;

	$self->{last_op} = 'subsection';
}

sub get_link {
	my ($self, $command, $section, $option) = @_;

	if ($option eq 'external') {
		my $links = $self->{links};
		push(@$links, $command);
		$command =~ s/\.html$//;
		$command =~ s/-/ /g;
		push(@$links, $command);
		return '\fI' . $command . '\fR\&[1]';
	} else {
		return '\fB' . $command . '\fR(' . $section . ')';
	}
}

sub common {
	my ($self, $text, $option) = @_;

	# escape backslashes, but not in "\n", "\&" or "\fB"
	$text =~ s/\\(?!n|f[A-Z]|&)/\\\\/g;
	# escape "-"
	$text =~ s/-/\\-/g;
	# handle ...
	$text =~ s/(\.\.\.)/\\&\1/g;
	# remove double space after full stop or comma
	$text =~ s/([\.,])  /\1 /g;

	if ($option ne 'no-markup') {
		# make 'italic'
		$text =~ s/'([^'\n]*)'/\\fI\1\\fR/g;
		# ignore `
		$text =~ s/`//g;
		# make *bold*
		$text =~ s/\*([^\*\n]*)\*/\\fB\1\\fR/g;
		# handle <<sections>
		$text =~ s/<<([^>]*)>>/the section called \\(lq\1\\(rq/g;
	}

	return $text;
}

sub normal {
	my ($self, $text) = @_;

	if ($text eq "") {
		return;
	}

	$text = $self->common($text);

	$text =~ s/ *\n(.)/ \1/g;

	if ($self->{last_op} eq 'normal') {
		print "\n";
	}

	print $text;

	$self->{last_op} = 'normal';
}

sub verse {
	my ($self, $text) = @_;

	$text = $self->common($text);
	$text =~ s/^\t/        /mg;

	print ".sp\n.RS 4\n.nf\n" . $text . ".fi\n.RE\n";

	$self->{last_op} = 'verse';
}

sub enumeration {
	my ($self, $text, $option) = @_;

	my $counter = 0;
	foreach $line (@$text) {
		$counter++;
		print ".TP 4\n"
			. ($option eq 'unnumbered' ? '\(bu' : $counter . '.')
			. "\n"
			. $self->common($line);
	}

	$self->{last_op} = 'enumeration';
}

sub begin_item {
	my ($self, $item, $text) = @_;

	$item = $self->common($item);
	$text = $self->common($text);

	$text =~ s/([^\n]) *\n([^\n])/\1 \2/g;

	print ".PP\n" . $item . "\n.RS 4\n" . $text;

	$self->{last_op} = 'item'; 
}

sub end_item {
	my ($self) = @_;

	print ".RE\n";

	$self->{last_op} = 'end_item';
}

sub indent {
	my ($self, $text) = @_;

	$text = $self->common($text, 'no-markup');
	$text =~ s/^\t/        /mg;

	if ($self->{last_op} eq 'normal') {
		print "\n";
	}

	print ".sp\n.RS 4\n.nf\n" . $text . ".fi\n.RE\n";

	$self->{last_op} = 'indent';
}

sub verbatim {
	my ($self, $text) = @_;

	$text = $self->common($text, 'no-markup');

	# convert tabs to spaces
	$text =~ s/^\t/        /mg;
	# remove trailing empty lines
	$text =~ s/\n\n*$/\n/;

	if ($self->{last_op} eq 'normal') {
		print "\n";
	}

	print ".sp\n.RS 4\n.nf\n.ft C\n" . $text . ".ft\n\n.fi\n.RE\n";

	$self->{last_op} = 'verbatim';
}

sub anchor {
	my ($self, $text) = @_;

	$self->{last_op} = 'anchor';
}

sub finish {
	my ($self) = @_;
	my $links = $self->{links};

	if ($#$links >= 0) {
		print '.SH "REFERENCES"' . "\n";
		my $i = 1;
		while ($#$links >= 0) {
			my $ref = shift(@$links);
			$ref =~ s/-/\\-/g;
			my $label = shift(@$links);
			printf (".IP \"% 2d.\" 4\n%s\n.RS 4\n\\%%%s\n.RE\n",
				$i++, $label, $ref);
		}
	} else {
		print "\n";
	}
}

package html_page;

sub new {
	my ($class) = @_;
	my $self = {};
	bless $self, $class;
	return $self;
}

-- snap --

Ciao,
Dscho

P.S.: I need to catch some Zs, and do some real work, so do not be 
surprised if I do not respond within the next 24 hours.

Re: WIP: asciidoc replacement

[Sam Vilain]

Johannes Schindelin wrote:
> Hi,
> 
> I do not want to depend on more than necessary in msysGit, and therefore I 
> started to write an asciidoc replacement.
> 
> So here it is: a perl script that does a good job on many .txt files in 
> Documentation/, although for some it deviates from "make man"'s output, 
> and for others it is outright broken.  It is meant to be run in 
> Documentation/.
> 
> My intention is not to fix the script for all cases, but to make patches 
> to Documentation/*.txt themselves, so that they are more consistent (and 
> incidentally nicer to the script).
> 
> Now, I hear you already moan: "But Dscho, you know you suck at Perl!"
> 
> Yeah, I know, but maybe instead of bashing on me (pun intended), you may 
> want to enlighten me with tips how to make it nicer to read.  (Yes, there 
> are no comments; yes, I will gladly add them where appropriate; yes, html 
> is just a stub.)
> 
> So here, without further ado, da script:

It's pretty good, I certainly wouldn't have trouble reading or
maintaining it, but I'll give you suggestions anyway.

nice work, replacing a massive XML/XSL/etc stack with a small Perl
script ;-)

Sam.

> 
> -- snip --
> #!/usr/bin/perl

Add -w for warnings, also use strict;

> $conv = new man_page();
> $conv->{manual} = 'Git Manual';
> $conv->{git_version} = 'Git ' . `cat ../GIT-VERSION-FILE`;
> $conv->{git_version} =~ s/GIT_VERSION = //;
> $conv->{git_version} =~ s/-/\\-/;
> $conv->{git_version} =~ s/\n//;
> $conv->{date} = `date +%m/%d/%Y`;
> $conv->{date} =~ s/\n//;
> 
> $par = '';
> handle_file($ARGV[0]);
> $conv->finish();
> 
> sub handle_text {

this function acts on globals; make them explicit arguments to the function.

> 	if ($par =~ /^\. /s) {
> 		my @lines = split(/^\. /m, $par);
> 		shift @lines;
> 		$conv->enumeration(\@lines);
> 	} elsif ($par =~ /^\* /s) {

uncuddle your elsif's; also consider making this a "tabular ternary"
with the actions in separate functions.

ie

$result = ( $par =~ /^\. /s      ? $conv->do_enum($par)    :
            $par =~ /^\[verse\]/ ? $conv->do_verse($par)  :
            ... )

However I have a suspicion that your script is doing line-based parsing
instead of recursive descent; I don't know whether that's the right
thing for asciidoc.  It's actually fairly easy to convert a grammar to
code blocks using tricks from MJD's _Higher Order Perl_.  Is it
necessary for the asciidoc grammar?

> 		my @lines = split(/^\* /m, $par);
> 		shift @lines;
> 		$conv->enumeration(\@lines, 'unnumbered');
> 	} elsif ($par =~ /^\[verse\]/) {
> 		$par =~ s/\[verse\] *\n?//;
> 		$conv->verse($par);
> 	} elsif ($par =~ /^(\t|  +)/s) {
> 		$par =~ s/^$1//mg;
> 		$par =~ s/^\+$//mg;
> 		$conv->indent($par);
> 	} elsif ($par =~ /^([^\n]*)::\n((\t|  +).*)$/s) {
> 		my ($first, $rest, $indent) = ($1, $2, $3);
> 		$rest =~ s/^\+$//mg;
> 		while ($rest =~ /^(.*?\n\n)--+\n(.*?\n)--+\n\n(.*)$/s) {
> 			my ($pre, $verb, $post) = ($1, $2, $3);
> 
> 			$pre =~ s/^(\t|$indent)//mg;
> 			if ($first ne '') {
> 				$conv->begin_item($first, $pre);
> 				$first = '';
> 			} else {
> 				$conv->normal($pre);
> 			}
> 
> 			$conv->verbatim($verb);
> 			$rest = $post;
> 		}
> 		$rest =~ s/^(\t|$indent)//mg;
> 		if ($first ne '') {
> 			$conv->begin_item($first, $rest);
> 		} else {
> 			$conv->normal($rest);
> 		}
> 		$conv->end_item();
> 	} elsif ($par =~ /^-+\n(.*\n)-+\n$/s) {
> 		$conv->verbatim($1);
> 	} else {
> 		$conv->normal($par);
> 	}
> 	$par = '';
> }
> 
> sub handle_file {
> 	my $in;
> 	open($in, '<' . $_[0]);
> 	while (<$in>) {
> 		if (/^=+$/) {
> 			if ($par ne '' && length($_) >= length($par)) {
> 				$conv->header($par);
> 				$par = '';
> 				next;
> 			}
> 		} elsif (/^-+$/) {
> 			if ($par ne '' && length($_) >= length($par)) {
> 				$conv->section($par);
> 				$par = '';
> 				next;
> 			}
> 		} elsif (/^~+$/) {
> 			if ($par ne '' && length($_) >= length($par)) {
> 				$conv->subsection($par);
> 				$par = '';
> 				next;
> 			}
> 		} elsif (/^\[\[(.*)\]\]$/) {
> 			handle_text();
> 			$conv->anchor($1);
> 			next;
> 		} elsif (/^$/) {
> 			if ($par =~ /^-+\n.*[^-]\n$/s) {
> 				# fallthru; is verbatim, but needs more.
> 			} elsif ($par =~ /::\n$/s) {
> 				# is item, but needs more.
> 				next;
> 			} else {
> 				handle_text();
> 				next;
> 			}
> 		} elsif (/^include::(.*)\[\]$/) {
> 			handle_text();
> 			handle_file($1);
> 			next;
> 		}
> 
> 		# convert "\--" to "--"
> 		s/\\--/--/g;
> 		# convert "\*" to "*"
> 		s/\\\*/*/g;
> 
> 		# handle gitlink:
> 		s/gitlink:([^\[ ]*)\[(\d+)\]/sprintf "%s",
> 			$conv->get_link($1, $2)/ge;
> 		# handle link:
> 		s/link:([^\[ ]*)\[(.+)\]/sprintf "%s",
> 			$conv->get_link($1, $2, 'external')/ge;

These REs suffer from LTS (Leaning Toothpick Syndrome).  Consider using
s{foo}{bar} and adding the 'x' modifier to space out groups.

> 
> 		$par .= $_;
> 	}
> 	close($in);
> 	handle_text();
> }
> 
> package man_page;
> 
> sub new {
> 	my ($class) = @_;
> 	my $self = {
> 		sep => '',
> 		links => [],
> #		generator => 'Home grown git txt2man converter'
> 		generator => 'DocBook XSL Stylesheets v1.71.1 <http://docbook.sf.net/>'
> 	};
> 	bless $self, $class;
> 	return $self;
> }
> 
> sub header {
> 	my ($self, $text) = @_;
> 	$text =~ s/-/\\-/g;
> 
> 	if ($self->{preamble_shown} == undef) {
> 		$title = $text;
> 		$title =~ s/\(\d+\)$//;
> 		print '.\"     Title: ' . $title
> 			. '.\"    Author: ' . "\n"
> 			. '.\" Generator: ' . $self->{generator} . "\n"
> 			. '.\"      Date: ' . $self->{date} . "\n"
> 			. '.\"    Manual: ' . $self->{manual} . "\n"
> 			. '.\"    Source: ' . $self->{git_version} . "\n"
> 			. '.\"' . "\n";
> 	}

I'd consider a HERE-doc, or multi-line qq{ } more readable than this.

> 
> 	$text =~ tr/a-z/A-Z/;
> 	my $suffix = "\"$self->{date}\" \"$self->{git_version}\""
> 		. " \"$self->{manual}\"";

Use qq{} when making strings with lots of embedded double quotes and
interpolation.

> 	$text =~ s/^(.*)\((\d+)\)$/.TH "\1" "\2" $suffix/;
> 	print $text;
> 
> 	if ($self->{preamble_shown} == undef) {
> 		print '.\" disable hyphenation' . "\n"
> 			. '.nh' . "\n"
> 			. '.\" disable justification (adjust text to left'
> 				. ' margin only)' . "\n"
> 			. '.ad l' . "\n";

Using commas rather than "." will safe you a concat when printing to
filehandles, but that's a very small nit to pick :)

> 		$self->{preamble_shown} = 1;
> 	}
> 
> 	$self->{last_op} = 'header';
> }
> 
> sub section {
> 	my ($self, $text) = @_;
> 
> 	$text =~ tr/a-z/A-Z/;
> 	$text =~ s/^(.*)$/.SH "\1"/;
> 
> 	print $text;
> 
> 	$self->{last_op} = 'section';
> }
> 
> sub subsection {
> 	my ($self, $text) = @_;
> 
> 	$text =~ s/^(.*)$/.SS "\1"/;
> 
> 	print $text;
> 
> 	$self->{last_op} = 'subsection';
> }
> 
> sub get_link {
> 	my ($self, $command, $section, $option) = @_;
> 
> 	if ($option eq 'external') {
> 		my $links = $self->{links};
> 		push(@$links, $command);
> 		$command =~ s/\.html$//;
> 		$command =~ s/-/ /g;
> 		push(@$links, $command);
> 		return '\fI' . $command . '\fR\&[1]';
> 	} else {
> 		return '\fB' . $command . '\fR(' . $section . ')';
> 	}
> }
> 
> sub common {
> 	my ($self, $text, $option) = @_;
> 
> 	# escape backslashes, but not in "\n", "\&" or "\fB"
> 	$text =~ s/\\(?!n|f[A-Z]|&)/\\\\/g;
> 	# escape "-"
> 	$text =~ s/-/\\-/g;
> 	# handle ...
> 	$text =~ s/(\.\.\.)/\\&\1/g;
> 	# remove double space after full stop or comma
> 	$text =~ s/([\.,])  /\1 /g;
> 
> 	if ($option ne 'no-markup') {
> 		# make 'italic'
> 		$text =~ s/'([^'\n]*)'/\\fI\1\\fR/g;
> 		# ignore `
> 		$text =~ s/`//g;
> 		# make *bold*
> 		$text =~ s/\*([^\*\n]*)\*/\\fB\1\\fR/g;
> 		# handle <<sections>
> 		$text =~ s/<<([^>]*)>>/the section called \\(lq\1\\(rq/g;

Hmm, that regex would not match for <<foo > bar>>, if you care you'd
need to write something like <<((?:[^>]+|>[^>])*)>>

> 	}
> 
> 	return $text;
> }
> 
> sub normal {
> 	my ($self, $text) = @_;
> 
> 	if ($text eq "") {
> 		return;
> 	}
> 
> 	$text = $self->common($text);
> 
> 	$text =~ s/ *\n(.)/ \1/g;
> 
> 	if ($self->{last_op} eq 'normal') {
> 		print "\n";
> 	}
> 
> 	print $text;
> 
> 	$self->{last_op} = 'normal';
> }
> 
> sub verse {
> 	my ($self, $text) = @_;
> 
> 	$text = $self->common($text);
> 	$text =~ s/^\t/        /mg;
> 
> 	print ".sp\n.RS 4\n.nf\n" . $text . ".fi\n.RE\n";
> 
> 	$self->{last_op} = 'verse';
> }
> 
> sub enumeration {
> 	my ($self, $text, $option) = @_;
> 
> 	my $counter = 0;
> 	foreach $line (@$text) {
> 		$counter++;
> 		print ".TP 4\n"
> 			. ($option eq 'unnumbered' ? '\(bu' : $counter . '.')
> 			. "\n"
> 			. $self->common($line);
> 	}
> 
> 	$self->{last_op} = 'enumeration';
> }
> 
> sub begin_item {
> 	my ($self, $item, $text) = @_;
> 
> 	$item = $self->common($item);
> 	$text = $self->common($text);
> 
> 	$text =~ s/([^\n]) *\n([^\n])/\1 \2/g;

"." is the same as [^\n] (without the 's' modifier).

> 
> 	print ".PP\n" . $item . "\n.RS 4\n" . $text;
> 
> 	$self->{last_op} = 'item'; 
> }
> 
> sub end_item {
> 	my ($self) = @_;
> 
> 	print ".RE\n";
> 
> 	$self->{last_op} = 'end_item';
> }
> 
> sub indent {
> 	my ($self, $text) = @_;
> 
> 	$text = $self->common($text, 'no-markup');
> 	$text =~ s/^\t/        /mg;
> 
> 	if ($self->{last_op} eq 'normal') {
> 		print "\n";
> 	}
> 
> 	print ".sp\n.RS 4\n.nf\n" . $text . ".fi\n.RE\n";
> 
> 	$self->{last_op} = 'indent';
> }
> 
> sub verbatim {
> 	my ($self, $text) = @_;
> 
> 	$text = $self->common($text, 'no-markup');
> 
> 	# convert tabs to spaces
> 	$text =~ s/^\t/        /mg;
> 	# remove trailing empty lines
> 	$text =~ s/\n\n*$/\n/;
> 
> 	if ($self->{last_op} eq 'normal') {
> 		print "\n";
> 	}
> 
> 	print ".sp\n.RS 4\n.nf\n.ft C\n" . $text . ".ft\n\n.fi\n.RE\n";
> 
> 	$self->{last_op} = 'verbatim';
> }
> 
> sub anchor {
> 	my ($self, $text) = @_;
> 
> 	$self->{last_op} = 'anchor';
> }
> 
> sub finish {
> 	my ($self) = @_;
> 	my $links = $self->{links};
> 
> 	if ($#$links >= 0) {
> 		print '.SH "REFERENCES"' . "\n";
> 		my $i = 1;
> 		while ($#$links >= 0) {

just use if (@$links) and while (@$links)

> 			my $ref = shift(@$links);
> 			$ref =~ s/-/\\-/g;
> 			my $label = shift(@$links);
> 			printf (".IP \"% 2d.\" 4\n%s\n.RS 4\n\\%%%s\n.RE\n",
> 				$i++, $label, $ref);
> 		}
> 	} else {
> 		print "\n";
> 	}
> }
> 
> package html_page;
> 
> sub new {
> 	my ($class) = @_;
> 	my $self = {};
> 	bless $self, $class;
> 	return $self;
> }
> 
> -- snap --
> 
> Ciao,
> Dscho
> 
> P.S.: I need to catch some Zs, and do some real work, so do not be 
> surprised if I do not respond within the next 24 hours.
> -
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: WIP: asciidoc replacement

[Johannes Schindelin]

Hi,

On Wed, 3 Oct 2007, Sam Vilain wrote:

> Johannes Schindelin wrote:
> 
> > I do not want to depend on more than necessary in msysGit, and 
> > therefore I started to write an asciidoc replacement.
> 
> It's pretty good, I certainly wouldn't have trouble reading or 
> maintaining it, but I'll give you suggestions anyway.

Thank you very much!  (On both accounts...)

> nice work, replacing a massive XML/XSL/etc stack with a small Perl 
> script ;-)

Uhm... It is less capable, though...

> > -- snip --
> > #!/usr/bin/perl
> 
> Add -w for warnings, also use strict;

<dumb>What does "use strict;" imply?</dumb>

> > sub handle_text {
> 
> this function acts on globals; make them explicit arguments to the 
> function.

Actually, it resets the global $par.  Should I rather make it a class?

> > 	if ($par =~ /^\. /s) {
> > 		my @lines = split(/^\. /m, $par);
> > 		shift @lines;
> > 		$conv->enumeration(\@lines);
> > 	} elsif ($par =~ /^\* /s) {
> 
> uncuddle your elsif's;

I'm sorry... What do you mean?

> also consider making this a "tabular ternary" with the actions in 
> separate functions.
> 
> ie
> 
> $result = ( $par =~ /^\. /s      ? $conv->do_enum($par)    :
>             $par =~ /^\[verse\]/ ? $conv->do_verse($par)  :
>             ... )

I do not like that way... is it Perl standard to code like that?

> However I have a suspicion that your script is doing line-based parsing 
> instead of recursive descent; I don't know whether that's the right 
> thing for asciidoc.  It's actually fairly easy to convert a grammar to 
> code blocks using tricks from MJD's _Higher Order Perl_.  Is it 
> necessary for the asciidoc grammar?

I wanted to keep it simple.  So I'll try to stay away from any fancy 
grammar parsing, and stay with "read lines until you have something to 
process".

> > 		# handle gitlink:
> > 		s/gitlink:([^\[ ]*)\[(\d+)\]/sprintf "%s",
> > 			$conv->get_link($1, $2)/ge;
> > 		# handle link:
> > 		s/link:([^\[ ]*)\[(.+)\]/sprintf "%s",
> > 			$conv->get_link($1, $2, 'external')/ge;
> 
> These REs suffer from LTS (Leaning Toothpick Syndrome).  Consider using 
> s{foo}{bar} and adding the 'x' modifier to space out groups.

I guess you mean the forward slash.  Alas, that's what I'm used to, and 
I'd rather not change it unless forced to... lest I stop understanding my 
own code!

(Besides, I did not find _any_ example showing why "x" should be useful.)

> > 	if ($self->{preamble_shown} == undef) {
> > 		$title = $text;
> > 		$title =~ s/\(\d+\)$//;
> > 		print '.\"     Title: ' . $title
> > 			. '.\"    Author: ' . "\n"
> > 			. '.\" Generator: ' . $self->{generator} . "\n"
> > 			. '.\"      Date: ' . $self->{date} . "\n"
> > 			. '.\"    Manual: ' . $self->{manual} . "\n"
> > 			. '.\"    Source: ' . $self->{git_version} . "\n"
> > 			. '.\"' . "\n";
> > 	}
> 
> I'd consider a HERE-doc, or multi-line qq{ } more readable than this.

Can you give me an example of a HERE-doc?  (What I tried to avoid is 
having ugly indentation-breaking tlobs.)

> > 	$text =~ tr/a-z/A-Z/;
> > 	my $suffix = "\"$self->{date}\" \"$self->{git_version}\""
> > 		. " \"$self->{manual}\"";
> 
> Use qq{} when making strings with lots of embedded double quotes and
> interpolation.

I'll try to find something about qq{} in the docs.

> > 	$text =~ s/^(.*)\((\d+)\)$/.TH "\1" "\2" $suffix/;
> > 	print $text;
> > 
> > 	if ($self->{preamble_shown} == undef) {
> > 		print '.\" disable hyphenation' . "\n"
> > 			. '.nh' . "\n"
> > 			. '.\" disable justification (adjust text to left'
> > 				. ' margin only)' . "\n"
> > 			. '.ad l' . "\n";
> 
> Using commas rather than "." will safe you a concat when printing to
> filehandles, but that's a very small nit to pick :)

Does that also work with older perl?  IIRC there was some strange problem 
with my perl when lots of code in git.git was changed to using commata.

> > 		# handle <<sections>
> > 		$text =~ s/<<([^>]*)>>/the section called \\(lq\1\\(rq/g;
> 
> Hmm, that regex would not match for <<foo > bar>>, if you care you'd 
> need to write something like <<((?:[^>]+|>[^>])*)>>

I'd rather leave it as is -- this script is not meant to grok all kind of 
sh*t.  It is meant to make translating the docs as fast and uncumbersome 
as possible.  Which will involve making the documentation more consistent 
(in and of itself something I rather like).

So unless there comes a compelling reason, I'd rather leave it.

> > sub begin_item {
> > 	my ($self, $item, $text) = @_;
> > 
> > 	$item = $self->common($item);
> > 	$text = $self->common($text);
> > 
> > 	$text =~ s/([^\n]) *\n([^\n])/\1 \2/g;
> 
> "." is the same as [^\n] (without the 's' modifier).

But I need the (implicit) 's' modifier, otherwise the "\n" in the middle 
is not interpreted correctly.  This regsub is meant to unwrap the 
paragraph and put it into a very long line (but leaving \n\n alone).

> > sub finish {
> > 	my ($self) = @_;
> > 	my $links = $self->{links};
> > 
> > 	if ($#$links >= 0) {
> > 		print '.SH "REFERENCES"' . "\n";
> > 		my $i = 1;
> > 		while ($#$links >= 0) {
> 
> just use if (@$links) and while (@$links)

Thanks.  I hoped there would be something like this.

Another thing: if I want to add some documentation, what would be the 
common way to do it?  =pod...=cut?

Thank you for all your tips!

Ciao,
Dscho

Re: WIP: asciidoc replacement

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> So here it is: a perl script that does a good job on many .txt files in 
> Documentation/, although for some it deviates from "make man"'s output, 
> and for others it is outright broken.  It is meant to be run in 
> Documentation/.
>
> My intention is not to fix the script for all cases, but to make patches 
> to Documentation/*.txt themselves, so that they are more consistent (and 
> incidentally nicer to the script).

How you spend your time is up to you, but I need to wonder...

 - Is "man" format important for msysGit aka Windows
   environment?  I had an impression that their helpfile format
   were closer to "html" output.

 - Does it make sense in the longer term for us to maintain
   in-house documentation tools?  Can we afford it?

It appears that we heard about breakages for every minor docbook
updates, and it is really appealing if we do not have to rely on
xsl toolchain for manpage generation.  But if patching the text
means making it compatible with the in-house script _and_
incompatible with AsciiDoc, hmmm...

Re: WIP: asciidoc replacement

[Jeff King]

On Wed, Oct 03, 2007 at 05:23:35AM +0100, Johannes Schindelin wrote:

> > > #!/usr/bin/perl
> > 
> > Add -w for warnings, also use strict;
> 
> <dumb>What does "use strict;" imply?</dumb>

Try "perldoc strict" for details.

> > > 	if ($par =~ /^\. /s) {
> > > 		my @lines = split(/^\. /m, $par);
> > > 		shift @lines;
> > > 		$conv->enumeration(\@lines);
> > > 	} elsif ($par =~ /^\* /s) {
> > 
> > uncuddle your elsif's;
> 
> I'm sorry... What do you mean?

I think he means reformatting to

  if (condition) {
  }
  elsif (condition) {
  }

> > $result = ( $par =~ /^\. /s      ? $conv->do_enum($par)    :
> >             $par =~ /^\[verse\]/ ? $conv->do_verse($par)  :
> >             ... )
> 
> I do not like that way... is it Perl standard to code like that?

It's quite common if you have a dispatch function, but obviously not
required.

> > > 		$title =~ s/\(\d+\)$//;
> > > 		print '.\"     Title: ' . $title
> > > 			. '.\"    Author: ' . "\n"
> > > 			. '.\" Generator: ' . $self->{generator} . "\n"
> > > 			. '.\"      Date: ' . $self->{date} . "\n"
> > > 			. '.\"    Manual: ' . $self->{manual} . "\n"
> > > 			. '.\"    Source: ' . $self->{git_version} . "\n"
> > > 			. '.\"' . "\n";
> > > 	}
> > 
> > I'd consider a HERE-doc, or multi-line qq{ } more readable than this.
> 
> Can you give me an example of a HERE-doc?  (What I tried to avoid is 
> having ugly indentation-breaking tlobs.)

print <<EOF;
foo
EOF

HERE-docs necessarily break indentation unless you strip it out manually
(which is inefficient and ugly).

But two things that might make that look better are using qq// (to avoid
having to escape quotes) and interpolating the variables:

  . qq/." Generator: $self->{generator}\n/

> I'll try to find something about qq{} in the docs.

It's in perlop, but it's basically a fancy way of double-quoting, except
that you get to choose the delimiter.

> > > 	$text =~ s/([^\n]) *\n([^\n])/\1 \2/g;
> > 
> > "." is the same as [^\n] (without the 's' modifier).
> 
> But I need the (implicit) 's' modifier, otherwise the "\n" in the middle 
> is not interpreted correctly.  This regsub is meant to unwrap the 
> paragraph and put it into a very long line (but leaving \n\n alone).

I think you might be confused about how the 's' modifier works. You are
not using it, so '.' is the same as '[^\n]'. Perl will always match a
newline if it's in your regex. If you specify 'm', then it will also
allow '^' and '$' to match at line boundaries (instead of just at the
beginning and end of the string).

-Peff

Re: WIP: asciidoc replacement

[Wincent Colaiuta]

El 3/10/2007, a las 6:48, Junio C Hamano escribió:

>  - Does it make sense in the longer term for us to maintain
>    in-house documentation tools?  Can we afford it?
>
> It appears that we heard about breakages for every minor docbook
> updates, and it is really appealing if we do not have to rely on
> xsl toolchain for manpage generation.

Indeed, especially seeing as asciidoc and the xsl toolchain are the  
trickiest build dependencies to install. If all that could be  
replaced by a single simple script like this one then that would be  
awesome, and probably more maintainable in the long run seeing as it  
would eliminate those intermittent breakages caused by changes in  
third-party tools.

Cheers,
Wincent

Re: WIP: asciidoc replacement

[Wincent Colaiuta]

El 3/10/2007, a las 3:56, Sam Vilain escribió:

> However I have a suspicion that your script is doing line-based  
> parsing
> instead of recursive descent; I don't know whether that's the right
> thing for asciidoc.  It's actually fairly easy to convert a grammar to
> code blocks using tricks from MJD's _Higher Order Perl_.  Is it
> necessary for the asciidoc grammar?

I haven't looked at all of the asciidoc source for the Git  
documentation but I suspect that almost all of it is entirely  
regular, and if there is any nesting in it is is probably of a  
limited scope (ie. sections, subsections, subsubsections in the user  
manual) and so you can avoid the complexity of a full recursive  
descent parser. I'd also expect Johannes' approach to be faster (is  
it faster than the existing tool chain? I would expect so).

Cheers,
Wincent

Re: WIP: asciidoc replacement

[David Kastrup]

Wincent Colaiuta <win@wincent.com> writes:

> El 3/10/2007, a las 6:48, Junio C Hamano escribió:
>
>>  - Does it make sense in the longer term for us to maintain
>>    in-house documentation tools?  Can we afford it?
>>
>> It appears that we heard about breakages for every minor docbook
>> updates, and it is really appealing if we do not have to rely on
>> xsl toolchain for manpage generation.
>
> Indeed, especially seeing as asciidoc and the xsl toolchain are the  
> trickiest build dependencies to install. If all that could be  
> replaced by a single simple script like this one then that would be  
> awesome, and probably more maintainable in the long run seeing as it  
> would eliminate those intermittent breakages caused by changes in  
> third-party tools.

What with output in print, HTML, info?  The advantage of a toolchain
in that it is flexible.  I am the first to admit that getting the
AsciiDoc/Docbook/Docbook2X toolchain to get it to do what one wants to
is like baking cake in a lightless kitchen.  But it is not like we go
through that pain without any reason.

Personally, I think it might make sense to just step away from the
AsciiDoc documentation to Docbook: plain text (without cutified
formatting control like in AsciiDoc) can be generated _from_ Docbook.

And AsciiDoc keeps us from documenting the formatting: Docbook, which
is a source format and looks it, can easily admit comments that won't
get through to the formatted versions.  Sure, the first version would
likely be generated with AsciiDoc and thus basically uncommented.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: WIP: asciidoc replacement

[Wincent Colaiuta]

El 3/10/2007, a las 10:12, David Kastrup escribió:

> What with output in print, HTML, info?

Yes, that's still a problem...

> Personally, I think it might make sense to just step away from the
> AsciiDoc documentation to Docbook: plain text (without cutified
> formatting control like in AsciiDoc) can be generated _from_ Docbook.

Yes, but editing DocBook (XML) is relatively painful compared to  
editing plain text. You either have to rely on a bloated XML- 
validating editor or instead ask your doc authors to manually write  
valid XML (and I totally agree with Terrence Parr that, "XML makes a  
lousy human interface
"; see <http://www.ibm.com/developerworks/xml/library/x-sbxml.html>  
for his full take).

I know that Linus has argued for AsciiDoc because the source *is* the  
plain text documentation and is therefore easily readable, but for me  
the real benefit lies in the fact that *because* the source is plain  
text it is easily edited (ie. that the source is easily *writeable*),  
and things like documentation patches are very neat with AsciiDoc.

Cheers,
Wincent

Re: WIP: asciidoc replacement

[David Kastrup]

Wincent Colaiuta <win@wincent.com> writes:

> El 3/10/2007, a las 10:12, David Kastrup escribió:
>
>> What with output in print, HTML, info?
>
> Yes, that's still a problem...
>
>> Personally, I think it might make sense to just step away from the
>> AsciiDoc documentation to Docbook: plain text (without cutified
>> formatting control like in AsciiDoc) can be generated _from_ Docbook.
>
> Yes, but editing DocBook (XML) is relatively painful compared to
> editing plain text.

The problem is that we are not editing plain text, but Docbook source
masquerading as plain text.

> You either have to rely on a bloated XML- validating editor or
> instead ask your doc authors to manually write valid XML (and I
> totally agree with Terrence Parr that, "XML makes a lousy human
> interface "; see
> <http://www.ibm.com/developerworks/xml/library/x-sbxml.html> for his
> full take).
>
> I know that Linus has argued for AsciiDoc because the source *is*
> the plain text documentation and is therefore easily readable, but
> for me the real benefit lies in the fact that *because* the source
> is plain text it is easily edited (ie. that the source is easily
> *writeable*), and things like documentation patches are very neat
> with AsciiDoc.

But it is not all _all_ easily writeable the moment you try to do
something with _structural_ impact.  In fact, it is pretty much
impossible for anybody except wizards to do that.  And when the
wizards do it, they can't actually document what they have been doing
since that would mean cluttering the purported "plain text
documentation" with formatting comments.

Maybe it would help to rename *.txt to *.asciidoc and generate *.txt
in future.  That would at least make it possible to document stuff in
the AsciiDoc source, and also would make it possible to add indexing
info and other stuff without cluttering up the plain text use case.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: WIP: asciidoc replacement

[Sam Ravnborg]

Hi David.
> 
> But it is not all _all_ easily writeable the moment you try to do
> something with _structural_ impact.  In fact, it is pretty much
> impossible for anybody except wizards to do that.  And when the
> wizards do it, they can't actually document what they have been doing
> since that would mean cluttering the purported "plain text
> documentation" with formatting comments.
When I converted part of the kbuild documentation to ascii doc
is was a strightforward excercise.
The txt file was equally readable before and after,
and the generated HTML looks OK.

Up until 3.6 in following link were properly converted:
http://www.ravnborg.org/kbuild/makefiles.html

But I did not convince asciidoc to generate an index - is it this
you refer to as magic?

Using the kbuild doc as my background I would say that with
or without asciidoc there were a requirment for a consistent
style used throughout the file.
And that style was and are simple to use just by copying
relevant examples.

What asciidoc gave me was a simple syntax chack of what I did.
I found wrong references when playing with asciidoc as
one example.

If the asciidoc replacement prove a success for git I would
consider suggesting it for the kernel too.
It would be good to generate some nicer loking online documentation
and here an asii tool like thing would be a help.

	Sam

Re: WIP: asciidoc replacement

[Junio C Hamano]

Wincent Colaiuta <win@wincent.com> writes:

> Yes, but editing DocBook (XML) is relatively painful compared to  
> editing plain text. You either have to rely on a bloated XML- 
> validating editor or instead ask your doc authors to manually write  
> valid XML (and I totally agree with Terrence Parr that, "XML makes a  
> lousy human interface
> "; see <http://www.ibm.com/developerworks/xml/library/x-sbxml.html>  
> for his full take).
>
> I know that Linus has argued for AsciiDoc because the source *is* the  
> plain text documentation and is therefore easily readable, but for me  
> the real benefit lies in the fact that *because* the source is plain  
> text it is easily edited (ie. that the source is easily *writeable*),  
> and things like documentation patches are very neat with AsciiDoc.

To give credits to where they are due, most of the structure of
the initial documentation was done during the first week of May
2005 by David Greaves while Linus was vacationing, and the first
person who brought up AsciiDoc was Bert Hubert.

    http://article.gmane.org/gmane.comp.version-control.git/2323

One thing Linus had to say about the issue from early on, and I
still agree with, is the last paragraph in:

    http://article.gmane.org/gmane.comp.version-control.git/2298

There was another thread in the recent past

    http://article.gmane.org/gmane.comp.version-control.git/55059

I've seen markdown used elsewhere, and I regularly read pod.
These are both used by other people (I _care_ about good
external support and user community), are readable as straight
text (I personally care more about this than prettyprinted
output), and can be alternative candidates if we consider
switching.

How good are HTML and manpage output support from these (or
other candidates) formats these days?  Output to help page
format Windows folks use (I am assuming Mac people are happy as
long as man is available) would be a definite plus.

Another alternative could be to build on AsciiDoc to get manpage
and html output without relying too heavily on docbook
toolchain, and considering the fact that we still seem to be one
of the most important customers listed on its home page, we
might be able to get help from Stuart himself to improve the
situation (I am assuming that mingwgit folks do not have problem
with Python, but I may be mistaken).

In short, although I do appreciate Johannes's and Sam's attempt,
I would really prefer to see us pick some externally maintained
alternative, instead of inventing a homebrew system that we need
to maintain ourselves.  It is rumored that git has much higher
developer count vs loc count ratio than many other open source
projects, doing the documentation format is not part of our
project, and I'd rather see them spend time working on git, not
building and maintaining AsciiDoc lookalike.

Re: [msysGit] Re: WIP: asciidoc replacement

[Johannes Schindelin]

Hi,

On Tue, 2 Oct 2007, Junio C Hamano wrote:

> 
> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
> 
> > So here it is: a perl script that does a good job on many .txt files 
> > in Documentation/, although for some it deviates from "make man"'s 
> > output, and for others it is outright broken.  It is meant to be run 
> > in Documentation/.
> >
> > My intention is not to fix the script for all cases, but to make 
> > patches to Documentation/*.txt themselves, so that they are more 
> > consistent (and incidentally nicer to the script).
> 
> How you spend your time is up to you, but I need to wonder...
> 
>  - Is "man" format important for msysGit aka Windows
>    environment?  I had an impression that their helpfile format
>    were closer to "html" output.

I wanted something that can output both "man" and "html" output (and if 
some suck^Wlos^Wtexi-fan wants to provide it, also a "texi" or even "info" 
backend).

IMHO "man" needs a stricter framework in place, so I went with that.

>  - Does it make sense in the longer term for us to maintain
>    in-house documentation tools?  Can we afford it?

In the long run, I expect only few bugs (and I will try hard to squash 
them when they crop up, _and_ make this beast more maintainable whenever 
somebody has an idea how to do that).

However, it should definitely help keeping the docs clean, as now nobody 
has an excuse to test doc changes a la "I do not have asciidoc, so I do 
not know if it works, so please test".

> It appears that we heard about breakages for every minor docbook 
> updates, and it is really appealing if we do not have to rely on xsl 
> toolchain for manpage generation.

Exactly.

> But if patching the text means making it compatible with the in-house 
> script _and_ incompatible with AsciiDoc, hmmm...

No, I do not want it _incompatible_.  I want it _stricter_.  For example, 
you can do this in asciidoc:


    This is some paragraph that is indented,
    but the funny thing is:

This paragraph:
---------------

	is indented all the same!


So one thing I absolutely detest here is that you are free to use one, 
two, three or more spaces, or tabs, and asciidoc does the DWIMery of 
handling them the same.  But _not_ if there was any indentation before 
that with _less_ spaces and/or tabs!

Therefore I'd like to enforce strict rules here: Tab it is.  One tab per 
indentation level.  No spaces, no ambiguities.

Ciao,
Dscho

Re: [msysGit] Re: WIP: asciidoc replacement

[David Kastrup]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> On Tue, 2 Oct 2007, Junio C Hamano wrote:
>
>> 
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> 
>> > So here it is: a perl script that does a good job on many .txt files 
>> > in Documentation/, although for some it deviates from "make man"'s 
>> > output, and for others it is outright broken.  It is meant to be run 
>> > in Documentation/.
>> >
>> > My intention is not to fix the script for all cases, but to make 
>> > patches to Documentation/*.txt themselves, so that they are more 
>> > consistent (and incidentally nicer to the script).
>> 
>> How you spend your time is up to you, but I need to wonder...
>> 
>>  - Is "man" format important for msysGit aka Windows
>>    environment?  I had an impression that their helpfile format
>>    were closer to "html" output.
>
> I wanted something that can output both "man" and "html" output (and
> if some suck^Wlos^Wtexi-fan wants to provide it, also a "texi" or
> even "info" backend).

And you have not even talked about print.  The thing is that high
quality output is a lot of work (including design and design
decisions) for _every_ _single_ backend.  If the only target were
"man" and text, then one would be better off writing as groff source
and be done.

We really can't afford another time sink.  That nobody can actually
solve structural problems ("how do we include the man pages as an
appendix in the user manual?") also implies a waste of time, but at
some point of time people just throw up their hands and give up in
disgust.

You propose a larger time sink where people won't be forced to give
up.  But we really don't want to waste time reinventing the wheel.  It
would be better spent communicating with the wheelbuilders.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: WIP: asciidoc replacement

[J. Bruce Fields]

On Wed, Oct 03, 2007 at 12:25:44PM +0200, David Kastrup wrote:
> Wincent Colaiuta <win@wincent.com> writes:
> 
> > El 3/10/2007, a las 10:12, David Kastrup escribió:
> >
> >> What with output in print, HTML, info?
> >
> > Yes, that's still a problem...
> >
> >> Personally, I think it might make sense to just step away from the
> >> AsciiDoc documentation to Docbook: plain text (without cutified
> >> formatting control like in AsciiDoc) can be generated _from_ Docbook.
> >
> > Yes, but editing DocBook (XML) is relatively painful compared to
> > editing plain text.
> 
> The problem is that we are not editing plain text, but Docbook source
> masquerading as plain text.

I do a fair amount of editing of the asciidoc source, but 99% of it is
done by just blind imitation of what's already there.  I've never
learned docbook (I've barely learned asciidoc, to be honest), and with a
few (now forgotten) exceptions haven't tried to understand how the
toolchain works.

Maybe my experience would be the same with Docbook--I have no idea,
never having worked with it--but if you're suggesting that knowledge of
Docbook is a prerequisite for working with asciidoc, that certainly
hasn't been my experience.

> But it is not all _all_ easily writeable the moment you try to do
> something with _structural_ impact.  In fact, it is pretty much
> impossible for anybody except wizards to do that.  And when the
> wizards do it, they can't actually document what they have been doing
> since that would mean cluttering the purported "plain text
> documentation" with formatting comments.

I'm not sure what you're talking about here.  Example?

--b.

Re: WIP: asciidoc replacement

[J. Bruce Fields]

On Wed, Oct 03, 2007 at 05:23:35AM +0100, Johannes Schindelin wrote:
> On Wed, 3 Oct 2007, Sam Vilain wrote:
> > Johannes Schindelin wrote:
> > 
> > > I do not want to depend on more than necessary in msysGit, and 
> > > therefore I started to write an asciidoc replacement.
> > 
> > It's pretty good, I certainly wouldn't have trouble reading or 
> > maintaining it, but I'll give you suggestions anyway.
> 
> Thank you very much!  (On both accounts...)
> 
> > nice work, replacing a massive XML/XSL/etc stack with a small Perl 
> > script ;-)
> 
> Uhm... It is less capable, though...

By the way, without having looked at your script to see it does, a
couple things I know of that the user manual relies on that other docs
may not as much are automatic table of contents generation, and cross
references.

I'd be similarly inclined to work on improving asciidoc rather than
inventing something new, but I guess it's at least interesting to see
how far your approach gets us.

--b.

Re: WIP: asciidoc replacement

[David Kastrup]

"J. Bruce Fields" <bfields@fieldses.org> writes:

> On Wed, Oct 03, 2007 at 12:25:44PM +0200, David Kastrup wrote:
>
>> The problem is that we are not editing plain text, but Docbook
>> source masquerading as plain text.
>
> I do a fair amount of editing of the asciidoc source, but 99% of it
> is done by just blind imitation of what's already there.

But not everything is already there, and when something surprising
happens, there is little chance to see how it came about.

> Maybe my experience would be the same with Docbook--I have no idea,
> never having worked with it--but if you're suggesting that knowledge
> of Docbook is a prerequisite for working with asciidoc, that
> certainly hasn't been my experience.

"making use of" and "working with" are two different things.

>> But it is not all _all_ easily writeable the moment you try to do
>> something with _structural_ impact.  In fact, it is pretty much
>> impossible for anybody except wizards to do that.  And when the
>> wizards do it, they can't actually document what they have been
>> doing since that would mean cluttering the purported "plain text
>> documentation" with formatting comments.
>
> I'm not sure what you're talking about here.  Example?

Try including the manual pages as a (properly linked when man pages
are referenced) appendix in the user manual, so that the printed form
(or PDF) of the user manual is a single coherent document with all
information inside.  That's what I tried for about a week, digging
into the various available (and unavailable) documentation and then
postponing the project indefinitely because it both exceeded my
current capability as well as demonstrating that there was no
reasonably outlined path for acquiring the necessary skills.

In Texinfo, this takes few commands, all of which are well-documented
and in a reasonable place in the Texinfo manual (which is all you need
to consult in order to write Texinfo documents).

But with git's AsciiDoc information, not only is the required
information scattered through half a dozen of different manuals all
describing completely different systems, but the necessary other
documentation is, at best, only mentioned in passing in every single
relevant document.  So while you may know where you want to start and
end your journey, there is nothing which would tell you how to get
from start to end.  You have to randomly pick your road until you may
or may not find something closer to the end.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: WIP: asciidoc replacement

[Sam Ravnborg]

Hi Junio.
> 
> In short, although I do appreciate Johannes's and Sam's attempt,
> I would really prefer to see us pick some externally maintained
> alternative, instead of inventing a homebrew system that we need
> to maintain ourselves.  It is rumored that git has much higher
> developer count vs loc count ratio than many other open source
> projects, doing the documentation format is not part of our
> project, and I'd rather see them spend time working on git, not
> building and maintaining AsciiDoc lookalike.

For the kernel I would like to see a tool that does:

o Based on nicely formatted ascii/utf-8 be able to:
 - Generate good looking and easy to read HTML
 - Possible generate other output formats too

And if the kernel folks do not like it then at least the possibility
to run this on kbuild documentation locally so I can generate nice
html docs for that part.

For a kernel integrated tool the dependencies shall be minimal which
is where asciidoc fails today. If asciidoc people could address this
issue for the simpler output format then the tool IMO would
have a much stronger position.

	Sam

Re: WIP: asciidoc replacement

[Johannes Schindelin]

Hi,

On Wed, 3 Oct 2007, Sam Ravnborg wrote:

> For a kernel integrated tool the dependencies shall be minimal which is 
> where asciidoc fails today.

Is perl not too much already?

Ciao,
Dscho

Re: WIP: asciidoc replacement

[Sam Ravnborg]

On Wed, Oct 03, 2007 at 07:57:48PM +0100, Johannes Schindelin wrote:
> Hi,
> 
> On Wed, 3 Oct 2007, Sam Ravnborg wrote:
> 
> > For a kernel integrated tool the dependencies shall be minimal which is 
> > where asciidoc fails today.
> 
> Is perl not too much already?

No - perl has not proved to be a problem.
But like git we have had a smaller share of docbook
incompatibilities.

We even require perl for a regular kernel build
for some arch's these days.

	Sam

Re: WIP: asciidoc replacement

[Sam Vilain]

Johannes,

Given other people have answered some points, I'll answer the rest.

Johannes Schindelin wrote:
>>> -- snip --
>>> #!/usr/bin/perl
>> Add -w for warnings, also use strict;
> 
> <dumb>What does "use strict;" imply?</dumb>

Three things;

1. variables must be declared with 'my', 'our' or 'use var' before they
are used, to catch typos

2. when subroutine calls are found, they are checked to exist otherwise
they throw a compile-time error

3. force all dereferences to follow real references and not allow symbol
table access (don't worry about that ;-))


>>> sub handle_text {
>> this function acts on globals; make them explicit arguments to the 
>> function.
> 
> Actually, it resets the global $par.  Should I rather make it a class?

Well, just channelling Dijkstra really.  Functions should take all their
input as formal arguments rather than globals.

ie

  sub handle_text {
      my $par = shift;
  }

If it really should be a global, it is perhaps best declared up front
with "our" or "use vars".  "use strict" will force you to do one of these.

>> also consider making this a "tabular ternary" with the actions in 
>> separate functions.
>>
>> ie
>>
>> $result = ( $par =~ /^\. /s      ? $conv->do_enum($par)    :
>>             $par =~ /^\[verse\]/ ? $conv->do_verse($par)  :
>>             ... )
> 
> I do not like that way... is it Perl standard to code like that?

It's in Perl Best Practices, but these are always suggestions and not
hard and fast rules.  It just means that you have a big table of regex
-> function that you can quickly check rather than looking at a lot of
spaced out 'elsif's

>>> 		s/gitlink:([^\[ ]*)\[(\d+)\]/sprintf "%s",
>>> 			$conv->get_link($1, $2)/ge;
>>> 		# handle link:
>>> 		s/link:([^\[ ]*)\[(.+)\]/sprintf "%s",
>>> 			$conv->get_link($1, $2, 'external')/ge;
>> These REs suffer from LTS (Leaning Toothpick Syndrome).  Consider using 
>> s{foo}{bar} and adding the 'x' modifier to space out groups.
> 
> I guess you mean the forward slash.  Alas, that's what I'm used to, and 
> I'd rather not change it unless forced to... lest I stop understanding my 
> own code!
> 
> (Besides, I did not find _any_ example showing why "x" should be useful.)

Before:

s/link:([^\[ ]*)\[(.+)\]/sprintf "%s",
 			$conv->get_link($1, $2, 'external')/ge;

After:

s{ link: ([^\[\040]*) \[(.+)\] }
 { sprintf "%s", $conv->get_link($1, $2, 'external') }gex;

>>> 	if ($self->{preamble_shown} == undef) {
>>> 		print '.\" disable hyphenation' . "\n"
>>> 			. '.nh' . "\n"
>>> 			. '.\" disable justification (adjust text to left'
>>> 				. ' margin only)' . "\n"
>>> 			. '.ad l' . "\n";
>> Using commas rather than "." will safe you a concat when printing to
>> filehandles, but that's a very small nit to pick :)
> 
> Does that also work with older perl?  IIRC there was some strange problem 
> with my perl when lots of code in git.git was changed to using commata.

That should go back all the way to perl 4, if not earlier.  If you're
assigning to a scalar, then you need to use concat.  But very minor.

>> Hmm, that regex would not match for <<foo > bar>>, if you care you'd 
>> need to write something like <<((?:[^>]+|>[^>])*)>>
> 
> I'd rather leave it as is -- this script is not meant to grok all kind of 
> sh*t.  It is meant to make translating the docs as fast and uncumbersome 
> as possible.  Which will involve making the documentation more consistent 
> (in and of itself something I rather like).
> 
> So unless there comes a compelling reason, I'd rather leave it.

Sure, KISS.

> Another thing: if I want to add some documentation, what would be the 
> common way to do it?  =pod...=cut?

That's right.  If you're an emacs user, in cperl-mode with abbrevs on
you type '=head1' and you'll get:

=head1 NAME

scriptname

=head1 SYNOPSIS

=head1 DESCRIPTION

=cut

See perlpod(3) for more!

Sam.

Re: WIP: asciidoc replacement

[Martin Langhoff]

Junio,

great summary!

On 10/3/07, Junio C Hamano <gitster@pobox.com> wrote:
> One thing Linus had to say about the issue from early on, and I
> still agree with, is the last paragraph in:
>
>     http://article.gmane.org/gmane.comp.version-control.git/2298

I'm in complete agreement here...

...
> I've seen markdown used elsewhere, and I regularly read pod.
...
> How good are HTML and manpage output support from these (or
> other candidates) formats these days?  Output to help page
> format Windows folks use (I am assuming Mac people are happy as
> long as man is available) would be a definite plus.

And something that leads to PDF (perhaps via latex).

The problem I see here -- and that I've bumped into several times in
other projects -- is that readable & easy to edit text formats for the
source are key, and those can do most of what we need for
man/info/html outputs. But documentation formats that produce high
quality print output usually need  arcane formats _and_ a
high-maintenance toolchain.

With AsciiDoc we've managed to avoid the arcane format, but we are
still laden with a horrid toolchain. In that light, I actually like
what Johannes is doing, even though it's a timesink.

Do the other text based alternatives these days have a workable high
quality PDF/latex output format without pulling in brittle
dependencies like XSLT?

cheers


martin

Re: WIP: asciidoc replacement

[Johannes Schindelin]

Hi,

On Thu, 4 Oct 2007, Sam Vilain wrote:

> Given other people have answered some points, I'll answer the rest.

Thanks for your patient explanations!

Ciao,
Dscho

Re: WIP: asciidoc replacement

[David Kastrup]

"Martin Langhoff" <martin.langhoff@gmail.com> writes:

> With AsciiDoc we've managed to avoid the arcane format, but we are
> still laden with a horrid toolchain.

Let's put this somewhat into perspective: the toolchain is horrid with
regard to the complexity and documentation (well, AsciiDoc
documentation itself is quite thorough and reasonably organized, but
it does not buy you much without learning Docbook, and learning
Docbook is such a chore that people would rather use AsciiDoc in order
to avoid it), not horrid regarding the usability or flexibility of the
results.

If we had a few people specializing in Docbook/AsciiDoc/XSLT available
constantly on the team, we could probably get along fine.

> In that light, I actually like what Johannes is doing, even though
> it's a timesink.

The problem is not just that it is a timesink now, but that it will
remain a timesink.  There is a reason that we don't have so many
formats around with multiple high-quality backends.

> Do the other text based alternatives these days have a workable high
> quality PDF/latex output format without pulling in brittle
> dependencies like XSLT?

Texinfo produces good info and plain text, tolerable PDF and HTML and
not-quite-usable Docbook.  The output for HTML from the current git
documentation toolchain certainly looks better.  I don't see how to
generate PDF right now, but there must be a way (xmlto complains about
not seeing passivetex though I have it installed in TeXlive).

The source is uglier than AsciiDoc, but then there is no cleverly
disguised information in it: every formatting detail is quite out in
the open.  The same is true for Docbook, but Docbook really eats the
cake, platter and all concerning unreadability of the input.  On the
other hand, there are more special-purpose editors that know how to
deal with Docbook/XML than there are for Texinfo.

Texinfo gives a reasonable subset of Linus-thinkalikes the cooties,
and that pretty much rules it out: we need a format that people are
willing to write in.

There are not really many options for versatile formats, I am afraid.

-- 
David Kastrup, Kriemhildstr. 15, 44793 Bochum

Re: WIP: asciidoc replacement

[Martin Langhoff]

On 10/5/07, David Kastrup <dak@gnu.org> wrote:
> "Martin Langhoff" <martin.langhoff@gmail.com> writes:
>
> > With AsciiDoc we've managed to avoid the arcane format, but we are
> > still laden with a horrid toolchain.
>
> Let's put this somewhat into perspective: the toolchain is horrid with
> regard to the complexity and documentation

Exactly. I'm not complaining about asciidoc itself. But the toolchain
is very fragile, and not crossplatform. Git compiles and works on many
unixen, win32, and some embedded posixy OSs if IIRC.

OTOH asciidoc can be pretty hard to get going even on modern
linuxen.The asciidoc toolchain doesn't even work on Debian Sarge,
which isn't *that* old, while I'm pretty sure git itself can be built
and used on older linuxen. That's where a good old regex-insanity
Perl-based parser beats anything else: no dependencies, works
everywhere.

In that sense, this is close to being a rehash of the "let's use
autoconf" argument...

cheers



martin