[PATCH 0/4] doc: data flow graph and doc linkification

[PATCH 0/4] doc: data flow graph and doc linkification

[Eric Wong]

A couple of minor doc improvements.  Hopefully the flow graph
will make things easier-to-understand for new users.

And the txt2pre changes are a precursor to configurable
linkification in the WWW interface for email.

Eric Wong (4):
  doc: add data flow diagram using Graph::Easy
  doc: remove .x/ subdirectory for Xapian manpages
  doc: txt2pre: auto-linkify manpage references
  doc: v1: add a reference to git-filter-repo(1), too

 Documentation/.gitignore                 |  2 +-
 Documentation/flow.ge                    | 27 ++++++++
 Documentation/flow.txt                   | 32 +++++++++
 Documentation/include.mk                 | 21 ++++--
 Documentation/public-inbox-compact.pod   |  3 +-
 Documentation/public-inbox-edit.pod      |  2 +-
 Documentation/public-inbox-v1-format.pod |  3 +-
 Documentation/public-inbox-xcpdb.pod     |  4 +-
 Documentation/txt2pre                    | 88 +++++++++++++++++++++++-
 Makefile.PL                              |  7 +-
 10 files changed, 173 insertions(+), 16 deletions(-)
 create mode 100644 Documentation/flow.ge
 create mode 100644 Documentation/flow.txt

[PATCH 1/4] doc: add data flow diagram using Graph::Easy

[Eric Wong]

Maybe this can make it easier for new and potential
users to understand what's going on.
---
 Documentation/flow.ge    | 27 +++++++++++++++++++++++++++
 Documentation/flow.txt   | 32 ++++++++++++++++++++++++++++++++
 Documentation/include.mk | 18 +++++++++++++++++-
 Makefile.PL              |  1 +
 4 files changed, 77 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/flow.ge
 create mode 100644 Documentation/flow.txt

diff --git a/Documentation/flow.ge b/Documentation/flow.ge
new file mode 100644
index 00000000..27f2bfcb
--- /dev/null
+++ b/Documentation/flow.ge
@@ -0,0 +1,27 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir.  Combining them for the SAME inboxdir
+# will cause conflicts.  Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+graph { flow: down }
+
+[delivery tools:\n
+ public-inbox-mda\n
+ public-inbox-watch\n
+ public-inbox-learn] -> [inboxdir]
+
+[git mirroring tools:\n
+ grok-pull,\n
+ various scripts
+] -- git (clone|fetch) &&\n
+ public-inbox-index --> [inboxdir]
+
+[inboxdir] ->
+[read-only daemons:\n
+ public-inbox-httpd\n
+ public-inbox-nntpd]
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/flow.txt b/Documentation/flow.txt
new file mode 100644
index 00000000..ac3459e4
--- /dev/null
+++ b/Documentation/flow.txt
@@ -0,0 +1,32 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir.  Combining them for the SAME inboxdir
+# will cause conflicts.  Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+                                                 +--------------------+
+                                                 |  delivery tools:   |
+                                                 |  public-inbox-mda  |
+                                                 | public-inbox-watch |
+                                                 | public-inbox-learn |
+                                                 +--------------------+
+                                                   |
+                                                   |
+                                                   v
++----------------------+                         +--------------------+
+| git mirroring tools: |  git (clone|fetch) &&   |                    |
+|      grok-pull,      |  public-inbox-index     |      inboxdir      |
+|   various scripts    | ----------------------> |                    |
++----------------------+                         +--------------------+
+                                                   |
+                                                   |
+                                                   v
+                                                 +--------------------+
+                                                 | read-only daemons: |
+                                                 | public-inbox-httpd |
+                                                 | public-inbox-nntpd |
+                                                 +--------------------+
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 3d88fcec..41de0a72 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -10,6 +10,9 @@ MAN = man
 # this is "xml" on FreeBSD and maybe some other distros:
 XMLSTARLET = xmlstarlet
 
+# libgraph-easy-perl from Debian, Graph::Easy from CPAN
+GRAPH_EASY = graph-easy
+
 # same as pod2text
 COLUMNS = 76
 
@@ -62,6 +65,17 @@ Documentation/standards.txt : Documentation/standards.perl
 	touch -r Documentation/standards.perl $@+
 	mv $@+ $@
 
+# flow.txt is checked into git since Graph::Easy isn't in many distros
+Documentation/flow.txt : Documentation/flow.ge
+	(sed -ne '1,/^$$/p' <Documentation/flow.ge; \
+		$(GRAPH_EASY) Documentation/flow.ge || \
+			cat Documentation/flow.txt; \
+		echo; \
+		sed -ne '/^# Copyright/,$$p' <Documentation/flow.ge \
+		) >$@+
+	touch -r Documentation/flow.ge $@+
+	mv $@+ $@
+
 NEWS NEWS.atom NEWS.html : $(news_deps)
 	$(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES)
 
@@ -99,7 +113,9 @@ rsync-doc: NEWS.atom.gz
 clean-doc:
 	$(RM_F) $(man1) $(man5) $(man7) $(man8) $(gz_docs) $(docs_html) \
 		$(mantxt) $(rsync_xdocs) \
-		NEWS NEWS.atom NEWS.html Documentation/standards.txt
+		NEWS NEWS.atom NEWS.html Documentation/standards.txt \
+		Documentation/flow.html Documentation/flow.html.gz \
+		Documentation/flow.txt.gz
 
 clean :: clean-doc
 
diff --git a/Makefile.PL b/Makefile.PL
index d626aef4..59874170 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -13,6 +13,7 @@ $v->{news_deps} = [ map { "Documentation/RelNotes/$_.eml" } @RELEASES ];
 $v->{txt} = [ qw(INSTALL README COPYING TODO HACKING) ];
 my @dtxt = grep(m!\ADocumentation/[^/]+\.txt\z!, @manifest);
 push @dtxt, 'Documentation/standards.txt';
+push @dtxt, 'Documentation/flow.txt';
 push @dtxt, @{$v->{txt}};
 for my $txt (@dtxt) {
 	my $html = $txt;

[PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages

[Eric Wong]

There's no need to keep Xapian manpage renderings in a separate
subdirectory, after all.

Eliminating this difference between the local FS and URL path
will allow relative URLs to the Xapian manpages in our local
HTML documentation to work smoothly, since there was never any
".x/" path component for files served from public-inbox.org
---
 Documentation/.gitignore | 2 +-
 Documentation/include.mk | 3 ---
 Makefile.PL              | 6 +++---
 3 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/Documentation/.gitignore b/Documentation/.gitignore
index f26169ac..92510039 100644
--- a/Documentation/.gitignore
+++ b/Documentation/.gitignore
@@ -1,4 +1,4 @@
 /public-inbox-*.txt
 /public-inbox.cgi.txt
 /standards.txt
-/.x
+/.*.txt
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 41de0a72..8c5f3ca8 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -89,9 +89,6 @@ check :: NEWS .NEWS.atom.check NEWS.html
 
 html: $(docs_html)
 
-Documentation/.x:
-	mkdir -p $@
-
 doc: $(docs)
 
 %.gz: %
diff --git a/Makefile.PL b/Makefile.PL
index 59874170..8f1bc4b4 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -71,10 +71,10 @@ $v->{rsync_docs} = [ @{$v->{gz_docs}}, @{$v->{docs}},
 # external manpages which we host ourselves, since some packages
 # (currently just Xapian) doesn't host manpages themselves.
 my @xman = qw(copydatabase.1 xapian-compact.1);
-$v->{xdocs} = [ map { "Documentation/.x/.$_.txt" } @xman ];
-$v->{xdocs_html} = [ map { "Documentation/.x/.$_.html" } @xman ];
+$v->{xdocs} = [ map { "Documentation/.$_.txt" } @xman ];
+$v->{xdocs_html} = [ map { "Documentation/.$_.html" } @xman ];
 for (@{$v->{xdocs}}) {
-	$t->{"$_ : | Documentation/.x"} = [
+	$t->{"$_:"} = [
 		'$(PERL) -w Documentation/extman.perl $@ >$@+',
 		'mv $@+ $@'
 	];

[PATCH 3/4] doc: txt2pre: auto-linkify manpage references

[Eric Wong]

This can be more convenient for people browsing HTML docs
remotely or locally.
---
 Documentation/public-inbox-compact.pod |  3 +-
 Documentation/public-inbox-edit.pod    |  2 +-
 Documentation/public-inbox-xcpdb.pod   |  4 +-
 Documentation/txt2pre                  | 86 +++++++++++++++++++++++++-
 4 files changed, 88 insertions(+), 7 deletions(-)

diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 6b0c7376..956069fd 100644
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -61,5 +61,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/public-inbox-edit.pod b/Documentation/public-inbox-edit.pod
index 120f3c3b..46179778 100644
--- a/Documentation/public-inbox-edit.pod
+++ b/Documentation/public-inbox-edit.pod
@@ -106,4 +106,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<public-inbox-purge(1)|https://public-inbox.org/public-inbox-purge.html>
+L<public-inbox-purge(1)>
diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index a13c4efa..b19470cc 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -89,6 +89,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<copydatabase(1)|https://public-inbox.org/.copydatabase.1.html>,
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<copydatabase(1)>, L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index cb6a7613..01b49379 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -9,13 +9,97 @@ use strict;
 use warnings;
 use PublicInbox::Linkify;
 use PublicInbox::Hval qw(ascii_html);
+my %xurls;
+for (qw[public-inbox.cgi(1)
+	public-inbox-compact(1)
+	public-inbox-config(5)
+	public-inbox-convert(1)
+	public-inbox-daemon(8)
+	public-inbox-edit(1)
+	public-inbox-httpd(1)
+	public-inbox-index(1)
+	public-inbox-init(1)
+	public-inbox-learn(1)
+	public-inbox-mda(1)
+	public-inbox-nntpd(1)
+	public-inbox-overview(7)
+	public-inbox-purge(1)
+	public-inbox-v1-format(5)
+	public-inbox-v2-format(5)
+	public-inbox-watch(1)
+	public-inbox-xcpdb(1)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "$n.html";
+	$xurls{$n} = "$n.html";
+}
+
+for (qw[copydatabase(1) xapian-compact(1)]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = ".$n.1.html"
+}
+
+for (qw[flock(2) setrlimit(2) vfork(2)]) {
+	my ($n, $s) = (/([\w\-]+)\((\d)\)/);
+	$xurls{$_} = "http://www.man7.org/linux/man-pages/man2/$n.$s.html";
+}
+
+for (qw[git(1)
+	git-am(1)
+	git-config(1)
+	git-daemon(1)
+	git-fetch(1)
+	git-filter-branch(1)
+	git-format-patch(1)
+	git-gc(1)
+	git-http-backend(1)
+	git-imap-send(1)
+	git-init(1)
+	git-send-email(1)
+	gitrepository-layout(5)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "https://kernel.org/pub/software/scm/git/docs/$n.html"
+}
 
-my $str = eval { local $/; <> };
+for (qw[
+	sd_listen_fds(3)
+	systemd(1)
+	systemd.unit(5)
+	systemd.socket(5)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "https://www.freedesktop.org/software/systemd/man/$n.html";
+}
+
+$xurls{'spamc(1)'} =
+	'https://spamassassin.apache.org/full/3.4.x/doc/spamc.html';
+$xurls{'grok-pull'} =
+	'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
+	'/tree/man/grok-pull.1.rst';
+$xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
+$xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';
+
+my $str = do { local $/; <STDIN> };
 my ($title) = ($str =~ /\A([^\n]+)/);
+if ($str =~ /^NAME\n\s+([^\n]+)/sm) {
+	# don't link to ourselves
+	$title = $1;
+	if ($title =~ /([\w\.\-]+)/) {
+		delete $xurls{$1};
+	}
+}
 $title = ascii_html($title);
 my $l = PublicInbox::Linkify->new;
 $str = $l->linkify_1($str);
 $str = ascii_html($str);
+
+# longest matches, first
+my @keys = sort { length($b) <=> length($a) } keys %xurls;
+my $xkeys = join('|', map { quotemeta } @keys);
+$str =~ s,(?<![>\w_])($xkeys)(?!(?:[\w<\-]|\.html)),
+	qq(<a\nhref=").$xurls{$1}.qq(">$1).($2//'').'</a>',sge;
+
 $str = $l->linkify_2($str);
 
 print '<html><head>',

[PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too

[Eric Wong]

The git-filter-branch(1) manpage itself recommends git-filter-repo,
nowadays due to performance and safety problems.
---
 Documentation/public-inbox-v1-format.pod | 3 ++-
 Documentation/txt2pre                    | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod
index 507c9451..47a6c94a 100644
--- a/Documentation/public-inbox-v1-format.pod
+++ b/Documentation/public-inbox-v1-format.pod
@@ -169,7 +169,8 @@ there may be many files.
 
 It is impossible to completely expunge messages, even spam, as git
 retains full history.  Projects may (with adequate notice) cycle to new
-repositories/branches with history cleaned up via L<git-filter-branch(1)>.
+repositories/branches with history cleaned up via L<git-filter-repo(1)>
+or L<git-filter-branch(1)>.
 This is up to the administrators.
 
 =head1 COPYRIGHT
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 01b49379..e058125a 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -77,6 +77,8 @@ $xurls{'spamc(1)'} =
 $xurls{'grok-pull'} =
 	'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
 	'/tree/man/grok-pull.1.rst';
+$xurls{'git-filter-repo(1)'} = 'https://github.com/newren/git-filter-repo'.
+			'./blob/master/Documentation/git-filter-repo.txt';
 $xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
 $xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';