user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH 0/4] doc: data flow graph and doc linkification
@ 2020-02-06  4:03 Eric Wong
  2020-02-06  4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
                   ` (3 more replies)
  0 siblings, 4 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06  4:03 UTC (permalink / raw)
  To: meta

A couple of minor doc improvements.  Hopefully the flow graph
will make things easier-to-understand for new users.

And the txt2pre changes are a precursor to configurable
linkification in the WWW interface for email.

Eric Wong (4):
  doc: add data flow diagram using Graph::Easy
  doc: remove .x/ subdirectory for Xapian manpages
  doc: txt2pre: auto-linkify manpage references
  doc: v1: add a reference to git-filter-repo(1), too

 Documentation/.gitignore                 |  2 +-
 Documentation/flow.ge                    | 27 ++++++++
 Documentation/flow.txt                   | 32 +++++++++
 Documentation/include.mk                 | 21 ++++--
 Documentation/public-inbox-compact.pod   |  3 +-
 Documentation/public-inbox-edit.pod      |  2 +-
 Documentation/public-inbox-v1-format.pod |  3 +-
 Documentation/public-inbox-xcpdb.pod     |  4 +-
 Documentation/txt2pre                    | 88 +++++++++++++++++++++++-
 Makefile.PL                              |  7 +-
 10 files changed, 173 insertions(+), 16 deletions(-)
 create mode 100644 Documentation/flow.ge
 create mode 100644 Documentation/flow.txt


^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 1/4] doc: add data flow diagram using Graph::Easy
  2020-02-06  4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
@ 2020-02-06  4:03 ` Eric Wong
  2020-02-06  4:03 ` [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages Eric Wong
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06  4:03 UTC (permalink / raw)
  To: meta

Maybe this can make it easier for new and potential
users to understand what's going on.
---
 Documentation/flow.ge    | 27 +++++++++++++++++++++++++++
 Documentation/flow.txt   | 32 ++++++++++++++++++++++++++++++++
 Documentation/include.mk | 18 +++++++++++++++++-
 Makefile.PL              |  1 +
 4 files changed, 77 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/flow.ge
 create mode 100644 Documentation/flow.txt

diff --git a/Documentation/flow.ge b/Documentation/flow.ge
new file mode 100644
index 00000000..27f2bfcb
--- /dev/null
+++ b/Documentation/flow.ge
@@ -0,0 +1,27 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir.  Combining them for the SAME inboxdir
+# will cause conflicts.  Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+graph { flow: down }
+
+[delivery tools:\n
+ public-inbox-mda\n
+ public-inbox-watch\n
+ public-inbox-learn] -> [inboxdir]
+
+[git mirroring tools:\n
+ grok-pull,\n
+ various scripts
+] -- git (clone|fetch) &&\n
+ public-inbox-index --> [inboxdir]
+
+[inboxdir] ->
+[read-only daemons:\n
+ public-inbox-httpd\n
+ public-inbox-nntpd]
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/flow.txt b/Documentation/flow.txt
new file mode 100644
index 00000000..ac3459e4
--- /dev/null
+++ b/Documentation/flow.txt
@@ -0,0 +1,32 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir.  Combining them for the SAME inboxdir
+# will cause conflicts.  Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+                                                 +--------------------+
+                                                 |  delivery tools:   |
+                                                 |  public-inbox-mda  |
+                                                 | public-inbox-watch |
+                                                 | public-inbox-learn |
+                                                 +--------------------+
+                                                   |
+                                                   |
+                                                   v
++----------------------+                         +--------------------+
+| git mirroring tools: |  git (clone|fetch) &&   |                    |
+|      grok-pull,      |  public-inbox-index     |      inboxdir      |
+|   various scripts    | ----------------------> |                    |
++----------------------+                         +--------------------+
+                                                   |
+                                                   |
+                                                   v
+                                                 +--------------------+
+                                                 | read-only daemons: |
+                                                 | public-inbox-httpd |
+                                                 | public-inbox-nntpd |
+                                                 +--------------------+
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 3d88fcec..41de0a72 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -10,6 +10,9 @@ MAN = man
 # this is "xml" on FreeBSD and maybe some other distros:
 XMLSTARLET = xmlstarlet
 
+# libgraph-easy-perl from Debian, Graph::Easy from CPAN
+GRAPH_EASY = graph-easy
+
 # same as pod2text
 COLUMNS = 76
 
@@ -62,6 +65,17 @@ Documentation/standards.txt : Documentation/standards.perl
 	touch -r Documentation/standards.perl $@+
 	mv $@+ $@
 
+# flow.txt is checked into git since Graph::Easy isn't in many distros
+Documentation/flow.txt : Documentation/flow.ge
+	(sed -ne '1,/^$$/p' <Documentation/flow.ge; \
+		$(GRAPH_EASY) Documentation/flow.ge || \
+			cat Documentation/flow.txt; \
+		echo; \
+		sed -ne '/^# Copyright/,$$p' <Documentation/flow.ge \
+		) >$@+
+	touch -r Documentation/flow.ge $@+
+	mv $@+ $@
+
 NEWS NEWS.atom NEWS.html : $(news_deps)
 	$(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES)
 
@@ -99,7 +113,9 @@ rsync-doc: NEWS.atom.gz
 clean-doc:
 	$(RM_F) $(man1) $(man5) $(man7) $(man8) $(gz_docs) $(docs_html) \
 		$(mantxt) $(rsync_xdocs) \
-		NEWS NEWS.atom NEWS.html Documentation/standards.txt
+		NEWS NEWS.atom NEWS.html Documentation/standards.txt \
+		Documentation/flow.html Documentation/flow.html.gz \
+		Documentation/flow.txt.gz
 
 clean :: clean-doc
 
diff --git a/Makefile.PL b/Makefile.PL
index d626aef4..59874170 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -13,6 +13,7 @@ $v->{news_deps} = [ map { "Documentation/RelNotes/$_.eml" } @RELEASES ];
 $v->{txt} = [ qw(INSTALL README COPYING TODO HACKING) ];
 my @dtxt = grep(m!\ADocumentation/[^/]+\.txt\z!, @manifest);
 push @dtxt, 'Documentation/standards.txt';
+push @dtxt, 'Documentation/flow.txt';
 push @dtxt, @{$v->{txt}};
 for my $txt (@dtxt) {
 	my $html = $txt;

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages
  2020-02-06  4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
  2020-02-06  4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
@ 2020-02-06  4:03 ` Eric Wong
  2020-02-06  4:03 ` [PATCH 3/4] doc: txt2pre: auto-linkify manpage references Eric Wong
  2020-02-06  4:03 ` [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06  4:03 UTC (permalink / raw)
  To: meta

There's no need to keep Xapian manpage renderings in a separate
subdirectory, after all.

Eliminating this difference between the local FS and URL path
will allow relative URLs to the Xapian manpages in our local
HTML documentation to work smoothly, since there was never any
".x/" path component for files served from public-inbox.org
---
 Documentation/.gitignore | 2 +-
 Documentation/include.mk | 3 ---
 Makefile.PL              | 6 +++---
 3 files changed, 4 insertions(+), 7 deletions(-)

diff --git a/Documentation/.gitignore b/Documentation/.gitignore
index f26169ac..92510039 100644
--- a/Documentation/.gitignore
+++ b/Documentation/.gitignore
@@ -1,4 +1,4 @@
 /public-inbox-*.txt
 /public-inbox.cgi.txt
 /standards.txt
-/.x
+/.*.txt
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 41de0a72..8c5f3ca8 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -89,9 +89,6 @@ check :: NEWS .NEWS.atom.check NEWS.html
 
 html: $(docs_html)
 
-Documentation/.x:
-	mkdir -p $@
-
 doc: $(docs)
 
 %.gz: %
diff --git a/Makefile.PL b/Makefile.PL
index 59874170..8f1bc4b4 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -71,10 +71,10 @@ $v->{rsync_docs} = [ @{$v->{gz_docs}}, @{$v->{docs}},
 # external manpages which we host ourselves, since some packages
 # (currently just Xapian) doesn't host manpages themselves.
 my @xman = qw(copydatabase.1 xapian-compact.1);
-$v->{xdocs} = [ map { "Documentation/.x/.$_.txt" } @xman ];
-$v->{xdocs_html} = [ map { "Documentation/.x/.$_.html" } @xman ];
+$v->{xdocs} = [ map { "Documentation/.$_.txt" } @xman ];
+$v->{xdocs_html} = [ map { "Documentation/.$_.html" } @xman ];
 for (@{$v->{xdocs}}) {
-	$t->{"$_ : | Documentation/.x"} = [
+	$t->{"$_:"} = [
 		'$(PERL) -w Documentation/extman.perl $@ >$@+',
 		'mv $@+ $@'
 	];

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 3/4] doc: txt2pre: auto-linkify manpage references
  2020-02-06  4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
  2020-02-06  4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
  2020-02-06  4:03 ` [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages Eric Wong
@ 2020-02-06  4:03 ` Eric Wong
  2020-02-06  4:03 ` [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06  4:03 UTC (permalink / raw)
  To: meta

This can be more convenient for people browsing HTML docs
remotely or locally.
---
 Documentation/public-inbox-compact.pod |  3 +-
 Documentation/public-inbox-edit.pod    |  2 +-
 Documentation/public-inbox-xcpdb.pod   |  4 +-
 Documentation/txt2pre                  | 86 +++++++++++++++++++++++++-
 4 files changed, 88 insertions(+), 7 deletions(-)

diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 6b0c7376..956069fd 100644
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -61,5 +61,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/public-inbox-edit.pod b/Documentation/public-inbox-edit.pod
index 120f3c3b..46179778 100644
--- a/Documentation/public-inbox-edit.pod
+++ b/Documentation/public-inbox-edit.pod
@@ -106,4 +106,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<public-inbox-purge(1)|https://public-inbox.org/public-inbox-purge.html>
+L<public-inbox-purge(1)>
diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index a13c4efa..b19470cc 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -89,6 +89,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
 
 =head1 SEE ALSO
 
-L<copydatabase(1)|https://public-inbox.org/.copydatabase.1.html>,
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<copydatabase(1)>, L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index cb6a7613..01b49379 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -9,13 +9,97 @@ use strict;
 use warnings;
 use PublicInbox::Linkify;
 use PublicInbox::Hval qw(ascii_html);
+my %xurls;
+for (qw[public-inbox.cgi(1)
+	public-inbox-compact(1)
+	public-inbox-config(5)
+	public-inbox-convert(1)
+	public-inbox-daemon(8)
+	public-inbox-edit(1)
+	public-inbox-httpd(1)
+	public-inbox-index(1)
+	public-inbox-init(1)
+	public-inbox-learn(1)
+	public-inbox-mda(1)
+	public-inbox-nntpd(1)
+	public-inbox-overview(7)
+	public-inbox-purge(1)
+	public-inbox-v1-format(5)
+	public-inbox-v2-format(5)
+	public-inbox-watch(1)
+	public-inbox-xcpdb(1)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "$n.html";
+	$xurls{$n} = "$n.html";
+}
+
+for (qw[copydatabase(1) xapian-compact(1)]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = ".$n.1.html"
+}
+
+for (qw[flock(2) setrlimit(2) vfork(2)]) {
+	my ($n, $s) = (/([\w\-]+)\((\d)\)/);
+	$xurls{$_} = "http://www.man7.org/linux/man-pages/man2/$n.$s.html";
+}
+
+for (qw[git(1)
+	git-am(1)
+	git-config(1)
+	git-daemon(1)
+	git-fetch(1)
+	git-filter-branch(1)
+	git-format-patch(1)
+	git-gc(1)
+	git-http-backend(1)
+	git-imap-send(1)
+	git-init(1)
+	git-send-email(1)
+	gitrepository-layout(5)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "https://kernel.org/pub/software/scm/git/docs/$n.html"
+}
 
-my $str = eval { local $/; <> };
+for (qw[
+	sd_listen_fds(3)
+	systemd(1)
+	systemd.unit(5)
+	systemd.socket(5)
+]) {
+	my ($n) = (/([\w\-\.]+)/);
+	$xurls{$_} = "https://www.freedesktop.org/software/systemd/man/$n.html";
+}
+
+$xurls{'spamc(1)'} =
+	'https://spamassassin.apache.org/full/3.4.x/doc/spamc.html';
+$xurls{'grok-pull'} =
+	'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
+	'/tree/man/grok-pull.1.rst';
+$xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
+$xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';
+
+my $str = do { local $/; <STDIN> };
 my ($title) = ($str =~ /\A([^\n]+)/);
+if ($str =~ /^NAME\n\s+([^\n]+)/sm) {
+	# don't link to ourselves
+	$title = $1;
+	if ($title =~ /([\w\.\-]+)/) {
+		delete $xurls{$1};
+	}
+}
 $title = ascii_html($title);
 my $l = PublicInbox::Linkify->new;
 $str = $l->linkify_1($str);
 $str = ascii_html($str);
+
+# longest matches, first
+my @keys = sort { length($b) <=> length($a) } keys %xurls;
+my $xkeys = join('|', map { quotemeta } @keys);
+$str =~ s,(?<![>\w_])($xkeys)(?!(?:[\w<\-]|\.html)),
+	qq(<a\nhref=").$xurls{$1}.qq(">$1).($2//'').'</a>',sge;
+
 $str = $l->linkify_2($str);
 
 print '<html><head>',

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too
  2020-02-06  4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
                   ` (2 preceding siblings ...)
  2020-02-06  4:03 ` [PATCH 3/4] doc: txt2pre: auto-linkify manpage references Eric Wong
@ 2020-02-06  4:03 ` Eric Wong
  3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06  4:03 UTC (permalink / raw)
  To: meta

The git-filter-branch(1) manpage itself recommends git-filter-repo,
nowadays due to performance and safety problems.
---
 Documentation/public-inbox-v1-format.pod | 3 ++-
 Documentation/txt2pre                    | 2 ++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod
index 507c9451..47a6c94a 100644
--- a/Documentation/public-inbox-v1-format.pod
+++ b/Documentation/public-inbox-v1-format.pod
@@ -169,7 +169,8 @@ there may be many files.
 
 It is impossible to completely expunge messages, even spam, as git
 retains full history.  Projects may (with adequate notice) cycle to new
-repositories/branches with history cleaned up via L<git-filter-branch(1)>.
+repositories/branches with history cleaned up via L<git-filter-repo(1)>
+or L<git-filter-branch(1)>.
 This is up to the administrators.
 
 =head1 COPYRIGHT
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 01b49379..e058125a 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -77,6 +77,8 @@ $xurls{'spamc(1)'} =
 $xurls{'grok-pull'} =
 	'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
 	'/tree/man/grok-pull.1.rst';
+$xurls{'git-filter-repo(1)'} = 'https://github.com/newren/git-filter-repo'.
+			'./blob/master/Documentation/git-filter-repo.txt';
 $xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
 $xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';
 

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, back to index

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-06  4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
2020-02-06  4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
2020-02-06  4:03 ` [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages Eric Wong
2020-02-06  4:03 ` [PATCH 3/4] doc: txt2pre: auto-linkify manpage references Eric Wong
2020-02-06  4:03 ` [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git