* [PATCH 1/4] doc: add data flow diagram using Graph::Easy
2020-02-06 4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
@ 2020-02-06 4:03 ` Eric Wong
2020-02-06 4:03 ` [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages Eric Wong
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06 4:03 UTC (permalink / raw)
To: meta
Maybe this can make it easier for new and potential
users to understand what's going on.
---
Documentation/flow.ge | 27 +++++++++++++++++++++++++++
Documentation/flow.txt | 32 ++++++++++++++++++++++++++++++++
Documentation/include.mk | 18 +++++++++++++++++-
Makefile.PL | 1 +
4 files changed, 77 insertions(+), 1 deletion(-)
create mode 100644 Documentation/flow.ge
create mode 100644 Documentation/flow.txt
diff --git a/Documentation/flow.ge b/Documentation/flow.ge
new file mode 100644
index 00000000..27f2bfcb
--- /dev/null
+++ b/Documentation/flow.ge
@@ -0,0 +1,27 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir. Combining them for the SAME inboxdir
+# will cause conflicts. Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+graph { flow: down }
+
+[delivery tools:\n
+ public-inbox-mda\n
+ public-inbox-watch\n
+ public-inbox-learn] -> [inboxdir]
+
+[git mirroring tools:\n
+ grok-pull,\n
+ various scripts
+] -- git (clone|fetch) &&\n
+ public-inbox-index --> [inboxdir]
+
+[inboxdir] ->
+[read-only daemons:\n
+ public-inbox-httpd\n
+ public-inbox-nntpd]
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/flow.txt b/Documentation/flow.txt
new file mode 100644
index 00000000..ac3459e4
--- /dev/null
+++ b/Documentation/flow.txt
@@ -0,0 +1,32 @@
+# public-inbox data flow
+#
+# Note: choose either "delivery tools" OR "git mirroring tools"
+# for a given inboxdir. Combining them for the SAME inboxdir
+# will cause conflicts. Of course, different inboxdirs may
+# choose different means of getting mail into them.
+
+ +--------------------+
+ | delivery tools: |
+ | public-inbox-mda |
+ | public-inbox-watch |
+ | public-inbox-learn |
+ +--------------------+
+ |
+ |
+ v
++----------------------+ +--------------------+
+| git mirroring tools: | git (clone|fetch) && | |
+| grok-pull, | public-inbox-index | inboxdir |
+| various scripts | ----------------------> | |
++----------------------+ +--------------------+
+ |
+ |
+ v
+ +--------------------+
+ | read-only daemons: |
+ | public-inbox-httpd |
+ | public-inbox-nntpd |
+ +--------------------+
+
+# Copyright 2020 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 3d88fcec..41de0a72 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -10,6 +10,9 @@ MAN = man
# this is "xml" on FreeBSD and maybe some other distros:
XMLSTARLET = xmlstarlet
+# libgraph-easy-perl from Debian, Graph::Easy from CPAN
+GRAPH_EASY = graph-easy
+
# same as pod2text
COLUMNS = 76
@@ -62,6 +65,17 @@ Documentation/standards.txt : Documentation/standards.perl
touch -r Documentation/standards.perl $@+
mv $@+ $@
+# flow.txt is checked into git since Graph::Easy isn't in many distros
+Documentation/flow.txt : Documentation/flow.ge
+ (sed -ne '1,/^$$/p' <Documentation/flow.ge; \
+ $(GRAPH_EASY) Documentation/flow.ge || \
+ cat Documentation/flow.txt; \
+ echo; \
+ sed -ne '/^# Copyright/,$$p' <Documentation/flow.ge \
+ ) >$@+
+ touch -r Documentation/flow.ge $@+
+ mv $@+ $@
+
NEWS NEWS.atom NEWS.html : $(news_deps)
$(PERL) -I lib -w Documentation/mknews.perl $@ $(RELEASES)
@@ -99,7 +113,9 @@ rsync-doc: NEWS.atom.gz
clean-doc:
$(RM_F) $(man1) $(man5) $(man7) $(man8) $(gz_docs) $(docs_html) \
$(mantxt) $(rsync_xdocs) \
- NEWS NEWS.atom NEWS.html Documentation/standards.txt
+ NEWS NEWS.atom NEWS.html Documentation/standards.txt \
+ Documentation/flow.html Documentation/flow.html.gz \
+ Documentation/flow.txt.gz
clean :: clean-doc
diff --git a/Makefile.PL b/Makefile.PL
index d626aef4..59874170 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -13,6 +13,7 @@ $v->{news_deps} = [ map { "Documentation/RelNotes/$_.eml" } @RELEASES ];
$v->{txt} = [ qw(INSTALL README COPYING TODO HACKING) ];
my @dtxt = grep(m!\ADocumentation/[^/]+\.txt\z!, @manifest);
push @dtxt, 'Documentation/standards.txt';
+push @dtxt, 'Documentation/flow.txt';
push @dtxt, @{$v->{txt}};
for my $txt (@dtxt) {
my $html = $txt;
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages
2020-02-06 4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
2020-02-06 4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
@ 2020-02-06 4:03 ` Eric Wong
2020-02-06 4:03 ` [PATCH 3/4] doc: txt2pre: auto-linkify manpage references Eric Wong
2020-02-06 4:03 ` [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too Eric Wong
3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06 4:03 UTC (permalink / raw)
To: meta
There's no need to keep Xapian manpage renderings in a separate
subdirectory, after all.
Eliminating this difference between the local FS and URL path
will allow relative URLs to the Xapian manpages in our local
HTML documentation to work smoothly, since there was never any
".x/" path component for files served from public-inbox.org
---
Documentation/.gitignore | 2 +-
Documentation/include.mk | 3 ---
Makefile.PL | 6 +++---
3 files changed, 4 insertions(+), 7 deletions(-)
diff --git a/Documentation/.gitignore b/Documentation/.gitignore
index f26169ac..92510039 100644
--- a/Documentation/.gitignore
+++ b/Documentation/.gitignore
@@ -1,4 +1,4 @@
/public-inbox-*.txt
/public-inbox.cgi.txt
/standards.txt
-/.x
+/.*.txt
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 41de0a72..8c5f3ca8 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -89,9 +89,6 @@ check :: NEWS .NEWS.atom.check NEWS.html
html: $(docs_html)
-Documentation/.x:
- mkdir -p $@
-
doc: $(docs)
%.gz: %
diff --git a/Makefile.PL b/Makefile.PL
index 59874170..8f1bc4b4 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -71,10 +71,10 @@ $v->{rsync_docs} = [ @{$v->{gz_docs}}, @{$v->{docs}},
# external manpages which we host ourselves, since some packages
# (currently just Xapian) doesn't host manpages themselves.
my @xman = qw(copydatabase.1 xapian-compact.1);
-$v->{xdocs} = [ map { "Documentation/.x/.$_.txt" } @xman ];
-$v->{xdocs_html} = [ map { "Documentation/.x/.$_.html" } @xman ];
+$v->{xdocs} = [ map { "Documentation/.$_.txt" } @xman ];
+$v->{xdocs_html} = [ map { "Documentation/.$_.html" } @xman ];
for (@{$v->{xdocs}}) {
- $t->{"$_ : | Documentation/.x"} = [
+ $t->{"$_:"} = [
'$(PERL) -w Documentation/extman.perl $@ >$@+',
'mv $@+ $@'
];
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 3/4] doc: txt2pre: auto-linkify manpage references
2020-02-06 4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
2020-02-06 4:03 ` [PATCH 1/4] doc: add data flow diagram using Graph::Easy Eric Wong
2020-02-06 4:03 ` [PATCH 2/4] doc: remove .x/ subdirectory for Xapian manpages Eric Wong
@ 2020-02-06 4:03 ` Eric Wong
2020-02-06 4:03 ` [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too Eric Wong
3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06 4:03 UTC (permalink / raw)
To: meta
This can be more convenient for people browsing HTML docs
remotely or locally.
---
Documentation/public-inbox-compact.pod | 3 +-
Documentation/public-inbox-edit.pod | 2 +-
Documentation/public-inbox-xcpdb.pod | 4 +-
Documentation/txt2pre | 86 +++++++++++++++++++++++++-
4 files changed, 88 insertions(+), 7 deletions(-)
diff --git a/Documentation/public-inbox-compact.pod b/Documentation/public-inbox-compact.pod
index 6b0c7376..956069fd 100644
--- a/Documentation/public-inbox-compact.pod
+++ b/Documentation/public-inbox-compact.pod
@@ -61,5 +61,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/public-inbox-edit.pod b/Documentation/public-inbox-edit.pod
index 120f3c3b..46179778 100644
--- a/Documentation/public-inbox-edit.pod
+++ b/Documentation/public-inbox-edit.pod
@@ -106,4 +106,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<public-inbox-purge(1)|https://public-inbox.org/public-inbox-purge.html>
+L<public-inbox-purge(1)>
diff --git a/Documentation/public-inbox-xcpdb.pod b/Documentation/public-inbox-xcpdb.pod
index a13c4efa..b19470cc 100644
--- a/Documentation/public-inbox-xcpdb.pod
+++ b/Documentation/public-inbox-xcpdb.pod
@@ -89,6 +89,4 @@ License: AGPL-3.0+ L<https://www.gnu.org/licenses/agpl-3.0.txt>
=head1 SEE ALSO
-L<copydatabase(1)|https://public-inbox.org/.copydatabase.1.html>,
-L<xapian-compact(1)|https://public-inbox.org/.xapian-compact.1.html>,
-L<public-inbox-index(1)|https://public-inbox.org/public-inbox-index.html>
+L<copydatabase(1)>, L<xapian-compact(1)>, L<public-inbox-index(1)>
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index cb6a7613..01b49379 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -9,13 +9,97 @@ use strict;
use warnings;
use PublicInbox::Linkify;
use PublicInbox::Hval qw(ascii_html);
+my %xurls;
+for (qw[public-inbox.cgi(1)
+ public-inbox-compact(1)
+ public-inbox-config(5)
+ public-inbox-convert(1)
+ public-inbox-daemon(8)
+ public-inbox-edit(1)
+ public-inbox-httpd(1)
+ public-inbox-index(1)
+ public-inbox-init(1)
+ public-inbox-learn(1)
+ public-inbox-mda(1)
+ public-inbox-nntpd(1)
+ public-inbox-overview(7)
+ public-inbox-purge(1)
+ public-inbox-v1-format(5)
+ public-inbox-v2-format(5)
+ public-inbox-watch(1)
+ public-inbox-xcpdb(1)
+]) {
+ my ($n) = (/([\w\-\.]+)/);
+ $xurls{$_} = "$n.html";
+ $xurls{$n} = "$n.html";
+}
+
+for (qw[copydatabase(1) xapian-compact(1)]) {
+ my ($n) = (/([\w\-\.]+)/);
+ $xurls{$_} = ".$n.1.html"
+}
+
+for (qw[flock(2) setrlimit(2) vfork(2)]) {
+ my ($n, $s) = (/([\w\-]+)\((\d)\)/);
+ $xurls{$_} = "http://www.man7.org/linux/man-pages/man2/$n.$s.html";
+}
+
+for (qw[git(1)
+ git-am(1)
+ git-config(1)
+ git-daemon(1)
+ git-fetch(1)
+ git-filter-branch(1)
+ git-format-patch(1)
+ git-gc(1)
+ git-http-backend(1)
+ git-imap-send(1)
+ git-init(1)
+ git-send-email(1)
+ gitrepository-layout(5)
+]) {
+ my ($n) = (/([\w\-\.]+)/);
+ $xurls{$_} = "https://kernel.org/pub/software/scm/git/docs/$n.html"
+}
-my $str = eval { local $/; <> };
+for (qw[
+ sd_listen_fds(3)
+ systemd(1)
+ systemd.unit(5)
+ systemd.socket(5)
+]) {
+ my ($n) = (/([\w\-\.]+)/);
+ $xurls{$_} = "https://www.freedesktop.org/software/systemd/man/$n.html";
+}
+
+$xurls{'spamc(1)'} =
+ 'https://spamassassin.apache.org/full/3.4.x/doc/spamc.html';
+$xurls{'grok-pull'} =
+ 'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
+ '/tree/man/grok-pull.1.rst';
+$xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
+$xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';
+
+my $str = do { local $/; <STDIN> };
my ($title) = ($str =~ /\A([^\n]+)/);
+if ($str =~ /^NAME\n\s+([^\n]+)/sm) {
+ # don't link to ourselves
+ $title = $1;
+ if ($title =~ /([\w\.\-]+)/) {
+ delete $xurls{$1};
+ }
+}
$title = ascii_html($title);
my $l = PublicInbox::Linkify->new;
$str = $l->linkify_1($str);
$str = ascii_html($str);
+
+# longest matches, first
+my @keys = sort { length($b) <=> length($a) } keys %xurls;
+my $xkeys = join('|', map { quotemeta } @keys);
+$str =~ s,(?<![>\w_])($xkeys)(?!(?:[\w<\-]|\.html)),
+ qq(<a\nhref=").$xurls{$1}.qq(">$1).($2//'').'</a>',sge;
+
$str = $l->linkify_2($str);
print '<html><head>',
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 4/4] doc: v1: add a reference to git-filter-repo(1), too
2020-02-06 4:03 [PATCH 0/4] doc: data flow graph and doc linkification Eric Wong
` (2 preceding siblings ...)
2020-02-06 4:03 ` [PATCH 3/4] doc: txt2pre: auto-linkify manpage references Eric Wong
@ 2020-02-06 4:03 ` Eric Wong
3 siblings, 0 replies; 5+ messages in thread
From: Eric Wong @ 2020-02-06 4:03 UTC (permalink / raw)
To: meta
The git-filter-branch(1) manpage itself recommends git-filter-repo,
nowadays due to performance and safety problems.
---
Documentation/public-inbox-v1-format.pod | 3 ++-
Documentation/txt2pre | 2 ++
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/Documentation/public-inbox-v1-format.pod b/Documentation/public-inbox-v1-format.pod
index 507c9451..47a6c94a 100644
--- a/Documentation/public-inbox-v1-format.pod
+++ b/Documentation/public-inbox-v1-format.pod
@@ -169,7 +169,8 @@ there may be many files.
It is impossible to completely expunge messages, even spam, as git
retains full history. Projects may (with adequate notice) cycle to new
-repositories/branches with history cleaned up via L<git-filter-branch(1)>.
+repositories/branches with history cleaned up via L<git-filter-repo(1)>
+or L<git-filter-branch(1)>.
This is up to the administrators.
=head1 COPYRIGHT
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 01b49379..e058125a 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -77,6 +77,8 @@ $xurls{'spamc(1)'} =
$xurls{'grok-pull'} =
'https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git' .
'/tree/man/grok-pull.1.rst';
+$xurls{'git-filter-repo(1)'} = 'https://github.com/newren/git-filter-repo'.
+ './blob/master/Documentation/git-filter-repo.txt';
$xurls{'ssoma(1)'} = 'https://ssoma.public-inbox.org/ssoma.txt';
$xurls{'cgitrc(5)'} = 'https://git.zx2c4.com/cgit/tree/cgitrc.5.txt';
^ permalink raw reply related [flat|nested] 5+ messages in thread