user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH 0/9] install updates
@ 2019-05-09  7:50 Eric Wong
  2019-05-09  7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
                   ` (8 more replies)
  0 siblings, 9 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

A couple of build and installation related updates
to clarify package dependencies.  The INSTALL document
is hopefully less intimidating, now.

I've found ci/deps.perl pretty useful; maybe even for
ordinary users on Debian and FreeBSD systems.

I still haven't tested on any RPM-based systems, yet;
but FreeBSD 11.2 and 12.0 on amd64 seem fine; along
with Debian sid.

Eric Wong (9):
  INSTALL: update with FreeBSD pkg names and reorder
  INSTALL: depend on Encode instead of Encode::MIME::Header
  INSTALL: note xapian-compact(1) tool
  t/purge.t: fix unreferenced variable
  doc: use bullet list for wwwlisting options
  doc: give txt files proper titles
  Makefile.PL: improve make(1) portability
  doc/include.mk: remove git use and redundant declarations
  ci: include some scripts to make CI easier

 Documentation/design_www.txt          |   2 +
 Documentation/hosted.txt              |   2 +
 Documentation/include.mk              |  31 +++-
 Documentation/public-inbox-config.pod |  16 +-
 Documentation/txt2pre                 |   3 +-
 INSTALL                               | 205 +++++++++++++--------
 MANIFEST                              |   4 +
 Makefile.PL                           |  17 +-
 ci/README                             |  33 ++++
 ci/deps.perl                          | 254 ++++++++++++++++++++++++++
 ci/profiles.sh                        |  72 ++++++++
 ci/run.sh                             |  21 +++
 t/purge.t                             |   2 +-
 13 files changed, 558 insertions(+), 104 deletions(-)
 create mode 100644 ci/README
 create mode 100755 ci/deps.perl
 create mode 100755 ci/profiles.sh
 create mode 100755 ci/run.sh

-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
@ 2019-05-09  7:50 ` Eric Wong
  2019-05-09  7:50 ` [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header Eric Wong
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

Fortunately, there is a pattern to most of these package names
in all distros I've tested (and hopefully other BSDs have them,
too).

Then, reorder the INSTALL document to mention the top-level
modules, first, so users can avoid needing to type extra
dependencies.  However, we still list some implicit dependencies
in case the upstream package drops dependencies independently of
us.

Finally, Devel::Peek is not a dependency worth making optional
since it's bundled by Perl upstream.  Fedora/RH-based distros
are the only one which turn it into a non-standard package when
Perl5 is installed.
---
 INSTALL     | 197 ++++++++++++++++++++++++++++++++--------------------
 Makefile.PL |   8 +++
 2 files changed, 128 insertions(+), 77 deletions(-)

diff --git a/INSTALL b/INSTALL
index fafaf57..99836c1 100644
--- a/INSTALL
+++ b/INSTALL
@@ -9,8 +9,8 @@ public-inbox is developed on Debian GNU/Linux systems and will
 never depend on packages outside of the "main" component of
 the "stable" distribution, currently Debian 9.x ("stretch")
 
-Most packages are available in other GNU/Linux distributions;
-and FreeBSD support can happen.
+Most packages are available in other GNU/Linux distributions
+and FreeBSD.
 
 TODO: this still needs to be documented better,
 also see the scripts/ and sa_config/ directories in the source tree
@@ -23,7 +23,7 @@ functionality.  The core tools are, of course:
 
 * Git (1.8.0+, 2.6+ for writing v2 repositories)
 * Perl 5.10.1+
-* SQLite (needed for Xapian use)
+* DBD::SQLite (needed for NNTP, message threading, and v2 repositories)
 
 To accept incoming mail into a public inbox, you'll likely want:
 
@@ -32,88 +32,131 @@ To accept incoming mail into a public inbox, you'll likely want:
 
 Beyond that, there is a long list of Perl modules required, starting with:
 
-* Date::Parse                   deb: libtimedate-perl
-                                rpm: perl-Time-ParseDate
+* Date::Parse                      deb: libtimedate-perl
+                                   pkg: p5-TimeDate
+                                   rpm: perl-TimeDate
 
-* Email::MIME                   deb: libemail-mime-perl
-                                rpm: perl-Email-MIME
+* Devel::Peek                      deb: libperl5.$MINOR (e.g. libperl5.24)
+                                   pkg: perl5
+                                   rpm: perl-Devel-Peek
+                                   (typically installed alongside Perl5)
 
-* Email::MIME::ContentType      deb: libemail-mime-contenttype-perl
-                                rpm: perl-Email-MIME-ContentType
+* Email::MIME                      deb: libemail-mime-perl
+                                   pkg: p5-Email-MIME
+                                   rpm: perl-Email-MIME
 
-* Encode::MIME::Header          deb: libencode-perl
-                                rpm: perl-Encode
+* Encode::MIME::Header             deb: libencode-perl
+                                   pkg: perl5
+                                   rpm: perl-Encode
 
-* Plack                         deb: libplack-perl
-                                rpm: perl-Plack, perl-Plack-Test,
-                                (for HTML/Atom generation)
+* Plack                            deb: libplack-perl
+                                   pkg: p5-Plack
+                                   rpm: perl-Plack, perl-Plack-Test,
+                                   (for HTML/Atom generation)
 
-* URI::Escape                   deb: liburi-perl
-                                rpm: perl-URI
-                                (for HTML/Atom generation)
+* URI::Escape                      deb: liburi-perl
+                                   pkg: p5-URI
+                                   rpm: perl-URI
+                                   (for HTML/Atom generation)
 
-Where "deb" indicates package names for Debian-derived distributions and
+Where "deb" indicates package names for Debian-derived distributions,
+"pkg" is for the FreeBSD package (maybe other common BSDs, too), and
 "rpm" is for RPM-based distributions (only known to work on Fedora).
 
 Numerous optional modules are likely to be useful as well:
 
-  - Search::Xapian             deb: libsearch-xapian-perl
-                               rpm: perl-Search-Xapian
-                               (for NNTP service or gzipped mbox over HTTP)
-
-  - IO::Compress::Gzip         deb: perl-modules (or libio-compress-perl)
-                               rpm: perl-PerlIO-gzip
-                               (for gzipped mbox over HTTP)
-
-  - DBI                        deb: libdbi-perl
-                               rpm: perl-DBI
-                               (for gzipped mbox over HTTP)
-
-  - DBD::SQLite                deb: libdbd-sqlite3-perl
-                               rpm: perl-DBD-SQLite
-                               (for NNTP service or gzipped mbox over HTTP)
-
-  - Net::Server                deb: libnet-server-perl
-                               rpm: perl-Net-Server
-                               (for HTTP/NNTP servers as standalone daemons,
-                                not needed as systemd services)
-
-  - Filesys::Notify::Simple    deb: libfilesys-notify-simple-perl
-                               rpm: perl-Filesys-Notify-Simple
-                               (for public-inbox-watch)
-
-  - Inline::C[7]               deb: libinline-c-perl
-                               (speeds up spawning of git(1) on Linux
-                                (see public-inbox-daemon(8))
-
-  - Plack::Middleware::ReverseProxy
-
-                               deb: libplack-middleware-reverseproxy-perl
-                               rpm: perl-Plack-Middleware-ReverseProxy
-                               (ensures redirects are correct when running
-                                behind nginx or Varnish)
-
-  - Plack::Middleware::Deflater
-
-                               deb: libplack-middleware-deflater-perl
-                               rpm: perl-Plack-Middleware-Deflater
-                               (saves bandwidth on responses)
-
-  - Socket6                    deb: libsocket6-perl
-                               rpm: perl-Socket6
-                               (pulled in by SpamAssassin and Net::Server,
-                                only necessary if using IPv6 with
-                                Plack::Middleware::AccessLog or similar
-                                on Perl <= 5.12)
-
-  - Crypt::CBC                 deb: libcrypt-cbc-perl
-                               (for the rarely-used PublicInbox::Unsubscribe)
-
-On Fedora systems, you'll probably also end up wanting
-perl-Test-HTTP-Server-Simple, perl-Devel-Peek, and perl-IPC-Run to run the
-test suite.  On Debian systems, libxml-feed-perl and libipc-run-perl
-will aid in running the test suite (XML::Feed and IPC::Run respectively,
-on CPAN).
+- DBD::SQLite                      deb: libdbd-sqlite3-perl
+                                   pkg: p5-DBD-SQLite
+                                   rpm: perl-DBD-SQLite
+                                   (for v2, NNTP, or gzipped mboxes)
+
+- Search::Xapian                   deb: libsearch-xapian-perl
+                                   pkg: p5-Search-Xapian
+                                   rpm: perl-Search-Xapian
+                                   (for v2, HTTP search)
+
+- Net::Server                      deb: libnet-server-perl
+                                   pkg: pkg-Net-Server
+                                   rpm: perl-Net-Server
+                                   (for HTTP/NNTP background daemons,
+                                    not needed as systemd services or
+                                    foreground servers)
+
+- Inline::C[7]                     deb: libinline-c-perl
+                                   pkg: pkg-Inline-C
+                                   (speeds up process spawning on Linux,
+                                    see public-inbox-daemon(8))
+
+- Plack::Middleware::ReverseProxy  deb: libplack-middleware-reverseproxy-perl
+                                   pkg: p5-Plack-Middleware-ReverseProxy
+                                   rpm: perl-Plack-Middleware-ReverseProxy
+                                   (ensures redirects are correct when running
+                                    behind nginx or Varnish)
+
+- Plack::Middleware::Deflater      deb: libplack-middleware-deflater-perl
+                                   pkg: p5 -Plack-Middleware-Deflater
+                                   rpm: perl-Plack-Middleware-Deflater
+                                   (saves bandwidth on responses)
+
+* highlight                        deb: libhighlight-perl
+                                   (for syntax highlighting with coderepo)
+
+The following modules are typically pulled in by dependencies listed
+above, so there is no need to explicitly install them:
+
+- Email::MIME::ContentType         deb: libemail-mime-contenttype-perl
+                                   pkg: p5-Email-MIME-ContentType
+                                   rpm: perl-Email-MIME-ContentType
+                                   (pulled in by Email::MIME)
+
+- Email::Simple                    deb: libemail-simple-perl
+                                   pkg: p5-Email-Simple
+                                   rpm: perl-Email-Simple
+                                   (pulled in by Email::MIME)
+
+- DBI                              deb: libdbi-perl
+                                   pkg: p5-DBI
+                                   rpm: perl-DBI
+                                   (pulled in by DBD::SQLite)
+
+- Filesys::Notify::Simple          deb: libfilesys-notify-simple-perl
+                                   pkg: pkg-Filesys-Notify-Simple
+                                   rpm: perl-Filesys-Notify-Simple
+                                   (for public-inbox-watch, pulled in by Plack)
+
+- IO::Compress::Gzip               deb: perl-modules (or libio-compress-perl)
+                                   rpm: perl-PerlIO-gzip
+                                   pkg: perl5
+                                   (for gzipped mbox over HTTP)
+
+Uncommonly needed modules:
+
+- Socket6                          deb: libsocket6-perl
+                                   pkg: p5-Socket6
+                                   rpm: perl-Socket6
+                                   (pulled in by SpamAssassin and Net::Server,
+                                    only necessary if using IPv6 with
+                                    Plack::Middleware::AccessLog or similar
+                                    on Perl <= 5.12)
+
+- Crypt::CBC                       deb: libcrypt-cbc-perl
+                                   pkg: p5-Crypt-CBC
+                                   (for PublicInbox::Unsubscribe (rarely used))
+
+Optional packages testing and development:
+
+- IPC::Run                         deb: libipc-run-perl
+                                   pkg: p5-IPC-Run
+                                   rpm: perl-IPC-Run
+
+- XML::Feed                        deb: libxml-feed-perl
+                                   pkg: p5-XML-Feed
+                                   rpm: perl-XML-Feed
+
+- Test::HTTP::Server::Simple       deb: libtest-http-server-simple-perl
+                                   pkg: p5-Test-HTTP-Server-Simple
+                                   rpm: perl-Test-HTTP-Server-Simple
+                                   (XXX is this really needed?)
 
 standard MakeMaker installation (Perl)
 --------------------------------------
@@ -137,8 +180,8 @@ public-inbox will never store unregeneratable data in Xapian
 or any other search database we might use; Xapian corruption
 will not destroy critical data.
 
-See the public-inbox-overview(7) man page for the next steps once the
-installation is complete.
+See the public-inbox-overview(7) man page for the next steps once
+the installation is complete.
 
 Copyright
 ---------
diff --git a/Makefile.PL b/Makefile.PL
index 3bb0072..b857667 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -29,10 +29,18 @@ WriteMakefile(
 		# Keep this sorted and synced to the INSTALL document
 		'Date::Parse' => 0,
 		'Email::MIME' => 0,
+
+		# the following should be pulled in by Email::MIME:
 		'Email::MIME::ContentType' => 0,
 		'Email::Simple' => 0,
+
+		# "libencode-perl" on Debian, `perl5' on FreeBSD
 		'Encode::MIME::Header' => 0,
 
+		# libperl$PERL_VERSION on Debian, `perl5' on FreeBSD,
+		# but Fedora seems to need this separately
+		'Devel::Peek' => 0,
+
 		# TODO: these should really be made optional...
 		'Plack' => 0,
 		'URI::Escape' => 0,
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
  2019-05-09  7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
@ 2019-05-09  7:50 ` Eric Wong
  2019-05-09  7:50 ` [PATCH 3/9] INSTALL: note xapian-compact(1) tool Eric Wong
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

Email::MIME uses Encode::MIME::Header and depends on that
appropriately; however we depend on other parts of the Encode
distribution, but that's bundled with Perl by upstream, anyways;
and should place no additional burden on users.
---
 INSTALL     | 9 +++++----
 Makefile.PL | 5 +++--
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/INSTALL b/INSTALL
index 99836c1..4574311 100644
--- a/INSTALL
+++ b/INSTALL
@@ -45,10 +45,6 @@ Beyond that, there is a long list of Perl modules required, starting with:
                                    pkg: p5-Email-MIME
                                    rpm: perl-Email-MIME
 
-* Encode::MIME::Header             deb: libencode-perl
-                                   pkg: perl5
-                                   rpm: perl-Encode
-
 * Plack                            deb: libplack-perl
                                    pkg: p5-Plack
                                    rpm: perl-Plack, perl-Plack-Test,
@@ -114,6 +110,11 @@ above, so there is no need to explicitly install them:
                                    rpm: perl-Email-Simple
                                    (pulled in by Email::MIME)
 
+* Encode                           deb: libperl5.$MINOR (or libencode-perl)
+                                   pkg: perl5
+                                   rpm: perl-Encode
+                                   (likely installed with Perl)
+
 - DBI                              deb: libdbi-perl
                                    pkg: p5-DBI
                                    rpm: perl-DBI
diff --git a/Makefile.PL b/Makefile.PL
index b857667..eda7319 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -34,8 +34,9 @@ WriteMakefile(
 		'Email::MIME::ContentType' => 0,
 		'Email::Simple' => 0,
 
-		# "libencode-perl" on Debian, `perl5' on FreeBSD
-		'Encode::MIME::Header' => 0,
+		# libperl$PERL_VERSION or libencode-perl on Debian,
+		# `perl5' on FreeBSD
+		'Encode' => 0,
 
 		# libperl$PERL_VERSION on Debian, `perl5' on FreeBSD,
 		# but Fedora seems to need this separately
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 3/9] INSTALL: note xapian-compact(1) tool
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
  2019-05-09  7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
  2019-05-09  7:50 ` [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header Eric Wong
@ 2019-05-09  7:50 ` Eric Wong
  2019-05-09  7:50 ` [PATCH 4/9] t/purge.t: fix unreferenced variable Eric Wong
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

The Perl Xapian bindings do not support compacting, yet,
so we rely on the xapian-compact tool.
---
 INSTALL | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/INSTALL b/INSTALL
index 4574311..313a295 100644
--- a/INSTALL
+++ b/INSTALL
@@ -97,6 +97,11 @@ Numerous optional modules are likely to be useful as well:
 * highlight                        deb: libhighlight-perl
                                    (for syntax highlighting with coderepo)
 
+* xapian-compact (tool)            deb: xapian-tools
+                                   pkg: xapian-core
+                                   rpm: xapian-core
+                                   (optional, for public-inbox-compact(1))
+
 The following modules are typically pulled in by dependencies listed
 above, so there is no need to explicitly install them:
 
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 4/9] t/purge.t: fix unreferenced variable
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (2 preceding siblings ...)
  2019-05-09  7:50 ` [PATCH 3/9] INSTALL: note xapian-compact(1) tool Eric Wong
@ 2019-05-09  7:50 ` Eric Wong
  2019-05-09  7:50 ` [PATCH 5/9] doc: use bullet list for wwwlisting options Eric Wong
                   ` (4 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

---
 t/purge.t | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/purge.t b/t/purge.t
index 9406005..b518c26 100644
--- a/t/purge.t
+++ b/t/purge.t
@@ -9,7 +9,7 @@ require_git(2.6);
 my @mods = qw(IPC::Run DBI DBD::SQLite Search::Xapian);
 foreach my $mod (@mods) {
 	eval "require $mod";
-	plan skip_all => "missing $_ for t/purge.t" if $@;
+	plan skip_all => "missing $mod for t/purge.t" if $@;
 };
 use Cwd qw(abs_path);
 my $purge = abs_path('blib/script/public-inbox-purge');
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 5/9] doc: use bullet list for wwwlisting options
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (3 preceding siblings ...)
  2019-05-09  7:50 ` [PATCH 4/9] t/purge.t: fix unreferenced variable Eric Wong
@ 2019-05-09  7:50 ` Eric Wong
  2019-05-09  7:51 ` [PATCH 6/9] doc: give txt files proper titles Eric Wong
                   ` (3 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:50 UTC (permalink / raw)
  To: meta

Otherwise, pod2man complains about "=item 404" not starting
with a letter and thinking it's part of a numbered list.
---
 Documentation/public-inbox-config.pod | 16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)

diff --git a/Documentation/public-inbox-config.pod b/Documentation/public-inbox-config.pod
index d44c8f3..db81bf1 100644
--- a/Documentation/public-inbox-config.pod
+++ b/Documentation/public-inbox-config.pod
@@ -241,19 +241,15 @@ Valid values are:
 
 =over 8
 
-=item all
+=item * all
+- Show all inboxes
 
-Show all inboxes
-
-=item 404
-
-Return a 404 page.  This is useful to allow customization with
+=item * 404
+- Return a 404 page.  This is useful to allow customization with
 L<Plack::App::Cascade(3pm)>
 
-=item match=domain
-
-Only show inboxes with URLs which belong to the domain of the HTTP
-request
+=item * match=domain
+- Only show inboxes with URLs which belong to the domain of the HTTP request
 
 =for TODO comment
 
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 6/9] doc: give txt files proper titles
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (4 preceding siblings ...)
  2019-05-09  7:50 ` [PATCH 5/9] doc: use bullet list for wwwlisting options Eric Wong
@ 2019-05-09  7:51 ` Eric Wong
  2019-05-09  7:51 ` [PATCH 7/9] Makefile.PL: improve make(1) portability Eric Wong
                   ` (2 subsequent siblings)
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:51 UTC (permalink / raw)
  To: meta

We can fix the redundant rule in include.mk which causes
make(1) on FreeBSD to complain; but HTML docs will likely
still require GNU make.
---
 Documentation/design_www.txt | 2 ++
 Documentation/hosted.txt     | 2 ++
 Documentation/include.mk     | 5 +++--
 Documentation/txt2pre        | 3 +--
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/Documentation/design_www.txt b/Documentation/design_www.txt
index cd715b2..f15a556 100644
--- a/Documentation/design_www.txt
+++ b/Documentation/design_www.txt
@@ -1,3 +1,5 @@
+PublicInbox::WWW (PSGI interface) design notes
+
 URL and anchor naming
 ---------------------
 
diff --git a/Documentation/hosted.txt b/Documentation/hosted.txt
index 42926d2..0b1fb92 100644
--- a/Documentation/hosted.txt
+++ b/Documentation/hosted.txt
@@ -1,3 +1,5 @@
+unofficially hosted mirrors at public-inbox.org
+
 In addition to eating our own dogfood at <https://public-inbox.org/meta/>,
 public-inbox.org hosts unofficial archives for several other projects
 as a service to their users and to further test our own software.
diff --git a/Documentation/include.mk b/Documentation/include.mk
index 02cbef3..9089e06 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -87,8 +87,9 @@ dtxt := $(addprefix Documentation/, $(dtxt)) $(mantxt)
 Documentation/standards.txt : Documentation/standards.perl
 	$(PERL) $< >$@+ && mv $@+ $@
 
-%.html: %.txt
-	TITLE="$(basename $(<F))" $(txt2pre)
+Documentation/%.html: Documentation/%.txt
+	$(txt2pre)
+
 %.html: %
 	$(txt2pre)
 
diff --git a/Documentation/txt2pre b/Documentation/txt2pre
index 4c4b2ca..9a81a24 100755
--- a/Documentation/txt2pre
+++ b/Documentation/txt2pre
@@ -11,8 +11,7 @@ use PublicInbox::Linkify;
 use PublicInbox::Hval qw(ascii_html);
 
 my $str = eval { local $/; <> };
-my $title = $ENV{TITLE};
-($title) = ($str =~ /\A([^\n]+)/) unless $title;
+my ($title) = ($str =~ /\A([^\n]+)/);
 $title = ascii_html($title);
 my $l = PublicInbox::Linkify->new;
 $str = $l->linkify_1($str);
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 7/9] Makefile.PL: improve make(1) portability
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (5 preceding siblings ...)
  2019-05-09  7:51 ` [PATCH 6/9] doc: give txt files proper titles Eric Wong
@ 2019-05-09  7:51 ` Eric Wong
  2019-05-09  7:51 ` [PATCH 8/9] doc/include.mk: remove git use and redundant declarations Eric Wong
  2019-05-09  7:51 ` [PATCH 9/9] ci: include some scripts to make CI easier Eric Wong
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:51 UTC (permalink / raw)
  To: meta

The `shell' function appears missing, so we'll rely on Bourne
shell expansioN, instead.

Use "$?" instead of "$<" since the latter is only specified
for inference and .DEFAULT rules, not target rules.

Tested on FreeBSD make(1) and bmake(1) on Debian.
---
 Makefile.PL | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile.PL b/Makefile.PL
index eda7319..6be913b 100644
--- a/Makefile.PL
+++ b/Makefile.PL
@@ -59,7 +59,7 @@ sub MY::postamble {
 EATMYDATA =
 -include config.mak
 -include Documentation/include.mk
-N ?= \$(shell echo \$\$(( \$\$(nproc 2>/dev/null || echo 2) + 1)))
+N = \$\$(( \$\$(nproc 2>/dev/null || gnproc 2>/dev/null || echo 2) + 1 ))
 SCRIPTS := scripts/ssoma-replay
 my_syntax := \$(addsuffix .syntax, $PM_FILES \$(EXE_FILES) \$(SCRIPTS))
 
@@ -70,13 +70,13 @@ my_syntax := \$(addsuffix .syntax, $PM_FILES \$(EXE_FILES) \$(SCRIPTS))
 syntax:: \$(my_syntax)
 
 check-manifest :: MANIFEST
-	if git ls-files >\$<.gen 2>&1; then diff -u \$< \$<.gen; fi
+	if git ls-files >\$?.gen 2>&1; then diff -u \$? \$?.gen; fi
 
 check:: pure_all check-manifest
 	\$(EATMYDATA) prove -lv -j\$(N)
 
 lib/PublicInbox/UserContent.pm :: contrib/css/216dark.css
-	@\$(PERL) -I lib \$@ \$<
+	\$(PERL) -I lib \$@ \$?
 
 EOF
 }
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 8/9] doc/include.mk: remove git use and redundant declarations
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (6 preceding siblings ...)
  2019-05-09  7:51 ` [PATCH 7/9] Makefile.PL: improve make(1) portability Eric Wong
@ 2019-05-09  7:51 ` Eric Wong
  2019-05-09  7:51 ` [PATCH 9/9] ci: include some scripts to make CI easier Eric Wong
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:51 UTC (permalink / raw)
  To: meta

And document that we still have GNU-isms in that
include.mk Makefile (and may continue to do so).
Finally, take advantage of GNU-isms to warn users
to run "gmake" to build all manpages.
---
 Documentation/include.mk | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/Documentation/include.mk b/Documentation/include.mk
index 9089e06..2a02611 100644
--- a/Documentation/include.mk
+++ b/Documentation/include.mk
@@ -2,10 +2,18 @@
 # License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
 all::
 
+# Note: some GNU-isms present and required to build docs
+# (including manpages), but at least this should not trigger
+# warnings with BSD make(1) when running "make check"
+# Maybe it's not worth it to support non-GNU make, though...
 RSYNC = rsync
 RSYNC_DEST = public-inbox.org:/srv/public-inbox/
-docs := README COPYING INSTALL TODO HACKING
-docs += $(shell git ls-files 'Documentation/*.txt')
+txt := INSTALL README COPYING TODO HACKING
+dtxt := design_notes.txt design_www.txt dc-dlvr-spam-flow.txt hosted.txt
+dtxt += standards.txt
+dtxt := $(addprefix Documentation/, $(dtxt))
+docs := $(txt) $(dtxt)
+
 INSTALL = install
 PODMAN = pod2man
 PODMAN_OPTS = -v --stderr -d 1993-10-02 -c 'public-inbox user manual'
@@ -71,6 +79,7 @@ manuals += $(m8)
 
 mantxt = $(addprefix Documentation/, $(addsuffix .txt, $(manuals)))
 docs += $(mantxt)
+dtxt += $(mantxt)
 
 all :: $(mantxt)
 
@@ -79,10 +88,6 @@ Documentation/%.txt : Documentation/%.pod
 
 txt2pre = $(PERL) -I lib ./Documentation/txt2pre <$< >$@+ && \
 	touch -r $< $@+ && mv $@+ $@
-txt := INSTALL README COPYING TODO
-dtxt := design_notes.txt design_www.txt dc-dlvr-spam-flow.txt hosted.txt
-dtxt += standards.txt
-dtxt := $(addprefix Documentation/, $(dtxt)) $(mantxt)
 
 Documentation/standards.txt : Documentation/standards.perl
 	$(PERL) $< >$@+ && mv $@+ $@
@@ -96,7 +101,10 @@ Documentation/%.html: Documentation/%.txt
 docs_html := $(addsuffix .html, $(subst .txt,,$(dtxt)) $(txt))
 html: $(docs_html)
 gz_docs := $(addsuffix .gz, $(docs) $(docs_html))
-rsync_docs := $(gz_docs) $(docs) $(txt) $(docs_html) $(dtxt)
+rsync_docs := $(gz_docs) $(docs) $(docs_html)
+
+doc: $(docs)
+
 %.gz: %
 	gzip -9 --rsyncable <$< >$@+
 	touch -r $< $@+
@@ -111,3 +119,7 @@ clean-doc:
 	$(RM) $(man1) $(man5) $(man7) $(gz_docs) $(docs_html) $(mantxt)
 
 clean :: clean-doc
+
+pure_all ::
+	@if test x"$(addprefix g, make)" != xgmake; then \
+	echo W: gmake is currently required to build manpages; fi
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

* [PATCH 9/9] ci: include some scripts to make CI easier
  2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
                   ` (7 preceding siblings ...)
  2019-05-09  7:51 ` [PATCH 8/9] doc/include.mk: remove git use and redundant declarations Eric Wong
@ 2019-05-09  7:51 ` Eric Wong
  8 siblings, 0 replies; 10+ messages in thread
From: Eric Wong @ 2019-05-09  7:51 UTC (permalink / raw)
  To: meta

This should make it easier to test a bunch of package
installation profiles across whatever OS isolation
one chooses (chroots, containers, jails, VMs).
---
 MANIFEST       |   4 +
 ci/README      |  33 +++++++
 ci/deps.perl   | 254 +++++++++++++++++++++++++++++++++++++++++++++++++
 ci/profiles.sh |  72 ++++++++++++++
 ci/run.sh      |  21 ++++
 5 files changed, 384 insertions(+)
 create mode 100644 ci/README
 create mode 100755 ci/deps.perl
 create mode 100755 ci/profiles.sh
 create mode 100755 ci/run.sh

diff --git a/MANIFEST b/MANIFEST
index da9e364..29cea10 100644
--- a/MANIFEST
+++ b/MANIFEST
@@ -27,6 +27,10 @@ MANIFEST
 Makefile.PL
 README
 TODO
+ci/README
+ci/deps.perl
+ci/profiles.sh
+ci/run.sh
 contrib/css/216dark.css
 contrib/css/216light.css
 contrib/css/README
diff --git a/ci/README b/ci/README
new file mode 100644
index 0000000..4687fbc
--- /dev/null
+++ b/ci/README
@@ -0,0 +1,33 @@
+various scripts for automated testing in chroots/VMs/jails
+
+TL;DR: ./ci/run.sh
+
+By default, `sudo' is used to install/uninstall packages.  It may be
+overridden with the `SUDO' environment variable.  These scripts should
+run in the top-level source tree, that is, as `./ci/run.sh'.
+
+* ci/run.sh - runs tests against all profiles for the current OS
+
+	Environment options may override behavior:
+
+	* DO - may be set to ":" to print commands instead of running
+	       default: "" (empty)
+
+	Common commands can be overridden by the environment, too:
+
+	* MAKE - default: "make"
+	* PERL - default: "perl"
+	* SUDO - default: "sudo"
+
+* ci/deps.perl - script to mass-install/remove packages (requires root/sudo)
+
+	Called automatically by ci/run.sh
+
+	There is no need to run this manually unless you are debugging
+	or doing development.  However, it can be convenient to for
+	users to mass-install several packages.
+
+* ci/profiles.sh - prints to-be tested package profile for the current OS
+
+	Called automatically by ci/run.sh
+	The output is read by ci/run.sh
diff --git a/ci/deps.perl b/ci/deps.perl
new file mode 100755
index 0000000..e0fda01
--- /dev/null
+++ b/ci/deps.perl
@@ -0,0 +1,254 @@
+#!/usr/bin/perl -w
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+# Helper script for installing/uninstalling packages for CI use
+# Intended for use on non-production chroots or VMs since it
+# changes installed packages
+use strict;
+my $usage = "$0 PKG_FMT PROFILE [PROFILE_MOD]";
+my $pkg_fmt = shift;
+@ARGV or die $usage, "\n";
+
+# package profiles
+my $profiles = {
+	# the smallest possible profile
+	# TODO: trim this, Plack pulls in Filesys::Notify::Simple,
+	# and we don't need that for mda-only installs
+	essential => [ qw(
+		git
+		perl
+		Date::Parse
+		Devel::Peek
+		Email::Simple
+		Email::MIME
+		Email::MIME::ContentType
+		Encode
+		Filesys::Notify::Simple
+		Plack
+		URI::Escape
+		) ],
+
+	# everything optional for normal use
+	optional => [ qw(
+		BSD::Resource
+		DBD::SQLite
+		DBI
+		IO::Compress::Gzip
+		Inline::C
+		Net::Server
+		Plack::Middleware::Deflater
+		Plack::Middleware::ReverseProxy
+		Search::Xapian
+		Socket6
+		highlight.pm
+		xapian-compact
+		) ],
+
+	# developer stuff
+	devtest => [ qw(
+		IPC::Run
+		Test::HTTP::Server::Simple
+		XML::Feed
+		curl
+		w3m
+		) ],
+};
+
+# account for granularity differences between package systems and OSes
+my @precious;
+if ($^O eq 'freebsd') {
+	@precious = qw(perl curl Socket6 IO::Compress::Gzip);
+}
+
+if (@precious) {
+	my $re = join('|', map { quotemeta($_) } @precious);
+	for my $list (values %$profiles) {
+		@$list = grep(!/\A(?:$re)\z/, @$list);
+	}
+	push @{$profiles->{essential}}, @precious;
+}
+
+
+# bare minimum for v2
+$profiles->{v2essential} = [ @{$profiles->{essential}}, qw(
+	DBD::SQLite
+	DBI
+	Search::Xapian
+	xapian-compact
+) ];
+
+# package names which can't be mapped automatically:
+my $non_auto = {
+	'perl' => { pkg => 'perl5' },
+	'Date::Parse' => {
+		deb => 'libtimedate-perl',
+		pkg => 'p5-TimeDate',
+		rpm => 'perl-TimeDate',
+	},
+	'Devel::Peek' => {
+		deb => 'perl', # libperl5.XX, but the XX varies
+		pkg => 'perl5',
+	},
+	'Encode' => {
+		deb => 'perl', # libperl5.XX, but the XX varies
+		pkg => 'perl5',
+		rpm => 'perl-Encode',
+	},
+	'IO::Compress::Gzip' => {
+		deb => 'perl', # perl-modules-5.xx
+		pkg => 'perl5',
+		rpm => 'perl-PerlIO-gzip',
+	},
+	'DBD::SQLite' => { deb => 'libdbd-sqlite3-perl' },
+	'URI::Escape' => {
+		deb => 'liburi-perl',
+		pkg => 'p5-URI',
+		rpm => 'perl-URI',
+	},
+	'highlight.pm' => {
+		deb => 'libhighlight-perl',
+		pkg => [],
+		rpm => [],
+	},
+
+	# we call xapian-compact(1) in public-inbox-compact(1)
+	'xapian-compact' => {
+		deb => 'xapian-tools',
+		pkg => 'xapian-core',
+		rpm => 'xapian-core', # ???
+	},
+
+	# OS-specific
+	'IO::KQueue' => {
+		deb => [],
+		pkg => 'p5-IO-KQueue',
+		rpm => [],
+	},
+};
+
+my (@pkg_install, @pkg_remove, %all);
+for my $ary (values %$profiles) {
+	$all{$_} = \@pkg_remove for @$ary;
+}
+if ($^O eq 'freebsd') {
+	$all{'IO::KQueue'} = \@pkg_remove;
+}
+$profiles->{all} = [ keys %all ]; # pseudo-profile for all packages
+
+# parse the profile list from the command-line
+for my $profile (@ARGV) {
+	if ($profile =~ s/-\z//) {
+		# like apt-get, trailing "-" means remove
+		profile2dst($profile, \@pkg_remove);
+	} else {
+		profile2dst($profile, \@pkg_install);
+	}
+}
+
+# fill in @pkg_install and @pkg_remove:
+while (my ($pkg, $dst_pkg_list) = each %all) {
+	push @$dst_pkg_list, list(pkg2ospkg($pkg, $pkg_fmt));
+}
+
+my @apt_opts =
+	qw(-o APT::Install-Recommends=false -o APT::Install-Suggests=false);
+
+# OS-specific cleanups appreciated
+
+if ($pkg_fmt eq 'deb') {
+	my @quiet = $ENV{V} ? () : ('-q');
+	root('apt-get', @apt_opts, qw(install --purge -y), @quiet,
+		@pkg_install,
+		# apt-get lets you suffix a package with "-" to
+		# remove it in an "install" sub-command:
+		map { "$_-" } @pkg_remove);
+	root('apt-get', @apt_opts, qw(autoremove --purge -y), @quiet);
+} elsif ($pkg_fmt eq 'pkg') {
+	my @quiet = $ENV{V} ? () : ('-q');
+	# FreeBSD, maybe other *BSDs are similar?
+
+	# don't remove stuff that isn't installed:
+	exclude_uninstalled(\@pkg_remove);
+	root(qw(pkg remove -y), @quiet, @pkg_remove) if @pkg_remove;
+	root(qw(pkg install -y), @quiet, @pkg_install) if @pkg_install;
+	root(qw(pkg autoremove -y), @quiet);
+# TODO: yum / rpm support
+} else {
+	die "unsupported package format: $pkg_fmt\n";
+}
+exit 0;
+
+
+# map a generic package name to an OS package name
+sub pkg2ospkg {
+	my ($pkg, $fmt) = @_;
+
+	# check explicit overrides, first:
+	if (my $ospkg = $non_auto->{$pkg}->{$fmt}) {
+		return $ospkg;
+	}
+
+	# check common Perl module name patterns:
+	if ($pkg =~ /::/ || $pkg =~ /\A[A-Z]/) {
+		if ($fmt eq 'deb') {
+			$pkg =~ s/::/-/g;
+			$pkg =~ tr/A-Z/a-z/;
+			return "lib$pkg-perl";
+		} elsif ($fmt eq 'rpm') {
+			$pkg =~ s/::/-/g;
+			return "perl-$pkg"
+		} elsif ($fmt eq 'pkg') {
+			$pkg =~ s/::/-/g;
+			return "p5-$pkg"
+		} else {
+			die "unsupported package format: $fmt for $pkg\n"
+		}
+	}
+
+	# use package name as-is (e.g. 'curl' or 'w3m')
+	$pkg;
+}
+
+# maps a install profile to a package list (@pkg_remove or @pkg_install)
+sub profile2dst {
+	my ($profile, $dst_pkg_list) = @_;
+	if (my $pkg_list = $profiles->{$profile}) {
+		$all{$_} = $dst_pkg_list for @$pkg_list;
+	} elsif ($all{$profile}) { # $profile is just a package name
+		$all{$profile} = $dst_pkg_list;
+	} else {
+		die "unrecognized profile or package: $profile\n";
+	}
+}
+
+sub exclude_uninstalled {
+	my ($list) = @_;
+	my %inst_check = (
+		pkg => sub { system(qw(pkg info -q), $_[0]) == 0 },
+		deb => sub { system("dpkg -s $_[0] >/dev/null 2>&1") == 0 },
+		rpm => sub { system("rpm -qs $_[0] >/dev/null 2>&1") == 0 },
+	);
+
+	my $cb = $inst_check{$pkg_fmt} || die <<"";
+don't know how to check install status for $pkg_fmt
+
+	my @tmp;
+	for my $pkg (@$list) {
+		push @tmp, $pkg if $cb->($pkg);
+	}
+	@$list = @tmp;
+}
+
+sub root {
+	print join(' ', @_), "\n";
+	return if $ENV{DRY_RUN};
+	return if system(@_) == 0;
+	warn 'command failed: ', join(' ', @_), "\n";
+	exit($? >> 8);
+}
+
+# ensure result can be pushed into an array:
+sub list {
+	my ($pkg) = @_;
+	ref($pkg) eq 'ARRAY' ? @$pkg : $pkg;
+}
diff --git a/ci/profiles.sh b/ci/profiles.sh
new file mode 100755
index 0000000..31140f3
--- /dev/null
+++ b/ci/profiles.sh
@@ -0,0 +1,72 @@
+#!/bin/sh
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+
+# Prints OS-specific package profiles to stdout (one per-newline) to use
+# as command-line args for ci/deps.perl.  Called automatically by ci/run.sh
+
+# set by os-release(5) or similar
+ID= VERSION_ID=
+case $(uname -o) in
+GNU/Linux)
+	for f in /etc/os-release /usr/lib/os-release
+	do
+		test -f $f || continue
+		. $f
+
+		# Debian sid (and testing) have no VERSION_ID
+		case $ID--$VERSION_ID in
+		debian--)
+			case $PRETTY_NAME in
+			*/sid) VERSION_ID=sid ;;
+			*)
+				echo >&2 "$ID, but no VERSION_ID"
+				echo >&2 "==> $f <=="
+				cat >&2 $f
+				exit 1
+				;;
+			esac
+			;;
+		esac
+
+		case $ID--$VERSION_ID in
+		-|*--|--*) continue ;;
+		*--*) break ;;
+		esac
+	done
+	;;
+FreeBSD)
+	ID=freebsd
+	VERSION_ID=$(uname -r | cut -d . -f 1)
+	test "$VERSION_ID" -lt 11 && {
+		echo >&2 "ID=$ID $(uname -r) too old to support";
+		exit 1
+	}
+esac
+
+case $ID in
+freebsd) PKG_FMT=pkg ;;
+debian|ubuntu) PKG_FMT=deb ;;
+centos|redhat|fedora) PKG_FMT=rpm ;;
+*) echo >&2 "PKG_FMT undefined for ID=$ID in $0"
+esac
+
+case $ID-$VERSION_ID in
+freebsd-11|freebsd-12) sed "s/^/$PKG_FMT /" <<EOF
+all devtest-
+all devtest IO::KQueue-
+all devtest IO::KQueue
+v2essential
+essential
+essential devtest-
+EOF
+	;;
+debian-sid|debian-9|debian-10) sed "s/^/$PKG_FMT /" <<EOF
+all devtest-
+all devtest
+v2essential
+essential
+essential devtest-
+EOF
+	;;
+esac
diff --git a/ci/run.sh b/ci/run.sh
new file mode 100755
index 0000000..87d9561
--- /dev/null
+++ b/ci/run.sh
@@ -0,0 +1,21 @@
+#!/bin/sh
+# Copyright (C) 2019 all contributors <meta@public-inbox.org>
+# License: AGPL-3.0+ <https://www.gnu.org/licenses/agpl-3.0.txt>
+set -e
+SUDO=${SUDO-'sudo'} PERL=${PERL-'perl'} MAKE=${MAKE-'make'}
+DO=${DO-''}
+
+set -x
+if test -f Makefile
+then
+	$DO $MAKE clean
+fi
+
+./ci/profiles.sh | while read args
+do
+	$DO $SUDO $PERL -w ci/deps.perl $args
+	$DO $PERL Makefile.PL
+	$DO $MAKE
+	$DO $MAKE check
+	$DO $MAKE clean
+done
-- 
EW


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, back to index

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-05-09  7:50 [PATCH 0/9] install updates Eric Wong
2019-05-09  7:50 ` [PATCH 1/9] INSTALL: update with FreeBSD pkg names and reorder Eric Wong
2019-05-09  7:50 ` [PATCH 2/9] INSTALL: depend on Encode instead of Encode::MIME::Header Eric Wong
2019-05-09  7:50 ` [PATCH 3/9] INSTALL: note xapian-compact(1) tool Eric Wong
2019-05-09  7:50 ` [PATCH 4/9] t/purge.t: fix unreferenced variable Eric Wong
2019-05-09  7:50 ` [PATCH 5/9] doc: use bullet list for wwwlisting options Eric Wong
2019-05-09  7:51 ` [PATCH 6/9] doc: give txt files proper titles Eric Wong
2019-05-09  7:51 ` [PATCH 7/9] Makefile.PL: improve make(1) portability Eric Wong
2019-05-09  7:51 ` [PATCH 8/9] doc/include.mk: remove git use and redundant declarations Eric Wong
2019-05-09  7:51 ` [PATCH 9/9] ci: include some scripts to make CI easier Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git