announce-gen and OpenPGP key servers

announce-gen and OpenPGP key servers

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 2178 bytes --]

Hi.  Our announce-gen contains:

  If that command fails because you don't have the required public key,
  then run this command to import it:
  gpg --keyserver keys.gnupg.net --recv-keys $gpg_key_id

Given recent OpenPGP key server issues, that doesn't work reliably any
more, and behave different for different GnuPG versions.  What should we
recommend instead?  Werner Koch said:

https://lists.gnupg.org/pipermail/gnupg-devel/2021-July/034937.html

I like WKD, but not all of us has published their OpenPGP key there, and
some may never be able to (it requires that you can put a file on your
e-mail domains' https server).  Still, I think it is the best long-term
solution.

How about the patch below?  It is not meant to be commited, but to start
discussion.

I think we should do more than the patch.  The OpenPGP web of trust
seems to be under attack and is not as usable any more.

Our announcements doesn't contain the full OpenPGP key fingerprint,
which they should.

The release announcement could include hash checksums of the files too.

Some of us publish our OpenPGP keys at a https URL, and including that
link in the announcement would also help.  That could point to the
Savannah PGP page, but I think few of us keep that maintained and the
URL looks horrible.

Maybe we should involve the ftp-upload@gnu.org people.  Having the
OpenPGP key database they use be published on gnu.org would help.

Let's discuss and see what we can do.

/Simon

diff --git a/build-aux/announce-gen b/build-aux/announce-gen
index daa478c8e..a696bff89 100755
--- a/build-aux/announce-gen
+++ b/build-aux/announce-gen
@@ -549,7 +549,12 @@ then run this command to import it:
 
   gpg --keyserver keys.gnupg.net --recv-keys $gpg_key_id
 
-and rerun the 'gpg --verify' command.
+You may also try other key servers such as keyserver.ubuntu.com or
+pgp.mit.edu.  With newer GnuPG versions you may use the following
+command to download and refresh any expired key:
+
+  gpg --auto-key-locate=clear,wkd,nodefault --locate-key simon@josefsson.org
+
 EOF
 
   my @tool_versions = get_tool_versions (\@tool_list, $gnulib_version);

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: announce-gen and OpenPGP key servers

[Paul Eggert]

I agree that the current situation is bad and that your suggestions 
would be improvement.

Particularly the part about involving the ftp-upload people. My 
impression is that they're quite conservative about changing things (and 
rightly so) but we really need a more-reliable distribution mechanism, 
given all the bad actors out there.

Re: announce-gen and OpenPGP key servers

[Jim Meyering]

On Tue, Jul 27, 2021 at 2:38 AM Simon Josefsson via Gnulib discussion
list <bug-gnulib@gnu.org> wrote:
> Hi.  Our announce-gen contains:
>
>   If that command fails because you don't have the required public key,
>   then run this command to import it:
>   gpg --keyserver keys.gnupg.net --recv-keys $gpg_key_id
>
> Given recent OpenPGP key server issues, that doesn't work reliably any
> more, and behave different for different GnuPG versions.  What should we
> recommend instead?  Werner Koch said:
>
> https://lists.gnupg.org/pipermail/gnupg-devel/2021-July/034937.html
>
> I like WKD, but not all of us has published their OpenPGP key there, and
> some may never be able to (it requires that you can put a file on your
> e-mail domains' https server).  Still, I think it is the best long-term
> solution.
>
> How about the patch below?  It is not meant to be commited, but to start
> discussion.
>
> I think we should do more than the patch.  The OpenPGP web of trust
> seems to be under attack and is not as usable any more.
>
> Our announcements doesn't contain the full OpenPGP key fingerprint,
> which they should.
>
> The release announcement could include hash checksums of the files too.
>
> Some of us publish our OpenPGP keys at a https URL, and including that
> link in the announcement would also help.  That could point to the
> Savannah PGP page, but I think few of us keep that maintained and the
> URL looks horrible.
>
> Maybe we should involve the ftp-upload@gnu.org people.  Having the
> OpenPGP key database they use be published on gnu.org would help.
>
> Let's discuss and see what we can do.
>
> /Simon
>
> diff --git a/build-aux/announce-gen b/build-aux/announce-gen
> index daa478c8e..a696bff89 100755
> --- a/build-aux/announce-gen
> +++ b/build-aux/announce-gen
> @@ -549,7 +549,12 @@ then run this command to import it:
>
>    gpg --keyserver keys.gnupg.net --recv-keys $gpg_key_id
>
> -and rerun the 'gpg --verify' command.
> +You may also try other key servers such as keyserver.ubuntu.com or
> +pgp.mit.edu.  With newer GnuPG versions you may use the following
> +command to download and refresh any expired key:
> +
> +  gpg --auto-key-locate=clear,wkd,nodefault --locate-key simon@josefsson.org

I've just run that, and it failed like this:

  gpg: error retrieving 'simon@josefsson.org' via WKD: General error

I too agree. We must make changes to improve matters.
I was rather dismayed to see recently how hard it was to find a usable
keyserver.

Feel free to make the script generate a full fingerprint and even
(though it feels a little like giving up) add a checksum or two.

Re: announce-gen and OpenPGP key servers

[Bernhard Voelker]

On 7/27/21 11:38 AM, Simon Josefsson via Gnulib discussion list wrote:
> Let's discuss and see what we can do.
Isn't this what the "release GPG keys" on Savannah are for?

Each project maintainer can set them up correctly under "Edit public info":
  "https://savannah.gnu.org/project/admin/editgroupinfo.php?group=${PROJECT}"

The result can be downloaded by users for verification:
  "https://savannah.gnu.org/project/release-gpgkeys.php?group=${PROJECT}&download=1"
e.g. coreutils':
  "https://savannah.gnu.org/project/release-gpgkeys.php?group=coreutils&download=1"

Downstream sometimes use them, e.g. SUSE and openSUSE are verifying the keys
on their Open Build Service.

Have a nice day,
Berny

Re: announce-gen and OpenPGP key servers

[Simon Josefsson via Gnulib discussion list]

sön 2021-08-01 klockan 17:47 +0200 skrev Bernhard Voelker:
> On 7/27/21 11:38 AM, Simon Josefsson via Gnulib discussion list
> wrote:
> > Let's discuss and see what we can do.
> Isn't this what the "release GPG keys" on Savannah are for?
> 
> Each project maintainer can set them up correctly under "Edit public
> info":
>   "
> https://savannah.gnu.org/project/admin/editgroupinfo.php?group=${PROJECT}
> "
> 
> The result can be downloaded by users for verification:
>   "
> https://savannah.gnu.org/project/release-gpgkeys.php?group=${PROJECT}&download=1
> "
> e.g. coreutils':
>   "
> https://savannah.gnu.org/project/release-gpgkeys.php?group=coreutils&download=1
> "
> 
> Downstream sometimes use them, e.g. SUSE and openSUSE are verifying
> the keys on their Open Build Service.

Right, thanks for pointing that out.  It can be used in some
situations.  One concern is that the set of keys trusted for uploads to
ftp.gnu.org is a a different set of keys, and not all GNU projects use
Savannah.

/Simon

Re: announce-gen and OpenPGP key servers

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1.1: Type: text/plain, Size: 1478 bytes --]

Jim Meyering <jim@meyering.net> writes:

> Feel free to make the script generate a full fingerprint and even
> (though it feels a little like giving up) add a checksum or two.

I think checksums still serve a purpose.

Many announcement e-mails are OpenPGP signed (and sometimes with a
different key than the release tarballs, thus creating another way to
verify tarballs).

Checksums also makes it harder to replace the tarball on the server with
a fake (or, after a key compromise, a genuine) signature.

I don't think it is a either-or situation, but rather a
belt-and-suspender case.  Ideally, people downloading a release should
verify both the signature (to know it comes from a trusted origin) and
checksum (to know it is the intended release, in case multiple signed
versions co-exists).

The patches below make the maintainer-makefile announcements contain
SHA1 and B64(SHA256) checksums by default.  The MD5 checksums are
dropped; they are completely insecure now.  The B64(SHA256) output is
inspired by OpenSSH which started this practice with release 6.5 in 2014
and still today prints similar outputs, see:

https://www.openssh.com/txt/release-6.5
https://www.openssh.com/txt/release-8.6

Unfortunately, 'sha256sum' can't verify these outputs, but I recall
earlier discussions around 'sha256sum --base64' so I will resume work on
that.

We could opt to simply use the "standard" sha256sum output instead, if
people here don't like the base64 output format.

/Simon

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-announce-gen-Print-SHA1-B64-SHA256-instead-of-MD5-SH.patch --]
[-- Type: text/x-diff, Size: 3999 bytes --]

From 4adae938b8dbe01750698109bcbf5f1c9eb045b1 Mon Sep 17 00:00:00 2001
From: Simon Josefsson <simon@josefsson.org>
Date: Tue, 3 Aug 2021 17:15:16 +0200
Subject: [PATCH 1/2] announce-gen: Print SHA1/B64(SHA256) instead of MD5/SHA1.

* build-aux/announce-gen (%digest_classes): Removed.
(usage): Doc fix.
(print_checksums): Instead of MD5/SHA1, print SHA1 and
B64(SHA256), inspired by OpenSSH announcements.
---
 ChangeLog              |  8 ++++++++
 build-aux/announce-gen | 33 +++++++++++----------------------
 2 files changed, 19 insertions(+), 22 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 06f139a54..079a5b71c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2021-08-03  Simon Josefsson  <simon@josefsson.org>
+
+	announce-gen: Print SHA1/B64(SHA256) instead of MD5/SHA1.
+	* build-aux/announce-gen (%digest_classes): Removed.
+	(usage): Doc fix.
+	(print_checksums): Instead of MD5/SHA1, print SHA1 and
+	B64(SHA256), inspired by OpenSSH announcements.
+
 2021-08-02  Paul Eggert  <eggert@cs.ucla.edu>
 
 	manywarnings: enable some malloc warnings
diff --git a/build-aux/announce-gen b/build-aux/announce-gen
index daa478c8e..b07cbd742 100755
--- a/build-aux/announce-gen
+++ b/build-aux/announce-gen
@@ -35,7 +35,7 @@
 eval 'exec perl -wSx "$0" "$@"'
      if 0;
 
-my $VERSION = '2021-04-11 8:42'; # UTC
+my $VERSION = '2021-08-03 15:13'; # UTC
 # The definition above must lie within the first 8 lines in order
 # for the Emacs time-stamp write hook (at end) to update it.
 # If you change this file with Emacs, please let the write hook
@@ -51,12 +51,6 @@ use POSIX qw(strftime);
 
 my %valid_release_types = map {$_ => 1} qw (alpha beta stable);
 my @archive_suffixes = qw (tar.gz tar.bz2 tar.lz tar.lzma tar.xz);
-my %digest_classes =
-  (
-   'md5' => (eval { require Digest::MD5; } and 'Digest::MD5'),
-   'sha1' => ((eval { require Digest::SHA; } and 'Digest::SHA')
-              or (eval { require Digest::SHA1; } and 'Digest::SHA1'))
-  );
 my $srcdir = '.';
 
 sub usage ($)
@@ -96,7 +90,7 @@ The following are optional:
                                 VERSION is the result of running git describe
                                 in the gnulib source directory.
                                 required if gnulib is in TOOL_LIST.
-   --no-print-checksums         do not emit MD5 or SHA1 checksums
+   --no-print-checksums         do not emit SHA1 or SHA256 checksums
    --archive-suffix=SUF         add SUF to the list of archive suffixes
    --mail-headers=HEADERS       a space-separated list of mail headers, e.g.,
                                 To: x\@example.com Cc: y-announce\@example.com,...
@@ -163,7 +157,7 @@ sub print_locations ($\@\%@)
 
 =item C<print_checksums (@file)
 
-Print the MD5 and SHA1 signature section for each C<@file>.
+Print the SHA1 and SHA256 signature section for each C<@file>.
 
 =cut
 
@@ -171,23 +165,18 @@ sub print_checksums (@)
 {
   my (@file) = @_;
 
-  print "Here are the MD5 and SHA1 checksums:\n";
+  print "Here are the SHA1 and SHA256 checksums:\n";
   print "\n";
 
-  foreach my $meth (qw (md5 sha1))
+  use Digest::file qw(digest_file_hex digest_file_base64);
+
+  foreach my $f (@file)
     {
-      my $class = $digest_classes{$meth} or next;
-      foreach my $f (@file)
-        {
-          open IN, '<', $f
-            or die "$ME: $f: cannot open for reading: $!\n";
-          binmode IN;
-          my $dig = $class->new->addfile(*IN)->hexdigest;
-          close IN;
-          print "$dig  $f\n";
-        }
+      print digest_file_hex($f, "SHA-1"), "  $f\n";
+      print digest_file_base64($f, "SHA-256"), "  $f\n";
     }
-  print "\n";
+  print "\nPlease note that the SHA256 checksum is base64 encoded and not\n";
+  print "hexadecimal (which is the default for most checksum tools).\n\n";
 }
 
 =item C<print_news_deltas ($news_file, $prev_version, $curr_version)
-- 
2.30.2


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.3: 0002-maintainer-makefile-Print-checksums-by-default.patch --]
[-- Type: text/x-diff, Size: 1305 bytes --]

From 3ace7783656f3e38b6db4e44881959116e581a2b Mon Sep 17 00:00:00 2001
From: Simon Josefsson <simon@josefsson.org>
Date: Tue, 3 Aug 2021 17:16:42 +0200
Subject: [PATCH 2/2] maintainer-makefile: Print checksums by default.

* top/maint.mk (announcement): Drop --no-print-checksums.
---
 ChangeLog    | 3 +++
 top/maint.mk | 1 -
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/ChangeLog b/ChangeLog
index 079a5b71c..cb65d202b 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -6,6 +6,9 @@
 	(print_checksums): Instead of MD5/SHA1, print SHA1 and
 	B64(SHA256), inspired by OpenSSH announcements.
 
+	maintainer-makefile: Print checksums by default.
+	* top/maint.mk (announcement): Drop --no-print-checksums.
+
 2021-08-02  Paul Eggert  <eggert@cs.ucla.edu>
 
 	manywarnings: enable some malloc warnings
diff --git a/top/maint.mk b/top/maint.mk
index 044254bdc..6a3ea9606 100644
--- a/top/maint.mk
+++ b/top/maint.mk
@@ -1426,7 +1426,6 @@ announcement: NEWS ChangeLog $(rel-files)
 	    --bootstrap-tools=$(bootstrap-tools)			\
 	    $$(case ,$(bootstrap-tools), in (*,gnulib,*)		\
 	       echo --gnulib-version=$(gnulib-version);; esac)		\
-	    --no-print-checksums					\
 	    $(addprefix --url-dir=, $(url_dir_list))
 
 .PHONY: release-commit
-- 
2.30.2


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: announce-gen and OpenPGP key servers

[Jim Meyering]

On Tue, Aug 3, 2021 at 8:40 AM Simon Josefsson <simon@josefsson.org> wrote:
> Jim Meyering <jim@meyering.net> writes:
>
> > Feel free to make the script generate a full fingerprint and even
> > (though it feels a little like giving up) add a checksum or two.
>
> I think checksums still serve a purpose.
>
> Many announcement e-mails are OpenPGP signed (and sometimes with a
> different key than the release tarballs, thus creating another way to
> verify tarballs).
>
> Checksums also makes it harder to replace the tarball on the server with
> a fake (or, after a key compromise, a genuine) signature.
>
> I don't think it is a either-or situation, but rather a
> belt-and-suspender case.  Ideally, people downloading a release should

Agreed.

> verify both the signature (to know it comes from a trusted origin) and
> checksum (to know it is the intended release, in case multiple signed
> versions co-exists).
>
> The patches below make the maintainer-makefile announcements contain
> SHA1 and B64(SHA256) checksums by default.  The MD5 checksums are
> dropped; they are completely insecure now.  The B64(SHA256) output is
> inspired by OpenSSH which started this practice with release 6.5 in 2014
> and still today prints similar outputs, see:
>
> https://www.openssh.com/txt/release-6.5
> https://www.openssh.com/txt/release-8.6
>
> Unfortunately, 'sha256sum' can't verify these outputs, but I recall
> earlier discussions around 'sha256sum --base64' so I will resume work on
> that.
>
> We could opt to simply use the "standard" sha256sum output instead, if
> people here don't like the base64 output format.

Thanks, Simon! I too am all for B64-formatted checksums.
You may want to coordinate with Pádraig.
I think he is planning a unification of the checksum-generating tools.
Your patches look fine. One nit: please drop the "Please" here :-)

+  print "\nPlease note that the SHA256 checksum is base64 encoded and not\n";

Re: announce-gen and OpenPGP key servers

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 834 bytes --]

Jim Meyering <jim@meyering.net> writes:

> Thanks, Simon! I too am all for B64-formatted checksums.

Good, it is a trade-off between output readability and code complexity.
Aligning 'sha*sum' with OpenBSD's 'sha*' tools would be nice, and base64
support is one missing piece.

> You may want to coordinate with Pádraig.
> I think he is planning a unification of the checksum-generating tools.

Okay!

> Your patches look fine. One nit: please drop the "Please" here :-)
>
> +  print "\nPlease note that the SHA256 checksum is base64 encoded and not\n";

Thanks, I reworded it a bit further, but I'm not a native speaker so I
appreciate improvements from others.

+  print "\nThe SHA256 checksum is base64 encoded and not hexadecimal,\n";
+  print "which is the default for most checksum tools.\n\n";


/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: announce-gen and OpenPGP key servers

[Paul Eggert]

On 8/3/21 12:20 PM, Simon Josefsson via Gnulib discussion list wrote:
> +  print "\nThe SHA256 checksum is base64 encoded and not hexadecimal,\n";
> +  print "which is the default for most checksum tools.\n\n";

Perhaps this?

   print "\nThe SHA256 checksum is base64 encoded, instead of the\n";
   print "hexadecimal encoding that most checksum tools default to.\n\n";

Re: announce-gen and OpenPGP key servers

[Jim Meyering]

On Tue, Aug 3, 2021 at 12:25 PM Paul Eggert <eggert@cs.ucla.edu> wrote:
> On 8/3/21 12:20 PM, Simon Josefsson via Gnulib discussion list wrote:
> > +  print "\nThe SHA256 checksum is base64 encoded and not hexadecimal,\n";
> > +  print "which is the default for most checksum tools.\n\n";
>
> Perhaps this?
>
>    print "\nThe SHA256 checksum is base64 encoded, instead of the\n";
>    print "hexadecimal encoding that most checksum tools default to.\n\n";

Thanks. That is clearer, indeed.

Re: announce-gen and OpenPGP key servers

[Simon Josefsson via Gnulib discussion list]

tis 2021-08-03 klockan 16:51 -0700 skrev Jim Meyering:
> On Tue, Aug 3, 2021 at 12:25 PM Paul Eggert <eggert@cs.ucla.edu>
> wrote:
> > On 8/3/21 12:20 PM, Simon Josefsson via Gnulib discussion list
> > wrote:
> > > +  print "\nThe SHA256 checksum is base64 encoded and not
> > > hexadecimal,\n";
> > > +  print "which is the default for most checksum tools.\n\n";
> > 
> > Perhaps this?
> > 
> >    print "\nThe SHA256 checksum is base64 encoded, instead of
> > the\n";
> >    print "hexadecimal encoding that most checksum tools default
> > to.\n\n";
> 
> Thanks. That is clearer, indeed.

Pushed, thanks.

/Simon