parse-datetime.y - Military Timezones are inverted from the correct sense

parse-datetime.y - Military Timezones are inverted from the correct sense

[Neil Hoggarth]

I have observed incorrect handling of "Military" timezones when
exercising the --date=... option of the GNU coreutils "date" utility.
I believe the underlying problem is with initialization of
"military_table[]" in the parse-datetime.y file of gnulib.

In Military Time, timezone letters "A" through "M" are supposed to
indicate zones progressively east of the prime meridian (local clocks
read some hours ahead of UTC), and "N" through "Y" progressively west
of the prime meridian (local clocks read some hours behind UTC), with
"Z" standing for the UTC+0 timezone. Reference:
https://aa.usno.navy.mil/faq/docs/world_tzones.php

Utilities using parse-datetime routines interpret correctly interpret
"Z" as UTC, but all the letters the wrong way around: zone "A" should
be one hour ahead of UTC (Paris, France when not observing DST for
example), but parse-datetime has it one hour behind:

$ TZ=UTC ./date --date='12:00A 2019-01-01'
Tue  1 Jan 13:00:00 UTC 2019
$ TZ=UTC ./date --date='12:00+0100 2019-01-01'
Tue  1 Jan 11:00:00 UTC 2019
$ TZ=UTC ./date --date='TZ="Europe/Paris" 1200 2019-01-01'
Tue  1 Jan 11:00:00 UTC 2019
$ TZ=UTC ./date --date='12:00CET 2019-01-01'
Tue  1 Jan 11:00:00 UTC 2019

Similarly, when not observing DST then US Eastern Standard Time is
5-hours behind UTC, in the "R for Romeo" timezone, but an "R" passed
to parse-datetime shifts the time 5 hours in the wrong direction:

$ TZ=UTC ./date --date='12:00R 2019-01-01'
Tue  1 Jan 07:00:00 UTC 2019
$ TZ=UTC ./date --date='12:00-0500 2019-01-01'
Tue  1 Jan 17:00:00 UTC 2019
$ TZ=UTC ./date --date='TZ="America/New_York" 12:00 2019-01-01'
Tue  1 Jan 17:00:00 UTC 2019
$ TZ=UTC ./date --date='12:00EST 2019-01-01'
Tue  1 Jan 17:00:00 UTC 2019

The following trivial change makes the routine and utilities using it
work as expected:

--- a/lib/parse-datetime.y
+++ b/lib/parse-datetime.y
@@ -1164,30 +1164,30 @@ static table const time_zone_table[] =
    8601 date and time of day representation.  */
 static table const military_table[] =
 {
-  { "A", tZONE, -HOUR ( 1) },
-  { "B", tZONE, -HOUR ( 2) },
-  { "C", tZONE, -HOUR ( 3) },
-  { "D", tZONE, -HOUR ( 4) },
-  { "E", tZONE, -HOUR ( 5) },
-  { "F", tZONE, -HOUR ( 6) },
-  { "G", tZONE, -HOUR ( 7) },
-  { "H", tZONE, -HOUR ( 8) },
-  { "I", tZONE, -HOUR ( 9) },
-  { "K", tZONE, -HOUR (10) },
-  { "L", tZONE, -HOUR (11) },
-  { "M", tZONE, -HOUR (12) },
-  { "N", tZONE,  HOUR ( 1) },
-  { "O", tZONE,  HOUR ( 2) },
-  { "P", tZONE,  HOUR ( 3) },
-  { "Q", tZONE,  HOUR ( 4) },
-  { "R", tZONE,  HOUR ( 5) },
-  { "S", tZONE,  HOUR ( 6) },
+  { "A", tZONE,  HOUR ( 1) },
+  { "B", tZONE,  HOUR ( 2) },
+  { "C", tZONE,  HOUR ( 3) },
+  { "D", tZONE,  HOUR ( 4) },
+  { "E", tZONE,  HOUR ( 5) },
+  { "F", tZONE,  HOUR ( 6) },
+  { "G", tZONE,  HOUR ( 7) },
+  { "H", tZONE,  HOUR ( 8) },
+  { "I", tZONE,  HOUR ( 9) },
+  { "K", tZONE,  HOUR (10) },
+  { "L", tZONE,  HOUR (11) },
+  { "M", tZONE,  HOUR (12) },
+  { "N", tZONE, -HOUR ( 1) },
+  { "O", tZONE, -HOUR ( 2) },
+  { "P", tZONE, -HOUR ( 3) },
+  { "Q", tZONE, -HOUR ( 4) },
+  { "R", tZONE, -HOUR ( 5) },
+  { "S", tZONE, -HOUR ( 6) },
   { "T", 'T',    0 },
-  { "U", tZONE,  HOUR ( 8) },
-  { "V", tZONE,  HOUR ( 9) },
-  { "W", tZONE,  HOUR (10) },
-  { "X", tZONE,  HOUR (11) },
-  { "Y", tZONE,  HOUR (12) },
+  { "U", tZONE, -HOUR ( 8) },
+  { "V", tZONE, -HOUR ( 9) },
+  { "W", tZONE, -HOUR (10) },
+  { "X", tZONE, -HOUR (11) },
+  { "Y", tZONE, -HOUR (12) },
   { "Z", tZONE,  HOUR ( 0) },
   { NULL, 0, 0 }
 };

Regards,

Neil Hoggarth

Re: parse-datetime.y - Military Timezones are inverted from the correct sense

[Assaf Gordon]

Hello,

On 2019-08-09 5:31 a.m., Neil Hoggarth wrote:
> I have observed incorrect handling of "Military" timezones when
> exercising the --date=... option of the GNU coreutils "date" utility.
> I believe the underlying problem is with initialization of
> "military_table[]" in the parse-datetime.y file of gnulib.
[...]
 >
> Utilities using parse-datetime routines interpret correctly interpret
> "Z" as UTC, but all the letters the wrong way around: zone "A" should
> be one hour ahead of UTC (Paris, France when not observing DST for
> example), but parse-datetime has it one hour behind:


Thank you for the report.

For reference,
it seems this change happened 20 years less 20 days ago,
in commit c6cb26ec2993800dda1b95c213fc4539acd4fa51 from August 29, 1999:

https://git.savannah.gnu.org/cgit/gnulib.git/commit/?id=c6cb26ec2

This change rewrote the "lib/getdate.y" file (the precursor to 
lib/parse-datetime.y) from the public domain version into
a more GNU-like style.

It contained the following changes:
-----
-/* Military timezone table. */
-static TABLE const MilitaryTable[] = {
-    { "a",     tZONE,  HOUR (  1) },
-    { "b",     tZONE,  HOUR (  2) },
-    { "c",     tZONE,  HOUR (  3) },
-    { "d",     tZONE,  HOUR (  4) },
-    { "e",     tZONE,  HOUR (  5) },
[....]
+/* Military time zone table. */
+static table const military_table[] =
  {
+  { "A", tZONE,        -HOUR ( 1) },
+  { "B", tZONE,        -HOUR ( 2) },
+  { "C", tZONE,        -HOUR ( 3) },
+  { "D", tZONE,        -HOUR ( 4) },
+  { "E", tZONE,        -HOUR ( 5) },
-----

Since it was in 1999, it predates all "coreutils" versions
(they were separate sh-utils,text-utils,file-utils packages).

On GNU's ftp server ( https://ftp.gnu.org/old-gnu/sh-utils/ ),
the last available version of sh-utils (version 2.0) from
August 15, 1999 had the positive (A=1) military time zones.
There are mentions (on the website and in coreutils' NEWS file)
of later version sh-tuils-2.0.15 which likely had negative (A=-1)
military time zones.

The first public version of the unified coreutils-5.0 from 2003
already used the negative values.

-assaf

Re: parse-datetime.y - Military Timezones are inverted from the correct sense

[Paul Eggert]

[-- Attachment #1: Type: text/plain, Size: 838 bytes --]

Thanks for mentioning this problem. The military timezone indicators 
(other than "Z") were purposely done backwards in 1999, to be compatible 
with Internet RFC 822 which also had them backwards and which was the 
standard for email at the time. RFC 5322, which is RFC 822's current 
version, says that because of the error these timezone indicators should 
all be treated as equivalent to "Z" now. So coreutils should be changed, 
either to follow the RFC 5322 recommendation, or to be compatible with 
traditional military use. The latter would make more sense since it is 
somewhat useful behavior and the RFC 5322 requirement is merely a 
SHOULD, not a MUST.

Since the RFC 822 error was fixed in 2001 when RFC 2822 came out, it is 
long past time to fix parse-datetime.y accordingly, so I installed the 
attached patch into Gnulib.

[-- Attachment #2: 0001-parse-datetime-fix-military-timezone-letters.patch --]
[-- Type: text/x-patch, Size: 3929 bytes --]

From 61bdaa139a38d5accdcca852bc58880966611335 Mon Sep 17 00:00:00 2001
From: Paul Eggert <eggert@cs.ucla.edu>
Date: Fri, 9 Aug 2019 13:47:41 -0700
Subject: [PATCH] parse-datetime: fix military timezone letters

Problem and trivial fix reported by Neil Hoggarth in:
https://lists.gnu.org/r/bug-gnulib/2019-08/msg00005.html
* lib/parse-datetime.y (military_table):
Do it the right way, not the RFC 822 way.
---
 ChangeLog               |  8 +++++++
 doc/parse-datetime.texi |  4 +++-
 lib/parse-datetime.y    | 49 ++++++++++++++++++++++-------------------
 3 files changed, 37 insertions(+), 24 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 35f870abe..7616b5efd 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,11 @@
+2019-08-09  Paul Eggert  <eggert@cs.ucla.edu>
+
+	parse-datetime: fix military timezone letters
+	Problem and trivial fix reported by Neil Hoggarth in:
+	https://lists.gnu.org/r/bug-gnulib/2019-08/msg00005.html
+	* lib/parse-datetime.y (military_table):
+	Do it the right way, not the RFC 822 way.
+
 2019-08-08  Eric Blake  <eblake@redhat.com>
 
 	configmake: Avoid namespace pollution issue on mingw.
diff --git a/doc/parse-datetime.texi b/doc/parse-datetime.texi
index 193575edc..2f1147572 100644
--- a/doc/parse-datetime.texi
+++ b/doc/parse-datetime.texi
@@ -306,7 +306,9 @@ only for @samp{UTC}; for example, @samp{UTC+05:30} is equivalent to
 Time zone items other than @samp{UTC} and @samp{Z}
 are obsolescent and are not recommended, because they
 are ambiguous; for example, @samp{EST} has a different meaning in
-Australia than in the United States.  Instead, it's better to use
+Australia than in the United States, and @samp{A} has different
+meaning as a military time zone than as an obsolescent
+RFC 822 time zone.  Instead, it's better to use
 unambiguous numeric time zone corrections like @samp{-0500}, as
 described in the previous section.
 
diff --git a/lib/parse-datetime.y b/lib/parse-datetime.y
index 780575919..d371b9cb1 100644
--- a/lib/parse-datetime.y
+++ b/lib/parse-datetime.y
@@ -1160,34 +1160,37 @@ static table const time_zone_table[] =
 
 /* Military time zone table.
 
+   RFC 822 got these backwards, but RFC 5322 makes the incorrect
+   treatment optional, so do them the right way here.
+
    Note 'T' is a special case, as it is used as the separator in ISO
    8601 date and time of day representation.  */
 static table const military_table[] =
 {
-  { "A", tZONE, -HOUR ( 1) },
-  { "B", tZONE, -HOUR ( 2) },
-  { "C", tZONE, -HOUR ( 3) },
-  { "D", tZONE, -HOUR ( 4) },
-  { "E", tZONE, -HOUR ( 5) },
-  { "F", tZONE, -HOUR ( 6) },
-  { "G", tZONE, -HOUR ( 7) },
-  { "H", tZONE, -HOUR ( 8) },
-  { "I", tZONE, -HOUR ( 9) },
-  { "K", tZONE, -HOUR (10) },
-  { "L", tZONE, -HOUR (11) },
-  { "M", tZONE, -HOUR (12) },
-  { "N", tZONE,  HOUR ( 1) },
-  { "O", tZONE,  HOUR ( 2) },
-  { "P", tZONE,  HOUR ( 3) },
-  { "Q", tZONE,  HOUR ( 4) },
-  { "R", tZONE,  HOUR ( 5) },
-  { "S", tZONE,  HOUR ( 6) },
+  { "A", tZONE,  HOUR ( 1) },
+  { "B", tZONE,  HOUR ( 2) },
+  { "C", tZONE,  HOUR ( 3) },
+  { "D", tZONE,  HOUR ( 4) },
+  { "E", tZONE,  HOUR ( 5) },
+  { "F", tZONE,  HOUR ( 6) },
+  { "G", tZONE,  HOUR ( 7) },
+  { "H", tZONE,  HOUR ( 8) },
+  { "I", tZONE,  HOUR ( 9) },
+  { "K", tZONE,  HOUR (10) },
+  { "L", tZONE,  HOUR (11) },
+  { "M", tZONE,  HOUR (12) },
+  { "N", tZONE, -HOUR ( 1) },
+  { "O", tZONE, -HOUR ( 2) },
+  { "P", tZONE, -HOUR ( 3) },
+  { "Q", tZONE, -HOUR ( 4) },
+  { "R", tZONE, -HOUR ( 5) },
+  { "S", tZONE, -HOUR ( 6) },
   { "T", 'T',    0 },
-  { "U", tZONE,  HOUR ( 8) },
-  { "V", tZONE,  HOUR ( 9) },
-  { "W", tZONE,  HOUR (10) },
-  { "X", tZONE,  HOUR (11) },
-  { "Y", tZONE,  HOUR (12) },
+  { "U", tZONE, -HOUR ( 8) },
+  { "V", tZONE, -HOUR ( 9) },
+  { "W", tZONE, -HOUR (10) },
+  { "X", tZONE, -HOUR (11) },
+  { "Y", tZONE, -HOUR (12) },
   { "Z", tZONE,  HOUR ( 0) },
   { NULL, 0, 0 }
 };
-- 
2.21.0

Re: parse-datetime.y - Military Timezones are inverted from the correct sense

[Assaf Gordon]

[-- Attachment #1: Type: text/plain, Size: 351 bytes --]

Hello,

On Fri, Aug 09, 2019 at 02:01:35PM -0700, Paul Eggert wrote:
> Since the RFC 822 error was fixed in 2001 when RFC 2822 came out, it is long
> past time to fix parse-datetime.y accordingly, so I installed the attached
> patch into Gnulib.

This results in a user-visible change for gnu date,
I suggest the attached patch for coreutils.

-assaf

[-- Attachment #2: date-mention-military-timezone-changes-from-gnulib.patch --]
[-- Type: text/plain, Size: 2966 bytes --]

From 19f7eab06af234641a2927514c03570c07a311db Mon Sep 17 00:00:00 2001
From: Assaf Gordon <assafgordon@gmail.com>
Date: Fri, 9 Aug 2019 19:51:42 -0600
Subject: [PATCH 1/2] gnulib: update to latest

---
 gnulib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gnulib b/gnulib
index c7d0b4506..f1f10d47b 160000
--- a/gnulib
+++ b/gnulib
@@ -1 +1 @@
-Subproject commit c7d0b4506574887be5835ae9ae892d365afbb98c
+Subproject commit f1f10d47be8762e4ca17c8957a0520b08d28abfb
-- 
2.20.1


From 6eb1118f00a7018f08f69c7ace86cd92f89ca961 Mon Sep 17 00:00:00 2001
From: Assaf Gordon <assafgordon@gmail.com>
Date: Fri, 9 Aug 2019 20:16:06 -0600
Subject: [PATCH 2/2] date: mention military timezone changes from gnulib

Gnulib commit f1f10d47be8762e4ca17c8957a0520b08d28abfb (based on
https://lists.gnu.org/r/bug-gnulib/2019-08/msg00005.html) negated the
meaning of military timezones parsed in gnu date.

* NEWS: Mention this user-visible change.
* tests/misc/date.pl: Add tests for the new behavior.
---
 NEWS               | 16 ++++++++++++++++
 tests/misc/date.pl |  9 +++++++++
 2 files changed, 25 insertions(+)

diff --git a/NEWS b/NEWS
index 97c9d18bd..d4904d20b 100644
--- a/NEWS
+++ b/NEWS
@@ -49,6 +49,22 @@ GNU coreutils NEWS                                    -*- outline -*-
   coherency of file system attributes, useful on network file systems.
 
 
+** Changes in behavior
+
+  date now parses military time zones in accordance with rfc5322:
+    "A" to "M"  are equivalent to UTC+1 to UTC+12
+    "N" to "S"  are equivalent to UTC-1 to UTC-6
+    "U" to "Y"  are equivalent to UTC-8 to UTC-12
+    "T" is parsed as a ISO-8601 format representation,
+        and should not be used for military time zones in gnu date.
+    "Z" is "zulu" time (UTC).
+  For example, 'date -d "09:00B" is now equivalent to 9am in UTC+2 time zone.
+  Previously, military time zones were parsed according to the obsolete
+  rfc822, with their value negated (e.g., "B" was equivalent to UTC-2).
+  [The old behavior was introduced in sh-utils 2.0.15 ca. 1999, predating
+  coreutils package.]
+
+
 * Noteworthy changes in release 8.31 (2019-03-10) [stable]
 
 ** Bug fixes
diff --git a/tests/misc/date.pl b/tests/misc/date.pl
index 9ba3d3983..e11753347 100755
--- a/tests/misc/date.pl
+++ b/tests/misc/date.pl
@@ -300,6 +300,15 @@ my @Tests =
 
      # https://bugs.gnu.org/34608
      ['date-century-plus', '-d @0 +.%+4C.', {OUT => '.+019.'}],
+
+
+     # Military time zones, new behavior (since 8.32)
+     # https://lists.gnu.org/r/bug-gnulib/2019-08/msg00005.html
+     ['mtz1', '-u -d "09:00B" +%T', {OUT => '07:00:00'}],
+     ['mtz2', '-u -d "09:00L" +%T', {OUT => '22:00:00'}],
+     ['mtz3', '-u -d "09:00N" +%T', {OUT => '10:00:00'}],
+     ['mtz4', '-u -d "09:00X" +%T', {OUT => '20:00:00'}],
+     ['mtz5', '-u -d "09:00Z" +%T', {OUT => '09:00:00'}],
     );
 
 # Repeat the cross-dst test, using Jan 1, 2005 and every interval from 1..364.
-- 
2.20.1

Re: parse-datetime.y - Military Timezones are inverted from the correct sense

[Paul Eggert]

Assaf Gordon wrote:
> I suggest the attached patch for coreutils.

OK, except I'd remove "in accordance with rfc5322" since RFC 5322 recommends 
treating all these zones as if they were UTC. Also, "T" continues to have its 
military meaning (i.e., between "S" and "U") if it's used properly.