localtime on native Windows

localtime on native Windows

[Bruno Haible]

[-- Attachment #1: Type: text/plain, Size: 1802 bytes --]

I wrote:
> For these modules, the next function to provide in an MT-safe way is
> localtime_r.

Our gmtime_r and localtime_r are MT-safe on native Windows. I ran the
test-gmtime_r-mt and test-localtime_r-mt tests for 2000 seconds each, and
they did not crash.

But the problem is that localtime() and localtime_r() on native Windows
produce nonsensical results:
  - They pretend that in France, in 2007, DST began on 2007-03-11. When in
    fact, it started on 2007-03-25.
  - The hour is wrong.
Witness: The attached program loc.c.

> On native Windows, when the 'localtime_s' function [1][2]
> is not available, such as on the older Windows versions that Emacs cares
> about, the solution is to use GetTimeZoneInformation [3].

None of the GetTimeZoneInformation APIs from the Windows DLLs works either.
They pretend that in German and French time zones, DST starts on March 5,
in all years. Witness: The attached program tzi.c.

So, there is no way around implementing a correct localtime_r, based on
tzdata, in Gnulib.

It will be useful
  - for localtime_r on native Windows,
  - for nstrftime, c_nstrftime, parse-datetime, which all take a timezone_t
    argument.

For reading tzdata: The first question is how to include tzdata in gnulib.
  - AFAICS, the main data file (without comments) is tzdata.zi and is about
    100 KB large. It can be upgraded simply by copying the newest tzdata.zi
    from a newer tzdata distribution. Including such a file in gnulib would
    be OK (re copyright, number of files, total size), right?
  - Whereas including all files from /usr/share/zoneinfo is probably not
    acceptable (> 1300 files, ca. 6 MB total size).
  - Access pattern: In a running program, very few among the time zones will
    be used. Therefore, caching in memory is essential.

Bruno

[-- Attachment #2: loc.c --]
[-- Type: text/x-csrc, Size: 1519 bytes --]

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>


/* Some common time zone name.  */

#if defined _WIN32 && !defined __CYGWIN__
/* Cf. <https://ss64.com/timezones.html>  */
# define FRENCH_TZ  "Romance Standard Time"
#else
/* Cf. <https://en.wikipedia.org/wiki/List_of_tz_database_time_zones>  */
//# define FRENCH_TZ  "Europe/Paris"
# define FRENCH_TZ  "Europe/Berlin"
#endif


static void
show_localtime_result (time_t t)
{
  struct tm *result = localtime (&t);
  printf ("%ld -> %04d-%02d-%02d %02d:%02d:%02d DST=%d\n",
          (long) t,
          result->tm_year + 1900, result->tm_mon + 1, result->tm_mday,
          result->tm_hour, result->tm_min, result->tm_sec,
          result->tm_isdst);
}

int
main (int argc, char *argv[])
{
#if defined _WIN32 && !defined __CYGWIN__
  _putenv ("TZ" "=" FRENCH_TZ);
#else
  setenv ("TZ", FRENCH_TZ, 1);
#endif

  show_localtime_result (1173578399); /* 2007-03-11 02:59:59 */
  show_localtime_result (1173578401); /* 2007-03-11 03:00:01 */

  show_localtime_result (1174784399); /* 2007-03-25 01:59:59 */
  show_localtime_result (1174784401); /* 2007-03-25 03:00:01 */

  return 0;
}
/*
glibc, Cygwin:
1173578399 -> 2007-03-11 02:59:59 DST=0
1173578401 -> 2007-03-11 03:00:01 DST=0
1174784399 -> 2007-03-25 01:59:59 DST=0
1174784401 -> 2007-03-25 03:00:01 DST=1

Native Windows:
1173578399 -> 2007-03-11 01:59:59 DST=0
1173578401 -> 2007-03-11 03:00:01 DST=1
1174784399 -> 2007-03-25 01:59:59 DST=1
1174784401 -> 2007-03-25 02:00:01 DST=1
*/

[-- Attachment #3: tzi.c --]
[-- Type: text/x-csrc, Size: 4372 bytes --]

#include <stdio.h>
#include <windows.h>
#include <timezoneapi.h>
#include <wchar.h>

/* Cf. <https://ss64.com/timezones.html>  */
# define FRENCH_TZ  "Romance Standard Time"

int main ()
{
  _putenv ("TZ=" FRENCH_TZ);

  DWORD ret;

  TIME_ZONE_INFORMATION info1;
  ret = GetTimeZoneInformation (&info1);
  printf ("GetTimeZoneInformation:\n"
          "ret = %lu  info1 =\n"
          "standard: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n"
          "daylight: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n",
          ret,
          info1.StandardName,
          info1.StandardBias,
          info1.StandardDate.wYear,
          info1.StandardDate.wMonth,
          info1.StandardDate.wDay,
          info1.StandardDate.wHour,
          info1.StandardDate.wMinute,
          info1.StandardDate.wSecond,
          info1.StandardDate.wMilliseconds,
          info1.DaylightName,
          info1.DaylightBias,
          info1.DaylightDate.wYear,
          info1.DaylightDate.wMonth,
          info1.DaylightDate.wDay,
          info1.DaylightDate.wHour,
          info1.DaylightDate.wMinute,
          info1.DaylightDate.wSecond,
          info1.DaylightDate.wMilliseconds);

  DYNAMIC_TIME_ZONE_INFORMATION info2;
  ret = GetDynamicTimeZoneInformation (&info2);
  printf ("GetDynamicTimeZoneInformation:\n"
          "ret = %lu  info2 = bias=%ld timezonekey=%ls dynamicdisabled=%d\n"
          "standard: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n"
          "daylight: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n",
          ret,
          info2.Bias,
          info2.TimeZoneKeyName,
          info2.DynamicDaylightTimeDisabled,
          info2.StandardName,
          info2.StandardBias,
          info2.StandardDate.wYear,
          info2.StandardDate.wMonth,
          info2.StandardDate.wDay,
          info2.StandardDate.wHour,
          info2.StandardDate.wMinute,
          info2.StandardDate.wSecond,
          info2.StandardDate.wMilliseconds,
          info2.DaylightName,
          info2.DaylightBias,
          info2.DaylightDate.wYear,
          info2.DaylightDate.wMonth,
          info2.DaylightDate.wDay,
          info2.DaylightDate.wHour,
          info2.DaylightDate.wMinute,
          info2.DaylightDate.wSecond,
          info2.DaylightDate.wMilliseconds);

  DYNAMIC_TIME_ZONE_INFORMATION info3i;
  TIME_ZONE_INFORMATION info3;
  {
    DWORD i;
    for (i = 0; ; i++)
      {
        if (EnumDynamicTimeZoneInformation (i, &info3i) == ERROR_SUCCESS // Link error in mingw, OK in MSVC.
            && wcscmp (info3i.TimeZoneKeyName, L"Romance Standard Time") == 0)
          break;
      }
  }
  ret = GetTimeZoneInformationForYear (2007, &info3i, &info3);
  printf ("GetTimeZoneInformationForYear(2007):\n"
          "ret = %lu  info3 =\n"
          "standard: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n"
          "daylight: |%ls| bias=%ld date=%4d-%02d-%02d %02d:%02d:%02d.%03d\n",
          ret,
          info3.StandardName,
          info3.StandardBias,
          info3.StandardDate.wYear,
          info3.StandardDate.wMonth,
          info3.StandardDate.wDay,
          info3.StandardDate.wHour,
          info3.StandardDate.wMinute,
          info3.StandardDate.wSecond,
          info3.StandardDate.wMilliseconds,
          info3.DaylightName,
          info3.DaylightBias,
          info3.DaylightDate.wYear,
          info3.DaylightDate.wMonth,
          info3.DaylightDate.wDay,
          info3.DaylightDate.wHour,
          info3.DaylightDate.wMinute,
          info3.DaylightDate.wSecond,
          info3.DaylightDate.wMilliseconds);

}
/* Compile:
   $CC tzi.c -Wall -D_WIN32_WINNT=_WIN32_WINNT_WIN8 -ladvapi32
 */
/* Results:

GetTimeZoneInformation:
ret = 1  info1 =
standard: |W. Europe Standard Time| bias=0 date=   0-10-05 03:00:00.000
daylight: |W. Europe Daylight Time| bias=-60 date=   0-03-05 02:00:00.000
GetDynamicTimeZoneInformation:
ret = 1  info2 = bias=-60 timezonekey=W. Europe Standard Time dynamicdisabled=0
standard: |W. Europe Standard Time| bias=0 date=   0-10-05 03:00:00.000
daylight: |W. Europe Daylight Time| bias=-60 date=   0-03-05 02:00:00.000
GetTimeZoneInformationForYear(2007):
ret = 1  info3 =
standard: |Romance Standard Time| bias=0 date=   0-10-05 03:00:00.000
daylight: |Romance Daylight Time| bias=-60 date=   0-03-05 02:00:00.000

*/

Re: localtime on native Windows

[Paul Eggert]

On 2024-02-11 04:46, Bruno Haible wrote:
> The first question is how to include tzdata in gnulib.
>    - AFAICS, the main data file (without comments) is tzdata.zi and is about
>      100 KB large. It can be upgraded simply by copying the newest tzdata.zi
>      from a newer tzdata distribution. Including such a file in gnulib would
>      be OK (re copyright, number of files, total size), right?

No problem with copyright; the file is in the public domain.

>   - Whereas including all files from /usr/share/zoneinfo is probably not
>     acceptable (> 1300 files, ca. 6 MB total size).

That can be automatically generated from tzdata.zi, by using zic, the 
source code of which is also in the public domain. This is common in 
GNU/Linux systems. If memory serves Android uses another approach: a 
single binary file indexed by timezone name.

Re: localtime on native Windows

[Brian Inglis]

On Sun, 11 Feb 2024 13:46:50 +0100, Bruno Haible wrote on bug-gnulib:
> I wrote:
>> For these modules, the next function to provide in an MT-safe way is
>> localtime_r.
> 
> Our gmtime_r and localtime_r are MT-safe on native Windows. I ran the
> test-gmtime_r-mt and test-localtime_r-mt tests for 2000 seconds each, and
> they did not crash.
> 
> But the problem is that localtime() and localtime_r() on native Windows
> produce nonsensical results:
>   - They pretend that in France, in 2007, DST began on 2007-03-11. When in
>     fact, it started on 2007-03-25.
>   - The hour is wrong.
> Witness: The attached program loc.c.
> 
>> On native Windows, when the 'localtime_s' function [1][2]
>> is not available, such as on the older Windows versions that Emacs cares
>> about, the solution is to use GetTimeZoneInformation [3].
> 
> None of the GetTimeZoneInformation APIs from the Windows DLLs works either.
> They pretend that in German and French time zones, DST starts on March 5,
> in all years. Witness: The attached program tzi.c.
> 
> So, there is no way around implementing a correct localtime_r, based on
> tzdata, in Gnulib.
> 
> It will be useful
>   - for localtime_r on native Windows,
>   - for nstrftime, c_nstrftime, parse-datetime, which all take a timezone_t
>     argument.
> 
> For reading tzdata: The first question is how to include tzdata in gnulib.
>   - AFAICS, the main data file (without comments) is tzdata.zi and is about
>     100 KB large. It can be upgraded simply by copying the newest tzdata.zi
>     from a newer tzdata distribution. Including such a file in gnulib would
>     be OK (re copyright, number of files, total size), right?
>   - Whereas including all files from /usr/share/zoneinfo is probably not
>     acceptable (> 1300 files, ca. 6 MB total size).
>   - Access pattern: In a running program, very few among the time zones will
>     be used. Therefore, caching in memory is essential.

I was looking around to see if there was some way to leverage Windows UWP/.NET 
tzdata provided in:

	https://github.com/microsoft/icu

https://github.com/search?q=repo%3Amicrosoft%2Ficu%20tzdata&type=code

and noticed that some comments under:

https://learn.microsoft.com/en-ca/dotnet/core/extensions/globalization-icu#icu-on-webassembly

suggest that required data can be shrunk to ~300KB using Brotli!?

Perhaps gnulib could leverage some of the interfaces in:

	https://github.com/sillsdev/icu-dotnet

as an alternative interface to access tzdata on native Windows or .NET 5+ Core 
platforms, without building in tzdata and requiring updates?

Other options would be for some selection of generated data such as using 
zonenow.tab, supporting only the current time onward, for the 449 time zones 
currently in:

https://github.com/unicode-org/cldr/blob/main/common/supplemental/windowsZones.xml

perhaps using CBOR format and/or Brotli compression to improve on using a zip or 
tar container format:

	470KB -> 221KB zip, 134KB tar.gz, 69KB tar.lz, 68KB tar.xz;

or using the usual built-in POSIX TZ parser and the POSIX TZ rules output by:

	tail -1 /usr/share/zoneinfo/**/*

for the selected time zones, including time zone ids and rules:

	12.6KB, 4KB zip, 3.8KB gz, 3.5KB xz/lz/bz2.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                  -- Antoine de Saint-Exupéry

Re: localtime on native Windows

[Bruno Haible]

Brian Inglis wrote:
> I was looking around ... that required data can be shrunk to ~300KB using Brotli!?

Whereas the entire tzdata.zi (without comments, and with abbreviations) is
only around 100 KB. I'll definitely prefer the latter.

> Other options would be for some selection of generated data such as using 
> zonenow.tab, supporting only the current time onward

This is a simplification that one may do when working on an embedded system
or such. But not in gnulib. When the entire data file is 100 KB, it is not
a good compromise to give up correctness for dates in the past, in exchange
for further data size reduction.

> https://github.com/unicode-org/cldr/blob/main/common/supplemental/windowsZones.xml

This file is interesting for its mapping table from Windows time zones.

Bruno

Re: localtime on native Windows

[Bruno Haible]

Paul Eggert wrote:
> > The first question is how to include tzdata in gnulib.
> >    - AFAICS, the main data file (without comments) is tzdata.zi and is about
> >      100 KB large. It can be upgraded simply by copying the newest tzdata.zi
> >      from a newer tzdata distribution. Including such a file in gnulib would
> >      be OK (re copyright, number of files, total size), right?
> 
> No problem with copyright; the file is in the public domain.
> 
> >   - Whereas including all files from /usr/share/zoneinfo is probably not
> >     acceptable (> 1300 files, ca. 6 MB total size).
> 
> That can be automatically generated from tzdata.zi, by using zic, the 
> source code of which is also in the public domain.

Thanks for the confirmation.

My current plan, in order to cover the two goals
  - localtime_r on native Windows,
  - nstrftime, c_nstrftime, parse-datetime, which all take a timezone_t
    argument,
is to write four pieces of code:
  1) code that represents the data for a single time zone in memory,
     (based on glibc's tzfile.c),
  2) code that reads a binary time zone file, for use on Unix,
     (based on glibc's tzfile.c), returning it in the format 1).
  3) code that reads a tzdata.zi, for use on Windows,
     (based on glibc's zic.c and tzfile.c), returning it in the format 1).
  4) an implementation of localtime_r for a given time zone
     (based on glibc's tzfile.c:__tzfile_compute, tzset.c:__tz_convert, and
     localtime_r).

This should fulfil Gnulib's needs (code under LGPLv2+, small number
of source files, small number and size of data files to include).

Note that this will have to wait for a few weeks. I have more urgent work on
gettext to do first.

Note also: The above approach will need locking, in order to protect caches
(in 1) the data for a single time zone, in 3) the contents of tzdata.zi).

Bruno

Re: localtime on native Windows

[Paul Eggert]

On 2024-02-17 18:38, Bruno Haible wrote:
> Note also: The above approach will need locking, in order to protect caches
> (in 1) the data for a single time zone, in 3) the contents of tzdata.zi).

The idea I had was to go more in the NetBSD / TZDB direction. These 
implementations don't need either locks or lock-free caching algorithms, 
because it's the caller's responsibility to cache.

Re: localtime on native Windows

[Brian.Inglis]

On 2024-02-17 19:14, Bruno Haible wrote:
> Brian Inglis wrote:
>> I was looking around ... that required data can be shrunk to ~300KB using Brotli!?
> 
> Whereas the entire tzdata.zi (without comments, and with abbreviations) is
> only around 100 KB. I'll definitely prefer the latter.
> 
>> Other options would be for some selection of generated data such as using
>> zonenow.tab, supporting only the current time onward
> 
> This is a simplification that one may do when working on an embedded system
> or such. But not in gnulib. When the entire data file is 100 KB, it is not
> a good compromise to give up correctness for dates in the past, in exchange
> for further data size reduction.
> 
>> https://github.com/unicode-org/cldr/blob/main/common/supplemental/windowsZones.xml
> 
> This file is interesting for its mapping table from Windows time zones.

My point was that this data is already built into Windows, and accessible via 
its ICU API, so gnulib does not need to include, update, and compile tzdata to 
support Windows.

It just has to adapt its localization, as does Cygwin, which uses the Windows 
zones to look up tz ids, which it uses to load tzdata to provide POSIX 
interfaces, using otherwise mostly BSD code adapted from tzcode, required by 
newlib to allow commercial and other uses by RedHat and other embedded vendors.

-- 
Take care. Thanks, Brian Inglis              Calgary, Alberta, Canada

La perfection est atteinte                   Perfection is achieved
non pas lorsqu'il n'y a plus rien à ajouter  not when there is no more to add
mais lorsqu'il n'y a plus rien à retirer     but when there is no more to cut
                                 -- Antoine de Saint-Exupéry

Re: localtime on native Windows

[Bruno Haible]

Paul Eggert wrote:
> > Note also: The above approach will need locking, in order to protect caches
> > (in 1) the data for a single time zone, in 3) the contents of tzdata.zi).
> 
> The idea I had was to go more in the NetBSD / TZDB direction. These 
> implementations don't need either locks or lock-free caching algorithms, 
> because it's the caller's responsibility to cache.

I don't understand what you mean:
  - How can the caller do the caching for something internal to the callee?
    I've never seen callbacks being used for that purpose. I've once seen
    a calling convention where the caller allocates memory for a cache and
    passes it to the callee; but that was for a case where the cache's
    lifetime ends when the callee returns.
  - The NetBSD code [1][2] does locking in the top-level functions (tzset,
    mktime, strftime). I guess that it is because NetBSD uses global variables
    (tzname, timezone, daylight, altzone).

For Gnulib, I intend to avoid global variables of this kind — they belong
in data allocated for each time zone — and therefore move the locking to
lower levels. In the frequent case that a timezone is used for the second
time or later, no locking will be needed at all; this will thus be faster
than what NetBSD does.

Bruno

[1] http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/src/lib/libc/time/localtime.c?rev=1.141&only_with_tag=MAIN
[2] http://cvsweb.netbsd.org/bsdweb.cgi/~checkout~/src/lib/libc/time/strftime.c?rev=1.55

Re: localtime on native Windows

[Paul Eggert]

On 2024-02-18 06:38, Bruno Haible wrote:
>    - How can the caller do the caching for something internal to the callee?

The caller calls tzalloc, gets a timezone object, and calls tzfree when 
it no longer needs that timezone object.

>   - The NetBSD code [1][2] does locking in the top-level functions (tzset,
>     mktime, strftime). I guess that it is because NetBSD uses global variables
>     (tzname, timezone, daylight, altzone).

This is for backward compatibility, and is required for POSIX 
compatibility. It has obvious problems with threads. New code should not 
use these vestigial interfaces. They should use localtime_z and 
mktime_z. (There should be no need for strftime_z.)

https://data.iana.org/time-zones/theory.html#vestigial

> For Gnulib, I intend to avoid global variables of this kind — they belong
> in data allocated for each time zone — and therefore move the locking to
> lower levels. In the frequent case that a timezone is used for the second
> time or later, no locking will be needed at all; this will thus be faster
> than what NetBSD does.

You're right to avoid the global variables, but I don't see why tzalloc 
needs to lock, even in MS-Windows. Perhaps there could be an "I'll do 
the caching" layer above tzalloc, but the lower-level code need not lock 
at all.