Accelerating Y2038 glibc fixes

Accelerating Y2038 glibc fixes

[Wolfgang Denk]

Hi all,

this is a somewhat desperate call for help...

We have been trying for some time to help some of our customers to
get patches into glibc to fix the remaining Y2038 issues.

But progress is way too slow...

The problems we are facing are not on a technical level - we believe
we could deal with such.

See for example [1] - there are just 7 lines of "code".  But Joseph
does not accept our patches.  The arguments he gives are not on a
technical level; instead he says that only a native English speaker
who has a with deep understanding of glibc internals and the
previous development process will be capable to provide such a patch
in an acceptable way.

To be honest - I think only himself would fulfill this requirement
good enough.

But Joseph has not enough resources available, and he cannot
allocate more resources even if such efforts would be paid for by
DENX.  And he does not know of anyone to help.

In short, we are stuck, and not even proper funding can accelerate
the process - not to mention, that it's impossible to give any
deadlines at all.


Obviously, this is just my personal and external view - I guess I
must be missing something, because I cannot imagine that the progress
of a project that plays such a central role to the whole FOSS
community suffers from such bottlenecks, and that these cannot be
helped.

Is there anything we can do to accelerate this process?  As
mentioned, we are willing to dedicate both man-power and/or money,
if this helps.

Thanks in advance!

[1] https://patchwork.ozlabs.org/patch/1114061/



Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
An age is called Dark not because  the  light  fails  to  shine,  but
because people refuse to see it.           -- James Michener, "Space"

Re: Accelerating Y2038 glibc fixes

[Wolfgang Denk]

Hi all,

In message <20190712072103.D3DBC24003A@gemini.denx.de> I wrote:
>
> this is a somewhat desperate call for help...
>
> We have been trying for some time to help some of our customers to
> get patches into glibc to fix the remaining Y2038 issues.
>
> But progress is way too slow...
>
> The problems we are facing are not on a technical level - we believe
> we could deal with such.
>
> See for example [1] - there are just 7 lines of "code".  But Joseph
> does not accept our patches.  The arguments he gives are not on a
> technical level; instead he says that only a native English speaker
> who has a with deep understanding of glibc internals and the
> previous development process will be capable to provide such a patch
> in an acceptable way.
>
> To be honest - I think only himself would fulfill this requirement
> good enough.
>
> But Joseph has not enough resources available, and he cannot
> allocate more resources even if such efforts would be paid for by
> DENX.  And he does not know of anyone to help.
>
> In short, we are stuck, and not even proper funding can accelerate
> the process - not to mention, that it's impossible to give any
> deadlines at all.
>
>
> Obviously, this is just my personal and external view - I guess I
> must be missing something, because I cannot imagine that the progress
> of a project that plays such a central role to the whole FOSS
> community suffers from such bottlenecks, and that these cannot be
> helped.
>
> Is there anything we can do to accelerate this process?  As
> mentioned, we are willing to dedicate both man-power and/or money,
> if this helps.
>
> Thanks in advance!
>
> [1] https://patchwork.ozlabs.org/patch/1114061/

Is there nobody here who has any idea what could be done?

No comments at all?


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
The only perfect science is hind-sight.

Re: Accelerating Y2038 glibc fixes

[Siddhesh Poyarekar]

Hello Wolfgang,

I haven't been involved in the Y2038 issue so I cannot comment on the
content or quality of the patches or the way forward in that context.
However I want to assure you that we're collectively trying our best to
review patches that come on the list, just that there aren't enough
hands to help.  glibc is a central project in the Linux ecosystem but
unfortunately it's not as well staffed as other projects such as the
kernel or gcc.  Things have gotten much better in the last few years,
but we're still short of people as many of us are doing other things
alongside glibc.

Hopefully someone with more context and will respond soon and help
unblock things but until then, please don't lose hope, we will get there
eventually!

Siddhesh


On 12/07/19 12:51 PM, Wolfgang Denk wrote:
> See for example [1] - there are just 7 lines of "code".  But Joseph
> does not accept our patches.  The arguments he gives are not on a
> technical level; instead he says that only a native English speaker
> who has a with deep understanding of glibc internals and the
> previous development process will be capable to provide such a patch
> in an acceptable way.
> 
> To be honest - I think only himself would fulfill this requirement
> good enough.
> 
> But Joseph has not enough resources available, and he cannot
> allocate more resources even if such efforts would be paid for by
> DENX.  And he does not know of anyone to help.
> 
> In short, we are stuck, and not even proper funding can accelerate
> the process - not to mention, that it's impossible to give any
> deadlines at all.
> 
> 
> Obviously, this is just my personal and external view - I guess I
> must be missing something, because I cannot imagine that the progress
> of a project that plays such a central role to the whole FOSS
> community suffers from such bottlenecks, and that these cannot be
> helped.
> 
> Is there anything we can do to accelerate this process?  As
> mentioned, we are willing to dedicate both man-power and/or money,
> if this helps.
> 
> Thanks in advance!
> 
> [1] https://patchwork.ozlabs.org/patch/1114061/
> 
> 
> 
> Best regards,
> 
> Wolfgang Denk
> 

Re: Accelerating Y2038 glibc fixes

[Wolfgang Denk]

Dear Siddhesh,

In message <2a129837-9fe4-8f37-2b0d-2086a2f8253b@gotplt.org> you wrote:
>
> I haven't been involved in the Y2038 issue so I cannot comment on the
> content or quality of the patches or the way forward in that context.
> However I want to assure you that we're collectively trying our best to
> review patches that come on the list, just that there aren't enough
> hands to help.

Thanks for taking time to read and respond.

I never meant to suggest that you folks are not all trying to do
your best - belive me, I know such a situation very well.  I've been
long enough maintainer of a bigger FOSS project myself (U-Boot)...

> glibc is a central project in the Linux ecosystem but
> unfortunately it's not as well staffed as other projects such as the
> kernel or gcc.  Things have gotten much better in the last few years,
> but we're still short of people as many of us are doing other things
> alongside glibc.

This is something I can understand and accept.  And we're trying to
offer help - both with man-power and with funding.  only nobody
seems to be interested :-(

> Hopefully someone with more context and will respond soon and help
> unblock things but until then, please don't lose hope, we will get there
> eventually!

Thanks!

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
I program, therefore I am.

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Wolfgang Denk:

> We have been trying for some time to help some of our customers to
> get patches into glibc to fix the remaining Y2038 issues.

> Is there anything we can do to accelerate this process?  As
> mentioned, we are willing to dedicate both man-power and/or money,
> if this helps.

One difficult trade-off is that for you, this is just one-time
enablement work, but the regular contributors will be stuck with the
code you add forever.  Especially since it touches 64-bit architectures
as well.

For me personally, the whole project is quite baffling.  I'm interested
in 32-bit changes only to support legacy binaries, and this is new work
targeting new binaries.  Particularly for i386, it *will* lead to ABI
fragmentation.  We can add dual ABI support for glibc, but for the rest
of the system, distributions will have to pick a default (probably with
their build flags selection), and they will make different choices.
Some distributions will focus on backwards compatibility with legacy
binaries, but others want to keep the i386 architecture alive for many
more years.  I really do not want to deal with the resulting confusion,
but it seems that I do not have any choice there.

Since I'm opposed to this entire project, I have largely refrained from
reviews, except for things that looked obviously wrong to me (e.g.,
things that definitely break compatibility with older kernels or
existing ABIs), but even for those cases, my feedback probably wasn't
very helpful.

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Wolfgang Denk]

Dear Florian,

In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
>
> One difficult trade-off is that for you, this is just one-time
> enablement work, but the regular contributors will be stuck with the
> code you add forever.  Especially since it touches 64-bit architectures
> as well.

I am aware of this, and I'm willing to go all necessary procedures
to get things right.  If someone points out a problem, we can
address it, and move on.  But we haven't seen any progress for a
long time...

> For me personally, the whole project is quite baffling.  I'm interested
> in 32-bit changes only to support legacy binaries, and this is new work
> targeting new binaries.

Indeed it is easy for all the big distros who dropped (at least
commercial) support for 32 bit systems years ago.  But this is only
a fraction of the systems that are affected, and it's not only x86.
There is a zillion of 32 bit ARM, Power architecture and MIPS
devices out there, with more of them being sold for many more years
to come, and many of these are used in products a) with product
lifetimes exceeding Y2038 and b) where being Y2038 clean is a
critical feature.

I think this is part of the problem - the big distros are not really
interested in this work, they see it only as a nuisance to their
business cases.  I cannot stop myself from stating that such an
approach is egotistic at best, if not ignorant, or both.

> Particularly for i386, it *will* lead to ABI
> fragmentation.  We can add dual ABI support for glibc, but for the rest
> of the system, distributions will have to pick a default (probably with
> their build flags selection), and they will make different choices.
> Some distributions will focus on backwards compatibility with legacy
> binaries, but others want to keep the i386 architecture alive for many
> more years.  I really do not want to deal with the resulting confusion,
> but it seems that I do not have any choice there.

As mentioned, it is not only about i386.  There are countless
Embedded Systems running 32 bit ARM, PPC and MIPS systems.

> Since I'm opposed to this entire project, I have largely refrained from
> reviews, except for things that looked obviously wrong to me (e.g.,
> things that definitely break compatibility with older kernels or
> existing ABIs), but even for those cases, my feedback probably wasn't
> very helpful.

I can understand your point of view, but indeed this is not helpful
here.  glibc is a central resource for the whole FOSS system, and we
are not pushing these changes for a fancy, but because a solution
is needed for a large number of affected projects and products.


We need to find a constructive way to proceed with this matter.  It
will not go away if we try to ignore it.

Thanks.


Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
"Text processing has made it possible to right-justify any idea, even
one which cannot be justified on any other grounds."
                                                 -- J. Finnegan, USC.

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Wolfgang Denk:

> In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
>>
>> One difficult trade-off is that for you, this is just one-time
>> enablement work, but the regular contributors will be stuck with the
>> code you add forever.  Especially since it touches 64-bit architectures
>> as well.
>
> I am aware of this, and I'm willing to go all necessary procedures
> to get things right.  If someone points out a problem, we can
> address it, and move on.  But we haven't seen any progress for a
> long time...

This is not what I meant, or maybe I'm misunderstanding you.  Are you
offering a long-term collaboration on a varied range of topics?  Or just
this one contribution of a single feature?

>> For me personally, the whole project is quite baffling.  I'm interested
>> in 32-bit changes only to support legacy binaries, and this is new work
>> targeting new binaries.
>
> Indeed it is easy for all the big distros who dropped (at least
> commercial) support for 32 bit systems years ago.

Red Hat still supports 32-bit legacy applications.  I'm sure SUSE does
as well.

Newer operating system releases do not come with a 32-bit kernel
anymore, but we have a 32-bit userspace (mostly consisting of libraries
to support existing legacy application), so we still need glibc.

> I think this is part of the problem - the big distros are not really
> interested in this work, they see it only as a nuisance to their
> business cases.  I cannot stop myself from stating that such an
> approach is egotistic at best, if not ignorant, or both.

I would even go one step further: For i386, the upcoming ABI
fragmentation impacts our users even if they are never going to use the
new capabilities.

There also appears to be an expectation that all applications which
invoke the futex system call directly need to be changed at the source
code level.  This will affect many upstream projects, even if they
primarily target 64-bit architectures.

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Andrew Pinski]

On Tue, Jul 16, 2019 at 8:09 AM Florian Weimer <fweimer@redhat.com> wrote:
>
> * Wolfgang Denk:
>
> > In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
> >>
> >> One difficult trade-off is that for you, this is just one-time
> >> enablement work, but the regular contributors will be stuck with the
> >> code you add forever.  Especially since it touches 64-bit architectures
> >> as well.
> >
> > I am aware of this, and I'm willing to go all necessary procedures
> > to get things right.  If someone points out a problem, we can
> > address it, and move on.  But we haven't seen any progress for a
> > long time...
>
> This is not what I meant, or maybe I'm misunderstanding you.  Are you
> offering a long-term collaboration on a varied range of topics?  Or just
> this one contribution of a single feature?
>
> >> For me personally, the whole project is quite baffling.  I'm interested
> >> in 32-bit changes only to support legacy binaries, and this is new work
> >> targeting new binaries.
> >
> > Indeed it is easy for all the big distros who dropped (at least
> > commercial) support for 32 bit systems years ago.
>
> Red Hat still supports 32-bit legacy applications.  I'm sure SUSE does
> as well.
>
> Newer operating system releases do not come with a 32-bit kernel
> anymore, but we have a 32-bit userspace (mostly consisting of libraries
> to support existing legacy application), so we still need glibc.

I should point out a recent case where Ubuntu mentioned they would be
removing the 32bit userspace and some closed source commerical
software (Steam) said they would be pulling support for Ubuntu.  This
forced Ubuntu to change their mind.
This happened within the last few weeks.  I don't know if this is
something we as glibc project as to worry about but it does give
second thoughts to the bigger distros at removing 32bit userland
support or even supporting a new framented ABI.

Thanks,
Andrew Pinski

>
> > I think this is part of the problem - the big distros are not really
> > interested in this work, they see it only as a nuisance to their
> > business cases.  I cannot stop myself from stating that such an
> > approach is egotistic at best, if not ignorant, or both.
>
> I would even go one step further: For i386, the upcoming ABI
> fragmentation impacts our users even if they are never going to use the
> new capabilities.
>
> There also appears to be an expectation that all applications which
> invoke the futex system call directly need to be changed at the source
> code level.  This will affect many upstream projects, even if they
> primarily target 64-bit architectures.
>
> Thanks,
> Florian

Re: Accelerating Y2038 glibc fixes

[Arnd Bergmann]

On Tue, Jul 16, 2019 at 4:52 PM Wolfgang Denk <wd@denx.de> wrote:
> In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
>
> > Since I'm opposed to this entire project, I have largely refrained from
> > reviews, except for things that looked obviously wrong to me (e.g.,
> > things that definitely break compatibility with older kernels or
> > existing ABIs), but even for those cases, my feedback probably wasn't
> > very helpful.
>
> I can understand your point of view, but indeed this is not helpful
> here.  glibc is a central resource for the whole FOSS system, and we
> are not pushing these changes for a fancy, but because a solution
> is needed for a large number of affected projects and products.
>
> We need to find a constructive way to proceed with this matter.  It
> will not go away if we try to ignore it.

My impression is that we have two mostly separate groups
of users for 32-bit user space:

a) Those that absolutely require 100% ABI compatibility to run
    legacy code but will eventually migrate to 64-bit user space,
    almost certainly within the coming ten years before we really
    need to worry (too much) about things like key expiration dates
    beyond 2038.

b) Those that already need support for 64-bit time_t because
    they are deploying new 32-bit binaries that are expected to run
    beyond 2038, while not caring at all about compatibility
    with existing binaries that are already known to be broken
    for this purpose.

If we just work to satisfy both these groups but ignore the intersection
(those that have existing 32-bit binaries and want those to run
after 2038?), things could become noticeably easier.

The price for that is a mid-2030s EOL time on any existing 32-bit
glibc ports, any binaries linked to them, and any distros built upon
that, and replacing them with something else that is binary
incompatible but can coexist within e.g. a container.

There are several options what such a replacement might look
like, some that come to mind would include:

- A patchset against glibc that replaces all the time32 interfaces
  with time64 ones and that is maintained outside of the glibc
  code base by whoever is interested in this working.

- A set of glibc architecture ports that can coexist with the
  existing ports but have new target triples and/or soname.
  These could be implemented and merged one architecture
  at a time and share most of the code with the existing
  ports.

- Leave glibc unchanged and require users/distros to migrate to
  musl or something else if they need the time64 support.

Any of the above approaches (including the current plan of
supporting time32 and time64 in a single glibc binary) can of
coexist, but the more incompatible approaches get implemented,
the more fragmentation of ABIs we get.

Note that we have never had such a forced transition on x86, but
there are several examples on other architectures:
- arm oabi/gnueabi/gnueabihf
- ppc64/ppc64le
- mips o32/n32/64
The migration is not much fun for users to say the least, but it's
quite possible that the alternative is even worse.

       Arnd

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Arnd Bergmann:

> b) Those that already need support for 64-bit time_t because
>     they are deploying new 32-bit binaries that are expected to run
>     beyond 2038, while not caring at all about compatibility
>     with existing binaries that are already known to be broken
>     for this purpose.

There is also c), new 32-bit architectures which need 64-bit time_t
support today due to kernel limitations.  Whether those binaries need to
run for two years or twenty does not matter to them.

I have reviewed patches for the c) case, but that doesn't seem to be
work that interests Wolfgang.

> Any of the above approaches (including the current plan of
> supporting time32 and time64 in a single glibc binary) can of
> coexist, but the more incompatible approaches get implemented,
> the more fragmentation of ABIs we get.
>
> Note that we have never had such a forced transition on x86,

We did change the stack alignment requirements on x86.  Without any ABI
markup.  For the non-executable stack, we used proper markup, but hat
approach will not work for time_t.

Several of the transitions you mentioned have ELF markup for the
transition, and at least in the case of ELF v2 for POWER, it is really
clear and consistent (it's also an easy scenarion admittedly, and very
different from the time_t situation).

Part of my reservations about the update to legacy ABIs is there hasn't
been any discussion about markup.  I'm not saying that we absolutely
need to have it.  It's just something that contributes to my general
unease.

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Wolfgang Denk]

Dear Florian,

In message <875zo0911b.fsf@oldenburg2.str.redhat.com> you wrote:
> * Arnd Bergmann:
>
> > b) Those that already need support for 64-bit time_t because
> >     they are deploying new 32-bit binaries that are expected to run
> >     beyond 2038, while not caring at all about compatibility
> >     with existing binaries that are already known to be broken
> >     for this purpose.
>
> There is also c), new 32-bit architectures which need 64-bit time_t
> support today due to kernel limitations.  Whether those binaries need to
> run for two years or twenty does not matter to them.
>
> I have reviewed patches for the c) case, but that doesn't seem to be
> work that interests Wolfgang.

Correct - our situation is Arnd's case b).

But my understanding is that for c) glibc has to modify the generic
syscalls wrapper (like clock_gettime/nanosleep/settime, etc.), and
for b) we also need to do that first.  But currently we are stuck at
the point where the __ASSUME_TIME64_SYSCALLS flag is not accepted /
pulled.

So b) and c) align in development...

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd@denx.de
An Elephant is a mouse with an Operating System.              - Knuth

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Wolfgang Denk:

> Dear Florian,
>
> In message <875zo0911b.fsf@oldenburg2.str.redhat.com> you wrote:
>> * Arnd Bergmann:
>>
>> > b) Those that already need support for 64-bit time_t because
>> >     they are deploying new 32-bit binaries that are expected to run
>> >     beyond 2038, while not caring at all about compatibility
>> >     with existing binaries that are already known to be broken
>> >     for this purpose.
>>
>> There is also c), new 32-bit architectures which need 64-bit time_t
>> support today due to kernel limitations.  Whether those binaries need to
>> run for two years or twenty does not matter to them.
>>
>> I have reviewed patches for the c) case, but that doesn't seem to be
>> work that interests Wolfgang.
>
> Correct - our situation is Arnd's case b).
>
> But my understanding is that for c) glibc has to modify the generic
> syscalls wrapper (like clock_gettime/nanosleep/settime, etc.), and
> for b) we also need to do that first.  But currently we are stuck at
> the point where the __ASSUME_TIME64_SYSCALLS flag is not accepted /
> pulled.
>
> So b) and c) align in development...

Can you do without __ASSUME_TIME64_SYSCALLS?  Most other __ASSUME_*
macros are an optimization, and if your interest is b),
__ASSUME_TIME64_SYSCALLS will not be the default for glibc distribution
builds anyway because defining it would negatively impact host kernel
compatibility.  It's not just about containers in the fashionable sense,
but simple build chroots are problematic as well in this context.

Or have you received different guidance that __ASSUME_TIME64_SYSCALLS
markup is absolutely required for the initial contribution?

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 3048 bytes --]

Hi Florian,

> * Wolfgang Denk:
> 
> > Dear Florian,
> >
> > In message <875zo0911b.fsf@oldenburg2.str.redhat.com> you wrote:  
> >> * Arnd Bergmann:
> >>  
> >> > b) Those that already need support for 64-bit time_t because
> >> >     they are deploying new 32-bit binaries that are expected to
> >> > run beyond 2038, while not caring at all about compatibility
> >> >     with existing binaries that are already known to be broken
> >> >     for this purpose.  
> >>
> >> There is also c), new 32-bit architectures which need 64-bit time_t
> >> support today due to kernel limitations.  Whether those binaries
> >> need to run for two years or twenty does not matter to them.
> >>
> >> I have reviewed patches for the c) case, but that doesn't seem to
> >> be work that interests Wolfgang.  
> >
> > Correct - our situation is Arnd's case b).
> >
> > But my understanding is that for c) glibc has to modify the generic
> > syscalls wrapper (like clock_gettime/nanosleep/settime, etc.), and
> > for b) we also need to do that first.  But currently we are stuck at
> > the point where the __ASSUME_TIME64_SYSCALLS flag is not accepted /
> > pulled.
> >
> > So b) and c) align in development...  
> 
> Can you do without __ASSUME_TIME64_SYSCALLS?  Most other __ASSUME_*
> macros are an optimization, and if your interest is b),
> __ASSUME_TIME64_SYSCALLS will not be the default for glibc
> distribution builds anyway because defining it would negatively
> impact host kernel compatibility.  It's not just about containers in
> the fashionable sense, but simple build chroots are problematic as
> well in this context.
> 
> Or have you received different guidance that __ASSUME_TIME64_SYSCALLS
> markup is absolutely required for the initial contribution?

The __ASSUME_TIME64_SYSCALLS was discussed with Joseph and Stepan (both
CC'ed) for a long time on the libc-alpha mailing list. The discussion
can be found here [1] (last link is the newest one).



As fair as I understood from the previous discussion, adding
__ASSUME_TIME64_SYSCALLS is a first step to add Y2038 support (or 64 bit
time support to 32 bit archs in general).

The latest patch (v8) with semantics explanation of
__ASSUME_TIME64_SYSCALLS: [2]



Note:

[1] Evolution of __ASSUME_TIME64_SYSCALLS up till v7:

https://patchwork.ozlabs.org/patch/1092579/
https://patchwork.ozlabs.org/patch/1096343/
https://patchwork.ozlabs.org/patch/1096349/
https://patchwork.ozlabs.org/patch/1097132/
https://patchwork.ozlabs.org/patch/1100097/
https://patchwork.ozlabs.org/patch/1107125/
https://patchwork.ozlabs.org/patch/1114061/
https://patchwork.ozlabs.org/patch/1117100/

[2] - https://patchwork.ozlabs.org/patch/1117100/

> 
> Thanks,
> Florian




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Wed, Jul 17, 2019 at 04:15:25PM +0200, Arnd Bergmann wrote:
> On Tue, Jul 16, 2019 at 4:52 PM Wolfgang Denk <wd@denx.de> wrote:
> > In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
> >
> > > Since I'm opposed to this entire project, I have largely refrained from
> > > reviews, except for things that looked obviously wrong to me (e.g.,
> > > things that definitely break compatibility with older kernels or
> > > existing ABIs), but even for those cases, my feedback probably wasn't
> > > very helpful.
> >
> > I can understand your point of view, but indeed this is not helpful
> > here.  glibc is a central resource for the whole FOSS system, and we
> > are not pushing these changes for a fancy, but because a solution
> > is needed for a large number of affected projects and products.
> >
> > We need to find a constructive way to proceed with this matter.  It
> > will not go away if we try to ignore it.
> 
> My impression is that we have two mostly separate groups
> of users for 32-bit user space:
> 
> a) Those that absolutely require 100% ABI compatibility to run
>     legacy code but will eventually migrate to 64-bit user space,
>     almost certainly within the coming ten years before we really
>     need to worry (too much) about things like key expiration dates
>     beyond 2038.
> 
> b) Those that already need support for 64-bit time_t because
>     they are deploying new 32-bit binaries that are expected to run
>     beyond 2038, while not caring at all about compatibility
>     with existing binaries that are already known to be broken
>     for this purpose.
> 
> If we just work to satisfy both these groups but ignore the intersection
> (those that have existing 32-bit binaries and want those to run
> after 2038?), things could become noticeably easier.

Well it replaces one sticky problem with two separate, individually
easier problems. But I'm not sure it reduces the overall amount of
work needed; it might increase it.

The above roughly corresponds to what my original idea for musl and
the ".2 ABI" was, and I'm now fairly skeptical of it. Binary distros
are probably not going to want to expend extra effort to satisfy the
needs of both groups, so they'll either EOL the 32-bit ports they have
now (effectively leaving group (a) screwed, not because glibc dropped
them, but because their distro did) to produce something that will
continue to be relevant into the future, or they'll stick with the old
ABI and EOL 32-bit entirely in the near-ish future.

The segment of group (b) that's building everything themselves
probably doesn't care in the latter case, but then do they even care
about using glibc (vs something else) anyway?

From this analysis it seems like, if you attack the problems
separately, "no action" is the right choice for distros, but that also
leads to EOL for distro support of all existing 32-bit targets.

As a result, I'm leaning strongly towards a conclusion that a solution
that doesn't split the two cases is better, and it's probably what
we'll do for musl.

> The price for that is a mid-2030s EOL time on any existing 32-bit

That's really wishful thinking. I suspect we'll hit critical failures
much sooner due to need to represent dates in the future.

> glibc ports, any binaries linked to them, and any distros built upon
> that, and replacing them with something else that is binary
> incompatible but can coexist within e.g. a container.

I'm not sure how "coexist within a container" is helpful; that might
as well just be a separate system. Multi-arch type approaches might
work a little bit better, but in order for these to be useful to end
users, there need to be automated ways of setting them up. That's a
big burden for distros to spend on archs that many of them would
rather just EOL/drop...

> There are several options what such a replacement might look
> like, some that come to mind would include:
> 
> - A patchset against glibc that replaces all the time32 interfaces
>   with time64 ones and that is maintained outside of the glibc
>   code base by whoever is interested in this working.

If you're trying to use incompatible old and new ABI alongside each
other, I would think you only want to do that as long as you still
have old stuff around. But if you're using patches with ad-hoc ABI
that have bypassed the process of upstream review to ensure that
they're making good decisions compatible with the long term, it's
likely that it's going to keep breaking ABI, and you're going to need
to do multiple painful replacements or keep N ABIs alongside each
other rather than 2 (possibly using something like nix to make it
practical).

So I don't think "use out-of-tree patches for time64 support" is a
good approach.

> - A set of glibc architecture ports that can coexist with the
>   existing ports but have new target triples and/or soname.
>   These could be implemented and merged one architecture
>   at a time and share most of the code with the existing
>   ports.

This is basically the same proposal as the musl ".2 ABI", and it's one
of the least-bad, but as explained above I no longer really favor it.

> - Leave glibc unchanged and require users/distros to migrate to
>   musl or something else if they need the time64 support.

As much as I'd like this, I don't think distros that have stability
contracts with their users for how things work can replace the libc
with one which has significantly different behaviors in application-
and user-facing ways. It's a viable solution for users to run new
32-bit stuff alongside a 32-bit glibc target that's being phased-out
and nearing EOL, but if this is the only option I think it would just
lead to distros dropping the targets.

> Any of the above approaches (including the current plan of
> supporting time32 and time64 in a single glibc binary) can of
> coexist, but the more incompatible approaches get implemented,
> the more fragmentation of ABIs we get.

I'm moderately in favor of supporting both in a single binary. As
noted elsewhere, this fundamentally does not break ABI between libc
consumers (main apps or 3rd party libs) and libc, only between pairs
of libc consumers that have used the affected types as part of their
public interfaces.

To answer Wolfgang Denk's original question, in my opinion, the single
most valuable thing for accelerating the time64 process that third
parties could contribute is detailed analysis of the interfaces that
will break. This (1) will give us an idea of the extent of breakage
that could happen in the worst case, and (2) can be translated
directly into a graph of "requires/conflicts-with" relationships
between library packages that a distro can use to ensure consistency
when updating packages.

Are others willing to start a discussion of exactly what form this
analysis should take, how results should be reported, and how to use
them to make such a transition go smoothly?

> Note that we have never had such a forced transition on x86, but
> there are several examples on other architectures:
> - arm oabi/gnueabi/gnueabihf
> - ppc64/ppc64le
> - mips o32/n32/64
> The migration is not much fun for users to say the least, but it's
> quite possible that the alternative is even worse.

I wouldn't characterize all of those as transitions, more as different
options. As far as I know glibc still supports them all except ARM
OABI, which was dropped long before there was serious interest in
end-user-facing Linux on ARM.

Rich

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 10452 bytes --]

Hi Rich,

Thanks for sharing your view.

> On Wed, Jul 17, 2019 at 04:15:25PM +0200, Arnd Bergmann wrote:
> > On Tue, Jul 16, 2019 at 4:52 PM Wolfgang Denk <wd@denx.de> wrote:  
> > > In message <874l3mjgi6.fsf@oldenburg2.str.redhat.com> you wrote:
> > >  
> > > > Since I'm opposed to this entire project, I have largely
> > > > refrained from reviews, except for things that looked obviously
> > > > wrong to me (e.g., things that definitely break compatibility
> > > > with older kernels or existing ABIs), but even for those cases,
> > > > my feedback probably wasn't very helpful.  
> > >
> > > I can understand your point of view, but indeed this is not
> > > helpful here.  glibc is a central resource for the whole FOSS
> > > system, and we are not pushing these changes for a fancy, but
> > > because a solution is needed for a large number of affected
> > > projects and products.
> > >
> > > We need to find a constructive way to proceed with this matter.
> > > It will not go away if we try to ignore it.  
> > 
> > My impression is that we have two mostly separate groups
> > of users for 32-bit user space:
> > 
> > a) Those that absolutely require 100% ABI compatibility to run
> >     legacy code but will eventually migrate to 64-bit user space,
> >     almost certainly within the coming ten years before we really
> >     need to worry (too much) about things like key expiration dates
> >     beyond 2038.
> > 
> > b) Those that already need support for 64-bit time_t because
> >     they are deploying new 32-bit binaries that are expected to run
> >     beyond 2038, while not caring at all about compatibility
> >     with existing binaries that are already known to be broken
> >     for this purpose.
> > 
> > If we just work to satisfy both these groups but ignore the
> > intersection (those that have existing 32-bit binaries and want
> > those to run after 2038?), things could become noticeably easier.  
> 
> Well it replaces one sticky problem with two separate, individually
> easier problems. But I'm not sure it reduces the overall amount of
> work needed; it might increase it.
> 
> The above roughly corresponds to what my original idea for musl and
> the ".2 ABI" was, and I'm now fairly skeptical of it. Binary distros
> are probably not going to want to expend extra effort to satisfy the
> needs of both groups, so they'll either EOL the 32-bit ports they have
> now (effectively leaving group (a) screwed, not because glibc dropped
> them, but because their distro did) to produce something that will
> continue to be relevant into the future, or they'll stick with the old
> ABI and EOL 32-bit entirely in the near-ish future.
> 
> The segment of group (b) that's building everything themselves
> probably doesn't care in the latter case, but then do they even care
> about using glibc (vs something else) anyway?

If I may add my 2$ - I think they care. Otherwise there wouldn't be the
effort to adapt "upstream first" policy with Y2038 support.

> 
> From this analysis it seems like, if you attack the problems
> separately, "no action" is the right choice for distros, but that also
> leads to EOL for distro support of all existing 32-bit targets.

But as it was mentioned in the other mail. There are _real_ users of 32
bit targets, so distros may re-consider dropping them.

> 
> As a result, I'm leaning strongly towards a conclusion that a solution
> that doesn't split the two cases is better, and it's probably what
> we'll do for musl.

The approach would be very similar to what was done with
_FILE_OFFSET_BITS==64. The end user would need to add -D_TIME_BITS==64
when compiling programs against glibc on 32 bit architecture.

> 
> > The price for that is a mid-2030s EOL time on any existing 32-bit  
> 
> That's really wishful thinking. I suspect we'll hit critical failures
> much sooner due to need to represent dates in the future.
> 
> > glibc ports, any binaries linked to them, and any distros built upon
> > that, and replacing them with something else that is binary
> > incompatible but can coexist within e.g. a container.  
> 
> I'm not sure how "coexist within a container" is helpful; that might
> as well just be a separate system.
> 1 Multi-arch type approaches might
> work a little bit better, but in order for these to be useful to end
> users, there need to be automated ways of setting them up. That's a
> big burden for distros to spend on archs that many of them would
> rather just EOL/drop...
> 
> > There are several options what such a replacement might look
> > like, some that come to mind would include:
> > 
> > - A patchset against glibc that replaces all the time32 interfaces
> >   with time64 ones and that is maintained outside of the glibc
> >   code base by whoever is interested in this working.  
> 
> If you're trying to use incompatible old and new ABI alongside each
> other, I would think you only want to do that as long as you still
> have old stuff around. But if you're using patches with ad-hoc ABI
> that have bypassed the process of upstream review to ensure that
> they're making good decisions compatible with the long term, it's
> likely that it's going to keep breaking ABI, and you're going to need
> to do multiple painful replacements or keep N ABIs alongside each
> other rather than 2 (possibly using something like nix to make it
> practical).
> 
> So I don't think "use out-of-tree patches for time64 support" is a
> good approach.

I do agree here - the "out-of-tree" is not a good approach. Basically
one would need to rebase those "hacks" on top of each glibc release.

If you are wondering how such set of "out-of-tree" patches ("hacks" is
a better word I think) look like - please check following link [1].

They provide support for Y2038 on 32 bit arch (and even passes some
tests) [2].

> 
> > - A set of glibc architecture ports that can coexist with the
> >   existing ports but have new target triples and/or soname.
> >   These could be implemented and merged one architecture
> >   at a time and share most of the code with the existing
> >   ports.  
> 
> This is basically the same proposal as the musl ".2 ABI", and it's one
> of the least-bad, but as explained above I no longer really favor it.
> 
> > - Leave glibc unchanged and require users/distros to migrate to
> >   musl or something else if they need the time64 support.  
> 
> As much as I'd like this, I don't think distros that have stability
> contracts with their users for how things work can replace the libc
> with one which has significantly different behaviors in application-
> and user-facing ways.

+1

> It's a viable solution for users to run new
> 32-bit stuff alongside a 32-bit glibc target that's being phased-out
> and nearing EOL, but if this is the only option I think it would just
> lead to distros dropping the targets.
> 
> > Any of the above approaches (including the current plan of
> > supporting time32 and time64 in a single glibc binary) can of
> > coexist, but the more incompatible approaches get implemented,
> > the more fragmentation of ABIs we get.  
> 
> I'm moderately in favor of supporting both in a single binary. As
> noted elsewhere, this fundamentally does not break ABI between libc
> consumers (main apps or 3rd party libs) and libc, only between pairs
> of libc consumers that have used the affected types as part of their
> public interfaces.

As I've written above. For the end user it shall be enough to add
-D_TIME_BITS==64 during compilation.

> 
> To answer Wolfgang Denk's original question, in my opinion, the single
> most valuable thing for accelerating the time64 process that third
> parties could contribute is detailed analysis of the interfaces that
> will break.

Such analysis is already done and available on glib'c wiki page [3].

> This (1) will give us an idea of the extent of breakage
> that could happen in the worst case, and (2) can be translated
> directly into a graph of "requires/conflicts-with" relationships
> between library packages that a distro can use to ensure consistency
> when updating packages.

As stated above - please visit [3].

> 
> Are others willing to start a discussion of exactly what form this
> analysis should take, how results should be reported, and how to use
> them to make such a transition go smoothly?

There is even the OE/Yocto's meta-y2038 layer (thud YPRR) [4] , which
allows building glibc for testing with:

1. Kernel 5.1+ (with 64 bit time syscalls support on 32 bit archs). 

2. Kernel < 5.1 (to check if it works with older syscalls)

3. A set of tests for syscalls mentioned in [3][2] (with and without
Y2038 support in glibc)

4. Several different architectures for testing (qemu):
	- ARM 32 bit - vexpress-a9
	- i386 / x86_64
	- x32 - the "special" one
	- All the OE/Yocto supported qemu* targets (mips, ppc)


5. One can run the glibc xtest with mentioned above qemu running system
with test-wrapper='/opt/Y2038/glibc/src/scripts/cross-test-ssh.sh
(detailed instruction in [4]).

6. It is also possible to run plain glibc tests for archs mentioned in
point 4 above.

> 
> > Note that we have never had such a forced transition on x86, but
> > there are several examples on other architectures:
> > - arm oabi/gnueabi/gnueabihf
> > - ppc64/ppc64le
> > - mips o32/n32/64
> > The migration is not much fun for users to say the least, but it's
> > quite possible that the alternative is even worse.  
> 
> I wouldn't characterize all of those as transitions, more as different
> options. As far as I know glibc still supports them all except ARM
> OABI, which was dropped long before there was serious interest in
> end-user-facing Linux on ARM.
> 
> Rich


Note:

[1] -
https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019

[2] - https://github.com/lmajewski/y2038-tests

[3] -
https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29

[4] - https://github.com/lmajewski/meta-y2038/tree/master


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
> > As a result, I'm leaning strongly towards a conclusion that a solution
> > that doesn't split the two cases is better, and it's probably what
> > we'll do for musl.
> 
> The approach would be very similar to what was done with
> _FILE_OFFSET_BITS==64. The end user would need to add -D_TIME_BITS==64
> when compiling programs against glibc on 32 bit architecture.

Yes, except that I'd switch the default from the beginning. glibc made
the mistake of not doing this with _FILE_OFFSET_BITS, and it's still
the wrong default 20 years later, with every application's build
system containing workarounds...

> > It's a viable solution for users to run new
> > 32-bit stuff alongside a 32-bit glibc target that's being phased-out
> > and nearing EOL, but if this is the only option I think it would just
> > lead to distros dropping the targets.
> > 
> > > Any of the above approaches (including the current plan of
> > > supporting time32 and time64 in a single glibc binary) can of
> > > coexist, but the more incompatible approaches get implemented,
> > > the more fragmentation of ABIs we get.  
> > 
> > I'm moderately in favor of supporting both in a single binary. As
> > noted elsewhere, this fundamentally does not break ABI between libc
> > consumers (main apps or 3rd party libs) and libc, only between pairs
> > of libc consumers that have used the affected types as part of their
> > public interfaces.
> 
> As I've written above. For the end user it shall be enough to add
> -D_TIME_BITS==64 during compilation.

I think you missed the whole thing about ABI between pairs of libc
consumers...

> > To answer Wolfgang Denk's original question, in my opinion, the single
> > most valuable thing for accelerating the time64 process that third
> > parties could contribute is detailed analysis of the interfaces that
> > will break.
> 
> Such analysis is already done and available on glib'c wiki page [3].

...because there's no such analysis here. Such analysis is basically a
grep -r of /usr/include on a Debian system with *-dev installed,
combined with some automated analysis of what packages depend on each
matching package's library, then manual analysis to determine impact
(and in many cases, whether the lib is even a real lib that's used by
multiple programs rather than just by the program it ships with).

Arnd sent me a really preliminary grep like this a couple weeks ago,
but doing it right is going to take a lot more time.

Rich

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 3280 bytes --]

Hi Rich,

> On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
> > > As a result, I'm leaning strongly towards a conclusion that a
> > > solution that doesn't split the two cases is better, and it's
> > > probably what we'll do for musl.  
> > 
> > The approach would be very similar to what was done with
> > _FILE_OFFSET_BITS==64. The end user would need to add
> > -D_TIME_BITS==64 when compiling programs against glibc on 32 bit
> > architecture.  
> 
> Yes, except that I'd switch the default from the beginning. glibc made
> the mistake of not doing this with _FILE_OFFSET_BITS, and it's still
> the wrong default 20 years later, with every application's build
> system containing workarounds...
> 
> > > It's a viable solution for users to run new
> > > 32-bit stuff alongside a 32-bit glibc target that's being
> > > phased-out and nearing EOL, but if this is the only option I
> > > think it would just lead to distros dropping the targets.
> > >   
> > > > Any of the above approaches (including the current plan of
> > > > supporting time32 and time64 in a single glibc binary) can of
> > > > coexist, but the more incompatible approaches get implemented,
> > > > the more fragmentation of ABIs we get.    
> > > 
> > > I'm moderately in favor of supporting both in a single binary. As
> > > noted elsewhere, this fundamentally does not break ABI between
> > > libc consumers (main apps or 3rd party libs) and libc, only
> > > between pairs of libc consumers that have used the affected types
> > > as part of their public interfaces.  
> > 
> > As I've written above. For the end user it shall be enough to add
> > -D_TIME_BITS==64 during compilation.  
> 
> I think you missed the whole thing about ABI between pairs of libc
> consumers...

You are probably right, as I'm now biased to the systems where the
Y2038 safe BSP is build with OE/Yocto and deployed (the "self containing
release").

Thanks for extra explanation.

> 
> > > To answer Wolfgang Denk's original question, in my opinion, the
> > > single most valuable thing for accelerating the time64 process
> > > that third parties could contribute is detailed analysis of the
> > > interfaces that will break.  
> > 
> > Such analysis is already done and available on glib'c wiki page
> > [3].  
> 
> ...because there's no such analysis here. Such analysis is basically a
> grep -r of /usr/include on a Debian system with *-dev installed,
> combined with some automated analysis of what packages depend on each
> matching package's library, then manual analysis to determine impact
> (and in many cases, whether the lib is even a real lib that's used by
> multiple programs rather than just by the program it ships with).
> 
> Arnd sent me a really preliminary grep like this a couple weeks ago,
> but doing it right is going to take a lot more time.

Is this script available somewhere? I'm wondering if I could adapt it
(and run) to meta-y2038?

> 
> Rich




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Thu, Jul 18, 2019 at 09:20:48AM +0200, Lukasz Majewski wrote:
> > > > To answer Wolfgang Denk's original question, in my opinion, the
> > > > single most valuable thing for accelerating the time64 process
> > > > that third parties could contribute is detailed analysis of the
> > > > interfaces that will break.  
> > > 
> > > Such analysis is already done and available on glib'c wiki page
> > > [3].  
> > 
> > ...because there's no such analysis here. Such analysis is basically a
> > grep -r of /usr/include on a Debian system with *-dev installed,
> > combined with some automated analysis of what packages depend on each
> > matching package's library, then manual analysis to determine impact
> > (and in many cases, whether the lib is even a real lib that's used by
> > multiple programs rather than just by the program it ships with).
> > 
> > Arnd sent me a really preliminary grep like this a couple weeks ago,
> > but doing it right is going to take a lot more time.
> 
> Is this script available somewhere? I'm wondering if I could adapt it
> (and run) to meta-y2038?

See this thread: https://www.openwall.com/lists/musl/2019/07/02/2

Again, the point here is identifying libraries that have
time_t-derived structures as part of their public APIs, since that
means their *ABIs* are functions of how time_t is defined. There are
fairly many, but there's also good reason to believe that lots of them
aren't actually "public" in any meaningful sense.

Assuming the list is produced correctly and is complete, updating
these in sync with the apps (or other libs) that consume them avoids
any breakage, and is a much smaller task than replacing the whole
binary ecosystem all at once.

Rich

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
> Note:
> 
> [1] -
> https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019
> 
> [2] - https://github.com/lmajewski/y2038-tests
> 
> [3] -
> https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29
> 
> [4] - https://github.com/lmajewski/meta-y2038/tree/master

Some findings here that need correction:

[1] is completely missing the sysvipc interfaces affected, and [3]
fails to document them as affected because the structs are variadic
arguments not declared ones. Fortunately, this means we can get away
without actually replacing the functions, and instead define new
command numbers to perform the translation. When doing this, glibc
should follow musl and correct other bugs in these structs: for
example, struct ipc_perm's mode field has the wrong type on some archs
(short instead of mode_t; only makes a difference on big endian).

Rich

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Rich Felker:

> On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
>> Note:
>> 
>> [1] -
>> https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019
>> 
>> [2] - https://github.com/lmajewski/y2038-tests
>> 
>> [3] -
>> https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29
>> 
>> [4] - https://github.com/lmajewski/meta-y2038/tree/master
>
> Some findings here that need correction:
>
> [1] is completely missing the sysvipc interfaces affected, and [3]
> fails to document them as affected because the structs are variadic
> arguments not declared ones. Fortunately, this means we can get away
> without actually replacing the functions, and instead define new
> command numbers to perform the translation. When doing this, glibc
> should follow musl and correct other bugs in these structs: for
> example, struct ipc_perm's mode field has the wrong type on some archs
> (short instead of mode_t; only makes a difference on big endian).

Do the musl types match the kernel types?

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Thu, Jul 18, 2019 at 04:49:31PM +0200, Florian Weimer wrote:
> * Rich Felker:
> 
> > On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
> >> Note:
> >> 
> >> [1] -
> >> https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019
> >> 
> >> [2] - https://github.com/lmajewski/y2038-tests
> >> 
> >> [3] -
> >> https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29
> >> 
> >> [4] - https://github.com/lmajewski/meta-y2038/tree/master
> >
> > Some findings here that need correction:
> >
> > [1] is completely missing the sysvipc interfaces affected, and [3]
> > fails to document them as affected because the structs are variadic
> > arguments not declared ones. Fortunately, this means we can get away
> > without actually replacing the functions, and instead define new
> > command numbers to perform the translation. When doing this, glibc
> > should follow musl and correct other bugs in these structs: for
> > example, struct ipc_perm's mode field has the wrong type on some archs
> > (short instead of mode_t; only makes a difference on big endian).
> 
> Do the musl types match the kernel types?

Largely, since the kernel also left padding, but in the wrong order
for big endian. So instead of short+padding for mode on affected
archs, we have mode_t mode. On big endian variants of these archs,
this requires some fixup code to swap the upper/lower 16 bits.

With time64, if you're bothering to decode the extra time bits (on
most archs these are in adjacent padding and just need endian swapping
on big endian; on some archs they're in non-adjacent padding) already
then it's no big deal to fix the mode_t at the same time.

Note that on most archs, fixing these structs for time64 is just
replacing the 32-bit time_t and adjacent padding with a 64-bit time_t.
But on the exceptions (probably mips, because eew..) the struct needs
more significant changes.

Rich

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

On 18/07/2019 12:46, Rich Felker wrote:
> On Thu, Jul 18, 2019 at 04:49:31PM +0200, Florian Weimer wrote:
>> * Rich Felker:
>>
>>> On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
>>>> Note:
>>>>
>>>> [1] -
>>>> https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019
>>>>
>>>> [2] - https://github.com/lmajewski/y2038-tests
>>>>
>>>> [3] -
>>>> https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29
>>>>
>>>> [4] - https://github.com/lmajewski/meta-y2038/tree/master
>>>
>>> Some findings here that need correction:
>>>
>>> [1] is completely missing the sysvipc interfaces affected, and [3]
>>> fails to document them as affected because the structs are variadic
>>> arguments not declared ones. Fortunately, this means we can get away
>>> without actually replacing the functions, and instead define new
>>> command numbers to perform the translation. When doing this, glibc
>>> should follow musl and correct other bugs in these structs: for
>>> example, struct ipc_perm's mode field has the wrong type on some archs
>>> (short instead of mode_t; only makes a difference on big endian).
>>
>> Do the musl types match the kernel types?
> 
> Largely, since the kernel also left padding, but in the wrong order
> for big endian. So instead of short+padding for mode on affected
> archs, we have mode_t mode. On big endian variants of these archs,
> this requires some fixup code to swap the upper/lower 16 bits.
> 
> With time64, if you're bothering to decode the extra time bits (on
> most archs these are in adjacent padding and just need endian swapping
> on big endian; on some archs they're in non-adjacent padding) already
> then it's no big deal to fix the mode_t at the same time.
> 
> Note that on most archs, fixing these structs for time64 is just
> replacing the 32-bit time_t and adjacent padding with a 64-bit time_t.
> But on the exceptions (probably mips, because eew..) the struct needs
> more significant changes.
> 

This is BZ#18231 and I started to work on this some time ago. The main
issue is not really fix it, but consolidate all the implementations to
use a generic header for msg.h, sem.h, and shm.h, while providing 
compat symbol if it were the case.  I will try to check this out again,
since eventually I think we will need to get back to avoid a complicate
support for time64.

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

[-- Attachment #1.1: Type: text/plain, Size: 4286 bytes --]



On 17/07/2019 13:18, Lukasz Majewski wrote:
> Hi Florian,
> 
>> * Wolfgang Denk:
>>
>>> Dear Florian,
>>>
>>> In message <875zo0911b.fsf@oldenburg2.str.redhat.com> you wrote:  
>>>> * Arnd Bergmann:
>>>>  
>>>>> b) Those that already need support for 64-bit time_t because
>>>>>     they are deploying new 32-bit binaries that are expected to
>>>>> run beyond 2038, while not caring at all about compatibility
>>>>>     with existing binaries that are already known to be broken
>>>>>     for this purpose.  
>>>>
>>>> There is also c), new 32-bit architectures which need 64-bit time_t
>>>> support today due to kernel limitations.  Whether those binaries
>>>> need to run for two years or twenty does not matter to them.
>>>>
>>>> I have reviewed patches for the c) case, but that doesn't seem to
>>>> be work that interests Wolfgang.  
>>>
>>> Correct - our situation is Arnd's case b).
>>>
>>> But my understanding is that for c) glibc has to modify the generic
>>> syscalls wrapper (like clock_gettime/nanosleep/settime, etc.), and
>>> for b) we also need to do that first.  But currently we are stuck at
>>> the point where the __ASSUME_TIME64_SYSCALLS flag is not accepted /
>>> pulled.
>>>
>>> So b) and c) align in development...  
>>
>> Can you do without __ASSUME_TIME64_SYSCALLS?  Most other __ASSUME_*
>> macros are an optimization, and if your interest is b),
>> __ASSUME_TIME64_SYSCALLS will not be the default for glibc
>> distribution builds anyway because defining it would negatively
>> impact host kernel compatibility.  It's not just about containers in
>> the fashionable sense, but simple build chroots are problematic as
>> well in this context.
>>
>> Or have you received different guidance that __ASSUME_TIME64_SYSCALLS
>> markup is absolutely required for the initial contribution?
> 
> The __ASSUME_TIME64_SYSCALLS was discussed with Joseph and Stepan (both
> CC'ed) for a long time on the libc-alpha mailing list. The discussion
> can be found here [1] (last link is the newest one).
> 
> 
> 
> As fair as I understood from the previous discussion, adding
> __ASSUME_TIME64_SYSCALLS is a first step to add Y2038 support (or 64 bit
> time support to 32 bit archs in general).
> 
> The latest patch (v8) with semantics explanation of
> __ASSUME_TIME64_SYSCALLS: [2]
>  
> Note:
> 
> [1] Evolution of __ASSUME_TIME64_SYSCALLS up till v7:
> 
> https://patchwork.ozlabs.org/patch/1092579/
> https://patchwork.ozlabs.org/patch/1096343/
> https://patchwork.ozlabs.org/patch/1096349/
> https://patchwork.ozlabs.org/patch/1097132/
> https://patchwork.ozlabs.org/patch/1100097/
> https://patchwork.ozlabs.org/patch/1107125/
> https://patchwork.ozlabs.org/patch/1114061/
> https://patchwork.ozlabs.org/patch/1117100/
> 
> [2] - https://patchwork.ozlabs.org/patch/1117100/

More and more I see that a possible disruption in a release due to a 
possible ABI change should be a net gain over time by moving to a better
alternative.  The off64 migration required a lot of efforts and its 
issues still hit us in a way or another (BZ#23960), especially because 
off64 is *not* default for 32-bit builds neither on new 32-bits ports 
(on glibc side). Even now, from BZ#23960 comments, it seems that 
distributions still do not use LFS for some packages builds, it required 
Linux to force us to move forward.

So what about to not add a user-selected way to set the time_t size
(as off_t) and just enable time64_t support and sets is as default for 
build with a minimum kernel of v5.1 (if I recall correctly as being the 
one that added time64 support)? 

It will move all time32 support as compat symbols for the required ABI, 
without an option to select it through a build flag. Newer ports will
just have the expected symbol names, no tinkering needed.

And yes, it will mostly break stuff once people start to build distributions 
with these options. However, it will save us time to figure out all the bits
and joints required at cost of an initially high cost of adaptation plus
the all required code complexity and testing to support the multiple 
permutation possible.

I also wish we could also move forward with off_t and set LFS as default
as well.


  


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Adhemerval Zanella:

> So what about to not add a user-selected way to set the time_t size
> (as off_t) and just enable time64_t support and sets is as default for 
> build with a minimum kernel of v5.1 (if I recall correctly as being the 
> one that added time64 support)?

Does this mean that some developers see a glibc 2.31 with a 32-bit
time_t on i386, and others see a 64-bit time_t?

> It will move all time32 support as compat symbols for the required ABI, 
> without an option to select it through a build flag. Newer ports will
> just have the expected symbol names, no tinkering needed.

But if we build glibc with pre-5.1 kernel headers, you will get the
legacy interface?

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

On 18/07/2019 16:13, Florian Weimer wrote:
> * Adhemerval Zanella:
> 
>> So what about to not add a user-selected way to set the time_t size
>> (as off_t) and just enable time64_t support and sets is as default for 
>> build with a minimum kernel of v5.1 (if I recall correctly as being the 
>> one that added time64 support)?
> 
> Does this mean that some developers see a glibc 2.31 with a 32-bit
> time_t on i386, and others see a 64-bit time_t?
> 
>> It will move all time32 support as compat symbols for the required ABI, 
>> without an option to select it through a build flag. Newer ports will
>> just have the expected symbol names, no tinkering needed.
> 
> But if we build glibc with pre-5.1 kernel headers, you will get the
> legacy interface?

My idea is to setup based on --enable-kernel and add a new EI_ABIVERSION
when time64 is used.  So, depending on how glibc is built, developers
will indeed see a different time_t.

Legacy builds won't build any time64 support, even with newer kernels.
Build set with --enable-kernel will automatically set time_t as time64
and redirect the required interfaces to the new symbols ones (similar
to LFS, but transparently as _FILE_OFFSET_BITS=64 as being set as default).

Newer 32-bit ports won't require anything like this as expected.

I still think that in long term this initial bump transition is better
than to the complexity from mixed support.

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Adhemerval Zanella:

> On 18/07/2019 16:13, Florian Weimer wrote:
>> * Adhemerval Zanella:
>> 
>>> So what about to not add a user-selected way to set the time_t size
>>> (as off_t) and just enable time64_t support and sets is as default for 
>>> build with a minimum kernel of v5.1 (if I recall correctly as being the 
>>> one that added time64 support)?
>> 
>> Does this mean that some developers see a glibc 2.31 with a 32-bit
>> time_t on i386, and others see a 64-bit time_t?
>> 
>>> It will move all time32 support as compat symbols for the required ABI, 
>>> without an option to select it through a build flag. Newer ports will
>>> just have the expected symbol names, no tinkering needed.
>> 
>> But if we build glibc with pre-5.1 kernel headers, you will get the
>> legacy interface?
>
> My idea is to setup based on --enable-kernel and add a new EI_ABIVERSION
> when time64 is used.  So, depending on how glibc is built, developers
> will indeed see a different time_t.
>
> Legacy builds won't build any time64 support, even with newer kernels.
> Build set with --enable-kernel will automatically set time_t as time64
> and redirect the required interfaces to the new symbols ones (similar
> to LFS, but transparently as _FILE_OFFSET_BITS=64 as being set as default).
>
> Newer 32-bit ports won't require anything like this as expected.
>
> I still think that in long term this initial bump transition is better
> than to the complexity from mixed support.

I've relayed this proposal to Debian and Fedora:

  <https://lists.debian.org/debian-devel/2019/07/msg00384.html>
  <https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/WMNZEREJU5JHB5DH6YDAHFLZJEU3IFEF/>

I suggest others do this for the distributions they care about.

Re: Accelerating Y2038 glibc fixes

[Paul Eggert]

Adhemerval Zanella wrote:
> I still think that in long term this initial bump transition is better
> than to the complexity from mixed support.

Yes, that is what NetBSD did in release 6.0 (2012). OpenBSD release 5.5 (2014) 
followed suit. Admittedly these are smaller ecosystems, but the transitions went 
reasonably well as I recall. With my application-developer hat on, I prefer this 
simpler approach to the sort of complexity we went through with off_t in the 
1990s, a mess that we're still having to deal with.

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Thu, Jul 18, 2019 at 05:31:51PM -0300, Adhemerval Zanella wrote:
> 
> 
> On 18/07/2019 16:13, Florian Weimer wrote:
> > * Adhemerval Zanella:
> > 
> >> So what about to not add a user-selected way to set the time_t size
> >> (as off_t) and just enable time64_t support and sets is as default for 
> >> build with a minimum kernel of v5.1 (if I recall correctly as being the 
> >> one that added time64 support)?
> > 
> > Does this mean that some developers see a glibc 2.31 with a 32-bit
> > time_t on i386, and others see a 64-bit time_t?
> > 
> >> It will move all time32 support as compat symbols for the required ABI, 
> >> without an option to select it through a build flag. Newer ports will
> >> just have the expected symbol names, no tinkering needed.
> > 
> > But if we build glibc with pre-5.1 kernel headers, you will get the
> > legacy interface?
> 
> My idea is to setup based on --enable-kernel and add a new EI_ABIVERSION
> when time64 is used.  So, depending on how glibc is built, developers
> will indeed see a different time_t.
> 
> Legacy builds won't build any time64 support, even with newer kernels.
> Build set with --enable-kernel will automatically set time_t as time64
> and redirect the required interfaces to the new symbols ones (similar
> to LFS, but transparently as _FILE_OFFSET_BITS=64 as being set as default).
> 
> Newer 32-bit ports won't require anything like this as expected.
> 
> I still think that in long term this initial bump transition is better
> than to the complexity from mixed support.

I see no reason for it to depend on the minimum kernel version glibc
is built for. You should be able to build glibc that runs on the
oldest still-supported kernels and that uses 64-bit time_t in the
application-facing ABI.

If built with --enable-kernel new enough, the fallback code to use old
syscalls when the time64 ones fail with ENOSYS can be omitted. If not,
the time64 syscalls are tried first, and if they fail, the old
syscalls are used and the results converted.

This is what musl has always planned to do regardless of whether we go
with a hard ABI break or symbol redirection. "Only works on
bleeding-edge kernels" is a sure way to make sure nobody uses the new
thing, resulting in continued production of legacy binaries that are
ticking timebombs far into the future, and all the exploding popcorn
when 2038 approaches...

Rich

Re: Accelerating Y2038 glibc fixes

[Andreas Schwab]

On Jul 18 2019, Paul Eggert <eggert@cs.ucla.edu> wrote:

> Adhemerval Zanella wrote:
>> I still think that in long term this initial bump transition is better
>> than to the complexity from mixed support.
>
> Yes, that is what NetBSD did in release 6.0 (2012). OpenBSD release 5.5
> (2014) followed suit. Admittedly these are smaller ecosystems,

Rather non-existent.  *BSD doesn't care about backward compatibility.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

On 19/07/2019 00:06, Rich Felker wrote:
> On Thu, Jul 18, 2019 at 05:31:51PM -0300, Adhemerval Zanella wrote:
>>
>>
>> On 18/07/2019 16:13, Florian Weimer wrote:
>>> * Adhemerval Zanella:
>>>
>>>> So what about to not add a user-selected way to set the time_t size
>>>> (as off_t) and just enable time64_t support and sets is as default for 
>>>> build with a minimum kernel of v5.1 (if I recall correctly as being the 
>>>> one that added time64 support)?
>>>
>>> Does this mean that some developers see a glibc 2.31 with a 32-bit
>>> time_t on i386, and others see a 64-bit time_t?
>>>
>>>> It will move all time32 support as compat symbols for the required ABI, 
>>>> without an option to select it through a build flag. Newer ports will
>>>> just have the expected symbol names, no tinkering needed.
>>>
>>> But if we build glibc with pre-5.1 kernel headers, you will get the
>>> legacy interface?
>>
>> My idea is to setup based on --enable-kernel and add a new EI_ABIVERSION
>> when time64 is used.  So, depending on how glibc is built, developers
>> will indeed see a different time_t.
>>
>> Legacy builds won't build any time64 support, even with newer kernels.
>> Build set with --enable-kernel will automatically set time_t as time64
>> and redirect the required interfaces to the new symbols ones (similar
>> to LFS, but transparently as _FILE_OFFSET_BITS=64 as being set as default).
>>
>> Newer 32-bit ports won't require anything like this as expected.
>>
>> I still think that in long term this initial bump transition is better
>> than to the complexity from mixed support.
> 
> I see no reason for it to depend on the minimum kernel version glibc
> is built for. You should be able to build glibc that runs on the
> oldest still-supported kernels and that uses 64-bit time_t in the
> application-facing ABI.
> 
> If built with --enable-kernel new enough, the fallback code to use old
> syscalls when the time64 ones fail with ENOSYS can be omitted. If not,
> the time64 syscalls are tried first, and if they fail, the old
> syscalls are used and the results converted.
> 
> This is what musl has always planned to do regardless of whether we go
> with a hard ABI break or symbol redirection. "Only works on
> bleeding-edge kernels" is a sure way to make sure nobody uses the new
> thing, resulting in continued production of legacy binaries that are
> ticking timebombs far into the future, and all the exploding popcorn
> when 2038 approaches...

I am aiming for make the required changes less complex and allow the
distributions to have a way to actually set or not whether to support
time64, since you can still build glibc aiming for default minimum
(3.2) while still providing a time64 enabled kernel.  I personally prefer
to just make the switch based on release rather than a configure option.

And I do agree with you that it is feasible to just make the switch
on glibc 2.31 for all 32-bit architectures and move on. However
we will need to take care of glibc some conventions, so instead of just 

--
#if __LINUX_KERNEL_VERSION >= 0x050100
# define __ASSUME_TIME64_SYSCALLS 1
#endif

int
syscall_name (...)
{
#ifdef __ASSUME_TIME64_SYSCALLS
  return INLINE_SYSCALL_CALL (syscall64, ...);
#else
  return INLINE_SYSCALL_CALL (syscall, ...);
#endif
}
---

We will need to have something like:

--
#if __LINUX_KERNEL_VERSION >= 0x050100 >= 0x050100
# define __ASSUME_TIME64_SYSCALLS 1
#endif

int
syscall_name (args...)
{
#ifdef __ASSUME_TIME64_SYSCALLS
  return INLINE_SYSCALL_CALL (syscall64, args...);
#else  
  int ret;
# ifdef __NR_syscall64
  static int syscall_name_supported = 1;
  if (atomic_read_relaxed (&syscall_name_supported) == 1)
    {
      ret = INLINE_SYSCALL_CALL (syscall64, args...);
      if (ret == 0 || errno != ENOSYS)
	return ret;

      atomic_store_relaxed (&syscall_name_supported, 0)
    }
# endif /* __NR_syscall64  */
  
  ret = INLINE_SYSCALL_CALL (syscall, args...);
  if (ret == 0)
    ret = convert_syscall_time_to_time64 (args, ...);
  return ret;
#endif
}
--

For the syscall implementation. We will have some more complexity to handle
vDSO, but it should be simplified with my refactor.

Re: Accelerating Y2038 glibc fixes

[Alistair Francis]

On Fri, Jul 19, 2019 at 10:44 AM Adhemerval Zanella
<adhemerval.zanella@linaro.org> wrote:
>
>
>
> On 19/07/2019 00:06, Rich Felker wrote:
> > On Thu, Jul 18, 2019 at 05:31:51PM -0300, Adhemerval Zanella wrote:
> >>
> >>
> >> On 18/07/2019 16:13, Florian Weimer wrote:
> >>> * Adhemerval Zanella:
> >>>
> >>>> So what about to not add a user-selected way to set the time_t size
> >>>> (as off_t) and just enable time64_t support and sets is as default for
> >>>> build with a minimum kernel of v5.1 (if I recall correctly as being the
> >>>> one that added time64 support)?
> >>>
> >>> Does this mean that some developers see a glibc 2.31 with a 32-bit
> >>> time_t on i386, and others see a 64-bit time_t?
> >>>
> >>>> It will move all time32 support as compat symbols for the required ABI,
> >>>> without an option to select it through a build flag. Newer ports will
> >>>> just have the expected symbol names, no tinkering needed.
> >>>
> >>> But if we build glibc with pre-5.1 kernel headers, you will get the
> >>> legacy interface?
> >>
> >> My idea is to setup based on --enable-kernel and add a new EI_ABIVERSION
> >> when time64 is used.  So, depending on how glibc is built, developers
> >> will indeed see a different time_t.
> >>
> >> Legacy builds won't build any time64 support, even with newer kernels.
> >> Build set with --enable-kernel will automatically set time_t as time64
> >> and redirect the required interfaces to the new symbols ones (similar
> >> to LFS, but transparently as _FILE_OFFSET_BITS=64 as being set as default).
> >>
> >> Newer 32-bit ports won't require anything like this as expected.
> >>
> >> I still think that in long term this initial bump transition is better
> >> than to the complexity from mixed support.
> >
> > I see no reason for it to depend on the minimum kernel version glibc
> > is built for. You should be able to build glibc that runs on the
> > oldest still-supported kernels and that uses 64-bit time_t in the
> > application-facing ABI.
> >
> > If built with --enable-kernel new enough, the fallback code to use old
> > syscalls when the time64 ones fail with ENOSYS can be omitted. If not,
> > the time64 syscalls are tried first, and if they fail, the old
> > syscalls are used and the results converted.
> >
> > This is what musl has always planned to do regardless of whether we go
> > with a hard ABI break or symbol redirection. "Only works on
> > bleeding-edge kernels" is a sure way to make sure nobody uses the new
> > thing, resulting in continued production of legacy binaries that are
> > ticking timebombs far into the future, and all the exploding popcorn
> > when 2038 approaches...
>
> I am aiming for make the required changes less complex and allow the
> distributions to have a way to actually set or not whether to support
> time64, since you can still build glibc aiming for default minimum
> (3.2) while still providing a time64 enabled kernel.  I personally prefer
> to just make the switch based on release rather than a configure option.
>
> And I do agree with you that it is feasible to just make the switch
> on glibc 2.31 for all 32-bit architectures and move on. However
> we will need to take care of glibc some conventions, so instead of just
>
> --
> #if __LINUX_KERNEL_VERSION >= 0x050100
> # define __ASSUME_TIME64_SYSCALLS 1
> #endif
>
> int
> syscall_name (...)
> {
> #ifdef __ASSUME_TIME64_SYSCALLS
>   return INLINE_SYSCALL_CALL (syscall64, ...);
> #else
>   return INLINE_SYSCALL_CALL (syscall, ...);
> #endif
> }
> ---
>
> We will need to have something like:
>
> --
> #if __LINUX_KERNEL_VERSION >= 0x050100 >= 0x050100
> # define __ASSUME_TIME64_SYSCALLS 1
> #endif
>
> int
> syscall_name (args...)
> {
> #ifdef __ASSUME_TIME64_SYSCALLS
>   return INLINE_SYSCALL_CALL (syscall64, args...);
> #else
>   int ret;
> # ifdef __NR_syscall64
>   static int syscall_name_supported = 1;
>   if (atomic_read_relaxed (&syscall_name_supported) == 1)
>     {
>       ret = INLINE_SYSCALL_CALL (syscall64, args...);
>       if (ret == 0 || errno != ENOSYS)
>         return ret;
>
>       atomic_store_relaxed (&syscall_name_supported, 0)
>     }
> # endif /* __NR_syscall64  */
>
>   ret = INLINE_SYSCALL_CALL (syscall, args...);
>   if (ret == 0)
>     ret = convert_syscall_time_to_time64 (args, ...);
>   return ret;
> #endif
> }
> --

Great! This was my understanding as well (although not every patch in
the latest RV32 RFC does this, that is an oversight).

>
> For the syscall implementation. We will have some more complexity to handle
> vDSO, but it should be simplified with my refactor.

Also good news!

Alistair

Re: Accelerating Y2038 glibc fixes

[Rich Felker]

On Thu, Jul 18, 2019 at 10:47:15AM -0400, Rich Felker wrote:
> On Wed, Jul 17, 2019 at 11:57:48PM +0200, Lukasz Majewski wrote:
> > Note:
> > 
> > [1] -
> > https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-11-03-2019
> > 
> > [2] - https://github.com/lmajewski/y2038-tests
> > 
> > [3] -
> > https://sourceware.org/glibc/wiki/Y2038ProofnessDesign?highlight=%28y2038%29
> > 
> > [4] - https://github.com/lmajewski/meta-y2038/tree/master
> 
> Some findings here that need correction:
> 
> [1] is completely missing the sysvipc interfaces affected, and [3]
> fails to document them as affected because the structs are variadic
> arguments not declared ones. Fortunately, this means we can get away
> without actually replacing the functions, and instead define new
> command numbers to perform the translation. When doing this, glibc
> should follow musl and correct other bugs in these structs: for
> example, struct ipc_perm's mode field has the wrong type on some archs
> (short instead of mode_t; only makes a difference on big endian).

Another omission I found: ftw/nftw. They pass stat structures back to
a callback. If the time64 stat structures start with the old stat64
structure and just append the new timespecs to the end, no action is
required here for compatibility. This is what I'm leaning towards
doing in musl, but it doesn't seem to be what the above glibc branch
does. If glibc is going to completely redefine the stat structure for
time64, then new versions ftw and nftw are needed along with
redirections for them. There may be other interfaces like this too. I
seem to recall glibc having another nonstandard interface like ftw,
but it being broken and not even supporting _FILE_OFFSET_BITS=64
yet, so no idea what would happen to it...

Rich

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Fri, 12 Jul 2019, Wolfgang Denk wrote:

> See for example [1] - there are just 7 lines of "code".  But Joseph
> does not accept our patches.  The arguments he gives are not on a
> technical level;

The patch is providing a technical specification that is supposed to 
underpin both subsequent patches in the series and maintenance of code in 
this area of glibc over several years - an area where complete clarity of 
the intended interface is critical.  That the specification in question is 
seriously vague with emphasis on the wrong concepts is very much a 
technical issue.  The code is very much the easiest and least important 
part of this patch.

> instead he says that only a native English speaker
> who has a with deep understanding of glibc internals and the
> previous development process will be capable to provide such a patch
> in an acceptable way.

That is not what I said.  I said it makes more sense for someone with that 
familiarity to do the editing - that is, that would enable the 
specification to achieve consensus sooner than going through a long 
sequence of patch reviews.

Writing detailed, precise explanations of exactly how something vague is 
vague and how the concepts referenced aren't quite the right ones is 
itself very time-consuming (listing all the deficiencies in a sentence 
such as "To be more specific - this flag focuses on higer level of 
abstraction, which indicates the system's ability to support 64 bit time." 
in the specification would result in an enumeration much longer than that 
sentence itself), but if you'd rather proceed with such reviews we can do 
so.  Based on previous experience with reviews of patches in this series, 
that would likely take several more iterations to get a specification of 
reasonable quality than simply rewriting the text in question.

> [1] https://patchwork.ozlabs.org/patch/1114061/

This is not the most recent version of the patch posted.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Thu, 18 Jul 2019, Adhemerval Zanella wrote:

> So what about to not add a user-selected way to set the time_t size
> (as off_t) and just enable time64_t support and sets is as default for 
> build with a minimum kernel of v5.1 (if I recall correctly as being the 
> one that added time64 support)? 

I think duplicating ABIs like this is a very bad idea - the ABI supported 
by glibc for a configuration that currently has 32-bit time_t should not 
change to have two different, incompatible variants depending on how glibc 
is configured.  The default API provided by glibc should also not vary 
like that depending on how glibc is configured.

Given that you have compatibility with existing binaries (as opposed to a 
complete new incompatible libc.so.7 ABI with building libc.so.6 for that 
architecture ceasing to be supported by glibc), eliminating support for 
building with _TIME_BITS=32 makes the changes much *harder*, not easier.  
There are three logical steps in the time_t transition.

(a) Support building applications for 64-bit time_t, using _TIME_BITS=64.

(b) Change the default to _TIME_BITS=64 (also requires defaulting to 
_FILE_OFFSET_BITS=64).

(c) Remove support for building applications with _TIME_BITS=32.

The hardest of those steps is (c), not (a), because of the difficulty in 
both building and testing all the compatibility symbols after step (c), 
without all the header redirections of symbols getting in the way when 
building those functions and the tests for them.

In view of the difficulty of both (c) and (a), it clearly makes sense to 
separate them, and start with (a), with (c) to follow some time later.

> I also wish we could also move forward with off_t and set LFS as default 
> as well.

We can (it's the equivalent of (c), not of (b), that's particularly hard, 
though even when doing (b) you need to be careful you keep sufficient test 
coverage for both function variants).  See what I said in 
<https://sourceware.org/ml/libc-alpha/2019-01/msg00124.html>.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 3254 bytes --]

Hi Joseph,

> On Fri, 12 Jul 2019, Wolfgang Denk wrote:
> 

Wolfgang went for his holidays, so I hope that you don't mind if I
reply.

> > See for example [1] - there are just 7 lines of "code".  But Joseph
> > does not accept our patches.  The arguments he gives are not on a
> > technical level;  
> 
> The patch is providing a technical specification that is supposed to 
> underpin both subsequent patches in the series and maintenance of
> code in this area of glibc over several years - an area where
> complete clarity of the intended interface is critical.  That the
> specification in question is seriously vague with emphasis on the
> wrong concepts is very much a technical issue.  The code is very much
> the easiest and least important part of this patch.
> 

Yes. We are aware of this. However, we discuss this problem since the
beginning of the year (and yes, I'm aware about your limited time,
which you can spent on this). 

> > instead he says that only a native English speaker
> > who has a with deep understanding of glibc internals and the
> > previous development process will be capable to provide such a patch
> > in an acceptable way.  
> 
> That is not what I said.  I said it makes more sense for someone with
> that familiarity to do the editing - that is, that would enable the 
> specification to achieve consensus sooner than going through a long 
> sequence of patch reviews.

The problem is your limited "time budget" on this issue as well as the
very low interest from the glibc community (as Alistair tries to add
RISC-V Y2038 safe 32 bit port).

Our goal is to add a solid foundation for the Y2038 work, so we would
know the direction where we are heading.

> 
> Writing detailed, precise explanations of exactly how something vague
> is vague and how the concepts referenced aren't quite the right ones
> is itself very time-consuming (listing all the deficiencies in a
> sentence such as "To be more specific - this flag focuses on higer
> level of abstraction, which indicates the system's ability to support
> 64 bit time." in the specification would result in an enumeration
> much longer than that sentence itself), but if you'd rather proceed
> with such reviews we can do so.  Based on previous experience with
> reviews of patches in this series, that would likely take several
> more iterations to get a specification of reasonable quality than
> simply rewriting the text in question.

If you think that it would be better and most of all faster if you
rewrite the description, then I don't mind. 

It would be great if you could do it sooner than latter as this slows
down our development.

> 
> > [1] https://patchwork.ozlabs.org/patch/1114061/  
> 
> This is not the most recent version of the patch posted.

Yes, correct. I must have shared wrong link with Wolfgang.

The most recent version of this patch set (v8):
https://patchwork.ozlabs.org/patch/1117100/

> 




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

On 25/07/2019 17:40, Joseph Myers wrote:
> On Thu, 18 Jul 2019, Adhemerval Zanella wrote:
> 
>> So what about to not add a user-selected way to set the time_t size
>> (as off_t) and just enable time64_t support and sets is as default for 
>> build with a minimum kernel of v5.1 (if I recall correctly as being the 
>> one that added time64 support)? 
> 
> I think duplicating ABIs like this is a very bad idea - the ABI supported 
> by glibc for a configuration that currently has 32-bit time_t should not 
> change to have two different, incompatible variants depending on how glibc 
> is configured.  The default API provided by glibc should also not vary 
> like that depending on how glibc is configured.

Later on the thread [1] I did state I would prefer switch based on release
rather than a configure option, the suggestion was initially as a way to
easier the transition (at the cost of complexity I give you).

> 
> Given that you have compatibility with existing binaries (as opposed to a 
> complete new incompatible libc.so.7 ABI with building libc.so.6 for that 
> architecture ceasing to be supported by glibc), eliminating support for 
> building with _TIME_BITS=32 makes the changes much *harder*, not easier.  
> There are three logical steps in the time_t transition.
> 
> (a) Support building applications for 64-bit time_t, using _TIME_BITS=64.
> 
> (b) Change the default to _TIME_BITS=64 (also requires defaulting to 
> _FILE_OFFSET_BITS=64).
> 
> (c) Remove support for building applications with _TIME_BITS=32.
> 
> The hardest of those steps is (c), not (a), because of the difficulty in 
> both building and testing all the compatibility symbols after step (c), 
> without all the header redirections of symbols getting in the way when 
> building those functions and the tests for them.
> 
> In view of the difficulty of both (c) and (a), it clearly makes sense to 
> separate them, and start with (a), with (c) to follow some time later.

Since we require to have both time32 and time64 implementation for the
'legacy' 32-bit architectures, the change to implement (c) is mainly to
make the symbol compat ones.  And since we will need to internal logic
to select whether to build time32 implementations (since for newer 32-bit
ABI time32 is not a option from kernel side and it does not make sense
to add time32 interfaces for such cases), the same applies for make them
a compat symbol.

The tricky part is testing, where it would require some more boilerplate 
to redirect to compat symbol instead of just define the _TIME_BITS, but
 with proper infrastructure on libsupport I still think this is feasible.

One extra step we will need to implement is to extend the abilist to take
in consideration whether the symbol is not the default version.

So I am not sure if (c) is really the most difficult part of the required
changes.

The question I have is what is the real gain of still supporting _TIME_BITS=32
as a build option, if the idea is default to _TIME_BITS=64.  It open a 
possibility to programs that are not time64 ready to just add _TIME_BITS=32
as a workaround, which is clearly *not* the desirable long-term desirable
option.

> 
>> I also wish we could also move forward with off_t and set LFS as default 
>> as well.
> 
> We can (it's the equivalent of (c), not of (b), that's particularly hard, 
> though even when doing (b) you need to be careful you keep sufficient test 
> coverage for both function variants).  See what I said in 
> <https://sourceware.org/ml/libc-alpha/2019-01/msg00124.html>.
> 

I recall the thread and your suggestion as the guideline to move forward [2].
However the getdents64 regression (BZ#23960) shows us that even after two
decades, at lot of environments still don't use _FILE_OFFSET_BITS=64 as default
and it most likely will take even more time to actually make it to land of
some projects.

This is most likely the usual usercases don't actually required LFS support and
only handful scenarios might trigger some issues with non-LFS support.  And
*now* is true for time_t interfaces as well, programs will continue to be build
and deployed with default options, or even worse, some pieces will be build
with different time_t abi which IMHO it even more difficult to debug. They will 
move forward to a safer ABI only when things start breaks.

So, different than LFS, I see no point in not moving forward to time64 and just
make time32 deprecated/compat.

[1] https://sourceware.org/ml/libc-alpha/2019-07/msg00456.html
[2] https://sourceware.org/ml/libc-alpha/2019-01/msg00177.html

Re: Accelerating Y2038 glibc fixes

[Zack Weinberg]

On Fri, Jul 26, 2019 at 6:39 AM Lukasz Majewski <lukma@denx.de> wrote:
...
> > > See for example [1] - there are just 7 lines of "code".  But Joseph
> > > does not accept our patches.  The arguments he gives are not on a
> > > technical level;
...
> Our goal is to add a solid foundation for the Y2038 work, so we would
> know the direction where we are heading.
...
> If you think that it would be better and most of all faster if you
> rewrite the description, then I don't mind.
>
> It would be great if you could do it sooner than latter as this slows
> down our development.
...
> The most recent version of this patch set (v8):
> https://patchwork.ozlabs.org/patch/1117100/

I haven’t been following the details of these patches super carefully,
and I’m not sure I understand what _Joseph’s_ concerns with your
writing is.  However, I’m a native English speaker, I’ve read over the
text in the patch at <https://patchwork.ozlabs.org/patch/1117100/>, I
do think I understand the issues at a high level, and I do think the
meaning of __ASSUME_TIME64_SYSCALLS could be explained more clearly.
I’m prepared to work with you to come up with better wording but I
need to ask you a bunch of questions.  Could you please reply to each
of the queries marked Qn below?

As I understand it, we have five distinct cases to consider:

1. All future LP64 ABIs and all but one existing LP64 ABI, identified
   by __WORDSIZE == 64: time_t is already a 64-bit integer type and
   all of the relevant system calls already accept it in that form.
   glibc’s implementation of, for instance, clock_gettime may continue
   to invoke the system call numbered __NR_clock_gettime.

2. The exception to (1) is Alpha.  That is an exclusively LP64
   architecture, but in glibc 2.0 it used 32-bit time_t, and we still
   have compatibility code for that case.  The compatibility symbols
   invoke a set of compatibility syscalls with ‘osf’ in their names:
   for instance, gettimeofday@GLIBC_2.0 invokes __NR_osf_gettimeofday.
   Not all of the time-related functions in glibc have compatibility
   symbols, only those that existed in version 2.0.

   Your patches do not touch this compatibility code at all, as far as
   I can see.  Alpha being out of production, and binaries compiled
   against glibc 2.0 being rare anyway, it would only make sense to
   involve this code in your patches if it reduced the overall volume
   of compatibility code somehow, but regardless we need to make sure
   it doesn't break.

3. x32 (recently new 32-bit ABI for x86): like case 1, time_t is
   already 64-bit and we use unsuffixed system calls.  The text says
   this case is identified by __WORDSIZE == 32 && __TIMESIZE == 64,
   but the code actually checks __SYSCALL_WORDSIZE.

   Q1: Which condition correctly identifies this case, __TIMESIZE == 64 or
   __SYSCALL_WORDSIZE == 64?

   Q2: Could we perhaps ensure that __TIMESIZE and/or __SYSCALL_WORDSIZE
   is defined to 64 whenever __WORDSIZE == 64? Then we could collapse
   this into case 1.

4. Brand-new (added in kernel 5.1 or later) 32-bit ABIs: time_t will
   always be 64-bit, _but_ glibc’s implementation of time-related APIs
   may need to invoke system calls with suffixed names: clock_gettime
   invoking __NR_clock_gettime64, for instance.  Also, the kernel will
   not provide all of the time-related system calls that have
   historically existed; glibc must, for instance, implement
   gettimeofday in terms of clock_gettime.

   Q3: What macros are defined for this case?

   Q4: Does glibc need to call system calls with suffixed names in
   this case?

   Q4a: If the answer to Q4 is ‘yes’, why is that, and can we change
   the kernel so that it’s the same as x32 and the LP64 architectures?
   (Either by removing the suffixes, or by _adding_ suffixed aliases
   to asm/unistd.h for x32 and LP64 architectures.)

5. Historical 32-bit ABIs, where the existing set of system calls
   takes 32-bit time_t, and Linux 5.1 added a matching set that takes
   64-bit time_t.  For compatibility with old programs that make
   direct system calls, the kernel will not rename the __NR_ constants
   for the old (32-bit) system calls; instead new constants with ‘64’
   or ‘time64’ suffixes will be added.  As in case 4, the new set of
   system calls does not cover all of the historic time-related system
   calls.

   In this case, and only this case, glibc’s code needs to account for
   the possibility that the new __NR_ constants are not defined
   (because glibc is being compiled against kernel headers from a
   version older than 5.1), or that the new system calls are not
   available at runtime (glibc was compiled against new kernel headers
   but is running with an old kernel).

   The #if is complicated enough that I’m not sure, but I _think_ your
   code only defines __ASSUME_TIME64_SYSCALLS when the new constants
   are _guaranteed_ to be defined.

   Q5: Is it correct that __ASSUME_TIME64_SYSCALLS is only defined
   when the new constants are guaranteed to be defined?

   Q6: All of the other __ASSUME_ constants mean both that we assume
   the kernel headers are new enough to provide all the necessary
   declarations, and that we assume the feature is available at
   runtime: no fallback code will be included in the library.  Is this
   also the intended meaning of __ASSUME_TIME64_SYSCALLS?

zw

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Mon, 29 Jul 2019, Adhemerval Zanella wrote:

> > I think duplicating ABIs like this is a very bad idea - the ABI supported 
> > by glibc for a configuration that currently has 32-bit time_t should not 
> > change to have two different, incompatible variants depending on how glibc 
> > is configured.  The default API provided by glibc should also not vary 
> > like that depending on how glibc is configured.
> 
> Later on the thread [1] I did state I would prefer switch based on release
> rather than a configure option, the suggestion was initially as a way to
> easier the transition (at the cost of complexity I give you).

I'm not clear what "switch based on release" means.

> Since we require to have both time32 and time64 implementation for the
> 'legacy' 32-bit architectures, the change to implement (c) is mainly to
> make the symbol compat ones.  And since we will need to internal logic

No, it's mainly (for a large number of functions) finding some way to 
avoid the unconditional (for platforms with __TIMESIZE == 32) header 
redirects from <func> to __<func>_time64 applying to the definitions of 
those compat symbols (and, likewise, for all the functions that have 
variants for _FILE_OFFSET_BITS=32, because we don't want to support the 
combination of 32-bit offsets with 64-bit times, and requiring 64-bit 
times implies first requiring 64-bit offsets).  You can do that with a 
suitable #define before including the header and #undef after, but there 
are many functions, and different implementations of those functions to 
deal with - and then there is the testing issue, where the changes are 
probably even more involved.

Just adding new function variants with new names and header redirection to 
provide optional support for using them is much simpler than anything that 
also obsoletes the old functions.

> The question I have is what is the real gain of still supporting _TIME_BITS=32
> as a build option, if the idea is default to _TIME_BITS=64.  It open a 

The gain is supporting building glibc itself for such configurations 
without a large amount of complicated work to build and test compat 
symbols needing to be done up front - allowing the transition to be broken 
down into more reasonably sized pieces.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

[-- Attachment #1: Type: text/plain, Size: 1863 bytes --]

On Mon, 29 Jul 2019, Zack Weinberg wrote:

> I’m prepared to work with you to come up with better wording but I
> need to ask you a bunch of questions.  Could you please reply to each
> of the queries marked Qn below?

Note that while these cases are things we should think about in working 
out (for example) what the best semantics for __ASSUME_TIME64_SYSCALLS 
are, they shouldn't be part of how it's defined (a definition that says 
this is what happens in each of five cases is not a cleanly defined 
interface); the definition should rather be such that readers can see what 
the answer would be for each of those cases - and for any other cases that 
may arise in future.  (I think the current definition is that it means 
either a specified set of suffixed syscalls using 64-bit time are 
guaranteed to be available at runtime, *or* that the corresponding 
unsuffixed syscalls use 64-bit time and are guaranteed to be available, so 
that #define of the suffixed names to the unsuffixed ones is OK in that 
case.)

There is at least one case you didn't list (or a variant of case 5 that's 
different as far as glibc's concerned but not as far as the kernel's 
concerned): new glibc ports for ILP32 ABIs where the oldest kernel version 
supported is older than 5.1, should we wish for any such port to support 
only 64-bit time and not 32-bit time (so __TIMESIZE == 64, __WORDSIZE == 
64, __SYSCALL_WORDSIZE == 64) - and once we have the support in glibc to 
make it possible not to support 32-bit time in such a case, it seems a 
good idea for any such new ports to use it.

__WORDSIZE is the size, in bits, of "long int".  __SYSCALL_WORDSIZE (when 
defined) is the size, in bits, of __syscall_slong_t (the "long int" type 
in the syscall interface, which is the same as userspace "long int" except 
for x32).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Adhemerval Zanella]

On 29/07/2019 16:58, Joseph Myers wrote:
> On Mon, 29 Jul 2019, Adhemerval Zanella wrote:
> 
>>> I think duplicating ABIs like this is a very bad idea - the ABI supported 
>>> by glibc for a configuration that currently has 32-bit time_t should not 
>>> change to have two different, incompatible variants depending on how glibc 
>>> is configured.  The default API provided by glibc should also not vary 
>>> like that depending on how glibc is configured.
>>
>> Later on the thread [1] I did state I would prefer switch based on release
>> rather than a configure option, the suggestion was initially as a way to
>> easier the transition (at the cost of complexity I give you).
> 
> I'm not clear what "switch based on release" means.

To make the time_t change its type on a new glibc release rather than the
configure option.

> 
>> Since we require to have both time32 and time64 implementation for the
>> 'legacy' 32-bit architectures, the change to implement (c) is mainly to
>> make the symbol compat ones.  And since we will need to internal logic
> 
> No, it's mainly (for a large number of functions) finding some way to 
> avoid the unconditional (for platforms with __TIMESIZE == 32) header 
> redirects from <func> to __<func>_time64 applying to the definitions of 
> those compat symbols (and, likewise, for all the functions that have 
> variants for _FILE_OFFSET_BITS=32, because we don't want to support the 
> combination of 32-bit offsets with 64-bit times, and requiring 64-bit 
> times implies first requiring 64-bit offsets).  You can do that with a 
> suitable #define before including the header and #undef after, but there 
> are many functions, and different implementations of those functions to 
> deal with - and then there is the testing issue, where the changes are 
> probably even more involved.
> 
> Just adding new function variants with new names and header redirection to 
> provide optional support for using them is much simpler than anything that 
> also obsoletes the old functions.

I think it would be easier than what you described because we won't need
to actually to add any header redefinition, all symbol affect will just
use the new time_t regardless of the ABI.  The compat implementation will
use an internal-only type to use the old one.

What it would require is to add compat implementations with a different
type, time32_t for instance.  Something like:

---
* sysdeps/unix/sysv/linux/generic/bits/typesizes.h
[...]
#define __TIME_T_TYPE           __UQUAD_TYPE
[...]

* time/time.h
[...]
extern time_t time (time_t *__timer) __THROW;
[...]

* sysdeps/unix/sysv/linux/time.c
[...]
time_t
__time (time_t *t)
{ 
#ifdef __ASSUME_TIME64_SYSCALLS
  return INLINE_SYSCALL_CALL (time64, t);
#else  
  int ret;
# ifdef __NR_time64
  /* Maybe we can use 'sets' of 'supported' to enable/disable
     multiple syscalls.  */
  static int time64_supported = 1;
  if (atomic_read_relaxed (&time64_supported) == 1)
    {
      ret = INLINE_SYSCALL_CALL (time64, t);
      if (ret == 0 || errno != ENOSYS)
	return ret;

      atomic_store_relaxed (&time64_supported, 0)
    }
# endif /* __NR_syscall64  */
  
  ret = INLINE_SYSCALL_CALL (time, t);
  /* I assume kernel will return EOVERFLOW if the case.  Other symbol
     will require further handling.  */
  return ret;
#endif
}
libc_hidden_def (time)

/* At first it would be a arch-specific definition, on kernel-features.h.
   Maybe there is a clever way to accomplish it.  */
#ifdef __REQUIRE_TIME32_COMPAT
/* Define time32_t internally somewhere.  */
time32_t
__time32 (time32_t *t)
{
  return INLINE_SYSCALL_CALL (time, t);
}
compat_symbol (libc, __time32, time, GLIBC_2_0);
versioned_symbol (libc, __time, time, GLIBC_2_SOMETHING);
#else
weak_alias (__time, time)
#endif
[...]
---

I used the simplest symbol I can think off, other symbol that use time_t embedded
inside more complex struct will need to replicate it on some compat header (as
we do for some internal definition such sigaction).

Testing will require more boilerplate to link against the compat symbol, but
we already have some support on libsupport and I think we can improve to
make it simpler.

> 
>> The question I have is what is the real gain of still supporting _TIME_BITS=32
>> as a build option, if the idea is default to _TIME_BITS=64.  It open a 
> 
> The gain is supporting building glibc itself for such configurations 
> without a large amount of complicated work to build and test compat 
> symbols needing to be done up front - allowing the transition to be broken 
> down into more reasonably sized pieces.
> 

I don't have a strong opinion if a patch proposal use the _TIME_BITS=32
as a initial transition to enable time64 support, however I see no point
in make it available either on a release point neither in long term. 

In fact I see that support_TIME_BITS=32 on long term is a detriment for 
the ecosystem as whole. It is error prone, it will be glibc specific 
(since other system that used to have the same issue already did that
transition without similar complexity, such as BSD), and it will lead 
to subtle bugs with the multiple build systems that we have in Linux 
ecosystem.

That's why I still think that an initial rough transition now to move
forward definitely is better than still support broken interfaces 
indefinitely (not considering the required compat implementations).

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Mon, 29 Jul 2019, Adhemerval Zanella wrote:

> I think it would be easier than what you described because we won't need
> to actually to add any header redefinition, all symbol affect will just
> use the new time_t regardless of the ABI.  The compat implementation will
> use an internal-only type to use the old one.
> 
> What it would require is to add compat implementations with a different
> type, time32_t for instance.  Something like:

That seems much harder to implement through incremental development, 
whereas with _TIME_BITS=64 things can readily be implemented incrementally 
(adding 64-bit functions one-by-one with 32-bit ones as wrappers - already 
done for various mktime / timezone functions, for example - with only the 
final addition of _TIME_BITS=64 support in the headers and exporting 
symbols at public symbol versions needing to be one big monolithic patch).

> I don't have a strong opinion if a patch proposal use the _TIME_BITS=32
> as a initial transition to enable time64 support, however I see no point
> in make it available either on a release point neither in long term. 

The point is _TIME_BITS=64 as an API, not _TIME_BITS=32.  Since the change 
to flip the default is small, I don't think it's very significant whether 
or not there is a release where the default is 64-bit but _TIME_BITS=32 is 
available as well (whereas having _TIME_BITS=64 available before it 
becomes the default is a practical matter of both reviewability and 
allowing users to plan the timing of their own transition to 64-bit 
times).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Paul Eggert]

On 7/29/19 4:08 PM, Joseph Myers wrote:
>> What it would require is to add compat implementations with a different
>> type, time32_t for instance.  Something like:
> That seems much harder to implement through incremental development,
> whereas with _TIME_BITS=64 things can readily be implemented incrementally

I worry that this puts the incremental cart before the incremental 
horse, at least for portable applications. If we take the _TIME_BITS=64 
approach, portable applications should add the equivalent of "#define 
_TIME_BITS 64" to their config.h files or ihatever, because they should 
be Y2038-safe. But if glibc development is incremental so that some 
time_t uses are glitchy when _TIME_BITS=64, these applications will have 
problems. Conversely, if glibc doesn't publicize the time_t changes 
until they're ready (which I think is the intent), the _TIME_BITS=32 
approach would be less hassle overall for portable applications; they 
won't have to change their source code.

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Mon, 29 Jul 2019, Paul Eggert wrote:

> On 7/29/19 4:08 PM, Joseph Myers wrote:
> > > What it would require is to add compat implementations with a different
> > > type, time32_t for instance.  Something like:
> > That seems much harder to implement through incremental development,
> > whereas with _TIME_BITS=64 things can readily be implemented incrementally
> 
> I worry that this puts the incremental cart before the incremental horse, at
> least for portable applications. If we take the _TIME_BITS=64 approach,
> portable applications should add the equivalent of "#define _TIME_BITS 64" to
> their config.h files or ihatever, because they should be Y2038-safe. But if
> glibc development is incremental so that some time_t uses are glitchy when
> _TIME_BITS=64, these applications will have problems. Conversely, if glibc
> doesn't publicize the time_t changes until they're ready (which I think is the
> intent), the _TIME_BITS=32 approach would be less hassle overall for portable
> applications; they won't have to change their source code.

Header support for _TIME_BITS=64 shouldn't be added until all functions 
fully support it.  At that point, having it not be the default for some 
number of releases helps users plan the ABI transition for their libraries 
(rather than getting an ABI break by default for any library that gets 
rebuilt with newer glibc unless a special option is used, they can plan a 
whole-distribution transition on their own timescale).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 2339 bytes --]

Hi Joseph, Zack,

> On Mon, 29 Jul 2019, Zack Weinberg wrote:
> 
> > I’m prepared to work with you to come up with better wording but I
> > need to ask you a bunch of questions.  Could you please reply to
> > each of the queries marked Qn below?  
> 
> Note that while these cases are things we should think about in
> working out (for example) what the best semantics for
> __ASSUME_TIME64_SYSCALLS are, they shouldn't be part of how it's
> defined (a definition that says this is what happens in each of five
> cases is not a cleanly defined interface); the definition should
> rather be such that readers can see what the answer would be for each
> of those cases - and for any other cases that may arise in future.
> (I think the current definition is that it means either a specified
> set of suffixed syscalls using 64-bit time are guaranteed to be
> available at runtime, *or* that the corresponding unsuffixed syscalls
> use 64-bit time and are guaranteed to be available, so that #define
> of the suffixed names to the unsuffixed ones is OK in that case.)
> 

The above definition is the concise description of
__ASSUME_TIME64_SYSCALLS.

> There is at least one case you didn't list (or a variant of case 5
> that's different as far as glibc's concerned but not as far as the
> kernel's concerned): new glibc ports for ILP32 ABIs where the oldest
> kernel version supported is older than 5.1, should we wish for any
> such port to support only 64-bit time and not 32-bit time (so
> __TIMESIZE == 64, __WORDSIZE == 64, __SYSCALL_WORDSIZE == 64) - and
                    ^^^^^^^^^^^^^^^^ - shouldn't be WORDSIZE == 32 ?

> once we have the support in glibc to make it possible not to support
> 32-bit time in such a case, it seems a good idea for any such new
> ports to use it.
> 
> __WORDSIZE is the size, in bits, of "long int".  __SYSCALL_WORDSIZE
> (when defined) is the size, in bits, of __syscall_slong_t (the "long
> int" type in the syscall interface, which is the same as userspace
> "long int" except for x32).
> 


Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Tue, 30 Jul 2019, Lukasz Majewski wrote:

> > There is at least one case you didn't list (or a variant of case 5
> > that's different as far as glibc's concerned but not as far as the
> > kernel's concerned): new glibc ports for ILP32 ABIs where the oldest
> > kernel version supported is older than 5.1, should we wish for any
> > such port to support only 64-bit time and not 32-bit time (so
> > __TIMESIZE == 64, __WORDSIZE == 64, __SYSCALL_WORDSIZE == 64) - and
>                     ^^^^^^^^^^^^^^^^ - shouldn't be WORDSIZE == 32 ?

Sorry, yes - __WORDSIZE == 32 and __SYSCALL_WORDSIZE == 32.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 9278 bytes --]

Hi Zack,

> On Fri, Jul 26, 2019 at 6:39 AM Lukasz Majewski <lukma@denx.de> wrote:
> ...
> > > > See for example [1] - there are just 7 lines of "code".  But
> > > > Joseph does not accept our patches.  The arguments he gives are
> > > > not on a technical level;  
> ...
> > Our goal is to add a solid foundation for the Y2038 work, so we
> > would know the direction where we are heading.  
> ...
> > If you think that it would be better and most of all faster if you
> > rewrite the description, then I don't mind.
> >
> > It would be great if you could do it sooner than latter as this
> > slows down our development.  
> ...
> > The most recent version of this patch set (v8):
> > https://patchwork.ozlabs.org/patch/1117100/  
> 
> I haven’t been following the details of these patches super carefully,
> and I’m not sure I understand what _Joseph’s_ concerns with your
> writing is.  However, I’m a native English speaker, I’ve read over the
> text in the patch at <https://patchwork.ozlabs.org/patch/1117100/>, I
> do think I understand the issues at a high level, and I do think the
> meaning of __ASSUME_TIME64_SYSCALLS could be explained more clearly.
> I’m prepared to work with you to come up with better wording 

Thanks for offering your help.

> but I
> need to ask you a bunch of questions.  Could you please reply to each
> of the queries marked Qn below?
> 
> As I understand it, we have five distinct cases to consider:
> 
> 1. All future LP64 ABIs and all but one existing LP64 ABI, identified
>    by __WORDSIZE == 64: time_t is already a 64-bit integer type and
>    all of the relevant system calls already accept it in that form.
>    glibc’s implementation of, for instance, clock_gettime may continue
>    to invoke the system call numbered __NR_clock_gettime.

This is exactly how we shall proceed with machines having
__WORDSIZE==64 (e.g. x86_64, armv8, etc).

They now support 64 bit time with non suffixed syscalls.

In the patch [1] the __WORDSIZE == 64 check covers this.

> 
> 2. The exception to (1) is Alpha.  That is an exclusively LP64
>    architecture, but in glibc 2.0 it used 32-bit time_t, and we still
>    have compatibility code for that case.  The compatibility symbols
>    invoke a set of compatibility syscalls with ‘osf’ in their names:
>    for instance, gettimeofday@GLIBC_2.0 invokes __NR_osf_gettimeofday.
>    Not all of the time-related functions in glibc have compatibility
>    symbols, only those that existed in version 2.0.
> 
>    Your patches do not touch this compatibility code at all, as far as
>    I can see. 

Yes, you are correct. I was not even aware of such a case (as I found
Alpha as 64 bit arch when I did my checking).

> Alpha being out of production, and binaries compiled
>    against glibc 2.0 being rare anyway, it would only make sense to
>    involve this code in your patches if it reduced the overall volume
>    of compatibility code somehow, but regardless we need to make sure
>    it doesn't break.

As you mentioned - we shall not break existing binaries. However, I'm
not sure if we shall spent more time/effort on the arch being near EOL
(or at least being out of production now).

> 
> 3. x32 (recently new 32-bit ABI for x86): like case 1, time_t is
>    already 64-bit and we use unsuffixed system calls.  The text says
>    this case is identified by __WORDSIZE == 32 && __TIMESIZE == 64,
>    but the code actually checks __SYSCALL_WORDSIZE.
> 
>    Q1: Which condition correctly identifies this case, __TIMESIZE ==
> 64 or __SYSCALL_WORDSIZE == 64?

It is:

(__WORDSIZE == 32) && ((defined __SYSCALL_WORDSIZE &&__SYSCALL_WORDSIZE
== 64))

Only x32 defines the __SYSCALL_WORDSIZE == 64 (as it has __WORDSIZE ==
32, but supports 64 bit syscalls ABI).

> 
>    Q2: Could we perhaps ensure that __TIMESIZE and/or
> __SYSCALL_WORDSIZE is defined to 64 whenever __WORDSIZE == 64? Then
> we could collapse this into case 1.

__TIMESIZE == 64 for x32. 

The x32 uses the same set of syscalls (e.g. clock_gettime) as in point 1
(as for example x86_64).

> 
> 4. Brand-new (added in kernel 5.1 or later) 32-bit ABIs: time_t will
>    always be 64-bit,

This would be true after we make the "switch" to support Y2038 aware
systems. Please find example implementation [2] from this patch series
[3] (it adds example code for converting __clock_settime to support 64
bit time when __ASSUME_TIME64_SYSCALLS is defined).

> _but_ glibc’s implementation of time-related APIs
>    may need to invoke system calls with suffixed names: clock_gettime
>    invoking __NR_clock_gettime64, for instance.  Also, the kernel will
>    not provide all of the time-related system calls that have
>    historically existed; glibc must, for instance, implement
>    gettimeofday in terms of clock_gettime.

Yes, correct. Some syscalls would be emulated (as they are not or will
not be converted to 64 bit version).

> 
>    Q3: What macros are defined for this case?

There are no macros yet available.

> 
>    Q4: Does glibc need to call system calls with suffixed names in
>    this case?

I think yes - for example the gettimeofday would internally use
clock_gettime64 (vDSO if available).

> 
>    Q4a: If the answer to Q4 is ‘yes’, why is that, and can we change
>    the kernel so that it’s the same as x32 and the LP64 architectures?

We need new set of syscalls for 64 bit time support on 32 bit archs
(WORDSIZE==32); for example x32/LP64 would still use clock_settime
syscall (number X). To have the same functionality (64 bit time
support) 32 bit archs would need to use clock_settime64 (number 404 on
armv7)

And here the __ASSUME_TIME64_SYSCALLS comes into play. If the arch is
capable of providing 64 bit time, (no matter if it uses clock_settime
or clock_settime64), then __ASSUME_TIME64_SYSCALLS is defined.

It is also assumed that both clock_settime64 and clock_settime provide
the same ABI, so no code needs to be adjusted in glibc.

If code needs to be adjusted (as the calls are not compatible) - a new
flag will be introduced (like with semtimedop)

>    (Either by removing the suffixes, or by _adding_ suffixed aliases
>    to asm/unistd.h for x32 and LP64 architectures.)

Wouldn't this caused the ABI break?

> 
> 5. Historical 32-bit ABIs, where the existing set of system calls
>    takes 32-bit time_t, and Linux 5.1 added a matching set that takes
>    64-bit time_t.  For compatibility with old programs that make
>    direct system calls, the kernel will not rename the __NR_ constants
>    for the old (32-bit) system calls; instead new constants with ‘64’
>    or ‘time64’ suffixes will be added.  As in case 4, the new set of
>    system calls does not cover all of the historic time-related system
>    calls.
> 
>    In this case, and only this case, glibc’s code needs to account for
>    the possibility that the new __NR_ constants are not defined
>    (because glibc is being compiled against kernel headers from a
>    version older than 5.1), or that the new system calls are not
>    available at runtime (glibc was compiled against new kernel headers
>    but is running with an old kernel).
> 
>    The #if is complicated enough that I’m not sure, but I _think_ your
>    code only defines __ASSUME_TIME64_SYSCALLS when the new constants
>    are _guaranteed_ to be defined.
> 
>    Q5: Is it correct that __ASSUME_TIME64_SYSCALLS is only defined
>    when the new constants are guaranteed to be defined?

No.

The __ASSUME_TIME64_SYSCALLS is defined only when the architecture
supports 64 bit time related ABI.

(either via clock_settime on e.g. x86_64/arm64 or clock_settime64 on
arm).

Please consult the code for clock_settime [4]. It shows how the
__ASSUME_TIME64_SYSCALLS flag is used in practice.

> 
>    Q6: All of the other __ASSUME_ constants mean both that we assume
>    the kernel headers are new enough to provide all the necessary
>    declarations, and that we assume the feature is available at
>    runtime: no fallback code will be included in the library.  Is this
>    also the intended meaning of __ASSUME_TIME64_SYSCALLS?

The patch [1] defines the __ASSUME_TIME64_SYSCALLS as the ability of
the architecture (via kernel syscalls) to provide 64 bit time support.

As shown in [4] - the fallback code is called when
__ASSUME_TIME64_SYSCALLS is NOT defined and if architecture doesn't
support clock_settime64.

> 
> zw


Note:

[1] - https://patchwork.ozlabs.org/patch/1117100/ 

[2] -
https://github.com/lmajewski/y2038_glibc/commit/3d5f3512438de7426acba58c1edf53f756f8570b#diff-c051022b496f12bd9028edb46b8ec04d

[3] -
https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-__clock-internal-struct-timespec-v6

[4] -
https://github.com/lmajewski/y2038_glibc/commit/69f842a8519ca13ed11fab0ff1bcc6fa1a524192



Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Accelerating Y2038 glibc fixes

[Joseph Myers]

On Mon, 29 Jul 2019, Zack Weinberg wrote:

>    Q5: Is it correct that __ASSUME_TIME64_SYSCALLS is only defined
>    when the new constants are guaranteed to be defined?

Either the new constants, *or* constants such as __NR_clock_settime if 
those syscalls use 64-bit time with the same semantics as the suffixed 
syscalls.

Note that if Florian's changes to include syscall lists in glibc go in 
then we could assume the constants for the new syscalls are defined in all 
cases where the old syscalls don't use 64-bit time.  That might slightly 
simplify some of the code, by eliminating the case "__TIMESIZE == 32, 
constants for syscalls with 64-bit time are not defined".

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: Accelerating Y2038 glibc fixes

[Florian Weimer]

* Joseph Myers:

> On Mon, 29 Jul 2019, Zack Weinberg wrote:
>
>>    Q5: Is it correct that __ASSUME_TIME64_SYSCALLS is only defined
>>    when the new constants are guaranteed to be defined?
>
> Either the new constants, *or* constants such as __NR_clock_settime if 
> those syscalls use 64-bit time with the same semantics as the suffixed 
> syscalls.
>
> Note that if Florian's changes to include syscall lists in glibc go in 
> then we could assume the constants for the new syscalls are defined in all 
> cases where the old syscalls don't use 64-bit time.  That might slightly 
> simplify some of the code, by eliminating the case "__TIMESIZE == 32, 
> constants for syscalls with 64-bit time are not defined".

Yes, that was part of the motivation for this proposal.  I will try to
resubmit something that follows your suggestions.

Thanks,
Florian

Re: Accelerating Y2038 glibc fixes

[Lukasz Majewski]

[-- Attachment #1: Type: text/plain, Size: 10060 bytes --]

Hi Zack,

> Hi Zack,
> 
> > On Fri, Jul 26, 2019 at 6:39 AM Lukasz Majewski <lukma@denx.de>
> > wrote: ...  
> > > > > See for example [1] - there are just 7 lines of "code".  But
> > > > > Joseph does not accept our patches.  The arguments he gives
> > > > > are not on a technical level;    
> > ...  
> > > Our goal is to add a solid foundation for the Y2038 work, so we
> > > would know the direction where we are heading.    
> > ...  
> > > If you think that it would be better and most of all faster if you
> > > rewrite the description, then I don't mind.
> > >
> > > It would be great if you could do it sooner than latter as this
> > > slows down our development.    
> > ...  
> > > The most recent version of this patch set (v8):
> > > https://patchwork.ozlabs.org/patch/1117100/    
> > 
> > I haven’t been following the details of these patches super
> > carefully, and I’m not sure I understand what _Joseph’s_ concerns
> > with your writing is.  However, I’m a native English speaker, I’ve
> > read over the text in the patch at
> > <https://patchwork.ozlabs.org/patch/1117100/>, I do think I
> > understand the issues at a high level, and I do think the meaning
> > of __ASSUME_TIME64_SYSCALLS could be explained more clearly. I’m
> > prepared to work with you to come up with better wording   
> 
> Thanks for offering your help.

Shall I provide more input to this issue?

> 
> > but I
> > need to ask you a bunch of questions.  Could you please reply to
> > each of the queries marked Qn below?
> > 
> > As I understand it, we have five distinct cases to consider:
> > 
> > 1. All future LP64 ABIs and all but one existing LP64 ABI,
> > identified by __WORDSIZE == 64: time_t is already a 64-bit integer
> > type and all of the relevant system calls already accept it in that
> > form. glibc’s implementation of, for instance, clock_gettime may
> > continue to invoke the system call numbered __NR_clock_gettime.  
> 
> This is exactly how we shall proceed with machines having
> __WORDSIZE==64 (e.g. x86_64, armv8, etc).
> 
> They now support 64 bit time with non suffixed syscalls.
> 
> In the patch [1] the __WORDSIZE == 64 check covers this.
> 
> > 
> > 2. The exception to (1) is Alpha.  That is an exclusively LP64
> >    architecture, but in glibc 2.0 it used 32-bit time_t, and we
> > still have compatibility code for that case.  The compatibility
> > symbols invoke a set of compatibility syscalls with ‘osf’ in their
> > names: for instance, gettimeofday@GLIBC_2.0 invokes
> > __NR_osf_gettimeofday. Not all of the time-related functions in
> > glibc have compatibility symbols, only those that existed in
> > version 2.0.
> > 
> >    Your patches do not touch this compatibility code at all, as far
> > as I can see.   
> 
> Yes, you are correct. I was not even aware of such a case (as I found
> Alpha as 64 bit arch when I did my checking).
> 
> > Alpha being out of production, and binaries compiled
> >    against glibc 2.0 being rare anyway, it would only make sense to
> >    involve this code in your patches if it reduced the overall
> > volume of compatibility code somehow, but regardless we need to
> > make sure it doesn't break.  
> 
> As you mentioned - we shall not break existing binaries. However, I'm
> not sure if we shall spent more time/effort on the arch being near EOL
> (or at least being out of production now).
> 
> > 
> > 3. x32 (recently new 32-bit ABI for x86): like case 1, time_t is
> >    already 64-bit and we use unsuffixed system calls.  The text says
> >    this case is identified by __WORDSIZE == 32 && __TIMESIZE == 64,
> >    but the code actually checks __SYSCALL_WORDSIZE.
> > 
> >    Q1: Which condition correctly identifies this case, __TIMESIZE ==
> > 64 or __SYSCALL_WORDSIZE == 64?  
> 
> It is:
> 
> (__WORDSIZE == 32) && ((defined __SYSCALL_WORDSIZE
> &&__SYSCALL_WORDSIZE == 64))
> 
> Only x32 defines the __SYSCALL_WORDSIZE == 64 (as it has __WORDSIZE ==
> 32, but supports 64 bit syscalls ABI).
> 
> > 
> >    Q2: Could we perhaps ensure that __TIMESIZE and/or
> > __SYSCALL_WORDSIZE is defined to 64 whenever __WORDSIZE == 64? Then
> > we could collapse this into case 1.  
> 
> __TIMESIZE == 64 for x32. 
> 
> The x32 uses the same set of syscalls (e.g. clock_gettime) as in
> point 1 (as for example x86_64).
> 
> > 
> > 4. Brand-new (added in kernel 5.1 or later) 32-bit ABIs: time_t will
> >    always be 64-bit,  
> 
> This would be true after we make the "switch" to support Y2038 aware
> systems. Please find example implementation [2] from this patch series
> [3] (it adds example code for converting __clock_settime to support 64
> bit time when __ASSUME_TIME64_SYSCALLS is defined).
> 
> > _but_ glibc’s implementation of time-related APIs
> >    may need to invoke system calls with suffixed names:
> > clock_gettime invoking __NR_clock_gettime64, for instance.  Also,
> > the kernel will not provide all of the time-related system calls
> > that have historically existed; glibc must, for instance, implement
> >    gettimeofday in terms of clock_gettime.  
> 
> Yes, correct. Some syscalls would be emulated (as they are not or will
> not be converted to 64 bit version).
> 
> > 
> >    Q3: What macros are defined for this case?  
> 
> There are no macros yet available.
> 
> > 
> >    Q4: Does glibc need to call system calls with suffixed names in
> >    this case?  
> 
> I think yes - for example the gettimeofday would internally use
> clock_gettime64 (vDSO if available).
> 
> > 
> >    Q4a: If the answer to Q4 is ‘yes’, why is that, and can we change
> >    the kernel so that it’s the same as x32 and the LP64
> > architectures?  
> 
> We need new set of syscalls for 64 bit time support on 32 bit archs
> (WORDSIZE==32); for example x32/LP64 would still use clock_settime
> syscall (number X). To have the same functionality (64 bit time
> support) 32 bit archs would need to use clock_settime64 (number 404 on
> armv7)
> 
> And here the __ASSUME_TIME64_SYSCALLS comes into play. If the arch is
> capable of providing 64 bit time, (no matter if it uses clock_settime
> or clock_settime64), then __ASSUME_TIME64_SYSCALLS is defined.
> 
> It is also assumed that both clock_settime64 and clock_settime provide
> the same ABI, so no code needs to be adjusted in glibc.
> 
> If code needs to be adjusted (as the calls are not compatible) - a new
> flag will be introduced (like with semtimedop)
> 
> >    (Either by removing the suffixes, or by _adding_ suffixed aliases
> >    to asm/unistd.h for x32 and LP64 architectures.)  
> 
> Wouldn't this caused the ABI break?
> 
> > 
> > 5. Historical 32-bit ABIs, where the existing set of system calls
> >    takes 32-bit time_t, and Linux 5.1 added a matching set that
> > takes 64-bit time_t.  For compatibility with old programs that make
> >    direct system calls, the kernel will not rename the __NR_
> > constants for the old (32-bit) system calls; instead new constants
> > with ‘64’ or ‘time64’ suffixes will be added.  As in case 4, the
> > new set of system calls does not cover all of the historic
> > time-related system calls.
> > 
> >    In this case, and only this case, glibc’s code needs to account
> > for the possibility that the new __NR_ constants are not defined
> >    (because glibc is being compiled against kernel headers from a
> >    version older than 5.1), or that the new system calls are not
> >    available at runtime (glibc was compiled against new kernel
> > headers but is running with an old kernel).
> > 
> >    The #if is complicated enough that I’m not sure, but I _think_
> > your code only defines __ASSUME_TIME64_SYSCALLS when the new
> > constants are _guaranteed_ to be defined.
> > 
> >    Q5: Is it correct that __ASSUME_TIME64_SYSCALLS is only defined
> >    when the new constants are guaranteed to be defined?  
> 
> No.
> 
> The __ASSUME_TIME64_SYSCALLS is defined only when the architecture
> supports 64 bit time related ABI.
> 
> (either via clock_settime on e.g. x86_64/arm64 or clock_settime64 on
> arm).
> 
> Please consult the code for clock_settime [4]. It shows how the
> __ASSUME_TIME64_SYSCALLS flag is used in practice.
> 
> > 
> >    Q6: All of the other __ASSUME_ constants mean both that we assume
> >    the kernel headers are new enough to provide all the necessary
> >    declarations, and that we assume the feature is available at
> >    runtime: no fallback code will be included in the library.  Is
> > this also the intended meaning of __ASSUME_TIME64_SYSCALLS?  
> 
> The patch [1] defines the __ASSUME_TIME64_SYSCALLS as the ability of
> the architecture (via kernel syscalls) to provide 64 bit time support.
> 
> As shown in [4] - the fallback code is called when
> __ASSUME_TIME64_SYSCALLS is NOT defined and if architecture doesn't
> support clock_settime64.
> 
> > 
> > zw  
> 
> 
> Note:
> 
> [1] - https://patchwork.ozlabs.org/patch/1117100/ 
> 
> [2] -
> https://github.com/lmajewski/y2038_glibc/commit/3d5f3512438de7426acba58c1edf53f756f8570b#diff-c051022b496f12bd9028edb46b8ec04d
> 
> [3] -
> https://github.com/lmajewski/y2038_glibc/commits/Y2038-2.29-glibc-__clock-internal-struct-timespec-v6
> 
> [4] -
> https://github.com/lmajewski/y2038_glibc/commit/69f842a8519ca13ed11fab0ff1bcc6fa1a524192
> 
> 
> 
> Best regards,
> 
> Lukasz Majewski
> 
> --
> 
> DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
> HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
> Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email:
> lukma@denx.de




Best regards,

Lukasz Majewski

--

DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-59 Fax: (+49)-8142-66989-80 Email: lukma@denx.de

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Fwd: Accelerating Y2038 glibc fixes

[Zack Weinberg]

[forwarding accidental off-list reply]

On Fri, Aug 9, 2019 at 3:26 AM Lukasz Majewski <lukma@denx.de> wrote:
> > > I haven’t been following the details of these patches super
> > > carefully, and I’m not sure I understand what _Joseph’s_ concerns
> > > with your writing is.  However, I’m a native English speaker, I’ve
> > > read over the text in the patch at
> > > <https://patchwork.ozlabs.org/patch/1117100/>, I do think I
> > > understand the issues at a high level, and I do think the meaning
> > > of __ASSUME_TIME64_SYSCALLS could be explained more clearly. I’m
> > > prepared to work with you to come up with better wording
> >
> > Thanks for offering your help.
>
> Shall I provide more input to this issue?

Not yet.  Your answers to the questions I asked were helpful, but they
made it clear that I need to read over your entire patch set before I
can respond.  I did not have time to do that in the past couple weeks
because I've been busy with my actual job.  I hope to do it early next
week.  Thank you for your patience.

zw