[PATCH] Deprecate 32-bit off_t support

[PATCH] Deprecate 32-bit off_t support

[Florian Weimer]

diff --git a/NEWS b/NEWS
index cc20102fda..2f601c6217 100644
--- a/NEWS
+++ b/NEWS
@@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
   as all functions that call vscanf, vfscanf, or vsscanf are annotated with
   __attribute__ ((format (scanf, ...))).
 
+* A future release of glibc will use a 64-bit off_t type on all
+  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
+  32-bit architectures).  Building new applications with
+  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
+  the 64-bit function aliases (such as fstat64) will remain available under
+  the appropriate feature test macros.  In preparation, libraries should
+  stop using off_t in public header files, and use off64_t (or a fixed-width
+  type such as int64_t or uint64_t) instead.
+
 Changes to build and runtime requirements:
 
 * Python 3.4 or later is required to build the GNU C Library.

Re: [PATCH] Deprecate 32-bit off_t support

[Siddhesh Poyarekar]

On 04/01/19 6:09 PM, Florian Weimer wrote:
> diff --git a/NEWS b/NEWS
> index cc20102fda..2f601c6217 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>     as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>     __attribute__ ((format (scanf, ...))).
>   
> +* A future release of glibc will use a 64-bit off_t type on all

Wouldn't it be clearer to fix the 'future release' to a specific 
timeline, say, 2.31 (to give a year+ to change offending code) so that 
this doesn't sound like a vague threat and there's actually a time bound 
for people to work with?

> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
> +  32-bit architectures).  Building new applications with
> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
> +  the 64-bit function aliases (such as fstat64) will remain available under
> +  the appropriate feature test macros.  In preparation, libraries should
> +  stop using off_t in public header files, and use off64_t (or a fixed-width
> +  type such as int64_t or uint64_t) instead.
> +
>   Changes to build and runtime requirements:
>   
>   * Python 3.4 or later is required to build the GNU C Library.
> 

Re: [PATCH] Deprecate 32-bit off_t support

[Siddhesh Poyarekar]

On 04/01/19 6:15 PM, Siddhesh Poyarekar wrote:
> On 04/01/19 6:09 PM, Florian Weimer wrote:
>> diff --git a/NEWS b/NEWS
>> index cc20102fda..2f601c6217 100644
>> --- a/NEWS
>> +++ b/NEWS
>> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes 
>> affecting compatibility:
>>     as all functions that call vscanf, vfscanf, or vsscanf are 
>> annotated with
>>     __attribute__ ((format (scanf, ...))).
>> +* A future release of glibc will use a 64-bit off_t type on all
> 
> Wouldn't it be clearer to fix the 'future release' to a specific 
> timeline, say, 2.31 (to give a year+ to change offending code) so that 
> this doesn't sound like a vague threat and there's actually a time bound 
> for people to work with?

Oh, and a warning under #if _FILE_OFFSET_BITS == 32 that warns about the 
deprecation?

Siddhesh

Re: [PATCH] Deprecate 32-bit off_t support

[Florian Weimer]

* Siddhesh Poyarekar:

> On 04/01/19 6:15 PM, Siddhesh Poyarekar wrote:
>> On 04/01/19 6:09 PM, Florian Weimer wrote:
>>> diff --git a/NEWS b/NEWS
>>> index cc20102fda..2f601c6217 100644
>>> --- a/NEWS
>>> +++ b/NEWS
>>> @@ -85,6 +85,15 @@ Deprecated and removed features, and other
>>> changes affecting compatibility:
>>>     as all functions that call vscanf, vfscanf, or vsscanf are
>>> annotated with
>>>     __attribute__ ((format (scanf, ...))).
>>> +* A future release of glibc will use a 64-bit off_t type on all
>>
>> Wouldn't it be clearer to fix the 'future release' to a specific
>> timeline, say, 2.31 (to give a year+ to change offending code) so
>> that this doesn't sound like a vague threat and there's actually a
>> time bound for people to work with?

2.31 sounds okay to me.  2.30 would work for me as well.  What do others
think?

> Oh, and a warning under #if _FILE_OFFSET_BITS == 32 that warns about
> the deprecation?

Due to -Werror, that alone would be rather disruptive.  It's too late to
add this for 2.29, too.

Thanks,
Florian

Re: [PATCH] Deprecate 32-bit off_t support

[Siddhesh Poyarekar]

On 04/01/19 6:28 PM, Florian Weimer wrote:
> Due to -Werror, that alone would be rather disruptive.  It's too late to
> add this for 2.29, too.

Yeah, this would have to be for 2.30.

Siddhesh

Re: [PATCH] Deprecate 32-bit off_t support

[Joseph Myers]

On Fri, 4 Jan 2019, Siddhesh Poyarekar wrote:

> Wouldn't it be clearer to fix the 'future release' to a specific timeline,
> say, 2.31 (to give a year+ to change offending code) so that this doesn't
> sound like a vague threat and there's actually a time bound for people to work
> with?

There should be one or more releases with the default changed but the old 
API still available with -D_FILE_OFFSET_BITS=32, before any release with 
_FILE_OFFSET_BITS=32 no longer available and the old ABIs turned into 
compat symbols.  Giving concrete future release versions for a complicated 
change without patches in existence seems dubious.

Note 1: changing the default for user programs, without changing the mode 
used to build glibc and its tests, is easy.  Mike Frysinger posted patches 
in March 2014, and Paul Eggert did some investigation of the effects on 
the ABIs of libraries in a GNU/Linux distribution (many of which are in 
fact already built with _FILE_OFFSET_BITS=64).  Bug 13047.

Note 2: changing the mode used to build glibc itself, as required for 
eliminating the other mode as a supported API in the headers, is much more 
involved, because you need to avoid the header redirections affecting the 
compilation of all the functions for the 32-bit ABI, and keep all the old 
types and structures available in internal headers for building those 
functions although they'll no longer be accessible with installed headers.

Note 3: changing the default mode as used for building testcases (which 
might or might not be the same as the default used for building glibc 
itself) requires fixing the associated linknamespace issues by adding new 
public __*64 symbol names and using them instead of *64 in the header 
redirections (bug 14106).

Note 4: a change of default for building testcases would need to be 
accompanied by systematically making sure that all relevant interfaces are 
tested in all three cases (_FILE_OFFSET_BITS=32, _FILE_OFFSET_BITS=64 and 
the explicit *64 APIs).  Once _FILE_OFFSET_BITS=32 is no longer supported, 
the corresponding ABIs would still need to be tested (using 
compat_symbol_reference etc.) to make sure they continue to work, using 
the expected structure layouts and other types which would need to move to 
internal headers (and care is needed to verify in some automated way that 
the layouts put in internal headers really do match the existing ones in 
installed headers in all cases).

Note 5: presumably the change would also make the _FILE_OFFSET_BITS=64 C++ 
name mangling the default on 64-bit platforms where the C ABI doesn't 
change (see bug 15766), and, similarly, while both APIs are supported, 
-D_FILE_OFFSET_BITS=32 would result in the old mangling on 64-bit 
platforms (despite off_t still being 64-bit so the macro name / value 
being a bit misleading in that case).

Note 6: much the same would apply with struct stat layout on mips64 (n64 
ABI has stat and stat64 layouts different despite off_t being 64-bit by 
default, so the default struct stat layout would change to what's 
currently stat64 and _FILE_OFFSET_BITS=64 stat).

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Deprecate 32-bit off_t support

[Szabolcs Nagy]

On 04/01/2019 12:39, Florian Weimer wrote:
> diff --git a/NEWS b/NEWS
> index cc20102fda..2f601c6217 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>    as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>    __attribute__ ((format (scanf, ...))).
>  
> +* A future release of glibc will use a 64-bit off_t type on all
> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
> +  32-bit architectures).  Building new applications with
> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
> +  the 64-bit function aliases (such as fstat64) will remain available under
> +  the appropriate feature test macros.  In preparation, libraries should
> +  stop using off_t in public header files, and use off64_t (or a fixed-width

please do not recommend the use of a non-standard type
in place of a standard one (off64_t is not even available
with default feature test macros in posix headers).

instead i'd recommend the consistent use of
-D_FILE_OFFSET_BITS=64 and warn
#if _FILE_OFFSET_BITS==32
where appropriate.

> +  type such as int64_t or uint64_t) instead.
> +
>  Changes to build and runtime requirements:
>  
>  * Python 3.4 or later is required to build the GNU C Library.
> 

Re: [PATCH] Deprecate 32-bit off_t support

[Szabolcs Nagy]

On 04/01/2019 13:15, Szabolcs Nagy wrote:
> On 04/01/2019 12:39, Florian Weimer wrote:
>> diff --git a/NEWS b/NEWS
>> index cc20102fda..2f601c6217 100644
>> --- a/NEWS
>> +++ b/NEWS
>> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>>    as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>>    __attribute__ ((format (scanf, ...))).
>>  
>> +* A future release of glibc will use a 64-bit off_t type on all
>> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
>> +  32-bit architectures).  Building new applications with
>> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
>> +  the 64-bit function aliases (such as fstat64) will remain available under
>> +  the appropriate feature test macros.  In preparation, libraries should
>> +  stop using off_t in public header files, and use off64_t (or a fixed-width
> 
> please do not recommend the use of a non-standard type
> in place of a standard one (off64_t is not even available
> with default feature test macros in posix headers).
> 
> instead i'd recommend the consistent use of
> -D_FILE_OFFSET_BITS=64 and warn
> #if _FILE_OFFSET_BITS==32
> where appropriate.

we could recommend

#ifndef _FILE_OFFSET_BITS
_Static_assert(sizeof(off_t)>4, "32bit off_t is deprecated, on platforms where it is still supported use explicit -D_FILE_OFFSET_BITS=32.");
#endif

or similar when off_t is used in public apis.

then first change the default, then drop support.

>> +  type such as int64_t or uint64_t) instead.
>> +
>>  Changes to build and runtime requirements:
>>  
>>  * Python 3.4 or later is required to build the GNU C Library.
>>
> 

Re: [PATCH] Deprecate 32-bit off_t support

[Florian Weimer]

* Szabolcs Nagy:

> please do not recommend the use of a non-standard type
> in place of a standard one (off64_t is not even available
> with default feature test macros in posix headers).

What do you propose instead?  Mention int64_t and uint64_t only?

What do others think about this aspect?

Thanks,
Florian

Re: [PATCH] Deprecate 32-bit off_t support

[Joseph Myers]

On Mon, 7 Jan 2019, Florian Weimer wrote:

> * Szabolcs Nagy:
> 
> > please do not recommend the use of a non-standard type
> > in place of a standard one (off64_t is not even available
> > with default feature test macros in posix headers).
> 
> What do you propose instead?  Mention int64_t and uint64_t only?

Programs and libraries should use types such as off_t, and arrange to be 
built with _FILE_OFFSET_BITS=64, which would make them unaffected by a 
future change to the default.

-- 
Joseph S. Myers
joseph@codesourcery.com

Re: [PATCH] Deprecate 32-bit off_t support

[Paul Eggert]

On 1/7/19 2:17 PM, Joseph Myers wrote:
>> What do you propose instead?  Mention int64_t and uint64_t only?
> Programs and libraries should use types such as off_t, and arrange to be
> built with _FILE_OFFSET_BITS=64, which would make them unaffected by a
> future change to the default.

+1. GNU applications have been doing this for nearly two decades. It's 
by far the best approach for free software.

I suppose I should start adding "#define _TIME_BITS 64" to GNU 
applications, to future-proof them so that they will take advantage of 
64-bit time_t on 32-bit platforms the instant that glibc starts 
publishing it. Are we definitely going to use this as the feature-test 
macro, or is it still possible that we'll use something else?

Re: [PATCH] Deprecate 32-bit off_t support

[Florian Weimer]

* Joseph Myers:

> On Mon, 7 Jan 2019, Florian Weimer wrote:
>
>> * Szabolcs Nagy:
>> 
>> > please do not recommend the use of a non-standard type
>> > in place of a standard one (off64_t is not even available
>> > with default feature test macros in posix headers).
>> 
>> What do you propose instead?  Mention int64_t and uint64_t only?
>
> Programs and libraries should use types such as off_t, and arrange to be 
> built with _FILE_OFFSET_BITS=64, which would make them unaffected by a 
> future change to the default.

If this concerns a public header file, is there a preprocessor check we
can recommend to add to the header file?  Something that causes an
#error if the ABI would not be correct?

Thanks,
Florian

Re: [PATCH] Deprecate 32-bit off_t support

[Guillem Jover]

Hi!

On Fri, 2019-01-04 at 13:39:59 +0100, Florian Weimer wrote:
> diff --git a/NEWS b/NEWS
> index cc20102fda..2f601c6217 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>    as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>    __attribute__ ((format (scanf, ...))).
>  
> +* A future release of glibc will use a 64-bit off_t type on all
> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
> +  32-bit architectures).  Building new applications with
> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
> +  the 64-bit function aliases (such as fstat64) will remain available under
> +  the appropriate feature test macros.  In preparation, libraries should
> +  stop using off_t in public header files, and use off64_t (or a fixed-width
> +  type such as int64_t or uint64_t) instead.
> +
>  Changes to build and runtime requirements:
>  
>  * Python 3.4 or later is required to build the GNU C Library.

While I'm fully behind further pushing to get software to switch to
build (and work) with LFS (that's the reason I proposed the below lintian
tag some time ago), it feels that something like the above might still be
a bit too aggressive? :/

For example in Debian that would probably imply not being able to
upgrade to such glibc until at least all shared libraries and packages
supporting and implementing plugins listed at

  <https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html>

have been transitioned (with either SONAME bumps, or package renames
with Breaks/Conflicts), or determined to not affect their ABI. Please
notice at the top, the comment about that page only having a truncated
view of the affected packages; in addition there might be other packages
not being currently detected.

Thanks,
Guillem

Re: [PATCH] Deprecate 32-bit off_t support

[Florian Weimer]

* Guillem Jover:

> Hi!
>
> On Fri, 2019-01-04 at 13:39:59 +0100, Florian Weimer wrote:
>> diff --git a/NEWS b/NEWS
>> index cc20102fda..2f601c6217 100644
>> --- a/NEWS
>> +++ b/NEWS
>> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>>    as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>>    __attribute__ ((format (scanf, ...))).
>>  
>> +* A future release of glibc will use a 64-bit off_t type on all
>> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
>> +  32-bit architectures).  Building new applications with
>> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
>> +  the 64-bit function aliases (such as fstat64) will remain available under
>> +  the appropriate feature test macros.  In preparation, libraries should
>> +  stop using off_t in public header files, and use off64_t (or a fixed-width
>> +  type such as int64_t or uint64_t) instead.
>> +
>>  Changes to build and runtime requirements:
>>  
>>  * Python 3.4 or later is required to build the GNU C Library.
>
> While I'm fully behind further pushing to get software to switch to
> build (and work) with LFS (that's the reason I proposed the below lintian
> tag some time ago), it feels that something like the above might still be
> a bit too aggressive? :/
>
> For example in Debian that would probably imply not being able to
> upgrade to such glibc until at least all shared libraries and packages
> supporting and implementing plugins listed at
>
>   <https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html>
>
> have been transitioned (with either SONAME bumps, or package renames
> with Breaks/Conflicts), or determined to not affect their ABI. Please
> notice at the top, the comment about that page only having a truncated
> view of the affected packages; in addition there might be other packages
> not being currently detected.

What's interesting is what is *not* in this list.  As far as I can see,
the non-cross binutils is missing, and so are Berkeley DB and
parts/versions of of LLVM.  What's common among them is that they
install public header files under /usr/include which use off_t in the
API.  So these ABIs already assume that client code is compiled with
-D_FILE_OFFSET_BITS=64.

Sure, some corner cases may break when we change the default, but other
things will magically start working.  I think it is worth a try.

Thanks,
Florian

Re: [PATCH] Deprecate 32-bit off_t support

[Adhemerval Zanella]

On 18/01/2019 08:09, Florian Weimer wrote:
> * Guillem Jover:
> 
>> Hi!
>>
>> On Fri, 2019-01-04 at 13:39:59 +0100, Florian Weimer wrote:
>>> diff --git a/NEWS b/NEWS
>>> index cc20102fda..2f601c6217 100644
>>> --- a/NEWS
>>> +++ b/NEWS
>>> @@ -85,6 +85,15 @@ Deprecated and removed features, and other changes affecting compatibility:
>>>    as all functions that call vscanf, vfscanf, or vsscanf are annotated with
>>>    __attribute__ ((format (scanf, ...))).
>>>  
>>> +* A future release of glibc will use a 64-bit off_t type on all
>>> +  architectures (as currently available with -D_FILE_OFFSET_BITS=64 on
>>> +  32-bit architectures).  Building new applications with
>>> +  -D_FILE_OFFSET_BITS=32 will no longer be supported.  The off64_t type and
>>> +  the 64-bit function aliases (such as fstat64) will remain available under
>>> +  the appropriate feature test macros.  In preparation, libraries should
>>> +  stop using off_t in public header files, and use off64_t (or a fixed-width
>>> +  type such as int64_t or uint64_t) instead.
>>> +
>>>  Changes to build and runtime requirements:
>>>  
>>>  * Python 3.4 or later is required to build the GNU C Library.
>>
>> While I'm fully behind further pushing to get software to switch to
>> build (and work) with LFS (that's the reason I proposed the below lintian
>> tag some time ago), it feels that something like the above might still be
>> a bit too aggressive? :/
>>
>> For example in Debian that would probably imply not being able to
>> upgrade to such glibc until at least all shared libraries and packages
>> supporting and implementing plugins listed at
>>
>>   <https://lintian.debian.org/tags/binary-file-built-without-LFS-support.html>
>>
>> have been transitioned (with either SONAME bumps, or package renames
>> with Breaks/Conflicts), or determined to not affect their ABI. Please
>> notice at the top, the comment about that page only having a truncated
>> view of the affected packages; in addition there might be other packages
>> not being currently detected.
> 
> What's interesting is what is *not* in this list.  As far as I can see,
> the non-cross binutils is missing, and so are Berkeley DB and
> parts/versions of of LLVM.  What's common among them is that they
> install public header files under /usr/include which use off_t in the
> API.  So these ABIs already assume that client code is compiled with
> -D_FILE_OFFSET_BITS=64.
> 
> Sure, some corner cases may break when we change the default, but other
> things will magically start working.  I think it is worth a try.

It is not clear to me if packages are being built without LFS due wrong
configure/build options, lack of support from distro built system, or
explicit disabling in package build (due either lack of testing or
ABI limitation).

Also, it is really questionable to have a system with mixed LFS support 
for libraries specially the one that might export off_t in its API.

My impression is, for system consistency, have all libraries with explicit
LFS support and handle outliers as why the program/libraries is building 
*without* non-LFS support (not the other way around).