fts in gnulib behave different than glibc

fts in gnulib behave different than glibc

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

Hi.  I replaced GNU InetUtils' old custom fts implementation with the
one from gnulib, but self-tests started failing.  Looking at the code,
it seems gnulib's fts implementation has diverged compared to glibc, and
has some optimizations that (I think) change the API (wrt stat and
chdir).  Also, gnulib's fts module is always enabled, even on modern
glibc systems.  InetUtils's usage of fts works fine with modern glibc,
but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
gnulib manual for the fts replacement module isn't terribly clear about
this.  Is there a reason for this behaviour?

I would prefer if there were two fts modules in gnulib:

1) One module 'fts' based on glibc's code, that is only enabled in on
systems that doesn't have fts, or where fts is known to be buggy.

2) One 'fts-faster' that is the current code, which can be used when you
always wants to pull in the optimized implementation.

Then InetUtils would use system fts on glibc platforms, and not always
have to pull in a replacement copy.

What do you think?

I could live with a new module 'fts-optional' that only pulls in the
current 'fts' module when the system is lacking it.  That doesn't fix
the API confusion, but is probably sufficient for InetUtils.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: fts in gnulib behave different than glibc

[Jim Meyering]

On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion
list <bug-gnulib@gnu.org> wrote:
> Hi.  I replaced GNU InetUtils' old custom fts implementation with the
> one from gnulib, but self-tests started failing.  Looking at the code,
> it seems gnulib's fts implementation has diverged compared to glibc, and
> has some optimizations that (I think) change the API (wrt stat and
> chdir).  Also, gnulib's fts module is always enabled, even on modern
> glibc systems.  InetUtils's usage of fts works fine with modern glibc,
> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
> gnulib manual for the fts replacement module isn't terribly clear about
> this.  Is there a reason for this behaviour?
>
> I would prefer if there were two fts modules in gnulib:
>
> 1) One module 'fts' based on glibc's code, that is only enabled in on
> systems that doesn't have fts, or where fts is known to be buggy.
>
> 2) One 'fts-faster' that is the current code, which can be used when you
> always wants to pull in the optimized implementation.
>
> Then InetUtils would use system fts on glibc platforms, and not always
> have to pull in a replacement copy.
>
> What do you think?
>
> I could live with a new module 'fts-optional' that only pulls in the
> current 'fts' module when the system is lacking it.  That doesn't fix
> the API confusion, but is probably sufficient for InetUtils.

There are fundamental flaws in the ABI of glibc's fts that make it
unsuitable for use in any tool I care about.
Those flaws make it easy to hit silly limits or to provoke inordinate
resource usage/DoS.
Is it ok for InetUtil's fts to be unable to do these things? (each of
which afflicts glibc fts, from what I recall)
- process files in a tree more than 2^16 levels deep
- detect certain cycles efficiently
- delete (in reasonable time) a hierarchy with too many entries in a
single directory.

Re: fts in gnulib behave different than glibc

[Simon Josefsson via Gnulib discussion list]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

Jim Meyering <jim@meyering.net> writes:

> On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion
> list <bug-gnulib@gnu.org> wrote:
>> Hi.  I replaced GNU InetUtils' old custom fts implementation with the
>> one from gnulib, but self-tests started failing.  Looking at the code,
>> it seems gnulib's fts implementation has diverged compared to glibc, and
>> has some optimizations that (I think) change the API (wrt stat and
>> chdir).  Also, gnulib's fts module is always enabled, even on modern
>> glibc systems.  InetUtils's usage of fts works fine with modern glibc,
>> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
>> gnulib manual for the fts replacement module isn't terribly clear about
>> this.  Is there a reason for this behaviour?
>>
>> I would prefer if there were two fts modules in gnulib:
>>
>> 1) One module 'fts' based on glibc's code, that is only enabled in on
>> systems that doesn't have fts, or where fts is known to be buggy.
>>
>> 2) One 'fts-faster' that is the current code, which can be used when you
>> always wants to pull in the optimized implementation.
>>
>> Then InetUtils would use system fts on glibc platforms, and not always
>> have to pull in a replacement copy.
>>
>> What do you think?
>>
>> I could live with a new module 'fts-optional' that only pulls in the
>> current 'fts' module when the system is lacking it.  That doesn't fix
>> the API confusion, but is probably sufficient for InetUtils.
>
> There are fundamental flaws in the ABI of glibc's fts that make it
> unsuitable for use in any tool I care about.

Ouch -- is there disagreement from the glibc people on fixing glibc fts?
Maybe the reaction will be different now.

Are the problems inherent with the glibc ABI, or can they be fixed?  If
it isn't possible to fix, maybe the entire API should be declared
deprecated and eventually removed.

The glibc manual doesn't seem to document fts?!

> Those flaws make it easy to hit silly limits or to provoke inordinate
> resource usage/DoS.

It would be nice to document these problems in more detail in the gnulib
manual.

Is there any known behavioural difference between glibc fts and gnulib
fts?  Documenting that too would be useful.

InetUtils' required a FTS_NOCHDIR flag in order to continue behave as
before (a simple command like 'ls foo' where foo is a directory failed).
I don't see any self-tests in gnulib without that flag, so maybe this
suggests there is some API/ABI difference.

> Is it ok for InetUtil's fts to be unable to do these things? (each of
> which afflicts glibc fts, from what I recall)
> - process files in a tree more than 2^16 levels deep
> - detect certain cycles efficiently
> - delete (in reasonable time) a hierarchy with too many entries in a
> single directory.

InetUtils only uses fts for "DIR" in ftpd, when it emulate /bin/ls
internally (based on some old BSD implementation of /bin/ls that uses
fts).  The first two applies, but not the third, I think, but it sounds
like corner-cases.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]

Re: fts in gnulib behave different than glibc

[Jim Meyering]

On Thu, Jul 29, 2021 at 3:22 AM Simon Josefsson <simon@josefsson.org> wrote:
>
> Jim Meyering <jim@meyering.net> writes:
>
> > On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion
> > list <bug-gnulib@gnu.org> wrote:
> >> Hi.  I replaced GNU InetUtils' old custom fts implementation with the
> >> one from gnulib, but self-tests started failing.  Looking at the code,
> >> it seems gnulib's fts implementation has diverged compared to glibc, and
> >> has some optimizations that (I think) change the API (wrt stat and
> >> chdir).  Also, gnulib's fts module is always enabled, even on modern
> >> glibc systems.  InetUtils's usage of fts works fine with modern glibc,
> >> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
> >> gnulib manual for the fts replacement module isn't terribly clear about
> >> this.  Is there a reason for this behaviour?
> >>
> >> I would prefer if there were two fts modules in gnulib:
> >>
> >> 1) One module 'fts' based on glibc's code, that is only enabled in on
> >> systems that doesn't have fts, or where fts is known to be buggy.
> >>
> >> 2) One 'fts-faster' that is the current code, which can be used when you
> >> always wants to pull in the optimized implementation.
> >>
> >> Then InetUtils would use system fts on glibc platforms, and not always
> >> have to pull in a replacement copy.
> >>
> >> What do you think?
> >>
> >> I could live with a new module 'fts-optional' that only pulls in the
> >> current 'fts' module when the system is lacking it.  That doesn't fix
> >> the API confusion, but is probably sufficient for InetUtils.
> >
> > There are fundamental flaws in the ABI of glibc's fts that make it
> > unsuitable for use in any tool I care about.
>
> Ouch -- is there disagreement from the glibc people on fixing glibc fts?
> Maybe the reaction will be different now.

It's intrinsic in the current ABI. There would have to be a new interface.
I chatted with Uli about this many years ago, and he would have been
happy to receive a patch adding the new interfaces, but gnulib's fts
solved all of my problems, so I never made time to add this to glibc.

> Are the problems inherent with the glibc ABI, or can they be fixed?  If
> it isn't possible to fix, maybe the entire API should be declared
> deprecated and eventually removed.
>
> The glibc manual doesn't seem to document fts?!
>
> > Those flaws make it easy to hit silly limits or to provoke inordinate
> > resource usage/DoS.
>
> It would be nice to document these problems in more detail in the gnulib
> manual.
>
> Is there any known behavioural difference between glibc fts and gnulib
> fts?  Documenting that too would be useful.

Yes, see each item on the list quoted below.
I think I can dig up some documentation on gnulib's fts.

> InetUtils' required a FTS_NOCHDIR flag in order to continue behave as
> before (a simple command like 'ls foo' where foo is a directory failed).
> I don't see any self-tests in gnulib without that flag, so maybe this
> suggests there is some API/ABI difference.
>
> > Is it ok for InetUtil's fts to be unable to do these things? (each of
> > which afflicts glibc fts, from what I recall)
> > - process files in a tree more than 2^16 levels deep
> > - detect certain cycles efficiently
> > - delete (in reasonable time) a hierarchy with too many entries in a
> > single directory.
>
> InetUtils only uses fts for "DIR" in ftpd, when it emulate /bin/ls
> internally (based on some old BSD implementation of /bin/ls that uses
> fts).  The first two applies, but not the third, I think, but it sounds
> like corner-cases.

They are corner cases, indeed.

Correcting that last item: s/delete/traverse/:
- traverse (in reasonable time) a hierarchy with too many entries in a
single directory.