Re: fts in gnulib behave different than glibc

[Simon Josefsson via Gnulib discussion list <bug-gnulib@gnu.org>]

[-- Attachment #1: Type: text/plain, Size: 3031 bytes --]

Jim Meyering <jim@meyering.net> writes:

> On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion
> list <bug-gnulib@gnu.org> wrote:
>> Hi.  I replaced GNU InetUtils' old custom fts implementation with the
>> one from gnulib, but self-tests started failing.  Looking at the code,
>> it seems gnulib's fts implementation has diverged compared to glibc, and
>> has some optimizations that (I think) change the API (wrt stat and
>> chdir).  Also, gnulib's fts module is always enabled, even on modern
>> glibc systems.  InetUtils's usage of fts works fine with modern glibc,
>> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR).  The
>> gnulib manual for the fts replacement module isn't terribly clear about
>> this.  Is there a reason for this behaviour?
>>
>> I would prefer if there were two fts modules in gnulib:
>>
>> 1) One module 'fts' based on glibc's code, that is only enabled in on
>> systems that doesn't have fts, or where fts is known to be buggy.
>>
>> 2) One 'fts-faster' that is the current code, which can be used when you
>> always wants to pull in the optimized implementation.
>>
>> Then InetUtils would use system fts on glibc platforms, and not always
>> have to pull in a replacement copy.
>>
>> What do you think?
>>
>> I could live with a new module 'fts-optional' that only pulls in the
>> current 'fts' module when the system is lacking it.  That doesn't fix
>> the API confusion, but is probably sufficient for InetUtils.
>
> There are fundamental flaws in the ABI of glibc's fts that make it
> unsuitable for use in any tool I care about.

Ouch -- is there disagreement from the glibc people on fixing glibc fts?
Maybe the reaction will be different now.

Are the problems inherent with the glibc ABI, or can they be fixed?  If
it isn't possible to fix, maybe the entire API should be declared
deprecated and eventually removed.

The glibc manual doesn't seem to document fts?!

> Those flaws make it easy to hit silly limits or to provoke inordinate
> resource usage/DoS.

It would be nice to document these problems in more detail in the gnulib
manual.

Is there any known behavioural difference between glibc fts and gnulib
fts?  Documenting that too would be useful.

InetUtils' required a FTS_NOCHDIR flag in order to continue behave as
before (a simple command like 'ls foo' where foo is a directory failed).
I don't see any self-tests in gnulib without that flag, so maybe this
suggests there is some API/ABI difference.

> Is it ok for InetUtil's fts to be unable to do these things? (each of
> which afflicts glibc fts, from what I recall)
> - process files in a tree more than 2^16 levels deep
> - detect certain cycles efficiently
> - delete (in reasonable time) a hierarchy with too many entries in a
> single directory.

InetUtils only uses fts for "DIR" in ftpd, when it emulate /bin/ls
internally (based on some old BSD implementation of /bin/ls that uses
fts).  The first two applies, but not the third, I think, but it sounds
like corner-cases.

/Simon

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 255 bytes --]