From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: AS22989 209.51.188.0/24 X-Spam-Status: No, score=-3.9 required=3.0 tests=AWL,BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_PASS,SPF_PASS shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dcvr.yhbt.net (Postfix) with ESMTPS id D02FB1F8C6 for ; Thu, 29 Jul 2021 18:11:00 +0000 (UTC) Received: from localhost ([::1]:50906 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m9AUl-0005S7-KV for normalperson@yhbt.net; Thu, 29 Jul 2021 14:10:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:55602) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m9AUi-0005Rm-FT for bug-gnulib@gnu.org; Thu, 29 Jul 2021 14:10:56 -0400 Received: from mail-wm1-f48.google.com ([209.85.128.48]:52080) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1m9AUg-00012k-KQ for bug-gnulib@gnu.org; Thu, 29 Jul 2021 14:10:56 -0400 Received: by mail-wm1-f48.google.com with SMTP id u15so4296734wmj.1 for ; Thu, 29 Jul 2021 11:10:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=926ASoTCeoBvoDekIGxbwoh3JR06GDvlqbyvZMSDm5U=; b=Qymt2cg5+ZDycrWiV4MVM2aP1hxnIVtzWqiCyJRHtzA0IbwwQaKs40HzolHlhULB6F rop8vYdchPlfx+tPjpYeuYENRm4bgK39Y637mpyKAY/XIxFV0Eva+tRkogWdk/ownTbD 7QzdJtCupH2Zm6QQh2ZvV8YQ8dlpsS7WklOXCzMHBA+B2J1vFZlrYIXN7vfyst+lwD09 GsLIrvVb68zm5lkdI6SYwojtyNPsxMr7kjiGZE+iAUGE4bOBZatzjJoB5gNWQtdoH2kD 1X/sFgtnKGvvu16QM28KB3KPAnWIOux52SpOQctcMKR2IOzmHwwYW5e/78hUduFljpte pXbA== X-Gm-Message-State: AOAM531zX7mjhJGbXt3KKD1Cf2I/zr3IdL5tTWNrUTEsZVJvKYXwTl4t i4aN8IrHSMZgndLsA569hbYW7rUQOngtNjYI1Ho= X-Google-Smtp-Source: ABdhPJxGplBPxN/IHpq1aJbWeCeFKc/aFPxX1LmVuIg88D0G7nA4mM4VBPiMj6Pflxh7ccwvGoe+4RUYMqVROAlDkIA= X-Received: by 2002:a05:600c:206:: with SMTP id 6mr16057399wmi.137.1627582252862; Thu, 29 Jul 2021 11:10:52 -0700 (PDT) MIME-Version: 1.0 References: <87r1fig8sj.fsf@latte.josefsson.org> <87lf5p4dyr.fsf@latte.josefsson.org> In-Reply-To: <87lf5p4dyr.fsf@latte.josefsson.org> From: Jim Meyering Date: Thu, 29 Jul 2021 11:10:41 -0700 Message-ID: Subject: Re: fts in gnulib behave different than glibc To: Simon Josefsson Content-Type: text/plain; charset="UTF-8" Received-SPF: pass client-ip=209.85.128.48; envelope-from=meyering@gmail.com; helo=mail-wm1-f48.google.com X-Spam_score_int: -13 X-Spam_score: -1.4 X-Spam_bar: - X-Spam_report: (-1.4 / 5.0 requ) BAYES_00=-1.9, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.248, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: bug-gnulib@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Gnulib discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "bug-gnulib@gnu.org List" Errors-To: bug-gnulib-bounces+normalperson=yhbt.net@gnu.org Sender: "bug-gnulib" On Thu, Jul 29, 2021 at 3:22 AM Simon Josefsson wrote: > > Jim Meyering writes: > > > On Wed, Jul 28, 2021 at 1:08 AM Simon Josefsson via Gnulib discussion > > list wrote: > >> Hi. I replaced GNU InetUtils' old custom fts implementation with the > >> one from gnulib, but self-tests started failing. Looking at the code, > >> it seems gnulib's fts implementation has diverged compared to glibc, and > >> has some optimizations that (I think) change the API (wrt stat and > >> chdir). Also, gnulib's fts module is always enabled, even on modern > >> glibc systems. InetUtils's usage of fts works fine with modern glibc, > >> but it didn't work with gnulib's version (it needed a FTS_NOCHDIR). The > >> gnulib manual for the fts replacement module isn't terribly clear about > >> this. Is there a reason for this behaviour? > >> > >> I would prefer if there were two fts modules in gnulib: > >> > >> 1) One module 'fts' based on glibc's code, that is only enabled in on > >> systems that doesn't have fts, or where fts is known to be buggy. > >> > >> 2) One 'fts-faster' that is the current code, which can be used when you > >> always wants to pull in the optimized implementation. > >> > >> Then InetUtils would use system fts on glibc platforms, and not always > >> have to pull in a replacement copy. > >> > >> What do you think? > >> > >> I could live with a new module 'fts-optional' that only pulls in the > >> current 'fts' module when the system is lacking it. That doesn't fix > >> the API confusion, but is probably sufficient for InetUtils. > > > > There are fundamental flaws in the ABI of glibc's fts that make it > > unsuitable for use in any tool I care about. > > Ouch -- is there disagreement from the glibc people on fixing glibc fts? > Maybe the reaction will be different now. It's intrinsic in the current ABI. There would have to be a new interface. I chatted with Uli about this many years ago, and he would have been happy to receive a patch adding the new interfaces, but gnulib's fts solved all of my problems, so I never made time to add this to glibc. > Are the problems inherent with the glibc ABI, or can they be fixed? If > it isn't possible to fix, maybe the entire API should be declared > deprecated and eventually removed. > > The glibc manual doesn't seem to document fts?! > > > Those flaws make it easy to hit silly limits or to provoke inordinate > > resource usage/DoS. > > It would be nice to document these problems in more detail in the gnulib > manual. > > Is there any known behavioural difference between glibc fts and gnulib > fts? Documenting that too would be useful. Yes, see each item on the list quoted below. I think I can dig up some documentation on gnulib's fts. > InetUtils' required a FTS_NOCHDIR flag in order to continue behave as > before (a simple command like 'ls foo' where foo is a directory failed). > I don't see any self-tests in gnulib without that flag, so maybe this > suggests there is some API/ABI difference. > > > Is it ok for InetUtil's fts to be unable to do these things? (each of > > which afflicts glibc fts, from what I recall) > > - process files in a tree more than 2^16 levels deep > > - detect certain cycles efficiently > > - delete (in reasonable time) a hierarchy with too many entries in a > > single directory. > > InetUtils only uses fts for "DIR" in ftpd, when it emulate /bin/ls > internally (based on some old BSD implementation of /bin/ls that uses > fts). The first two applies, but not the third, I think, but it sounds > like corner-cases. They are corner cases, indeed. Correcting that last item: s/delete/traverse/: - traverse (in reasonable time) a hierarchy with too many entries in a single directory.