Re: [RFC PATCH] http: use xmalloc with cURL

[Jeff King <peff@peff.net>]

On Sun, Aug 11, 2019 at 12:08:53PM -0700, Carlo Arenas wrote:

> there is also the risk that xmalloc might not be sufficiently thread
> safe (ex: when it triggers unuse_one_window() through the use of a
> try_to_free_routine in packfile.c but we could mitigate the risk for
> now by doing it only first #ifdef NED (I promise to take it out before
> it even gets to next, assuming Junio would trust me enough after the
> last blow up to even get it into pu)

Yes, our xmalloc() is very much not thread-safe, and I suspect
multithreaded bits that don't override try_to_free_routine, like
git-grep, are probably buggy right now (assuming they xmalloc() outside
of the big global lock). This has been the case for ages (see
[1] from 2011, for example) and I don't think anybody has ever
complained. Which leads me to believe that try_to_free_routine() rarely
actually kicks in.

I've long suspected that the whole try_to_free_pack_memory thing is more
trouble than it's worth these days. We're dealing with mmap'd files, so
the OS is going to do a much better job at responding to memory pressure
by dropping individual pages.

So the only thing we're really freeing is address space. On 64-bit
systems, we have plenty of that to go around. On 32-bit systems, it
might. But we introduced it[2] at the same time as sliding windows[3],
so it's not clear how much it helps (keep in mind that even without this
we'd still free old pack windows to make room for more mmap'd packs).

And as a cost, xmalloc() isn't thread safe, and is a lot harder to
reason about (e.g., something simple like strbuf_grow() might trigger
quite a bit more work than you'd expect).

I think it might be worth just eliminating the whole idea.

[1] https://public-inbox.org/git/20110831015936.GB2519@sigill.intra.peff.net/
[2] 97bfeb34df (Release pack windows before reporting out of memory., 2006-12-24)
[3] 60bb8b1453 (Fully activate the sliding window pack access., 2006-12-23)

-Peff