Re: git fast-import crashing on big imports

["Ulrich Spörlein" <uqs@freebsd.org>]

2017-01-18 22:51 GMT+01:00 Jeff King <peff@peff.net>:
>
> On Wed, Jan 18, 2017 at 03:27:04PM -0500, Jeff King wrote:
>
> > On Wed, Jan 18, 2017 at 09:20:07PM +0100, Ulrich Spörlein wrote:
> >
> > > I found your commit via bisect in case you were wondering. Thanks for
> > > picking this up.
> >
> > Still downloading. However, just looking at the code, the obvious
> > culprit would be clear_delta_base_cache(), which is called from
> > literally nowhere except fast-import, and then only when checkpointing.

Sorry for the bad URL, pesky last minute changes ...

> Hmm. I haven't reproduced your exact issue, but I was able to produce
> some hijinks in that function.
>
> The problem is that the hashmap_iter interface is unreliable if entries
> are added or removed from the map during the iteration.
>
> I suspect the patch below may fix things for you. It works around it by
> walking over the lru list (either is fine, as they both contain all
> entries, and since we're clearing everything, we don't care about the
> order).

Confirmed. With the patch applied, I can import the whole 55G in one go
without any crashes or aborts. Thanks much!

>
> ---
>  sha1_file.c | 9 ++++-----
>  1 file changed, 4 insertions(+), 5 deletions(-)
>
> diff --git a/sha1_file.c b/sha1_file.c
> index 1eb47f611..d20714d6b 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -2342,11 +2342,10 @@ static inline void release_delta_base_cache(struct delta_base_cache_entry *ent)
>
>  void clear_delta_base_cache(void)
>  {
> -       struct hashmap_iter iter;
> -       struct delta_base_cache_entry *entry;
> -       for (entry = hashmap_iter_first(&delta_base_cache, &iter);
> -            entry;
> -            entry = hashmap_iter_next(&iter)) {
> +       struct list_head *lru, *tmp;
> +       list_for_each_safe(lru, tmp, &delta_base_cache_lru) {
> +               struct delta_base_cache_entry *entry =
> +                       list_entry(lru, struct delta_base_cache_entry, lru);
>                 release_delta_base_cache(entry);
>         }
>  }
> --
> 2.11.0.698.gd6b48ab4c
>
>
>
>
> >
> > -Peff