Re: [PATCH v4 5/5] unpack-trees: reuse (still valid) cache-tree from src_index

[Duy Nguyen <pclouds@gmail.com>]

On Mon, Aug 13, 2018 at 5:48 PM Elijah Newren <newren@gmail.com> wrote:
>
> On Sun, Aug 12, 2018 at 1:16 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> >
> > We do n-way merge by walking the source index and n trees at the same
> > time and add merge results to a new temporary index called o->result.
> > The merge result for any given path could be either
> >
> > - keep_entry(): same old index entry in o->src_index is reused
> > - merged_entry(): either a new entry is added, or an existing one updated
> > - deleted_entry(): one entry from o->src_index is removed
> >
> > For some reason [1] we keep making sure that the source index's
> > cache-tree is still valid if used by o->result: for all those
> > merged/deleted entries, we invalidate the same path in o->src_index,
> > so only cache-trees covering the "keep_entry" parts remain good.
> >
> > Because of this, the cache-tree from o->src_index can be perfectly
> > reused in o->result. And in fact we already rely on this logic to
> > reuse untracked cache in edf3b90553 (unpack-trees: preserve index
> > extensions - 2017-05-08). Move the cache-tree to o->result before
> > doing cache_tree_update() to reduce hashing cost.
> >
> > Since cache_tree_update() has risen up as one of the most expensive
> > parts in unpack_trees() after the last few patches. This does help
> > reduce unpack_trees() time significantly (on webkit.git):
> >
> >     before       after
> >   --------------------------------------------------------------------
> >     0.080394752  0.051258167 s:  read cache .git/index
> >     0.216010838  0.212106298 s:  preload index
> >     0.008534301  0.280521764 s:  refresh index
> >     0.251992198  0.218160442 s:   traverse_trees
> >     0.377031383  0.374948191 s:   check_updates
> >     0.372768105  0.037040114 s:   cache_tree_update
> >     1.045887251  0.672031609 s:  unpack_trees
>
> Cool, nice drop in both cache_tree_update() and unpack_trees().  But
> why did refresh_index() go up so much?  That should have been
> unaffected by this patch to, so it seems like something odd is going
> on.  Any ideas?

Probably fs cache and stuff. This is a laptop with just 4GB RAM and a
very slow disk so if something triggers in the background and evicts
some webkit.git's stat info, refresh_index will get hot fast (and with
275k files, webkit.git needs quite a bit of ram to make sure stat()
calls don't hit the disk).
-- 
Duy