git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / code / Atom feed
* [INVESTIGATION] why is fsck --connectivity-only so much more expensive than rev-list --objects --all?
@ 2022-09-20 19:27 John Cai
  2022-09-20 20:41 ` Jeff King
  0 siblings, 1 reply; 10+ messages in thread
From: John Cai @ 2022-09-20 19:27 UTC (permalink / raw)
  To: git; +Cc: Christian Couder

Hi everyone,

I would love your thoughts about something I'm noticing in git fsck
--connectivity-only. I noticed that for large repositories, this operation can
take up alot of memory.

Here at GitLab we are exploring using `git rev-list --all --objects` as a proxy
for checking connectivity. In doing so, I noticed `git fsck --connectivity-only`
is much more expensive than `git rev-list --all --objects`.

Using Valgrind, here are some memory profile results from running both on the
linux repo:

`git fsck --connectivity-only`:

    GB
1.916^                                                                       :
     |                                                                @@:@@:#:
     |                                                         :::@@:@@@:@@:#:
     |                                                  :::@::@:::@@:@@@:@@:#:
     |                                           :::@@@@:: @::@:::@@:@@@:@@:#:
     |                                   @@::::::: :@ @@:: @::@:::@@:@@@:@@:#:
     |                            :::::::@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     |                     ::::@@::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     |              ::::::@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     |        :::::::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     |   ::::::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | ::::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
     | : ::: ::::: ::::: :@::::@ ::: :: :@@: ::: : :@ @@:: @::@:::@@:@@@:@@:#:
   0 +----------------------------------------------------------------------->Gi
     0                                                                   233.7


`git rev-list --all --objects`:

    MB
979.8^                                                                   #
     |                                                                   #
     |                                                                   #
     |                                                                   #::::
     |                                                                   #::::
     |                                                                   #::::
     |                                                             @@@:@@#::::
     |                                                    ::@@@:@@@@@@:@@#::::
     |                                           ::::@::::::@@@:@@@@@@:@@#::::
     |                             @@   :::::::::: ::@::::::@@@:@@@@@@:@@#::::
     |                             @ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |                          @@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |                ::::@::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |          @@::::::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |          @ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |          @ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |        :@@ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |       ::@@ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |    :::::@@ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
     |  :::: ::@@ ::: ::: @::::@@@:@ :@:::::: :::: ::@::::::@@@:@@@@@@:@@#::::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   245.5

As we can see, `git fsck` uses about twice the amount of memory as `git
rev-list`, which was a bit surprising to me.

Digging in a bit, here is the stack trace analysis:

`git fsck --connectivity-only`:

98.37% (2,008,484,573B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->84.04% (1,715,907,794B) 0x35779F: do_xmalloc (wrapper.c:51)
| ->50.73% (1,035,816,788B) 0x35781F: do_xmallocz.part.0 (wrapper.c:85)
| | ->40.48% (826,445,820B) 0x2B2447: unpack_compressed_entry (packfile.c:1601)
| | | ->40.48% (826,445,820B) 0x2B46AB: unpack_entry (packfile.c:1768)
| | | | ->40.48% (826,445,820B) 0x2B4A23: cache_or_unpack_entry (packfile.c:1438)
| | | |   ->40.48% (826,445,820B) 0x2B4A23: packed_object_info (packfile.c:1516)
| | | |     ->40.48% (826,445,820B) 0x29EA83: do_oid_object_info_extended (object-file.c:1620)
| | | |       ->40.48% (826,445,820B) 0x29EBDB: oid_object_info_extended (object-file.c:1639)
| | | |         ->40.48% (826,445,820B) 0x29EDC3: read_object (object-file.c:1671)
| | | |           ->40.48% (826,445,820B) 0x29EE5B: read_object_file_extended (object-file.c:1714)
| | | |             ->40.12% (819,056,147B) 0x2143BB: repo_read_object_file (object-store.h:253)
| | | |             | ->40.12% (819,056,147B) 0x2143BB: repo_parse_commit_internal (commit.c:511)
| | | |             |   ->40.12% (819,056,147B) 0x2143BB: repo_parse_commit_internal (commit.c:495)
| | | |             |     ->40.12% (819,056,147B) 0x25A747: repo_parse_commit (commit.h:90)
| | | |             |       ->40.12% (819,056,147B) 0x25A747: fsck_walk_commit (fsck.c:355)
| | | |             |         ->40.12% (819,056,147B) 0x25A747: fsck_walk (fsck.c:441)
| | | |             |           ->40.12% (819,056,147B) 0x16C7BB: traverse_one_object (fsck.c:175)
| | | |             |             ->40.12% (819,056,147B) 0x16C7BB: traverse_reachable (fsck.c:192)
| | | |             |               ->40.12% (819,056,147B) 0x16C7BB: check_connectivity (fsck.c:363)
| | | |             |                 ->40.12% (819,056,147B) 0x16C7BB: cmd_fsck (fsck.c:982)
| | | |             |                   ->40.12% (819,056,147B) 0x128007: run_builtin (git.c:466)
| | | |             |                     ->40.12% (819,056,147B) 0x128007: handle_builtin (git.c:720)
| | | |             |                       ->40.12% (819,056,147B) 0x129127: cmd_main (git.c:888)
| | | |             |                         ->40.12% (819,056,147B) 0x127B67: main (common-main.c:56)
| | | |             |
| | | |             ->00.36% (7,389,673B) in 1+ places, all below ms_print's threshold (01.00%)
| | | |
| | | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->08.61% (175,870,114B) 0x2B9DC3: patch_delta (patch-delta.c:36)
| | | ->08.61% (175,870,114B) 0x2B42EB: unpack_entry (packfile.c:1829)
| | |   ->08.61% (175,870,114B) 0x2B4A23: cache_or_unpack_entry (packfile.c:1438)
| | |     ->08.61% (175,870,114B) 0x2B4A23: packed_object_info (packfile.c:1516)
| | |       ->08.61% (175,870,114B) 0x29EA83: do_oid_object_info_extended (object-file.c:1620)
| | |         ->08.61% (175,870,114B) 0x29EBDB: oid_object_info_extended (object-file.c:1639)
| | |           ->08.61% (175,870,114B) 0x29EDC3: read_object (object-file.c:1671)
| | |             ->08.61% (175,870,114B) 0x29EE5B: read_object_file_extended (object-file.c:1714)
| | |               ->04.52% (92,219,588B) 0x3461CB: repo_read_object_file (object-store.h:253)
| | |               | ->04.52% (92,219,588B) 0x3461CB: parse_tree_gently.part.0 (tree.c:132)
| | |               |   ->04.52% (92,219,588B) 0x25A4EB: parse_tree (tree.h:24)
| | |               |     ->04.52% (92,219,588B) 0x25A4EB: fsck_walk_tree (fsck.c:307)
| | |               |       ->04.52% (92,219,588B) 0x25A4EB: fsck_walk (fsck.c:439)
| | |               |         ->04.52% (92,219,588B) 0x16C7BB: traverse_one_object (fsck.c:175)
| | |               |           ->04.52% (92,219,588B) 0x16C7BB: traverse_reachable (fsck.c:192)
| | |               |             ->04.52% (92,219,588B) 0x16C7BB: check_connectivity (fsck.c:363)
| | |               |               ->04.52% (92,219,588B) 0x16C7BB: cmd_fsck (fsck.c:982)
| | |               |                 ->04.52% (92,219,588B) 0x128007: run_builtin (git.c:466)
| | |               |                   ->04.52% (92,219,588B) 0x128007: handle_builtin (git.c:720)
| | |               |                     ->04.52% (92,219,588B) 0x129127: cmd_main (git.c:888)
| | |               |                       ->04.52% (92,219,588B) 0x127B67: main (common-main.c:56)
| | |               |
| | |               ->04.10% (83,650,526B) 0x2143BB: repo_read_object_file (object-store.h:253)
| | |               | ->04.10% (83,650,526B) 0x2143BB: repo_parse_commit_internal (commit.c:511)
| | |               |   ->04.10% (83,650,526B) 0x2143BB: repo_parse_commit_internal (commit.c:495)
| | |               |     ->04.10% (83,650,526B) 0x25A747: repo_parse_commit (commit.h:90)
| | |               |       ->04.10% (83,650,526B) 0x25A747: fsck_walk_commit (fsck.c:355)
| | |               |         ->04.10% (83,650,526B) 0x25A747: fsck_walk (fsck.c:441)
| | |               |           ->04.10% (83,650,526B) 0x16C7BB: traverse_one_object (fsck.c:175)
| | |               |             ->04.10% (83,650,526B) 0x16C7BB: traverse_reachable (fsck.c:192)
| | |               |               ->04.10% (83,650,526B) 0x16C7BB: check_connectivity (fsck.c:363)
| | |               |                 ->04.10% (83,650,526B) 0x16C7BB: cmd_fsck (fsck.c:982)
| | |               |                   ->04.10% (83,650,526B) 0x128007: run_builtin (git.c:466)
| | |               |                     ->04.10% (83,650,526B) 0x128007: handle_builtin (git.c:720)
| | |               |                       ->04.10% (83,650,526B) 0x129127: cmd_main (git.c:888)
| | |               |                         ->04.10% (83,650,526B) 0x127B67: main (common-main.c:56)
| | |               |
| | |               ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->01.64% (33,500,854B) 0x357903: do_xmallocz (wrapper.c:83)
| | | ->01.64% (33,500,854B) 0x357903: xmallocz (wrapper.c:93)
| | |   ->01.64% (33,500,854B) 0x357903: xmemdupz (wrapper.c:109)
| | |     ->01.64% (33,493,416B) 0x2B476F: cache_or_unpack_entry (packfile.c:1444)
| | |     | ->01.64% (33,493,416B) 0x2B476F: packed_object_info (packfile.c:1516)
| | |     |   ->01.64% (33,493,416B) 0x29EA83: do_oid_object_info_extended (object-file.c:1620)
| | |     |     ->01.64% (33,493,416B) 0x29EBDB: oid_object_info_extended (object-file.c:1639)
| | |     |       ->01.64% (33,493,416B) 0x29EDC3: read_object (object-file.c:1671)
| | |     |         ->01.64% (33,493,416B) 0x29EE5B: read_object_file_extended (object-file.c:1714)
| | |     |           ->01.64% (33,438,291B) 0x2143BB: repo_read_object_file (object-store.h:253)
| | |     |           | ->01.64% (33,438,291B) 0x2143BB: repo_parse_commit_internal (commit.c:511)
| | |     |           |   ->01.64% (33,438,291B) 0x2143BB: repo_parse_commit_internal (commit.c:495)
| | |     |           |     ->01.64% (33,438,291B) 0x25A747: repo_parse_commit (commit.h:90)
| | |     |           |       ->01.64% (33,438,291B) 0x25A747: fsck_walk_commit (fsck.c:355)
| | |     |           |         ->01.64% (33,438,291B) 0x25A747: fsck_walk (fsck.c:441)
| | |     |           |           ->01.64% (33,438,291B) 0x16C7BB: traverse_one_object (fsck.c:175)
| | |     |           |             ->01.64% (33,438,291B) 0x16C7BB: traverse_reachable (fsck.c:192)
| | |     |           |               ->01.64% (33,438,291B) 0x16C7BB: check_connectivity (fsck.c:363)
| | |     |           |                 ->01.64% (33,438,291B) 0x16C7BB: cmd_fsck (fsck.c:982)
| | |     |           |                   ->01.64% (33,438,291B) 0x128007: run_builtin (git.c:466)
| | |     |           |                     ->01.64% (33,438,291B) 0x128007: handle_builtin (git.c:720)
| | |     |           |                       ->01.64% (33,438,291B) 0x129127: cmd_main (git.c:888)
| | |     |           |                         ->01.64% (33,438,291B) 0x127B67: main (common-main.c:56)
| | |     |           |
| | |     |           ->00.00% (55,125B) in 1+ places, all below ms_print's threshold (01.00%)
| | |     |
| | |     ->00.00% (7,438B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->31.69% (646,963,200B) 0x369A07: alloc_node (alloc.c:59)
| | ->31.69% (646,963,200B) 0x369A07: alloc_object_node (alloc.c:95)
| |   ->31.69% (646,963,200B) 0x2A3A0F: lookup_unknown_object (object.c:184)
| |     ->31.69% (646,963,200B) 0x16B19B: mark_object_for_connectivity (fsck.c:800)
| |       ->31.69% (646,963,200B) 0x16B19B: mark_packed_for_connectivity (fsck.c:817)
| |         ->31.69% (646,963,200B) 0x2B50AB: for_each_object_in_pack (packfile.c:2176)
| |           ->31.69% (646,963,200B) 0x2B5217: for_each_packed_object (packfile.c:2207)
| |             ->31.69% (646,963,200B) 0x16C6A3: cmd_fsck (fsck.c:881)
| |               ->31.69% (646,963,200B) 0x128007: run_builtin (git.c:466)
| |                 ->31.69% (646,963,200B) 0x128007: handle_builtin (git.c:720)
| |                   ->31.69% (646,963,200B) 0x129127: cmd_main (git.c:888)
| |                     ->31.69% (646,963,200B) 0x127B67: main (common-main.c:56)
| |
| ->01.62% (33,127,806B) in 14 places, all below massif's threshold (1.00%)
|
->14.12% (288,287,504B) 0x357A37: xcalloc (wrapper.c:150)
| ->13.15% (268,435,456B) 0x2A386B: grow_object_hash (object.c:130)
| | ->13.15% (268,435,456B) 0x2A386B: create_object (object.c:152)
| |   ->13.15% (268,435,456B) 0x16B19B: mark_object_for_connectivity (fsck.c:800)
| |     ->13.15% (268,435,456B) 0x16B19B: mark_packed_for_connectivity (fsck.c:817)
| |       ->13.15% (268,435,456B) 0x2B50AB: for_each_object_in_pack (packfile.c:2176)
| |         ->13.15% (268,435,456B) 0x2B5217: for_each_packed_object (packfile.c:2207)
| |           ->13.15% (268,435,456B) 0x16C6A3: cmd_fsck (fsck.c:881)
| |             ->13.15% (268,435,456B) 0x128007: run_builtin (git.c:466)
| |               ->13.15% (268,435,456B) 0x128007: handle_builtin (git.c:720)
| |                 ->13.15% (268,435,456B) 0x129127: cmd_main (git.c:888)
| |                   ->13.15% (268,435,456B) 0x127B67: main (common-main.c:56)
| |
| ->00.97% (19,852,048B) in 1+ places, all below ms_print's threshold (01.00%)
|
->00.21% (4,289,275B) in 1+ places, all below ms_print's threshold (01.00%)

`git rev-list --all --objects`:

99.88% (1,026,125,720B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
->54.04% (555,159,796B) 0x35779F: do_xmalloc (wrapper.c:51)
| ->27.44% (281,960,448B) 0x369803: alloc_node (alloc.c:59)
| | ->27.44% (281,960,448B) 0x369803: alloc_tree_node (alloc.c:81)
| |   ->27.44% (281,960,448B) 0x3462DF: lookup_tree (tree.c:109)
| |     ->27.44% (281,960,448B) 0x3462DF: lookup_tree (tree.c:105)
| |       ->21.38% (219,684,864B) 0x270707: process_tree_contents (list-objects.c:142)
| |       | ->21.38% (219,684,864B) 0x270707: process_tree (list-objects.c:221)
| |       |   ->15.11% (155,287,552B) 0x27072B: process_tree_contents (list-objects.c:149)
| |       |   | ->15.11% (155,287,552B) 0x27072B: process_tree (list-objects.c:221)
| |       |   |   ->08.64% (88,768,512B) 0x27072B: process_tree_contents (list-objects.c:149)
| |       |   |   | ->08.64% (88,768,512B) 0x27072B: process_tree (list-objects.c:221)
| |       |   |   |   ->04.77% (49,029,120B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |       |   |   |   | ->04.77% (49,029,120B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |       |   |   |   |   ->04.77% (49,029,120B) 0x270D5B: do_traverse (list-objects.c:432)
| |       |   |   |   |     ->04.77% (49,029,120B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |       |   |   |   |       ->04.77% (49,029,120B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       |   |   |   |         ->04.77% (49,029,120B) 0x128007: run_builtin (git.c:466)
| |       |   |   |   |           ->04.77% (49,029,120B) 0x128007: handle_builtin (git.c:720)
| |       |   |   |   |             ->04.77% (49,029,120B) 0x129127: cmd_main (git.c:888)
| |       |   |   |   |               ->04.77% (49,029,120B) 0x127B67: main (common-main.c:56)
| |       |   |   |   |
| |       |   |   |   ->03.87% (39,739,392B) 0x27072B: process_tree_contents (list-objects.c:149)
| |       |   |   |     ->03.87% (39,739,392B) 0x27072B: process_tree (list-objects.c:221)
| |       |   |   |       ->02.45% (25,174,016B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |       |   |   |       | ->02.45% (25,174,016B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |       |   |   |       |   ->02.45% (25,174,016B) 0x270D5B: do_traverse (list-objects.c:432)
| |       |   |   |       |     ->02.45% (25,174,016B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |       |   |   |       |       ->02.45% (25,174,016B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       |   |   |       |         ->02.45% (25,174,016B) 0x128007: run_builtin (git.c:466)
| |       |   |   |       |           ->02.45% (25,174,016B) 0x128007: handle_builtin (git.c:720)
| |       |   |   |       |             ->02.45% (25,174,016B) 0x129127: cmd_main (git.c:888)
| |       |   |   |       |               ->02.45% (25,174,016B) 0x127B67: main (common-main.c:56)
| |       |   |   |       |
| |       |   |   |       ->01.42% (14,565,376B) 0x27072B: process_tree_contents (list-objects.c:149)
| |       |   |   |         ->01.42% (14,565,376B) 0x27072B: process_tree (list-objects.c:221)
| |       |   |   |           ->01.00% (10,321,920B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |       |   |   |           | ->01.00% (10,321,920B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |       |   |   |           |   ->01.00% (10,321,920B) 0x270D5B: do_traverse (list-objects.c:432)
| |       |   |   |           |     ->01.00% (10,321,920B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |       |   |   |           |       ->01.00% (10,321,920B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       |   |   |           |         ->01.00% (10,321,920B) 0x128007: run_builtin (git.c:466)
| |       |   |   |           |           ->01.00% (10,321,920B) 0x128007: handle_builtin (git.c:720)
| |       |   |   |           |             ->01.00% (10,321,920B) 0x129127: cmd_main (git.c:888)
| |       |   |   |           |               ->01.00% (10,321,920B) 0x127B67: main (common-main.c:56)
| |       |   |   |           |
| |       |   |   |           ->00.41% (4,243,456B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |   |   |
| |       |   |   ->06.47% (66,519,040B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |       |   |     ->06.47% (66,519,040B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |       |   |       ->06.47% (66,519,040B) 0x270D5B: do_traverse (list-objects.c:432)
| |       |   |         ->06.47% (66,519,040B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |       |   |           ->06.47% (66,519,040B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       |   |             ->06.47% (66,519,040B) 0x128007: run_builtin (git.c:466)
| |       |   |               ->06.47% (66,519,040B) 0x128007: handle_builtin (git.c:720)
| |       |   |                 ->06.47% (66,519,040B) 0x129127: cmd_main (git.c:888)
| |       |   |                   ->06.47% (66,519,040B) 0x127B67: main (common-main.c:56)
| |       |   |
| |       |   ->06.27% (64,397,312B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |       |     ->06.27% (64,397,312B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |       |       ->06.27% (64,397,312B) 0x270D5B: do_traverse (list-objects.c:432)
| |       |         ->06.27% (64,397,312B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |       |           ->06.27% (64,397,312B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       |             ->06.27% (64,397,312B) 0x128007: run_builtin (git.c:466)
| |       |               ->06.27% (64,397,312B) 0x128007: handle_builtin (git.c:720)
| |       |                 ->06.27% (64,397,312B) 0x129127: cmd_main (git.c:888)
| |       |                   ->06.27% (64,397,312B) 0x127B67: main (common-main.c:56)
| |       |
| |       ->06.06% (62,275,584B) 0x213F93: parse_commit_buffer (commit.c:440)
| |         ->06.06% (62,218,240B) 0x2143E7: repo_parse_commit_internal (commit.c:522)
| |         | ->06.06% (62,218,240B) 0x2143E7: repo_parse_commit_internal (commit.c:495)
| |         |   ->06.06% (62,218,240B) 0x2FC77B: process_parents (revision.c:1169)
| |         |     ->06.06% (62,218,240B) 0x2FFEBB: get_revision_1 (revision.c:4080)
| |         |       ->06.06% (62,218,240B) 0x3000EB: get_revision_internal (revision.c:4188)
| |         |         ->06.06% (62,218,240B) 0x30026F: get_revision (revision.c:4266)
| |         |           ->06.06% (62,218,240B) 0x270C63: do_traverse (list-objects.c:397)
| |         |             ->06.06% (62,218,240B) 0x270C63: traverse_commit_list_filtered (list-objects.c:453)
| |         |               ->06.06% (62,218,240B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |         |                 ->06.06% (62,218,240B) 0x128007: run_builtin (git.c:466)
| |         |                   ->06.06% (62,218,240B) 0x128007: handle_builtin (git.c:720)
| |         |                     ->06.06% (62,218,240B) 0x129127: cmd_main (git.c:888)
| |         |                       ->06.06% (62,218,240B) 0x127B67: main (common-main.c:56)
| |         |
| |         ->00.01% (57,344B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->09.80% (100,703,562B) 0x35781F: do_xmallocz.part.0 (wrapper.c:85)
| | ->09.47% (97,307,344B) 0x2B9DC3: patch_delta (patch-delta.c:36)
| | | ->09.47% (97,307,344B) 0x2B42EB: unpack_entry (packfile.c:1829)
| | |   ->09.47% (97,307,344B) 0x2B4A23: cache_or_unpack_entry (packfile.c:1438)
| | |     ->09.47% (97,307,344B) 0x2B4A23: packed_object_info (packfile.c:1516)
| | |       ->09.47% (97,307,344B) 0x29EA83: do_oid_object_info_extended (object-file.c:1620)
| | |         ->09.47% (97,307,344B) 0x29EBDB: oid_object_info_extended (object-file.c:1639)
| | |           ->09.47% (97,307,344B) 0x29EDC3: read_object (object-file.c:1671)
| | |             ->09.47% (97,307,344B) 0x29EE5B: read_object_file_extended (object-file.c:1714)
| | |               ->09.47% (97,307,344B) 0x3461CB: repo_read_object_file (object-store.h:253)
| | |               | ->09.47% (97,307,344B) 0x3461CB: parse_tree_gently.part.0 (tree.c:132)
| | |               |   ->09.47% (97,307,344B) 0x2703EF: process_tree (list-objects.c:188)
| | |               |     ->09.07% (93,167,206B) 0x27072B: process_tree_contents (list-objects.c:149)
| | |               |     | ->09.07% (93,167,206B) 0x27072B: process_tree (list-objects.c:221)
| | |               |     |   ->07.49% (76,907,030B) 0x27072B: process_tree_contents (list-objects.c:149)
| | |               |     |   | ->07.49% (76,907,030B) 0x27072B: process_tree (list-objects.c:221)
| | |               |     |   |   ->05.69% (58,476,176B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| | |               |     |   |   | ->05.69% (58,476,176B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| | |               |     |   |   |   ->05.69% (58,476,176B) 0x270D5B: do_traverse (list-objects.c:432)
| | |               |     |   |   |     ->05.69% (58,476,176B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| | |               |     |   |   |       ->05.69% (58,476,176B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| | |               |     |   |   |         ->05.69% (58,476,176B) 0x128007: run_builtin (git.c:466)
| | |               |     |   |   |           ->05.69% (58,476,176B) 0x128007: handle_builtin (git.c:720)
| | |               |     |   |   |             ->05.69% (58,476,176B) 0x129127: cmd_main (git.c:888)
| | |               |     |   |   |               ->05.69% (58,476,176B) 0x127B67: main (common-main.c:56)
| | |               |     |   |   |
| | |               |     |   |   ->01.79% (18,430,854B) 0x27072B: process_tree_contents (list-objects.c:149)
| | |               |     |   |     ->01.79% (18,430,854B) 0x27072B: process_tree (list-objects.c:221)
| | |               |     |   |       ->01.26% (12,951,424B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| | |               |     |   |       | ->01.26% (12,951,424B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| | |               |     |   |       |   ->01.26% (12,951,424B) 0x270D5B: do_traverse (list-objects.c:432)
| | |               |     |   |       |     ->01.26% (12,951,424B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| | |               |     |   |       |       ->01.26% (12,951,424B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| | |               |     |   |       |         ->01.26% (12,951,424B) 0x128007: run_builtin (git.c:466)
| | |               |     |   |       |           ->01.26% (12,951,424B) 0x128007: handle_builtin (git.c:720)
| | |               |     |   |       |             ->01.26% (12,951,424B) 0x129127: cmd_main (git.c:888)
| | |               |     |   |       |               ->01.26% (12,951,424B) 0x127B67: main (common-main.c:56)
| | |               |     |   |       |
| | |               |     |   |       ->00.53% (5,479,430B) in 1+ places, all below ms_print's threshold (01.00%)
| | |               |     |   |
| | |               |     |   ->01.58% (16,260,176B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| | |               |     |     ->01.58% (16,260,176B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| | |               |     |       ->01.58% (16,260,176B) 0x270D5B: do_traverse (list-objects.c:432)
| | |               |     |         ->01.58% (16,260,176B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| | |               |     |           ->01.58% (16,260,176B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| | |               |     |             ->01.58% (16,260,176B) 0x128007: run_builtin (git.c:466)
| | |               |     |               ->01.58% (16,260,176B) 0x128007: handle_builtin (git.c:720)
| | |               |     |                 ->01.58% (16,260,176B) 0x129127: cmd_main (git.c:888)
| | |               |     |                   ->01.58% (16,260,176B) 0x127B67: main (common-main.c:56)
| | |               |     |
| | |               |     ->00.40% (4,140,138B) in 1+ places, all below ms_print's threshold (01.00%)
| | |               |
| | |               ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| | |
| | ->00.33% (3,396,218B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->08.68% (89,210,880B) 0x36970B: alloc_node (alloc.c:59)
| | ->08.68% (89,210,880B) 0x36970B: alloc_blob_node (alloc.c:74)
| |   ->08.68% (89,210,880B) 0x1FEE17: lookup_blob (blob.c:12)
| |     ->08.68% (89,210,880B) 0x27066F: process_tree_contents (list-objects.c:155)
| |       ->08.68% (89,210,880B) 0x27066F: process_tree (list-objects.c:221)
| |         ->08.59% (88,227,840B) 0x27072B: process_tree_contents (list-objects.c:149)
| |         | ->08.59% (88,227,840B) 0x27072B: process_tree (list-objects.c:221)
| |         |   ->08.24% (84,623,360B) 0x27072B: process_tree_contents (list-objects.c:149)
| |         |   | ->08.24% (84,623,360B) 0x27072B: process_tree (list-objects.c:221)
| |         |   |   ->05.85% (60,088,320B) 0x27072B: process_tree_contents (list-objects.c:149)
| |         |   |   | ->05.85% (60,088,320B) 0x27072B: process_tree (list-objects.c:221)
| |         |   |   |   ->03.17% (32,604,160B) 0x27072B: process_tree_contents (list-objects.c:149)
| |         |   |   |   | ->03.17% (32,604,160B) 0x27072B: process_tree (list-objects.c:221)
| |         |   |   |   |   ->01.92% (19,742,720B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |         |   |   |   |   | ->01.92% (19,742,720B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |         |   |   |   |   |   ->01.92% (19,742,720B) 0x270D5B: do_traverse (list-objects.c:432)
| |         |   |   |   |   |     ->01.92% (19,742,720B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |         |   |   |   |   |       ->01.92% (19,742,720B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |         |   |   |   |   |         ->01.92% (19,742,720B) 0x128007: run_builtin (git.c:466)
| |         |   |   |   |   |           ->01.92% (19,742,720B) 0x128007: handle_builtin (git.c:720)
| |         |   |   |   |   |             ->01.92% (19,742,720B) 0x129127: cmd_main (git.c:888)
| |         |   |   |   |   |               ->01.92% (19,742,720B) 0x127B67: main (common-main.c:56)
| |         |   |   |   |   |
| |         |   |   |   |   ->01.25% (12,861,440B) 0x27072B: process_tree_contents (list-objects.c:149)
| |         |   |   |   |     ->01.25% (12,861,440B) 0x27072B: process_tree (list-objects.c:221)
| |         |   |   |   |       ->01.25% (12,861,440B) in 2 places, all below massif's threshold (1.00%)
| |         |   |   |   |
| |         |   |   |   ->02.68% (27,484,160B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |         |   |   |     ->02.68% (27,484,160B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |         |   |   |       ->02.68% (27,484,160B) 0x270D5B: do_traverse (list-objects.c:432)
| |         |   |   |         ->02.68% (27,484,160B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |         |   |   |           ->02.68% (27,484,160B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |         |   |   |             ->02.68% (27,484,160B) 0x128007: run_builtin (git.c:466)
| |         |   |   |               ->02.68% (27,484,160B) 0x128007: handle_builtin (git.c:720)
| |         |   |   |                 ->02.68% (27,484,160B) 0x129127: cmd_main (git.c:888)
| |         |   |   |                   ->02.68% (27,484,160B) 0x127B67: main (common-main.c:56)
| |         |   |   |
| |         |   |   ->02.39% (24,535,040B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |         |   |     ->02.39% (24,535,040B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |         |   |       ->02.39% (24,535,040B) 0x270D5B: do_traverse (list-objects.c:432)
| |         |   |         ->02.39% (24,535,040B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |         |   |           ->02.39% (24,535,040B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |         |   |             ->02.39% (24,535,040B) 0x128007: run_builtin (git.c:466)
| |         |   |               ->02.39% (24,535,040B) 0x128007: handle_builtin (git.c:720)
| |         |   |                 ->02.39% (24,535,040B) 0x129127: cmd_main (git.c:888)
| |         |   |                   ->02.39% (24,535,040B) 0x127B67: main (common-main.c:56)
| |         |   |
| |         |   ->00.35% (3,604,480B) in 1+ places, all below ms_print's threshold (01.00%)
| |         |
| |         ->00.10% (983,040B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->07.87% (80,879,616B) 0x369B27: alloc_node (alloc.c:59)
| | ->07.87% (80,879,616B) 0x369B27: alloc_commit_node (alloc.c:119)
| |   ->07.87% (80,879,616B) 0x2133B7: lookup_commit (commit.c:66)
| |     ->07.87% (80,879,616B) 0x2133B7: lookup_commit (commit.c:62)
| |       ->07.87% (80,805,888B) 0x21411B: parse_commit_buffer (commit.c:466)
| |       | ->07.86% (80,732,160B) 0x2143E7: repo_parse_commit_internal (commit.c:522)
| |       | | ->07.86% (80,732,160B) 0x2143E7: repo_parse_commit_internal (commit.c:495)
| |       | |   ->07.86% (80,732,160B) 0x2FC77B: process_parents (revision.c:1169)
| |       | |     ->07.86% (80,732,160B) 0x2FFEBB: get_revision_1 (revision.c:4080)
| |       | |       ->07.86% (80,732,160B) 0x3000EB: get_revision_internal (revision.c:4188)
| |       | |         ->07.86% (80,732,160B) 0x30026F: get_revision (revision.c:4266)
| |       | |           ->07.86% (80,732,160B) 0x270C63: do_traverse (list-objects.c:397)
| |       | |             ->07.86% (80,732,160B) 0x270C63: traverse_commit_list_filtered (list-objects.c:453)
| |       | |               ->07.86% (80,732,160B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |       | |                 ->07.86% (80,732,160B) 0x128007: run_builtin (git.c:466)
| |       | |                   ->07.86% (80,732,160B) 0x128007: handle_builtin (git.c:720)
| |       | |                     ->07.86% (80,732,160B) 0x129127: cmd_main (git.c:888)
| |       | |                       ->07.86% (80,732,160B) 0x127B67: main (common-main.c:56)
| |       | |
| |       | ->00.01% (73,728B) in 1+ places, all below ms_print's threshold (01.00%)
| |       |
| |       ->00.01% (73,728B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.23% (2,405,290B) in 1+ places, all below ms_print's threshold (01.00%)
|
->39.29% (403,708,926B) 0x357A37: xcalloc (wrapper.c:150)
| ->39.19% (402,653,184B) 0x2A386B: grow_object_hash (object.c:130)
| | ->39.19% (402,653,184B) 0x2A386B: create_object (object.c:152)
| |   ->39.19% (402,653,184B) 0x270707: process_tree_contents (list-objects.c:142)
| |   | ->39.19% (402,653,184B) 0x270707: process_tree (list-objects.c:221)
| |   |   ->26.13% (268,435,456B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |   |   | ->26.13% (268,435,456B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |   |   |   ->26.13% (268,435,456B) 0x270D5B: do_traverse (list-objects.c:432)
| |   |   |     ->26.13% (268,435,456B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |   |   |       ->26.13% (268,435,456B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |   |   |         ->26.13% (268,435,456B) 0x128007: run_builtin (git.c:466)
| |   |   |           ->26.13% (268,435,456B) 0x128007: handle_builtin (git.c:720)
| |   |   |             ->26.13% (268,435,456B) 0x129127: cmd_main (git.c:888)
| |   |   |               ->26.13% (268,435,456B) 0x127B67: main (common-main.c:56)
| |   |   |
| |   |   ->13.06% (134,217,728B) 0x27072B: process_tree_contents (list-objects.c:149)
| |   |     ->13.06% (134,217,728B) 0x27072B: process_tree (list-objects.c:221)
| |   |       ->13.06% (134,217,728B) 0x27072B: process_tree_contents (list-objects.c:149)
| |   |         ->13.06% (134,217,728B) 0x27072B: process_tree (list-objects.c:221)
| |   |           ->13.06% (134,217,728B) 0x2708D3: traverse_non_commits (list-objects.c:378)
| |   |             ->13.06% (134,217,728B) 0x2708D3: traverse_non_commits (list-objects.c:357)
| |   |               ->13.06% (134,217,728B) 0x270D5B: do_traverse (list-objects.c:432)
| |   |                 ->13.06% (134,217,728B) 0x270D5B: traverse_commit_list_filtered (list-objects.c:453)
| |   |                   ->13.06% (134,217,728B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |   |                     ->13.06% (134,217,728B) 0x128007: run_builtin (git.c:466)
| |   |                       ->13.06% (134,217,728B) 0x128007: handle_builtin (git.c:720)
| |   |                         ->13.06% (134,217,728B) 0x129127: cmd_main (git.c:888)
| |   |                           ->13.06% (134,217,728B) 0x127B67: main (common-main.c:56)
| |   |
| |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.10% (1,055,742B) in 1+ places, all below ms_print's threshold (01.00%)
|
->06.54% (67,228,381B) 0x3579CF: xrealloc (wrapper.c:136)
| ->06.53% (67,106,816B) 0x2A406B: add_object_array_with_path (object.c:353)
| | ->06.53% (67,106,816B) 0x2F8427: add_pending_object_with_path (revision.c:343)
| |   ->06.53% (67,106,816B) 0x270D17: add_pending_tree (list-objects.c:354)
| |   | ->06.53% (67,106,816B) 0x270D17: do_traverse (list-objects.c:413)
| |   |   ->06.53% (67,106,816B) 0x270D17: traverse_commit_list_filtered (list-objects.c:453)
| |   |     ->06.53% (67,106,816B) 0x1BD3D3: cmd_rev_list (rev-list.c:721)
| |   |       ->06.53% (67,106,816B) 0x128007: run_builtin (git.c:466)
| |   |         ->06.53% (67,106,816B) 0x128007: handle_builtin (git.c:720)
| |   |           ->06.53% (67,106,816B) 0x129127: cmd_main (git.c:888)
| |   |             ->06.53% (67,106,816B) 0x127B67: main (common-main.c:56)
| |   |
| |   ->00.00% (0B) in 1+ places, all below ms_print's threshold (01.00%)
| |
| ->00.01% (121,565B) in 1+ places, all below ms_print's threshold (01.00%)
|
->00.00% (28,617B) in 1+ places, all below ms_print's threshold (01.00%)

One observation is that `git-fsck` seems to use up alot more memory in calling
unpack_compressed_entry() than `git-rev-list` does.

But, not having delved too deeply into the code, I wanted to ask the list if
anything jumps out as to why `git-fsck` consumes so much more memory than
`git-rev-list`--perhaps there is opportunity for improvement/optimization?

thanks!
John

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2022-09-22 22:16 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-09-20 19:27 [INVESTIGATION] why is fsck --connectivity-only so much more expensive than rev-list --objects --all? John Cai
2022-09-20 20:41 ` Jeff King
2022-09-22 10:09   ` [PATCH 0/3] reducing fsck memory usage Jeff King
2022-09-22 10:11     ` [PATCH 1/3] fsck: free tree buffers after walking unreachable objects Jeff King
2022-09-22 18:40       ` Junio C Hamano
2022-09-22 18:58         ` Jeff King
2022-09-22 19:27           ` Junio C Hamano
2022-09-22 22:16             ` Jeff King
2022-09-22 10:13     ` [PATCH 2/3] fsck: turn off save_commit_buffer Jeff King
2022-09-22 10:15     ` [PATCH 3/3] parse_object_buffer(): respect save_commit_buffer Jeff King

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).