From: Derrick Stolee <stolee@gmail.com> To: Eric Sunshine <sunshine@sunshineco.com> Cc: "Git List" <git@vger.kernel.org>, "Junio C Hamano" <gitster@pobox.com>, "Stefan Beller" <sbeller@google.com>, "Nguyễn Thái Ngọc Duy" <pclouds@gmail.com>, "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>, "Derrick Stolee" <dstolee@microsoft.com> Subject: Re: [PATCH v3 07/24] multi-pack-index: expand test data Date: Thu, 12 Jul 2018 10:10:29 -0400 Message-ID: <3e82dd51-2e30-291f-8753-c1d711459c1d@gmail.com> (raw) In-Reply-To: <CAPig+cTO=UPuCRUAAWfBwaNhRD+=dgSwpgEB2fQJuWyFEd7Aew@mail.gmail.com> On 7/6/2018 12:36 AM, Eric Sunshine wrote: > On Thu, Jul 5, 2018 at 8:54 PM Derrick Stolee <stolee@gmail.com> wrote: > >> + for i in $(test_seq 6 10) >> + do >> + iii=$(printf '%03i' $i) >> + test-tool genrandom "bar" 200 >wide_delta_$iii && >> + test-tool genrandom "baz $iii" 50 >>wide_delta_$iii && >> + test-tool genrandom "foo"$i 100 >deep_delta_$iii && >> + test-tool genrandom "foo"$(expr $i + 1) 100 >>deep_delta_$iii && >> + test-tool genrandom "foo"$(expr $i + 2) 100 >>deep_delta_$iii && >> + echo $iii >file_$iii && >> + test-tool genrandom "$iii" 8192 >>file_$iii && >> + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii && >> + i=$(expr $i + 1) || return 1 >> + done && >> + { echo 101 && test-tool genrandom 100 8192; } >file_101 && >> + git update-index --add file_101 && >> + tree=$(git write-tree) && >> + commit=$(git commit-tree $tree -p HEAD</dev/null) && { >> + echo $tree && >> + git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\) .*/\\1/" >> + } >obj-list2 && >> + git update-ref HEAD $commit >> +' > There seems to be a fair bit of duplication in these tests which > create objects. Is it possible to factor out some of this code into a > shell function? In addition to the other small changes, this refactor in particular was a big change (but a good one). I'm sending my current progress in this direction, as I expect this can be improved. To make the commit_and_list_objects method more generic to all situations, I had to add an extra commit, which will cause some of the numbers to change in the later 'midx_read_expect' calls. Thanks, -Stolee -->8-- From cb38bb284fd05cf2230725b6cb9ead5795c913f2 Mon Sep 17 00:00:00 2001 From: Derrick Stolee <dstolee@microsoft.com> Date: Thu, 31 May 2018 15:05:00 -0400 Subject: [PATCH] t5319: expand test data As we build the multi-pack-index file format, we want to test the format on real repositories. Add tests that create repository data including multiple packfiles with both version 1 and version 2 formats. The current 'git multi-pack-index write' command will always write the same file with no "real" data. This will be expanded in future commits, along with the test expectations. Signed-off-by: Derrick Stolee <dstolee@microsoft.com> --- t/t5319-multi-pack-index.sh | 83 +++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 2ecc369529..a50be41bc0 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -13,9 +13,92 @@ midx_read_expect () { } test_expect_success 'write midx with no packs' ' + test_when_finished rm -f pack/multi-pack-index && git multi-pack-index --object-dir=. write && test_path_is_file pack/multi-pack-index && midx_read_expect ' +generate_objects () { + i=$1 + iii=$(printf '%03i' $i) + { + test-tool genrandom "bar" 200 && + test-tool genrandom "baz $iii" 50 + } >wide_delta_$iii && + { + test-tool genrandom "foo"$i 100 && + test-tool genrandom "foo"$(( $i + 1 )) 100 && + test-tool genrandom "foo"$(( $i + 2 )) 100 + } >>deep_delta_$iii && + echo $iii >file_$iii && + test-tool genrandom "$iii" 8192 >>file_$iii && + git update-index --add file_$iii deep_delta_$iii wide_delta_$iii +} + +commit_and_list_objects () { + { + echo 101 && + test-tool genrandom 100 8192; + } >file_101 && + git update-index --add file_101 && + tree=$(git write-tree) && + commit=$(git commit-tree $tree -p HEAD</dev/null) && + { + echo $tree && + git ls-tree $tree | sed -e "s/.* \\([0-9a-f]*\\) .*/\\1/" + } >obj-list && + git reset --hard $commit +} + +test_expect_success 'create objects' ' + test_commit initial && + for i in $(test_seq 1 5) + do + generate_objects $i + done && + commit_and_list_objects +' + +test_expect_success 'write midx with one v1 pack' ' + pack=$(git pack-objects --index-version=1 pack/test <obj-list) && + test_when_finished rm pack/test-$pack.pack pack/test-$pack.idx pack/multi-pack-index && + git multi-pack-index --object-dir=. write && + midx_read_expect +' + +test_expect_success 'write midx with one v2 pack' ' + git pack-objects --index-version=2,0x40 pack/test <obj-list && + git multi-pack-index --object-dir=. write && + midx_read_expect +' + +test_expect_success 'add more objects' ' + for i in $(test_seq 6 10) + do + generate_objects $i + done && + commit_and_list_objects +' + +test_expect_success 'write midx with two packs' ' + git pack-objects --index-version=1 pack/test-2 <obj-list && + git multi-pack-index --object-dir=. write && + midx_read_expect +' + +test_expect_success 'add more packs' ' + for j in $(test_seq 1 10) + do + generate_objects $j && + commit_and_list_objects && + git pack-objects --index-version=2 test-pack <obj-list + done +' + +test_expect_success 'write midx with twelve packs' ' + git multi-pack-index --object-dir=. write && + midx_read_expect +' + test_done -- 2.18.0.118.gd4f65b8d14
next prev parent reply other threads:[~2018-07-12 14:10 UTC|newest] Thread overview: 192+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-06-07 14:03 [PATCH 00/23] Multi-pack-index (MIDX) Derrick Stolee 2018-06-07 14:03 ` [PATCH 01/23] midx: add design document Derrick Stolee 2018-06-11 19:04 ` Stefan Beller 2018-06-18 18:48 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 02/23] midx: add midx format details to pack-format.txt Derrick Stolee 2018-06-11 19:19 ` Stefan Beller 2018-06-18 19:01 ` Derrick Stolee 2018-06-18 19:41 ` Stefan Beller 2018-06-07 14:03 ` [PATCH 03/23] midx: add midx builtin Derrick Stolee 2018-06-07 17:20 ` Duy Nguyen 2018-06-18 19:23 ` Derrick Stolee 2018-06-11 21:02 ` Stefan Beller 2018-06-18 19:40 ` Derrick Stolee 2018-06-18 19:55 ` Stefan Beller 2018-06-18 19:58 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 04/23] midx: add 'write' subcommand and basic wiring Derrick Stolee 2018-06-07 17:27 ` Duy Nguyen 2018-06-07 14:03 ` [PATCH 05/23] midx: write header information to lockfile Derrick Stolee 2018-06-07 17:35 ` Duy Nguyen 2018-06-12 15:00 ` Duy Nguyen 2018-06-19 12:54 ` Derrick Stolee 2018-06-19 14:59 ` Duy Nguyen 2018-06-19 15:24 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 06/23] midx: struct midxed_git and 'read' subcommand Derrick Stolee 2018-06-07 17:54 ` Duy Nguyen 2018-06-20 13:13 ` Derrick Stolee 2018-06-07 18:31 ` Duy Nguyen 2018-06-20 13:33 ` Derrick Stolee 2018-06-20 15:07 ` Duy Nguyen 2018-06-20 16:39 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 07/23] midx: expand test data Derrick Stolee 2018-06-07 14:03 ` [PATCH 08/23] midx: read packfiles from pack directory Derrick Stolee 2018-06-07 18:03 ` Duy Nguyen 2018-06-20 16:33 ` [PATCH] packfile: generalize pack directory list Derrick Stolee 2018-06-07 14:03 ` [PATCH 09/23] midx: write pack names in chunk Derrick Stolee 2018-06-07 18:26 ` Duy Nguyen 2018-06-21 15:25 ` Derrick Stolee 2018-06-21 17:38 ` Junio C Hamano 2018-06-22 18:25 ` Derrick Stolee 2018-06-22 18:31 ` Junio C Hamano 2018-06-22 18:32 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 10/23] midx: write a lookup into the pack names chunk Derrick Stolee 2018-06-09 16:43 ` Duy Nguyen 2018-06-21 17:23 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 11/23] midx: sort and deduplicate objects from packfiles Derrick Stolee 2018-06-09 17:07 ` Duy Nguyen 2018-06-21 17:54 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 12/23] midx: write object ids in a chunk Derrick Stolee 2018-06-09 17:25 ` Duy Nguyen 2018-06-07 14:03 ` [PATCH 13/23] midx: write object id fanout chunk Derrick Stolee 2018-06-09 17:28 ` Duy Nguyen 2018-06-21 19:49 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 14/23] midx: write object offsets Derrick Stolee 2018-06-09 17:41 ` Duy Nguyen 2018-06-07 14:03 ` [PATCH 15/23] midx: create core.midx config setting Derrick Stolee 2018-06-07 14:03 ` [PATCH 16/23] midx: prepare midxed_git struct Derrick Stolee 2018-06-09 17:47 ` Duy Nguyen 2018-06-07 14:03 ` [PATCH 17/23] midx: read objects from multi-pack-index Derrick Stolee 2018-06-09 17:56 ` Duy Nguyen 2018-06-21 20:03 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 18/23] midx: use midx in abbreviation calculations Derrick Stolee 2018-06-09 18:01 ` Duy Nguyen 2018-06-22 18:38 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 19/23] midx: use existing midx when writing new one Derrick Stolee 2018-06-07 14:03 ` [PATCH 20/23] midx: use midx in approximate_object_count Derrick Stolee 2018-06-09 18:03 ` Duy Nguyen 2018-06-22 18:39 ` Derrick Stolee 2018-06-07 14:03 ` [PATCH 21/23] midx: prevent duplicate packfile loads Derrick Stolee 2018-06-09 18:05 ` Duy Nguyen 2018-06-07 14:03 ` [PATCH 22/23] midx: use midx to find ref-deltas Derrick Stolee 2018-06-07 14:03 ` [PATCH 23/23] midx: clear midx on repack Derrick Stolee 2018-06-09 18:13 ` Duy Nguyen 2018-06-22 18:44 ` Derrick Stolee 2018-06-07 14:06 ` [PATCH 00/23] Multi-pack-index (MIDX) Derrick Stolee 2018-06-07 14:45 ` Ævar Arnfjörð Bjarmason 2018-06-07 14:54 ` Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 00/24] " Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 01/24] multi-pack-index: add design document Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 02/24] multi-pack-index: add format details Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 03/24] multi-pack-index: add builtin Derrick Stolee 2018-06-25 19:15 ` Junio C Hamano 2018-06-25 14:34 ` [PATCH v2 04/24] multi-pack-index: add 'write' verb Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 05/24] midx: write header information to lockfile Derrick Stolee 2018-06-25 19:19 ` Junio C Hamano 2018-07-05 19:13 ` Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 06/24] multi-pack-index: load into memory Derrick Stolee 2018-06-25 19:38 ` Junio C Hamano 2018-07-05 14:19 ` Derrick Stolee 2018-07-05 18:58 ` Eric Sunshine 2018-07-06 19:20 ` Junio C Hamano 2018-06-25 14:34 ` [PATCH v2 07/24] multi-pack-index: expand test data Derrick Stolee 2018-06-25 19:45 ` Junio C Hamano 2018-06-25 14:34 ` [PATCH v2 08/24] packfile: generalize pack directory list Derrick Stolee 2018-06-25 19:57 ` Junio C Hamano 2018-06-25 14:34 ` [PATCH v2 09/24] multi-pack-index: read packfile list Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 10/24] multi-pack-index: write pack names in chunk Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 11/24] midx: read pack names into array Derrick Stolee 2018-06-25 23:52 ` Eric Sunshine 2018-06-25 14:34 ` [PATCH v2 12/24] midx: sort and deduplicate objects from packfiles Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 13/24] midx: write object ids in a chunk Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 14/24] midx: write object id fanout chunk Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 15/24] midx: write object offsets Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 16/24] config: create core.multiPackIndex setting Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 17/24] midx: prepare midxed_git struct Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 18/24] midx: read objects from multi-pack-index Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 19/24] midx: use midx in abbreviation calculations Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 20/24] midx: use existing midx when writing new one Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 21/24] midx: use midx in approximate_object_count Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 22/24] midx: prevent duplicate packfile loads Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 23/24] packfile: skip loading index if in multi-pack-index Derrick Stolee 2018-06-25 14:34 ` [PATCH v2 24/24] midx: clear midx on repack Derrick Stolee 2018-07-06 0:52 ` [PATCH v3 00/24] Multi-pack-index (MIDX) Derrick Stolee 2018-07-06 0:52 ` [PATCH v3 01/24] multi-pack-index: add design document Derrick Stolee 2018-07-06 0:52 ` [PATCH v3 02/24] multi-pack-index: add format details Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 03/24] multi-pack-index: add builtin Derrick Stolee 2018-07-06 3:54 ` Eric Sunshine 2018-07-06 0:53 ` [PATCH v3 04/24] multi-pack-index: add 'write' verb Derrick Stolee 2018-07-06 4:07 ` Eric Sunshine 2018-07-06 0:53 ` [PATCH v3 05/24] midx: write header information to lockfile Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 06/24] multi-pack-index: load into memory Derrick Stolee 2018-07-06 4:19 ` Eric Sunshine 2018-07-06 5:18 ` Eric Sunshine 2018-07-09 19:08 ` Junio C Hamano 2018-07-12 16:06 ` Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 07/24] multi-pack-index: expand test data Derrick Stolee 2018-07-06 4:36 ` Eric Sunshine 2018-07-06 5:20 ` Eric Sunshine 2018-07-12 14:10 ` Derrick Stolee [this message] 2018-07-12 18:02 ` Eric Sunshine 2018-07-12 18:06 ` Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 08/24] packfile: generalize pack directory list Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 09/24] multi-pack-index: read packfile list Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 10/24] multi-pack-index: write pack names in chunk Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 11/24] midx: read pack names into array Derrick Stolee 2018-07-06 4:58 ` Eric Sunshine 2018-07-06 0:53 ` [PATCH v3 12/24] midx: sort and deduplicate objects from packfiles Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 13/24] midx: write object ids in a chunk Derrick Stolee 2018-07-06 5:04 ` Eric Sunshine 2018-07-06 0:53 ` [PATCH v3 14/24] midx: write object id fanout chunk Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 15/24] midx: write object offsets Derrick Stolee 2018-07-06 5:27 ` Eric Sunshine 2018-07-12 16:33 ` Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 16/24] config: create core.multiPackIndex setting Derrick Stolee 2018-07-06 5:39 ` Eric Sunshine 2018-07-12 13:19 ` Derrick Stolee 2018-07-12 16:30 ` Derrick Stolee 2018-07-11 9:48 ` SZEDER Gábor 2018-07-12 13:01 ` Derrick Stolee 2018-07-12 13:31 ` SZEDER Gábor 2018-07-12 15:40 ` Derrick Stolee 2018-07-12 17:29 ` Junio C Hamano 2018-07-06 0:53 ` [PATCH v3 17/24] midx: prepare midxed_git struct Derrick Stolee 2018-07-06 5:41 ` Eric Sunshine 2018-07-06 0:53 ` [PATCH v3 18/24] midx: read objects from multi-pack-index Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 19/24] midx: use midx in abbreviation calculations Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 20/24] midx: use existing midx when writing new one Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 21/24] midx: use midx in approximate_object_count Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 22/24] midx: prevent duplicate packfile loads Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 23/24] packfile: skip loading index if in multi-pack-index Derrick Stolee 2018-07-06 0:53 ` [PATCH v3 24/24] midx: clear midx on repack Derrick Stolee 2018-07-06 5:52 ` Eric Sunshine 2018-07-12 19:39 ` [PATCH v4 00/23] Multi-pack-index (MIDX) Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 01/23] multi-pack-index: add design document Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 02/23] multi-pack-index: add format details Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 03/23] multi-pack-index: add builtin Derrick Stolee 2018-07-20 18:22 ` Junio C Hamano 2018-07-20 22:15 ` brian m. carlson 2018-07-20 22:28 ` Junio C Hamano 2018-07-12 19:39 ` [PATCH v4 04/23] multi-pack-index: add 'write' verb Derrick Stolee 2018-07-12 22:56 ` Eric Sunshine 2018-07-12 19:39 ` [PATCH v4 05/23] midx: write header information to lockfile Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 06/23] multi-pack-index: load into memory Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 07/23] t5319: expand test data Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 08/23] packfile: generalize pack directory list Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 09/23] multi-pack-index: read packfile list Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 10/23] multi-pack-index: write pack names in chunk Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 11/23] midx: read pack names into array Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 12/23] midx: sort and deduplicate objects from packfiles Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 13/23] midx: write object ids in a chunk Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 14/23] midx: write object id fanout chunk Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 15/23] midx: write object offsets Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 16/23] config: create core.multiPackIndex setting Derrick Stolee 2018-07-12 21:05 ` Junio C Hamano 2018-07-13 0:50 ` Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 17/23] midx: read objects from multi-pack-index Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 18/23] midx: use midx in abbreviation calculations Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 19/23] midx: use existing midx when writing new one Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 20/23] midx: use midx in approximate_object_count Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 21/23] midx: prevent duplicate packfile loads Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 22/23] packfile: skip loading index if in multi-pack-index Derrick Stolee 2018-07-12 19:39 ` [PATCH v4 23/23] midx: clear midx on repack Derrick Stolee 2018-07-12 21:11 ` [PATCH v4 00/23] Multi-pack-index (MIDX) Junio C Hamano
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://vger.kernel.org/majordomo-info.html * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=3e82dd51-2e30-291f-8753-c1d711459c1d@gmail.com \ --to=stolee@gmail.com \ --cc=avarab@gmail.com \ --cc=dstolee@microsoft.com \ --cc=git@vger.kernel.org \ --cc=gitster@pobox.com \ --cc=pclouds@gmail.com \ --cc=sbeller@google.com \ --cc=sunshine@sunshineco.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
git@vger.kernel.org list mirror (unofficial, one of many) This inbox may be cloned and mirrored by anyone: git clone --mirror https://public-inbox.org/git git clone --mirror http://ou63pmih66umazou.onion/git git clone --mirror http://czquwvybam4bgbro.onion/git git clone --mirror http://hjrcffqmbrq6wope.onion/git # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V1 git git/ https://public-inbox.org/git \ git@vger.kernel.org public-inbox-index git Example config snippet for mirrors. Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.version-control.git nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git nntp://news.gmane.io/gmane.comp.version-control.git note: .onion URLs require Tor: https://www.torproject.org/ code repositories for the project(s) associated with this inbox: https://80x24.org/mirrors/git.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git