Calculate packfile and idxfile checksums

Calculate packfile and idxfile checksums

[Farhan Khan]

Hi all,

I am trying to understand how git's index-pack works, particularly how
it calculates the packfile checksum and idxfile checksum. I traced
back the packfile checksum in the source to the value char *sha1 that
is utilized in write_idx_file() in pack_file.c:45. However, I cannot
determine where this value is set.

My printf() debugging has it set at pack-write.c:171 (right before the
hashwrite call) but it does not seem to be utilized prior to that
point. Please assist.

Thanks!
--
Farhan Khan
PGP Fingerprint: B28D 2726 E2BC A97E 3854 5ABE 9A9F 00BC D525 16EE

Re: Calculate packfile and idxfile checksums

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 18 2019, Farhan Khan wrote:

> Hi all,
>
> I am trying to understand how git's index-pack works, particularly how
> it calculates the packfile checksum and idxfile checksum.

It's unclear if this is what you're asking, but the pack checksum is
just a 20 bytes of SHA1 of the preceding contents at the end for *.idx
and *.pack. As seen with this program:

    $ perl -MFile::Slurp=slurp -MDigest::SHA=sha1 -wE 'my $f = shift; my $c = slurp($f); my $cp = $c; $cp =~ s/.{20}$//s; my $n = $cp . sha1($cp); if ($n eq $c) { say "Computed checksum trailer for $f" } else { say "Failed trailer for new content for $f is different" }' pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
    Computed checksum trailer for pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx

You can also feed it *.pack files.

> I traced back the packfile checksum in the source to the value char
> *sha1 that is utilized in write_idx_file() in pack_file.c:45. However,
> I cannot

It seems you mean pack-write.c not pack_file.c

> determine where this value is set.

It's initialized in cmd_index_pack() and then passed down to that
function.

> My printf() debugging has it set at pack-write.c:171 (right before the
> hashwrite call) but it does not seem to be utilized prior to that
> point. Please assist.

I'm happy to help, but still not quite sure what the source of the
confusion is, maybe that the variable in index-pack.c has a different
name and is passed down to pack-write.c's function as a pointer?

Re: Calculate packfile and idxfile checksums

[Farhan Khan]

On Fri, Jan 18, 2019 at 3:36 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
>
> On Fri, Jan 18 2019, Farhan Khan wrote:
>
> > Hi all,
> >
> > I am trying to understand how git's index-pack works, particularly how
> > it calculates the packfile checksum and idxfile checksum.
>
> It's unclear if this is what you're asking, but the pack checksum is
> just a 20 bytes of SHA1 of the preceding contents at the end for *.idx
> and *.pack. As seen with this program:
>
>     $ perl -MFile::Slurp=slurp -MDigest::SHA=sha1 -wE 'my $f = shift; my $c = slurp($f); my $cp = $c; $cp =~ s/.{20}$//s; my $n = $cp . sha1($cp); if ($n eq $c) { say "Computed checksum trailer for $f" } else { say "Failed trailer for new content for $f is different" }' pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
>     Computed checksum trailer for pack-79c2ccce950e6676452dc9f0473f80003e7ccdef.idx
>
> You can also feed it *.pack files.
>
> > I traced back the packfile checksum in the source to the value char
> > *sha1 that is utilized in write_idx_file() in pack_file.c:45. However,
> > I cannot
>
> It seems you mean pack-write.c not pack_file.c
>
> > determine where this value is set.
>
> It's initialized in cmd_index_pack() and then passed down to that
> function.
>
> > My printf() debugging has it set at pack-write.c:171 (right before the
> > hashwrite call) but it does not seem to be utilized prior to that
> > point. Please assist.
>
> I'm happy to help, but still not quite sure what the source of the
> confusion is, maybe that the variable in index-pack.c has a different
> name and is passed down to pack-write.c's function as a pointer?

I apologize for my poor email and appreciate your willingness to help.
Let me try that again.

My objective is to identify how to calculate the packfile and idxfile
checksums and/or where this is performed in the git source code.

You said that this value is initialized in
builtin/index-pack.c:cmd_index_pack(). Is it correct that this value
is the 'pack_hash' variable? At a high level, how is this value
calculated?

That seems to be the packfile index, how and where is that idxfile
hash calculated?

Thanks!