Re: [PATCH 3/6] introduce pack metadata cache files

[Jeff King <peff@peff.net>]

On Tue, Jan 29, 2013 at 09:35:12AM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> > +static void write_meta_header(struct metapack_writer *mw, const char *id,
> > +			      uint32_t version)
> > +{
> > +	version = htonl(version);
> > +
> > +	sha1write(mw->out, "META", 4);
> > +	sha1write(mw->out, "\0\0\0\1", 4);
> > +	sha1write(mw->out, mw->pack->sha1, 20);
> > +	sha1write(mw->out, id, 4);
> > +	sha1write(mw->out, &version, 4);
> > +}
> 
> It seems that you are very close to actually having a plumbing that
> could also do the pack .idx files.  Until/unless that can be done, I
> am not sure how much benefit we would be getting from a file format
> that records a subtype "id" and a generic "META" type, instead of
> just a single "id" as the type ehader.  But it is OK to use 8 extra
> bytes if we can potentially gain something later.

Yeah, I considered going that route. I had initially envisioned having a
generic META file type that provided some services (like fixed-size
records), and then having individual subtypes below that. But as I
simplified the design, the META format became pretty much pointless. I
left it in as the 8 bytes are not really a big problem, and it means we
can treat metapacks generically in some cases without necessarily
knowing what is in them. But I don't have a specific use case in mind,
so perhaps it is just useless and confusing. I don't mind simplifying.

> Shouldn't id be validated with at least something like
> 
> 	if (strlen(id) < 3)
> 		die("Bad id: %s", id);
> 
> to catch a call
> 
> 	write_meta_header(&mw, "me", 47);
> 
> that will stuff 'm', 'e', NUL and the garbage the compiler/linker
> combo has placed after that constant string in the 4-byte id field?

Yes, the id does need to be at least 4 bytes. Since the id is intended
to be a static string, I had planned to just document the requirement in
the API documentation. I don't mind putting in a run-time check. I had
originally had a separate "id" parameter that could be "char id[4]", but
found that it was just redundant with the "name" parameter: you ended up
passing ("commit", "CMIT") or similar.

> > +	strbuf_addstr(&path, pack_idx);
> > +	strbuf_chompstr(&path, ".idx");
> > +	strbuf_addch(&path, '.');
> > +	strbuf_addstr(&path, name);
> 
> Your chompstr() does not even validate if the given name ends with
> ".idx",

Yeah, my intent was that it would be liberal in its input (i.e., take
just "pack-*"). E.g., you can run "git metapack pack/pack-XXXX".

> so this sounds like a glorified way to say
> 
> 	strbuf_splice(&path, path->len - strlen("idx"), strlen("idx"),
> 			 name, strlen(name));
> 
> to me.

Yup, though my version handles edge cases by not chomping (e.g., what
does splice do when path->len is less than 3?).

> > +void metapack_writer_finish(struct metapack_writer *mw)
> > +{
> > +	const char *tmp = mw->out->name;
> > +
> > +	sha1close(mw->out, NULL, CSUM_FSYNC);
> > +	if (rename(tmp, mw->path))
> > +		die_errno("unable to rename temporary metapack file");
> 
> Who is responsible for running adjust_shared_perm()?  The caller, or
> this function?

I didn't think about it at all, but it seems pretty obvious to me that
this function should do so. Thanks for pointing it out.

-Peff