Re: reftable [v2]: new ref storage format

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Sun, Jul 23 2017, Shawn Pearce jotted:

> My apologies for not responding to this piece of feedback earlier.
>
> On Wed, Jul 19, 2017 at 7:02 AM, Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> On Tue, Jul 18 2017, Shawn Pearce jotted:
>>> On Mon, Jul 17, 2017 at 12:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
>>>> Shawn Pearce <spearce@spearce.org> writes:
>>>>> where `time_sec` is the update time in seconds since the epoch.  The
>>>>> `reverse_int32` function inverses the value so lexographical ordering
>>>>> the network byte order time sorts more recent records first:
>>>>>
>>>>>     reverse_int(int32 t) {
>>>>>       return 0xffffffff - t;
>>>>>     }
>>>>
>>>> Is 2038 an issue, or by that time we'd all be retired together with
>>>> this file format and it won't be our problem?
>>>
>>> Based on discussion with Michael Haggerty, this is now an 8 byte field
>>> storing microseconds since the epoch. We should be good through year
>>> 9999.
>>
>> I think this should be s/microseconds/nanoseconds/, not because there's
>> some great need to get better resolution than nanoseconds, but because:
>>
>>  a) We already have WIP code (bp/fsmonitor) that's storing 64 bit
>>     nanoseconds since the epoch, albeit for the index, not for refs.
>>
>>  b) There are several filesystems that have nanosecond resolution now,
>>     and it's likely more will start using that.
>
> The time in a reflog and the time returned by lstat(2) to detect dirty
> files in the working tree are unrelated. Of course we want the
> dircache to be reflecting the highest precision available from lstat,
> to reduce the number of files that must be content hashed for racily
> clean detection. So if a filesystem is using nanoseconds, dircache
> maybe should support it.
>
>> Thus:
>>
>>  x) If you use such a filesystem you'll lose time resolution with this
>>     ref backend v.s. storing them on disk, which isn't itself a big
>>     deal, but more importantly you lose 1=1 time mapping as you
>>     transition and convert between the two.
>
> No, you won't. The reflog today ($GIT_DIR/logs) is storing second
> precision in the log record. What precision the filesystem is using as
> an mtime is irrelevant.

To this & the point above: Sorry about being unclear, I'm talking about
the mtime on the modified loose ref. This format proposes to replace
both loose & packed refs, does it not? The reflog time is not the only
place were we store the mtime of a ref. On my local ext4:

    $ tail -n 1 .git/logs/refs/heads/master
    <sha1> <sha1> Ævar Arnfjörð Bjarmason <avarab@gmail.com> 1500852355 +0200   commit: test
    $ perl -wE 'say ~~localtime shift' 1500852355
    Mon Jul 24 01:25:55 2017
    $ stat -c %y .git/logs/refs/heads/master
    2017-07-24 01:25:55.531379799 +0200

Of course you lose this information as soon as you "git pack-refs", but
it's there now & implicitly part of our current FS-backed on-disk
format.

So what I meant by "x" is that if to test this new reftable backend you
write a "git pack-reftable" you won't be able to 1=1 map it to the
mtimes you have on the fs showing when the ref was updated, but I see
now that you were perhaps never intending to use the more accurate FS
time at all for the loose refs, but just use the second resolution
reflog data.

> Further, microsecond is sufficient resolution for reflog data. From my
> benchmarking just reading a reference from a very hot reftable costs
> ~20.2 usec. Any update of a reference requires a read-compare-modify
> cycle, and so updates aren't going to be more frequent than 20 usec.

Right, I'm not arguing that it isn't sufficient, just that it's
introducing a needless variation by adding a third timestamp resolution
to git.

Even if it's not the same logical area in git (dir management v.s. ref
management) code to e.g. pretty format timestamps of sec/usec/nsec
resolution would tend to get shared, so we'd end up with 3 variants of
those instead of 2.

That's of course trivial, but so would be just deciding that ~500 years
of future proofing is good enough without any extra storage size for
those 64 bits and doing away with 1/3.

Just standardizing that makes more sense than picking the exact right
time resolution for every use case IMO. Otherwise we'll come up with
some other thingy in the future that just needs e.g. millisecond in its
format, and then end up with 4 variants....

I also see from "Update transactions" that unlike the current loose
backend the reftable backend wouldn't support multiple writers on
multiple machines (think NFS-mounted git master) updating unrelated
refs, which would break this usec assumption (but which holds due to the
locking involved in the new backend).

>>  y) Our own code will need to juggle second resolution epochs
>>     (traditional FSs, any 32bit epoch format), microseconds (this
>>     proposal), and nanoseconds (new FSs, bp/fsmonitor) internally in
>>     various places.
>
> But these are also unrelated areas. IMHO, the nanosecond stuff should
> be confined to the dircache management code and working tree
> comparison code, and not be leaking out of there. Commit objects are
> still recorded with second precision, and that isn't going to change.
>
> Therefore I decided to stick with microseconds in the reftable v3
> draft that I posted on July 22nd.