Millisecond precision in timestamps?

Millisecond precision in timestamps?

[Eric S. Raymond]

Because I do a lot of work on repository conversion tools, I've had
to learn a lot of detail about ontological mismatches between
version-control systems - especially places where you lose metadata
moving between them.

In general, git metadata can carry forward almost all the metadata in
a Subversion repository.  Among the handful of minor exceptions (empty
directories, flow structure, certain kinds of mergeinfos) there is one
that stands out because it seems to be an implementation detail rather
than a consequence of fundamentally different design decisions.

I refer to the one-second precision of git timestamps.  Subversion
stores its commit and property-change timestamps to microsecond
precision; conversion tools have to throw the subsecond part of
this information away.

Has going to timestamps with the full precision of the system clock
been considered and rejected, or am I the first to bring this up?

If I were to write refactoring patches that treated "timestamp" as
an ADT, with a view towards hiding the difference between int and
float timestamps and eventually experimenting with float ones, 
would they be accepted?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Every Communist must grasp the truth, 'Political power grows out of
the barrel of a gun.'
        -- Mao Tse-tung, 1938, inadvertently endorsing the Second Amendment.

Re: Millisecond precision in timestamps?

[Shawn Pearce]

On Tue, Nov 27, 2012 at 12:48 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Because I do a lot of work on repository conversion tools, I've had
> to learn a lot of detail about ontological mismatches between
> version-control systems - especially places where you lose metadata
> moving between them.
>
> In general, git metadata can carry forward almost all the metadata in
> a Subversion repository.  Among the handful of minor exceptions (empty
> directories, flow structure, certain kinds of mergeinfos) there is one
> that stands out because it seems to be an implementation detail rather
> than a consequence of fundamentally different design decisions.
>
> I refer to the one-second precision of git timestamps.  Subversion
> stores its commit and property-change timestamps to microsecond
> precision; conversion tools have to throw the subsecond part of
> this information away.
>
> Has going to timestamps with the full precision of the system clock
> been considered and rejected, or am I the first to bring this up?
>
> If I were to write refactoring patches that treated "timestamp" as
> an ADT, with a view towards hiding the difference between int and
> float timestamps and eventually experimenting with float ones,
> would they be accepted?

JGit would fortunately ignore a floating point timestamp specification
if given in a commit, but I don't know about other Git
implementations... like say git. :-)

RE: Millisecond precision in timestamps?

[Pyeron, Jason J CTR (US)]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

> -----Original Message-----
> From: Eric S. Raymond
> Sent: Tuesday, November 27, 2012 3:48 PM
> 
> Because I do a lot of work on repository conversion tools, I've had
> to learn a lot of detail about ontological mismatches between
> version-control systems - especially places where you lose metadata
> moving between them.
> 
> In general, git metadata can carry forward almost all the metadata in
> a Subversion repository.  Among the handful of minor exceptions (empty
> directories, flow structure, certain kinds of mergeinfos) there is one
> that stands out because it seems to be an implementation detail rather
> than a consequence of fundamentally different design decisions.
> 
> I refer to the one-second precision of git timestamps.  Subversion
> stores its commit and property-change timestamps to microsecond
> precision; conversion tools have to throw the subsecond part of
> this information away.
> 
> Has going to timestamps with the full precision of the system clock
> been considered and rejected, or am I the first to bring this up?
> 
> If I were to write refactoring patches that treated "timestamp" as
> an ADT, with a view towards hiding the difference between int and
> float timestamps and eventually experimenting with float ones,

Do you really mean floating point numbers with approximate imprecise values?



[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5615 bytes --]

Re: Millisecond precision in timestamps?

[Junio C Hamano]

Shawn Pearce <spearce@spearce.org> writes:

> JGit would fortunately ignore a floating point timestamp specification
> if given in a commit, but I don't know about other Git
> implementations... like say git. :-)

fsck_ident() in fsck.c rejects anything but " [1-9][0-9]* " after
the author and committer ident (i.e. the timestamp has to be
integral number of seconds since the epoch, not before it, nor
with fractional seconds).

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Junio C Hamano <gitster@pobox.com>:
> fsck_ident() in fsck.c rejects anything but " [1-9][0-9]* " after
> the author and committer ident (i.e. the timestamp has to be
> integral number of seconds since the epoch, not before it, nor
> with fractional seconds).

Is this architecturally significant?  It sounds like another
implementation detail.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Shawn Pearce]

On Tue, Nov 27, 2012 at 3:04 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Junio C Hamano <gitster@pobox.com>:
>> fsck_ident() in fsck.c rejects anything but " [1-9][0-9]* " after
>> the author and committer ident (i.e. the timestamp has to be
>> integral number of seconds since the epoch, not before it, nor
>> with fractional seconds).
>
> Is this architecturally significant?  It sounds like another
> implementation detail.

Well... if we added a fractional seconds to a commit, older versions
of Git will scream loudly and refuse to work with the new commit. That
would create a fork of Git.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Shawn Pearce <spearce@spearce.org>:
> Well... if we added a fractional seconds to a commit, older versions
> of Git will scream loudly and refuse to work with the new commit. That
> would create a fork of Git.

So much for that idea, I guess.  

Unless..I don't know how git's database representations work.  Are they
version-stamped in any way?  If so, some slightly painful hackery would
get around that problem.

I'm being exploratory, here. No proposal to code anything is in the
offing.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[David Lang]

On Tue, 27 Nov 2012, Eric S. Raymond wrote:

> Shawn Pearce <spearce@spearce.org>:
>> Well... if we added a fractional seconds to a commit, older versions
>> of Git will scream loudly and refuse to work with the new commit. That
>> would create a fork of Git.
>
> So much for that idea, I guess.
>
> Unless..I don't know how git's database representations work.  Are they
> version-stamped in any way?  If so, some slightly painful hackery would
> get around that problem.
>
> I'm being exploratory, here. No proposal to code anything is in the
> offing.

Apologies if this was covered earlier in the thread (I missed the beginning)

remember that git is dealing with timestamps generated across different 
machines, and since the times are not assumed to be in sync, let alone to the 
millisecond level, there's not much value to git in that level of presision.

git routinely deals with timestamps that are off by days. If the timestamps are 
within a minute or so, you are in pretty good shape.

David Lang

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Wed, Nov 28, 2012 at 1:12 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Shawn Pearce <spearce@spearce.org>:
>> Well... if we added a fractional seconds to a commit, older versions
>> of Git will scream loudly and refuse to work with the new commit. That
>> would create a fork of Git.
>
> So much for that idea, I guess.
>
> Unless..I don't know how git's database representations work.  Are they
> version-stamped in any way?  If so, some slightly painful hackery would
> get around that problem.

% git cat-file -p HEAD

You'll see exactly how git stores commits. Changing anything in there
must be done carefully.

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Shawn Pearce]

On Tue, Nov 27, 2012 at 4:26 PM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
> On Wed, Nov 28, 2012 at 1:12 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> Shawn Pearce <spearce@spearce.org>:
>>> Well... if we added a fractional seconds to a commit, older versions
>>> of Git will scream loudly and refuse to work with the new commit. That
>>> would create a fork of Git.
>>
>> So much for that idea, I guess.
>>
>> Unless..I don't know how git's database representations work.  Are they
>> version-stamped in any way?  If so, some slightly painful hackery would
>> get around that problem.
>
> % git cat-file -p HEAD
>
> You'll see exactly how git stores commits. Changing anything in there
> must be done carefully.

Apparently there is no room to change in these fields without breaking
compatibility with all current versions of Git. So its not just done
carefully... its deciding to make Git 2.0 that is not compatible with
any Git 1.x release.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> % git cat-file -p HEAD
> 
> You'll see exactly how git stores commits. Changing anything in there
> must be done carefully.

Oh, I've seen *that* before.  Are you telling me the database 
representation is actually textual?
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Jeff King]

On Tue, Nov 27, 2012 at 05:07:34PM -0800, Shawn O. Pearce wrote:

> On Tue, Nov 27, 2012 at 4:26 PM, Felipe Contreras
> <felipe.contreras@gmail.com> wrote:
> > On Wed, Nov 28, 2012 at 1:12 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> >> Shawn Pearce <spearce@spearce.org>:
> >>> Well... if we added a fractional seconds to a commit, older versions
> >>> of Git will scream loudly and refuse to work with the new commit. That
> >>> would create a fork of Git.
> >>
> >> So much for that idea, I guess.
> >>
> >> Unless..I don't know how git's database representations work.  Are they
> >> version-stamped in any way?  If so, some slightly painful hackery would
> >> get around that problem.
> >
> > % git cat-file -p HEAD
> >
> > You'll see exactly how git stores commits. Changing anything in there
> > must be done carefully.
> 
> Apparently there is no room to change in these fields without breaking
> compatibility with all current versions of Git. So its not just done
> carefully... its deciding to make Git 2.0 that is not compatible with
> any Git 1.x release.

There is room for new headers, and older versions of git will ignore
them. You could add a new "committer-timestamp" field that elaborates on
the timestamp included on the committer line. Newer versions of git
would respect it, and older versions would fall back to using the
committer timestamp.

But I really wonder if anybody actually cares about adding sub-second
timestamp support, or if it is merely "because SVN has it".

-Peff

RE: Millisecond precision in timestamps?

[Jason Pyeron]

> -----Original Message-----
> From: Jeff King
> Sent: Tuesday, November 27, 2012 20:18
> 
> On Tue, Nov 27, 2012 at 05:07:34PM -0800, Shawn O. Pearce wrote:
> 
> > On Tue, Nov 27, 2012 at 4:26 PM, Felipe Contreras 
> > <felipe.contreras@gmail.com> wrote:
> > > On Wed, Nov 28, 2012 at 1:12 AM, Eric S. Raymond 
> <esr@thyrsus.com> wrote:
> > >> Shawn Pearce <spearce@spearce.org>:
> > >>> Well... if we added a fractional seconds to a commit, older 
> > >>> versions of Git will scream loudly and refuse to work 
> with the new 
> > >>> commit. That would create a fork of Git.
> > >>
> > >> So much for that idea, I guess.
> > >>
> > >> Unless..I don't know how git's database representations 
> work.  Are 
> > >> they version-stamped in any way?  If so, some slightly painful 
> > >> hackery would get around that problem.
> > >
> > > % git cat-file -p HEAD
> > >
> > > You'll see exactly how git stores commits. Changing anything in 
> > > there must be done carefully.
> > 
> > Apparently there is no room to change in these fields 
> without breaking 
> > compatibility with all current versions of Git. So its not 
> just done 
> > carefully... its deciding to make Git 2.0 that is not 
> compatible with 
> > any Git 1.x release.
> 
> There is room for new headers, and older versions of git will 
> ignore them. You could add a new "committer-timestamp" field 
> that elaborates on the timestamp included on the committer 
> line. Newer versions of git would respect it, and older 
> versions would fall back to using the committer timestamp.

Suggestion add a ms offset field. Ex:

jpyeron@black /projects/git/git
$ git cat-file -p HEAD
tree 1e24acfbfcc05aa57e8cb2cfe3ffe01cb100961d
parent e98fa647aa5673cc95b6e9be1fdc13c0afa2cb37
author Junio C Hamano <gitster@pobox.com> 1350495361 -0700
committer Junio C Hamano <gitster@pobox.com> 1350495402 -0700
mstimestamps author 0 committer 1234

Git 1.7.12.4

Signed-off-by: Junio C Hamano <gitster@pobox.com>


> 
> But I really wonder if anybody actually cares about adding 
> sub-second timestamp support, or if it is merely "because SVN has it".

Not because subversion has it but because date != git(precisedate) and some
automation using git in a larger enterprise workflow may assume that date
1354065991.1234 going in should be the same when queried.

-Jason



--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Wed, Nov 28, 2012 at 2:11 AM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Felipe Contreras <felipe.contreras@gmail.com>:
>> % git cat-file -p HEAD
>>
>> You'll see exactly how git stores commits. Changing anything in there
>> must be done carefully.
>
> Oh, I've seen *that* before.  Are you telling me the database
> representation is actually textual?

http://git-scm.com/book/ch9-2.html

---
% ruby -e "require 'zlib'; puts
Zlib::Inflate.inflate(File.read('.git/objects/55/47f28602c9b07f69dfa4685945f71f660e8b25'))"

commit 382tree a14fe16bb9cb7d949d9eb7570bf56968b209f4a2
parent c3640f09011d969ac85753c8f0114ce9b9c86603
author Felipe Contreras <felipe.contreras@gmail.com> 1352767480 +0100
committer Felipe Contreras <felipe.contreras@gmail.com> 1354064281 +0100

remote-bzr: detect local repositories

So we don't create a clone  unnecessarily.

Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---

Yes, that's what I'm telling you.

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Wed, Nov 28, 2012 at 2:17 AM, Jeff King <peff@peff.net> wrote:

> But I really wonder if anybody actually cares about adding sub-second
> timestamp support, or if it is merely "because SVN has it".

I agree, I don't see any point.

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Junio C Hamano]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Junio C Hamano <gitster@pobox.com>:
>> fsck_ident() in fsck.c rejects anything but " [1-9][0-9]* " after
>> the author and committer ident (i.e. the timestamp has to be
>> integral number of seconds since the epoch, not before it, nor
>> with fractional seconds).
>
> Is this architecturally significant?  It sounds like another
> implementation detail.

No.

If you create a commit object that violatse it and have 47 million
existing users pull such a history, they not be able to use such a
history with the version of Git they have.  Don't go there.

As somebody else mentioned, in distributed environment millisecond
timestamps won't have much meaning, so it looks like a very low
priority to me from Git's perspective.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Jeff King <peff@peff.net>:
> But I really wonder if anybody actually cares about adding sub-second
> timestamp support, or if it is merely "because SVN has it".

There's actually one possible other reason to care.  1-second granularity 
isn't quite fine enough to guarantee that a (committer, timestamp)
pair is a unique key.  1 microsecond granularity would be.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Jeff King]

On Tue, Nov 27, 2012 at 10:23:37PM -0500, Eric S. Raymond wrote:

> Jeff King <peff@peff.net>:
> > But I really wonder if anybody actually cares about adding sub-second
> > timestamp support, or if it is merely "because SVN has it".
> 
> There's actually one possible other reason to care.  1-second granularity 
> isn't quite fine enough to guarantee that a (committer, timestamp)
> pair is a unique key.  1 microsecond granularity would be.

You can't guarantee that such a pair is unique, anyway, due to clock
skew.

A much more compelling argument to me would be that you are doing some
bidirectional magic between git and svn, and you want to make make sure
that an svn->git->svn translation will result in the exact same bytes.
Then the argument is still "because SVN has it", but at least it is "and
we interoperate with it" and not simply chasing a cool but useless
feature.

-Peff

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Wed, Nov 28, 2012 at 4:30 AM, Jeff King <peff@peff.net> wrote:
> On Tue, Nov 27, 2012 at 10:23:37PM -0500, Eric S. Raymond wrote:
>
>> Jeff King <peff@peff.net>:
>> > But I really wonder if anybody actually cares about adding sub-second
>> > timestamp support, or if it is merely "because SVN has it".
>>
>> There's actually one possible other reason to care.  1-second granularity
>> isn't quite fine enough to guarantee that a (committer, timestamp)
>> pair is a unique key.  1 microsecond granularity would be.
>
> You can't guarantee that such a pair is unique, anyway, due to clock
> skew.
>
> A much more compelling argument to me would be that you are doing some
> bidirectional magic between git and svn, and you want to make make sure
> that an svn->git->svn translation will result in the exact same bytes.
> Then the argument is still "because SVN has it", but at least it is "and
> we interoperate with it" and not simply chasing a cool but useless
> feature.

But the same can be said of mercurial and bzr. This can be solved
attaching some external SCM information in notes, and somehow make
fast-export throw that info along with the commit.

For now the solution has been to append the extra information into the
commit message, which is ugly and hacky, but it works.

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Jeff King <peff@peff.net>:
> A much more compelling argument to me would be that you are doing some
> bidirectional magic between git and svn, and you want to make make sure
> that an svn->git->svn translation will result in the exact same bytes.
> Then the argument is still "because SVN has it", but at least it is "and
> we interoperate with it" and not simply chasing a cool but useless
> feature.

Er, well, that *is* in fact the exact reason I want it.

I didn't put it exactly that way because I didn't expect anyone here
to particularly care about round-tripping like that.  But remember 
that I do a lot of stuff with repo surgery and conversion tools.

As a matter of fact (and this list is the first to hear about it) 
I'm working on code right now that massages a git import stream
into a Subversion dumpfile.  Soon, unless I hit a blocker I'm
not expecting, I'll ship it.

Yes, there will be serious limitations and unavoidable metadata loss.
But in every case *except timestamps* that loss is Subversion's fault
for having a weak ontology.  Timestamps are the one place git doesn't 
hold up its end.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Jeff King]

On Tue, Nov 27, 2012 at 10:47:00PM -0500, Eric S. Raymond wrote:

> Jeff King <peff@peff.net>:
> > A much more compelling argument to me would be that you are doing some
> > bidirectional magic between git and svn, and you want to make make sure
> > that an svn->git->svn translation will result in the exact same bytes.
> > Then the argument is still "because SVN has it", but at least it is "and
> > we interoperate with it" and not simply chasing a cool but useless
> > feature.
> 
> Er, well, that *is* in fact the exact reason I want it.
> 
> I didn't put it exactly that way because I didn't expect anyone here
> to particularly care about round-tripping like that.  But remember 
> that I do a lot of stuff with repo surgery and conversion tools.

If that's what we really care about, then that opens up the
possibilities for how we store the data. An extension header in the
object might be convenient, but it opens up a lot of questions about
what git will do with such a header (e.g., would it be part of git-log
output?).

Felipe suggested using git-notes to add the metadata, which I think is a
reasonable first step. The git side of the code is already written, and
the concept is nicely modularized away from the core of git. Nobody has
to care about it but your importer, and anybody who wants to query it[1]
can do so by requesting the note.

-Peff

[1] And you do not have to limit yourself to timestamps, if there is
    other metadata about each commit you end up wanting to store for a
    clean bi-directional conversion.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Jeff King <peff@peff.net>:
> Felipe suggested using git-notes to add the metadata, which I think is a
> reasonable first step. The git side of the code is already written, and
> the concept is nicely modularized away from the core of git. Nobody has
> to care about it but your importer, and anybody who wants to query it[1]
> can do so by requesting the note.
> 
> -Peff
> 
> [1] And you do not have to limit yourself to timestamps, if there is
>     other metadata about each commit you end up wanting to store for a
>     clean bi-directional conversion.

I have actually wanted something like this quite badly.  Not so much
for timestamps (though that would be nice), but it would be useful if
each commit could carry a fossil-ID attribute that points at the
Subversion commit it was derived from.

I've tried to make notes work for this, but couldn't beat it into
doing what I was after.  Shawn, is there a way that the import stream
syntax can declare a note with in-line data attached to the commit where
it's declared?  

I tried just using the mark of the current commit, but git throws an error
because it thinks that mark is not yet declared when the note fileop
is parsed.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

> There is room for new headers, and older versions of git will ignore
> them. You could add a new "committer-timestamp" field that elaborates on
> the timestamp included on the committer line. Newer versions of git
> would respect it, and older versions would fall back to using the
> committer timestamp.
>
> But I really wonder if anybody actually cares about adding sub-second
> timestamp support, or if it is merely "because SVN has it".

Roundtrip conversions may benefit from sub-second timestamps, but
personally I think negative timestamps are more interesting and of
practical use.  Prehistoric projects need them even if they intend
to switch to Git, never to go back to their original tarballs and
collection of RCS ,v files.

And if we were to add "committer-timestamp" and friends to support
negative timestamps anyway (because older tools will not support
them), supporting sub-second part might be something we want to
think about at the same time.

We would however need to be extra careful.  How should we express
half-second past Tue Nov 27 23:24:16 2012 (US/Pacific)?  Would we
spell it 1354087456.5?  1354087456.500?  Would we require decimal
representation of floating point numbers to be normalized in some
way (e.g. minimum number of digits without losing precision)?  The
same timestamp needs to be expressed the same way, or we will end up
with different commit objects, which defeats the whole purpose of
introducing subsecond timestamps to support round-trip conversions.

If we were to use a separate "subsecond" fields, another thing we
need to be careful about is the order of these extra fields, exactly
for the same reason.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Junio C Hamano <gitster@pobox.com>:
> Roundtrip conversions may benefit from sub-second timestamps, but
> personally I think negative timestamps are more interesting and of
> practical use. 

You mean, as in times before the Unix epoch 1970-01-01T00:00:00Z?  

Interesting.  I hadn't thought of that.  I've never seen a software
project under version control with bits that old, which is significant
because I've probably done more digging into ancient software than
anybody other than a specialist historian or two.

They would have to have been restrospective dates from the get-go.
SCCS wasn't built until 1972.

> And if we were to add "committer-timestamp" and friends to support
> negative timestamps anyway (because older tools will not support
> them), supporting sub-second part might be something we want to
> think about at the same time.

That seems eminently reasonable.

> We would however need to be extra careful.  How should we express
> half-second past Tue Nov 27 23:24:16 2012 (US/Pacific)?  Would we
> spell it 1354087456.5?  1354087456.500?  Would we require decimal
> representation of floating point numbers to be normalized in some
> way (e.g. minimum number of digits without losing precision)?  The
> same timestamp needs to be expressed the same way, or we will end up
> with different commit objects, which defeats the whole purpose of
> introducing subsecond timestamps to support round-trip conversions.
> 
> If we were to use a separate "subsecond" fields, another thing we
> need to be careful about is the order of these extra fields, exactly
> for the same reason.

I think minimum number of digits without losing precision is about the
only alternative that is future-proof - I was going to suggest it for
that reason.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[David Aguilar]

On Tue, Nov 27, 2012 at 11:58 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
> Junio C Hamano <gitster@pobox.com>:
>> Roundtrip conversions may benefit from sub-second timestamps, but
>> personally I think negative timestamps are more interesting and of
>> practical use.
>
> You mean, as in times before the Unix epoch 1970-01-01T00:00:00Z?
>
> Interesting.  I hadn't thought of that.  I've never seen a software
> project under version control with bits that old, which is significant
> because I've probably done more digging into ancient software than
> anybody other than a specialist historian or two.

One example I've heard is someone wanting to throw the history
of a country's laws into git so they can diff them.
-- 
David

Re: Millisecond precision in timestamps?

[Thomas Berg]

On Wed, Nov 28, 2012 at 8:29 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jeff King <peff@peff.net> writes:
>
>> There is room for new headers, and older versions of git will ignore
>> them. You could add a new "committer-timestamp" field that elaborates on
>> the timestamp included on the committer line. Newer versions of git
>> would respect it, and older versions would fall back to using the
>> committer timestamp.
>>
>> But I really wonder if anybody actually cares about adding sub-second
>> timestamp support, or if it is merely "because SVN has it".
>
> Roundtrip conversions may benefit from sub-second timestamps, but
> personally I think negative timestamps are more interesting and of
> practical use.  Prehistoric projects need them even if they intend
> to switch to Git, never to go back to their original tarballs and
> collection of RCS ,v files.

If roundtripping to other version control systems is an argument,
adding sub-second timestamps could potentially create as many problems
as it solves. For example, I've been using the hg-git bridge, and it
supports roundtripping between git and mercurial today (for most repos
I've tried anyway). I may have missed something, but this could imply
that mercurial doesn't care about sub-second timestamps either. If so,
and if git suddenly were to record it, it would no longer be as
straight forward to represent git history in hg.

In my opinion it would be a shame to sacrifice this compatibility just
to reduce the distance to svn, which is much larger anyway.

- Thomas

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Wed, Nov 28, 2012 at 9:19 AM, Thomas Berg <merlin66b@gmail.com> wrote:
> On Wed, Nov 28, 2012 at 8:29 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Jeff King <peff@peff.net> writes:
>>
>>> There is room for new headers, and older versions of git will ignore
>>> them. You could add a new "committer-timestamp" field that elaborates on
>>> the timestamp included on the committer line. Newer versions of git
>>> would respect it, and older versions would fall back to using the
>>> committer timestamp.
>>>
>>> But I really wonder if anybody actually cares about adding sub-second
>>> timestamp support, or if it is merely "because SVN has it".
>>
>> Roundtrip conversions may benefit from sub-second timestamps, but
>> personally I think negative timestamps are more interesting and of
>> practical use.  Prehistoric projects need them even if they intend
>> to switch to Git, never to go back to their original tarballs and
>> collection of RCS ,v files.
>
> If roundtripping to other version control systems is an argument,
> adding sub-second timestamps could potentially create as many problems
> as it solves. For example, I've been using the hg-git bridge, and it
> supports roundtripping between git and mercurial today (for most repos
> I've tried anyway). I may have missed something, but this could imply
> that mercurial doesn't care about sub-second timestamps either. If so,
> and if git suddenly were to record it, it would no longer be as
> straight forward to represent git history in hg.

I'm not entirely sure. The API seems to return a float for the time,
but at least as far I can see, it never has any decimals anyway.

But it doesn't really matter, mercurial doesn't have a committer
information either. This is solved by tools like hg-git by storing the
information in an 'extra' field, which can store anything.

Unfortunately git doesn't have a similar field, so people have been
using the commit message to store extra information.

Either way, I don't see the point in changing git's commit format for
external tools. The git-notes functionality works just fine for that,
it just needs to be attached in the relevant places, like 'git
fast-export'.

BTW. Have you checked git's native support for hg?[1]

Cheers.

[1] http://felipec.wordpress.com/2012/11/13/git-remote-hg-bzr-2/

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Thomas Berg]

On Wed, Nov 28, 2012 at 9:44 AM, Felipe Contreras
<felipe.contreras@gmail.com> wrote:
>> If roundtripping to other version control systems is an argument,
>> adding sub-second timestamps could potentially create as many problems
>> as it solves. For example, I've been using the hg-git bridge, and it
>> supports roundtripping between git and mercurial today (for most repos
>> I've tried anyway). I may have missed something, but this could imply
>> that mercurial doesn't care about sub-second timestamps either. If so,
>> and if git suddenly were to record it, it would no longer be as
>> straight forward to represent git history in hg.
>
> I'm not entirely sure. The API seems to return a float for the time,
> but at least as far I can see, it never has any decimals anyway.
>
> But it doesn't really matter, mercurial doesn't have a committer
> information either. This is solved by tools like hg-git by storing the
> information in an 'extra' field, which can store anything.

True. For many commits though, hg-git doesn't need any extra fields,
as far as I've seen. A timestamp incompatibility would require extra
info on every commit.

> Either way, I don't see the point in changing git's commit format for
> external tools. The git-notes functionality works just fine for that,
> it just needs to be attached in the relevant places, like 'git
> fast-export'.

I agree. Even encoding info in the commit message works fine, and
git-svn already does that.

> BTW. Have you checked git's native support for hg?[1]

That's been added after I played with this last, I'll have a look.

Cheers,
Thomas

Re: Millisecond precision in timestamps?

[Andreas Ericsson]

On 11/28/2012 08:29 AM, Junio C Hamano wrote:
> Jeff King <peff@peff.net> writes:
> 
>> There is room for new headers, and older versions of git will ignore
>> them. You could add a new "committer-timestamp" field that elaborates on
>> the timestamp included on the committer line. Newer versions of git
>> would respect it, and older versions would fall back to using the
>> committer timestamp.
>>
>> But I really wonder if anybody actually cares about adding sub-second
>> timestamp support, or if it is merely "because SVN has it".
> 
> Roundtrip conversions may benefit from sub-second timestamps, but
> personally I think negative timestamps are more interesting and of
> practical use.  Prehistoric projects need them even if they intend
> to switch to Git, never to go back to their original tarballs and
> collection of RCS ,v files.
> 
> And if we were to add "committer-timestamp" and friends to support
> negative timestamps anyway (because older tools will not support
> them), supporting sub-second part might be something we want to
> think about at the same time.
> 
> We would however need to be extra careful.  How should we express
> half-second past Tue Nov 27 23:24:16 2012 (US/Pacific)?  Would we
> spell it 1354087456.5?  1354087456.500?  Would we require decimal
> representation of floating point numbers to be normalized in some
> way (e.g. minimum number of digits without losing precision)?  The
> same timestamp needs to be expressed the same way, or we will end up
> with different commit objects, which defeats the whole purpose of
> introducing subsecond timestamps to support round-trip conversions.
> 
> If we were to use a separate "subsecond" fields, another thing we
> need to be careful about is the order of these extra fields, exactly
> for the same reason.
> 

If we're going to support pre-epoch timestamps, we'll have to do that
for 2.0 anyway, since we'll otherwise have two conflicting dates in
the commit object.

Adding support for parsing them now and start writing them in 2.0
would make sense.

In that case, we'd have to print timestamps as
printf("%lu.%06lu", tv.tv_sec, tv.tv_usec);

I'm unsure how useful it is to support pre-epoch dates though. It's
hard to find software where anyone really cares about the code from
43 years ago with anything but historical interest, and for those
who take the museum road, I'm betting it's more interesting to see
how people worked back then than it is to see what they wrote.

Aside from that, it would be trivial to support museum style history
viewing with a special flag that treats the timestamps as minutes
since 1900-01-01 or some such, giving us plenty of time before even
the first punch-card was invented. It wouldn't be much harder to
let the user specify the timeunit and the start-point either, and
then we could store the history of carbon-based lifeforms on earth
in git.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Millisecond precision in timestamps?

[Andreas Ericsson]

On 11/28/2012 09:04 AM, David Aguilar wrote:
> On Tue, Nov 27, 2012 at 11:58 PM, Eric S. Raymond <esr@thyrsus.com> wrote:
>> Junio C Hamano <gitster@pobox.com>:
>>> Roundtrip conversions may benefit from sub-second timestamps, but
>>> personally I think negative timestamps are more interesting and of
>>> practical use.
>>
>> You mean, as in times before the Unix epoch 1970-01-01T00:00:00Z?
>>
>> Interesting.  I hadn't thought of that.  I've never seen a software
>> project under version control with bits that old, which is significant
>> because I've probably done more digging into ancient software than
>> anybody other than a specialist historian or two.
> 
> One example I've heard is someone wanting to throw the history
> of a country's laws into git so they can diff them.
> 

That'll get tricky if you try it in Sweden. Our oldest written law
dates back to 1281. Quite fun reading. Apparently it was against the
law to shoot your slaves with stone arrows back then.

See my other proposal for how this could be done, which would only
affect the output layer (and some care would have to be taken with
the input, naturally).

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231

Considering the successes of the wars on alcohol, poverty, drugs and
terror, I think we should give some serious thought to declaring war
on peace.

Re: Millisecond precision in timestamps?

[Junio C Hamano]

Thomas Berg <merlin66b@gmail.com> writes:

> If roundtripping to other version control systems is an argument,
> adding sub-second timestamps could potentially create as many problems
> as it solves. For example, I've been using the hg-git bridge, and it
> supports roundtripping between git and mercurial today (for most repos
> I've tried anyway). I may have missed something,...

What I left unsaid was that the use of extra subsecond resolution is
optional.  I do not see any reason for *us* to create commits with
subsecond resolution when we are writing native commits.  Only when
the end users and/or import tools tell us to.  If you assume all
foreign SCM you care about have at least one second resolution, you
would be fine.

Having said all that, given that this, if implemented, would not be
used by us but only for recording other people's times, and that the
set of meta information we record in our history will never be
superset of everybody else's anyway, I do not see much point in
supporting subsecond timestamps in the first place.

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Steven Michalske <smichalske@gmail.com>:
> Would having arbitrary key value pairs be useful in the git data
> model?  We could have ones that affect the sha1 and others that are
> transparent.

My tools would have several uses for these.

bzr's implementation of import streams has a commit-propperties extension.
reposurgeon can read, display.. and manipulate these key/value pairs.
I do wish they were in core git.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Junio C Hamano]

Steven Michalske <smichalske@gmail.com> writes:

> Would having arbitrary key value pairs be useful in the git data
> model?

My answer to the question is that it is harmful to the data model,
but the benefit of going against the data model _may_ outweigh the
downside.  It is all relative.

The first of very small number of principles of the git data model
is that the object name is derived solely from the contents, hence
we can tell two different things apart with object names without
looking at object contents.

This is actively broken by adding "junk" fields left and right.
Adding arbitrary pieces of data that are optional (and largely
ignored by core operations) means you can record objects with
essentially the same contents under different object names, so
object names no longer help us telling two moral-equivalent objects
apart.

But "if two objects have different names, they are not the same"
does not have to be the only and the absolute truth in all contexts;
the world is not so black and white.  Depending on the application
and the context, you may want to treat two things that are not the
same as equivalents.

For example, at the blob level, two blob objects that store the same
text (say, one original and the other typed in double-space) would
be different objects and have different object names, but you may
want to treat them as "equivalents" (not same but interchangeable),
by applying textconv filter to normalize their contents when
comparing them.  We still keep the "two objects with different names
are different" principle, but at the same time, allow users to treat
them as equivalent in specific contexts.

Introducing a hack to exclude selective "junk" fields from hashing
done for object name computation is not a solution and is out of the
question, but that does not necessarily mean that commit objects
should never be extended with new types of header fields.  When a
commit object is made with a "junk" field, it will have a name that
is different from the one it would get without the "junk" field, but
the benefit of the ability to store extra data _may_ outweigh the
downside of having to always compare the contents of two objects
with different names to find out that they are different but
equivalent.

Re: Millisecond precision in timestamps?

[Felipe Contreras]

On Thu, Nov 29, 2012 at 8:11 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Steven Michalske <smichalske@gmail.com> writes:
>
>> Would having arbitrary key value pairs be useful in the git data
>> model?
>
> My answer to the question is that it is harmful to the data model,
> but the benefit of going against the data model _may_ outweigh the
> downside.  It is all relative.

If git doesn't provide the capability, people will keep using the
commit message to store that extra information, which I would think is
even more harmful. An standard 'commit-extra' note or something would
help deal with that.

Cheers.

-- 
Felipe Contreras

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Felipe Contreras <felipe.contreras@gmail.com>:
> On Thu, Nov 29, 2012 at 8:11 AM, Junio C Hamano <gitster@pobox.com> wrote:
> > Steven Michalske <smichalske@gmail.com> writes:
> >
> >> Would having arbitrary key value pairs be useful in the git data
> >> model?
> >
> > My answer to the question is that it is harmful to the data model,
> > but the benefit of going against the data model _may_ outweigh the
> > downside.  It is all relative.
> 
> If git doesn't provide the capability, people will keep using the
> commit message to store that extra information, which I would think is
> even more harmful. An standard 'commit-extra' note or something would
> help deal with that.

Agreed.  

My use case for a capability like this is one of the more common ones.
I want to be able to store a fossil commit-ID inherited from another
VCS outside the commit comment.  The absence of a key/value store forces
me into some annoying kludges.
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Junio C Hamano]

"Eric S. Raymond" <esr@thyrsus.com> writes:

> Felipe Contreras <felipe.contreras@gmail.com>:
>> On Thu, Nov 29, 2012 at 8:11 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> > Steven Michalske <smichalske@gmail.com> writes:
>> >
>> >> Would having arbitrary key value pairs be useful in the git data
>> >> model?
>> >
>> > My answer to the question is that it is harmful to the data model,
>> > but the benefit of going against the data model _may_ outweigh the
>> > downside.  It is all relative.
>
> My use case for a capability like this is one of the more common ones.
> I want to be able to store a fossil commit-ID inherited from another
> VCS outside the commit comment.

That is exactly why I said it is all relative.  If it helps your
application, you can weigh the pros-and-cons yourself and choose to
throw "junk" extended header fields in the commit objects you
create, using hash-object (or commit-tree).  You can read it out
using cat-file and do whatever you want to do with it, and modern
Git (v1.5.0 was from early 2007) and tools that are designed to work
with Git know to ignore such "junk" field.

> The absence of a key/value store forces me into some annoying
> kludges.

Do not do annoying kludge, then.  Come up with a method to encode
your list of (key,value) tuples into a single string, throw a
custom extra header after all the standard header fields in, perhaps
like this:

    tree 0664b9c82d87269b335ff78f32d0e4a504f58cfc
    author A U Thor <author@example.xz> 1355999999 +0900
    committer C O Mitter <committer@example.xz> 1355999999 +0900
    encoding iso-2022-jp
    reposurgeon-metadata your-serialized-list-of-key-value-tuples
     second-line-of-such-serialization
     third-line-of-such-serialization

    My first commit

    Signed-off-by: A U Thor <author@example.xz>
    Signed-off-by: C O Mitter <committer@example.xz>

Re: Millisecond precision in timestamps?

[Eric S. Raymond]

Junio C Hamano <gitster@pobox.com>:
> That is exactly why I said it is all relative.  If it helps your
> application, you can weigh the pros-and-cons yourself and choose to
> throw "junk" extended header fields in the commit objects you
> create, using hash-object (or commit-tree).  You can read it out
> using cat-file and do whatever you want to do with it, and modern
> Git (v1.5.0 was from early 2007) and tools that are designed to work
> with Git know to ignore such "junk" field.

A good start.  But remember that reposurgeon's entire interface to the
git object level is through fast-export/fast-import.  I need import-
stream syntax for these.

bzr's syntax would do:

-------------------------------------------
mark :1
committer Eric S. Raymond <esr@thyrsus.com> 1289147634 -0500
data 14
First commit.

property branch-nick 12 bzr-testrepo
M 644 inline README
data 41
This is a test file in a dummy bzr repo.
-------------------------------------------

If we actually care about keys being full utf-8 with embedded whitespace
it should look more like this:

-------------------------------------------
mark :1
committer Eric S. Raymond <esr@thyrsus.com> 1289147634 -0500
data 14
First commit.

property 11
branch-nick
propval 12 
bzr-testrepo
M 644 inline README
data 41
This is a test file in a dummy bzr repo.
-------------------------------------------
-- 
		<a href="http://www.catb.org/~esr/">Eric S. Raymond</a>

Re: Millisecond precision in timestamps?

[Phil Hord]

On Wed, Nov 28, 2012 at 2:29 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jeff King <peff@peff.net> writes:
>
>> There is room for new headers, and older versions of git will ignore
>> them. You could add a new "committer-timestamp" field that elaborates on
>> the timestamp included on the committer line. Newer versions of git
>> would respect it, and older versions would fall back to using the
>> committer timestamp.
>>
>> But I really wonder if anybody actually cares about adding sub-second
>> timestamp support, or if it is merely "because SVN has it".
>
> Roundtrip conversions may benefit from sub-second timestamps, but
> personally I think negative timestamps are more interesting and of
> practical use.  Prehistoric projects need them even if they intend
> to switch to Git, never to go back to their original tarballs and
> collection of RCS ,v files.
>
> And if we were to add "committer-timestamp" and friends to support
> negative timestamps anyway (because older tools will not support
> them), supporting sub-second part might be something we want to
> think about at the same time.

Posix-time is signed, but I suppose the git tools do not expect/allow
a '-' character in the stream.  Has git considered the year-2038
problem?

No hurry...

Phil

Re: Millisecond precision in timestamps?

[Jeff King]

On Thu, Nov 29, 2012 at 02:14:40PM -0500, Phil Hord wrote:

> > And if we were to add "committer-timestamp" and friends to support
> > negative timestamps anyway (because older tools will not support
> > them), supporting sub-second part might be something we want to
> > think about at the same time.
> 
> Posix-time is signed, but I suppose the git tools do not expect/allow
> a '-' character in the stream.  Has git considered the year-2038
> problem?

Yes. The timestamp is in base-10 ASCII, so there is no Y2038 problem in
the data format (it is up to the implementation to read it into a
sufficiently large time_t internally, of course[1]).

But negative timestamps are a different story. We use "unsigned long"
internally for timestamps, and fsck will complain about it.

-Peff

[1] We use "unsigned long", which means we are Y2038-fine on I32/LP64
    systems, but not on 32-bit or IL32/LLP64 systems. I do not use
    Windows, but my understanding is that LLP64 is the norm there, so it
    would eventually be a problem. But since we are unsigned, it is
    actually a Y2106 problem.

Re: Millisecond precision in timestamps?

[Robin Rosenberg]

----- Ursprungligt meddelande -----
> On Tue, Nov 27, 2012 at 11:58 PM, Eric S. Raymond <esr@thyrsus.com>
> wrote:
> > Junio C Hamano <gitster@pobox.com>:
> >> Roundtrip conversions may benefit from sub-second timestamps, but
> >> personally I think negative timestamps are more interesting and of
> >> practical use.
> >
> > You mean, as in times before the Unix epoch 1970-01-01T00:00:00Z?
> >
> > Interesting.  I hadn't thought of that.  I've never seen a software
> > project under version control with bits that old, which is
> > significant
> > because I've probably done more digging into ancient software than
> > anybody other than a specialist historian or two.
> 
> One example I've heard is someone wanting to throw the history
> of a country's laws into git so they can diff them.

Not sure any laws were passed on Feb 30th 1712 in sweden, but perhaps
you can define new time zones to handle that, but I doubt it is practically
doable when you get to countries and regions with less precise boundaries.

Seconds-since as a representation for dates is a dangerous and very
messy game. Java gets it wrong somewhere in 1910 and my guess is others
get it wrong too. There is change in time zones which triggers the bug.

-- robin

Re: Millisecond precision in timestamps?

[James Cloos]

>>>>> "ESR" == Eric S Raymond <esr@thyrsus.com> writes:

ESR> I've never seen a software project under version control with bits
ESR> that old,

They do exist, but the vcs timestamps are (at least for those in git :)
not (always) correlated to when the files were first added to the project.

Maxima, as an example, has code which was written in the '60s.  (A
couple of years ago a bug was fixed in a contrib module which had
been added to MACSYMA back in '62 or so.)

I beleive axiom also has some similarly ancient code.

Those two are now managed in git.  (Except for the openaxiom fork.)

And there is a high-energy physics package still under development
with code going back to the '50s.  I'm pretty sure they moved to a
vcs sometime in the last decade or two. :)

-JimC
-- 
James Cloos <cloos@jhcloos.com>         OpenPGP: 1024D/ED7DAEA6