[QUESTION] Tracking HEAD changes?

[QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

Hi,

Is there any possible way to track changes to HEAD using hooks?

Being able to listen using hooks to events such as pre-head-checkout
and post-head-checkout would be the best option (from my perspective).

To my knowledge, the only possible way to do that today is by adding a
file watch over the refs directory.

Thank you!
    Yaron

Re: [QUESTION] Tracking HEAD changes?

[Jeff King]

On Mon, Feb 22, 2021 at 11:12:11AM +0200, Yaron Wittenstein wrote:

> Is there any possible way to track changes to HEAD using hooks?
> 
> Being able to listen using hooks to events such as pre-head-checkout
> and post-head-checkout would be the best option (from my perspective).
> 
> To my knowledge, the only possible way to do that today is by adding a
> file watch over the refs directory.

No, I don't think there is currently a better way.

If you do go the file watch route, make sure you put one on HEAD, too
(to catch the case of checking out a different branch, as opposed to
updating it with a new commit). I use:

  gitdir=$(git rev-parse --git-dir)
  # We need delete_self to pick up changes to HEAD (since it gets renamed
  # over), and "move" to pick up changes in the refs directories.
  inotifywait -qq -t 60 -e delete_self -e move -r "$gitdir/HEAD" "$gitdir/refs"

in a script that I use for automatically running tests against the
current commit. (if you are doing something similar, I also recommend
https://github.com/mhagger/git-test for avoiding re-running against the
same commits repeatedly).

-Peff

Re: [QUESTION] Tracking HEAD changes?

[Jeff King]

On Mon, Feb 22, 2021 at 07:38:30PM -0500, Jeff King wrote:

> On Mon, Feb 22, 2021 at 11:12:11AM +0200, Yaron Wittenstein wrote:
> 
> > Is there any possible way to track changes to HEAD using hooks?
> > 
> > Being able to listen using hooks to events such as pre-head-checkout
> > and post-head-checkout would be the best option (from my perspective).
> > 
> > To my knowledge, the only possible way to do that today is by adding a
> > file watch over the refs directory.
> 
> No, I don't think there is currently a better way.

Actually, I completely forgot about Patrick's recent ref-transaction
hook. See the "reference-transaction" section of githooks(7). They do
more than you'd need, but you should be able to write a hook that just
looks for updates to HEAD, or updates to the ref that HEAD is pointing
at.

The code was introduced in 6754159767 (refs: implement reference
transaction hook, 2020-06-19), so you'll need Git v2.28.0 or later.

-Peff

Re: [QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

That indeed seems to do the trick.
I've done a little experiment and saw that when doing git reset the
hook gets called.

However, when switching branches the hook doesn't execute :(

I don't understand if it's intentional, since when I've moved to a new
branch HEAD pointed to another commit id.
The only workaround I see here is using the post-checkout hook in addition.

Thank you very much for helping me. I really appreciate it!!
     Yaron

On Tue, Feb 23, 2021 at 10:42 PM Jeff King <peff@peff.net> wrote:
>
> On Mon, Feb 22, 2021 at 07:38:30PM -0500, Jeff King wrote:
>
> > On Mon, Feb 22, 2021 at 11:12:11AM +0200, Yaron Wittenstein wrote:
> >
> > > Is there any possible way to track changes to HEAD using hooks?
> > >
> > > Being able to listen using hooks to events such as pre-head-checkout
> > > and post-head-checkout would be the best option (from my perspective).
> > >
> > > To my knowledge, the only possible way to do that today is by adding a
> > > file watch over the refs directory.
> >
> > No, I don't think there is currently a better way.
>
> Actually, I completely forgot about Patrick's recent ref-transaction
> hook. See the "reference-transaction" section of githooks(7). They do
> more than you'd need, but you should be able to write a hook that just
> looks for updates to HEAD, or updates to the ref that HEAD is pointing
> at.
>
> The code was introduced in 6754159767 (refs: implement reference
> transaction hook, 2020-06-19), so you'll need Git v2.28.0 or later.
>
> -Peff

Re: [QUESTION] Tracking HEAD changes?

[Jeff King]

On Wed, Feb 24, 2021 at 10:21:55PM +0200, Yaron Wittenstein wrote:

> That indeed seems to do the trick.
> I've done a little experiment and saw that when doing git reset the
> hook gets called.
> 
> However, when switching branches the hook doesn't execute :(
> 
> I don't understand if it's intentional, since when I've moved to a new
> branch HEAD pointed to another commit id.
> The only workaround I see here is using the post-checkout hook in addition.

Hmm, I would have thought that the branch switch would trigger the hooks
because they're updating HEAD. I wonder if that is a bug (or lack of
feature :) ) in the transaction hooks, or something Patrick did
intentionally.

-Peff

Re: [QUESTION] Tracking HEAD changes?

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 1164 bytes --]

On Wed, Feb 24, 2021 at 04:03:14PM -0500, Jeff King wrote:
> On Wed, Feb 24, 2021 at 10:21:55PM +0200, Yaron Wittenstein wrote:
> 
> > That indeed seems to do the trick.
> > I've done a little experiment and saw that when doing git reset the
> > hook gets called.
> > 
> > However, when switching branches the hook doesn't execute :(
> > 
> > I don't understand if it's intentional, since when I've moved to a new
> > branch HEAD pointed to another commit id.
> > The only workaround I see here is using the post-checkout hook in addition.
> 
> Hmm, I would have thought that the branch switch would trigger the hooks
> because they're updating HEAD. I wonder if that is a bug (or lack of
> feature :) ) in the transaction hooks, or something Patrick did
> intentionally.
> 
> -Peff

It was done semi-intentionally, or at least with the knowledge that
symrefs aren't covered. This is mostly because they're not covered by
the reference transaction mechanism itself.

But this again reminds me that I still have to update the documentation
of the hook to at least make it more explicit what's currently covered
and what's not.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

Thank you for the clarification!

Is there any plan to support such a case in the future?

Also, I've observed that the "post-index-change" hook is being
triggered before calling the "reference-transaction" hook (with the
"prepared" state).
It seems not intuitive to me since the index and working dirs are
being updated before approving the transaction.
(the HEAD still points to an "old" reference while the
"post-index-change" hook is executing).

Thank you,
   Yaron

On Thu, Feb 25, 2021 at 7:37 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Wed, Feb 24, 2021 at 04:03:14PM -0500, Jeff King wrote:
> > On Wed, Feb 24, 2021 at 10:21:55PM +0200, Yaron Wittenstein wrote:
> >
> > > That indeed seems to do the trick.
> > > I've done a little experiment and saw that when doing git reset the
> > > hook gets called.
> > >
> > > However, when switching branches the hook doesn't execute :(
> > >
> > > I don't understand if it's intentional, since when I've moved to a new
> > > branch HEAD pointed to another commit id.
> > > The only workaround I see here is using the post-checkout hook in addition.
> >
> > Hmm, I would have thought that the branch switch would trigger the hooks
> > because they're updating HEAD. I wonder if that is a bug (or lack of
> > feature :) ) in the transaction hooks, or something Patrick did
> > intentionally.
> >
> > -Peff
>
> It was done semi-intentionally, or at least with the knowledge that
> symrefs aren't covered. This is mostly because they're not covered by
> the reference transaction mechanism itself.
>
> But this again reminds me that I still have to update the documentation
> of the hook to at least make it more explicit what's currently covered
> and what's not.
>
> Patrick

Re: [QUESTION] Tracking HEAD changes?

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

On Thu, Feb 25, 2021 at 08:01:11AM +0200, Yaron Wittenstein wrote:
> Thank you for the clarification!
> 
> Is there any plan to support such a case in the future?

I personally don't have any plans to do so, but I can see usecases where
it would make sense. So I wouldn't be opposing any such efforts either.

> Also, I've observed that the "post-index-change" hook is being
> triggered before calling the "reference-transaction" hook (with the
> "prepared" state). It seems not intuitive to me since the index and
> working dirs are being updated before approving the transaction. (the
> HEAD still points to an "old" reference while the "post-index-change"
> hook is executing).

This is something that cannot really be helped though. The
reference-transaction hook directly hooks into the reference transaction
mechanism, which is how git updates all references. So we do not really
have any control over when the hook will get executed: it will simply
get executed whenever the reference transaction itself gets prepared
(refs get locked) and committed (written updated refs got moved into
place).

And that's by design: my objective was to catch _all_ reference updates
such that one can coordinate across multiple git nodes which all perform
the same action to assert they're moving from the same state to the same
state, regardless of whether they're doing a git-push(1), git-merge(1)
or git-update-ref(1).

So what you're observing is simply mirroring "reality": the order in
which git does its things here. There can be arbitrarily many
transactions in a given git command, and the only way this can be
changed is by changing how the command operating the transcations works.

Patrick

> Thank you,
>    Yaron
> 
> On Thu, Feb 25, 2021 at 7:37 AM Patrick Steinhardt <ps@pks.im> wrote:
> >
> > On Wed, Feb 24, 2021 at 04:03:14PM -0500, Jeff King wrote:
> > > On Wed, Feb 24, 2021 at 10:21:55PM +0200, Yaron Wittenstein wrote:
> > >
> > > > That indeed seems to do the trick.
> > > > I've done a little experiment and saw that when doing git reset the
> > > > hook gets called.
> > > >
> > > > However, when switching branches the hook doesn't execute :(
> > > >
> > > > I don't understand if it's intentional, since when I've moved to a new
> > > > branch HEAD pointed to another commit id.
> > > > The only workaround I see here is using the post-checkout hook in addition.
> > >
> > > Hmm, I would have thought that the branch switch would trigger the hooks
> > > because they're updating HEAD. I wonder if that is a bug (or lack of
> > > feature :) ) in the transaction hooks, or something Patrick did
> > > intentionally.
> > >
> > > -Peff
> >
> > It was done semi-intentionally, or at least with the knowledge that
> > symrefs aren't covered. This is mostly because they're not covered by
> > the reference transaction mechanism itself.
> >
> > But this again reminds me that I still have to update the documentation
> > of the hook to at least make it more explicit what's currently covered
> > and what's not.
> >
> > Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [QUESTION] Tracking HEAD changes?

[Junio C Hamano]

Patrick Steinhardt <ps@pks.im> writes:

> So what you're observing is simply mirroring "reality": the order in
> which git does its things here. There can be arbitrarily many
> transactions in a given git command, and the only way this can be
> changed is by changing how the command operating the transcations works.

In other words, ref transaction is only about changes to the refs.
There is no such "transaction" that treats a series of operations
like object creation and index manipulation that may happen before a
group of refs are updated as a single unit and make it atomic.

Re: [QUESTION] Tracking HEAD changes?

[Patrick Steinhardt]

[-- Attachment #1: Type: text/plain, Size: 757 bytes --]

On Wed, Feb 24, 2021 at 11:13:47PM -0800, Junio C Hamano wrote:
> Patrick Steinhardt <ps@pks.im> writes:
> 
> > So what you're observing is simply mirroring "reality": the order in
> > which git does its things here. There can be arbitrarily many
> > transactions in a given git command, and the only way this can be
> > changed is by changing how the command operating the transcations works.
> 
> In other words, ref transaction is only about changes to the refs.
> There is no such "transaction" that treats a series of operations
> like object creation and index manipulation that may happen before a
> group of refs are updated as a single unit and make it atomic.

Exactly. Thanks for putting it into better words than I did.

Patrick

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

Thank you all!

I'd like to make sure I understand the way things are right now:

1. The reference-transaction hook as it's today doesn't intercept
symbolic-references changes.
It means that when HEAD changes due to branch-switching the hook won't
get called.

Are there any other cases that today the transaction won't execute? (I
couldn't think of one)

2. The mechanisms that handle the changes to index and working-dir are
isolated from the one that manages the refs updates.

A side-effect to that is the post-index-change hook running before the
reference-transaction one.
Future code changes to the way git operates could result in the
reference-transaction hook running before the post-index-change one.

Is that correct?

  Thank you!
      Yaron

On Thu, Feb 25, 2021 at 9:28 AM Patrick Steinhardt <ps@pks.im> wrote:
>
> On Wed, Feb 24, 2021 at 11:13:47PM -0800, Junio C Hamano wrote:
> > Patrick Steinhardt <ps@pks.im> writes:
> >
> > > So what you're observing is simply mirroring "reality": the order in
> > > which git does its things here. There can be arbitrarily many
> > > transactions in a given git command, and the only way this can be
> > > changed is by changing how the command operating the transcations works.
> >
> > In other words, ref transaction is only about changes to the refs.
> > There is no such "transaction" that treats a series of operations
> > like object creation and index manipulation that may happen before a
> > group of refs are updated as a single unit and make it atomic.
>
> Exactly. Thanks for putting it into better words than I did.
>
> Patrick

Re: [QUESTION] Tracking HEAD changes?

[Jeff King]

On Thu, Feb 25, 2021 at 06:36:54AM +0100, Patrick Steinhardt wrote:

> > Hmm, I would have thought that the branch switch would trigger the hooks
> > because they're updating HEAD. I wonder if that is a bug (or lack of
> > feature :) ) in the transaction hooks, or something Patrick did
> > intentionally.
> 
> It was done semi-intentionally, or at least with the knowledge that
> symrefs aren't covered. This is mostly because they're not covered by
> the reference transaction mechanism itself.

Ah, right, I forgot about that. That might be something that we'd want
to fix if anybody wanted to use the transaction hooks to track the
complete state of refs.

At GitHub, our repository replication does track symbolic ref updates,
but we handle it outside of Git. Our normal ref updates use a few custom
bits of code similar to your transaction hooks in order to implement a
3-phase commit. But our symref updates don't; they just take a lock, run
git-symbolic-ref on each, and then vote on the result. This is slower
and less robust, but symbolic ref updates are relatively rare, so it
hasn't been a big deal.

-Peff

Re: [QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

Thank you all!

I'd like to make sure I understand the way things are right now:

1. The reference-transaction hook as it's today doesn't intercept
symbolic-references changes. It means that when HEAD changes due to
branch-switching the hook won't
get called.

Are there any other cases that today the transaction won't execute? (I
couldn't think of one)

2. The mechanisms that handle the changes to index and working-dir are
isolated from the one that manages the refs updates.

A side-effect to that is the post-index-change hook running before the
reference-transaction one.
Future code changes to the way git operates could result in the
reference-transaction hook running before the post-index-change one.

Is that correct?

  Thank you!
      Yaron

On Fri, Feb 26, 2021 at 7:59 AM Jeff King <peff@peff.net> wrote:
>
> On Thu, Feb 25, 2021 at 06:36:54AM +0100, Patrick Steinhardt wrote:
>
> > > Hmm, I would have thought that the branch switch would trigger the hooks
> > > because they're updating HEAD. I wonder if that is a bug (or lack of
> > > feature :) ) in the transaction hooks, or something Patrick did
> > > intentionally.
> >
> > It was done semi-intentionally, or at least with the knowledge that
> > symrefs aren't covered. This is mostly because they're not covered by
> > the reference transaction mechanism itself.
>
> Ah, right, I forgot about that. That might be something that we'd want
> to fix if anybody wanted to use the transaction hooks to track the
> complete state of refs.
>
> At GitHub, our repository replication does track symbolic ref updates,
> but we handle it outside of Git. Our normal ref updates use a few custom
> bits of code similar to your transaction hooks in order to implement a
> 3-phase commit. But our symref updates don't; they just take a lock, run
> git-symbolic-ref on each, and then vote on the result. This is slower
> and less robust, but symbolic ref updates are relatively rare, so it
> hasn't been a big deal.
>
> -Peff

Re: [QUESTION] Tracking HEAD changes?

[Jeff King]

On Fri, Feb 26, 2021 at 10:58:37PM +0200, Yaron Wittenstein wrote:

> I'd like to make sure I understand the way things are right now:
> 
> 1. The reference-transaction hook as it's today doesn't intercept
> symbolic-references changes. It means that when HEAD changes due to
> branch-switching the hook won't
> get called.
> 
> Are there any other cases that today the transaction won't execute? (I
> couldn't think of one)

No, I don't think so.

> 2. The mechanisms that handle the changes to index and working-dir are
> isolated from the one that manages the refs updates.
> 
> A side-effect to that is the post-index-change hook running before the
> reference-transaction one.
> Future code changes to the way git operates could result in the
> reference-transaction hook running before the post-index-change one.
> 
> Is that correct?

Yes, that matches my understanding.

-Peff

Re: [QUESTION] Tracking HEAD changes?

[Yaron Wittenstein]

Cool, thanks!

On Mon, Mar 1, 2021 at 11:03 AM Jeff King <peff@peff.net> wrote:
>
> On Fri, Feb 26, 2021 at 10:58:37PM +0200, Yaron Wittenstein wrote:
>
> > I'd like to make sure I understand the way things are right now:
> >
> > 1. The reference-transaction hook as it's today doesn't intercept
> > symbolic-references changes. It means that when HEAD changes due to
> > branch-switching the hook won't
> > get called.
> >
> > Are there any other cases that today the transaction won't execute? (I
> > couldn't think of one)
>
> No, I don't think so.
>
> > 2. The mechanisms that handle the changes to index and working-dir are
> > isolated from the one that manages the refs updates.
> >
> > A side-effect to that is the post-index-change hook running before the
> > reference-transaction one.
> > Future code changes to the way git operates could result in the
> > reference-transaction hook running before the post-index-change one.
> >
> > Is that correct?
>
> Yes, that matches my understanding.
>
> -Peff