[PATCH 0/5] shallow: extract a header file

[PATCH 0/5] shallow: extract a header file

[Taylor Blau]

Hi,

These few patches are based on a suggestion in [1] from Jonathan Nieder
and in [2] Junio. The gist of this series is to do the following three
things:

  1. Extract a new 'shallow.h' header to collect all of the functions
     that are related to shallow-ness, but defined in 'commit.h'.

  2. Add some documentation from the previous series where it could have
     benefited from it.

  3. Finally, introduce a thin wrapper type 'struct shallow_lock' which
     makes it a type-level error to unlock a non-shallow lock with
     '{commit,rollback}_shallow_file'.

The remainder of suggestions in [1] about the structure of t5537 I
decided to not address in this series, since I anticipate there is more
work there than I want to take on in this series alone.

Thanks in advance for your review!

[1]: https://lore.kernel.org/git/20200423011444.GG140314@google.com/
[2]: https://lore.kernel.org/git/xmqqy2qnidyy.fsf@gitster.c.googlers.com/

Taylor Blau (5):
  commit: make 'commit_graft_pos' non-static
  shallow: extract a header file for shallow-related functions
  commit: move 'unregister_shallow' to 'shallow.h'
  shallow.h: document '{commit,rollback}_shallow_file'
  shallow: use struct 'shallow_lock' for additional safety

 builtin.h              |  1 +
 builtin/receive-pack.c |  2 +-
 commit.c               | 16 ++-------
 commit.h               | 49 +-------------------------
 fetch-pack.c           |  3 +-
 send-pack.c            |  1 +
 shallow.c              | 36 +++++++++++++------
 shallow.h              | 78 ++++++++++++++++++++++++++++++++++++++++++
 upload-pack.c          |  1 +
 9 files changed, 112 insertions(+), 75 deletions(-)
 create mode 100644 shallow.h

--
2.26.0.113.ge9739cdccc

[PATCH 1/5] commit: make 'commit_graft_pos' non-static

[Taylor Blau]

In the next patch, some functions will be moved from 'commit.c' to have
prototypes in a new 'shallow.h' and their implementations in
'shallow.c'.

Three functions in 'commit.c' use 'commit_graft_pos()' (they are
'register_commit_graft()', 'lookup_commit_graft()', and
'unregister_shallow()'). The first two of these will stay in 'commit.c',
but the latter will move to 'shallow.c', and thus needs
'commit_graft_pos' to be non-static.

Prepare for that by making 'commit_graft_pos' non-static so that it can
be called from both 'commit.c' and 'shallow.c'.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit.c | 2 +-
 commit.h | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git a/commit.c b/commit.c
index c7099daeac..b76f7d72be 100644
--- a/commit.c
+++ b/commit.c
@@ -110,7 +110,7 @@ static const unsigned char *commit_graft_sha1_access(size_t index, void *table)
 	return commit_graft_table[index]->oid.hash;
 }
 
-static int commit_graft_pos(struct repository *r, const unsigned char *sha1)
+int commit_graft_pos(struct repository *r, const unsigned char *sha1)
 {
 	return sha1_pos(sha1, r->parsed_objects->grafts,
 			r->parsed_objects->grafts_nr,
diff --git a/commit.h b/commit.h
index ab91d21131..eb42e8b6d2 100644
--- a/commit.h
+++ b/commit.h
@@ -236,6 +236,7 @@ struct commit_graft {
 typedef int (*each_commit_graft_fn)(const struct commit_graft *, void *);
 
 struct commit_graft *read_graft_line(struct strbuf *line);
+int commit_graft_pos(struct repository *r, const unsigned char *sha1);
 int register_commit_graft(struct repository *r, struct commit_graft *, int);
 void prepare_commit_graft(struct repository *r);
 struct commit_graft *lookup_commit_graft(struct repository *r, const struct object_id *oid);
-- 
2.26.0.113.ge9739cdccc

Re: [PATCH 1/5] commit: make 'commit_graft_pos' non-static

[Jonathan Nieder]

Hi,

Taylor Blau wrote:


>            [...] by making 'commit_graft_pos' non-static so that it can
> be called from both 'commit.c' and 'shallow.c'.
>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  commit.c | 2 +-
>  commit.h | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/commit.h b/commit.h
> index ab91d21131..eb42e8b6d2 100644
> --- a/commit.h
> +++ b/commit.h
> @@ -236,6 +236,7 @@ struct commit_graft {
>  typedef int (*each_commit_graft_fn)(const struct commit_graft *, void *);
>  
>  struct commit_graft *read_graft_line(struct strbuf *line);
> +int commit_graft_pos(struct repository *r, const unsigned char *sha1);

Now that this function isn't file-local, its name becomes more
significant.  What array does this represent a position in?  What does
it return if the graft isn't found?  From a call site it's not
necessarily obvious.

Ideas:

- could include a comment saying that it's an index into
  r->parsed_objects->grafts

- I'm usually loathe to suggest unnecessary duplication of code, but
  it might make sense to duplicate the function into shallow.c.  Or
  even to inline it there (in the single call site, that ends up
  being pretty readable).

Thoughts?

Thanks,
Jonathan

Re: [PATCH 1/5] commit: make 'commit_graft_pos' non-static

[Taylor Blau]

On Wed, Apr 29, 2020 at 08:26:11PM -0700, Jonathan Nieder wrote:
> Hi,
>
> Taylor Blau wrote:
>
>
> >            [...] by making 'commit_graft_pos' non-static so that it can
> > be called from both 'commit.c' and 'shallow.c'.
> >
> > Signed-off-by: Taylor Blau <me@ttaylorr.com>
> > ---
> >  commit.c | 2 +-
> >  commit.h | 1 +
> >  2 files changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/commit.h b/commit.h
> > index ab91d21131..eb42e8b6d2 100644
> > --- a/commit.h
> > +++ b/commit.h
> > @@ -236,6 +236,7 @@ struct commit_graft {
> >  typedef int (*each_commit_graft_fn)(const struct commit_graft *, void *);
> >
> >  struct commit_graft *read_graft_line(struct strbuf *line);
> > +int commit_graft_pos(struct repository *r, const unsigned char *sha1);
>
> Now that this function isn't file-local, its name becomes more
> significant.  What array does this represent a position in?  What does
> it return if the graft isn't found?  From a call site it's not
> necessarily obvious.
>
> Ideas:
>
> - could include a comment saying that it's an index into
>   r->parsed_objects->grafts

This and the below are both good ideas to me. I prefer this one, since
we'd have to duplicate yet another static function
('commit_graft_sha1_access()' directly above) that is called by this
one.

> - I'm usually loathe to suggest unnecessary duplication of code, but
>   it might make sense to duplicate the function into shallow.c.  Or
>   even to inline it there (in the single call site, that ends up
>   being pretty readable).

I am not at all offended by duplication of code where it makes sense to
do so, but having to duplicate two functions seems like we'd be better
off simply documenting the function in commit.h.

> Thoughts?
>
> Thanks,
> Jonathan

Thanks,
Taylor

[PATCH 2/5] shallow: extract a header file for shallow-related functions

[Taylor Blau]

There are many functions in commit.h that are more related to shallow
repositories than they are to any sort of generic commit machinery.
Likely this began when there were only a few shallow-related functions,
and commit.h seemed a reasonable enough place to put them.

But, now there are a good number of shallow-related functions, and
placing them all in 'commit.h' doesn't make sense.

This patch extracts a 'shallow.h', which takes all of the headers from
'commit.h' for functions which already exist in 'shallow.c'. We will
bring the remaining shallow-related functions defined in 'commit.c' in a
subsequent patch.

For now, move only the ones that already are implemented in 'shallow.c',
and update the necessary includes.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin.h     |  1 +
 commit.c      |  1 +
 commit.h      | 48 ---------------------------------------
 fetch-pack.c  |  1 +
 send-pack.c   |  1 +
 shallow.c     |  1 +
 shallow.h     | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++
 upload-pack.c |  1 +
 8 files changed, 69 insertions(+), 48 deletions(-)
 create mode 100644 shallow.h

diff --git a/builtin.h b/builtin.h
index 2b25a80cde..2e701a023c 100644
--- a/builtin.h
+++ b/builtin.h
@@ -5,6 +5,7 @@
 #include "strbuf.h"
 #include "cache.h"
 #include "commit.h"
+#include "shallow.h"
 
 /*
  * builtin API
diff --git a/commit.c b/commit.c
index b76f7d72be..eebfbeb45d 100644
--- a/commit.c
+++ b/commit.c
@@ -20,6 +20,7 @@
 #include "refs.h"
 #include "commit-reach.h"
 #include "run-command.h"
+#include "shallow.h"
 
 static struct commit_extra_header *read_commit_extra_header_lines(const char *buf, size_t len, const char **);
 
diff --git a/commit.h b/commit.h
index eb42e8b6d2..5cd984939b 100644
--- a/commit.h
+++ b/commit.h
@@ -248,55 +248,7 @@ struct commit *get_fork_point(const char *refname, struct commit *commit);
 
 struct oid_array;
 struct ref;
-int register_shallow(struct repository *r, const struct object_id *oid);
-int unregister_shallow(const struct object_id *oid);
-int commit_shallow_file(struct repository *r, struct lock_file *lk);
-void rollback_shallow_file(struct repository *r, struct lock_file *lk);
 int for_each_commit_graft(each_commit_graft_fn, void *);
-int is_repository_shallow(struct repository *r);
-struct commit_list *get_shallow_commits(struct object_array *heads,
-					int depth, int shallow_flag, int not_shallow_flag);
-struct commit_list *get_shallow_commits_by_rev_list(
-		int ac, const char **av, int shallow_flag, int not_shallow_flag);
-void set_alternate_shallow_file(struct repository *r, const char *path, int override);
-int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
-			  const struct oid_array *extra);
-void setup_alternate_shallow(struct lock_file *shallow_lock,
-			     const char **alternate_shallow_file,
-			     const struct oid_array *extra);
-const char *setup_temporary_shallow(const struct oid_array *extra);
-void advertise_shallow_grafts(int);
-
-/*
- * Initialize with prepare_shallow_info() or zero-initialize (equivalent to
- * prepare_shallow_info with a NULL oid_array).
- */
-struct shallow_info {
-	struct oid_array *shallow;
-	int *ours, nr_ours;
-	int *theirs, nr_theirs;
-	struct oid_array *ref;
-
-	/* for receive-pack */
-	uint32_t **used_shallow;
-	int *need_reachability_test;
-	int *reachable;
-	int *shallow_ref;
-	struct commit **commits;
-	int nr_commits;
-};
-
-void prepare_shallow_info(struct shallow_info *, struct oid_array *);
-void clear_shallow_info(struct shallow_info *);
-void remove_nonexistent_theirs_shallow(struct shallow_info *);
-void assign_shallow_commits_to_refs(struct shallow_info *info,
-				    uint32_t **used,
-				    int *ref_status);
-int delayed_reachability_test(struct shallow_info *si, int c);
-#define PRUNE_SHOW_ONLY 1
-#define PRUNE_QUICK 2
-void prune_shallow(unsigned options);
-extern struct trace_key trace_shallow;
 
 int interactive_add(int argc, const char **argv, const char *prefix, int patch);
 int run_add_interactive(const char *revision, const char *patch_mode,
diff --git a/fetch-pack.c b/fetch-pack.c
index a618f3b029..401d028e41 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -22,6 +22,7 @@
 #include "connected.h"
 #include "fetch-negotiator.h"
 #include "fsck.h"
+#include "shallow.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;
diff --git a/send-pack.c b/send-pack.c
index 0407841ae8..e0ccfef75a 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -15,6 +15,7 @@
 #include "sha1-array.h"
 #include "gpg-interface.h"
 #include "cache.h"
+#include "shallow.h"
 
 int option_parse_push_signed(const struct option *opt,
 			     const char *arg, int unset)
diff --git a/shallow.c b/shallow.c
index 5010a6c732..d0191c7609 100644
--- a/shallow.c
+++ b/shallow.c
@@ -14,6 +14,7 @@
 #include "commit-slab.h"
 #include "list-objects.h"
 #include "commit-reach.h"
+#include "shallow.h"
 
 void set_alternate_shallow_file(struct repository *r, const char *path, int override)
 {
diff --git a/shallow.h b/shallow.h
new file mode 100644
index 0000000000..14dd748625
--- /dev/null
+++ b/shallow.h
@@ -0,0 +1,63 @@
+#ifndef SHALLOW_H
+#define SHALLOW_H
+
+#include "lockfile.h"
+#include "object.h"
+#include "repository.h"
+#include "strbuf.h"
+
+void set_alternate_shallow_file(struct repository *r, const char *path, int override);
+int register_shallow(struct repository *r, const struct object_id *oid);
+int is_repository_shallow(struct repository *r);
+int commit_shallow_file(struct repository *r, struct lock_file *lk);
+void rollback_shallow_file(struct repository *r, struct lock_file *lk);
+
+struct commit_list *get_shallow_commits(struct object_array *heads,
+					int depth, int shallow_flag, int not_shallow_flag);
+struct commit_list *get_shallow_commits_by_rev_list(
+		int ac, const char **av, int shallow_flag, int not_shallow_flag);
+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
+			  const struct oid_array *extra);
+
+void setup_alternate_shallow(struct lock_file *shallow_lock,
+			     const char **alternate_shallow_file,
+			     const struct oid_array *extra);
+
+const char *setup_temporary_shallow(const struct oid_array *extra);
+
+void advertise_shallow_grafts(int);
+
+#define PRUNE_SHOW_ONLY 1
+#define PRUNE_QUICK 2
+void prune_shallow(unsigned options);
+
+/*
+ * Initialize with prepare_shallow_info() or zero-initialize (equivalent to
+ * prepare_shallow_info with a NULL oid_array).
+ */
+struct shallow_info {
+	struct oid_array *shallow;
+	int *ours, nr_ours;
+	int *theirs, nr_theirs;
+	struct oid_array *ref;
+
+	/* for receive-pack */
+	uint32_t **used_shallow;
+	int *need_reachability_test;
+	int *reachable;
+	int *shallow_ref;
+	struct commit **commits;
+	int nr_commits;
+};
+
+void prepare_shallow_info(struct shallow_info *, struct oid_array *);
+void clear_shallow_info(struct shallow_info *);
+void remove_nonexistent_theirs_shallow(struct shallow_info *);
+void assign_shallow_commits_to_refs(struct shallow_info *info,
+				    uint32_t **used,
+				    int *ref_status);
+int delayed_reachability_test(struct shallow_info *si, int c);
+
+extern struct trace_key trace_shallow;
+
+#endif /* SHALLOW_H */
diff --git a/upload-pack.c b/upload-pack.c
index c53249cac1..e71b068291 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -26,6 +26,7 @@
 #include "serve.h"
 #include "commit-graph.h"
 #include "commit-reach.h"
+#include "shallow.h"
 
 /* Remember to update object flag allocation in object.h */
 #define THEY_HAVE	(1u << 11)
-- 
2.26.0.113.ge9739cdccc

Re: [PATCH 2/5] shallow: extract a header file for shallow-related functions

[Eric Sunshine]

On Wed, Apr 29, 2020 at 6:39 PM Taylor Blau <me@ttaylorr.com> wrote:
> There are many functions in commit.h that are more related to shallow
> repositories than they are to any sort of generic commit machinery.
> Likely this began when there were only a few shallow-related functions,
> and commit.h seemed a reasonable enough place to put them.
>
> But, now there are a good number of shallow-related functions, and
> placing them all in 'commit.h' doesn't make sense.
>
> This patch extracts a 'shallow.h', which takes all of the headers from

Did you mean s/headers/declarations/ ?

> 'commit.h' for functions which already exist in 'shallow.c'. We will
> bring the remaining shallow-related functions defined in 'commit.c' in a
> subsequent patch.
>
> For now, move only the ones that already are implemented in 'shallow.c',
> and update the necessary includes.
>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>

Re: [PATCH 2/5] shallow: extract a header file for shallow-related functions

[Jonathan Tan]

First of all, I couldn't apply these patches neither on latest master
(86ab15cb15 ("The fourth batch", 2020-04-28)) nor on its 2 immediate
ancestors - what is the base of these patches? (Or is there something
wrong with my workflow?)

I'll review based on the patches themselves, but what I've seen looks
good so far.

> diff --git a/builtin.h b/builtin.h
> index 2b25a80cde..2e701a023c 100644
> --- a/builtin.h
> +++ b/builtin.h
> @@ -5,6 +5,7 @@
>  #include "strbuf.h"
>  #include "cache.h"
>  #include "commit.h"
> +#include "shallow.h"

It's a pity that builtin.h has to be modified in this way, but I see
that it's necessary - a few files that include builtin.h (at least git.c
and builtin/bisect--helper.c) assume that they have shallow function
access through this header.

Once I manage to apply these patches myself, I'll verify with
--color-moved that the lines moved as I expect, but otherwise patch 1
and 2 look good.

Re: [PATCH 2/5] shallow: extract a header file for shallow-related functions

[Taylor Blau]

Hi Jonathan,

On Wed, Apr 29, 2020 at 05:21:54PM -0700, Jonathan Tan wrote:
> First of all, I couldn't apply these patches neither on latest master
> (86ab15cb15 ("The fourth batch", 2020-04-28)) nor on its 2 immediate
> ancestors - what is the base of these patches? (Or is there something
> wrong with my workflow?)

Yes, I should have mentioned in the cover letter that this is based on
'tb/reset-shallow' which is should be part of Junio's next push to
master. I didn't mention it because I figured that it would be on master
by the time anyone wanted to look at these patches ;).

Anyhow, apologies. It should apply cleanly on any branch that has
'tb/reset-shallow' already.

> I'll review based on the patches themselves, but what I've seen looks
> good so far.
>
> > diff --git a/builtin.h b/builtin.h
> > index 2b25a80cde..2e701a023c 100644
> > --- a/builtin.h
> > +++ b/builtin.h
> > @@ -5,6 +5,7 @@
> >  #include "strbuf.h"
> >  #include "cache.h"
> >  #include "commit.h"
> > +#include "shallow.h"
>
> It's a pity that builtin.h has to be modified in this way, but I see
> that it's necessary - a few files that include builtin.h (at least git.c
> and builtin/bisect--helper.c) assume that they have shallow function
> access through this header.

Yeah :/.

> Once I manage to apply these patches myself, I'll verify with
> --color-moved that the lines moved as I expect, but otherwise patch 1
> and 2 look good.

Thanks.

Thanks,
Taylor

Re: [PATCH 2/5] shallow: extract a header file for shallow-related functions

[Jonathan Tan]

> Yes, I should have mentioned in the cover letter that this is based on
> 'tb/reset-shallow' which is should be part of Junio's next push to
> master. I didn't mention it because I figured that it would be on master
> by the time anyone wanted to look at these patches ;).
> 
> Anyhow, apologies. It should apply cleanly on any branch that has
> 'tb/reset-shallow' already.

Thanks. Now that I'm able to apply the patches, I see that the prototype
for unregister_shallow() was removed with no corresponding addition.
This causes a compile error when running "make" with DEVELOPER=1. Other
than that, things look fine.

Re: [PATCH 2/5] shallow: extract a header file for shallow-related functions

[Jonathan Nieder]

Taylor Blau wrote:

> There are many functions in commit.h that are more related to shallow
> repositories than they are to any sort of generic commit machinery.
> Likely this began when there were only a few shallow-related functions,
> and commit.h seemed a reasonable enough place to put them.
>
> But, now there are a good number of shallow-related functions, and
> placing them all in 'commit.h' doesn't make sense.

Sure.  For me, there are a few additional sources of motivation:

- shallow clone is a bit of a thorny feature, so I like having the
  indication of which source files are interacting with it

- this will give us a good place to put any overview documentation on
  the shallow API

> This patch extracts a 'shallow.h', which takes all of the headers from
> 'commit.h' for functions which already exist in 'shallow.c'. We will
> bring the remaining shallow-related functions defined in 'commit.c' in a
> subsequent patch.
>
> For now, move only the ones that already are implemented in 'shallow.c',
> and update the necessary includes.

It's probably worth a mention of the builtin.h part here.

(By the way, I wouldn't be against propagating that to the callers,
to better match what https://include-what-you-use.org/ would enforce.)

Thanks,
Jonathan

[PATCH 3/5] commit: move 'unregister_shallow' to 'shallow.h'

[Taylor Blau]

In the last commit, we introduced a header for the functions defined in
'shallow.c'. There is one remaining shallow-related function in
commit.c which should be moved, too. This patch moves that function.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 commit.c  | 13 -------------
 shallow.c | 13 +++++++++++++
 shallow.h |  1 +
 3 files changed, 14 insertions(+), 13 deletions(-)

diff --git a/commit.c b/commit.c
index eebfbeb45d..87686a7055 100644
--- a/commit.c
+++ b/commit.c
@@ -246,19 +246,6 @@ int for_each_commit_graft(each_commit_graft_fn fn, void *cb_data)
 	return ret;
 }
 
-int unregister_shallow(const struct object_id *oid)
-{
-	int pos = commit_graft_pos(the_repository, oid->hash);
-	if (pos < 0)
-		return -1;
-	if (pos + 1 < the_repository->parsed_objects->grafts_nr)
-		MOVE_ARRAY(the_repository->parsed_objects->grafts + pos,
-			   the_repository->parsed_objects->grafts + pos + 1,
-			   the_repository->parsed_objects->grafts_nr - pos - 1);
-	the_repository->parsed_objects->grafts_nr--;
-	return 0;
-}
-
 struct commit_buffer {
 	void *buffer;
 	unsigned long size;
diff --git a/shallow.c b/shallow.c
index d0191c7609..76e00893fe 100644
--- a/shallow.c
+++ b/shallow.c
@@ -39,6 +39,19 @@ int register_shallow(struct repository *r, const struct object_id *oid)
 	return register_commit_graft(r, graft, 0);
 }
 
+int unregister_shallow(const struct object_id *oid)
+{
+	int pos = commit_graft_pos(the_repository, oid->hash);
+	if (pos < 0)
+		return -1;
+	if (pos + 1 < the_repository->parsed_objects->grafts_nr)
+		MOVE_ARRAY(the_repository->parsed_objects->grafts + pos,
+			   the_repository->parsed_objects->grafts + pos + 1,
+			   the_repository->parsed_objects->grafts_nr - pos - 1);
+	the_repository->parsed_objects->grafts_nr--;
+	return 0;
+}
+
 int is_repository_shallow(struct repository *r)
 {
 	FILE *fp;
diff --git a/shallow.h b/shallow.h
index 14dd748625..b50a85ed7e 100644
--- a/shallow.h
+++ b/shallow.h
@@ -8,6 +8,7 @@
 
 void set_alternate_shallow_file(struct repository *r, const char *path, int override);
 int register_shallow(struct repository *r, const struct object_id *oid);
+int unregister_shallow(const struct object_id *oid);
 int is_repository_shallow(struct repository *r);
 int commit_shallow_file(struct repository *r, struct lock_file *lk);
 void rollback_shallow_file(struct repository *r, struct lock_file *lk);
-- 
2.26.0.113.ge9739cdccc

Re: [PATCH 3/5] commit: move 'unregister_shallow' to 'shallow.h'

[Jonathan Nieder]

Taylor Blau wrote:

> In the last commit, we introduced a header for the functions defined in
> 'shallow.c'. There is one remaining shallow-related function in
> commit.c which should be moved, too. This patch moves that function.
> 
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  commit.c  | 13 -------------
>  shallow.c | 13 +++++++++++++
>  shallow.h |  1 +
>  3 files changed, 14 insertions(+), 13 deletions(-)

Yes, an obviously good thing to do.

Perhaps could be squashed with patch 1 (but also see my review of that
one).

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Should this take a "struct repository" parameter?

Re: [PATCH 3/5] commit: move 'unregister_shallow' to 'shallow.h'

[Taylor Blau]

On Wed, Apr 29, 2020 at 08:13:50PM -0700, Jonathan Nieder wrote:
> Taylor Blau wrote:
>
> > In the last commit, we introduced a header for the functions defined in
> > 'shallow.c'. There is one remaining shallow-related function in
> > commit.c which should be moved, too. This patch moves that function.
> >
> > Signed-off-by: Taylor Blau <me@ttaylorr.com>
> > ---
> >  commit.c  | 13 -------------
> >  shallow.c | 13 +++++++++++++
> >  shallow.h |  1 +
> >  3 files changed, 14 insertions(+), 13 deletions(-)
>
> Yes, an obviously good thing to do.
>
> Perhaps could be squashed with patch 1 (but also see my review of that
> one).

Do you mean patch 2/5? If so, I think that it probably makes more sense
to go there than in the first patch.

(I originally thought that it would be useful for people reading the
diff to have this change broken out into its own patch, but I think that
it's more juggling to do in terms of figuring out which headers to
include than is worth.)

> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>
>
> Should this take a "struct repository" parameter?

Probably, but I'd rather leave this series to be just focused on moving
this code around while being minimally invasive.

Thanks,
Taylor

[PATCH 4/5] shallow.h: document '{commit,rollback}_shallow_file'

[Taylor Blau]

When 'commit_shallow_file()' and 'rollback_shallow_file()' were
introduced, they did not have an documenting comment, when they could
have benefited from one.

Add a brief note about what these functions do, and make a special note
that they reset stat-validity checks.

Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 shallow.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/shallow.h b/shallow.h
index b50a85ed7e..08e1bc4fd0 100644
--- a/shallow.h
+++ b/shallow.h
@@ -10,6 +10,10 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
 int register_shallow(struct repository *r, const struct object_id *oid);
 int unregister_shallow(const struct object_id *oid);
 int is_repository_shallow(struct repository *r);
+/*
+ * {commit,rollback}_shallow_file commits or performs a rollback to the
+ * '.git/shallow' file, respectively, and resets stat-validity checks.
+ */
 int commit_shallow_file(struct repository *r, struct lock_file *lk);
 void rollback_shallow_file(struct repository *r, struct lock_file *lk);
 
-- 
2.26.0.113.ge9739cdccc

Re: [PATCH 4/5] shallow.h: document '{commit,rollback}_shallow_file'

[Eric Sunshine]

On Wed, Apr 29, 2020 at 6:39 PM Taylor Blau <me@ttaylorr.com> wrote:
> When 'commit_shallow_file()' and 'rollback_shallow_file()' were
> introduced, they did not have an documenting comment, when they could
> have benefited from one.

"an" seems wrong; did you mean "a" or "any"? Or, rewrite as: "...they
were not documented...".

> Add a brief note about what these functions do, and make a special note
> that they reset stat-validity checks.
>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
> diff --git a/shallow.h b/shallow.h
> @@ -10,6 +10,10 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
>  int register_shallow(struct repository *r, const struct object_id *oid);
>  int unregister_shallow(const struct object_id *oid);
>  int is_repository_shallow(struct repository *r);
> +/*
> + * {commit,rollback}_shallow_file commits or performs a rollback to the
> + * '.git/shallow' file, respectively, and resets stat-validity checks.
> + */
>  int commit_shallow_file(struct repository *r, struct lock_file *lk);
>  void rollback_shallow_file(struct repository *r, struct lock_file *lk);

Or, simpler:

    /* commit .git/shallow and reset stat-validity checks */
    int commit_shallow_file(...);
    /* rollback .git/shallow and reset stat-validity checks */
    void rollback_shallow_file(...);

[PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Taylor Blau]

In previous patches, the functions 'commit_shallow_file' and
'rollback_shallow_file' were introduced to reset the shallowness
validity checks on a repository after potentially modifying
'.git/shallow'.

These functions can be made safer by wrapping the 'struct lockfile *' in
a new type, 'shallow_lock', so that they cannot be called with a raw
lock (and potentially misused by other code that happens to possess a
lockfile, but has nothing to do with shallowness).

This patch introduces that type as a thin wrapper around 'struct
lockfile', and updates the two aforementioned functions and their
callers to use it.

Suggested-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Taylor Blau <me@ttaylorr.com>
---
 builtin/receive-pack.c |  2 +-
 fetch-pack.c           |  2 +-
 shallow.c              | 22 +++++++++++-----------
 shallow.h              | 16 +++++++++++++---
 4 files changed, 26 insertions(+), 16 deletions(-)

diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 652661fa99..10fd2c52ec 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -855,7 +855,7 @@ static void refuse_unconfigured_deny_delete_current(void)
 static int command_singleton_iterator(void *cb_data, struct object_id *oid);
 static int update_shallow_ref(struct command *cmd, struct shallow_info *si)
 {
-	struct lock_file shallow_lock = LOCK_INIT;
+	struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT;
 	struct oid_array extra = OID_ARRAY_INIT;
 	struct check_connected_options opt = CHECK_CONNECTED_INIT;
 	uint32_t mask = 1 << (cmd->index % 32);
diff --git a/fetch-pack.c b/fetch-pack.c
index 401d028e41..27d2d08cc0 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -35,7 +35,7 @@ static int fetch_fsck_objects = -1;
 static int transfer_fsck_objects = -1;
 static int agent_supported;
 static int server_supports_filtering;
-static struct lock_file shallow_lock;
+static struct shallow_lock shallow_lock;
 static const char *alternate_shallow_file;
 static struct strbuf fsck_msg_types = STRBUF_INIT;
 
diff --git a/shallow.c b/shallow.c
index 76e00893fe..1acf8ce95b 100644
--- a/shallow.c
+++ b/shallow.c
@@ -92,16 +92,16 @@ static void reset_repository_shallow(struct repository *r)
 	stat_validity_clear(r->parsed_objects->shallow_stat);
 }
 
-int commit_shallow_file(struct repository *r, struct lock_file *lk)
+int commit_shallow_file(struct repository *r, struct shallow_lock *lk)
 {
-	int res = commit_lock_file(lk);
+	int res = commit_lock_file(&lk->lk);
 	reset_repository_shallow(r);
 	return res;
 }
 
-void rollback_shallow_file(struct repository *r, struct lock_file *lk)
+void rollback_shallow_file(struct repository *r, struct shallow_lock *lk)
 {
-	rollback_lock_file(lk);
+	rollback_lock_file(&lk->lk);
 	reset_repository_shallow(r);
 }
 
@@ -366,22 +366,22 @@ const char *setup_temporary_shallow(const struct oid_array *extra)
 	return "";
 }
 
-void setup_alternate_shallow(struct lock_file *shallow_lock,
+void setup_alternate_shallow(struct shallow_lock *shallow_lock,
 			     const char **alternate_shallow_file,
 			     const struct oid_array *extra)
 {
 	struct strbuf sb = STRBUF_INIT;
 	int fd;
 
-	fd = hold_lock_file_for_update(shallow_lock,
+	fd = hold_lock_file_for_update(&shallow_lock->lk,
 				       git_path_shallow(the_repository),
 				       LOCK_DIE_ON_ERROR);
 	check_shallow_file_for_update(the_repository);
 	if (write_shallow_commits(&sb, 0, extra)) {
 		if (write_in_full(fd, sb.buf, sb.len) < 0)
 			die_errno("failed to write to %s",
-				  get_lock_file_path(shallow_lock));
-		*alternate_shallow_file = get_lock_file_path(shallow_lock);
+				  get_lock_file_path(&shallow_lock->lk));
+		*alternate_shallow_file = get_lock_file_path(&shallow_lock->lk);
 	} else
 		/*
 		 * is_repository_shallow() sees empty string as "no
@@ -414,7 +414,7 @@ void advertise_shallow_grafts(int fd)
  */
 void prune_shallow(unsigned options)
 {
-	struct lock_file shallow_lock = LOCK_INIT;
+	struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT;
 	struct strbuf sb = STRBUF_INIT;
 	unsigned flags = SEEN_ONLY;
 	int fd;
@@ -428,14 +428,14 @@ void prune_shallow(unsigned options)
 		strbuf_release(&sb);
 		return;
 	}
-	fd = hold_lock_file_for_update(&shallow_lock,
+	fd = hold_lock_file_for_update(&shallow_lock.lk,
 				       git_path_shallow(the_repository),
 				       LOCK_DIE_ON_ERROR);
 	check_shallow_file_for_update(the_repository);
 	if (write_shallow_commits_1(&sb, 0, NULL, flags)) {
 		if (write_in_full(fd, sb.buf, sb.len) < 0)
 			die_errno("failed to write to %s",
-				  get_lock_file_path(&shallow_lock));
+				  get_lock_file_path(&shallow_lock.lk));
 		commit_shallow_file(the_repository, &shallow_lock);
 	} else {
 		unlink(git_path_shallow(the_repository));
diff --git a/shallow.h b/shallow.h
index 08e1bc4fd0..d12096fbc4 100644
--- a/shallow.h
+++ b/shallow.h
@@ -10,12 +10,22 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
 int register_shallow(struct repository *r, const struct object_id *oid);
 int unregister_shallow(const struct object_id *oid);
 int is_repository_shallow(struct repository *r);
+
+/*
+ * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
+ * which locks can be used with '{commit,rollback}_shallow_file()'.
+ */
+struct shallow_lock {
+	struct lock_file lk;
+};
+#define SHALLOW_LOCK_INIT { LOCK_INIT }
+
 /*
  * {commit,rollback}_shallow_file commits or performs a rollback to the
  * '.git/shallow' file, respectively, and resets stat-validity checks.
  */
-int commit_shallow_file(struct repository *r, struct lock_file *lk);
-void rollback_shallow_file(struct repository *r, struct lock_file *lk);
+int commit_shallow_file(struct repository *r, struct shallow_lock *lk);
+void rollback_shallow_file(struct repository *r, struct shallow_lock *lk);
 
 struct commit_list *get_shallow_commits(struct object_array *heads,
 					int depth, int shallow_flag, int not_shallow_flag);
@@ -24,7 +34,7 @@ struct commit_list *get_shallow_commits_by_rev_list(
 int write_shallow_commits(struct strbuf *out, int use_pack_protocol,
 			  const struct oid_array *extra);
 
-void setup_alternate_shallow(struct lock_file *shallow_lock,
+void setup_alternate_shallow(struct shallow_lock *shallow_lock,
 			     const char **alternate_shallow_file,
 			     const struct oid_array *extra);
 
-- 
2.26.0.113.ge9739cdccc

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Eric Sunshine]

On Wed, Apr 29, 2020 at 6:39 PM Taylor Blau <me@ttaylorr.com> wrote:
> [...]
> This patch introduces that type as a thin wrapper around 'struct
> lockfile', and updates the two aforementioned functions and their
> callers to use it.
>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
> diff --git a/shallow.h b/shallow.h
> @@ -10,12 +10,22 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
> +/*
> + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> + * which locks can be used with '{commit,rollback}_shallow_file()'.
> + */
> +struct shallow_lock {
> +       struct lock_file lk;
> +};

The documentation comment for 'shallow_lock' may help newcomers to C
but probably doesn't add much value for seasoned programmers. If this
is the sort of idiom we want to introduce (or exists already in this
codebase) -- declaring a specific C type to avoid accidental use of an
unrelated lock -- then it's probably better documented in
CodingGuidelines rather than repeating it at each point in the code
which employs the idiom.

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Taylor Blau]

On Wed, Apr 29, 2020 at 07:03:48PM -0400, Eric Sunshine wrote:
> On Wed, Apr 29, 2020 at 6:39 PM Taylor Blau <me@ttaylorr.com> wrote:
> > [...]
> > This patch introduces that type as a thin wrapper around 'struct
> > lockfile', and updates the two aforementioned functions and their
> > callers to use it.
> >
> > Signed-off-by: Taylor Blau <me@ttaylorr.com>
> > ---
> > diff --git a/shallow.h b/shallow.h
> > @@ -10,12 +10,22 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
> > +/*
> > + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> > + * which locks can be used with '{commit,rollback}_shallow_file()'.
> > + */
> > +struct shallow_lock {
> > +       struct lock_file lk;
> > +};
>
> The documentation comment for 'shallow_lock' may help newcomers to C
> but probably doesn't add much value for seasoned programmers. If this
> is the sort of idiom we want to introduce (or exists already in this
> codebase) -- declaring a specific C type to avoid accidental use of an
> unrelated lock -- then it's probably better documented in
> CodingGuidelines rather than repeating it at each point in the code
> which employs the idiom.

Fair enough; I definitely weighed the usefulness of this comment a
little bit when I was writing it. I figured that I didn't want to
commit to updating CodingGuidelines in this series, but that I didn't
want to leave the explanation only in the commit message.

I figure that it's probably fine to document it in the commit message,
and leave it at that for now.

Thanks for this suggestion (and the earlier ones, too). I have applied
them locally, and I'll sit on them for a day or two before sending out
v2. Thanks again.


Thanks,
Taylor

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Jonathan Tan]

> @@ -428,14 +428,14 @@ void prune_shallow(unsigned options)
>  		strbuf_release(&sb);
>  		return;
>  	}
> -	fd = hold_lock_file_for_update(&shallow_lock,
> +	fd = hold_lock_file_for_update(&shallow_lock.lk,
>  				       git_path_shallow(the_repository),
>  				       LOCK_DIE_ON_ERROR);
>  	check_shallow_file_for_update(the_repository);
>  	if (write_shallow_commits_1(&sb, 0, NULL, flags)) {
>  		if (write_in_full(fd, sb.buf, sb.len) < 0)
>  			die_errno("failed to write to %s",
> -				  get_lock_file_path(&shallow_lock));
> +				  get_lock_file_path(&shallow_lock.lk));
>  		commit_shallow_file(the_repository, &shallow_lock);
>  	} else {
>  		unlink(git_path_shallow(the_repository));

I was hoping that the inner lock ("lk") wouldn't need to be exposed, so
that it wouldn't be possible to inadvertently commit the lock without
going through commit_shallow_file(), but this patch is still a step
towards that.

> diff --git a/shallow.h b/shallow.h
> index 08e1bc4fd0..d12096fbc4 100644
> --- a/shallow.h
> +++ b/shallow.h
> @@ -10,12 +10,22 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
>  int register_shallow(struct repository *r, const struct object_id *oid);
>  int unregister_shallow(const struct object_id *oid);
>  int is_repository_shallow(struct repository *r);
> +
> +/*
> + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> + * which locks can be used with '{commit,rollback}_shallow_file()'.
> + */
> +struct shallow_lock {
> +	struct lock_file lk;
> +};

As far as I know, we don't use many abbreviations in struct members -
maybe s/lk/lock/.

I couldn't find any other issues with all 5 patches in this patch
series, but I would like to be able to apply the patches to be sure.

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Jonathan Nieder]

Hi,

Taylor Blau wrote:

> In previous patches, the functions 'commit_shallow_file' and
> 'rollback_shallow_file' were introduced to reset the shallowness
> validity checks on a repository after potentially modifying
> '.git/shallow'.
>
> These functions can be made safer by wrapping the 'struct lockfile *' in
> a new type, 'shallow_lock', so that they cannot be called with a raw
> lock (and potentially misused by other code that happens to possess a
> lockfile, but has nothing to do with shallowness).
>
> This patch introduces that type as a thin wrapper around 'struct
> lockfile', and updates the two aforementioned functions and their
> callers to use it.
>
> Suggested-by: Junio C Hamano <gitster@pobox.com>
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  builtin/receive-pack.c |  2 +-
>  fetch-pack.c           |  2 +-
>  shallow.c              | 22 +++++++++++-----------
>  shallow.h              | 16 +++++++++++++---
>  4 files changed, 26 insertions(+), 16 deletions(-)

Yay!  Thanks for indulging the suggestion.

[...]
> --- a/shallow.h
> +++ b/shallow.h
> @@ -10,12 +10,22 @@ void set_alternate_shallow_file(struct repository *r, const char *path, int over
>  int register_shallow(struct repository *r, const struct object_id *oid);
>  int unregister_shallow(const struct object_id *oid);
>  int is_repository_shallow(struct repository *r);
> +
> +/*
> + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> + * which locks can be used with '{commit,rollback}_shallow_file()'.
> + */
> +struct shallow_lock {
> +	struct lock_file lk;
> +};
> +#define SHALLOW_LOCK_INIT { LOCK_INIT }

I think I disagree with Eric here: it's useful to have a comment here
to describe the purpose of the struct (i.e., the "why" as opposed to
the "what").

I wonder if we can go further, though --- when using a shallow_lock,
how should I think of it as a caller?  In some sense, the use of
'struct lock_file' is an implementation detail, so we could say:

	/*
	 * Lock for updating the $GIT_DIR/shallow file.
	 *
	 * Use `commit_shallow_file()` to commit an update, or
	 * `rollback_shallow_file()` to roll it back.  In either case,
	 * any in-memory cached information about which commits are
	 * shallow will be appropriately invalidated so that future
	 * operations reflect the new state.
	 */

What do you think?

[...]
> --- a/shallow.c
> +++ b/shallow.c
[...]
> @@ -366,22 +366,22 @@ const char *setup_temporary_shallow(const struct oid_array *extra)
>  	return "";
>  }
>  
> -void setup_alternate_shallow(struct lock_file *shallow_lock,
> +void setup_alternate_shallow(struct shallow_lock *shallow_lock,
>  			     const char **alternate_shallow_file,
>  			     const struct oid_array *extra)
>  {
>  	struct strbuf sb = STRBUF_INIT;
>  	int fd;
>  
> -	fd = hold_lock_file_for_update(shallow_lock,
> +	fd = hold_lock_file_for_update(&shallow_lock->lk,
>  				       git_path_shallow(the_repository),
>  				       LOCK_DIE_ON_ERROR);

This is peeking into the underlying lock_file, so I should ask myself
whether it's hinting at some missing function in the shallow_lock
API.  My feeling is "no": setup_alternate_shallow is itself that
function. :)

[...]
> @@ -414,7 +414,7 @@ void advertise_shallow_grafts(int fd)
>   */
>  void prune_shallow(unsigned options)
>  {
> -	struct lock_file shallow_lock = LOCK_INIT;
> +	struct shallow_lock shallow_lock = SHALLOW_LOCK_INIT;
>  	struct strbuf sb = STRBUF_INIT;
>  	unsigned flags = SEEN_ONLY;
>  	int fd;
> @@ -428,14 +428,14 @@ void prune_shallow(unsigned options)
>  		strbuf_release(&sb);
>  		return;
>  	}
> -	fd = hold_lock_file_for_update(&shallow_lock,
> +	fd = hold_lock_file_for_update(&shallow_lock.lk,
>  				       git_path_shallow(the_repository),
>  				       LOCK_DIE_ON_ERROR);
>  	check_shallow_file_for_update(the_repository);
>  	if (write_shallow_commits_1(&sb, 0, NULL, flags)) {
>  		if (write_in_full(fd, sb.buf, sb.len) < 0)
>  			die_errno("failed to write to %s",
> -				  get_lock_file_path(&shallow_lock));
> +				  get_lock_file_path(&shallow_lock.lk));
>  		commit_shallow_file(the_repository, &shallow_lock);

There's no obvious helper to extract here either, so this looks good
to me.

With whatever tweaks based on Eric's and Jonathan's reviews seem
appropriate,

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

Thanks.

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Eric Sunshine]

On Wed, Apr 29, 2020 at 11:11 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
> Taylor Blau wrote:
> > +/*
> > + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> > + * which locks can be used with '{commit,rollback}_shallow_file()'.
> > + */
>
> I think I disagree with Eric here: it's useful to have a comment here
> to describe the purpose of the struct (i.e., the "why" as opposed to
> the "what").

I'm not, in general, opposed to the structure being documented; it's
just that the comment, as presented, doesn't seem to add value.

> I wonder if we can go further, though --- when using a shallow_lock,
> how should I think of it as a caller? In some sense, the use of
> 'struct lock_file' is an implementation detail, so we could say:
>
>     /*
>     * Lock for updating the $GIT_DIR/shallow file.
>     *
>     * Use `commit_shallow_file()` to commit an update, or
>     * `rollback_shallow_file()` to roll it back. In either case,
>     * any in-memory cached information about which commits are
>     * shallow will be appropriately invalidated so that future
>     * operations reflect the new state.
>     */
>
> What do you think?

This comment makes more sense and wouldn't have led to me questioning
its usefulness. Thanks.

Re: [PATCH 5/5] shallow: use struct 'shallow_lock' for additional safety

[Taylor Blau]

On Thu, Apr 30, 2020 at 01:32:34AM -0400, Eric Sunshine wrote:
> On Wed, Apr 29, 2020 at 11:11 PM Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Taylor Blau wrote:
> > > +/*
> > > + * shallow_lock is a thin wrapper around 'struct lock_file' in order to restrict
> > > + * which locks can be used with '{commit,rollback}_shallow_file()'.
> > > + */
> >
> > I think I disagree with Eric here: it's useful to have a comment here
> > to describe the purpose of the struct (i.e., the "why" as opposed to
> > the "what").
>
> I'm not, in general, opposed to the structure being documented; it's
> just that the comment, as presented, doesn't seem to add value.
>
> > I wonder if we can go further, though --- when using a shallow_lock,
> > how should I think of it as a caller? In some sense, the use of
> > 'struct lock_file' is an implementation detail, so we could say:
> >
> >     /*
> >     * Lock for updating the $GIT_DIR/shallow file.
> >     *
> >     * Use `commit_shallow_file()` to commit an update, or
> >     * `rollback_shallow_file()` to roll it back. In either case,
> >     * any in-memory cached information about which commits are
> >     * shallow will be appropriately invalidated so that future
> >     * operations reflect the new state.
> >     */
> >
> > What do you think?
>
> This comment makes more sense and wouldn't have led to me questioning
> its usefulness. Thanks.

Me either, thanks for the suggestion.

Thanks,
Taylor