git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / Atom feed
* [PATCH 00/26] Moving global state into the repository object (part 1)
@ 2018-02-13  1:22 Stefan Beller
  2018-02-13  1:22 ` [PATCH 01/26] repository: introduce raw object store field Stefan Beller
                   ` (30 more replies)
  0 siblings, 31 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster; +Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller

This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which 
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (8):
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to map_sha1_file
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (18):
  repository: introduce raw object store field
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c                                    |   2 +-
 builtin/clone.c                                 |   2 +-
 builtin/count-objects.c                         |   6 +-
 builtin/fetch.c                                 |   2 +-
 builtin/fsck.c                                  |  13 ++-
 builtin/gc.c                                    |   4 +-
 builtin/grep.c                                  |   2 +-
 builtin/index-pack.c                            |   1 +
 builtin/merge.c                                 |   2 +-
 builtin/pack-objects.c                          |  21 ++--
 builtin/pack-redundant.c                        |   6 +-
 builtin/receive-pack.c                          |   3 +-
 cache.h                                         |  46 ++------
 contrib/coccinelle/refactoring/packed_git.cocci |   7 ++
 environment.c                                   |   5 +-
 fast-import.c                                   |   6 +-
 http-backend.c                                  |   6 +-
 http-push.c                                     |   1 +
 http-walker.c                                   |   4 +-
 http.c                                          |   6 +-
 mru.h                                           |   1 +
 object-store.h                                  |  75 +++++++++++++
 object.c                                        |  26 +++++
 pack-bitmap.c                                   |   4 +-
 pack-check.c                                    |   1 +
 pack-revindex.c                                 |   1 +
 packfile.c                                      |  64 +++++------
 packfile.h                                      |   2 +-
 path.c                                          |   2 +-
 reachable.c                                     |   1 +
 repository.c                                    |  17 ++-
 repository.h                                    |   7 +-
 server-info.c                                   |   6 +-
 sha1_file.c                                     | 135 +++++++++++++-----------
 sha1_name.c                                     |  11 +-
 streaming.c                                     |   5 +-
 36 files changed, 321 insertions(+), 182 deletions(-)
 create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci
 create mode 100644 object-store.h

-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 01/26] repository: introduce raw object store field
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
                   ` (29 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 17 +++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 42 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index 63ac38a46f..5d7ffb3a3f 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -242,9 +243,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..cf35760ceb
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 0afdfd19b7..9b5b65e189 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index f66fcb1342..bd2ad578de 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
  2018-02-13  1:22 ` [PATCH 01/26] repository: introduce raw object store field Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13 18:51   ` Brandon Williams
  2018-02-13  1:22 ` [PATCH 03/26] object-store: free alt_odb_list Stefan Beller
                   ` (28 subsequent siblings)
  30 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 cache.h        |  4 ++--
 object-store.h |  7 ++++++-
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 6 files changed, 27 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 04846d46f9..1048255da1 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/cache.h b/cache.h
index d8b975a571..918c2f15b4 100644
--- a/cache.h
+++ b/cache.h
@@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -1591,7 +1591,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
diff --git a/object-store.h b/object-store.h
index cf35760ceb..44b8f84753 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,14 +1,19 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+#include "cache.h"
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 4a5fe7ab18..d61076faaf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "mru.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -880,7 +881,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 3da70ac650..2826d5d6ed 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "mru.h"
@@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -673,12 +671,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 03/26] object-store: free alt_odb_list
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
  2018-02-13  1:22 ` [PATCH 01/26] repository: introduce raw object store field Stefan Beller
  2018-02-13  1:22 ` [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 04/26] object-store: move packed_git and packed_git_mru to object store Stefan Beller
                   ` (27 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster; +Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 9b5b65e189..d778f09717 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 04/26] object-store: move packed_git and packed_git_mru to object store
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (2 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 03/26] object-store: free alt_odb_list Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 05/26] object-store: close all packs upon clearing the " Stefan Beller
                   ` (26 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch
    contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
    This semantic patch is placed in a sub directory of the coccinelle
    contrib dir, as this semantic patch is not expected to be of general
    usefulness; it is only useful during developing this series and
    merging it with other topics in flight. At a later date, just
    delete that semantic patch.

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c                         |  6 ++--
 builtin/fsck.c                                  |  7 +++--
 builtin/gc.c                                    |  4 ++-
 builtin/index-pack.c                            |  1 +
 builtin/pack-objects.c                          | 21 ++++++++-----
 builtin/pack-redundant.c                        |  6 ++--
 builtin/receive-pack.c                          |  1 +
 cache.h                                         | 28 ------------------
 contrib/coccinelle/refactoring/packed_git.cocci |  7 +++++
 fast-import.c                                   |  6 ++--
 http-backend.c                                  |  6 ++--
 http-push.c                                     |  1 +
 http-walker.c                                   |  1 +
 http.c                                          |  1 +
 mru.h                                           |  1 +
 object-store.h                                  | 32 +++++++++++++++++++-
 object.c                                        |  6 ++++
 pack-bitmap.c                                   |  4 ++-
 pack-check.c                                    |  1 +
 pack-revindex.c                                 |  1 +
 packfile.c                                      | 39 +++++++++++++------------
 reachable.c                                     |  1 +
 server-info.c                                   |  6 ++--
 sha1_name.c                                     |  5 ++--
 24 files changed, 120 insertions(+), 72 deletions(-)
 create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 1048255da1..d4d249c2ed 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -707,7 +708,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -715,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 3c5eae0edf..6970f325e5 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 4c51aec81f..1902d4e096 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 6b9cfc289d..2ee8b69238 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1023,7 +1025,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			return want;
 	}
 
-	for (entry = packed_git_mru.head; entry; entry = entry->next) {
+	for (entry = the_repository->objects.packed_git_mru.head; entry;
+	     entry = entry->next) {
 		struct packed_git *p = entry->item;
 		off_t offset;
 
@@ -1041,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				mru_mark(&packed_git_mru, entry);
+				mru_mark(&the_repository->objects.packed_git_mru,
+					 entry);
 			if (want != -1)
 				return want;
 		}
@@ -2648,7 +2652,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2711,7 +2715,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2720,7 +2725,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2756,7 +2761,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3119,7 +3124,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3132,7 +3137,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index 918c2f15b4..459fd01dbb 100644
--- a/cache.h
+++ b/cache.h
@@ -1633,34 +1633,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list, which can
- * be iterated instead of packed_git (and marked via mru_mark).
- */
-extern struct mru packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/contrib/coccinelle/refactoring/packed_git.cocci b/contrib/coccinelle/refactoring/packed_git.cocci
new file mode 100644
index 0000000000..da317a51a9
--- /dev/null
+++ b/contrib/coccinelle/refactoring/packed_git.cocci
@@ -0,0 +1,7 @@
+@@ @@
+- packed_git
++ the_repository->objects.packed_git
+
+@@ @@
+- packed_git_mru
++ the_repository->objects.packed_git_mru
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 14435ab65d..34c5d030b7 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 1ae8363de2..a1c6f2639b 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 5977712712..ab989b88dd 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/mru.h b/mru.h
index 42e4aeaa10..d7de81efb9 100644
--- a/mru.h
+++ b/mru.h
@@ -37,6 +37,7 @@ struct mru_entry {
 struct mru {
 	struct mru_entry *head, *tail;
 };
+#define MRU_INIT { NULL, NULL }
 
 void mru_append(struct mru *mru, void *item);
 void mru_mark(struct mru *mru, struct mru_entry *entry);
diff --git a/object-store.h b/object-store.h
index 44b8f84753..8778ab9685 100644
--- a/object-store.h
+++ b/object-store.h
@@ -2,6 +2,7 @@
 #define OBJECT_STORE_H
 
 #include "cache.h"
+#include "mru.h"
 
 struct raw_object_store {
 	/*
@@ -10,11 +11,40 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct mru packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+struct packed_git {
+	struct packed_git *next;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index d778f09717..81d083bec1 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	mru_clear(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index 1b7ebd8d7e..82c3119652 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index d61076faaf..11b0e6613c 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 
@@ -40,8 +41,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-struct mru packed_git_mru;
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -241,7 +240,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -311,7 +310,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -379,7 +378,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -677,8 +676,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -760,7 +759,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -810,7 +810,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -859,17 +859,18 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	mru_clear(&packed_git_mru);
-	for (p = packed_git; p; p = p->next)
-		mru_append(&packed_git_mru, p);
+	mru_clear(&the_repository->objects.packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		mru_append(&the_repository->objects.packed_git_mru, p);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1003,7 +1004,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1835,12 +1836,12 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct mru_entry *p;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	for (p = packed_git_mru.head; p; p = p->next) {
+	for (p = the_repository->objects.packed_git_mru.head; p; p = p->next) {
 		if (fill_pack_entry(sha1, e, p->item)) {
-			mru_mark(&packed_git_mru, p);
+			mru_mark(&the_repository->objects.packed_git_mru, p);
 			return 1;
 		}
 	}
@@ -1887,7 +1888,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if (open_pack_index(p)) {
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 05/26] object-store: close all packs upon clearing the object store
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (3 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 04/26] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 06/26] pack: move prepare_packed_git_run_once to " Stefan Beller
                   ` (25 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster; +Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index acfe9d3c8c..75e16dfec6 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 284651797e..7ec08d4696 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1200,7 +1200,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 7bbcd26faf..70015ccdc7 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1407,7 +1407,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index 81d083bec1..79c2c447bc 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -468,8 +469,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 	o->alt_odb_tail = NULL;
 
 	mru_clear(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index 11b0e6613c..2ce1b2cef7 100644
--- a/packfile.c
+++ b/packfile.c
@@ -306,11 +306,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index 0cdeb54dcd..fa547226b7 100644
--- a/packfile.h
+++ b/packfile.h
@@ -61,7 +61,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 06/26] pack: move prepare_packed_git_run_once to object store
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (4 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 05/26] object-store: close all packs upon clearing the " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 07/26] pack: move approximate object count " Stefan Beller
                   ` (24 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 8778ab9685..9fd109fa9c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,8 +20,14 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 2ce1b2cef7..c107fce171 100644
--- a/packfile.c
+++ b/packfile.c
@@ -873,12 +873,11 @@ static void prepare_packed_git_mru(void)
 		mru_append(&the_repository->objects.packed_git_mru, p);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -886,13 +885,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 07/26] pack: move approximate object count to object store
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (5 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 06/26] pack: move prepare_packed_git_run_once to " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 08/26] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
                   ` (23 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 9fd109fa9c..86f1ec2cd0 100644
--- a/object-store.h
+++ b/object-store.h
@@ -21,13 +21,21 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
 	 */
 	unsigned packed_git_initialized : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index c107fce171..a53706cf16 100644
--- a/packfile.c
+++ b/packfile.c
@@ -793,8 +793,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -804,8 +802,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -815,8 +813,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -890,7 +889,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 08/26] sha1_file: add raw_object_store argument to alt_odb_usable
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (6 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 07/26] pack: move approximate object count " Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 09/26] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
                   ` (22 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster; +Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 2826d5d6ed..d25b68a8fb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -350,7 +350,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -366,7 +368,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -418,7 +420,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 09/26] sha1_file: add repository argument to link_alt_odb_entry
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (7 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 08/26] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 10/26] sha1_file: add repository argument to read_info_alternates Stefan Beller
                   ` (21 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index d25b68a8fb..7dc5f690e2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -394,8 +394,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -492,7 +493,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 10/26] sha1_file: add repository argument to read_info_alternates
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (8 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 09/26] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 11/26] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
                   ` (20 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 7dc5f690e2..4fdfdd945a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -393,7 +393,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -434,7 +436,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -500,7 +502,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -684,7 +687,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 11/26] sha1_file: add repository argument to link_alt_odb_entries
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (9 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 10/26] sha1_file: add repository argument to read_info_alternates Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 12/26] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
                   ` (19 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 4fdfdd945a..b090f403d8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -471,8 +471,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -515,7 +519,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -569,7 +573,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -582,7 +587,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -685,7 +691,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 12/26] sha1_file: add repository argument to prepare_alt_odb
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (10 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 11/26] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 13/26] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
                   ` (18 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

See previous patch for explanation.

While at it, move the declaration to object-store.h,
where it should be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 cache.h        |  1 -
 object-store.h |  3 +++
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  3 ++-
 6 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index d4d249c2ed..00ec8eecf0 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -695,7 +695,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/cache.h b/cache.h
index 459fd01dbb..70518e24ce 100644
--- a/cache.h
+++ b/cache.h
@@ -1592,7 +1592,6 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 extern int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/object-store.h b/object-store.h
index 86f1ec2cd0..d96a16edd1 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,4 +61,7 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+extern void prepare_alt_odb_the_repository(void);
+
 #endif /* OBJECT_STORE_H */
diff --git a/packfile.c b/packfile.c
index a53706cf16..9c49b34d2b 100644
--- a/packfile.c
+++ b/packfile.c
@@ -879,7 +879,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index b090f403d8..d7f271fe6e 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "mru.h"
@@ -585,7 +586,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -671,7 +672,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -680,7 +681,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -729,7 +730,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -884,7 +885,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -911,7 +912,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..3e490ee8f6 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 13/26] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (11 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 12/26] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 14/26] sha1_file: allow prepare_alt_odb " Stefan Beller
                   ` (17 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h     |  4 ++++
 sha1_file.c | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/cache.h b/cache.h
index 70518e24ce..09d37cb651 100644
--- a/cache.h
+++ b/cache.h
@@ -1590,6 +1590,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
diff --git a/sha1_file.c b/sha1_file.c
index d7f271fe6e..d18ce3aeba 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -394,11 +394,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -424,7 +423,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -432,12 +431,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -472,12 +471,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -491,7 +486,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -500,15 +495,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -520,7 +516,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (12 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 13/26] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-14  0:37   ` Duy Nguyen
  2018-02-13  1:22 ` [PATCH 15/26] sha1_file: add repository argument to sha1_file_name Stefan Beller
                   ` (16 subsequent siblings)
  30 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster; +Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 21 +++++++++++----------
 2 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/object-store.h b/object-store.h
index d96a16edd1..add1d4e27c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,7 +61,6 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-extern void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index d18ce3aeba..f046d560f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+
+	if (!r->ignore_env) {
+		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+		if (!alt)
+			alt = "";
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	}
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 15/26] sha1_file: add repository argument to sha1_file_name
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (13 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 14/26] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 16/26] sha1_file: add repository argument to stat_sha1_file Stefan Beller
                   ` (15 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  8 --------
 http-walker.c  |  3 ++-
 http.c         |  5 +++--
 object-store.h |  9 +++++++++
 sha1_file.c    | 11 ++++++-----
 5 files changed, 20 insertions(+), 16 deletions(-)

diff --git a/cache.h b/cache.h
index 09d37cb651..4b01c0de10 100644
--- a/cache.h
+++ b/cache.h
@@ -956,14 +956,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Return the name of the file in the local object database that would
- * be used to store a loose object with the specified sha1.  The
- * return value is a pointer to a statically allocated buffer that is
- * overwritten each time the function is called.
- */
-extern const char *sha1_file_name(const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index a1c6f2639b..96873bdfed 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		ret = error("unable to write sha1 filename %s",
-			    sha1_file_name(req->sha1));
+			    sha1_file_name(the_repository, req->sha1));
 	}
 
 	release_http_object_request(req);
diff --git a/http.c b/http.c
index ab989b88dd..c7d2b74ce2 100644
--- a/http.c
+++ b/http.c
@@ -2181,7 +2181,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	filename = sha1_file_name(sha1);
+	filename = sha1_file_name(the_repository, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename);
 
@@ -2329,7 +2329,8 @@ int finish_http_object_request(struct http_object_request *freq)
 		return -1;
 	}
 	freq->rename =
-		finalize_object_file(freq->tmpfile, sha1_file_name(freq->sha1));
+		finalize_object_file(freq->tmpfile,
+				     sha1_file_name(the_repository, freq->sha1));
 
 	return freq->rename;
 }
diff --git a/object-store.h b/object-store.h
index add1d4e27c..5422e80c08 100644
--- a/object-store.h
+++ b/object-store.h
@@ -61,6 +61,15 @@ struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 };
 
+/*
+ * Return the name of the file in a repository's local object database
+ * that would be used to store a loose object with the specified sha1.
+ * The return value is a pointer to a statically allocated buffer that
+ * is overwritten each time the function is called.
+ */
+#define sha1_file_name(r, s) sha1_file_name_##r(s)
+const char *sha1_file_name_the_repository(const unsigned char *sha1);
+
 void prepare_alt_odb(struct repository *r);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index f046d560f8..65fefdf4ac 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-const char *sha1_file_name(const unsigned char *sha1)
+const char *sha1_file_name_the_repository(const unsigned char *sha1)
 {
 	static struct strbuf buf = STRBUF_INIT;
 
@@ -721,7 +721,8 @@ int check_and_freshen_file(const char *fn, int freshen)
 
 static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 {
-	return check_and_freshen_file(sha1_file_name(sha1), freshen);
+	return check_and_freshen_file(sha1_file_name(the_repository, sha1),
+				      freshen);
 }
 
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
@@ -878,7 +879,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 {
 	struct alternate_object_database *alt;
 
-	*path = sha1_file_name(sha1);
+	*path = sha1_file_name(the_repository, sha1);
 	if (!lstat(*path, st))
 		return 0;
 
@@ -903,7 +904,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	struct alternate_object_database *alt;
 	int most_interesting_errno;
 
-	*path = sha1_file_name(sha1);
+	*path = sha1_file_name(the_repository, sha1);
 	fd = git_open(*path);
 	if (fd >= 0)
 		return fd;
@@ -1568,7 +1569,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	git_SHA_CTX c;
 	unsigned char parano_sha1[20];
 	static struct strbuf tmp_file = STRBUF_INIT;
-	const char *filename = sha1_file_name(sha1);
+	const char *filename = sha1_file_name(the_repository, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename);
 	if (fd < 0) {
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 16/26] sha1_file: add repository argument to stat_sha1_file
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (14 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 15/26] sha1_file: add repository argument to sha1_file_name Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 17/26] sha1_file: add repository argument to open_sha1_file Stefan Beller
                   ` (14 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 65fefdf4ac..1d81bafe56 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 
@@ -1173,7 +1174,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1372,7 +1373,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 17/26] sha1_file: add repository argument to open_sha1_file
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (15 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 16/26] sha1_file: add repository argument to stat_sha1_file Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 18/26] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
                   ` (13 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1d81bafe56..e9e01fb471 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -899,7 +899,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -938,7 +940,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 18/26] sha1_file: add repository argument to map_sha1_file_1
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (16 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 17/26] sha1_file: add repository argument to open_sha1_file Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 19/26] sha1_file: add repository argument to map_sha1_file Stefan Beller
                   ` (12 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index e9e01fb471..f706526bc2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -930,9 +930,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -961,7 +962,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2173,7 +2174,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 19/26] sha1_file: add repository argument to map_sha1_file
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (17 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 18/26] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 20/26] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
                   ` (11 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 2 ++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index 4b01c0de10..d0a1236dac 100644
--- a/cache.h
+++ b/cache.h
@@ -1237,7 +1237,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 5422e80c08..0a4561f476 100644
--- a/object-store.h
+++ b/object-store.h
@@ -69,6 +69,8 @@ struct packed_git {
  */
 #define sha1_file_name(r, s) sha1_file_name_##r(s)
 const char *sha1_file_name_the_repository(const unsigned char *sha1);
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
 
 void prepare_alt_odb(struct repository *r);
 
diff --git a/sha1_file.c b/sha1_file.c
index f706526bc2..1ce4058644 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -960,7 +960,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1184,7 +1184,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..22d27df55e 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 20/26] sha1_file: add repository argument to sha1_loose_object_info
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (18 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 19/26] sha1_file: add repository argument to map_sha1_file Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 21/26] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
                   ` (10 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1ce4058644..d0bc09a089 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1151,9 +1151,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1266,7 +1267,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 
 	if (!find_pack_entry(real, &e)) {
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 21/26] sha1_file: allow sha1_file_name to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (19 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 20/26] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
@ 2018-02-13  1:22 ` Stefan Beller
  2018-02-13  1:22 ` [PATCH 22/26] sha1_file: allow stat_sha1_file " Stefan Beller
                   ` (9 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index 0a4561f476..b4756444bb 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,8 +67,7 @@ struct packed_git {
  * The return value is a pointer to a statically allocated buffer that
  * is overwritten each time the function is called.
  */
-#define sha1_file_name(r, s) sha1_file_name_##r(s)
-const char *sha1_file_name_the_repository(const unsigned char *sha1);
+const char *sha1_file_name(struct repository *r, const unsigned char *sha1);
 #define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
 void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
 
diff --git a/sha1_file.c b/sha1_file.c
index d0bc09a089..fddada5756 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,12 +323,12 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-const char *sha1_file_name_the_repository(const unsigned char *sha1)
+const char *sha1_file_name(struct repository *r, const unsigned char *sha1)
 {
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	strbuf_addf(&buf, "%s/", get_object_directory());
+	strbuf_addf(&buf, "%s/", r->objects.objectdir);
 
 	fill_sha1_path(&buf, sha1);
 	return buf.buf;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 22/26] sha1_file: allow stat_sha1_file to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (20 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 21/26] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 23/26] sha1_file: allow open_sha1_file " Stefan Beller
                   ` (8 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index fddada5756..21ddbff846 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,19 +874,18 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
-static int stat_sha1_file_the_repository(const unsigned char *sha1,
-					 struct stat *st, const char **path)
+static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 
-	*path = sha1_file_name(the_repository, sha1);
+	*path = sha1_file_name(r, sha1);
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(the_repository);
+	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 23/26] sha1_file: allow open_sha1_file to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (21 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 22/26] sha1_file: allow stat_sha1_file " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 24/26] sha1_file: allow map_sha1_file_1 " Stefan Beller
                   ` (7 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 21ddbff846..50202f0959 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -898,22 +898,21 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
-static int open_sha1_file_the_repository(const unsigned char *sha1,
-					 const char **path)
+static int open_sha1_file(struct repository *r,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
 	int most_interesting_errno;
 
-	*path = sha1_file_name(the_repository, sha1);
+	*path = sha1_file_name(r, sha1);
 	fd = git_open(*path);
 	if (fd >= 0)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 24/26] sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (22 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 23/26] sha1_file: allow open_sha1_file " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 25/26] sha1_file: allow map_sha1_file " Stefan Beller
                   ` (6 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 50202f0959..86b0ca7089 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -928,10 +928,8 @@ static int open_sha1_file(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
-static void *map_sha1_file_1_the_repository(const char *path,
-					    const unsigned char *sha1,
-					    unsigned long *size)
+static void *map_sha1_file_1(struct repository *r, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -939,7 +937,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(the_repository, sha1, &path);
+		fd = open_sha1_file(r, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 25/26] sha1_file: allow map_sha1_file to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (23 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 24/26] sha1_file: allow map_sha1_file_1 " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:22 ` [PATCH 26/26] sha1_file: allow sha1_loose_object_info " Stefan Beller
                   ` (5 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Stefan Beller, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 5 +++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index b4756444bb..f164c4e5c9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -68,8 +68,7 @@ struct packed_git {
  * is overwritten each time the function is called.
  */
 const char *sha1_file_name(struct repository *r, const unsigned char *sha1);
-#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+extern void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
 void prepare_alt_odb(struct repository *r);
 
diff --git a/sha1_file.c b/sha1_file.c
index 86b0ca7089..6237d59a59 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -956,9 +956,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,
 	return map;
 }
 
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct repository *r,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(the_repository, NULL, sha1, size);
+	return map_sha1_file_1(r, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 26/26] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (24 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 25/26] sha1_file: allow map_sha1_file " Stefan Beller
@ 2018-02-13  1:22 ` " Stefan Beller
  2018-02-13  1:38 ` [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (4 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:22 UTC (permalink / raw)
  To: gitster
  Cc: git, pclouds, sunshine, jonathantanmy, Jonathan Nieder, Stefan Beller

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6237d59a59..c3f35914ce 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1148,10 +1148,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
-static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
-						 struct object_info *oi,
-						 int flags)
+static int sha1_loose_object_info(struct repository *r,
+				  const unsigned char *sha1,
+				  struct object_info *oi, int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1175,14 +1174,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
+		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(the_repository, sha1, &mapsize);
+	map = map_sha1_file(r, sha1, &mapsize);
 	if (!map)
 		return -1;
 
-- 
2.16.1.73.ga2c3e9663f.dirty


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (25 preceding siblings ...)
  2018-02-13  1:22 ` [PATCH 26/26] sha1_file: allow sha1_loose_object_info " Stefan Beller
@ 2018-02-13  1:38 ` Stefan Beller
  2018-02-13 11:49 ` Duy Nguyen
                   ` (3 subsequent siblings)
  30 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13  1:38 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: git, Duy Nguyen, Eric Sunshine, Jonathan Tan, Stefan Beller

On Mon, Feb 12, 2018 at 5:22 PM, Stefan Beller <sbeller@google.com> wrote:

> I developed this series [...] on top of current master.

also available at https://github.com/stefanbeller/git/tree/object-store-part1

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (26 preceding siblings ...)
  2018-02-13  1:38 ` [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
@ 2018-02-13 11:49 ` Duy Nguyen
  2018-02-13 12:13   ` Duy Nguyen
  2018-02-13 19:26 ` Jonathan Nieder
                   ` (2 subsequent siblings)
  30 siblings, 1 reply; 239+ messages in thread
From: Duy Nguyen @ 2018-02-13 11:49 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, sunshine, jonathantanmy

On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
> This is a real take on the first part of the recent RFC[1].
>
> ...
> 
> Duy suggested that we shall not use the repository blindly, but
> should carefully examine whether to pass on an object store or the
> refstore or such[4], which I agree with if it makes sense. This
> series unfortunately has an issue with that as I would not want to
> pass down the `ignore_env` flag separately from the object store, so
> I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full
> repository.

Second proposal :) How about you store ignore_env in raw_object_store?
This would not be the first time an object has some configuration
passed in at construction time. And it has a "constructor" now,
raw_object_store_init() (I probably should merge _setup in it too)

The core changes look like this. I have a full commit on top of your
series [1] that keeps sha1_file.c functions take 'struct
raw_object_store' instead.

[1] https://github.com/pclouds/git/tree/object-store-part1

-- 8< --
diff --git a/object-store.h b/object-store.h
index f164c4e5c9..a8bf1738f2 100644
--- a/object-store.h
+++ b/object-store.h
@@ -34,9 +34,13 @@ struct raw_object_store {
 	 * packs.
 	 */
 	unsigned packed_git_initialized : 1;
+
+	unsigned ignore_env : 1;
 };
 #define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_INIT, NULL, NULL, 0, 0, 0 }
 
+void raw_object_store_init(struct raw_object_store *ros, struct repository *r);
+void raw_object_store_setup(struct raw_object_store *ros, char *);
 void raw_object_store_clear(struct raw_object_store *o);
 
 struct packed_git {
diff --git a/object.c b/object.c
index 79c2c447bc..a4534bf4c4 100644
--- a/object.c
+++ b/object.c
@@ -461,6 +461,20 @@ static void free_alt_odbs(struct raw_object_store *o)
 	}
 }
 
+void raw_object_store_init(struct raw_object_store *o,
+			   struct repository *r)
+{
+	/* FIXME: memset? */
+	o->ignore_env = r->ignore_env;
+}
+
+void raw_object_store_setup(struct raw_object_store *o,
+			    char *objectdir)
+{
+	free(o->objectdir);
+	o->objectdir = objectdir;
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
diff --git a/repository.c b/repository.c
index bd2ad578de..ac5863d7e1 100644
--- a/repository.c
+++ b/repository.c
@@ -49,9 +49,9 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objects.objectdir);
-	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-					    "objects", !repo->ignore_env);
+	raw_object_store_setup(&repo->objects,
+			       git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+						 "objects", !repo->ignore_env));
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	raw_object_store_init(&repo->objects, repo);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/sha1_file.c b/sha1_file.c
index c3f35914ce..3be58651a1 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -677,14 +677,14 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(struct repository *r)
+void prepare_alt_odb(struct raw_object_store *ros)
 {
 	if (r->objects.alt_odb_tail)
 		return;
 
 	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
 
-	if (!r->ignore_env) {
+	if (!ros->ignore_env) {
 		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 		if (!alt)
 			alt = "";
-- 8< --


--
Duy

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 11:49 ` Duy Nguyen
@ 2018-02-13 12:13   ` Duy Nguyen
  2018-02-13 16:52     ` Brandon Williams
  2018-02-13 17:47     ` Stefan Beller
  0 siblings, 2 replies; 239+ messages in thread
From: Duy Nguyen @ 2018-02-13 12:13 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Junio C Hamano, Git Mailing List, Eric Sunshine, Jonathan Tan

On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
>> This is a real take on the first part of the recent RFC[1].
>>
>> ...
>>
>> Duy suggested that we shall not use the repository blindly, but
>> should carefully examine whether to pass on an object store or the
>> refstore or such[4], which I agree with if it makes sense. This
>> series unfortunately has an issue with that as I would not want to
>> pass down the `ignore_env` flag separately from the object store, so
>> I made all functions that only take the object store to have the raw
>> object store as the first parameter, and others using the full
>> repository.
>
> Second proposal :) How about you store ignore_env in raw_object_store?
> This would not be the first time an object has some configuration
> passed in at construction time. And it has a "constructor" now,
> raw_object_store_init() (I probably should merge _setup in it too)

A bit more on this configuration parameters. Down the road I think we
need something like this anyway to delete global config vars like
packed_git_window_size, delta_base_cache_limit...  Either all these
end up in raw_object_store, or raw_object_store holds a link to
"struct config_set".

The ignore_env specifically though looks to me like a stop gap
solution until everything goes through repo_init() first. At that
point we don't have to delay getenv() anymore. We can getenv() all at
repo_init() then pass them in raw_object_store and ignore_env should
be gone. So sticking it inside raw_object_store _temporarily_ does not
sound so bad.
-- 
Duy

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 12:13   ` Duy Nguyen
@ 2018-02-13 16:52     ` Brandon Williams
  2018-02-13 17:47     ` Stefan Beller
  1 sibling, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-13 16:52 UTC (permalink / raw)
  To: Duy Nguyen
  Cc: Stefan Beller, Junio C Hamano, Git Mailing List, Eric Sunshine,
	Jonathan Tan

On 02/13, Duy Nguyen wrote:
> On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
> > On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
> >> This is a real take on the first part of the recent RFC[1].
> >>
> >> ...
> >>
> >> Duy suggested that we shall not use the repository blindly, but
> >> should carefully examine whether to pass on an object store or the
> >> refstore or such[4], which I agree with if it makes sense. This
> >> series unfortunately has an issue with that as I would not want to
> >> pass down the `ignore_env` flag separately from the object store, so
> >> I made all functions that only take the object store to have the raw
> >> object store as the first parameter, and others using the full
> >> repository.
> >
> > Second proposal :) How about you store ignore_env in raw_object_store?
> > This would not be the first time an object has some configuration
> > passed in at construction time. And it has a "constructor" now,
> > raw_object_store_init() (I probably should merge _setup in it too)
> 
> A bit more on this configuration parameters. Down the road I think we
> need something like this anyway to delete global config vars like
> packed_git_window_size, delta_base_cache_limit...  Either all these
> end up in raw_object_store, or raw_object_store holds a link to
> "struct config_set".
> 
> The ignore_env specifically though looks to me like a stop gap
> solution until everything goes through repo_init() first. At that
> point we don't have to delay getenv() anymore. We can getenv() all at
> repo_init() then pass them in raw_object_store and ignore_env should
> be gone. So sticking it inside raw_object_store _temporarily_ does not
> sound so bad.

I like this approach, I mean at the moment we are replicating a single
bit of data but that allows us to be able to limit the scope of where a
repository struct is passed, giving us a better abstraction layer.

> -- 
> Duy

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 12:13   ` Duy Nguyen
  2018-02-13 16:52     ` Brandon Williams
@ 2018-02-13 17:47     ` Stefan Beller
  2018-02-13 18:57       ` Junio C Hamano
  1 sibling, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-13 17:47 UTC (permalink / raw)
  To: Duy Nguyen; +Cc: Junio C Hamano, Git Mailing List, Eric Sunshine, Jonathan Tan

On Tue, Feb 13, 2018 at 4:13 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, Feb 13, 2018 at 6:49 PM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Mon, Feb 12, 2018 at 05:22:15PM -0800, Stefan Beller wrote:
>>> This is a real take on the first part of the recent RFC[1].
>>>
>>> ...
>>>
>>> Duy suggested that we shall not use the repository blindly, but
>>> should carefully examine whether to pass on an object store or the
>>> refstore or such[4], which I agree with if it makes sense. This
>>> series unfortunately has an issue with that as I would not want to
>>> pass down the `ignore_env` flag separately from the object store, so
>>> I made all functions that only take the object store to have the raw
>>> object store as the first parameter, and others using the full
>>> repository.
>>
>> Second proposal :) How about you store ignore_env in raw_object_store?
>> This would not be the first time an object has some configuration
>> passed in at construction time. And it has a "constructor" now,
>> raw_object_store_init() (I probably should merge _setup in it too)
>
> A bit more on this configuration parameters. Down the road I think we
> need something like this anyway to delete global config vars like
> packed_git_window_size, delta_base_cache_limit...  Either all these
> end up in raw_object_store, or raw_object_store holds a link to
> "struct config_set".

That makes sense long term.

> The ignore_env specifically though looks to me like a stop gap
> solution until everything goes through repo_init() first. At that
> point we don't have to delay getenv() anymore. We can getenv() all at
> repo_init() then pass them in raw_object_store and ignore_env should
> be gone. So sticking it inside raw_object_store _temporarily_ does not
> sound so bad.

Oh, that is an interesting perspective. Here is how I arrived at the opposite
conclusion initially: Searching for 'ignore_env' shows that we care about it
as well for the index and graft paths, which are not the object store, hence
it would be better kept in the repository. (The alternative would be to
duplicate it into the raw object store, but I do not like data duplication)

But maybe it is better to duplicate this one bit instead of passing through
a larger scoped object.

I'll rework the patches.

Thanks!
Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-13  1:22 ` [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-13 18:51   ` Brandon Williams
  2018-02-13 19:46     ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Brandon Williams @ 2018-02-13 18:51 UTC (permalink / raw)
  To: Stefan Beller
  Cc: gitster, git, pclouds, sunshine, jonathantanmy, Jonathan Nieder

On 02/12, Stefan Beller wrote:
> In a process with multiple repositories open, alternates should be
> associated to a single repository and not shared globally. Move
> alt_odb_list and alt_odb_tail into the_repository and adjust callers
> to reflect this.
> 
> Now that the alternative object data base is per repository, we're
> leaking its memory upon freeing a repository. The next patch plugs
> this hole.
> 
> No functional change intended.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  builtin/fsck.c |  4 +++-
>  cache.h        |  4 ++--
>  object-store.h |  7 ++++++-
>  packfile.c     |  3 ++-
>  sha1_file.c    | 25 ++++++++++++-------------
>  sha1_name.c    |  3 ++-
>  6 files changed, 27 insertions(+), 19 deletions(-)
> 
> diff --git a/builtin/fsck.c b/builtin/fsck.c
> index 04846d46f9..1048255da1 100644
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -1,5 +1,6 @@
>  #include "builtin.h"
>  #include "cache.h"
> +#include "repository.h"
>  #include "config.h"
>  #include "commit.h"
>  #include "tree.h"
> @@ -694,7 +695,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  		fsck_object_dir(get_object_directory());
>  
>  		prepare_alt_odb();
> -		for (alt = alt_odb_list; alt; alt = alt->next)
> +		for (alt = the_repository->objects.alt_odb_list;
> +				alt; alt = alt->next)
>  			fsck_object_dir(alt->path);
>  
>  		if (check_full) {
> diff --git a/cache.h b/cache.h
> index d8b975a571..918c2f15b4 100644
> --- a/cache.h
> +++ b/cache.h
> @@ -1573,7 +1573,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
>  extern void schedule_dir_for_removal(const char *name, int len);
>  extern void remove_scheduled_dirs(void);
>  
> -extern struct alternate_object_database {
> +struct alternate_object_database {
>  	struct alternate_object_database *next;
>  
>  	/* see alt_scratch_buf() */
> @@ -1591,7 +1591,7 @@ extern struct alternate_object_database {
>  	struct oid_array loose_objects_cache;
>  
>  	char path[FLEX_ARRAY];
> -} *alt_odb_list;
> +};
>  extern void prepare_alt_odb(void);
>  extern char *compute_alternate_path(const char *path, struct strbuf *err);
>  typedef int alt_odb_fn(struct alternate_object_database *, void *);
> diff --git a/object-store.h b/object-store.h
> index cf35760ceb..44b8f84753 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -1,14 +1,19 @@
>  #ifndef OBJECT_STORE_H
>  #define OBJECT_STORE_H
>  
> +#include "cache.h"
> +

Maybe we would want to move the definition of the alternate store to
this header file?  That way we don't need to include cache.h here.

>  struct raw_object_store {
>  	/*
>  	 * Path to the repository's object store.
>  	 * Cannot be NULL after initialization.
>  	 */
>  	char *objectdir;
> +
> +	struct alternate_object_database *alt_odb_list;
> +	struct alternate_object_database **alt_odb_tail;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
>  
>  void raw_object_store_clear(struct raw_object_store *o);
>  
> diff --git a/packfile.c b/packfile.c
> index 4a5fe7ab18..d61076faaf 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "mru.h"
>  #include "pack.h"
> +#include "repository.h"
>  #include "dir.h"
>  #include "mergesort.h"
>  #include "packfile.h"
> @@ -880,7 +881,7 @@ void prepare_packed_git(void)
>  		return;
>  	prepare_packed_git_one(get_object_directory(), 1);
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next)
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
>  		prepare_packed_git_one(alt->path, 0);
>  	rearrange_packed_git();
>  	prepare_packed_git_mru();
> diff --git a/sha1_file.c b/sha1_file.c
> index 3da70ac650..2826d5d6ed 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -22,6 +22,7 @@
>  #include "pack-revindex.h"
>  #include "sha1-lookup.h"
>  #include "bulk-checkin.h"
> +#include "repository.h"
>  #include "streaming.h"
>  #include "dir.h"
>  #include "mru.h"
> @@ -346,9 +347,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
>  	return buf->buf;
>  }
>  
> -struct alternate_object_database *alt_odb_list;
> -static struct alternate_object_database **alt_odb_tail;
> -
>  /*
>   * Return non-zero iff the path is usable as an alternate object database.
>   */
> @@ -368,7 +366,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
>  	 * Prevent the common mistake of listing the same
>  	 * thing twice, or object directory itself.
>  	 */
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		if (!fspathcmp(path->buf, alt->path))
>  			return 0;
>  	}
> @@ -428,8 +426,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
>  	ent = alloc_alt_odb(pathbuf.buf);
>  
>  	/* add the alternate entry */
> -	*alt_odb_tail = ent;
> -	alt_odb_tail = &(ent->next);
> +	*the_repository->objects.alt_odb_tail = ent;
> +	the_repository->objects.alt_odb_tail = &(ent->next);
>  	ent->next = NULL;
>  
>  	/* recursively add alternates */
> @@ -563,7 +561,7 @@ void add_to_alternates_file(const char *reference)
>  		fprintf_or_die(out, "%s\n", reference);
>  		if (commit_lock_file(&lock))
>  			die_errno("unable to move new alternates file into place");
> -		if (alt_odb_tail)
> +		if (the_repository->objects.alt_odb_tail)
>  			link_alt_odb_entries(reference, '\n', NULL, 0);
>  	}
>  	free(alts);
> @@ -661,7 +659,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
>  	int r = 0;
>  
>  	prepare_alt_odb();
> -	for (ent = alt_odb_list; ent; ent = ent->next) {
> +	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
>  		r = fn(ent, cb);
>  		if (r)
>  			break;
> @@ -673,12 +671,13 @@ void prepare_alt_odb(void)
>  {
>  	const char *alt;
>  
> -	if (alt_odb_tail)
> +	if (the_repository->objects.alt_odb_tail)
>  		return;
>  
>  	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>  
> -	alt_odb_tail = &alt_odb_list;
> +	the_repository->objects.alt_odb_tail =
> +			&the_repository->objects.alt_odb_list;
>  	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
>  
>  	read_info_alternates(get_object_directory(), 0);
> @@ -717,7 +716,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
>  {
>  	struct alternate_object_database *alt;
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		const char *path = alt_sha1_path(alt, sha1);
>  		if (check_and_freshen_file(path, freshen))
>  			return 1;
> @@ -873,7 +872,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
>  
>  	prepare_alt_odb();
>  	errno = ENOENT;
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		*path = alt_sha1_path(alt, sha1);
>  		if (!lstat(*path, st))
>  			return 0;
> @@ -899,7 +898,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
>  	most_interesting_errno = errno;
>  
>  	prepare_alt_odb();
> -	for (alt = alt_odb_list; alt; alt = alt->next) {
> +	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
>  		*path = alt_sha1_path(alt, sha1);
>  		fd = git_open(*path);
>  		if (fd >= 0)
> diff --git a/sha1_name.c b/sha1_name.c
> index 611c7d24dd..957ce25680 100644
> --- a/sha1_name.c
> +++ b/sha1_name.c
> @@ -10,6 +10,7 @@
>  #include "dir.h"
>  #include "sha1-array.h"
>  #include "packfile.h"
> +#include "repository.h"
>  
>  static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
>  
> @@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
>  		 */
>  		fakeent = alloc_alt_odb(get_object_directory());
>  	}
> -	fakeent->next = alt_odb_list;
> +	fakeent->next = the_repository->objects.alt_odb_list;
>  
>  	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
>  		int pos;
> -- 
> 2.16.1.73.ga2c3e9663f.dirty
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 17:47     ` Stefan Beller
@ 2018-02-13 18:57       ` Junio C Hamano
  2018-02-13 19:23         ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-13 18:57 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Duy Nguyen, Git Mailing List, Eric Sunshine, Jonathan Tan

Stefan Beller <sbeller@google.com> writes:

> Oh, that is an interesting perspective. Here is how I arrived at the opposite
> conclusion initially: Searching for 'ignore_env' shows that we care about it
> as well for the index and graft paths, which are not the object store, hence
> it would be better kept in the repository. (The alternative would be to
> duplicate it into the raw object store, but I do not like data duplication)
>
> But maybe it is better to duplicate this one bit instead of passing through
> a larger scoped object.

If a larger scoped repository refers to a smaller scoped
object-store, is there still a need to duplicate that bit, instead
of referring to the bit the smaller scoped one has when asking about
the bit in the larger scoped one?

I am not sure if these "do not look at environment variables" is an
attribute of these entities---it sounds more like an attribute for
each invocation of an operation, i.e. "I want to learn where the
index is but would ignore GIT_INDEX environment for this particular
query." and "What's the value of this ref?  Please honor the
common-dir environment during this query".

So from that point of view, it may not matter where the "bit" lives,
among repository, object-store, or ref-store.


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 18:57       ` Junio C Hamano
@ 2018-02-13 19:23         ` Stefan Beller
  2018-02-13 19:35           ` Junio C Hamano
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-13 19:23 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Duy Nguyen, Git Mailing List, Eric Sunshine, Jonathan Tan

On Tue, Feb 13, 2018 at 10:57 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> Oh, that is an interesting perspective. Here is how I arrived at the opposite
>> conclusion initially: Searching for 'ignore_env' shows that we care about it
>> as well for the index and graft paths, which are not the object store, hence
>> it would be better kept in the repository. (The alternative would be to
>> duplicate it into the raw object store, but I do not like data duplication)
>>
>> But maybe it is better to duplicate this one bit instead of passing through
>> a larger scoped object.
>
> If a larger scoped repository refers to a smaller scoped
> object-store, is there still a need to duplicate that bit, instead
> of referring to the bit the smaller scoped one has when asking about
> the bit in the larger scoped one?

No (in theory). But in practice it may be worthwhile:

    "What's the value of this ref?"

    "Oh let me check the ignore_env bit that happens
    to live in the object store first."

would be super confusing to me.

> I am not sure if these "do not look at environment variables" is an
> attribute of these entities---it sounds more like an attribute for
> each invocation of an operation, i.e. "I want to learn where the
> index is but would ignore GIT_INDEX environment for this particular
> query." and "What's the value of this ref?  Please honor the
> common-dir environment during this query".

That sounds like we want to have a configset struct eventually.

For now the ignore_env bit lives in the repository, as that helps
when working with submodules, when reading its comments.
Unfortunately 359efeffc1 (repository: introduce the repository
object, 2017-06-22) did not reason about the existence of the ignore_env
flag in its commit message.

> So from that point of view, it may not matter where the "bit" lives,
> among repository, object-store, or ref-store.

It matters on the scale of confusing the developer?

Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (27 preceding siblings ...)
  2018-02-13 11:49 ` Duy Nguyen
@ 2018-02-13 19:26 ` Jonathan Nieder
  2018-02-14  0:57   ` Duy Nguyen
  2018-02-13 19:33 ` Brandon Williams
  2018-02-15 20:42 ` Junio C Hamano
  30 siblings, 1 reply; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-13 19:26 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, pclouds, sunshine, jonathantanmy

Hi,

Stefan Beller wrote:

> This is a real take on the first part of the recent RFC[1].
>
> Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
> might be a good breaking point for a first part at that RFC at patch 38.
> This series is smaller and contains only 26 patches as the patches in the big
> RFC were slightly out of order.

Thanks.  This looks like a nice reviewable series, so I'm happy to see
it broken out.

[...]
> Comments in the early range of that RFC were on 003 where Junio pointed out
> that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
> in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

Can you say a little more about this?  Was the problem that the
semantic patch wasn't idempotent, that it was too slow to run, or
something else?

If we're including the semantic patch for reference but never running
it, I think I'd prefer it to go in the commit message.  But if it's
useful to run then we should make it idempotent so it can go in
contrib/coccinelle.

[...]
> Duy suggested that we shall not use the repository blindly, but should carefully
> examine whether to pass on an object store or the refstore or such[4], which
> I agree with if it makes sense. This series unfortunately has an issue with that
> as I would not want to pass down the `ignore_env` flag separately from the object
> store, so I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full repository.

I think I want to push back on this a little.

The advantage of a function taking e.g. an object_store as an argument
instead of a repository is that it increases its flexibility, since it
allows callers that do not have access to a repository to call it.  The
disadvantage is also that it increases the flexibility without any
callers benefitting from that:

 1. It ties us to assumptions from today.  If e.g. an object access in
    the future starts relying on some other information from the
    repository (e.g. its config) then we'd have to either add a
    back-pointer from the object store to its repository or add
    additional arguments for that additional data at that point.

    If all callers already have a repository, it is simpler to pass
    that repository as context so that we have the flexibility to make
    more use of it later.

 2. It complicates the caller.  Instead of consistently passing the
    same repository argument as context to functions that access that
    repository, the caller would have to pull out relevant fields like
    the object store from it.

 3. It prevents us from making opportunistic use of other information
    from the repository, such as its name for use in error messages.

In lower-level funcitons that need to be usable by callers without a
repository (e.g. to find packfiles in an alternate) it makes sense to
not pass a repository, but without such a use case in mind I don't
think it needs to be a general goal.

To put it another way, most callers do not *care* whether they are
working with a repository's object store, ref database, or some other
aspect of the repository.  They just know they want to e.g. read an
object from this repository.

It's similar to how FILE * works: some operations rely on the buffer
the FILE * manages and some other operations only rely on the
underlying file descriptor, but using the FILE * consistently provides
a clean abstraction that generally makes life easier.

> Eric Sunshine brought up memory leaks with the RFC, and I would think to
> have plugged all holes.

Yay, thank you!

I'll try to find time to look at the patches in detail soon, but no
promises (i.e. if someone else reviews them first, then even better
;-)).

Sincerely,
Jonathan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (28 preceding siblings ...)
  2018-02-13 19:26 ` Jonathan Nieder
@ 2018-02-13 19:33 ` Brandon Williams
  2018-02-15 20:42 ` Junio C Hamano
  30 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-13 19:33 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, pclouds, sunshine, jonathantanmy

On 02/12, Stefan Beller wrote:
> This is a real take on the first part of the recent RFC[1].
> 
> Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
> might be a good breaking point for a first part at that RFC at patch 38.
> This series is smaller and contains only 26 patches as the patches in the big
> RFC were slightly out of order.
> 
> I developed this series partly by writing patches, but mostly by cherrypicking
> from that RFC on top of current master. I noticed no external conflicts apart
> from one addition to the repositories _INIT macro, which was easy to resolve.
> 
> Comments in the early range of that RFC were on 003 where Junio pointed out
> that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
> in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.
> 
> brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
> without any suggestion to include into this series[3].
> 
> Duy suggested that we shall not use the repository blindly, but should carefully
> examine whether to pass on an object store or the refstore or such[4], which 
> I agree with if it makes sense. This series unfortunately has an issue with that
> as I would not want to pass down the `ignore_env` flag separately from the object
> store, so I made all functions that only take the object store to have the raw
> object store as the first parameter, and others using the full repository.
> 
> Eric Sunshine brought up memory leaks with the RFC, and I would think to
> have plugged all holes.

I've looked through the patches and I think they look good.  At the end
of the series all the #define tricks have been eliminated so we don't
have to worry about them possibly being left and forgotten :)


Thanks for getting the ball rolling on this.

> 
> [1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
> [2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
> [3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
> [4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/
> 
> Thanks,
> Stefan
> 
> Jonathan Nieder (8):
>   pack: move prepare_packed_git_run_once to object store
>   pack: move approximate object count to object store
>   sha1_file: add repository argument to sha1_file_name
>   sha1_file: add repository argument to map_sha1_file
>   sha1_file: allow stat_sha1_file to handle arbitrary repositories
>   sha1_file: allow open_sha1_file to handle arbitrary repositories
>   sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
>   sha1_file: allow sha1_loose_object_info to handle arbitrary
>     repositories
> 
> Stefan Beller (18):
>   repository: introduce raw object store field
>   object-store: move alt_odb_list and alt_odb_tail to object store
>   object-store: free alt_odb_list
>   object-store: move packed_git and packed_git_mru to object store
>   object-store: close all packs upon clearing the object store
>   sha1_file: add raw_object_store argument to alt_odb_usable
>   sha1_file: add repository argument to link_alt_odb_entry
>   sha1_file: add repository argument to read_info_alternates
>   sha1_file: add repository argument to link_alt_odb_entries
>   sha1_file: add repository argument to prepare_alt_odb
>   sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
>   sha1_file: allow prepare_alt_odb to handle arbitrary repositories
>   sha1_file: add repository argument to stat_sha1_file
>   sha1_file: add repository argument to open_sha1_file
>   sha1_file: add repository argument to map_sha1_file_1
>   sha1_file: add repository argument to sha1_loose_object_info
>   sha1_file: allow sha1_file_name to handle arbitrary repositories
>   sha1_file: allow map_sha1_file to handle arbitrary repositories
> 
>  builtin/am.c                                    |   2 +-
>  builtin/clone.c                                 |   2 +-
>  builtin/count-objects.c                         |   6 +-
>  builtin/fetch.c                                 |   2 +-
>  builtin/fsck.c                                  |  13 ++-
>  builtin/gc.c                                    |   4 +-
>  builtin/grep.c                                  |   2 +-
>  builtin/index-pack.c                            |   1 +
>  builtin/merge.c                                 |   2 +-
>  builtin/pack-objects.c                          |  21 ++--
>  builtin/pack-redundant.c                        |   6 +-
>  builtin/receive-pack.c                          |   3 +-
>  cache.h                                         |  46 ++------
>  contrib/coccinelle/refactoring/packed_git.cocci |   7 ++
>  environment.c                                   |   5 +-
>  fast-import.c                                   |   6 +-
>  http-backend.c                                  |   6 +-
>  http-push.c                                     |   1 +
>  http-walker.c                                   |   4 +-
>  http.c                                          |   6 +-
>  mru.h                                           |   1 +
>  object-store.h                                  |  75 +++++++++++++
>  object.c                                        |  26 +++++
>  pack-bitmap.c                                   |   4 +-
>  pack-check.c                                    |   1 +
>  pack-revindex.c                                 |   1 +
>  packfile.c                                      |  64 +++++------
>  packfile.h                                      |   2 +-
>  path.c                                          |   2 +-
>  reachable.c                                     |   1 +
>  repository.c                                    |  17 ++-
>  repository.h                                    |   7 +-
>  server-info.c                                   |   6 +-
>  sha1_file.c                                     | 135 +++++++++++++-----------
>  sha1_name.c                                     |  11 +-
>  streaming.c                                     |   5 +-
>  36 files changed, 321 insertions(+), 182 deletions(-)
>  create mode 100644 contrib/coccinelle/refactoring/packed_git.cocci
>  create mode 100644 object-store.h
> 
> -- 
> 2.16.1.73.ga2c3e9663f.dirty
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 19:23         ` Stefan Beller
@ 2018-02-13 19:35           ` Junio C Hamano
  2018-02-13 19:43             ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-13 19:35 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Duy Nguyen, Git Mailing List, Eric Sunshine, Jonathan Tan

Stefan Beller <sbeller@google.com> writes:

> For now the ignore_env bit lives in the repository, as that helps
> when working with submodules, when reading its comments.
> Unfortunately 359efeffc1 (repository: introduce the repository
> object, 2017-06-22) did not reason about the existence of the ignore_env
> flag in its commit message.
>
>> So from that point of view, it may not matter where the "bit" lives,
>> among repository, object-store, or ref-store.
>
> It matters on the scale of confusing the developer?

What I meant is that from the point of view, having the bit in the
data (not on the invocation) is confusing no matter which data
structure holds it--they are equally bad.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 19:35           ` Junio C Hamano
@ 2018-02-13 19:43             ` Stefan Beller
  2018-02-14  0:30               ` Junio C Hamano
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-13 19:43 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Duy Nguyen, Git Mailing List, Eric Sunshine, Jonathan Tan

On Tue, Feb 13, 2018 at 11:35 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> For now the ignore_env bit lives in the repository, as that helps
>> when working with submodules, when reading its comments.
>> Unfortunately 359efeffc1 (repository: introduce the repository
>> object, 2017-06-22) did not reason about the existence of the ignore_env
>> flag in its commit message.
>>
>>> So from that point of view, it may not matter where the "bit" lives,
>>> among repository, object-store, or ref-store.
>>
>> It matters on the scale of confusing the developer?
>
> What I meant is that from the point of view, having the bit in the
> data (not on the invocation) is confusing no matter which data
> structure holds it--they are equally bad.

Right.

Which is why I'd strongly consider having it only in the repository
object as that is the largest-scoped thing we'd want. e.g. submodules
should care about environment variables differently:

    GIT_WORK_TREE=~/mysuperproject git checkout \
            --recurse-submodules master

 So with such a command in mind, the environment variable would
only apply to the superproject and the nested submodules should
ignore the env, but compute their paths off the superproject, i.e.
the superproject repository, not its object store?

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 02/26] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-13 18:51   ` Brandon Williams
@ 2018-02-13 19:46     ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-13 19:46 UTC (permalink / raw)
  To: Brandon Williams
  Cc: Junio C Hamano, git, Duy Nguyen, Eric Sunshine, Jonathan Tan,
	Jonathan Nieder

On Tue, Feb 13, 2018 at 10:51 AM, Brandon Williams <bmwill@google.com> wrote:

>> +#include "cache.h"
>> +
>
> Maybe we would want to move the definition of the alternate store to
> this header file?  That way we don't need to include cache.h here.
>

Sounds good. I'll take a look at that.

Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 19:43             ` Stefan Beller
@ 2018-02-14  0:30               ` Junio C Hamano
  0 siblings, 0 replies; 239+ messages in thread
From: Junio C Hamano @ 2018-02-14  0:30 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Duy Nguyen, Git Mailing List, Eric Sunshine, Jonathan Tan

Stefan Beller <sbeller@google.com> writes:

> Which is why I'd strongly consider having it only in the repository
> object as that is the largest-scoped thing we'd want. e.g. submodules
> should care about environment variables differently:
>
>     GIT_WORK_TREE=~/mysuperproject git checkout \
>             --recurse-submodules master
>
>  So with such a command in mind, the environment variable would
> only apply to the superproject and the nested submodules should
> ignore the env, but compute their paths off the superproject, i.e.
> the superproject repository, not its object store?

In the longer term endgame state, what is in the repository object
cannot just be a simple "do we honor environment variable" bit
anymore.  

It will be more like "we may (or may not) look at environment when
we create a repository object", i.e. a bit passed to repo_init()
constructor, and from then on, the repository object knows where the
object store and its alternates are, where the top of the working
tree is, where the repository (i.e. the directory that has "refs/"
in it) is, what "worktree" of the repository we are talking about by
itself.  There is no need for a bit "do we or do we not look at
environment?" that needs to be consulted at runtime, which is quite
a round-about thing.

In your example, the repository object that represents the
superproject will pay attention to GIT_WORK_TREE environment when it
is consturcted, and repository objects dynamically constructed while
the command "recurse-submodules" through them will be constructed
with their "where is the top of my working tree?" set appropriately.
They won't be storing "when figuring out where my working tree is,
do not pay attention to GIT_WORK_TREE environment variable" bit.


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-13  1:22 ` [PATCH 14/26] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-14  0:37   ` Duy Nguyen
  2018-02-14 18:08     ` Brandon Williams
  0 siblings, 1 reply; 239+ messages in thread
From: Duy Nguyen @ 2018-02-14  0:37 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Junio C Hamano, Git Mailing List, Eric Sunshine, Jonathan Tan

On Tue, Feb 13, 2018 at 8:22 AM, Stefan Beller <sbeller@google.com> wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  object-store.h |  3 +--
>  sha1_file.c    | 21 +++++++++++----------
>  2 files changed, 12 insertions(+), 12 deletions(-)
>
> diff --git a/object-store.h b/object-store.h
> index d96a16edd1..add1d4e27c 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -61,7 +61,6 @@ struct packed_git {
>         char pack_name[FLEX_ARRAY]; /* more */
>  };
>
> -#define prepare_alt_odb(r) prepare_alt_odb_##r()
> -extern void prepare_alt_odb_the_repository(void);
> +void prepare_alt_odb(struct repository *r);
>
>  #endif /* OBJECT_STORE_H */
> diff --git a/sha1_file.c b/sha1_file.c
> index d18ce3aeba..f046d560f8 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
>         return r;
>  }
>
> -void prepare_alt_odb_the_repository(void)
> +void prepare_alt_odb(struct repository *r)
>  {
> -       const char *alt;
> -
> -       if (the_repository->objects.alt_odb_tail)
> +       if (r->objects.alt_odb_tail)
>                 return;
>
> -       alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +       r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> +
> +       if (!r->ignore_env) {
> +               const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);

If one day the majority of git moves to use 'struct repository', then
ALTERNATE_DB_ENVIRONMENT is always ignored because ignore_env is
always true. I think if you ignore_env, then you still need to get
this "alt" from  'struct raw_object_store' (or 'struct repository').

Since you get lots of getenv() in repo_setup_env(), I think this
getenv(ALTERNATE_DB_ENVIRONMENT) belongs there too. Then here, if
ignore_env is true, you read r->objects.alt or something instead of
doing getenv().

I really want to kill this getenv() in this code, which is basically
delayed initialization because we haven't done proper init on
the_repo. I realize that it cannot be done earlier, when
prepare_alt_odb() does not even have a  'struct repository *' to work
with. Would it be ok if I contributed a patch on top of your series to
basically do repo_init(&the_repo) for all builtin/external commands
(and fix all the bugs that come with it)? Then we would not need
ignore_env here anymore.

> +               if (!alt)
> +                       alt = "";
>
> -       the_repository->objects.alt_odb_tail =
> -                       &the_repository->objects.alt_odb_list;
> -       link_alt_odb_entries(the_repository, alt,
> -                            PATH_SEP, NULL, 0);
> +               link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> +       }
>
> -       read_info_alternates(the_repository, get_object_directory(), 0);
> +       read_info_alternates(r, r->objects.objectdir, 0);
>  }
>
>  /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
> --
> 2.16.1.73.ga2c3e9663f.dirty
>



-- 
Duy

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13 19:26 ` Jonathan Nieder
@ 2018-02-14  0:57   ` Duy Nguyen
  0 siblings, 0 replies; 239+ messages in thread
From: Duy Nguyen @ 2018-02-14  0:57 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: Stefan Beller, Junio C Hamano, Git Mailing List, Eric Sunshine,
	Jonathan Tan

On Wed, Feb 14, 2018 at 2:26 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> Duy suggested that we shall not use the repository blindly, but should carefully
>> examine whether to pass on an object store or the refstore or such[4], which
>> I agree with if it makes sense. This series unfortunately has an issue with that
>> as I would not want to pass down the `ignore_env` flag separately from the object
>> store, so I made all functions that only take the object store to have the raw
>> object store as the first parameter, and others using the full repository.
>
> I think I want to push back on this a little.
>
> The advantage of a function taking e.g. an object_store as an argument
> instead of a repository is that it increases its flexibility, since it
> allows callers that do not have access to a repository to call it.  The
> disadvantage is also that it increases the flexibility without any
> callers benefitting from that:
>
>  1. It ties us to assumptions from today.  If e.g. an object access in
>     the future starts relying on some other information from the
>     repository (e.g. its config) then we'd have to either add a
>     back-pointer from the object store to its repository or add
>     additional arguments for that additional data at that point.
>
>     If all callers already have a repository, it is simpler to pass
>     that repository as context so that we have the flexibility to make
>     more use of it later.

It's essentially putting all global variables in the same place again.
Only this time it's not global namespace, but "struct repository".
It's no worse than the current state though.

>
>  2. It complicates the caller.  Instead of consistently passing the
>     same repository argument as context to functions that access that
>     repository, the caller would have to pull out relevant fields like
>     the object store from it.

Well, I see that as a good point :)

>
>  3. It prevents us from making opportunistic use of other information
>     from the repository, such as its name for use in error messages.

It does not exactly prevent us. It's just more effort to pass this
around. The repository name for example, there's no reason we can't
have object store name (which could be initialized the same as repo
name).

> In lower-level funcitons that need to be usable by callers without a
> repository (e.g. to find packfiles in an alternate) it makes sense to
> not pass a repository, but without such a use case in mind

I do agree with your benefit argument. But I'd like to point out
though that having all input to object store visible from something
like "struct raw_object_store" makes it easier to reason about the
code. I know how object store works, but occasionally I'm still
surprised when it getenv (or read $GIT_DIR/index, but not in object
store code) behind the scene. Imagine how hard it is for newcomers.

I would count that as benefit, even though it's not a use case per se.
Another potential benefit is writing unit tests will be much easier
(you can configure object store through struct repository too, but
setting one piece here, one piece there to control object store
behavior is not a nice experience). It's a nice thing to have, but not
a deciding factor.

> I don't think it needs to be a general goal.

My stand is a bit more aggressive. We should try to achieve better
abstraction if possible. But if it makes Stefan's life hell, it's not
worth doing. Converting to 'struct repository' is already a step
forward. Actually if it discourages him from finishing this work, it's
already not worth doing.
-- 
Duy

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 14/26] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-14  0:37   ` Duy Nguyen
@ 2018-02-14 18:08     ` Brandon Williams
  2018-02-23  9:56       ` [PATCH 0/2] Fix initializing the_hash_algo Nguyễn Thái Ngọc Duy
  2018-02-26 10:30       ` [PATCH 0/4] Delete ignore_env member in struct repository Nguyễn Thái Ngọc Duy
  0 siblings, 2 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-14 18:08 UTC (permalink / raw)
  To: Duy Nguyen
  Cc: Stefan Beller, Junio C Hamano, Git Mailing List, Eric Sunshine,
	Jonathan Tan

On 02/14, Duy Nguyen wrote:
> On Tue, Feb 13, 2018 at 8:22 AM, Stefan Beller <sbeller@google.com> wrote:
> > Signed-off-by: Stefan Beller <sbeller@google.com>
> > ---
> >  object-store.h |  3 +--
> >  sha1_file.c    | 21 +++++++++++----------
> >  2 files changed, 12 insertions(+), 12 deletions(-)
> >
> > diff --git a/object-store.h b/object-store.h
> > index d96a16edd1..add1d4e27c 100644
> > --- a/object-store.h
> > +++ b/object-store.h
> > @@ -61,7 +61,6 @@ struct packed_git {
> >         char pack_name[FLEX_ARRAY]; /* more */
> >  };
> >
> > -#define prepare_alt_odb(r) prepare_alt_odb_##r()
> > -extern void prepare_alt_odb_the_repository(void);
> > +void prepare_alt_odb(struct repository *r);
> >
> >  #endif /* OBJECT_STORE_H */
> > diff --git a/sha1_file.c b/sha1_file.c
> > index d18ce3aeba..f046d560f8 100644
> > --- a/sha1_file.c
> > +++ b/sha1_file.c
> > @@ -677,21 +677,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
> >         return r;
> >  }
> >
> > -void prepare_alt_odb_the_repository(void)
> > +void prepare_alt_odb(struct repository *r)
> >  {
> > -       const char *alt;
> > -
> > -       if (the_repository->objects.alt_odb_tail)
> > +       if (r->objects.alt_odb_tail)
> >                 return;
> >
> > -       alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> > +       r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> > +
> > +       if (!r->ignore_env) {
> > +               const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> 
> If one day the majority of git moves to use 'struct repository', then
> ALTERNATE_DB_ENVIRONMENT is always ignored because ignore_env is
> always true. I think if you ignore_env, then you still need to get
> this "alt" from  'struct raw_object_store' (or 'struct repository').
> 
> Since you get lots of getenv() in repo_setup_env(), I think this
> getenv(ALTERNATE_DB_ENVIRONMENT) belongs there too. Then here, if
> ignore_env is true, you read r->objects.alt or something instead of
> doing getenv().
> 
> I really want to kill this getenv() in this code, which is basically
> delayed initialization because we haven't done proper init on
> the_repo. I realize that it cannot be done earlier, when
> prepare_alt_odb() does not even have a  'struct repository *' to work
> with. Would it be ok if I contributed a patch on top of your series to
> basically do repo_init(&the_repo) for all builtin/external commands
> (and fix all the bugs that come with it)? Then we would not need
> ignore_env here anymore.

At some point yes we would definitely want the setup code to fully
initialize a repository object (in this case the_repository).  And I
would even like to change the function signatures of all the builtin
commands to take a repository object so that they don't implicitly rely
on the_repository at all.

When I introduced struct repository I seem to remember there being a
couple things which were different about setup that made it difficult to
simply call repo_init() on the_repository.  If you can fix whatever
those issues with setup were (I can't remember all of them) then that
would be great :)

> 
> > +               if (!alt)
> > +                       alt = "";
> >
> > -       the_repository->objects.alt_odb_tail =
> > -                       &the_repository->objects.alt_odb_list;
> > -       link_alt_odb_entries(the_repository, alt,
> > -                            PATH_SEP, NULL, 0);
> > +               link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> > +       }
> >
> > -       read_info_alternates(the_repository, get_object_directory(), 0);
> > +       read_info_alternates(r, r->objects.objectdir, 0);
> >  }
> >
> >  /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
> > --
> > 2.16.1.73.ga2c3e9663f.dirty
> >
> 
> 
> 
> -- 
> Duy

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-13  1:22 [PATCH 00/26] Moving global state into the repository object (part 1) Stefan Beller
                   ` (29 preceding siblings ...)
  2018-02-13 19:33 ` Brandon Williams
@ 2018-02-15 20:42 ` Junio C Hamano
  2018-02-15 21:09   ` Stefan Beller
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
  30 siblings, 2 replies; 239+ messages in thread
From: Junio C Hamano @ 2018-02-15 20:42 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, pclouds, sunshine, jonathantanmy

Stefan Beller <sbeller@google.com> writes:

> This is a real take on the first part of the recent RFC[1].

For the patches remaining in this series, The scope is about right
and the size is more manageable.  With topics already on 'master',
they have some interactions:

 - ot/mru-on-list & gs/retire-mru

   The packed_git MRU has been greatly simplified by using list API
   directly.

 - cc/sha1-file-name

   The function signature of sha1_file_name() has been updated.

I could certainly carry evil merge resolutions going forward (these
changes need to be made to object-store.h that has no mechanical
conflicts), but it may be less distracting for later readers of the
series if we rebase it on top of a more recent master.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 00/26] Moving global state into the repository object (part 1)
  2018-02-15 20:42 ` Junio C Hamano
@ 2018-02-15 21:09   ` Stefan Beller
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
  1 sibling, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-15 21:09 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git, Duy Nguyen, Eric Sunshine, Jonathan Tan

On Thu, Feb 15, 2018 at 12:42 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> This is a real take on the first part of the recent RFC[1].
>
> For the patches remaining in this series, The scope is about right
> and the size is more manageable.

ok, thanks.

>  With topics already on 'master',
> they have some interactions:
>
>  - ot/mru-on-list & gs/retire-mru
>
>    The packed_git MRU has been greatly simplified by using list API
>    directly.
>
>  - cc/sha1-file-name
>
>    The function signature of sha1_file_name() has been updated.
>
> I could certainly carry evil merge resolutions going forward (these
> changes need to be made to object-store.h that has no mechanical
> conflicts), but it may be less distracting for later readers of the
> series if we rebase it on top of a more recent master.

I was debating when to reroll this series as I want to make the
change that Duy proposed, moving the 'ignore_env' into the
object store as well.

I'll rebase on top of the latest master or these topicc while at it.

Thanks for the heads up.
Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-15 20:42 ` Junio C Hamano
  2018-02-15 21:09   ` Stefan Beller
@ 2018-02-16 17:46   ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 01/16] repository: introduce raw object store field Stefan Beller
                       ` (17 more replies)
  1 sibling, 18 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.
  
Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Stefan Beller (16):
  repository: introduce raw object store field
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: allow link_alt_odb_entries to handle arbitrary object
    stores
  sha1_file: allow prepare_alt_odb to handle arbitrary object stores
  sha1_file: allow sha1_file_name to handle arbitrary object stores
  sha1_file: allow stat_sha1_file to handle arbitrary object stores
  sha1_file: allow open_sha1_file to handle arbitrary object stores
  sha1_file: allow map_sha1_file_1 to handle arbitrary object stores
  sha1_file: allow map_sha1_file to handle arbitrary object stores
  sha1_file: allow sha1_loose_object_info to handle arbitrary object
    stores

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   6 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  13 +++--
 builtin/gc.c             |   4 +-
 builtin/grep.c           |   2 +-
 builtin/index-pack.c     |   1 +
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  19 +++---
 builtin/pack-redundant.c |   6 +-
 builtin/receive-pack.c   |   3 +-
 cache.h                  |  45 ++------------
 environment.c            |   5 +-
 fast-import.c            |   6 +-
 http-backend.c           |   6 +-
 http-push.c              |   1 +
 http-walker.c            |   4 +-
 http.c                   |   6 +-
 object-store.h           |  80 +++++++++++++++++++++++++
 object.c                 |  27 +++++++++
 pack-bitmap.c            |   4 +-
 pack-check.c             |   1 +
 pack-revindex.c          |   1 +
 packfile.c               |  64 ++++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 reachable.c              |   1 +
 repository.c             |  22 +++++--
 repository.h             |   7 ++-
 server-info.c            |   6 +-
 sha1_file.c              | 123 +++++++++++++++++++++------------------
 sha1_name.c              |  11 ++--
 streaming.c              |   5 +-
 34 files changed, 314 insertions(+), 177 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 01/16] repository: introduce raw object store field
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 17:46     ` [PATCH 02/16] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
                       ` (16 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct as well as
'ignore_env' which is duplicated from the repository. A later refactoring
(not in this series) will hopefully get rid of that one bit, once all
the construction of repo objects goes through repo_init(). The reason for
duplication now is that the refactoring done in this series allows for
smaller scoped objects in the functions refactored, nameny passing around
the object store instead of the repository object.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 17 +++++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 19 +++++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 46 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e..ec10b062e6 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..5959d990fc
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,17 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+
+	unsigned ignore_env : 1;
+};
+#define RAW_OBJECT_STORE_INIT { NULL, 0 }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b..11d904c033 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..3b8f1b91be 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,9 +49,11 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
+	repo->objects.ignore_env = repo->ignore_env;
+
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -209,12 +218,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 02/16] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
  2018-02-16 17:46     ` [PATCH 01/16] repository: introduce raw object store field Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 17:46     ` [PATCH 03/16] object-store: free alt_odb_list Stefan Beller
                       ` (15 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 cache.h        |  4 ++--
 object-store.h |  7 ++++++-
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 6 files changed, 27 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 7a8a679d4f..908e4f092a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/cache.h b/cache.h
index 9cac7bb518..f373c30e25 100644
--- a/cache.h
+++ b/cache.h
@@ -1576,7 +1576,7 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -1594,7 +1594,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
diff --git a/object-store.h b/object-store.h
index 5959d990fc..a3f0d6ac15 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,8 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+#include "cache.h"
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -8,9 +10,12 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
+
 	unsigned ignore_env : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d1..216ea836ee 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 831d9e7343..1348dce68f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 03/16] object-store: free alt_odb_list
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
  2018-02-16 17:46     ` [PATCH 01/16] repository: introduce raw object store field Stefan Beller
  2018-02-16 17:46     ` [PATCH 02/16] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 17:46     ` [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store Stefan Beller
                       ` (14 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 11d904c033..17b1da6d6b 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (2 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 03/16] object-store: free alt_odb_list Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 20:27       ` Junio C Hamano
  2018-02-16 17:46     ` [PATCH 05/16] object-store: close all packs upon clearing the " Stefan Beller
                       ` (13 subsequent siblings)
  17 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch to adjust callers.
  @@ @@
  - packed_git
  + the_repository->objects.packed_git

  @@ @@
  - packed_git_mru
  + the_repository->objects.packed_git_mru

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  6 ++++--
 builtin/fsck.c           |  7 +++++--
 builtin/gc.c             |  4 +++-
 builtin/index-pack.c     |  1 +
 builtin/pack-objects.c   | 19 +++++++++++--------
 builtin/pack-redundant.c |  6 ++++--
 builtin/receive-pack.c   |  1 +
 cache.h                  | 29 -----------------------------
 fast-import.c            |  6 ++++--
 http-backend.c           |  6 ++++--
 http-push.c              |  1 +
 http-walker.c            |  1 +
 http.c                   |  1 +
 object-store.h           | 36 +++++++++++++++++++++++++++++++++++-
 object.c                 |  7 +++++++
 pack-bitmap.c            |  4 +++-
 pack-check.c             |  1 +
 pack-revindex.c          |  1 +
 packfile.c               | 39 ++++++++++++++++++++-------------------
 reachable.c              |  1 +
 repository.c             |  3 +++
 server-info.c            |  6 ++++--
 sha1_name.c              |  5 +++--
 23 files changed, 118 insertions(+), 73 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 908e4f092a..2e99e02128 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0..96ff790867 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 5ebd370c56..e60cc3ed98 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 83dcbc9773..d73528b583 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1026,7 +1028,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			return want;
 	}
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1046,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2675,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2738,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2748,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2784,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3155,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3168,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index f373c30e25..018a297e10 100644
--- a/cache.h
+++ b/cache.h
@@ -1636,35 +1636,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 0913f8ab86..1ca662390f 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af82..8bb5d991bb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 31755023a4..a4a9e583c7 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/object-store.h b/object-store.h
index a3f0d6ac15..024ccc91e9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -2,6 +2,7 @@
 #define OBJECT_STORE_H
 
 #include "cache.h"
+#include "list.h"
 
 struct raw_object_store {
 	/*
@@ -10,13 +11,46 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
 	unsigned ignore_env : 1;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
+
+#define MRU_LIST_INIT {NULL, NULL}
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+extern struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+} *packed_git;
+
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 17b1da6d6b..c76b62572a 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	while (!list_empty(&o->packed_git_mru))
+		list_del(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index ff5f62c033..bb521cf7fb 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index 216ea836ee..d41e4c83d0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +383,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/repository.c b/repository.c
index 3b8f1b91be..61b23558d0 100644
--- a/repository.c
+++ b/repository.c
@@ -144,6 +144,9 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	repo->objects.ignore_env = 1;
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 05/16] object-store: close all packs upon clearing the object store
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (3 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 06/16] pack: move prepare_packed_git_run_once to " Stefan Beller
                       ` (12 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 5bdd2d7578..4762a702e3 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593..13cfaa6488 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2e..4d72efca78 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index c76b62572a..34daaf37b3 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	while (!list_empty(&o->packed_git_mru))
 		list_del(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index d41e4c83d0..511a2b0cdf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d6..6a2c57045c 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 06/16] pack: move prepare_packed_git_run_once to object store
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (4 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 05/16] object-store: close all packs upon clearing the " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 07/16] pack: move approximate object count " Stefan Beller
                       ` (11 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 024ccc91e9..794cb0af59 100644
--- a/object-store.h
+++ b/object-store.h
@@ -22,10 +22,16 @@ struct raw_object_store {
 	struct alternate_object_database **alt_odb_tail;
 
 	unsigned ignore_env : 1;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
 #define MRU_LIST_INIT {NULL, NULL}
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 511a2b0cdf..a8a2e7fe43 100644
--- a/packfile.c
+++ b/packfile.c
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 07/16] pack: move approximate object count to object store
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (5 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 06/16] pack: move prepare_packed_git_run_once to " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 08/16] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
                       ` (10 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 794cb0af59..80a2ca6f75 100644
--- a/object-store.h
+++ b/object-store.h
@@ -23,6 +23,14 @@ struct raw_object_store {
 
 	unsigned ignore_env : 1;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -31,7 +39,7 @@ struct raw_object_store {
 };
 
 #define MRU_LIST_INIT {NULL, NULL}
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index a8a2e7fe43..693bafbc98 100644
--- a/packfile.c
+++ b/packfile.c
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 08/16] sha1_file: add raw_object_store argument to alt_odb_usable
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (6 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 07/16] pack: move approximate object count " Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 17:46     ` [PATCH 09/16] sha1_file: allow link_alt_odb_entries to handle arbitrary object stores Stefan Beller
                       ` (9 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1348dce68f..a8e23bd2f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 09/16] sha1_file: allow link_alt_odb_entries to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (7 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 08/16] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
@ 2018-02-16 17:46     ` Stefan Beller
  2018-02-16 17:46     ` [PATCH 10/16] sha1_file: allow prepare_alt_odb " Stefan Beller
                       ` (8 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary object stores, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h     |  4 ++++
 sha1_file.c | 42 +++++++++++++++++++++++++-----------------
 2 files changed, 29 insertions(+), 17 deletions(-)

diff --git a/cache.h b/cache.h
index 018a297e10..12bb42f951 100644
--- a/cache.h
+++ b/cache.h
@@ -1593,6 +1593,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 extern void prepare_alt_odb(void);
diff --git a/sha1_file.c b/sha1_file.c
index a8e23bd2f8..c4255a2da6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -389,9 +389,11 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+static void read_info_alternates(struct raw_object_store *o,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct raw_object_store *o, const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -416,7 +418,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(o, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -424,12 +426,12 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*o->alt_odb_tail = ent;
+	o->alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(o, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -464,8 +466,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+static void link_alt_odb_entries(struct raw_object_store *o, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -479,7 +481,7 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, o->objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -488,13 +490,16 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(o, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates(struct raw_object_store *o,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -506,7 +511,7 @@ static void read_info_alternates(const char * relative_base, int depth)
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(o, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -560,7 +565,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(&the_repository->objects, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -573,7 +579,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(&the_repository->objects, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -676,9 +683,10 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(&the_repository->objects, alt,
+			     PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(&the_repository->objects, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 10/16] sha1_file: allow prepare_alt_odb to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (8 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 09/16] sha1_file: allow link_alt_odb_entries to handle arbitrary object stores Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 11/16] sha1_file: allow sha1_file_name " Stefan Beller
                       ` (7 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/fsck.c |  2 +-
 cache.h        |  1 -
 object-store.h |  2 ++
 packfile.c     |  2 +-
 sha1_file.c    | 23 +++++++++++------------
 sha1_name.c    |  3 ++-
 6 files changed, 17 insertions(+), 16 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 2e99e02128..c0d967d6d7 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(&the_repository->objects);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/cache.h b/cache.h
index 12bb42f951..d0c1a533c5 100644
--- a/cache.h
+++ b/cache.h
@@ -1599,7 +1599,6 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-extern void prepare_alt_odb(void);
 extern char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 extern int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/object-store.h b/object-store.h
index 80a2ca6f75..280e8e16ad 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,4 +67,6 @@ extern struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 } *packed_git;
 
+void prepare_alt_odb(struct raw_object_store *o);
+
 #endif /* OBJECT_STORE_H */
diff --git a/packfile.c b/packfile.c
index 693bafbc98..83b10e55c0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -890,7 +890,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index c4255a2da6..04a81e29f2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -577,7 +578,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 
 	link_alt_odb_entries(&the_repository->objects, reference,
 			     '\n', NULL, 0);
@@ -663,7 +664,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -672,21 +673,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb(struct raw_object_store *o)
 {
 	const char *alt;
 
-	if (the_repository->objects.alt_odb_tail)
+	if (o->alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(&the_repository->objects, alt,
-			     PATH_SEP, NULL, 0);
+	o->alt_odb_tail = &o->alt_odb_list;
+	link_alt_odb_entries(o, alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(&the_repository->objects, get_object_directory(), 0);
+	read_info_alternates(o, o->objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
@@ -726,7 +725,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -885,7 +884,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -916,7 +915,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..f1e6c910ba 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(&the_repository->objects);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 11/16] sha1_file: allow sha1_file_name to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (9 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 10/16] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 12/16] sha1_file: allow stat_sha1_file " Stefan Beller
                       ` (6 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  6 ++++++
 sha1_file.c    | 12 ++++++------
 5 files changed, 16 insertions(+), 16 deletions(-)

diff --git a/cache.h b/cache.h
index d0c1a533c5..2495c7081d 100644
--- a/cache.h
+++ b/cache.h
@@ -961,12 +961,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 8bb5d991bb..59bc0417d2 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(&the_repository->objects, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index a4a9e583c7..058436d9d3 100644
--- a/http.c
+++ b/http.c
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(&the_repository->objects, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(&the_repository->objects, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 280e8e16ad..5f6beb9d91 100644
--- a/object-store.h
+++ b/object-store.h
@@ -67,6 +67,12 @@ extern struct packed_git {
 	char pack_name[FLEX_ARRAY]; /* more */
 } *packed_git;
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1);
+
 void prepare_alt_odb(struct raw_object_store *o);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 04a81e29f2..d64cdbb494 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, get_object_directory());
+	strbuf_addstr(buf, o->objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
@@ -717,7 +717,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -878,7 +878,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -907,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(&the_repository->objects, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1592,7 +1592,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(&the_repository->objects, &filename, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 12/16] sha1_file: allow stat_sha1_file to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (10 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 11/16] sha1_file: allow sha1_file_name " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 13/16] sha1_file: allow open_sha1_file " Stefan Beller
                       ` (5 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index d64cdbb494..bd8b0331f0 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -871,22 +871,23 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+static int stat_sha1_file(struct raw_object_store *o, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&the_repository->objects, &buf, sha1);
+	sha1_file_name(o, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(&the_repository->objects);
+	prepare_alt_odb(o);
+
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -1178,7 +1179,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(&the_repository->objects, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1392,7 +1393,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(&the_repository->objects, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 13/16] sha1_file: allow open_sha1_file to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (11 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 12/16] sha1_file: allow stat_sha1_file " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 14/16] sha1_file: allow map_sha1_file_1 " Stefan Beller
                       ` (4 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index bd8b0331f0..3356f70dd2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -900,7 +900,8 @@ static int stat_sha1_file(struct raw_object_store *o, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+static int open_sha1_file(struct raw_object_store *o,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -908,7 +909,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&the_repository->objects, &buf, sha1);
+	sha1_file_name(o, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -916,8 +917,8 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(&the_repository->objects);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(o);
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
@@ -943,7 +944,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(&the_repository->objects, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 14/16] sha1_file: allow map_sha1_file_1 to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (12 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 13/16] sha1_file: allow open_sha1_file " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 15/16] sha1_file: allow map_sha1_file " Stefan Beller
                       ` (3 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 3356f70dd2..aeb925a895 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -934,9 +934,8 @@ static int open_sha1_file(struct raw_object_store *o,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+static void *map_sha1_file_1(struct raw_object_store *o, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -944,7 +943,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(&the_repository->objects, sha1, &path);
+		fd = open_sha1_file(o, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
@@ -965,7 +964,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(&the_repository->objects, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2195,7 +2194,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(&the_repository->objects, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 15/16] sha1_file: allow map_sha1_file to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (13 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 14/16] sha1_file: allow map_sha1_file_1 " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 17:46     ` [PATCH 16/16] sha1_file: allow sha1_loose_object_info " Stefan Beller
                       ` (2 subsequent siblings)
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 2 ++
 sha1_file.c    | 7 ++++---
 streaming.c    | 5 ++++-
 4 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/cache.h b/cache.h
index 2495c7081d..df29ee72e9 100644
--- a/cache.h
+++ b/cache.h
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 5f6beb9d91..30b4cfdbf5 100644
--- a/object-store.h
+++ b/object-store.h
@@ -73,6 +73,8 @@ extern struct packed_git {
  */
 void sha1_file_name(struct raw_object_store *o, struct strbuf *buf, const unsigned char *sha1);
 
+void *map_sha1_file(struct raw_object_store *o, const unsigned char *sha1, unsigned long *size);
+
 void prepare_alt_odb(struct raw_object_store *o);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index aeb925a895..4cbff471a2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -962,9 +962,10 @@ static void *map_sha1_file_1(struct raw_object_store *o, const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct raw_object_store *o,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(&the_repository->objects, NULL, sha1, size);
+	return map_sha1_file_1(o, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -1186,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(&the_repository->objects, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..6888448b12 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(&the_repository->objects,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 16/16] sha1_file: allow sha1_loose_object_info to handle arbitrary object stores
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (14 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 15/16] sha1_file: allow map_sha1_file " Stefan Beller
@ 2018-02-16 17:46     ` " Stefan Beller
  2018-02-16 22:34     ` [PATCHv2 00/16] Moving global state into the repository object (part 1) Jonathan Nieder
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
  17 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-16 17:46 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, pclouds, sbeller, sunshine

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 4cbff471a2..31be57249f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1154,7 +1154,8 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
+static int sha1_loose_object_info(struct raw_object_store *o,
+				  const unsigned char *sha1,
 				  struct object_info *oi,
 				  int flags)
 {
@@ -1180,14 +1181,14 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(&the_repository->objects, sha1, &st, &path) < 0)
+		if (stat_sha1_file(o, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(&the_repository->objects, sha1, &mapsize);
+	map = map_sha1_file(o, sha1, &mapsize);
 	if (!map)
 		return -1;
 
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(&the_repository->objects, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store
  2018-02-16 17:46     ` [PATCH 04/16] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-16 20:27       ` Junio C Hamano
  0 siblings, 0 replies; 239+ messages in thread
From: Junio C Hamano @ 2018-02-16 20:27 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Stefan Beller <sbeller@google.com> writes:

>   @@ @@
>   - packed_git_mru
>   + the_repository->objects.packed_git_mru

Regarding this...

> diff --git a/object-store.h b/object-store.h
> index a3f0d6ac15..024ccc91e9 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -2,6 +2,7 @@
>  #define OBJECT_STORE_H
>  
>  #include "cache.h"
> +#include "list.h"
>  
>  struct raw_object_store {
> ... 
> +	struct packed_git *packed_git;
> +	/*
> +	 * A most-recently-used ordered version of the packed_git list, which can
> +	 * be iterated instead of packed_git (and marked via mru_mark).
> +	 */
> +	struct list_head packed_git_mru;
> +
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  
>  	unsigned ignore_env : 1;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL, 0 }
> +
> +#define MRU_LIST_INIT {NULL, NULL}
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, MRU_LIST_INIT, NULL, NULL, 0 }
> ...
> diff --git a/packfile.c b/packfile.c
> index 216ea836ee..d41e4c83d0 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -7,6 +7,7 @@
> -...
> -LIST_HEAD(packed_git_mru);

Given that the definition of LIST_HEAD() is

    /* Define a variable with the head and tail of the list. */
    #define LIST_HEAD(name) \
            struct list_head name = { &(name), &(name) }

doesn't the updated definition of RAW_OBJECT_STORE_INIT look fishy?


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (15 preceding siblings ...)
  2018-02-16 17:46     ` [PATCH 16/16] sha1_file: allow sha1_loose_object_info " Stefan Beller
@ 2018-02-16 22:34     ` Jonathan Nieder
  2018-02-20 18:55       ` Stefan Beller
  2018-02-20 19:03       ` Junio C Hamano
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
  17 siblings, 2 replies; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-16 22:34 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> v2:
> * duplicated the 'ignore_env' bit into the object store as well
> * the #define trick no longer works as we do not have a "the_objectstore" global,
>   which means there is just one patch per function that is converted.
>   As this follows the same structure of the previous series, I am still confident
>   there is no hidden dependencies to globals outside the object store in these
>   converted functions.
> * rebased on top of current master, resolving the merge conflicts.
>   I think I used the list.h APIs right, but please double check.

For what it's worth, I think I prefer v1.  I put some comments on why
on patch 0 of v1 and would be interested in your thoughts on them
(e.g. as a reply to that).  I also think that even if we want to
switch to a style that passes around object_store separately from
repository, it is easier to do the migration in two steps: first get
rid of hidden dependencies on the_repository, then do the (simpler)
automatic migration from

 f(the_repository)

to

 f(the_repository->object_store)

*afterwards*.

Thoughts?

Thanks,
Jonathan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-16 22:34     ` [PATCHv2 00/16] Moving global state into the repository object (part 1) Jonathan Nieder
@ 2018-02-20 18:55       ` Stefan Beller
  2018-02-20 19:00         ` Brandon Williams
  2018-02-20 19:03       ` Junio C Hamano
  1 sibling, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-20 18:55 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: Junio C Hamano, git, Jonathan Tan, Duy Nguyen, Eric Sunshine

On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Stefan Beller wrote:
>
>> v2:
>> * duplicated the 'ignore_env' bit into the object store as well
>> * the #define trick no longer works as we do not have a "the_objectstore" global,
>>   which means there is just one patch per function that is converted.
>>   As this follows the same structure of the previous series, I am still confident
>>   there is no hidden dependencies to globals outside the object store in these
>>   converted functions.
>> * rebased on top of current master, resolving the merge conflicts.
>>   I think I used the list.h APIs right, but please double check.
>
> For what it's worth, I think I prefer v1.  I put some comments on why
> on patch 0 of v1 and would be interested in your thoughts on them
> (e.g. as a reply to that).  I also think that even if we want to
> switch to a style that passes around object_store separately from
> repository, it is easier to do the migration in two steps: first get
> rid of hidden dependencies on the_repository, then do the (simpler)
> automatic migration from
>
>  f(the_repository)
>
> to
>
>  f(the_repository->object_store)
>
> *afterwards*.
>
> Thoughts?

I would prefer to not spend more time on these conversions.
If Duy and you would come to a conclusion to either pick this
or the previous version I would be happy.

I do not see the benefit in splitting up the series even further and
do this multistep f(repo) -> f(object store), as the cost in potential
merge conflicts is too high. Note that brian just sent another object
id conversion series, also touching sha1_file.c, which I am sure will
produce merge conflicts for Junio.

For the next part 2 and onwards I'd be happy to take either this
strategy or Duys strategy as requested.

Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-20 18:55       ` Stefan Beller
@ 2018-02-20 19:00         ` Brandon Williams
  2018-02-20 19:03           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Brandon Williams @ 2018-02-20 19:00 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Jonathan Nieder, Junio C Hamano, git, Jonathan Tan, Duy Nguyen,
	Eric Sunshine

On 02/20, Stefan Beller wrote:
> On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> > Hi,
> >
> > Stefan Beller wrote:
> >
> >> v2:
> >> * duplicated the 'ignore_env' bit into the object store as well
> >> * the #define trick no longer works as we do not have a "the_objectstore" global,
> >>   which means there is just one patch per function that is converted.
> >>   As this follows the same structure of the previous series, I am still confident
> >>   there is no hidden dependencies to globals outside the object store in these
> >>   converted functions.
> >> * rebased on top of current master, resolving the merge conflicts.
> >>   I think I used the list.h APIs right, but please double check.
> >
> > For what it's worth, I think I prefer v1.  I put some comments on why
> > on patch 0 of v1 and would be interested in your thoughts on them
> > (e.g. as a reply to that).  I also think that even if we want to
> > switch to a style that passes around object_store separately from
> > repository, it is easier to do the migration in two steps: first get
> > rid of hidden dependencies on the_repository, then do the (simpler)
> > automatic migration from
> >
> >  f(the_repository)
> >
> > to
> >
> >  f(the_repository->object_store)
> >
> > *afterwards*.
> >
> > Thoughts?
> 
> I would prefer to not spend more time on these conversions.
> If Duy and you would come to a conclusion to either pick this
> or the previous version I would be happy.
> 
> I do not see the benefit in splitting up the series even further and
> do this multistep f(repo) -> f(object store), as the cost in potential
> merge conflicts is too high. Note that brian just sent another object
> id conversion series, also touching sha1_file.c, which I am sure will
> produce merge conflicts for Junio.
> 
> For the next part 2 and onwards I'd be happy to take either this
> strategy or Duys strategy as requested.

I think Jonathan is trying to point out that converting to f(repo) maybe
easier than converting to f(repo->object_store) upfront which would make
it easier to write the patches (which most of them are already written)
and to review because you can use the #define trick to make some sort of
guarantees.

After we have successfully completed the migration to f(repo), then we
can revisit the subsystems which want to have a clearer abstraction
layer and make the jump to f(repo->object_store).

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-16 22:34     ` [PATCHv2 00/16] Moving global state into the repository object (part 1) Jonathan Nieder
  2018-02-20 18:55       ` Stefan Beller
@ 2018-02-20 19:03       ` Junio C Hamano
  2018-02-20 19:06         ` Stefan Beller
  1 sibling, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-20 19:03 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Stefan Beller, git, jonathantanmy, pclouds, sunshine

Jonathan Nieder <jrnieder@gmail.com> writes:

> For what it's worth, I think I prefer v1.  I put some comments on why
> on patch 0 of v1 and would be interested in your thoughts on them
> (e.g. as a reply to that).  I also think that even if we want to
> switch to a style that passes around object_store separately from
> repository, it is easier to do the migration in two steps: first get
> rid of hidden dependencies on the_repository, then do the (simpler)
> automatic migration from
>
>  f(the_repository)
>
> to
>
>  f(the_repository->object_store)
>
> *afterwards*.
>
> Thoughts?

Are we envisioning the future in which one repository has more than
one object-store (I am counting an object store and its alternates
that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a
single logical "object store")?  If not, doing f(the_repository)
migration, stopping there without f(the_repository->object_store)
may perfectly be adequate, I would think.



^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-20 19:00         ` Brandon Williams
@ 2018-02-20 19:03           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-20 19:03 UTC (permalink / raw)
  To: Brandon Williams
  Cc: Jonathan Nieder, Junio C Hamano, git, Jonathan Tan, Duy Nguyen,
	Eric Sunshine

On Tue, Feb 20, 2018 at 11:00 AM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> On Fri, Feb 16, 2018 at 2:34 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
>> > Hi,
>> >
>> > Stefan Beller wrote:
>> >
>> >> v2:
>> >> * duplicated the 'ignore_env' bit into the object store as well
>> >> * the #define trick no longer works as we do not have a "the_objectstore" global,
>> >>   which means there is just one patch per function that is converted.
>> >>   As this follows the same structure of the previous series, I am still confident
>> >>   there is no hidden dependencies to globals outside the object store in these
>> >>   converted functions.
>> >> * rebased on top of current master, resolving the merge conflicts.
>> >>   I think I used the list.h APIs right, but please double check.
>> >
>> > For what it's worth, I think I prefer v1.  I put some comments on why
>> > on patch 0 of v1 and would be interested in your thoughts on them
>> > (e.g. as a reply to that).  I also think that even if we want to
>> > switch to a style that passes around object_store separately from
>> > repository, it is easier to do the migration in two steps: first get
>> > rid of hidden dependencies on the_repository, then do the (simpler)
>> > automatic migration from
>> >
>> >  f(the_repository)
>> >
>> > to
>> >
>> >  f(the_repository->object_store)
>> >
>> > *afterwards*.
>> >
>> > Thoughts?
>>
>> I would prefer to not spend more time on these conversions.
>> If Duy and you would come to a conclusion to either pick this
>> or the previous version I would be happy.
>>
>> I do not see the benefit in splitting up the series even further and
>> do this multistep f(repo) -> f(object store), as the cost in potential
>> merge conflicts is too high. Note that brian just sent another object
>> id conversion series, also touching sha1_file.c, which I am sure will
>> produce merge conflicts for Junio.
>>
>> For the next part 2 and onwards I'd be happy to take either this
>> strategy or Duys strategy as requested.
>
> I think Jonathan is trying to point out that converting to f(repo) maybe
> easier than converting to f(repo->object_store) upfront

I agree.

> which would make
> it easier to write the patches (which most of them are already written)

true, but for this series we also have the conversion to f(object_store)
written already.

> and to review because you can use the #define trick to make some sort of
> guarantees.

That is true, so it would be a tradeoff between reviewers and maintainers time?

> After we have successfully completed the migration to f(repo), then we
> can revisit the subsystems which want to have a clearer abstraction
> layer and make the jump to f(repo->object_store).

I would think we can take this series as-is and then move on with making
f(repo) abstractions, eventually moving to f(specialized-subsystem) as those
patches are not written yet (neither direct conversions, nor the repo trick;
the patches I already have need adaption which takes enough time on its own.)


>
> --
> Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-20 19:03       ` Junio C Hamano
@ 2018-02-20 19:06         ` Stefan Beller
  2018-02-20 19:55           ` Junio C Hamano
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-20 19:06 UTC (permalink / raw)
  To: Junio C Hamano, Christian Couder
  Cc: Jonathan Nieder, git, Jonathan Tan, Duy Nguyen, Eric Sunshine

On Tue, Feb 20, 2018 at 11:03 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Jonathan Nieder <jrnieder@gmail.com> writes:
>
>> For what it's worth, I think I prefer v1.  I put some comments on why
>> on patch 0 of v1 and would be interested in your thoughts on them
>> (e.g. as a reply to that).  I also think that even if we want to
>> switch to a style that passes around object_store separately from
>> repository, it is easier to do the migration in two steps: first get
>> rid of hidden dependencies on the_repository, then do the (simpler)
>> automatic migration from
>>
>>  f(the_repository)
>>
>> to
>>
>>  f(the_repository->object_store)
>>
>> *afterwards*.
>>
>> Thoughts?
>
> Are we envisioning the future in which one repository has more than
> one object-store (I am counting an object store and its alternates
> that are pointed by its $GIT_OBJECT_DIRECTORY/info/alternates as a
> single logical "object store")?  If not, doing f(the_repository)
> migration, stopping there without f(the_repository->object_store)
> may perfectly be adequate, I would think.
>

I do not envision that, maybe Christian Couder does?
The partial clone world would be happy with just one object store
per repo, I would think.

The step to take an object store would just add expressiveness
to the code, which may help in understanding what part of the code is
related to what other part of the code, so it may be a readability gain
on its own?

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-20 19:06         ` Stefan Beller
@ 2018-02-20 19:55           ` Junio C Hamano
  2018-02-20 20:16             ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-20 19:55 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Christian Couder, Jonathan Nieder, git, Jonathan Tan, Duy Nguyen,
	Eric Sunshine

Stefan Beller <sbeller@google.com> writes:

> The step to take an object store would just add expressiveness
> to the code, which may help in understanding what part of the code is
> related to what other part of the code, so it may be a readability gain
> on its own?

It certainly would allow you to have a standalone object store that
is not associated with *any* repository, but if we are not using
such a layout, I doubt it would be a readability gain to add excess
and unexercised expressiveness to the code.


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv2 00/16] Moving global state into the repository object (part 1)
  2018-02-20 19:55           ` Junio C Hamano
@ 2018-02-20 20:16             ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-20 20:16 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: Christian Couder, Jonathan Nieder, git, Jonathan Tan, Duy Nguyen,
	Eric Sunshine

On Tue, Feb 20, 2018 at 11:55 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> The step to take an object store would just add expressiveness
>> to the code, which may help in understanding what part of the code is
>> related to what other part of the code, so it may be a readability gain
>> on its own?
>
> It certainly would allow you to have a standalone object store that
> is not associated with *any* repository, but if we are not using
> such a layout, I doubt it would be a readability gain to add excess
> and unexercised expressiveness to the code.

So you favor v1?
Duy seems to be ok with v1 too if there is consensus that it is best
(or rather "if it makes Stefan's life hell, it's not worth doing.")[1].

I will try to resend that v1 shortly[2], as the only actual concern about the
code was where one struct was defined[3]. All other discussion was
meta-level, which direction to go.

Thanks for the clarification!
Stefan

[1] https://public-inbox.org/git/CACsJy8CPKESE8atc_eWdNVknQYp9T6ebwKwCdzLHyaFKH2BnZA@mail.gmail.com/
[2] https://public-inbox.org/git/20180213012241.187007-1-sbeller@google.com/
[3] https://public-inbox.org/git/20180213185120.GA108526@google.com/

^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv3 00/27] Moving global state into the repository object (part 1)
  2018-02-16 17:46   ` [PATCHv2 00/16] " Stefan Beller
                       ` (16 preceding siblings ...)
  2018-02-16 22:34     ` [PATCHv2 00/16] Moving global state into the repository object (part 1) Jonathan Nieder
@ 2018-02-21  1:54     ` " Stefan Beller
  2018-02-21  1:54       ` [PATCH 01/27] repository: introduce raw object store field Stefan Beller
                         ` (28 more replies)
  17 siblings, 29 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

v3:
* reverted back to use the repository for most of the functions
  (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as
  I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to
  object-store.h, such that inclusion of cache.h (in object-store.h)
  is not required any more.

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.

Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (2):
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (25):
  repository: introduce raw object store field
  object-store: migrate alternates struct and functions from cache.h
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to map_sha1_file
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   6 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  13 ++--
 builtin/gc.c             |   4 +-
 builtin/grep.c           |   2 +-
 builtin/index-pack.c     |   1 +
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  20 +++---
 builtin/pack-redundant.c |   6 +-
 builtin/receive-pack.c   |   3 +-
 cache.h                  |  87 -------------------------
 environment.c            |   5 +-
 fast-import.c            |   6 +-
 http-backend.c           |   6 +-
 http-push.c              |   1 +
 http-walker.c            |   4 +-
 http.c                   |   6 +-
 object-store.h           | 134 +++++++++++++++++++++++++++++++++++++++
 object.c                 |  27 ++++++++
 pack-bitmap.c            |   4 +-
 pack-check.c             |   1 +
 pack-revindex.c          |   1 +
 packfile.c               |  64 +++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 reachable.c              |   1 +
 repository.c             |  19 ++++--
 repository.h             |   7 +-
 server-info.c            |   6 +-
 sha1_file.c              | 134 +++++++++++++++++++++------------------
 sha1_name.c              |  11 ++--
 streaming.c              |   5 +-
 34 files changed, 365 insertions(+), 231 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 01/27] repository: introduce raw object store field
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  6:26         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h Stefan Beller
                         ` (27 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 17 +++++++++++++----
 repository.h   |  7 ++++---
 7 files changed, 42 insertions(+), 11 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8..0f0c195705 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e..ec10b062e6 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 0000000000..cf35760ceb
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b..11d904c033 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	free(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730..81a42d9115 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..2255ff657e 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+	free(repo->objects.objectdir);
+	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
 					    "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
@@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+	memset(&repo->objects, 0, sizeof(repo->objects));
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f..1f8bc7a7cf 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to the object store.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
  2018-02-21  1:54       ` [PATCH 01/27] repository: introduce raw object store field Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:37         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
                         ` (26 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Migrate the struct alternate_object_database and all its related
functions to the object store as these functions are easier found in
that header. The migration is just a verbatim copy, no need to
include the object store header at any C file, because cache.h includes
repository.h which in turn includes the object-store.h

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h        | 51 --------------------------------------------------
 object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 51 deletions(-)

diff --git a/cache.h b/cache.h
index 9cac7bb518..964873faf2 100644
--- a/cache.h
+++ b/cache.h
@@ -1576,57 +1576,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
-	struct alternate_object_database *next;
-
-	/* see alt_scratch_buf() */
-	struct strbuf scratch;
-	size_t base_len;
-
-	/*
-	 * Used to store the results of readdir(3) calls when searching
-	 * for unique abbreviated hashes.  This cache is never
-	 * invalidated, thus it's racy and not necessarily accurate.
-	 * That's fine for its purpose; don't use it for tasks requiring
-	 * greater accuracy!
-	 */
-	char loose_objects_subdir_seen[256];
-	struct oid_array loose_objects_cache;
-
-	char path[FLEX_ARRAY];
-} *alt_odb_list;
-extern void prepare_alt_odb(void);
-extern char *compute_alternate_path(const char *path, struct strbuf *err);
-typedef int alt_odb_fn(struct alternate_object_database *, void *);
-extern int foreach_alt_odb(alt_odb_fn, void*);
-
-/*
- * Allocate a "struct alternate_object_database" but do _not_ actually
- * add it to the list of alternates.
- */
-struct alternate_object_database *alloc_alt_odb(const char *dir);
-
-/*
- * Add the directory to the on-disk alternates file; the new entry will also
- * take effect in the current process.
- */
-extern void add_to_alternates_file(const char *dir);
-
-/*
- * Add the directory to the in-memory list of alternates (along with any
- * recursive alternates it points to), but do not modify the on-disk alternates
- * file.
- */
-extern void add_to_alternates_memory(const char *dir);
-
-/*
- * Returns a scratch strbuf pre-filled with the alternate object directory,
- * including a trailing slash, which can be used to access paths in the
- * alternate. Always use this over direct access to alt->scratch, as it
- * cleans up any previous use of the scratch buffer.
- */
-extern struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
-
 struct pack_window {
 	struct pack_window *next;
 	unsigned char *base;
diff --git a/object-store.h b/object-store.h
index cf35760ceb..5678aa1136 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,57 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+extern struct alternate_object_database {
+	struct alternate_object_database *next;
+
+	/* see alt_scratch_buf() */
+	struct strbuf scratch;
+	size_t base_len;
+
+	/*
+	 * Used to store the results of readdir(3) calls when searching
+	 * for unique abbreviated hashes.  This cache is never
+	 * invalidated, thus it's racy and not necessarily accurate.
+	 * That's fine for its purpose; don't use it for tasks requiring
+	 * greater accuracy!
+	 */
+	char loose_objects_subdir_seen[256];
+	struct oid_array loose_objects_cache;
+
+	char path[FLEX_ARRAY];
+} *alt_odb_list;
+void prepare_alt_odb(void);
+char *compute_alternate_path(const char *path, struct strbuf *err);
+typedef int alt_odb_fn(struct alternate_object_database *, void *);
+int foreach_alt_odb(alt_odb_fn, void*);
+
+/*
+ * Allocate a "struct alternate_object_database" but do _not_ actually
+ * add it to the list of alternates.
+ */
+struct alternate_object_database *alloc_alt_odb(const char *dir);
+
+/*
+ * Add the directory to the on-disk alternates file; the new entry will also
+ * take effect in the current process.
+ */
+void add_to_alternates_file(const char *dir);
+
+/*
+ * Add the directory to the in-memory list of alternates (along with any
+ * recursive alternates it points to), but do not modify the on-disk alternates
+ * file.
+ */
+void add_to_alternates_memory(const char *dir);
+
+/*
+ * Returns a scratch strbuf pre-filled with the alternate object directory,
+ * including a trailing slash, which can be used to access paths in the
+ * alternate. Always use this over direct access to alt->scratch, as it
+ * cleans up any previous use of the scratch buffer.
+ */
+struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
  2018-02-21  1:54       ` [PATCH 01/27] repository: introduce raw object store field Stefan Beller
  2018-02-21  1:54       ` [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:41         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 04/27] object-store: free alt_odb_list Stefan Beller
                         ` (25 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 object-store.h |  9 ++++++---
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 5 files changed, 25 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 7a8a679d4f..908e4f092a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/object-store.h b/object-store.h
index 5678aa1136..e78eea1dde 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,7 +1,7 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 void prepare_alt_odb(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
@@ -58,8 +58,11 @@ struct raw_object_store {
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d1..216ea836ee 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 831d9e7343..1348dce68f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -342,9 +343,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -364,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -424,8 +422,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -559,7 +557,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -657,7 +655,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -669,12 +667,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -718,7 +717,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -878,7 +877,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -908,7 +907,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dd..957ce25680 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 04/27] object-store: free alt_odb_list
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (2 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:42         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
                         ` (24 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index 11d904c033..17b1da6d6b 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	free(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (3 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 04/27] object-store: free alt_odb_list Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21 21:51         ` Junio C Hamano
  2018-02-22  6:44         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 06/27] object-store: close all packs upon clearing the " Stefan Beller
                         ` (23 subsequent siblings)
  28 siblings, 2 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Applying the semantic patch
    contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
    This semantic patch is placed in a sub directory of the coccinelle
    contrib dir, as this semantic patch is not expected to be of general
    usefulness; it is only useful during developing this series and
    merging it with other topics in flight. At a later date, just
    delete that semantic patch.

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h and object-store.h
    where needed.

 5. As the packfiles are now owned by an objectstore/repository, which
    is ephemeral unlike globals, we introduce memory leaks. So address
    them in raw_object_store_clear().

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  6 ++++--
 builtin/fsck.c           |  7 +++++--
 builtin/gc.c             |  4 +++-
 builtin/index-pack.c     |  1 +
 builtin/pack-objects.c   | 20 +++++++++++---------
 builtin/pack-redundant.c |  6 ++++--
 builtin/receive-pack.c   |  1 +
 cache.h                  | 29 ----------------------------
 fast-import.c            |  6 ++++--
 http-backend.c           |  6 ++++--
 http-push.c              |  1 +
 http-walker.c            |  1 +
 http.c                   |  1 +
 object-store.h           | 41 +++++++++++++++++++++++++++++++++++++++-
 object.c                 |  7 +++++++
 pack-bitmap.c            |  4 +++-
 pack-check.c             |  1 +
 pack-revindex.c          |  1 +
 packfile.c               | 39 +++++++++++++++++++-------------------
 reachable.c              |  1 +
 repository.c             |  2 ++
 server-info.c            |  6 ++++--
 sha1_name.c              |  5 +++--
 23 files changed, 122 insertions(+), 74 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c8..9334648dcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,8 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
+#include "object-store.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +122,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 908e4f092a..2e99e02128 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
@@ -729,7 +730,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -737,7 +739,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0..96ff790867 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -18,6 +19,7 @@
 #include "run-command.h"
 #include "sigchain.h"
 #include "argv-array.h"
+#include "object-store.h"
 #include "commit.h"
 #include "packfile.h"
 
@@ -173,7 +175,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 5ebd370c56..e60cc3ed98 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -12,6 +12,7 @@
 #include "exec_cmd.h"
 #include "streaming.h"
 #include "thread-utils.h"
+#include "object-store.h"
 #include "packfile.h"
 
 static const char index_pack_usage[] =
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 83dcbc9773..7f376c2aef 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,6 +1,8 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "object.h"
 #include "blob.h"
@@ -1025,8 +1027,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa8136322..55462bc826 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,8 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +573,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +588,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b7ce7c7f52..b2eac79a6e 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -7,6 +7,7 @@
 #include "sideband.h"
 #include "run-command.h"
 #include "exec_cmd.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "remote.h"
diff --git a/cache.h b/cache.h
index 964873faf2..3fd9dc52cd 100644
--- a/cache.h
+++ b/cache.h
@@ -1585,35 +1585,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index b70ac025e0..1685fe59a2 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,8 +154,10 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
@@ -1110,7 +1112,7 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1305,7 +1307,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a..4d93126c15 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,7 +1,9 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "exec_cmd.h"
@@ -518,13 +520,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/http-push.c b/http-push.c
index 0913f8ab86..1ca662390f 100644
--- a/http-push.c
+++ b/http-push.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "blob.h"
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af82..8bb5d991bb 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -4,6 +4,7 @@
 #include "http.h"
 #include "list.h"
 #include "transport.h"
+#include "object-store.h"
 #include "packfile.h"
 
 struct alt_base {
diff --git a/http.c b/http.c
index 31755023a4..a4a9e583c7 100644
--- a/http.c
+++ b/http.c
@@ -1,6 +1,7 @@
 #include "git-compat-util.h"
 #include "http.h"
 #include "config.h"
+#include "object-store.h"
 #include "pack.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/object-store.h b/object-store.h
index e78eea1dde..1de9e07102 100644
--- a/object-store.h
+++ b/object-store.h
@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
  */
 struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
 
+struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -59,10 +83,25 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/*
+	 * A most-recently-used ordered version of the packed_git list, which can
+	 * be iterated instead of packed_git (and marked via mru_mark).
+	 */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+
+/*
+ * The mru list_head is supposed to be initialized using
+ * the LIST_HEAD macro, assigning prev/next to itself.
+ * However this doesn't work in this case as some compilers dislike
+ * that macro on member variables. Use NULL instead as that is defined
+ * and accepted, deferring the real init to prepare_packed_git_mru(). */
+#define __MRU_INIT { NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index 17b1da6d6b..c76b62572a 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	while (!list_empty(&o->packed_git_mru))
+		list_del(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f..d51172b1d5 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -5,11 +5,13 @@
 #include "revision.h"
 #include "progress.h"
 #include "list-objects.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +337,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/pack-check.c b/pack-check.c
index 073c1fbd46..c80c3ed8fd 100644
--- a/pack-check.c
+++ b/pack-check.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack.h"
+#include "object-store.h"
 #include "pack-revindex.h"
 #include "progress.h"
 #include "packfile.h"
diff --git a/pack-revindex.c b/pack-revindex.c
index ff5f62c033..bb521cf7fb 100644
--- a/pack-revindex.c
+++ b/pack-revindex.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "pack-revindex.h"
+#include "object-store.h"
 
 /*
  * Pack index for existing packs give us easy access to the offsets into
diff --git a/packfile.c b/packfile.c
index 216ea836ee..d41e4c83d0 100644
--- a/packfile.c
+++ b/packfile.c
@@ -7,6 +7,7 @@
 #include "packfile.h"
 #include "delta.h"
 #include "list.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "sha1-lookup.h"
 #include "commit.h"
@@ -45,8 +46,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +245,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +315,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +383,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +685,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +768,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +820,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +869,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1015,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1846,13 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru, &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/reachable.c b/reachable.c
index 88d7d679da..59ed16ba76 100644
--- a/reachable.c
+++ b/reachable.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/repository.c b/repository.c
index 2255ff657e..9798f4ffca 100644
--- a/repository.c
+++ b/repository.c
@@ -142,6 +142,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d..96885c45f6 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,5 +1,7 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "tag.h"
@@ -200,7 +202,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +212,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680..016c883b5c 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 06/27] object-store: close all packs upon clearing the object store
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (4 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:43         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 07/27] pack: move prepare_packed_git_run_once to " Stefan Beller
                         ` (22 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 6 ++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 5bdd2d7578..4762a702e3 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593..13cfaa6488 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2e..4d72efca78 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 30264cfd7c..907ae44ab5 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index b2eac79a6e..954fc72c7c 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index c76b62572a..34daaf37b3 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	while (!list_empty(&o->packed_git_mru))
 		list_del(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
 }
diff --git a/packfile.c b/packfile.c
index d41e4c83d0..511a2b0cdf 100644
--- a/packfile.c
+++ b/packfile.c
@@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d6..6a2c57045c 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 07/27] pack: move prepare_packed_git_run_once to object store
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (5 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 06/27] object-store: close all packs upon clearing the " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:20         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 08/27] pack: move approximate object count " Stefan Beller
                         ` (21 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 1de9e07102..6cecba3951 100644
--- a/object-store.h
+++ b/object-store.h
@@ -92,6 +92,12 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
 /*
@@ -101,7 +107,7 @@ struct raw_object_store {
  * that macro on member variables. Use NULL instead as that is defined
  * and accepted, deferring the real init to prepare_packed_git_mru(). */
 #define __MRU_INIT { NULL, NULL }
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 511a2b0cdf..a8a2e7fe43 100644
--- a/packfile.c
+++ b/packfile.c
@@ -884,12 +884,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -897,13 +896,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 08/27] pack: move approximate object count to object store
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (6 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 07/27] pack: move prepare_packed_git_run_once to " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:47         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
                         ` (20 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index 6cecba3951..bd1e4fcd8b 100644
--- a/object-store.h
+++ b/object-store.h
@@ -93,6 +93,14 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -107,7 +115,7 @@ struct raw_object_store {
  * that macro on member variables. Use NULL instead as that is defined
  * and accepted, deferring the real init to prepare_packed_git_mru(). */
 #define __MRU_INIT { NULL, NULL }
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index a8a2e7fe43..693bafbc98 100644
--- a/packfile.c
+++ b/packfile.c
@@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -814,8 +812,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -901,7 +900,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (7 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 08/27] pack: move approximate object count " Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:29         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 10/27] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
                         ` (19 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 1348dce68f..a8e23bd2f8 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -346,7 +346,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -362,7 +364,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -414,7 +416,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 10/27] sha1_file: add repository argument to link_alt_odb_entry
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (8 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21  1:54       ` [PATCH 11/27] sha1_file: add repository argument to read_info_alternates Stefan Beller
                         ` (18 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index a8e23bd2f8..aaa6276211 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -390,8 +390,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -488,7 +489,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 11/27] sha1_file: add repository argument to read_info_alternates
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (9 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 10/27] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21  1:54       ` [PATCH 12/27] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
                         ` (17 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index aaa6276211..6edcc3d3a7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -389,7 +389,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -430,7 +432,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -496,7 +498,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -680,7 +683,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 12/27] sha1_file: add repository argument to link_alt_odb_entries
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (10 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 11/27] sha1_file: add repository argument to read_info_alternates Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21  1:54       ` [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
                         ` (16 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6edcc3d3a7..a0e9116318 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -467,8 +467,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -511,7 +515,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -565,7 +569,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -578,7 +583,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -681,7 +687,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (11 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 12/27] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  6:51         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
                         ` (15 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

See previous patch for explanation.

While at it, move the declaration to object-store.h,
where it should be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 object-store.h |  3 ++-
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  3 ++-
 5 files changed, 13 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 2e99e02128..76c94e9b5a 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 				alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/object-store.h b/object-store.h
index bd1e4fcd8b..514ad94287 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,7 +20,8 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-void prepare_alt_odb(void);
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+void prepare_alt_odb_the_repository(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/packfile.c b/packfile.c
index 693bafbc98..ccfe084642 100644
--- a/packfile.c
+++ b/packfile.c
@@ -890,7 +890,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index a0e9116318..c26921df4a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -581,7 +582,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -667,7 +668,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -676,7 +677,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -730,7 +731,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -889,7 +890,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -920,7 +921,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c..3e490ee8f6 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -11,6 +11,7 @@
 #include "sha1-array.h"
 #include "packfile.h"
 #include "repository.h"
+#include "object-store.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -353,7 +354,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (12 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:49         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 15/27] sha1_file: allow prepare_alt_odb " Stefan Beller
                         ` (14 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  4 ++++
 sha1_file.c    | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/object-store.h b/object-store.h
index 514ad94287..f283fbdba9 100644
--- a/object-store.h
+++ b/object-store.h
@@ -18,6 +18,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 #define prepare_alt_odb(r) prepare_alt_odb_##r()
diff --git a/sha1_file.c b/sha1_file.c
index c26921df4a..6e5105a252 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (13 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:35         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 16/27] sha1_file: add repository argument to sha1_file_name Stefan Beller
                         ` (13 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 21 +++++++++++----------
 2 files changed, 12 insertions(+), 12 deletions(-)

diff --git a/object-store.h b/object-store.h
index f283fbdba9..873b341774 100644
--- a/object-store.h
+++ b/object-store.h
@@ -24,8 +24,7 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/sha1_file.c b/sha1_file.c
index 6e5105a252..9cf3fffbeb 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -673,21 +673,22 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
-	const char *alt;
-
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
-	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+
+	if (!r->ignore_env) {
+		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
+		if (!alt)
+			alt = "";
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
+	}
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (14 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 15/27] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:51         ` Brandon Williams
  2018-02-21  1:54       ` [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file Stefan Beller
                         ` (12 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  7 +++++++
 sha1_file.c    | 10 +++++-----
 5 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/cache.h b/cache.h
index 3fd9dc52cd..e641c23b0a 100644
--- a/cache.h
+++ b/cache.h
@@ -961,12 +961,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 8bb5d991bb..76bfdbca76 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -546,7 +547,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(the_repository, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index a4a9e583c7..afe2707e86 100644
--- a/http.c
+++ b/http.c
@@ -2247,7 +2247,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2396,8 +2396,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(the_repository, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 873b341774..4f5060e768 100644
--- a/object-store.h
+++ b/object-store.h
@@ -123,4 +123,11 @@ struct raw_object_store {
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 9cf3fffbeb..811ed9e53f 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,7 +323,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
 {
 	strbuf_addstr(buf, get_object_directory());
 	strbuf_addch(buf, '/');
@@ -720,7 +720,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -881,7 +881,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -910,7 +910,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1595,7 +1595,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (15 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 16/27] sha1_file: add repository argument to sha1_file_name Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  6:52         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 18/27] sha1_file: add repository argument to open_sha1_file Stefan Beller
                         ` (11 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 811ed9e53f..ed297773e5 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,8 +874,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
@@ -1181,7 +1182,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1395,7 +1396,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 18/27] sha1_file: add repository argument to open_sha1_file
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (16 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21  1:54       ` [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
                         ` (10 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index ed297773e5..68aff90792 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -903,7 +903,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -946,7 +948,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (17 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 18/27] sha1_file: add repository argument to open_sha1_file Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:36         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 20/27] sha1_file: add repository argument to map_sha1_file Stefan Beller
                         ` (9 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 68aff90792..7fa9ed2f25 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -938,9 +938,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -969,7 +970,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2199,7 +2200,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 20/27] sha1_file: add repository argument to map_sha1_file
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (18 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-21  1:54       ` [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
                         ` (8 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 3 +++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index e641c23b0a..d338b65d75 100644
--- a/cache.h
+++ b/cache.h
@@ -1242,7 +1242,6 @@ extern int pretend_sha1_file(void *, unsigned long, enum object_type, unsigned c
 extern int force_object_loose(const unsigned char *sha1, time_t mtime);
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index 4f5060e768..a37d74a0c5 100644
--- a/object-store.h
+++ b/object-store.h
@@ -130,4 +130,7 @@ void raw_object_store_clear(struct raw_object_store *o);
 #define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
 void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
 
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 7fa9ed2f25..0d3ff99cf2 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -968,7 +968,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1192,7 +1192,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd8..22d27df55e 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (19 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 20/27] sha1_file: add repository argument to map_sha1_file Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  6:53         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
                         ` (7 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 0d3ff99cf2..d1f77a4c80 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1159,9 +1159,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1280,7 +1281,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (20 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
@ 2018-02-21  1:54       ` Stefan Beller
  2018-02-22  0:44         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 23/27] sha1_file: allow stat_sha1_file " Stefan Beller
                         ` (6 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 4 ++--
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index a37d74a0c5..146358fe33 100644
--- a/object-store.h
+++ b/object-store.h
@@ -127,8 +127,7 @@ void raw_object_store_clear(struct raw_object_store *o);
  * Put in `buf` the name of the file in the local object database that
  * would be used to store a loose object with the specified sha1.
  */
-#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
 #define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
 void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
diff --git a/sha1_file.c b/sha1_file.c
index d1f77a4c80..8fad523d85 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -323,9 +323,9 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
 {
-	strbuf_addstr(buf, get_object_directory());
+	strbuf_addstr(buf, r->objects.objectdir);
 	strbuf_addch(buf, '/');
 	fill_sha1_path(buf, sha1);
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 23/27] sha1_file: allow stat_sha1_file to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (21 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-21  1:54       ` [PATCH 24/27] sha1_file: allow open_sha1_file " Stefan Beller
                         ` (5 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8fad523d85..6879af8993 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -874,23 +874,22 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
-static int stat_sha1_file_the_repository(const unsigned char *sha1,
-					 struct stat *st, const char **path)
+static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
+			  struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb(the_repository);
+	prepare_alt_odb(r);
 	errno = ENOENT;
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (22 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 23/27] sha1_file: allow stat_sha1_file " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:45         ` Jonathan Tan
  2018-02-21  1:54       ` [PATCH 25/27] sha1_file: allow map_sha1_file_1 " Stefan Beller
                         ` (4 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6879af8993..598acc5410 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -902,9 +902,8 @@ static int stat_sha1_file(struct repository *r, const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
-static int open_sha1_file_the_repository(const unsigned char *sha1,
-					 const char **path)
+static int open_sha1_file(struct repository *r,
+			  const unsigned char *sha1, const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -912,7 +911,7 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(the_repository, &buf, sha1);
+	sha1_file_name(r, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -920,8 +919,8 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb(the_repository);
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	prepare_alt_odb(r);
+	for (alt = r->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 25/27] sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (23 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 24/27] sha1_file: allow open_sha1_file " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-21  1:54       ` [PATCH 26/27] sha1_file: allow map_sha1_file " Stefan Beller
                         ` (3 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 598acc5410..522804d9f0 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -936,10 +936,8 @@ static int open_sha1_file(struct repository *r,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
-static void *map_sha1_file_1_the_repository(const char *path,
-					    const unsigned char *sha1,
-					    unsigned long *size)
+static void *map_sha1_file_1(struct repository *r, const char *path,
+			     const unsigned char *sha1, unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -947,7 +945,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(the_repository, sha1, &path);
+		fd = open_sha1_file(r, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (24 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 25/27] sha1_file: allow map_sha1_file_1 " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  6:54         ` Jonathan Nieder
  2018-02-21  1:54       ` [PATCH 27/27] sha1_file: allow sha1_loose_object_info " Stefan Beller
                         ` (2 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c    | 5 +++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/object-store.h b/object-store.h
index 146358fe33..5d93677783 100644
--- a/object-store.h
+++ b/object-store.h
@@ -129,7 +129,6 @@ void raw_object_store_clear(struct raw_object_store *o);
  */
 void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1);
 
-#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 522804d9f0..a20f811a07 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -964,9 +964,10 @@ static void *map_sha1_file_1(struct repository *r, const char *path,
 	return map;
 }
 
-void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file(struct repository *r,
+		    const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(the_repository, NULL, sha1, size);
+	return map_sha1_file_1(r, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 27/27] sha1_file: allow sha1_loose_object_info to handle arbitrary repositories
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (25 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 26/27] sha1_file: allow map_sha1_file " Stefan Beller
@ 2018-02-21  1:54       ` " Stefan Beller
  2018-02-22  0:26       ` [PATCHv3 00/27] Moving global state into the repository object (part 1) Stefan Beller
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-21  1:54 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

From: Jonathan Nieder <jrnieder@gmail.com>

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index a20f811a07..36282acb1a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1156,10 +1156,9 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
-static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
-						 struct object_info *oi,
-						 int flags)
+static int sha1_loose_object_info(struct repository *r,
+				  const unsigned char *sha1,
+				  struct object_info *oi, int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1183,14 +1182,14 @@ static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
+		if (stat_sha1_file(r, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
 		return 0;
 	}
 
-	map = map_sha1_file(the_repository, sha1, &mapsize);
+	map = map_sha1_file(r, sha1, &mapsize);
 	if (!map)
 		return -1;
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-21  1:54       ` [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-21 21:51         ` Junio C Hamano
  2018-02-23 20:01           ` Stefan Beller
  2018-02-22  6:44         ` Jonathan Nieder
  1 sibling, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-21 21:51 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jonathantanmy, pclouds, sunshine, Jonathan Nieder

Stefan Beller <sbeller@google.com> writes:

> +
> +/*
> + * The mru list_head is supposed to be initialized using
> + * the LIST_HEAD macro, assigning prev/next to itself.
> + * However this doesn't work in this case as some compilers dislike
> + * that macro on member variables. Use NULL instead as that is defined
> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
> +#define __MRU_INIT { NULL, NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }

I do not think it has to be this way to abuse two NULLs, if you
faithfully mimicked how LIST_HEAD() macro is constructed.  The
reason why it does not try to introduce

	struct list_head x = LIST_HEAD_INIT;

and instead, uses

	LIST_HEAD(x);

is because of the need for self referral.  If we learn from it, we
can do the same, i.e. instead of doing

	struct raw_object_store x = RAW_OBJECT_STORE_INIT;

we can do

	RAW_OBJECT_STORE(x);

that expands to

	struct raw_object_store x = {
		...
		{ &x.packed_git_mru, &x.packed_git_mru },
		...
	};

no?  Then we do not need such a lengthy comment that reads only like
an excuse ;-)


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 07/27] pack: move prepare_packed_git_run_once to object store
  2018-02-21  1:54       ` [PATCH 07/27] pack: move prepare_packed_git_run_once to " Stefan Beller
@ 2018-02-22  0:20         ` Jonathan Tan
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:20 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine, Jonathan Nieder

On Tue, 20 Feb 2018 17:54:10 -0800
Stefan Beller <sbeller@google.com> wrote:

> Each repository's object store can be initialized independently, so
> they must not share a run_once variable.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }

Optional: Trailing zeros are not needed in struct initializers.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCHv3 00/27] Moving global state into the repository object (part 1)
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (26 preceding siblings ...)
  2018-02-21  1:54       ` [PATCH 27/27] sha1_file: allow sha1_loose_object_info " Stefan Beller
@ 2018-02-22  0:26       ` Stefan Beller
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-22  0:26 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine

On Tue, Feb 20, 2018 at 5:54 PM, Stefan Beller <sbeller@google.com> wrote:
> v3:
> * reverted back to use the repository for most of the functions
>   (including unduplicating 'ignore_env')
> * rebased on master again (I lost that state when doing v2, as
>   I did both rebase as well as conversion to object store in one go)
> * one additional patch, that moves the alternates related things to
>   object-store.h, such that inclusion of cache.h (in object-store.h)
>   is not required any more.

This is also available at
https://github.com/stefanbeller/git/tree/object-store-part1-v3

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable
  2018-02-21  1:54       ` [PATCH 09/27] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
@ 2018-02-22  0:29         ` Jonathan Tan
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:29 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine

On Tue, 20 Feb 2018 17:54:12 -0800
Stefan Beller <sbeller@google.com> wrote:

> Add a raw_object_store to alt_odb_usable to be more specific about which
> repository to act on. The choice of the repository is delegated to its
> only caller link_alt_odb_entry.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>

I checked that alt_odb_usable no longer depends on any repository-like
globals.

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-21  1:54       ` [PATCH 15/27] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-22  0:35         ` Jonathan Tan
  2018-02-23 23:18           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:35 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine

On Tue, 20 Feb 2018 17:54:18 -0800
Stefan Beller <sbeller@google.com> wrote:

> -void prepare_alt_odb_the_repository(void)
> +void prepare_alt_odb(struct repository *r)
>  {
> -	const char *alt;
> -
> -	if (the_repository->objects.alt_odb_tail)
> +	if (r->objects.alt_odb_tail)
>  		return;
>  
> -	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
> +
> +	if (!r->ignore_env) {
> +		const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
> +		if (!alt)
> +			alt = "";

alt can be NULL, just like in the existing code, so these 2 lines are
unnecessary. (link_alt_odb_entries() will just do nothing in either
case.)

(I also think that the check of absence of alt should be done by the
caller of link_alt_odb_entries(), not by link_alt_odb_entries() itself,
but that is much beyond the scope of this patch set.)

>  
> -	the_repository->objects.alt_odb_tail =
> -			&the_repository->objects.alt_odb_list;
> -	link_alt_odb_entries(the_repository, alt,
> -			     PATH_SEP, NULL, 0);
> +		link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
> +	}
>  
> -	read_info_alternates(the_repository, get_object_directory(), 0);
> +	read_info_alternates(r, r->objects.objectdir, 0);
>  }

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1
  2018-02-21  1:54       ` [PATCH 19/27] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
@ 2018-02-22  0:36         ` Jonathan Tan
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:36 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine, Jonathan Nieder

On Tue, 20 Feb 2018 17:54:22 -0800
Stefan Beller <sbeller@google.com> wrote:

> Add a repository argument to allow the map_sha1_file_1 caller to be
> more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
> 
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h
  2018-02-21  1:54       ` [PATCH 02/27] object-store: migrate alternates struct and functions from cache.h Stefan Beller
@ 2018-02-22  0:37         ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

On 02/20, Stefan Beller wrote:
> Migrate the struct alternate_object_database and all its related
> functions to the object store as these functions are easier found in
> that header. The migration is just a verbatim copy, no need to
> include the object store header at any C file, because cache.h includes
> repository.h which in turn includes the object-store.h

Nice.  Always love seeing cache.h get smaller


-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-21  1:54       ` [PATCH 03/27] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-22  0:41         ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:41 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

On 02/20, Stefan Beller wrote:
> In a process with multiple repositories open, alternates should be
> associated to a single repository and not shared globally. Move
> alt_odb_list and alt_odb_tail into the_repository and adjust callers
> to reflect this.
> 
> Now that the alternative object data base is per repository, we're
> leaking its memory upon freeing a repository. The next patch plugs
> this hole.
> 
> No functional change intended.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Looks good aside from the one spacing issue (though it's not that big of
a deal).

> ---
>  builtin/fsck.c |  4 +++-
>  object-store.h |  9 ++++++---
>  packfile.c     |  3 ++-
>  sha1_file.c    | 25 ++++++++++++-------------
>  sha1_name.c    |  3 ++-
>  5 files changed, 25 insertions(+), 19 deletions(-)
> 
> diff --git a/builtin/fsck.c b/builtin/fsck.c
> index 7a8a679d4f..908e4f092a 100644
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -1,5 +1,6 @@
>  #include "builtin.h"
>  #include "cache.h"
> +#include "repository.h"
>  #include "config.h"
>  #include "commit.h"
>  #include "tree.h"
> @@ -716,7 +717,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  		fsck_object_dir(get_object_directory());
>  
>  		prepare_alt_odb();
> -		for (alt = alt_odb_list; alt; alt = alt->next)
> +		for (alt = the_repository->objects.alt_odb_list;
> +				alt; alt = alt->next)

Indentation on this line looks odd.


-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 04/27] object-store: free alt_odb_list
  2018-02-21  1:54       ` [PATCH 04/27] object-store: free alt_odb_list Stefan Beller
@ 2018-02-22  0:42         ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:42 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

On 02/20, Stefan Beller wrote:
> Free the memory and reset alt_odb_{list, tail} to NULL.

Good to see memory leaks being avoided (well they will be on other
repository objects)

> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  object.c | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/object.c b/object.c
> index 11d904c033..17b1da6d6b 100644
> --- a/object.c
> +++ b/object.c
> @@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
>  	}
>  }
>  
> +static void free_alt_odb(struct alternate_object_database *alt)
> +{
> +	strbuf_release(&alt->scratch);
> +	oid_array_clear(&alt->loose_objects_cache);
> +}
> +
> +static void free_alt_odbs(struct raw_object_store *o)
> +{
> +	while (o->alt_odb_list) {
> +		free_alt_odb(o->alt_odb_list);
> +		o->alt_odb_list = o->alt_odb_list->next;
> +	}
> +}
> +
>  void raw_object_store_clear(struct raw_object_store *o)
>  {
>  	free(o->objectdir);
> +
> +	free_alt_odbs(o);
> +	o->alt_odb_tail = NULL;
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 06/27] object-store: close all packs upon clearing the object store
  2018-02-21  1:54       ` [PATCH 06/27] object-store: close all packs upon clearing the " Stefan Beller
@ 2018-02-22  0:43         ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:43 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

On 02/20, Stefan Beller wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>

Straight forward change, looks good.

> ---
>  builtin/am.c           | 2 +-
>  builtin/clone.c        | 2 +-
>  builtin/fetch.c        | 2 +-
>  builtin/merge.c        | 2 +-
>  builtin/receive-pack.c | 2 +-
>  object.c               | 6 ++----
>  packfile.c             | 4 ++--
>  packfile.h             | 2 +-
>  8 files changed, 10 insertions(+), 12 deletions(-)
> 
> diff --git a/builtin/am.c b/builtin/am.c
> index 5bdd2d7578..4762a702e3 100644
> --- a/builtin/am.c
> +++ b/builtin/am.c
> @@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
>  	 */
>  	if (!state->rebasing) {
>  		am_destroy(state);
> -		close_all_packs();
> +		close_all_packs(&the_repository->objects);
>  		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
>  	}
>  }
> diff --git a/builtin/clone.c b/builtin/clone.c
> index 101c27a593..13cfaa6488 100644
> --- a/builtin/clone.c
> +++ b/builtin/clone.c
> @@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
>  	transport_disconnect(transport);
>  
>  	if (option_dissociate) {
> -		close_all_packs();
> +		close_all_packs(&the_repository->objects);
>  		dissociate_from_references();
>  	}
>  
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 8ee998ea2e..4d72efca78 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>  
>  	string_list_clear(&list, 0);
>  
> -	close_all_packs();
> +	close_all_packs(&the_repository->objects);
>  
>  	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
>  	if (verbosity < 0)
> diff --git a/builtin/merge.c b/builtin/merge.c
> index 30264cfd7c..907ae44ab5 100644
> --- a/builtin/merge.c
> +++ b/builtin/merge.c
> @@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
>  			 * We ignore errors in 'gc --auto', since the
>  			 * user should see them.
>  			 */
> -			close_all_packs();
> +			close_all_packs(&the_repository->objects);
>  			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
>  		}
>  	}
> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
> index b2eac79a6e..954fc72c7c 100644
> --- a/builtin/receive-pack.c
> +++ b/builtin/receive-pack.c
> @@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
>  			proc.git_cmd = 1;
>  			proc.argv = argv_gc_auto;
>  
> -			close_all_packs();
> +			close_all_packs(&the_repository->objects);
>  			if (!start_command(&proc)) {
>  				if (use_sideband)
>  					copy_to_sideband(proc.err, -1, NULL);
> diff --git a/object.c b/object.c
> index c76b62572a..34daaf37b3 100644
> --- a/object.c
> +++ b/object.c
> @@ -4,6 +4,7 @@
>  #include "tree.h"
>  #include "commit.h"
>  #include "tag.h"
> +#include "packfile.h"
>  
>  static struct object **obj_hash;
>  static int nr_objs, obj_hash_size;
> @@ -469,8 +470,5 @@ void raw_object_store_clear(struct raw_object_store *o)
>  
>  	while (!list_empty(&o->packed_git_mru))
>  		list_del(&o->packed_git_mru);
> -	/*
> -	 * TODO: call close_all_packs once migrated to
> -	 * take an object store argument
> -	 */
> +	close_all_packs(o);
>  }
> diff --git a/packfile.c b/packfile.c
> index d41e4c83d0..511a2b0cdf 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -311,11 +311,11 @@ static void close_pack(struct packed_git *p)
>  	close_pack_index(p);
>  }
>  
> -void close_all_packs(void)
> +void close_all_packs(struct raw_object_store *o)
>  {
>  	struct packed_git *p;
>  
> -	for (p = the_repository->objects.packed_git; p; p = p->next)
> +	for (p = o->packed_git; p; p = p->next)
>  		if (p->do_not_close)
>  			die("BUG: want to close pack marked 'do-not-close'");
>  		else
> diff --git a/packfile.h b/packfile.h
> index a7fca598d6..6a2c57045c 100644
> --- a/packfile.h
> +++ b/packfile.h
> @@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
>  
>  extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
>  extern void close_pack_windows(struct packed_git *);
> -extern void close_all_packs(void);
> +extern void close_all_packs(struct raw_object_store *o);
>  extern void unuse_pack(struct pack_window **);
>  extern void clear_delta_base_cache(void);
>  extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
  2018-02-21  1:54       ` [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
@ 2018-02-22  0:44         ` Jonathan Tan
  2018-02-23 22:50           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:44 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine, Jonathan Nieder

On Tue, 20 Feb 2018 17:54:25 -0800
Stefan Beller <sbeller@google.com> wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

> -void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
> +void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
>  {
> -	strbuf_addstr(buf, get_object_directory());
> +	strbuf_addstr(buf, r->objects.objectdir);
>  	strbuf_addch(buf, '/');
>  	fill_sha1_path(buf, sha1);
>  }

In the future, we should probably have:
 - a function to get the object store out of a repo (so that it can
   lazily initialize the object store struct if necessary)
 - when the object store is obtained, its objectdir field is guaranteed
   to be populated
 - sha1_file_name should take the object store struct, not the repo
   struct

but this is outside the scope of this patch.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 24/27] sha1_file: allow open_sha1_file to handle arbitrary repositories
  2018-02-21  1:54       ` [PATCH 24/27] sha1_file: allow open_sha1_file " Stefan Beller
@ 2018-02-22  0:45         ` Jonathan Tan
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Tan @ 2018-02-22  0:45 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, pclouds, sunshine, Jonathan Nieder

On Tue, 20 Feb 2018 17:54:27 -0800
Stefan Beller <sbeller@google.com> wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 08/27] pack: move approximate object count to object store
  2018-02-21  1:54       ` [PATCH 08/27] pack: move approximate object count " Stefan Beller
@ 2018-02-22  0:47         ` Brandon Williams
  2018-02-23 22:22           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:47 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

On 02/20, Stefan Beller wrote:
> The approximate_object_count() function maintains a rough count of
> objects in a repository to estimate how long object name abbreviates
> should be.  Object names are scoped to a repository and the
> appropriate length may differ by repository, so the object count
> should not be global.
> 
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  object-store.h | 10 +++++++++-
>  packfile.c     | 11 +++++------
>  2 files changed, 14 insertions(+), 7 deletions(-)
> 
> diff --git a/object-store.h b/object-store.h
> index 6cecba3951..bd1e4fcd8b 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -93,6 +93,14 @@ struct raw_object_store {
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  
> +	/*
> +	 * A fast, rough count of the number of objects in the repository.
> +	 * These two fields are not meant for direct access. Use
> +	 * approximate_object_count() instead.
> +	 */
> +	unsigned long approximate_object_count;
> +	unsigned approximate_object_count_valid : 1;

Patch looks fine and is effectively a no-op, though what is the need for
both of these variables?  Maybe it can be simplified down to just use
one?  Just musing as its out of the scope of this patch and we probably
shouldn't try to change that in this series.

> +
>  	/*
>  	 * Whether packed_git has already been populated with this repository's
>  	 * packs.
> @@ -107,7 +115,7 @@ struct raw_object_store {
>   * that macro on member variables. Use NULL instead as that is defined
>   * and accepted, deferring the real init to prepare_packed_git_mru(). */
>  #define __MRU_INIT { NULL, NULL }
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0 }
> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL, 0, 0, 0 }
>  
>  void raw_object_store_clear(struct raw_object_store *o);
>  
> diff --git a/packfile.c b/packfile.c
> index a8a2e7fe43..693bafbc98 100644
> --- a/packfile.c
> +++ b/packfile.c
> @@ -803,8 +803,6 @@ static void prepare_packed_git_one(char *objdir, int local)
>  	strbuf_release(&path);
>  }
>  
> -static int approximate_object_count_valid;
> -
>  /*
>   * Give a fast, rough count of the number of objects in the repository. This
>   * ignores loose objects completely. If you have a lot of them, then either
> @@ -814,8 +812,8 @@ static int approximate_object_count_valid;
>   */
>  unsigned long approximate_object_count(void)
>  {
> -	static unsigned long count;
> -	if (!approximate_object_count_valid) {
> +	if (!the_repository->objects.approximate_object_count_valid) {
> +		unsigned long count;
>  		struct packed_git *p;
>  
>  		prepare_packed_git();
> @@ -825,8 +823,9 @@ unsigned long approximate_object_count(void)
>  				continue;
>  			count += p->num_objects;
>  		}
> +		the_repository->objects.approximate_object_count = count;
>  	}
> -	return count;
> +	return the_repository->objects.approximate_object_count;
>  }
>  
>  static void *get_next_packed_git(const void *p)
> @@ -901,7 +900,7 @@ void prepare_packed_git(void)
>  
>  void reprepare_packed_git(void)
>  {
> -	approximate_object_count_valid = 0;
> +	the_repository->objects.approximate_object_count_valid = 0;
>  	the_repository->objects.packed_git_initialized = 0;
>  	prepare_packed_git();
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  2018-02-21  1:54       ` [PATCH 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
@ 2018-02-22  0:49         ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:49 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

On 02/20, Stefan Beller wrote:
> Actually this also allows read_info_alternates and link_alt_odb_entry to
> handle arbitrary repositories, but link_alt_odb_entries is the most
> interesting function in this set of functions, hence the commit subject.
> 
> These functions span a strongly connected component in the function
> graph, i.e. the recursive call chain might look like
> 
>   -> link_alt_odb_entries
>     -> link_alt_odb_entry
>       -> read_info_alternates
>         -> link_alt_odb_entries
> 
> That is why we need to convert all these functions at the same time.
> 
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---

Looks good.

>  object-store.h |  4 ++++
>  sha1_file.c    | 36 ++++++++++++++++--------------------
>  2 files changed, 20 insertions(+), 20 deletions(-)
> 
> diff --git a/object-store.h b/object-store.h
> index 514ad94287..f283fbdba9 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -18,6 +18,10 @@ struct alternate_object_database {
>  	char loose_objects_subdir_seen[256];
>  	struct oid_array loose_objects_cache;
>  
> +	/*
> +	 * Path to the alternative object store. If this is a relative path,
> +	 * it is relative to the current working directory.
> +	 */

Thanks for adding the comment.

>  	char path[FLEX_ARRAY];
>  };
>  #define prepare_alt_odb(r) prepare_alt_odb_##r()
> diff --git a/sha1_file.c b/sha1_file.c
> index c26921df4a..6e5105a252 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -390,11 +390,10 @@ static int alt_odb_usable(struct raw_object_store *o,
>   * SHA1, an extra slash for the first level indirection, and the
>   * terminating NUL.
>   */
> -#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
> -static void read_info_alternates_the_repository(const char *relative_base,
> -						int depth);
> -#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
> -static int link_alt_odb_entry_the_repository(const char *entry,
> +static void read_info_alternates(struct repository *r,
> +				 const char *relative_base,
> +				 int depth);
> +static int link_alt_odb_entry(struct repository *r, const char *entry,
>  	const char *relative_base, int depth, const char *normalized_objdir)
>  {
>  	struct alternate_object_database *ent;
> @@ -420,7 +419,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
>  	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
>  		strbuf_setlen(&pathbuf, pathbuf.len - 1);
>  
> -	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
> +	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
>  		strbuf_release(&pathbuf);
>  		return -1;
>  	}
> @@ -428,12 +427,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
>  	ent = alloc_alt_odb(pathbuf.buf);
>  
>  	/* add the alternate entry */
> -	*the_repository->objects.alt_odb_tail = ent;
> -	the_repository->objects.alt_odb_tail = &(ent->next);
> +	*r->objects.alt_odb_tail = ent;
> +	r->objects.alt_odb_tail = &(ent->next);
>  	ent->next = NULL;
>  
>  	/* recursively add alternates */
> -	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
> +	read_info_alternates(r, pathbuf.buf, depth + 1);
>  
>  	strbuf_release(&pathbuf);
>  	return 0;
> @@ -468,12 +467,8 @@ static const char *parse_alt_odb_entry(const char *string,
>  	return end;
>  }
>  
> -#define link_alt_odb_entries(r, a, s, rb, d) \
> -	link_alt_odb_entries_##r(a, s, rb, d)
> -static void link_alt_odb_entries_the_repository(const char *alt,
> -						int sep,
> -						const char *relative_base,
> -						int depth)
> +static void link_alt_odb_entries(struct repository *r, const char *alt,
> +				 int sep, const char *relative_base, int depth)
>  {
>  	struct strbuf objdirbuf = STRBUF_INIT;
>  	struct strbuf entry = STRBUF_INIT;
> @@ -487,7 +482,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
>  		return;
>  	}
>  
> -	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
> +	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
>  	if (strbuf_normalize_path(&objdirbuf) < 0)
>  		die("unable to normalize object directory: %s",
>  		    objdirbuf.buf);
> @@ -496,15 +491,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
>  		alt = parse_alt_odb_entry(alt, sep, &entry);
>  		if (!entry.len)
>  			continue;
> -		link_alt_odb_entry(the_repository, entry.buf,
> +		link_alt_odb_entry(r, entry.buf,
>  				   relative_base, depth, objdirbuf.buf);
>  	}
>  	strbuf_release(&entry);
>  	strbuf_release(&objdirbuf);
>  }
>  
> -static void read_info_alternates_the_repository(const char *relative_base,
> -						int depth)
> +static void read_info_alternates(struct repository *r,
> +				 const char *relative_base,
> +				 int depth)
>  {
>  	char *path;
>  	struct strbuf buf = STRBUF_INIT;
> @@ -516,7 +512,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
>  		return;
>  	}
>  
> -	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
> +	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
>  	strbuf_release(&buf);
>  	free(path);
>  }
> -- 
> 2.16.1.291.g4437f3f132-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
  2018-02-21  1:54       ` [PATCH 16/27] sha1_file: add repository argument to sha1_file_name Stefan Beller
@ 2018-02-22  0:51         ` Brandon Williams
  2018-02-23 22:36           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Brandon Williams @ 2018-02-22  0:51 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, Jonathan Nieder

On 02/20, Stefan Beller wrote:
> Add a repository argument to allow sha1_file_name callers to be more
> specific about which repository to handle. This is a small mechanical
> change; it doesn't change the implementation to handle repositories
> other than the_repository yet.
> 
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
> 
> While at it, move the declaration to object-store.h, where it should
> be easier to find.

Seems like we may want to make a sha1-file.h or an oid-file.h or
something like that at some point as that seems like a better place for
the function than in the object-store.h file?


-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 01/27] repository: introduce raw object store field
  2018-02-21  1:54       ` [PATCH 01/27] repository: introduce raw object store field Stefan Beller
@ 2018-02-22  6:26         ` Jonathan Nieder
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:26 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> The raw object store field will contain any objects needed for
> access to objects in a given repository.
>
> This patch introduces the raw object store and populates it with the
> `objectdir`, which used to be part of the repository struct.
>
> As the struct gains members, we'll also populate the function to clear
> the memory for these members.
>
> In a later we'll introduce a struct object_parser, that will complement
> the object parsing in a repository struct: The raw object parser is the
> layer that will provide access to raw object content, while the higher
> level object parser code will parse raw objects and keeps track of
> parenthood and other object relationships using 'struct object'.
> For now only add the lower level to the repository struct.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---

Heh, I suppose that sign-off makes me not a great candidate for
reviewing this.  It's been long enough since I looked at the patch
that I feel okay trying anyway.

[...]
> --- /dev/null
> +++ b/object-store.h
> @@ -0,0 +1,15 @@
> +#ifndef OBJECT_STORE_H
> +#define OBJECT_STORE_H
> +
> +struct raw_object_store {
> +	/*
> +	 * Path to the repository's object store.
> +	 * Cannot be NULL after initialization.
> +	 */
> +	char *objectdir;
> +};
> +#define RAW_OBJECT_STORE_INIT { NULL }

Who owns and is responsible for freeing objectdir?

> +
> +void raw_object_store_clear(struct raw_object_store *o);

Oh, that answers that.

It could be worth a note in the doc comment anyway, but I don't mind
not having one.

[...]
> +
> +void raw_object_store_clear(struct raw_object_store *o)
> +{
> +	free(o->objectdir);
> +}

Should this use FREE_AND_NULL?

[...]
> --- a/repository.c
> +++ b/repository.c
[...]
> @@ -42,8 +49,8 @@ static void repo_setup_env(struct repository *repo)
>  						    !repo->ignore_env);
>  	free(repo->commondir);
>  	repo->commondir = strbuf_detach(&sb, NULL);
> -	free(repo->objectdir);
> -	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
> +	free(repo->objects.objectdir);

Should this call raw_object_store_clear instead of calling free
directly?

> +	repo->objects.objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
>  					    "objects", !repo->ignore_env);

Long line.  One way to break it would be

	repo->objects.objectdir =
		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
				  "objects", !repo->ignore_env);

>  	free(repo->graft_file);
>  	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
> @@ -209,12 +216,14 @@ void repo_clear(struct repository *repo)
>  {
>  	FREE_AND_NULL(repo->gitdir);
>  	FREE_AND_NULL(repo->commondir);
> -	FREE_AND_NULL(repo->objectdir);
>  	FREE_AND_NULL(repo->graft_file);
>  	FREE_AND_NULL(repo->index_file);
>  	FREE_AND_NULL(repo->worktree);
>  	FREE_AND_NULL(repo->submodule_prefix);
>  
> +	raw_object_store_clear(&repo->objects);
> +	memset(&repo->objects, 0, sizeof(repo->objects));

If raw_object_store_clear uses FREE_AND_NULL, this memset won't be necessary.

[...]
> --- a/repository.h
> +++ b/repository.h
> @@ -1,6 +1,8 @@
>  #ifndef REPOSITORY_H
>  #define REPOSITORY_H
>  
> +#include "object-store.h"
> +
>  struct config_set;
>  struct index_state;
>  struct submodule_cache;
> @@ -21,10 +23,9 @@ struct repository {
>  	char *commondir;
>  
>  	/*
> -	 * Path to the repository's object store.
> -	 * Cannot be NULL after initialization.
> +	 * Holds any information related to the object store.
>  	 */

This comment doesn't make it clear to me what the field represents.
Can it be replaced with some of the description from the commit
message?

> -	char *objectdir;
> +	struct raw_object_store objects;
>  

Thanks,
Jonathan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-21  1:54       ` [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
  2018-02-21 21:51         ` Junio C Hamano
@ 2018-02-22  6:44         ` Jonathan Nieder
  2018-02-23 21:42           ` Stefan Beller
  1 sibling, 1 reply; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:44 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> In a process with multiple repositories open, packfile accessors
> should be associated to a single repository and not shared globally.
> Move packed_git and packed_git_mru into the_repository and adjust
> callers to reflect this.
>
> Patch generated by
>
>  1. Moving the struct packed_git declaration to object-store.h
>     and packed_git, packed_git_mru globals to struct object_store.
>
>  2. Applying the semantic patch
>     contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
>     This semantic patch is placed in a sub directory of the coccinelle
>     contrib dir, as this semantic patch is not expected to be of general
>     usefulness; it is only useful during developing this series and
>     merging it with other topics in flight. At a later date, just
>     delete that semantic patch.

Can the semantic patch go in the commit message instead?  It is very
brief.

Actually, I don't see this semantic patch in the diffstat.  Is the
commit message stale?

>  3. Applying line wrapping fixes from "make style" to break the
>     resulting long lines.
>
>  4. Adding missing #includes of repository.h and object-store.h
>     where needed.

Is there a way to automate this step?  (I'm asking for my own
reference when writing future patches, not because of any concern
about the correctness of this one.)
>
>  5. As the packfiles are now owned by an objectstore/repository, which
>     is ephemeral unlike globals, we introduce memory leaks. So address
>     them in raw_object_store_clear().

The compound words are confusing me.  What is an
objectstore/repository?  Are these referring to particular identifiers
or something else?

Would some wording like the following work?

   5. Freeing packed_git and packed_git_mru in raw_object_store_clear
      to avoid a per-repository memory leak.  Previously they were
      global singletons, so code to free them did not exist.

[...]
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -12,6 +12,7 @@
>  #include "exec_cmd.h"
>  #include "streaming.h"
>  #include "thread-utils.h"
> +#include "object-store.h"
>  #include "packfile.h"
>  
>  static const char index_pack_usage[] =

Change from a different patch leaked into this one?

[...]
> +++ b/builtin/pack-objects.c
[...]
> @@ -1044,7 +1045,7 @@ static int want_object_in_pack(const struct object_id *oid,
>  			}
>  			want = want_found_object(exclude, p);
>  			if (!exclude && want > 0)
> -				list_move(&p->mru, &packed_git_mru);
> +				list_move(&p->mru, &the_repository->objects.packed_git_mru);

Long line.  Can "make style" catch this?

[...]
> +++ b/builtin/receive-pack.c
> @@ -7,6 +7,7 @@
>  #include "sideband.h"
>  #include "run-command.h"
>  #include "exec_cmd.h"
> +#include "object-store.h"
>  #include "commit.h"
>  #include "object.h"
>  #include "remote.h"

Another change leaked in?

[...]
> --- a/cache.h
> +++ b/cache.h
> @@ -1585,35 +1585,6 @@ struct pack_window {
>  	unsigned int inuse_cnt;
>  };
>  
> -extern struct packed_git {
[...]
> -} *packed_git;

Move detecting diff confirms that this wasn't modified.  Thanks for
creating it.

[...]
> +++ b/fast-import.c
[...]
> @@ -1110,7 +1112,7 @@ static int store_object(
>  	if (e->idx.offset) {
>  		duplicate_count_by_type[type]++;
>  		return 1;
> -	} else if (find_sha1_pack(oid.hash, packed_git)) {
> +	} else if (find_sha1_pack(oid.hash, the_repository->objects.packed_git)) {

Long line.  (I'll refrain from commenting about any further ones.)

[...]
> +++ b/http-push.c
> @@ -1,4 +1,5 @@
>  #include "cache.h"
> +#include "object-store.h"
>  #include "commit.h"
>  #include "tag.h"
>  #include "blob.h"

Stray change?

> diff --git a/http-walker.c b/http-walker.c
> index 07c2b1af82..8bb5d991bb 100644
> --- a/http-walker.c
> +++ b/http-walker.c
> @@ -4,6 +4,7 @@
>  #include "http.h"
>  #include "list.h"
>  #include "transport.h"
> +#include "object-store.h"
>  #include "packfile.h"
>  
>  struct alt_base {

Same question.

> diff --git a/http.c b/http.c
> index 31755023a4..a4a9e583c7 100644
> --- a/http.c
> +++ b/http.c
> @@ -1,6 +1,7 @@
>  #include "git-compat-util.h"
>  #include "http.h"
>  #include "config.h"
> +#include "object-store.h"
>  #include "pack.h"
>  #include "sideband.h"
>  #include "run-command.h"

Likewise.

> diff --git a/object-store.h b/object-store.h
> index e78eea1dde..1de9e07102 100644
> --- a/object-store.h
> +++ b/object-store.h
> @@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
>   */
>  struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
>  
> +struct packed_git {
> +	struct packed_git *next;
> +	struct list_head mru;
> +	struct pack_window *windows;
> +	off_t pack_size;
> +	const void *index_data;
> +	size_t index_size;
> +	uint32_t num_objects;
> +	uint32_t num_bad_objects;
> +	unsigned char *bad_object_sha1;
> +	int index_version;
> +	time_t mtime;
> +	int pack_fd;
> +	unsigned pack_local:1,
> +		 pack_keep:1,
> +		 freshened:1,
> +		 do_not_close:1,
> +		 pack_promisor:1;
> +	unsigned char sha1[20];
> +	struct revindex_entry *revindex;
> +	/* something like ".git/objects/pack/xxxxx.pack" */
> +	char pack_name[FLEX_ARRAY]; /* more */
> +};
> +
>  struct raw_object_store {
>  	/*
>  	 * Path to the repository's object store.
> @@ -59,10 +83,25 @@ struct raw_object_store {
>  	 */
>  	char *objectdir;
>  
> +	struct packed_git *packed_git;
> +	/*
> +	 * A most-recently-used ordered version of the packed_git list, which can
> +	 * be iterated instead of packed_git (and marked via mru_mark).
> +	 */
> +	struct list_head packed_git_mru;

I don't understand the new part of the comment.  Can you explain here,
for me?

Is this meant as a list of related functions, an explanation of what the
field is for, or something else?

> +
>  	struct alternate_object_database *alt_odb_list;
>  	struct alternate_object_database **alt_odb_tail;
>  };
> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
> +
> +/*
> + * The mru list_head is supposed to be initialized using
> + * the LIST_HEAD macro, assigning prev/next to itself.
> + * However this doesn't work in this case as some compilers dislike
> + * that macro on member variables. Use NULL instead as that is defined
> + * and accepted, deferring the real init to prepare_packed_git_mru(). */

style nit: '*/' should be on its own line.

More importantly, we can avoid such an issue as described by Junio. :)

> +#define __MRU_INIT { NULL, NULL }

Identifiers with leading underscores like this are in a reserved
namespace for the language implementation --- we can't count on them
being available for our own use.

> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
[...]
> --- a/object.c
> +++ b/object.c
> @@ -466,4 +466,11 @@ void raw_object_store_clear(struct raw_object_store *o)
>  
>  	free_alt_odbs(o);
>  	o->alt_odb_tail = NULL;
> +
> +	while (!list_empty(&o->packed_git_mru))
> +		list_del(&o->packed_git_mru);
> +	/*
> +	 * TODO: call close_all_packs once migrated to
> +	 * take an object store argument
> +	 */

Can you say more about this TODO?  Does this mean the patches are out
of order (i.e. that raw_object_store_clear leaves behind a leak until
a later patch)?

[...]
> --- a/pack-check.c
> +++ b/pack-check.c
> @@ -1,5 +1,6 @@
>  #include "cache.h"
>  #include "pack.h"
> +#include "object-store.h"
>  #include "pack-revindex.h"
>  #include "progress.h"
>  #include "packfile.h"

Another unexplained #include (I'll refrain from pointing out later
ones).

The rest looks good.

Thanks,
Jonathan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb
  2018-02-21  1:54       ` [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
@ 2018-02-22  6:51         ` Jonathan Nieder
  2018-02-23 22:33           ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:51 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> See previous patch for explanation.
>
> While at it, move the declaration to object-store.h,
> where it should be easier to find.

Which declaration?  It looks like prepare_alt_odb is already in
object-store.h.

[...]
> --- a/builtin/fsck.c
> +++ b/builtin/fsck.c
> @@ -717,7 +717,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
>  	} else {
>  		fsck_object_dir(get_object_directory());
>  
> -		prepare_alt_odb();
> +		prepare_alt_odb(the_repository);

Patch 2 added a #include of "repository.h".  Good.

(I checked because with the definition of prepare_alt_odb as a macro,
the function call would compile correctly even if the_repository
weren't in scope, but we want to include what we use as a matter of
style/maintainability.)

[...]
> --- a/packfile.c
> +++ b/packfile.c
> @@ -890,7 +890,7 @@ void prepare_packed_git(void)
>  	if (the_repository->objects.packed_git_initialized)
>  		return;
>  	prepare_packed_git_one(get_object_directory(), 1);
> -	prepare_alt_odb();
> +	prepare_alt_odb(the_repository);

Also good, since patch 3 added a #include of "repository.h".

[...]
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -23,6 +23,7 @@
>  #include "sha1-lookup.h"
>  #include "bulk-checkin.h"
>  #include "repository.h"
> +#include "object-store.h"

Should this #include have been added in an earlier patch, since the
file both uses and defines prepare_alt_odb, which is declared there?

The rest looks good.

Thanks,
Jonathan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file
  2018-02-21  1:54       ` [PATCH 17/27] sha1_file: add repository argument to stat_sha1_file Stefan Beller
@ 2018-02-22  6:52         ` Jonathan Nieder
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:52 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> Add a repository argument to allow the stat_sha1_file caller to be
> more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
>
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  sha1_file.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)

Simple and obviously correct.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info
  2018-02-21  1:54       ` [PATCH 21/27] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
@ 2018-02-22  6:53         ` Jonathan Nieder
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:53 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Hi,

Stefan Beller wrote:

> Add a repository argument to allow the sha1_loose_object_info caller
> to be more specific about which repository to act on. This is a small
> mechanical change; it doesn't change the implementation to handle
> repositories other than the_repository yet.
>
> As with the previous commits, use a macro to catch callers passing a
> repository other than the_repository at compile time.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  sha1_file.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)

Simple and obviously correct.
Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 26/27] sha1_file: allow map_sha1_file to handle arbitrary repositories
  2018-02-21  1:54       ` [PATCH 26/27] sha1_file: allow map_sha1_file " Stefan Beller
@ 2018-02-22  6:54         ` Jonathan Nieder
  0 siblings, 0 replies; 239+ messages in thread
From: Jonathan Nieder @ 2018-02-22  6:54 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine

Stefan Beller wrote:

> Signed-off-by: Stefan Beller <sbeller@google.com>
> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
> ---
>  object-store.h | 3 +--
>  sha1_file.c    | 5 +++--
>  2 files changed, 4 insertions(+), 4 deletions(-)

Thanks: this is short and simple, making my life as a reviewer very
easy.

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com>

^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 0/2] Fix initializing the_hash_algo
  2018-02-14 18:08     ` Brandon Williams
@ 2018-02-23  9:56       ` Nguyễn Thái Ngọc Duy
  2018-02-23  9:56         ` [PATCH 1/2] setup.c: initialize the_repository correctly in all cases Nguyễn Thái Ngọc Duy
                           ` (3 more replies)
  2018-02-26 10:30       ` [PATCH 0/4] Delete ignore_env member in struct repository Nguyễn Thái Ngọc Duy
  1 sibling, 4 replies; 239+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2018-02-23  9:56 UTC (permalink / raw)
  To: git
  Cc: Brandon Williams, Stefan Beller, Junio C Hamano, Eric Sunshine,
	Jonathan Tan, brian m . carlson,
	Nguyễn Thái Ngọc Duy

On Thu, Feb 15, 2018 at 1:08 AM, Brandon Williams <bmwill@google.com> wrote:
> At some point yes we would definitely want the setup code to fully
> initialize a repository object (in this case the_repository).  And I
> would even like to change the function signatures of all the builtin
> commands to take a repository object so that they don't implicitly rely
> on the_repository at all.
>
> When I introduced struct repository I seem to remember there being a
> couple things which were different about setup that made it difficult to
> simply call repo_init() on the_repository.  If you can fix whatever
> those issues with setup were (I can't remember all of them) then that
> would be great :)

I can certainly try! I start to remember all the hairy details in that
setup code.

The first step may be something like this, which identifies all the
"repo init" entry points. This is basically a revert of e26f7f19b6
(repository: pre-initialize hash algo pointer - 2018-01-19) and doing
things the proper way, hopefully.

This is on 'master', independent from Stefan's series. I have another
patch on top of that series to remove the use of ignore_env in
sha1_file.c (and things seem to work). Basically whenever you have to
initialize the hash algorithm, there's a good chance you need to
initialize object store as well. But I'll hold that off until
Stefan's and this one are both merged.

But yeah, it looks like we need some surgery in setup.c if we want
something as pretty as repo_submodule_init() but for the main repo.

Nguyễn Thái Ngọc Duy (2):
  setup.c: initialize the_repository correctly in all cases
  Revert "repository: pre-initialize hash algo pointer"

 builtin/index-pack.c             | 5 +++++
 builtin/init-db.c                | 3 ++-
 cache.h                          | 3 ++-
 common-main.c                    | 4 ++++
 diff-no-index.c                  | 5 +++++
 path.c                           | 2 +-
 repository.c                     | 2 +-
 setup.c                          | 5 ++++-
 t/helper/test-dump-split-index.c | 2 ++
 9 files changed, 26 insertions(+), 5 deletions(-)

-- 
2.16.1.435.g8f24da2e1a


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 1/2] setup.c: initialize the_repository correctly in all cases
  2018-02-23  9:56       ` [PATCH 0/2] Fix initializing the_hash_algo Nguyễn Thái Ngọc Duy
@ 2018-02-23  9:56         ` Nguyễn Thái Ngọc Duy
  2018-02-23 22:17           ` brian m. carlson
  2018-02-23  9:56         ` [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer" Nguyễn Thái Ngọc Duy
                           ` (2 subsequent siblings)
  3 siblings, 1 reply; 239+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2018-02-23  9:56 UTC (permalink / raw)
  To: git
  Cc: Brandon Williams, Stefan Beller, Junio C Hamano, Eric Sunshine,
	Jonathan Tan, brian m . carlson,
	Nguyễn Thái Ngọc Duy

There are many ways for any command to access a git repository:

- most of them will try to discover the .git dir via
  setup_git_directory() and friends

- the server side programs already know where the repo is and prepare
  with enter_repo()

- special commands that deal with repo creation (init/clone) use
  init_db() once the new repo is ready for access.

- somebody accesses $GIT_DIR before any of above functions are called
  and accidentally sets up a git repository by set_git_dir() alone

"the_repository" is partially set up via set_git_dir() at some point
in all four cases. The hash algorithm though is configured later after
.git/config is read.

So far proper repo initialization is done only for the first case [1].
The second case is not covered (but that's fine [3]). The third case
was found and worked around in [2]. The fourth case is a buggy one,
which should be fixed already by jk/no-looking-at-dotgit-outside-repo
and never happens again.

This patch makes sure all cases initialize the hash algorithm in
the_repository correctly. Both second and third cases must run
check_repo_format() before "entering" it. Eventually we probably just
rename this function to init_repo() or something.

[1] 78a6766802 (Integrate hash algorithm support with repo setup -
    2017-11-12)

[2] e26f7f19b6 (repository: pre-initialize hash algo pointer -
    2018-01-19)

[3] the reason server side is still running ok with no hash algo
    before [2] is because the programs that use enter_repo() do very
    little then spawn a new program (like pack-objects or
    upload-archive) to do the heavy lifting. These programs already
    use setup_git_dir..()

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/init-db.c | 3 ++-
 cache.h           | 3 ++-
 path.c            | 2 +-
 setup.c           | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/builtin/init-db.c b/builtin/init-db.c
index c9b7946bad..d119d9906b 100644
--- a/builtin/init-db.c
+++ b/builtin/init-db.c
@@ -9,6 +9,7 @@
 #include "builtin.h"
 #include "exec_cmd.h"
 #include "parse-options.h"
+#include "repository.h"
 
 #ifndef DEFAULT_GIT_TEMPLATE_DIR
 #define DEFAULT_GIT_TEMPLATE_DIR "/usr/share/git-core/templates"
@@ -369,7 +370,7 @@ int init_db(const char *git_dir, const char *real_git_dir,
 	 * config file, so this will not fail.  What we are catching
 	 * is an attempt to reinitialize new repository with an old tool.
 	 */
-	check_repository_format();
+	check_repository_format(the_repository);
 
 	reinit = create_default_files(template_dir, original_git_dir);
 
diff --git a/cache.h b/cache.h
index 21fbcc2414..6b97138264 100644
--- a/cache.h
+++ b/cache.h
@@ -894,6 +894,7 @@ extern int repository_format_precious_objects;
 extern char *repository_format_partial_clone;
 extern const char *core_partial_clone_filter_default;
 
+struct repository;
 struct repository_format {
 	int version;
 	int precious_objects;
@@ -926,7 +927,7 @@ int verify_repository_format(const struct repository_format *format,
  * set_git_dir() before calling this, and use it only for "are we in a valid
  * repo?".
  */
-extern void check_repository_format(void);
+extern void check_repository_format(struct repository *);
 
 #define MTIME_CHANGED	0x0001
 #define CTIME_CHANGED	0x0002
diff --git a/path.c b/path.c
index da8b655730..a544252198 100644
--- a/path.c
+++ b/path.c
@@ -827,7 +827,7 @@ const char *enter_repo(const char *path, int strict)
 
 	if (is_git_directory(".")) {
 		set_git_dir(".");
-		check_repository_format();
+		check_repository_format(the_repository);
 		return path;
 	}
 
diff --git a/setup.c b/setup.c
index c5d55dcee4..a82103832e 100644
--- a/setup.c
+++ b/setup.c
@@ -1180,11 +1180,14 @@ int git_config_perm(const char *var, const char *value)
 	return -(i & 0666);
 }
 
-void check_repository_format(void)
+/* optionally configure "repo" to the correct format */
+void check_repository_format(struct repository *repo)
 {
 	struct repository_format repo_fmt;
 	check_repository_format_gently(get_git_dir(), &repo_fmt, NULL);
 	startup_info->have_repository = 1;
+	if (repo)
+		repo_set_hash_algo(repo, repo_fmt.hash_algo);
 }
 
 /*
-- 
2.16.1.435.g8f24da2e1a


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23  9:56       ` [PATCH 0/2] Fix initializing the_hash_algo Nguyễn Thái Ngọc Duy
  2018-02-23  9:56         ` [PATCH 1/2] setup.c: initialize the_repository correctly in all cases Nguyễn Thái Ngọc Duy
@ 2018-02-23  9:56         ` Nguyễn Thái Ngọc Duy
  2018-02-23 18:24           ` Stefan Beller
  2018-02-23 22:29           ` brian m. carlson
  2018-02-23 22:47         ` [PATCH 0/2] Fix initializing the_hash_algo brian m. carlson
  2018-02-24  3:34         ` [PATCH v2 0/5] " Nguyễn Thái Ngọc Duy
  3 siblings, 2 replies; 239+ messages in thread
From: Nguyễn Thái Ngọc Duy @ 2018-02-23  9:56 UTC (permalink / raw)
  To: git
  Cc: Brandon Williams, Stefan Beller, Junio C Hamano, Eric Sunshine,
	Jonathan Tan, brian m . carlson,
	Nguyễn Thái Ngọc Duy

This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
problem, git clone not setting up the_hash_algo, has been fixed in the
previous patch.

As a result of the revert, some code paths that use the_hash_algo
without initialization is revealed and fixed here. It's basically
commands that are allowed to run without a repository. The fix here is
not the best. We probably could figure out the hash algorithm from input
somehow.

Since this is a dangerous move and could potentially break stuff after
release (and leads to workaround like the reverted commit), the
workaround technically remains, but is hidden behind a new environment
variable GIT_HASH_FIXUP. This should let the users continue to use git
while we fix the problem.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/index-pack.c             | 5 +++++
 common-main.c                    | 4 ++++
 diff-no-index.c                  | 5 +++++
 repository.c                     | 2 +-
 t/helper/test-dump-split-index.c | 2 ++
 5 files changed, 17 insertions(+), 1 deletion(-)

diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index 7e3e1a461c..8ee935504e 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
 	if (prefix && chdir(prefix))
 		die(_("Cannot come back to cwd"));
 
+	if (!the_hash_algo) {
+		warning(_("Running without a repository, assuming SHA-1 hash"));
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+	}
+
 	for (i = 1; i < argc; i++) {
 		const char *arg = argv[i];
 
diff --git a/common-main.c b/common-main.c
index 6a689007e7..12aec36794 100644
--- a/common-main.c
+++ b/common-main.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "exec_cmd.h"
 #include "attr.h"
+#include "repository.h"
 
 /*
  * Many parts of Git have subprograms communicate via pipe, expect the
@@ -40,5 +41,8 @@ int main(int argc, const char **argv)
 
 	restore_sigpipe_to_default();
 
+	if (getenv("GIT_HASH_FIXUP"))
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+
 	return cmd_main(argc, argv);
 }
diff --git a/diff-no-index.c b/diff-no-index.c
index 0ed5f0f496..f038f665bc 100644
--- a/diff-no-index.c
+++ b/diff-no-index.c
@@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
 	struct strbuf replacement = STRBUF_INIT;
 	const char *prefix = revs->prefix;
 
+	if (!the_hash_algo) {
+		warning(_("Running without a repository, assuming SHA-1 hash"));
+		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
+	}
+
 	diff_setup(&revs->diffopt);
 	for (i = 1; i < argc - 2; ) {
 		int j;
diff --git a/repository.c b/repository.c
index 4ffbe9bc94..0d715f4fdb 100644
--- a/repository.c
+++ b/repository.c
@@ -5,7 +5,7 @@
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0
 };
 struct repository *the_repository = &the_repo;
 
diff --git a/t/helper/test-dump-split-index.c b/t/helper/test-dump-split-index.c
index e44430b699..b759fe3fc1 100644
--- a/t/helper/test-dump-split-index.c
+++ b/t/helper/test-dump-split-index.c
@@ -12,6 +12,8 @@ int cmd_main(int ac, const char **av)
 	struct split_index *si;
 	int i;
 
+	setup_git_directory();
+
 	do_read_index(&the_index, av[1], 1);
 	printf("own %s\n", sha1_to_hex(the_index.sha1));
 	si = the_index.split_index;
-- 
2.16.1.435.g8f24da2e1a


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23  9:56         ` [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer" Nguyễn Thái Ngọc Duy
@ 2018-02-23 18:24           ` Stefan Beller
  2018-02-23 19:50             ` Junio C Hamano
  2018-02-23 22:29           ` brian m. carlson
  1 sibling, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 18:24 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy
  Cc: git, Brandon Williams, Junio C Hamano, Eric Sunshine,
	Jonathan Tan, brian m . carlson

On Fri, Feb 23, 2018 at 1:56 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> This reverts commit e26f7f19b6c7485f04234946a59ab8f4fd21d6d1. The root
> problem, git clone not setting up the_hash_algo, has been fixed in the
> previous patch.
>
> As a result of the revert, some code paths that use the_hash_algo
> without initialization is revealed and fixed here. It's basically
> commands that are allowed to run without a repository. The fix here is
> not the best. We probably could figure out the hash algorithm from input
> somehow.
>
> Since this is a dangerous move and could potentially break stuff after
> release (and leads to workaround like the reverted commit), the
> workaround technically remains, but is hidden behind a new environment
> variable GIT_HASH_FIXUP. This should let the users continue to use git
> while we fix the problem.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  builtin/index-pack.c             | 5 +++++
>  common-main.c                    | 4 ++++
>  diff-no-index.c                  | 5 +++++
>  repository.c                     | 2 +-
>  t/helper/test-dump-split-index.c | 2 ++
>  5 files changed, 17 insertions(+), 1 deletion(-)
>
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 7e3e1a461c..8ee935504e 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
>         if (prefix && chdir(prefix))
>                 die(_("Cannot come back to cwd"));
>
> +       if (!the_hash_algo) {
> +               warning(_("Running without a repository, assuming SHA-1 hash"));
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +       }
> +
>         for (i = 1; i < argc; i++) {
>                 const char *arg = argv[i];
>
> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..12aec36794 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
>
>         restore_sigpipe_to_default();
>
> +       if (getenv("GIT_HASH_FIXUP"))
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +
>         return cmd_main(argc, argv);
>  }
> diff --git a/diff-no-index.c b/diff-no-index.c
> index 0ed5f0f496..f038f665bc 100644
> --- a/diff-no-index.c
> +++ b/diff-no-index.c
> @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
>         struct strbuf replacement = STRBUF_INIT;
>         const char *prefix = revs->prefix;
>
> +       if (!the_hash_algo) {
> +               warning(_("Running without a repository, assuming SHA-1 hash"));
> +               repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +       }
> +
>         diff_setup(&revs->diffopt);
>         for (i = 1; i < argc - 2; ) {
>                 int j;
> diff --git a/repository.c b/repository.c
> index 4ffbe9bc94..0d715f4fdb 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -5,7 +5,7 @@
>
>  /* The main repository */
>  static struct repository the_repo = {
> -       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
> +       NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, NULL, 0, 0

I am not sure I agree with this defense in depth, because it would add
a lot to maintenance burden.
Specifically this part. The series that I sent out usually clashes
here as this is currently a hot area
of the code touched by many different series in flight.

However this is the long term correct thing to do? We assume no algorithm until
the repository can tell us from its config (or we default to sha1 if there is no
configuration present).

I wonder if there is yet another missing case in the enumeration of
the previous patch:
Some commands are able to operate on GIT_OBJECT_DIR instead
of GIT_DIR (git repack?), which may not even explore the full git directory,
and so doesn't know about the hash value.

In the cover letter you reference my series, but comparing the diffstats
(and looking through the patches), I would only expect this one place
to have merge conflicts, which ought to be easy to resolve.
(In my series I break the initializer into multiple lines to help the
future, too)

After some thought, I like this series.

Thanks,
Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23 18:24           ` Stefan Beller
@ 2018-02-23 19:50             ` Junio C Hamano
  2018-02-23 20:04               ` Stefan Beller
  0 siblings, 1 reply; 239+ messages in thread
From: Junio C Hamano @ 2018-02-23 19:50 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Nguyễn Thái Ngọc Duy, git, Brandon Williams,
	Eric Sunshine, Jonathan Tan, brian m . carlson

Stefan Beller <sbeller@google.com> writes:

> I wonder if there is yet another missing case in the enumeration of
> the previous patch:
> Some commands are able to operate on GIT_OBJECT_DIR instead
> of GIT_DIR (git repack?), which may not even explore the full git directory,
> and so doesn't know about the hash value.

... because GIT_DIR/config is not known?  "repack" is not one of
them, though---it needs to at least use refs as the starting point
so a standalone OBJECT_DIR is insufficient.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-21 21:51         ` Junio C Hamano
@ 2018-02-23 20:01           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 20:01 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: git, Jonathan Tan, Duy Nguyen, Eric Sunshine, Jonathan Nieder

On Wed, Feb 21, 2018 at 1:51 PM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> +
>> +/*
>> + * The mru list_head is supposed to be initialized using
>> + * the LIST_HEAD macro, assigning prev/next to itself.
>> + * However this doesn't work in this case as some compilers dislike
>> + * that macro on member variables. Use NULL instead as that is defined
>> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
>> +#define __MRU_INIT { NULL, NULL }
>> +#define RAW_OBJECT_STORE_INIT { NULL, NULL, __MRU_INIT, NULL, NULL }
>
> I do not think it has to be this way to abuse two NULLs, if you
> faithfully mimicked how LIST_HEAD() macro is constructed.  The
> reason why it does not try to introduce
>
>         struct list_head x = LIST_HEAD_INIT;
>
> and instead, uses
>
>         LIST_HEAD(x);
>
> is because of the need for self referral.  If we learn from it, we
> can do the same, i.e. instead of doing
>
>         struct raw_object_store x = RAW_OBJECT_STORE_INIT;
>
> we can do
>
>         RAW_OBJECT_STORE(x);
>
> that expands to
>
>         struct raw_object_store x = {
>                 ...
>                 { &x.packed_git_mru, &x.packed_git_mru },
>                 ...
>         };
>
> no?  Then we do not need such a lengthy comment that reads only like
> an excuse ;-)

We cannot do this, because the object store is directly
embedded into the the_repository (not as a pointer),
which is a global on its own. So we'd have to do this
at the repository level, which I would not want.
I'd want to have the #defines where the structs are declared.

Any guidance on how to do that correctly with self referential
definitions without making the object store a pointer then?

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23 19:50             ` Junio C Hamano
@ 2018-02-23 20:04               ` Stefan Beller
  2018-02-23 22:26                 ` Junio C Hamano
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 20:04 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: Nguyễn Thái Ngọc Duy, git, Brandon Williams,
	Eric Sunshine, Jonathan Tan, brian m . carlson

On Fri, Feb 23, 2018 at 11:50 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Stefan Beller <sbeller@google.com> writes:
>
>> I wonder if there is yet another missing case in the enumeration of
>> the previous patch:
>> Some commands are able to operate on GIT_OBJECT_DIR instead
>> of GIT_DIR (git repack?), which may not even explore the full git directory,
>> and so doesn't know about the hash value.
>
> ... because GIT_DIR/config is not known?  "repack" is not one of
> them, though---it needs to at least use refs as the starting point
> so a standalone OBJECT_DIR is insufficient.

Yes, I could have worded this as a question:
Is there any command that operates on GIT_OBJECT_DIR
without trying to discover GIT_DIR ?

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-22  6:44         ` Jonathan Nieder
@ 2018-02-23 21:42           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 21:42 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine

>>  2. Applying the semantic patch
>>     contrib/coccinelle/refactoring/packed_git.cocci to adjust callers.
>>     This semantic patch is placed in a sub directory of the coccinelle
>>     contrib dir, as this semantic patch is not expected to be of general
>>     usefulness; it is only useful during developing this series and
>>     merging it with other topics in flight. At a later date, just
>>     delete that semantic patch.
>
> Can the semantic patch go in the commit message instead?  It is very
> brief.

done

>
> Actually, I don't see this semantic patch in the diffstat.  Is the
> commit message stale?
>
>>  3. Applying line wrapping fixes from "make style" to break the
>>     resulting long lines.
>>
>>  4. Adding missing #includes of repository.h and object-store.h
>>     where needed.
>
> Is there a way to automate this step?  (I'm asking for my own
> reference when writing future patches, not because of any concern
> about the correctness of this one.)

no, for several reasons.
(a) I don't know how to write it in coccinelle, because
(b) I have not figured out the order in which we include headers, apart from
  "cache.h goes first", the rest of the includes sometimes looks like a random
  order, because different patches add new includes at different places.
  I have the impression, that (1) some add the include after all other
  existing includes or (2) try to figure out where to add the include to make
  sense in the existing include order or (3) sort alphabetically or (4) put it
  randomly, so the chance for a merge conflict with other series in flight
  decreases.
(c) I did it in a semi automated fashion:
    while make fails:
        add another include

The mess with including repository or object-store comes from the fact that
I had v2 based on object-store, not the repository and cherry-picked this patch
over to v3. Fixed all of the includes now.

>> @@ -59,10 +83,25 @@ struct raw_object_store {
>>        */
>>       char *objectdir;
>>
>> +     struct packed_git *packed_git;
>> +     /*
>> +      * A most-recently-used ordered version of the packed_git list, which can
>> +      * be iterated instead of packed_git (and marked via mru_mark).
>> +      */
>> +     struct list_head packed_git_mru;
>
> I don't understand the new part of the comment.  Can you explain here,
> for me?

cherrypicking error, fixed.

>> -#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
>> +
>> +/*
>> + * The mru list_head is supposed to be initialized using
>> + * the LIST_HEAD macro, assigning prev/next to itself.
>> + * However this doesn't work in this case as some compilers dislike
>> + * that macro on member variables. Use NULL instead as that is defined
>> + * and accepted, deferring the real init to prepare_packed_git_mru(). */
>
> style nit: '*/' should be on its own line.
>
> More importantly, we can avoid such an issue as described by Junio. :)

See reply to Junio, I am not quite sure I like that.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 1/2] setup.c: initialize the_repository correctly in all cases
  2018-02-23  9:56         ` [PATCH 1/2] setup.c: initialize the_repository correctly in all cases Nguyễn Thái Ngọc Duy
@ 2018-02-23 22:17           ` brian m. carlson
  2018-02-24  3:17             ` Duy Nguyen
  0 siblings, 1 reply; 239+ messages in thread
From: brian m. carlson @ 2018-02-23 22:17 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy
  Cc: git, Brandon Williams, Stefan Beller, Junio C Hamano,
	Eric Sunshine, Jonathan Tan

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

On Fri, Feb 23, 2018 at 04:56:39PM +0700, Nguyễn Thái Ngọc Duy wrote:
> [3] the reason server side is still running ok with no hash algo
>     before [2] is because the programs that use enter_repo() do very
>     little then spawn a new program (like pack-objects or
>     upload-archive) to do the heavy lifting. These programs already
>     use setup_git_dir..()

You have "..()" here.  Did you want to say "()." instead?
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 08/27] pack: move approximate object count to object store
  2018-02-22  0:47         ` Brandon Williams
@ 2018-02-23 22:22           ` Stefan Beller
  2018-02-26  8:55             ` Jeff King
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 22:22 UTC (permalink / raw)
  To: Brandon Williams, Jeff King
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine,
	Jonathan Nieder

On Wed, Feb 21, 2018 at 4:47 PM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> The approximate_object_count() function maintains a rough count of
>> objects in a repository to estimate how long object name abbreviates
>> should be.  Object names are scoped to a repository and the
>> appropriate length may differ by repository, so the object count
>> should not be global.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>> ---
>>  object-store.h | 10 +++++++++-
>>  packfile.c     | 11 +++++------
>>  2 files changed, 14 insertions(+), 7 deletions(-)
>>
>> diff --git a/object-store.h b/object-store.h
>> index 6cecba3951..bd1e4fcd8b 100644
>> --- a/object-store.h
>> +++ b/object-store.h
>> @@ -93,6 +93,14 @@ struct raw_object_store {
>>       struct alternate_object_database *alt_odb_list;
>>       struct alternate_object_database **alt_odb_tail;
>>
>> +     /*
>> +      * A fast, rough count of the number of objects in the repository.
>> +      * These two fields are not meant for direct access. Use
>> +      * approximate_object_count() instead.
>> +      */
>> +     unsigned long approximate_object_count;
>> +     unsigned approximate_object_count_valid : 1;
>
> Patch looks fine and is effectively a no-op, though what is the need for
> both of these variables?  Maybe it can be simplified down to just use
> one?  Just musing as its out of the scope of this patch and we probably
> shouldn't try to change that in this series.

I agree we should. It was introduced in e323de4ad7f (sha1_file:
allow sha1_loose_object_info to handle arbitrary repositories, 2017-08-30)
and I think it was seen as a clever optimization trick back then?

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23 20:04               ` Stefan Beller
@ 2018-02-23 22:26                 ` Junio C Hamano
  0 siblings, 0 replies; 239+ messages in thread
From: Junio C Hamano @ 2018-02-23 22:26 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Nguyễn Thái Ngọc Duy, git, Brandon Williams,
	Eric Sunshine, Jonathan Tan, brian m . carlson

Stefan Beller <sbeller@google.com> writes:

> On Fri, Feb 23, 2018 at 11:50 AM, Junio C Hamano <gitster@pobox.com> wrote:
>> Stefan Beller <sbeller@google.com> writes:
>>
>>> I wonder if there is yet another missing case in the enumeration of
>>> the previous patch:
>>> Some commands are able to operate on GIT_OBJECT_DIR instead
>>> of GIT_DIR (git repack?), which may not even explore the full git directory,
>>> and so doesn't know about the hash value.
>>
>> ... because GIT_DIR/config is not known?  "repack" is not one of
>> them, though---it needs to at least use refs as the starting point
>> so a standalone OBJECT_DIR is insufficient.
>
> Yes, I could have worded this as a question:
> Is there any command that operates on GIT_OBJECT_DIR
> without trying to discover GIT_DIR ?

If somebody finds one that would be a good argument not to pursue
the approach.  Lack of response to the question would not amount to
that much---it is possible all people who bothered to think of
overlooked something obvious, though.  "When I asked around nobody
thought of a possibly way for this to cause breakages, so let's
declare it is safe to do so and do it" is not how we want to do
things X-<.


^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23  9:56         ` [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer" Nguyễn Thái Ngọc Duy
  2018-02-23 18:24           ` Stefan Beller
@ 2018-02-23 22:29           ` brian m. carlson
  2018-02-23 23:16             ` Brandon Williams
  2018-02-24  3:44             ` Duy Nguyen
  1 sibling, 2 replies; 239+ messages in thread
From: brian m. carlson @ 2018-02-23 22:29 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy
  Cc: git, Brandon Williams, Stefan Beller, Junio C Hamano,
	Eric Sunshine, Jonathan Tan

[-- Attachment #1: Type: text/plain, Size: 2502 bytes --]

On Fri, Feb 23, 2018 at 04:56:40PM +0700, Nguyễn Thái Ngọc Duy wrote:
> diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> index 7e3e1a461c..8ee935504e 100644
> --- a/builtin/index-pack.c
> +++ b/builtin/index-pack.c
> @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
>  	if (prefix && chdir(prefix))
>  		die(_("Cannot come back to cwd"));
>  
> +	if (!the_hash_algo) {
> +		warning(_("Running without a repository, assuming SHA-1 hash"));
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +	}

Is this warning going to be visible to users in the normal course of
operation?  If so, people are probably going to find this bothersome or
alarming.

>  	for (i = 1; i < argc; i++) {
>  		const char *arg = argv[i];
>  
> diff --git a/common-main.c b/common-main.c
> index 6a689007e7..12aec36794 100644
> --- a/common-main.c
> +++ b/common-main.c
> @@ -1,6 +1,7 @@
>  #include "cache.h"
>  #include "exec_cmd.h"
>  #include "attr.h"
> +#include "repository.h"
>  
>  /*
>   * Many parts of Git have subprograms communicate via pipe, expect the
> @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
>  
>  	restore_sigpipe_to_default();
>  
> +	if (getenv("GIT_HASH_FIXUP"))
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);

I'm lukewarm on adding this environment variable, but considering our
history here, we had probably better.  We can always remove it after a
few releases.

>  	return cmd_main(argc, argv);
>  }
> diff --git a/diff-no-index.c b/diff-no-index.c
> index 0ed5f0f496..f038f665bc 100644
> --- a/diff-no-index.c
> +++ b/diff-no-index.c
> @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
>  	struct strbuf replacement = STRBUF_INIT;
>  	const char *prefix = revs->prefix;
>  
> +	if (!the_hash_algo) {
> +		warning(_("Running without a repository, assuming SHA-1 hash"));
> +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> +	}

Again, same concern.  I can imagine scripts that will blow up loudly if
git diff --no-index spews things to standard error.

I'm not opposed to making this more visible, but I wonder if maybe it
should only be visible to developers or such.  The only way I can think
of doing is that is with an advice options, but maybe there's a better
way.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 13/27] sha1_file: add repository argument to prepare_alt_odb
  2018-02-22  6:51         ` Jonathan Nieder
@ 2018-02-23 22:33           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 22:33 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine

>> --- a/sha1_file.c
>> +++ b/sha1_file.c
>> @@ -23,6 +23,7 @@
>>  #include "sha1-lookup.h"
>>  #include "bulk-checkin.h"
>>  #include "repository.h"
>> +#include "object-store.h"
>
> Should this #include have been added in an earlier patch, since the
> file both uses and defines prepare_alt_odb, which is declared there?

This is another superflous include, dropped it. Thanks for catching.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
  2018-02-22  0:51         ` Brandon Williams
@ 2018-02-23 22:36           ` Stefan Beller
  2018-02-23 23:11             ` Brandon Williams
  0 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 22:36 UTC (permalink / raw)
  To: Brandon Williams
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine,
	Jonathan Nieder

On Wed, Feb 21, 2018 at 4:51 PM, Brandon Williams <bmwill@google.com> wrote:
> On 02/20, Stefan Beller wrote:
>> Add a repository argument to allow sha1_file_name callers to be more
>> specific about which repository to handle. This is a small mechanical
>> change; it doesn't change the implementation to handle repositories
>> other than the_repository yet.
>>
>> As with the previous commits, use a macro to catch callers passing a
>> repository other than the_repository at compile time.
>>
>> While at it, move the declaration to object-store.h, where it should
>> be easier to find.
>
> Seems like we may want to make a sha1-file.h or an oid-file.h or
> something like that at some point as that seems like a better place for
> the function than in the object-store.h file?

It depends what our long term goal is.
Do we want header and source file name to match for each function?
Or do we want a coarser set of headers, such that we have a broad
object-store.h, but that is implemented in sha1_file.c, packfile.c
for the parts of the raw_objectstore and other .c files for the higher
levels of the object store?

For now I'd just keep it in object-store.h as moving out just a couple
functions seems less consistent?

Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 0/2] Fix initializing the_hash_algo
  2018-02-23  9:56       ` [PATCH 0/2] Fix initializing the_hash_algo Nguyễn Thái Ngọc Duy
  2018-02-23  9:56         ` [PATCH 1/2] setup.c: initialize the_repository correctly in all cases Nguyễn Thái Ngọc Duy
  2018-02-23  9:56         ` [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer" Nguyễn Thái Ngọc Duy
@ 2018-02-23 22:47         ` brian m. carlson
  2018-02-24  3:34         ` [PATCH v2 0/5] " Nguyễn Thái Ngọc Duy
  3 siblings, 0 replies; 239+ messages in thread
From: brian m. carlson @ 2018-02-23 22:47 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy
  Cc: git, Brandon Williams, Stefan Beller, Junio C Hamano,
	Eric Sunshine, Jonathan Tan

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Fri, Feb 23, 2018 at 04:56:38PM +0700, Nguyễn Thái Ngọc Duy wrote:
> I can certainly try! I start to remember all the hairy details in that
> setup code.
> 
> The first step may be something like this, which identifies all the
> "repo init" entry points. This is basically a revert of e26f7f19b6
> (repository: pre-initialize hash algo pointer - 2018-01-19) and doing
> things the proper way, hopefully.
> 
> This is on 'master', independent from Stefan's series. I have another
> patch on top of that series to remove the use of ignore_env in
> sha1_file.c (and things seem to work). Basically whenever you have to
> initialize the hash algorithm, there's a good chance you need to
> initialize object store as well. But I'll hold that off until
> Stefan's and this one are both merged.

I definitely think this series is an improvement over my previous patch.

My major concern is alarming users (or breaking scripts) with the
warning message.  I wonder if deferring the use of the warning until we
have multiple hash algorithms might be a better idea.  At that point,
the warning would become something people could act upon.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
https://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 867 bytes --]

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
  2018-02-22  0:44         ` Jonathan Tan
@ 2018-02-23 22:50           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 22:50 UTC (permalink / raw)
  To: Jonathan Tan
  Cc: git, Junio C Hamano, Duy Nguyen, Eric Sunshine, Jonathan Nieder

On Wed, Feb 21, 2018 at 4:44 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
> On Tue, 20 Feb 2018 17:54:25 -0800
> Stefan Beller <sbeller@google.com> wrote:
>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
>
> Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
>
>> -void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
>> +void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned char *sha1)
>>  {
>> -     strbuf_addstr(buf, get_object_directory());
>> +     strbuf_addstr(buf, r->objects.objectdir);
>>       strbuf_addch(buf, '/');
>>       fill_sha1_path(buf, sha1);
>>  }
>
> In the future, we should probably have:
>  - a function to get the object store out of a repo (so that it can
>    lazily initialize the object store struct if necessary)
>  - when the object store is obtained, its objectdir field is guaranteed
>    to be populated
>  - sha1_file_name should take the object store struct, not the repo
>    struct

We had that in v2, but I reverted it as the consensus seemed that we prefer
the_repository to be passed around.

>
> but this is outside the scope of this patch.

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 16/27] sha1_file: add repository argument to sha1_file_name
  2018-02-23 22:36           ` Stefan Beller
@ 2018-02-23 23:11             ` Brandon Williams
  0 siblings, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-23 23:11 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, Junio C Hamano, Jonathan Tan, Duy Nguyen, Eric Sunshine,
	Jonathan Nieder

On 02/23, Stefan Beller wrote:
> On Wed, Feb 21, 2018 at 4:51 PM, Brandon Williams <bmwill@google.com> wrote:
> > On 02/20, Stefan Beller wrote:
> >> Add a repository argument to allow sha1_file_name callers to be more
> >> specific about which repository to handle. This is a small mechanical
> >> change; it doesn't change the implementation to handle repositories
> >> other than the_repository yet.
> >>
> >> As with the previous commits, use a macro to catch callers passing a
> >> repository other than the_repository at compile time.
> >>
> >> While at it, move the declaration to object-store.h, where it should
> >> be easier to find.
> >
> > Seems like we may want to make a sha1-file.h or an oid-file.h or
> > something like that at some point as that seems like a better place for
> > the function than in the object-store.h file?
> 
> It depends what our long term goal is.
> Do we want header and source file name to match for each function?
> Or do we want a coarser set of headers, such that we have a broad
> object-store.h, but that is implemented in sha1_file.c, packfile.c
> for the parts of the raw_objectstore and other .c files for the higher
> levels of the object store?
> 
> For now I'd just keep it in object-store.h as moving out just a couple
> functions seems less consistent?

Fair enough :)

-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 2/2] Revert "repository: pre-initialize hash algo pointer"
  2018-02-23 22:29           ` brian m. carlson
@ 2018-02-23 23:16             ` Brandon Williams
  2018-02-24  3:44             ` Duy Nguyen
  1 sibling, 0 replies; 239+ messages in thread
From: Brandon Williams @ 2018-02-23 23:16 UTC (permalink / raw)
  To: brian m. carlson, Nguyễn Thái Ngọc Duy, git,
	Stefan Beller, Junio C Hamano, Eric Sunshine, Jonathan Tan

On 02/23, brian m. carlson wrote:
> On Fri, Feb 23, 2018 at 04:56:40PM +0700, Nguyễn Thái Ngọc Duy wrote:
> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c
> > index 7e3e1a461c..8ee935504e 100644
> > --- a/builtin/index-pack.c
> > +++ b/builtin/index-pack.c
> > @@ -1673,6 +1673,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)
> >  	if (prefix && chdir(prefix))
> >  		die(_("Cannot come back to cwd"));
> >  
> > +	if (!the_hash_algo) {
> > +		warning(_("Running without a repository, assuming SHA-1 hash"));
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> > +	}
> 
> Is this warning going to be visible to users in the normal course of
> operation?  If so, people are probably going to find this bothersome or
> alarming.
> 
> >  	for (i = 1; i < argc; i++) {
> >  		const char *arg = argv[i];
> >  
> > diff --git a/common-main.c b/common-main.c
> > index 6a689007e7..12aec36794 100644
> > --- a/common-main.c
> > +++ b/common-main.c
> > @@ -1,6 +1,7 @@
> >  #include "cache.h"
> >  #include "exec_cmd.h"
> >  #include "attr.h"
> > +#include "repository.h"
> >  
> >  /*
> >   * Many parts of Git have subprograms communicate via pipe, expect the
> > @@ -40,5 +41,8 @@ int main(int argc, const char **argv)
> >  
> >  	restore_sigpipe_to_default();
> >  
> > +	if (getenv("GIT_HASH_FIXUP"))
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> 
> I'm lukewarm on adding this environment variable, but considering our
> history here, we had probably better.  We can always remove it after a
> few releases.

Yeah I don't know what the best thing to do here would be.  I mean we
could always just init the hash_algo for the_repository here so that we
don't ever have to worry about it, but I don't know if even that is the
right approach.

> 
> >  	return cmd_main(argc, argv);
> >  }
> > diff --git a/diff-no-index.c b/diff-no-index.c
> > index 0ed5f0f496..f038f665bc 100644
> > --- a/diff-no-index.c
> > +++ b/diff-no-index.c
> > @@ -241,6 +241,11 @@ void diff_no_index(struct rev_info *revs,
> >  	struct strbuf replacement = STRBUF_INIT;
> >  	const char *prefix = revs->prefix;
> >  
> > +	if (!the_hash_algo) {
> > +		warning(_("Running without a repository, assuming SHA-1 hash"));
> > +		repo_set_hash_algo(the_repository, GIT_HASH_SHA1);
> > +	}
> 
> Again, same concern.  I can imagine scripts that will blow up loudly if
> git diff --no-index spews things to standard error.
> 
> I'm not opposed to making this more visible, but I wonder if maybe it
> should only be visible to developers or such.  The only way I can think
> of doing is that is with an advice options, but maybe there's a better
> way.
> -- 
> brian m. carlson / brian with sandals: Houston, Texas, US
> https://www.crustytoothpaste.net/~bmc | My opinion only
> OpenPGP: https://keybase.io/bk2204



-- 
Brandon Williams

^ permalink raw reply	[flat|nested] 239+ messages in thread

* Re: [PATCH 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-22  0:35         ` Jonathan Tan
@ 2018-02-23 23:18           ` Stefan Beller
  0 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-23 23:18 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: git, Junio C Hamano, Duy Nguyen, Eric Sunshine

On Wed, Feb 21, 2018 at 4:35 PM, Jonathan Tan <jonathantanmy@google.com> wrote:
> On Tue, 20 Feb 2018 17:54:18 -0800
> Stefan Beller <sbeller@google.com> wrote:
>
>> -void prepare_alt_odb_the_repository(void)
>> +void prepare_alt_odb(struct repository *r)
>>  {
>> -     const char *alt;
>> -
>> -     if (the_repository->objects.alt_odb_tail)
>> +     if (r->objects.alt_odb_tail)
>>               return;
>>
>> -     alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>> +     r->objects.alt_odb_tail = &r->objects.alt_odb_list;
>> +
>> +     if (!r->ignore_env) {
>> +             const char *alt = getenv(ALTERNATE_DB_ENVIRONMENT);
>> +             if (!alt)
>> +                     alt = "";
>
> alt can be NULL, just like in the existing code, so these 2 lines are
> unnecessary. (link_alt_odb_entries() will just do nothing in either
> case.)
>
> (I also think that the check of absence of alt should be done by the
> caller of link_alt_odb_entries(), not by link_alt_odb_entries() itself,
> but that is much beyond the scope of this patch set.)
>

reverted to a literal translation of s/the_repository/r/, I forget
why I implemented this change in the first place. dropped for now.

Thanks,
Stefan

^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 00/27] Moving global state into the repository object (part 1)
  2018-02-21  1:54     ` [PATCHv3 00/27] " Stefan Beller
                         ` (27 preceding siblings ...)
  2018-02-22  0:26       ` [PATCHv3 00/27] Moving global state into the repository object (part 1) Stefan Beller
@ 2018-02-24  0:47       ` " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 01/27] repository: introduce raw object store field Stefan Beller
                           ` (28 more replies)
  28 siblings, 29 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

v4:
* addressed feedback from the single patches (mostly nits)
* rebased on latest master

v3:
* reverted back to use the repository for most of the functions
  (including unduplicating 'ignore_env')
* rebased on master again (I lost that state when doing v2, as
  I did both rebase as well as conversion to object store in one go)
* one additional patch, that moves the alternates related things to
  object-store.h, such that inclusion of cache.h (in object-store.h)
  is not required any more.

v2:
* duplicated the 'ignore_env' bit into the object store as well
* the #define trick no longer works as we do not have a "the_objectstore" global,
  which means there is just one patch per function that is converted.
  As this follows the same structure of the previous series, I am still confident
  there is no hidden dependencies to globals outside the object store in these
  converted functions.
* rebased on top of current master, resolving the merge conflicts.
  I think I used the list.h APIs right, but please double check.

Thanks,
Stefan

v1:
This is a real take on the first part of the recent RFC[1].

Jonathan Tan suggested[2] that "sha1_loose_object_info to handle arbitrary repositories"
might be a good breaking point for a first part at that RFC at patch 38.
This series is smaller and contains only 26 patches as the patches in the big
RFC were slightly out of order.

I developed this series partly by writing patches, but mostly by cherrypicking
from that RFC on top of current master. I noticed no external conflicts apart
from one addition to the repositories _INIT macro, which was easy to resolve.

Comments in the early range of that RFC were on 003 where Junio pointed out
that the coccinelle patch ought to be not in contrib/coccinelle, so I put it
in a sub directory there, as 'make coccicheck' doesn't traverse subdirs.

brian had a questoin on patch 25 in the RFC, but that seemed to resolve itself
without any suggestion to include into this series[3].

Duy suggested that we shall not use the repository blindly, but should carefully
examine whether to pass on an object store or the refstore or such[4], which
I agree with if it makes sense. This series unfortunately has an issue with that
as I would not want to pass down the `ignore_env` flag separately from the object
store, so I made all functions that only take the object store to have the raw
object store as the first parameter, and others using the full repository.

Eric Sunshine brought up memory leaks with the RFC, and I would think to
have plugged all holes.

[1] https://public-inbox.org/git/20180205235508.216277-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180207143300.ce1c39ca07f6a0d64fe0e7ca@google.com/
[3] https://public-inbox.org/git/20180206011940.GD7904@genre.crustytoothpaste.net/
[4] https://public-inbox.org/git/CACsJy8CGgekpX4cZkyyTSPrj87uQVKZSOL7fyT__P2dh_1LmVQ@mail.gmail.com/

Thanks,
Stefan

Jonathan Nieder (2):
  sha1_file: allow map_sha1_file_1 to handle arbitrary repositories
  sha1_file: allow sha1_loose_object_info to handle arbitrary
    repositories

Stefan Beller (25):
  repository: introduce raw object store field
  object-store: migrate alternates struct and functions from cache.h
  object-store: move alt_odb_list and alt_odb_tail to object store
  object-store: free alt_odb_list
  object-store: move packed_git and packed_git_mru to object store
  object-store: close all packs upon clearing the object store
  pack: move prepare_packed_git_run_once to object store
  pack: move approximate object count to object store
  sha1_file: add raw_object_store argument to alt_odb_usable
  sha1_file: add repository argument to link_alt_odb_entry
  sha1_file: add repository argument to read_info_alternates
  sha1_file: add repository argument to link_alt_odb_entries
  sha1_file: add repository argument to prepare_alt_odb
  sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  sha1_file: add repository argument to sha1_file_name
  sha1_file: add repository argument to stat_sha1_file
  sha1_file: add repository argument to open_sha1_file
  sha1_file: add repository argument to map_sha1_file_1
  sha1_file: add repository argument to map_sha1_file
  sha1_file: add repository argument to sha1_loose_object_info
  sha1_file: allow sha1_file_name to handle arbitrary repositories
  sha1_file: allow stat_sha1_file to handle arbitrary repositories
  sha1_file: allow open_sha1_file to handle arbitrary repositories
  sha1_file: allow map_sha1_file to handle arbitrary repositories

 builtin/am.c             |   2 +-
 builtin/clone.c          |   2 +-
 builtin/count-objects.c  |   5 +-
 builtin/fetch.c          |   2 +-
 builtin/fsck.c           |  12 ++--
 builtin/gc.c             |   3 +-
 builtin/grep.c           |   2 +-
 builtin/merge.c          |   2 +-
 builtin/pack-objects.c   |  20 ++++---
 builtin/pack-redundant.c |   5 +-
 builtin/receive-pack.c   |   2 +-
 cache.h                  |  87 ---------------------------
 environment.c            |   5 +-
 fast-import.c            |   7 ++-
 http-backend.c           |   5 +-
 http-walker.c            |   3 +-
 http.c                   |   5 +-
 object-store.h           | 124 ++++++++++++++++++++++++++++++++++++++
 object.c                 |  27 +++++++++
 pack-bitmap.c            |   3 +-
 packfile.c               |  64 ++++++++++----------
 packfile.h               |   2 +-
 path.c                   |   2 +-
 repository.c             |  21 +++++--
 repository.h             |   7 ++-
 server-info.c            |   5 +-
 sha1_file.c              | 125 +++++++++++++++++++++------------------
 sha1_name.c              |  10 ++--
 streaming.c              |   5 +-
 29 files changed, 335 insertions(+), 229 deletions(-)
 create mode 100644 object-store.h

-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 01/27] repository: introduce raw object store field
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-26  9:30           ` Duy Nguyen
  2018-02-24  0:47         ` [PATCHv4 02/27] object-store: migrate alternates struct and functions from cache.h Stefan Beller
                           ` (27 subsequent siblings)
  28 siblings, 1 reply; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

The raw object store field will contain any objects needed for
access to objects in a given repository.

This patch introduces the raw object store and populates it with the
`objectdir`, which used to be part of the repository struct.

As the struct gains members, we'll also populate the function to clear
the memory for these members.

In a later we'll introduce a struct object_parser, that will complement
the object parsing in a repository struct: The raw object parser is the
layer that will provide access to raw object content, while the higher
level object parser code will parse raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
For now only add the lower level to the repository struct.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/grep.c |  2 +-
 environment.c  |  5 +++--
 object-store.h | 15 +++++++++++++++
 object.c       |  5 +++++
 path.c         |  2 +-
 repository.c   | 19 ++++++++++++++-----
 repository.h   |  7 ++++---
 7 files changed, 43 insertions(+), 12 deletions(-)
 create mode 100644 object-store.h

diff --git a/builtin/grep.c b/builtin/grep.c
index 3ca4ac80d8c..0f0c195705f 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -432,7 +432,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject,
 	 * object.
 	 */
 	grep_read_lock();
-	add_to_alternates_memory(submodule.objectdir);
+	add_to_alternates_memory(submodule.objects.objectdir);
 	grep_read_unlock();
 
 	if (oid) {
diff --git a/environment.c b/environment.c
index de8431e01e7..ec10b062e68 100644
--- a/environment.c
+++ b/environment.c
@@ -13,6 +13,7 @@
 #include "refs.h"
 #include "fmt-merge-msg.h"
 #include "commit.h"
+#include "object-store.h"
 
 int trust_executable_bit = 1;
 int trust_ctime = 1;
@@ -244,9 +245,9 @@ const char *get_git_work_tree(void)
 
 char *get_object_directory(void)
 {
-	if (!the_repository->objectdir)
+	if (!the_repository->objects.objectdir)
 		BUG("git environment hasn't been setup");
-	return the_repository->objectdir;
+	return the_repository->objects.objectdir;
 }
 
 int odb_mkstemp(struct strbuf *template, const char *pattern)
diff --git a/object-store.h b/object-store.h
new file mode 100644
index 00000000000..cf35760ceb6
--- /dev/null
+++ b/object-store.h
@@ -0,0 +1,15 @@
+#ifndef OBJECT_STORE_H
+#define OBJECT_STORE_H
+
+struct raw_object_store {
+	/*
+	 * Path to the repository's object store.
+	 * Cannot be NULL after initialization.
+	 */
+	char *objectdir;
+};
+#define RAW_OBJECT_STORE_INIT { NULL }
+
+void raw_object_store_clear(struct raw_object_store *o);
+
+#endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 9e6f9ff20b0..a2acdee1405 100644
--- a/object.c
+++ b/object.c
@@ -445,3 +445,8 @@ void clear_commit_marks_all(unsigned int flags)
 			obj->flags &= ~flags;
 	}
 }
+
+void raw_object_store_clear(struct raw_object_store *o)
+{
+	FREE_AND_NULL(o->objectdir);
+}
diff --git a/path.c b/path.c
index da8b655730d..81a42d91155 100644
--- a/path.c
+++ b/path.c
@@ -382,7 +382,7 @@ static void adjust_git_path(const struct repository *repo,
 		strbuf_splice(buf, 0, buf->len,
 			      repo->index_file, strlen(repo->index_file));
 	else if (dir_prefix(base, "objects"))
-		replace_dir(buf, git_dir_len + 7, repo->objectdir);
+		replace_dir(buf, git_dir_len + 7, repo->objects.objectdir);
 	else if (git_hooks_path && dir_prefix(base, "hooks"))
 		replace_dir(buf, git_dir_len + 5, git_hooks_path);
 	else if (repo->different_commondir)
diff --git a/repository.c b/repository.c
index 4ffbe9bc94e..fb65e8072d5 100644
--- a/repository.c
+++ b/repository.c
@@ -1,11 +1,18 @@
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 #include "config.h"
 #include "submodule-config.h"
 
 /* The main repository */
 static struct repository the_repo = {
-	NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0
+	NULL, NULL,
+	RAW_OBJECT_STORE_INIT,
+	NULL, NULL, NULL,
+	NULL, NULL, NULL,
+	&the_index,
+	&hash_algos[GIT_HASH_SHA1],
+	0, 0
 };
 struct repository *the_repository = &the_repo;
 
@@ -42,9 +49,10 @@ static void repo_setup_env(struct repository *repo)
 						    !repo->ignore_env);
 	free(repo->commondir);
 	repo->commondir = strbuf_detach(&sb, NULL);
-	free(repo->objectdir);
-	repo->objectdir = git_path_from_env(DB_ENVIRONMENT, repo->commondir,
-					    "objects", !repo->ignore_env);
+	raw_object_store_clear(&repo->objects);
+	repo->objects.objectdir =
+		git_path_from_env(DB_ENVIRONMENT, repo->commondir,
+				  "objects", !repo->ignore_env);
 	free(repo->graft_file);
 	repo->graft_file = git_path_from_env(GRAFT_ENVIRONMENT, repo->commondir,
 					     "info/grafts", !repo->ignore_env);
@@ -209,12 +217,13 @@ void repo_clear(struct repository *repo)
 {
 	FREE_AND_NULL(repo->gitdir);
 	FREE_AND_NULL(repo->commondir);
-	FREE_AND_NULL(repo->objectdir);
 	FREE_AND_NULL(repo->graft_file);
 	FREE_AND_NULL(repo->index_file);
 	FREE_AND_NULL(repo->worktree);
 	FREE_AND_NULL(repo->submodule_prefix);
 
+	raw_object_store_clear(&repo->objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index 0329e40c7f5..fa73ab8e938 100644
--- a/repository.h
+++ b/repository.h
@@ -1,6 +1,8 @@
 #ifndef REPOSITORY_H
 #define REPOSITORY_H
 
+#include "object-store.h"
+
 struct config_set;
 struct index_state;
 struct submodule_cache;
@@ -21,10 +23,9 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Path to the repository's object store.
-	 * Cannot be NULL after initialization.
+	 * Holds any information related to accessing the raw object content.
 	 */
-	char *objectdir;
+	struct raw_object_store objects;
 
 	/*
 	 * Path to the repository's graft file.
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 02/27] object-store: migrate alternates struct and functions from cache.h
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 01/27] repository: introduce raw object store field Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 03/27] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
                           ` (26 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

Migrate the struct alternate_object_database and all its related
functions to the object store as these functions are easier found in
that header. The migration is just a verbatim copy, no need to
include the object store header at any C file, because cache.h includes
repository.h which in turn includes the object-store.h

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 cache.h        | 51 --------------------------------------------------
 object-store.h | 51 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 51 insertions(+), 51 deletions(-)

diff --git a/cache.h b/cache.h
index 21fbcc24149..99eb2bdb893 100644
--- a/cache.h
+++ b/cache.h
@@ -1560,57 +1560,6 @@ extern int has_dirs_only_path(const char *name, int len, int prefix_len);
 extern void schedule_dir_for_removal(const char *name, int len);
 extern void remove_scheduled_dirs(void);
 
-extern struct alternate_object_database {
-	struct alternate_object_database *next;
-
-	/* see alt_scratch_buf() */
-	struct strbuf scratch;
-	size_t base_len;
-
-	/*
-	 * Used to store the results of readdir(3) calls when searching
-	 * for unique abbreviated hashes.  This cache is never
-	 * invalidated, thus it's racy and not necessarily accurate.
-	 * That's fine for its purpose; don't use it for tasks requiring
-	 * greater accuracy!
-	 */
-	char loose_objects_subdir_seen[256];
-	struct oid_array loose_objects_cache;
-
-	char path[FLEX_ARRAY];
-} *alt_odb_list;
-extern void prepare_alt_odb(void);
-extern char *compute_alternate_path(const char *path, struct strbuf *err);
-typedef int alt_odb_fn(struct alternate_object_database *, void *);
-extern int foreach_alt_odb(alt_odb_fn, void*);
-
-/*
- * Allocate a "struct alternate_object_database" but do _not_ actually
- * add it to the list of alternates.
- */
-struct alternate_object_database *alloc_alt_odb(const char *dir);
-
-/*
- * Add the directory to the on-disk alternates file; the new entry will also
- * take effect in the current process.
- */
-extern void add_to_alternates_file(const char *dir);
-
-/*
- * Add the directory to the in-memory list of alternates (along with any
- * recursive alternates it points to), but do not modify the on-disk alternates
- * file.
- */
-extern void add_to_alternates_memory(const char *dir);
-
-/*
- * Returns a scratch strbuf pre-filled with the alternate object directory,
- * including a trailing slash, which can be used to access paths in the
- * alternate. Always use this over direct access to alt->scratch, as it
- * cleans up any previous use of the scratch buffer.
- */
-extern struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
-
 struct pack_window {
 	struct pack_window *next;
 	unsigned char *base;
diff --git a/object-store.h b/object-store.h
index cf35760ceb6..5678aa1136c 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,6 +1,57 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
+extern struct alternate_object_database {
+	struct alternate_object_database *next;
+
+	/* see alt_scratch_buf() */
+	struct strbuf scratch;
+	size_t base_len;
+
+	/*
+	 * Used to store the results of readdir(3) calls when searching
+	 * for unique abbreviated hashes.  This cache is never
+	 * invalidated, thus it's racy and not necessarily accurate.
+	 * That's fine for its purpose; don't use it for tasks requiring
+	 * greater accuracy!
+	 */
+	char loose_objects_subdir_seen[256];
+	struct oid_array loose_objects_cache;
+
+	char path[FLEX_ARRAY];
+} *alt_odb_list;
+void prepare_alt_odb(void);
+char *compute_alternate_path(const char *path, struct strbuf *err);
+typedef int alt_odb_fn(struct alternate_object_database *, void *);
+int foreach_alt_odb(alt_odb_fn, void*);
+
+/*
+ * Allocate a "struct alternate_object_database" but do _not_ actually
+ * add it to the list of alternates.
+ */
+struct alternate_object_database *alloc_alt_odb(const char *dir);
+
+/*
+ * Add the directory to the on-disk alternates file; the new entry will also
+ * take effect in the current process.
+ */
+void add_to_alternates_file(const char *dir);
+
+/*
+ * Add the directory to the in-memory list of alternates (along with any
+ * recursive alternates it points to), but do not modify the on-disk alternates
+ * file.
+ */
+void add_to_alternates_memory(const char *dir);
+
+/*
+ * Returns a scratch strbuf pre-filled with the alternate object directory,
+ * including a trailing slash, which can be used to access paths in the
+ * alternate. Always use this over direct access to alt->scratch, as it
+ * cleans up any previous use of the scratch buffer.
+ */
+struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 03/27] object-store: move alt_odb_list and alt_odb_tail to object store
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 01/27] repository: introduce raw object store field Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 02/27] object-store: migrate alternates struct and functions from cache.h Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 04/27] object-store: free alt_odb_list Stefan Beller
                           ` (25 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

In a process with multiple repositories open, alternates should be
associated to a single repository and not shared globally. Move
alt_odb_list and alt_odb_tail into the_repository and adjust callers
to reflect this.

Now that the alternative object data base is per repository, we're
leaking its memory upon freeing a repository. The next patch plugs
this hole.

No functional change intended.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  4 +++-
 object-store.h |  9 ++++++---
 packfile.c     |  3 ++-
 sha1_file.c    | 25 ++++++++++++-------------
 sha1_name.c    |  3 ++-
 5 files changed, 25 insertions(+), 19 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 9981db22637..78edcb75c45 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "commit.h"
 #include "tree.h"
@@ -722,7 +723,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 		fsck_object_dir(get_object_directory());
 
 		prepare_alt_odb();
-		for (alt = alt_odb_list; alt; alt = alt->next)
+		for (alt = the_repository->objects.alt_odb_list;
+		     alt; alt = alt->next)
 			fsck_object_dir(alt->path);
 
 		if (check_full) {
diff --git a/object-store.h b/object-store.h
index 5678aa1136c..e78eea1dde3 100644
--- a/object-store.h
+++ b/object-store.h
@@ -1,7 +1,7 @@
 #ifndef OBJECT_STORE_H
 #define OBJECT_STORE_H
 
-extern struct alternate_object_database {
+struct alternate_object_database {
 	struct alternate_object_database *next;
 
 	/* see alt_scratch_buf() */
@@ -19,7 +19,7 @@ extern struct alternate_object_database {
 	struct oid_array loose_objects_cache;
 
 	char path[FLEX_ARRAY];
-} *alt_odb_list;
+};
 void prepare_alt_odb(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
@@ -58,8 +58,11 @@ struct raw_object_store {
 	 * Cannot be NULL after initialization.
 	 */
 	char *objectdir;
+
+	struct alternate_object_database *alt_odb_list;
+	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL }
+#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 7dbe8739d17..216ea836ee3 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "list.h"
 #include "pack.h"
+#include "repository.h"
 #include "dir.h"
 #include "mergesort.h"
 #include "packfile.h"
@@ -891,7 +892,7 @@ void prepare_packed_git(void)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next)
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
diff --git a/sha1_file.c b/sha1_file.c
index 826d7a0ae37..ac25146076e 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -22,6 +22,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "repository.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -340,9 +341,6 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 	return buf->buf;
 }
 
-struct alternate_object_database *alt_odb_list;
-static struct alternate_object_database **alt_odb_tail;
-
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
@@ -362,7 +360,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -422,8 +420,8 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*alt_odb_tail = ent;
-	alt_odb_tail = &(ent->next);
+	*the_repository->objects.alt_odb_tail = ent;
+	the_repository->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
@@ -557,7 +555,7 @@ void add_to_alternates_file(const char *reference)
 		fprintf_or_die(out, "%s\n", reference);
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
-		if (alt_odb_tail)
+		if (the_repository->objects.alt_odb_tail)
 			link_alt_odb_entries(reference, '\n', NULL, 0);
 	}
 	free(alts);
@@ -655,7 +653,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	int r = 0;
 
 	prepare_alt_odb();
-	for (ent = alt_odb_list; ent; ent = ent->next) {
+	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
 			break;
@@ -667,12 +665,13 @@ void prepare_alt_odb(void)
 {
 	const char *alt;
 
-	if (alt_odb_tail)
+	if (the_repository->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	alt_odb_tail = &alt_odb_list;
+	the_repository->objects.alt_odb_tail =
+			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
 	read_info_alternates(get_object_directory(), 0);
@@ -716,7 +715,7 @@ static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
 			return 1;
@@ -876,7 +875,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 
 	prepare_alt_odb();
 	errno = ENOENT;
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		if (!lstat(*path, st))
 			return 0;
@@ -906,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	most_interesting_errno = errno;
 
 	prepare_alt_odb();
-	for (alt = alt_odb_list; alt; alt = alt->next) {
+	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
 		if (fd >= 0)
diff --git a/sha1_name.c b/sha1_name.c
index 611c7d24dde..957ce25680b 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -10,6 +10,7 @@
 #include "dir.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "repository.h"
 
 static int get_oid_oneline(const char *, struct object_id *, struct commit_list *);
 
@@ -104,7 +105,7 @@ static void find_short_object_filename(struct disambiguate_state *ds)
 		 */
 		fakeent = alloc_alt_odb(get_object_directory());
 	}
-	fakeent->next = alt_odb_list;
+	fakeent->next = the_repository->objects.alt_odb_list;
 
 	for (alt = fakeent; alt && !ds->ambiguous; alt = alt->next) {
 		int pos;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 04/27] object-store: free alt_odb_list
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (2 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 03/27] object-store: move alt_odb_list and alt_odb_tail to object store Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
                           ` (24 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

Free the memory and reset alt_odb_{list, tail} to NULL.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/object.c b/object.c
index a2acdee1405..920dc4463fa 100644
--- a/object.c
+++ b/object.c
@@ -446,7 +446,24 @@ void clear_commit_marks_all(unsigned int flags)
 	}
 }
 
+static void free_alt_odb(struct alternate_object_database *alt)
+{
+	strbuf_release(&alt->scratch);
+	oid_array_clear(&alt->loose_objects_cache);
+}
+
+static void free_alt_odbs(struct raw_object_store *o)
+{
+	while (o->alt_odb_list) {
+		free_alt_odb(o->alt_odb_list);
+		o->alt_odb_list = o->alt_odb_list->next;
+	}
+}
+
 void raw_object_store_clear(struct raw_object_store *o)
 {
 	FREE_AND_NULL(o->objectdir);
+
+	free_alt_odbs(o);
+	o->alt_odb_tail = NULL;
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 05/27] object-store: move packed_git and packed_git_mru to object store
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (3 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 04/27] object-store: free alt_odb_list Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 06/27] object-store: close all packs upon clearing the " Stefan Beller
                           ` (23 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

In a process with multiple repositories open, packfile accessors
should be associated to a single repository and not shared globally.
Move packed_git and packed_git_mru into the_repository and adjust
callers to reflect this.

Patch generated by

 1. Moving the struct packed_git declaration to object-store.h
    and packed_git, packed_git_mru globals to struct object_store.

 2. Apply the following semantic patch to adjust callers:
    @@ @@
    - packed_git
    + the_repository->objects.packed_git

    @@ @@
    - packed_git_mru
    + the_repository->objects.packed_git_mru

 3. Applying line wrapping fixes from "make style" to break the
    resulting long lines.

 4. Adding missing #includes of repository.h where needed.

 5. As the packfiles are now owned by an objectstore, which is ephemeral
    unlike globals, we introduce memory leaks. So address them in
    raw_object_store_clear(). Defer freeing packed_git to the next
    patch due to the size of this patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/count-objects.c  |  5 +++--
 builtin/fsck.c           |  6 ++++--
 builtin/gc.c             |  3 ++-
 builtin/pack-objects.c   | 20 +++++++++++---------
 builtin/pack-redundant.c |  5 +++--
 cache.h                  | 29 -----------------------------
 fast-import.c            |  7 +++++--
 http-backend.c           |  5 +++--
 object-store.h           | 31 ++++++++++++++++++++++++++++++-
 object.c                 |  6 ++++++
 pack-bitmap.c            |  3 ++-
 packfile.c               | 39 ++++++++++++++++++++-------------------
 repository.c             |  4 +++-
 server-info.c            |  5 +++--
 sha1_name.c              |  5 +++--
 15 files changed, 98 insertions(+), 75 deletions(-)

diff --git a/builtin/count-objects.c b/builtin/count-objects.c
index 33343818c83..6aab8d08fcc 100644
--- a/builtin/count-objects.c
+++ b/builtin/count-objects.c
@@ -7,6 +7,7 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "repository.h"
 #include "builtin.h"
 #include "parse-options.h"
 #include "quote.h"
@@ -120,9 +121,9 @@ int cmd_count_objects(int argc, const char **argv, const char *prefix)
 		struct strbuf loose_buf = STRBUF_INIT;
 		struct strbuf pack_buf = STRBUF_INIT;
 		struct strbuf garbage_buf = STRBUF_INIT;
-		if (!packed_git)
+		if (!the_repository->objects.packed_git)
 			prepare_packed_git();
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local)
 				continue;
 			if (open_pack_index(p))
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 78edcb75c45..c3a339193c8 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -735,7 +735,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 			prepare_packed_git();
 
 			if (show_progress) {
-				for (p = packed_git; p; p = p->next) {
+				for (p = the_repository->objects.packed_git; p;
+				     p = p->next) {
 					if (open_pack_index(p))
 						continue;
 					total += p->num_objects;
@@ -743,7 +744,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 
 				progress = start_progress(_("Checking objects"), total);
 			}
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				/* verify gives error messages itself */
 				if (verify_pack(p, fsck_obj_buffer,
 						progress, count))
diff --git a/builtin/gc.c b/builtin/gc.c
index 77fa720bd0b..59bf5c2ea0b 100644
--- a/builtin/gc.c
+++ b/builtin/gc.c
@@ -11,6 +11,7 @@
  */
 
 #include "builtin.h"
+#include "repository.h"
 #include "config.h"
 #include "tempfile.h"
 #include "lockfile.h"
@@ -173,7 +174,7 @@ static int too_many_packs(void)
 		return 0;
 
 	prepare_packed_git();
-	for (cnt = 0, p = packed_git; p; p = p->next) {
+	for (cnt = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		if (p->pack_keep)
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 5c674b2843c..a64005760be 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "attr.h"
 #include "object.h"
@@ -1025,8 +1026,7 @@ static int want_object_in_pack(const struct object_id *oid,
 		if (want != -1)
 			return want;
 	}
-
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		off_t offset;
 
@@ -1044,7 +1044,8 @@ static int want_object_in_pack(const struct object_id *oid,
 			}
 			want = want_found_object(exclude, p);
 			if (!exclude && want > 0)
-				list_move(&p->mru, &packed_git_mru);
+				list_move(&p->mru,
+					  &the_repository->objects.packed_git_mru);
 			if (want != -1)
 				return want;
 		}
@@ -2673,7 +2674,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 
 	memset(&in_pack, 0, sizeof(in_pack));
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		struct object_id oid;
 		struct object *o;
 
@@ -2736,7 +2737,8 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 	static struct packed_git *last_found = (void *)1;
 	struct packed_git *p;
 
-	p = (last_found != (void *)1) ? last_found : packed_git;
+	p = (last_found != (void *)1) ? last_found :
+					the_repository->objects.packed_git;
 
 	while (p) {
 		if ((!p->pack_local || p->pack_keep) &&
@@ -2745,7 +2747,7 @@ static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid)
 			return 1;
 		}
 		if (p == last_found)
-			p = packed_git;
+			p = the_repository->objects.packed_git;
 		else
 			p = p->next;
 		if (p == last_found)
@@ -2781,7 +2783,7 @@ static void loosen_unused_packed_objects(struct rev_info *revs)
 	uint32_t i;
 	struct object_id oid;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local || p->pack_keep)
 			continue;
 
@@ -3152,7 +3154,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 	prepare_packed_git();
 	if (ignore_packed_keep) {
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next)
+		for (p = the_repository->objects.packed_git; p; p = p->next)
 			if (p->pack_local && p->pack_keep)
 				break;
 		if (!p) /* no keep-able packs found */
@@ -3165,7 +3167,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)
 		 * also covers non-local objects
 		 */
 		struct packed_git *p;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (!p->pack_local) {
 				have_non_local_packs = 1;
 				break;
diff --git a/builtin/pack-redundant.c b/builtin/pack-redundant.c
index aaa81363222..a196fae79d4 100644
--- a/builtin/pack-redundant.c
+++ b/builtin/pack-redundant.c
@@ -7,6 +7,7 @@
 */
 
 #include "builtin.h"
+#include "repository.h"
 #include "packfile.h"
 
 #define BLKSIZE 512
@@ -571,7 +572,7 @@ static struct pack_list * add_pack(struct packed_git *p)
 
 static struct pack_list * add_pack_file(const char *filename)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	if (strlen(filename) < 40)
 		die("Bad pack filename: %s", filename);
@@ -586,7 +587,7 @@ static struct pack_list * add_pack_file(const char *filename)
 
 static void load_all(void)
 {
-	struct packed_git *p = packed_git;
+	struct packed_git *p = the_repository->objects.packed_git;
 
 	while (p) {
 		add_pack(p);
diff --git a/cache.h b/cache.h
index 99eb2bdb893..63889acb591 100644
--- a/cache.h
+++ b/cache.h
@@ -1569,35 +1569,6 @@ struct pack_window {
 	unsigned int inuse_cnt;
 };
 
-extern struct packed_git {
-	struct packed_git *next;
-	struct list_head mru;
-	struct pack_window *windows;
-	off_t pack_size;
-	const void *index_data;
-	size_t index_size;
-	uint32_t num_objects;
-	uint32_t num_bad_objects;
-	unsigned char *bad_object_sha1;
-	int index_version;
-	time_t mtime;
-	int pack_fd;
-	unsigned pack_local:1,
-		 pack_keep:1,
-		 freshened:1,
-		 do_not_close:1,
-		 pack_promisor:1;
-	unsigned char sha1[20];
-	struct revindex_entry *revindex;
-	/* something like ".git/objects/pack/xxxxx.pack" */
-	char pack_name[FLEX_ARRAY]; /* more */
-} *packed_git;
-
-/*
- * A most-recently-used ordered version of the packed_git list.
- */
-extern struct list_head packed_git_mru;
-
 struct pack_entry {
 	off_t offset;
 	unsigned char sha1[20];
diff --git a/fast-import.c b/fast-import.c
index 0f818cd9328..3d7002d1cd6 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -154,6 +154,7 @@ Format of STDIN stream:
 
 #include "builtin.h"
 #include "cache.h"
+#include "repository.h"
 #include "config.h"
 #include "lockfile.h"
 #include "object.h"
@@ -1110,7 +1111,8 @@ static int store_object(
 	if (e->idx.offset) {
 		duplicate_count_by_type[type]++;
 		return 1;
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash,
+				  the_repository->objects.packed_git)) {
 		e->type = type;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
@@ -1307,7 +1309,8 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)
 		duplicate_count_by_type[OBJ_BLOB]++;
 		truncate_pack(&checkpoint);
 
-	} else if (find_sha1_pack(oid.hash, packed_git)) {
+	} else if (find_sha1_pack(oid.hash,
+				  the_repository->objects.packed_git)) {
 		e->type = OBJ_BLOB;
 		e->pack_id = MAX_PACK_ID;
 		e->idx.offset = 1; /* just not zero! */
diff --git a/http-backend.c b/http-backend.c
index f3dc218b2a3..85b6a97923f 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "config.h"
+#include "repository.h"
 #include "refs.h"
 #include "pkt-line.h"
 #include "object.h"
@@ -518,13 +519,13 @@ static void get_info_packs(struct strbuf *hdr, char *arg)
 
 	select_getanyfile(hdr);
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			cnt++;
 	}
 
 	strbuf_grow(&buf, cnt * 53 + 2);
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);
 	}
diff --git a/object-store.h b/object-store.h
index e78eea1dde3..4f768465a14 100644
--- a/object-store.h
+++ b/object-store.h
@@ -52,6 +52,30 @@ void add_to_alternates_memory(const char *dir);
  */
 struct strbuf *alt_scratch_buf(struct alternate_object_database *alt);
 
+struct packed_git {
+	struct packed_git *next;
+	struct list_head mru;
+	struct pack_window *windows;
+	off_t pack_size;
+	const void *index_data;
+	size_t index_size;
+	uint32_t num_objects;
+	uint32_t num_bad_objects;
+	unsigned char *bad_object_sha1;
+	int index_version;
+	time_t mtime;
+	int pack_fd;
+	unsigned pack_local:1,
+		 pack_keep:1,
+		 freshened:1,
+		 do_not_close:1,
+		 pack_promisor:1;
+	unsigned char sha1[20];
+	struct revindex_entry *revindex;
+	/* something like ".git/objects/pack/xxxxx.pack" */
+	char pack_name[FLEX_ARRAY]; /* more */
+};
+
 struct raw_object_store {
 	/*
 	 * Path to the repository's object store.
@@ -59,10 +83,15 @@ struct raw_object_store {
 	 */
 	char *objectdir;
 
+	struct packed_git *packed_git;
+	/* A most-recently-used ordered version of the packed_git list. */
+	struct list_head packed_git_mru;
+
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 };
-#define RAW_OBJECT_STORE_INIT { NULL, NULL, NULL }
+
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/object.c b/object.c
index 920dc4463fa..367441efa94 100644
--- a/object.c
+++ b/object.c
@@ -466,4 +466,10 @@ void raw_object_store_clear(struct raw_object_store *o)
 
 	free_alt_odbs(o);
 	o->alt_odb_tail = NULL;
+
+	INIT_LIST_HEAD(&o->packed_git_mru);
+	/*
+	 * TODO: call close_all_packs once migrated to
+	 * take an object store argument
+	 */
 }
diff --git a/pack-bitmap.c b/pack-bitmap.c
index 9270983e5f5..bbede32589a 100644
--- a/pack-bitmap.c
+++ b/pack-bitmap.c
@@ -10,6 +10,7 @@
 #include "pack-revindex.h"
 #include "pack-objects.h"
 #include "packfile.h"
+#include "repository.h"
 
 /*
  * An entry on the bitmap index, representing the bitmap for a given
@@ -335,7 +336,7 @@ static int open_pack_bitmap(void)
 	assert(!bitmap_git.map && !bitmap_git.loaded);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (open_pack_bitmap_1(p) == 0)
 			ret = 0;
 	}
diff --git a/packfile.c b/packfile.c
index 216ea836ee3..59210deaaf7 100644
--- a/packfile.c
+++ b/packfile.c
@@ -45,8 +45,6 @@ static unsigned int pack_open_fds;
 static unsigned int pack_max_fds;
 static size_t peak_pack_mapped;
 static size_t pack_mapped;
-struct packed_git *packed_git;
-LIST_HEAD(packed_git_mru);
 
 #define SZ_FMT PRIuMAX
 static inline uintmax_t sz_fmt(size_t s) { return s; }
@@ -246,7 +244,7 @@ static int unuse_one_window(struct packed_git *current)
 
 	if (current)
 		scan_windows(current, &lru_p, &lru_w, &lru_l);
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		scan_windows(p, &lru_p, &lru_w, &lru_l);
 	if (lru_p) {
 		munmap(lru_w->base, lru_w->len);
@@ -316,7 +314,7 @@ void close_all_packs(void)
 {
 	struct packed_git *p;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
@@ -384,7 +382,7 @@ static int close_one_pack(void)
 	struct pack_window *mru_w = NULL;
 	int accept_windows_inuse = 1;
 
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if (p->pack_fd == -1)
 			continue;
 		find_lru_pack(p, &lru_p, &mru_w, &accept_windows_inuse);
@@ -686,8 +684,8 @@ void install_packed_git(struct packed_git *pack)
 	if (pack->pack_fd != -1)
 		pack_open_fds++;
 
-	pack->next = packed_git;
-	packed_git = pack;
+	pack->next = the_repository->objects.packed_git;
+	the_repository->objects.packed_git = pack;
 }
 
 void (*report_garbage)(unsigned seen_bits, const char *path);
@@ -769,7 +767,8 @@ static void prepare_packed_git_one(char *objdir, int local)
 		base_len = path.len;
 		if (strip_suffix_mem(path.buf, &base_len, ".idx")) {
 			/* Don't reopen a pack we already have. */
-			for (p = packed_git; p; p = p->next) {
+			for (p = the_repository->objects.packed_git; p;
+			     p = p->next) {
 				size_t len;
 				if (strip_suffix(p->pack_name, ".pack", &len) &&
 				    len == base_len &&
@@ -820,7 +819,7 @@ unsigned long approximate_object_count(void)
 
 		prepare_packed_git();
 		count = 0;
-		for (p = packed_git; p; p = p->next) {
+		for (p = the_repository->objects.packed_git; p; p = p->next) {
 			if (open_pack_index(p))
 				continue;
 			count += p->num_objects;
@@ -869,18 +868,19 @@ static int sort_pack(const void *a_, const void *b_)
 
 static void rearrange_packed_git(void)
 {
-	packed_git = llist_mergesort(packed_git, get_next_packed_git,
-				     set_next_packed_git, sort_pack);
+	the_repository->objects.packed_git = llist_mergesort(
+		the_repository->objects.packed_git, get_next_packed_git,
+		set_next_packed_git, sort_pack);
 }
 
 static void prepare_packed_git_mru(void)
 {
 	struct packed_git *p;
 
-	INIT_LIST_HEAD(&packed_git_mru);
+	INIT_LIST_HEAD(&the_repository->objects.packed_git_mru);
 
-	for (p = packed_git; p; p = p->next)
-		list_add_tail(&p->mru, &packed_git_mru);
+	for (p = the_repository->objects.packed_git; p; p = p->next)
+		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
 static int prepare_packed_git_run_once = 0;
@@ -1014,7 +1014,7 @@ const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	struct packed_git *p;
 	unsigned i;
 
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		for (i = 0; i < p->num_bad_objects; i++)
 			if (!hashcmp(sha1, p->bad_object_sha1 + 20 * i))
 				return p;
@@ -1845,13 +1845,14 @@ int find_pack_entry(const unsigned char *sha1, struct pack_entry *e)
 	struct list_head *pos;
 
 	prepare_packed_git();
-	if (!packed_git)
+	if (!the_repository->objects.packed_git)
 		return 0;
 
-	list_for_each(pos, &packed_git_mru) {
+	list_for_each(pos, &the_repository->objects.packed_git_mru) {
 		struct packed_git *p = list_entry(pos, struct packed_git, mru);
 		if (fill_pack_entry(sha1, e, p)) {
-			list_move(&p->mru, &packed_git_mru);
+			list_move(&p->mru,
+				  &the_repository->objects.packed_git_mru);
 			return 1;
 		}
 	}
@@ -1898,7 +1899,7 @@ int for_each_packed_object(each_packed_object_fn cb, void *data, unsigned flags)
 	int pack_errors = 0;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		if ((flags & FOR_EACH_OBJECT_LOCAL_ONLY) && !p->pack_local)
 			continue;
 		if ((flags & FOR_EACH_OBJECT_PROMISOR_ONLY) &&
diff --git a/repository.c b/repository.c
index fb65e8072d5..a069b1b6404 100644
--- a/repository.c
+++ b/repository.c
@@ -7,7 +7,7 @@
 /* The main repository */
 static struct repository the_repo = {
 	NULL, NULL,
-	RAW_OBJECT_STORE_INIT,
+	RAW_OBJECT_STORE_INIT(the_repo.objects),
 	NULL, NULL, NULL,
 	NULL, NULL, NULL,
 	&the_index,
@@ -143,6 +143,8 @@ int repo_init(struct repository *repo, const char *gitdir, const char *worktree)
 
 	repo->ignore_env = 1;
 
+	INIT_LIST_HEAD(&repo->objects.packed_git_mru);
+
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
 
diff --git a/server-info.c b/server-info.c
index 26a6c20b7d4..c369ac27cbc 100644
--- a/server-info.c
+++ b/server-info.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "refs.h"
 #include "object.h"
 #include "commit.h"
@@ -200,7 +201,7 @@ static void init_pack_info(const char *infofile, int force)
 	objdirlen = strlen(objdir);
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next) {
+	for (p = the_repository->objects.packed_git; p; p = p->next) {
 		/* we ignore things on alternate path since they are
 		 * not available to the pullers in general.
 		 */
@@ -210,7 +211,7 @@ static void init_pack_info(const char *infofile, int force)
 	}
 	num_pack = i;
 	info = xcalloc(num_pack, sizeof(struct pack_info *));
-	for (i = 0, p = packed_git; p; p = p->next) {
+	for (i = 0, p = the_repository->objects.packed_git; p; p = p->next) {
 		if (!p->pack_local)
 			continue;
 		info[i] = xcalloc(1, sizeof(struct pack_info));
diff --git a/sha1_name.c b/sha1_name.c
index 957ce25680b..016c883b5c7 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -196,7 +196,8 @@ static void find_short_packed_object(struct disambiguate_state *ds)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p && !ds->ambiguous; p = p->next)
+	for (p = the_repository->objects.packed_git; p && !ds->ambiguous;
+	     p = p->next)
 		unique_in_pack(p, ds);
 }
 
@@ -566,7 +567,7 @@ static void find_abbrev_len_packed(struct min_abbrev_data *mad)
 	struct packed_git *p;
 
 	prepare_packed_git();
-	for (p = packed_git; p; p = p->next)
+	for (p = the_repository->objects.packed_git; p; p = p->next)
 		find_abbrev_len_for_pack(p, mad);
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 06/27] object-store: close all packs upon clearing the object store
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (4 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 05/27] object-store: move packed_git and packed_git_mru to object store Stefan Beller
@ 2018-02-24  0:47         ` " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 07/27] pack: move prepare_packed_git_run_once to " Stefan Beller
                           ` (22 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/am.c           | 2 +-
 builtin/clone.c        | 2 +-
 builtin/fetch.c        | 2 +-
 builtin/merge.c        | 2 +-
 builtin/receive-pack.c | 2 +-
 object.c               | 7 +++----
 packfile.c             | 4 ++--
 packfile.h             | 2 +-
 8 files changed, 11 insertions(+), 12 deletions(-)

diff --git a/builtin/am.c b/builtin/am.c
index 6661edc162b..fc1d9b43c51 100644
--- a/builtin/am.c
+++ b/builtin/am.c
@@ -1859,7 +1859,7 @@ static void am_run(struct am_state *state, int resume)
 	 */
 	if (!state->rebasing) {
 		am_destroy(state);
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 	}
 }
diff --git a/builtin/clone.c b/builtin/clone.c
index 101c27a593f..13cfaa6488a 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -1217,7 +1217,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)
 	transport_disconnect(transport);
 
 	if (option_dissociate) {
-		close_all_packs();
+		close_all_packs(&the_repository->objects);
 		dissociate_from_references();
 	}
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 8ee998ea2ee..4d72efca781 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1478,7 +1478,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
 
 	string_list_clear(&list, 0);
 
-	close_all_packs();
+	close_all_packs(&the_repository->objects);
 
 	argv_array_pushl(&argv_gc_auto, "gc", "--auto", NULL);
 	if (verbosity < 0)
diff --git a/builtin/merge.c b/builtin/merge.c
index 92ba99a1a5e..0af0c53a632 100644
--- a/builtin/merge.c
+++ b/builtin/merge.c
@@ -411,7 +411,7 @@ static void finish(struct commit *head_commit,
 			 * We ignore errors in 'gc --auto', since the
 			 * user should see them.
 			 */
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			run_command_v_opt(argv_gc_auto, RUN_GIT_CMD);
 		}
 	}
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index 75e7f18acef..8e25aae54c8 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -2027,7 +2027,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)
 			proc.git_cmd = 1;
 			proc.argv = argv_gc_auto;
 
-			close_all_packs();
+			close_all_packs(&the_repository->objects);
 			if (!start_command(&proc)) {
 				if (use_sideband)
 					copy_to_sideband(proc.err, -1, NULL);
diff --git a/object.c b/object.c
index 367441efa94..a7c238339bd 100644
--- a/object.c
+++ b/object.c
@@ -4,6 +4,7 @@
 #include "tree.h"
 #include "commit.h"
 #include "tag.h"
+#include "packfile.h"
 
 static struct object **obj_hash;
 static int nr_objs, obj_hash_size;
@@ -468,8 +469,6 @@ void raw_object_store_clear(struct raw_object_store *o)
 	o->alt_odb_tail = NULL;
 
 	INIT_LIST_HEAD(&o->packed_git_mru);
-	/*
-	 * TODO: call close_all_packs once migrated to
-	 * take an object store argument
-	 */
+	close_all_packs(o);
+	o->packed_git = NULL;
 }
diff --git a/packfile.c b/packfile.c
index 59210deaaf7..65d9a4f6c61 100644
--- a/packfile.c
+++ b/packfile.c
@@ -310,11 +310,11 @@ static void close_pack(struct packed_git *p)
 	close_pack_index(p);
 }
 
-void close_all_packs(void)
+void close_all_packs(struct raw_object_store *o)
 {
 	struct packed_git *p;
 
-	for (p = the_repository->objects.packed_git; p; p = p->next)
+	for (p = o->packed_git; p; p = p->next)
 		if (p->do_not_close)
 			die("BUG: want to close pack marked 'do-not-close'");
 		else
diff --git a/packfile.h b/packfile.h
index a7fca598d67..6a2c57045ca 100644
--- a/packfile.h
+++ b/packfile.h
@@ -63,7 +63,7 @@ extern void close_pack_index(struct packed_git *);
 
 extern unsigned char *use_pack(struct packed_git *, struct pack_window **, off_t, unsigned long *);
 extern void close_pack_windows(struct packed_git *);
-extern void close_all_packs(void);
+extern void close_all_packs(struct raw_object_store *o);
 extern void unuse_pack(struct pack_window **);
 extern void clear_delta_base_cache(void);
 extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local);
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 07/27] pack: move prepare_packed_git_run_once to object store
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (5 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 06/27] object-store: close all packs upon clearing the " Stefan Beller
@ 2018-02-24  0:47         ` " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 08/27] pack: move approximate object count " Stefan Beller
                           ` (21 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Each repository's object store can be initialized independently, so
they must not share a run_once variable.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 8 +++++++-
 packfile.c     | 7 +++----
 2 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/object-store.h b/object-store.h
index 4f768465a14..bd6441e525f 100644
--- a/object-store.h
+++ b/object-store.h
@@ -89,9 +89,15 @@ struct raw_object_store {
 
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
+
+	/*
+	 * Whether packed_git has already been populated with this repository's
+	 * packs.
+	 */
+	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 65d9a4f6c61..3ee349ab1bd 100644
--- a/packfile.c
+++ b/packfile.c
@@ -883,12 +883,11 @@ static void prepare_packed_git_mru(void)
 		list_add_tail(&p->mru, &the_repository->objects.packed_git_mru);
 }
 
-static int prepare_packed_git_run_once = 0;
 void prepare_packed_git(void)
 {
 	struct alternate_object_database *alt;
 
-	if (prepare_packed_git_run_once)
+	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
 	prepare_alt_odb();
@@ -896,13 +895,13 @@ void prepare_packed_git(void)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
 	prepare_packed_git_mru();
-	prepare_packed_git_run_once = 1;
+	the_repository->objects.packed_git_initialized = 1;
 }
 
 void reprepare_packed_git(void)
 {
 	approximate_object_count_valid = 0;
-	prepare_packed_git_run_once = 0;
+	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 08/27] pack: move approximate object count to object store
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (6 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 07/27] pack: move prepare_packed_git_run_once to " Stefan Beller
@ 2018-02-24  0:47         ` " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 09/27] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
                           ` (20 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

The approximate_object_count() function maintains a rough count of
objects in a repository to estimate how long object name abbreviates
should be.  Object names are scoped to a repository and the
appropriate length may differ by repository, so the object count
should not be global.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 10 +++++++++-
 packfile.c     | 11 +++++------
 2 files changed, 14 insertions(+), 7 deletions(-)

diff --git a/object-store.h b/object-store.h
index bd6441e525f..31ca4d3f340 100644
--- a/object-store.h
+++ b/object-store.h
@@ -90,6 +90,14 @@ struct raw_object_store {
 	struct alternate_object_database *alt_odb_list;
 	struct alternate_object_database **alt_odb_tail;
 
+	/*
+	 * A fast, rough count of the number of objects in the repository.
+	 * These two fields are not meant for direct access. Use
+	 * approximate_object_count() instead.
+	 */
+	unsigned long approximate_object_count;
+	unsigned approximate_object_count_valid : 1;
+
 	/*
 	 * Whether packed_git has already been populated with this repository's
 	 * packs.
@@ -97,7 +105,7 @@ struct raw_object_store {
 	unsigned packed_git_initialized : 1;
 };
 
-#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0 }
+#define RAW_OBJECT_STORE_INIT(o) { NULL, NULL, LIST_HEAD_INIT(o.packed_git_mru), NULL, NULL, 0, 0, 0 }
 
 void raw_object_store_clear(struct raw_object_store *o);
 
diff --git a/packfile.c b/packfile.c
index 3ee349ab1bd..8d7b1dd5f9c 100644
--- a/packfile.c
+++ b/packfile.c
@@ -802,8 +802,6 @@ static void prepare_packed_git_one(char *objdir, int local)
 	strbuf_release(&path);
 }
 
-static int approximate_object_count_valid;
-
 /*
  * Give a fast, rough count of the number of objects in the repository. This
  * ignores loose objects completely. If you have a lot of them, then either
@@ -813,8 +811,8 @@ static int approximate_object_count_valid;
  */
 unsigned long approximate_object_count(void)
 {
-	static unsigned long count;
-	if (!approximate_object_count_valid) {
+	if (!the_repository->objects.approximate_object_count_valid) {
+		unsigned long count;
 		struct packed_git *p;
 
 		prepare_packed_git();
@@ -824,8 +822,9 @@ unsigned long approximate_object_count(void)
 				continue;
 			count += p->num_objects;
 		}
+		the_repository->objects.approximate_object_count = count;
 	}
-	return count;
+	return the_repository->objects.approximate_object_count;
 }
 
 static void *get_next_packed_git(const void *p)
@@ -900,7 +899,7 @@ void prepare_packed_git(void)
 
 void reprepare_packed_git(void)
 {
-	approximate_object_count_valid = 0;
+	the_repository->objects.approximate_object_count_valid = 0;
 	the_repository->objects.packed_git_initialized = 0;
 	prepare_packed_git();
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 09/27] sha1_file: add raw_object_store argument to alt_odb_usable
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (7 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 08/27] pack: move approximate object count " Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 10/27] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
                           ` (19 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

Add a raw_object_store to alt_odb_usable to be more specific about which
repository to act on. The choice of the repository is delegated to its
only caller link_alt_odb_entry.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index ac25146076e..8e6f213f9d9 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -344,7 +344,9 @@ static const char *alt_sha1_path(struct alternate_object_database *alt,
 /*
  * Return non-zero iff the path is usable as an alternate object database.
  */
-static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
+static int alt_odb_usable(struct raw_object_store *o,
+			  struct strbuf *path,
+			  const char *normalized_objdir)
 {
 	struct alternate_object_database *alt;
 
@@ -360,7 +362,7 @@ static int alt_odb_usable(struct strbuf *path, const char *normalized_objdir)
 	 * Prevent the common mistake of listing the same
 	 * thing twice, or object directory itself.
 	 */
-	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
+	for (alt = o->alt_odb_list; alt; alt = alt->next) {
 		if (!fspathcmp(path->buf, alt->path))
 			return 0;
 	}
@@ -412,7 +414,7 @@ static int link_alt_odb_entry(const char *entry, const char *relative_base,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 10/27] sha1_file: add repository argument to link_alt_odb_entry
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (8 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 09/27] sha1_file: add raw_object_store argument to alt_odb_usable Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 11/27] sha1_file: add repository argument to read_info_alternates Stefan Beller
                           ` (18 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow the link_alt_odb_entry caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

Since the implementation does not yet work with other repositories,
use a wrapper macro to enforce that the caller passes in
the_repository as the first argument. It would be more appealing to
use BUILD_ASSERT_OR_ZERO to enforce this, but that doesn't work
because it requires a compile-time constant and common compilers like
gcc 4.8.4 do not consider "r == the_repository" a compile-time
constant.

This and the following three patches add repository arguments to
link_alt_odb_entry, read_info_alternates, link_alt_odb_entries
and prepare_alt_odb. Three out of the four functions are found
in a recursive call chain, calling each other, and one of them
accesses the repositories `objectdir` (which was migrated; it
was an obvious choice) and `ignore_env` (which we need to keep in
the repository struct for clarify); hence we will pass through the
repository unlike just the object store object + the ignore_env flag.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8e6f213f9d9..e5fe0aa04ef 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -388,8 +388,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * terminating NUL.
  */
 static void read_info_alternates(const char * relative_base, int depth);
-static int link_alt_odb_entry(const char *entry, const char *relative_base,
-	int depth, const char *normalized_objdir)
+#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
+static int link_alt_odb_entry_the_repository(const char *entry,
+	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
 	struct strbuf pathbuf = STRBUF_INIT;
@@ -486,7 +487,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(entry.buf, relative_base, depth, objdirbuf.buf);
+		link_alt_odb_entry(the_repository, entry.buf,
+				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 11/27] sha1_file: add repository argument to read_info_alternates
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (9 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 10/27] sha1_file: add repository argument to link_alt_odb_entry Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 12/27] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
                           ` (17 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index e5fe0aa04ef..f93d3b95b54 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -387,7 +387,9 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-static void read_info_alternates(const char * relative_base, int depth);
+#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth);
 #define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
 static int link_alt_odb_entry_the_repository(const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
@@ -428,7 +430,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(pathbuf.buf, depth + 1);
+	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -494,7 +496,8 @@ static void link_alt_odb_entries(const char *alt, int sep,
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates(const char * relative_base, int depth)
+static void read_info_alternates_the_repository(const char *relative_base,
+						int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -678,7 +681,7 @@ void prepare_alt_odb(void)
 			&the_repository->objects.alt_odb_list;
 	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(get_object_directory(), 0);
+	read_info_alternates(the_repository, get_object_directory(), 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 12/27] sha1_file: add repository argument to link_alt_odb_entries
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (10 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 11/27] sha1_file: add repository argument to read_info_alternates Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 13/27] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
                           ` (16 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index f93d3b95b54..8fb56ca0362 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -465,8 +465,12 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-static void link_alt_odb_entries(const char *alt, int sep,
-				 const char *relative_base, int depth)
+#define link_alt_odb_entries(r, a, s, rb, d) \
+	link_alt_odb_entries_##r(a, s, rb, d)
+static void link_alt_odb_entries_the_repository(const char *alt,
+						int sep,
+						const char *relative_base,
+						int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -509,7 +513,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
@@ -563,7 +567,8 @@ void add_to_alternates_file(const char *reference)
 		if (commit_lock_file(&lock))
 			die_errno("unable to move new alternates file into place");
 		if (the_repository->objects.alt_odb_tail)
-			link_alt_odb_entries(reference, '\n', NULL, 0);
+			link_alt_odb_entries(the_repository, reference,
+					     '\n', NULL, 0);
 	}
 	free(alts);
 }
@@ -576,7 +581,8 @@ void add_to_alternates_memory(const char *reference)
 	 */
 	prepare_alt_odb();
 
-	link_alt_odb_entries(reference, '\n', NULL, 0);
+	link_alt_odb_entries(the_repository, reference,
+			     '\n', NULL, 0);
 }
 
 /*
@@ -679,7 +685,8 @@ void prepare_alt_odb(void)
 
 	the_repository->objects.alt_odb_tail =
 			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(alt, PATH_SEP, NULL, 0);
+	link_alt_odb_entries(the_repository, alt,
+			     PATH_SEP, NULL, 0);
 
 	read_info_alternates(the_repository, get_object_directory(), 0);
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 13/27] sha1_file: add repository argument to prepare_alt_odb
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (11 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 12/27] sha1_file: add repository argument to link_alt_odb_entries Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
                           ` (15 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

See previous patch for explanation.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 builtin/fsck.c |  2 +-
 object-store.h |  3 ++-
 packfile.c     |  2 +-
 sha1_file.c    | 13 +++++++------
 sha1_name.c    |  2 +-
 5 files changed, 12 insertions(+), 10 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index c3a339193c8..cdcf2fa926c 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -722,7 +722,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)
 	} else {
 		fsck_object_dir(get_object_directory());
 
-		prepare_alt_odb();
+		prepare_alt_odb(the_repository);
 		for (alt = the_repository->objects.alt_odb_list;
 		     alt; alt = alt->next)
 			fsck_object_dir(alt->path);
diff --git a/object-store.h b/object-store.h
index 31ca4d3f340..95f328482aa 100644
--- a/object-store.h
+++ b/object-store.h
@@ -20,7 +20,8 @@ struct alternate_object_database {
 
 	char path[FLEX_ARRAY];
 };
-void prepare_alt_odb(void);
+#define prepare_alt_odb(r) prepare_alt_odb_##r()
+void prepare_alt_odb_the_repository(void);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/packfile.c b/packfile.c
index 8d7b1dd5f9c..1e02fe3b0bb 100644
--- a/packfile.c
+++ b/packfile.c
@@ -889,7 +889,7 @@ void prepare_packed_git(void)
 	if (the_repository->objects.packed_git_initialized)
 		return;
 	prepare_packed_git_one(get_object_directory(), 1);
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next)
 		prepare_packed_git_one(alt->path, 0);
 	rearrange_packed_git();
diff --git a/sha1_file.c b/sha1_file.c
index 8fb56ca0362..381eaf8221d 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -23,6 +23,7 @@
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
 #include "repository.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "dir.h"
 #include "list.h"
@@ -579,7 +580,7 @@ void add_to_alternates_memory(const char *reference)
 	 * Make sure alternates are initialized, or else our entry may be
 	 * overwritten when they are.
 	 */
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 
 	link_alt_odb_entries(the_repository, reference,
 			     '\n', NULL, 0);
@@ -665,7 +666,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	struct alternate_object_database *ent;
 	int r = 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (ent = the_repository->objects.alt_odb_list; ent; ent = ent->next) {
 		r = fn(ent, cb);
 		if (r)
@@ -674,7 +675,7 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb(void)
+void prepare_alt_odb_the_repository(void)
 {
 	const char *alt;
 
@@ -728,7 +729,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 static int check_and_freshen_nonlocal(const unsigned char *sha1, int freshen)
 {
 	struct alternate_object_database *alt;
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		const char *path = alt_sha1_path(alt, sha1);
 		if (check_and_freshen_file(path, freshen))
@@ -887,7 +888,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	if (!lstat(*path, st))
 		return 0;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	errno = ENOENT;
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
@@ -918,7 +919,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 		return fd;
 	most_interesting_errno = errno;
 
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	for (alt = the_repository->objects.alt_odb_list; alt; alt = alt->next) {
 		*path = alt_sha1_path(alt, sha1);
 		fd = git_open(*path);
diff --git a/sha1_name.c b/sha1_name.c
index 016c883b5c7..627e93f1447 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -353,7 +353,7 @@ static int init_object_disambiguation(const char *name, int len,
 
 	ds->len = len;
 	ds->hex_pfx[len] = '\0';
-	prepare_alt_odb();
+	prepare_alt_odb(the_repository);
 	return 0;
 }
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (12 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 13/27] sha1_file: add repository argument to prepare_alt_odb Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 15/27] sha1_file: allow prepare_alt_odb " Stefan Beller
                           ` (14 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Actually this also allows read_info_alternates and link_alt_odb_entry to
handle arbitrary repositories, but link_alt_odb_entries is the most
interesting function in this set of functions, hence the commit subject.

These functions span a strongly connected component in the function
graph, i.e. the recursive call chain might look like

  -> link_alt_odb_entries
    -> link_alt_odb_entry
      -> read_info_alternates
        -> link_alt_odb_entries

That is why we need to convert all these functions at the same time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  4 ++++
 sha1_file.c    | 36 ++++++++++++++++--------------------
 2 files changed, 20 insertions(+), 20 deletions(-)

diff --git a/object-store.h b/object-store.h
index 95f328482aa..08cd48ade11 100644
--- a/object-store.h
+++ b/object-store.h
@@ -18,6 +18,10 @@ struct alternate_object_database {
 	char loose_objects_subdir_seen[256];
 	struct oid_array loose_objects_cache;
 
+	/*
+	 * Path to the alternative object store. If this is a relative path,
+	 * it is relative to the current working directory.
+	 */
 	char path[FLEX_ARRAY];
 };
 #define prepare_alt_odb(r) prepare_alt_odb_##r()
diff --git a/sha1_file.c b/sha1_file.c
index 381eaf8221d..ab42d430b8a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -388,11 +388,10 @@ static int alt_odb_usable(struct raw_object_store *o,
  * SHA1, an extra slash for the first level indirection, and the
  * terminating NUL.
  */
-#define read_info_alternates(r, rb, d) read_info_alternates_##r(rb, d)
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth);
-#define link_alt_odb_entry(r, e, rb, d, n) link_alt_odb_entry_##r(e, rb, d, n)
-static int link_alt_odb_entry_the_repository(const char *entry,
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth);
+static int link_alt_odb_entry(struct repository *r, const char *entry,
 	const char *relative_base, int depth, const char *normalized_objdir)
 {
 	struct alternate_object_database *ent;
@@ -418,7 +417,7 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	while (pathbuf.len && pathbuf.buf[pathbuf.len - 1] == '/')
 		strbuf_setlen(&pathbuf, pathbuf.len - 1);
 
-	if (!alt_odb_usable(&the_repository->objects, &pathbuf, normalized_objdir)) {
+	if (!alt_odb_usable(&r->objects, &pathbuf, normalized_objdir)) {
 		strbuf_release(&pathbuf);
 		return -1;
 	}
@@ -426,12 +425,12 @@ static int link_alt_odb_entry_the_repository(const char *entry,
 	ent = alloc_alt_odb(pathbuf.buf);
 
 	/* add the alternate entry */
-	*the_repository->objects.alt_odb_tail = ent;
-	the_repository->objects.alt_odb_tail = &(ent->next);
+	*r->objects.alt_odb_tail = ent;
+	r->objects.alt_odb_tail = &(ent->next);
 	ent->next = NULL;
 
 	/* recursively add alternates */
-	read_info_alternates(the_repository, pathbuf.buf, depth + 1);
+	read_info_alternates(r, pathbuf.buf, depth + 1);
 
 	strbuf_release(&pathbuf);
 	return 0;
@@ -466,12 +465,8 @@ static const char *parse_alt_odb_entry(const char *string,
 	return end;
 }
 
-#define link_alt_odb_entries(r, a, s, rb, d) \
-	link_alt_odb_entries_##r(a, s, rb, d)
-static void link_alt_odb_entries_the_repository(const char *alt,
-						int sep,
-						const char *relative_base,
-						int depth)
+static void link_alt_odb_entries(struct repository *r, const char *alt,
+				 int sep, const char *relative_base, int depth)
 {
 	struct strbuf objdirbuf = STRBUF_INIT;
 	struct strbuf entry = STRBUF_INIT;
@@ -485,7 +480,7 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		return;
 	}
 
-	strbuf_add_absolute_path(&objdirbuf, get_object_directory());
+	strbuf_add_absolute_path(&objdirbuf, r->objects.objectdir);
 	if (strbuf_normalize_path(&objdirbuf) < 0)
 		die("unable to normalize object directory: %s",
 		    objdirbuf.buf);
@@ -494,15 +489,16 @@ static void link_alt_odb_entries_the_repository(const char *alt,
 		alt = parse_alt_odb_entry(alt, sep, &entry);
 		if (!entry.len)
 			continue;
-		link_alt_odb_entry(the_repository, entry.buf,
+		link_alt_odb_entry(r, entry.buf,
 				   relative_base, depth, objdirbuf.buf);
 	}
 	strbuf_release(&entry);
 	strbuf_release(&objdirbuf);
 }
 
-static void read_info_alternates_the_repository(const char *relative_base,
-						int depth)
+static void read_info_alternates(struct repository *r,
+				 const char *relative_base,
+				 int depth)
 {
 	char *path;
 	struct strbuf buf = STRBUF_INIT;
@@ -514,7 +510,7 @@ static void read_info_alternates_the_repository(const char *relative_base,
 		return;
 	}
 
-	link_alt_odb_entries(the_repository, buf.buf, '\n', relative_base, depth);
+	link_alt_odb_entries(r, buf.buf, '\n', relative_base, depth);
 	strbuf_release(&buf);
 	free(path);
 }
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 15/27] sha1_file: allow prepare_alt_odb to handle arbitrary repositories
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (13 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 14/27] sha1_file: allow link_alt_odb_entries to handle arbitrary repositories Stefan Beller
@ 2018-02-24  0:47         ` " Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 16/27] sha1_file: add repository argument to sha1_file_name Stefan Beller
                           ` (13 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object-store.h |  3 +--
 sha1_file.c    | 12 +++++-------
 2 files changed, 6 insertions(+), 9 deletions(-)

diff --git a/object-store.h b/object-store.h
index 08cd48ade11..24b9a750aef 100644
--- a/object-store.h
+++ b/object-store.h
@@ -24,8 +24,7 @@ struct alternate_object_database {
 	 */
 	char path[FLEX_ARRAY];
 };
-#define prepare_alt_odb(r) prepare_alt_odb_##r()
-void prepare_alt_odb_the_repository(void);
+void prepare_alt_odb(struct repository *r);
 char *compute_alternate_path(const char *path, struct strbuf *err);
 typedef int alt_odb_fn(struct alternate_object_database *, void *);
 int foreach_alt_odb(alt_odb_fn, void*);
diff --git a/sha1_file.c b/sha1_file.c
index ab42d430b8a..03b9bbe8bb7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -671,21 +671,19 @@ int foreach_alt_odb(alt_odb_fn fn, void *cb)
 	return r;
 }
 
-void prepare_alt_odb_the_repository(void)
+void prepare_alt_odb(struct repository *r)
 {
 	const char *alt;
 
-	if (the_repository->objects.alt_odb_tail)
+	if (r->objects.alt_odb_tail)
 		return;
 
 	alt = getenv(ALTERNATE_DB_ENVIRONMENT);
 
-	the_repository->objects.alt_odb_tail =
-			&the_repository->objects.alt_odb_list;
-	link_alt_odb_entries(the_repository, alt,
-			     PATH_SEP, NULL, 0);
+	r->objects.alt_odb_tail = &r->objects.alt_odb_list;
+	link_alt_odb_entries(r, alt, PATH_SEP, NULL, 0);
 
-	read_info_alternates(the_repository, get_object_directory(), 0);
+	read_info_alternates(r, r->objects.objectdir, 0);
 }
 
 /* Returns 1 if we have successfully freshened the file, 0 otherwise. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 16/27] sha1_file: add repository argument to sha1_file_name
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (14 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 15/27] sha1_file: allow prepare_alt_odb " Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 17/27] sha1_file: add repository argument to stat_sha1_file Stefan Beller
                           ` (12 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow sha1_file_name callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        |  6 ------
 http-walker.c  |  3 ++-
 http.c         |  5 ++---
 object-store.h |  7 +++++++
 sha1_file.c    | 10 +++++-----
 5 files changed, 16 insertions(+), 15 deletions(-)

diff --git a/cache.h b/cache.h
index 63889acb591..94d85a5c692 100644
--- a/cache.h
+++ b/cache.h
@@ -936,12 +936,6 @@ extern void check_repository_format(void);
 #define DATA_CHANGED    0x0020
 #define TYPE_CHANGED    0x0040
 
-/*
- * Put in `buf` the name of the file in the local object database that
- * would be used to store a loose object with the specified sha1.
- */
-extern void sha1_file_name(struct strbuf *buf, const unsigned char *sha1);
-
 /*
  * Return an abbreviated sha1 unique within this repository's object database.
  * The result will be at least `len` characters long, and will be NUL
diff --git a/http-walker.c b/http-walker.c
index 07c2b1af826..2c33969123a 100644
--- a/http-walker.c
+++ b/http-walker.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "repository.h"
 #include "commit.h"
 #include "walker.h"
 #include "http.h"
@@ -545,7 +546,7 @@ static int fetch_object(struct walker *walker, unsigned char *sha1)
 		ret = error("File %s has bad hash", hex);
 	} else if (req->rename < 0) {
 		struct strbuf buf = STRBUF_INIT;
-		sha1_file_name(&buf, req->sha1);
+		sha1_file_name(the_repository, &buf, req->sha1);
 		ret = error("unable to write sha1 filename %s", buf.buf);
 		strbuf_release(&buf);
 	}
diff --git a/http.c b/http.c
index 31755023a4f..df9dbea59c5 100644
--- a/http.c
+++ b/http.c
@@ -2246,7 +2246,7 @@ struct http_object_request *new_http_object_request(const char *base_url,
 	hashcpy(freq->sha1, sha1);
 	freq->localfile = -1;
 
-	sha1_file_name(&filename, sha1);
+	sha1_file_name(the_repository, &filename, sha1);
 	snprintf(freq->tmpfile, sizeof(freq->tmpfile),
 		 "%s.temp", filename.buf);
 
@@ -2395,8 +2395,7 @@ int finish_http_object_request(struct http_object_request *freq)
 		unlink_or_warn(freq->tmpfile);
 		return -1;
 	}
-
-	sha1_file_name(&filename, freq->sha1);
+	sha1_file_name(the_repository, &filename, freq->sha1);
 	freq->rename = finalize_object_file(freq->tmpfile, filename.buf);
 	strbuf_release(&filename);
 
diff --git a/object-store.h b/object-store.h
index 24b9a750aef..c0d1292cfc2 100644
--- a/object-store.h
+++ b/object-store.h
@@ -113,4 +113,11 @@ struct raw_object_store {
 
 void raw_object_store_clear(struct raw_object_store *o);
 
+/*
+ * Put in `buf` the name of the file in the local object database that
+ * would be used to store a loose object with the specified sha1.
+ */
+#define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index 03b9bbe8bb7..8ef2f856035 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -321,7 +321,7 @@ static void fill_sha1_path(struct strbuf *buf, const unsigned char *sha1)
 	}
 }
 
-void sha1_file_name(struct strbuf *buf, const unsigned char *sha1)
+void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1)
 {
 	strbuf_addstr(buf, get_object_directory());
 	strbuf_addch(buf, '/');
@@ -715,7 +715,7 @@ static int check_and_freshen_local(const unsigned char *sha1, int freshen)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 
 	return check_and_freshen_file(buf.buf, freshen);
 }
@@ -876,7 +876,7 @@ static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	if (!lstat(*path, st))
@@ -905,7 +905,7 @@ static int open_sha1_file(const unsigned char *sha1, const char **path)
 	static struct strbuf buf = STRBUF_INIT;
 
 	strbuf_reset(&buf);
-	sha1_file_name(&buf, sha1);
+	sha1_file_name(the_repository, &buf, sha1);
 	*path = buf.buf;
 
 	fd = git_open(*path);
@@ -1591,7 +1591,7 @@ static int write_loose_object(const struct object_id *oid, char *hdr,
 	static struct strbuf filename = STRBUF_INIT;
 
 	strbuf_reset(&filename);
-	sha1_file_name(&filename, oid->hash);
+	sha1_file_name(the_repository, &filename, oid->hash);
 
 	fd = create_tmpfile(&tmp_file, filename.buf);
 	if (fd < 0) {
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 17/27] sha1_file: add repository argument to stat_sha1_file
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (15 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 16/27] sha1_file: add repository argument to sha1_file_name Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 18/27] sha1_file: add repository argument to open_sha1_file Stefan Beller
                           ` (11 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow the stat_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 8ef2f856035..b7e6e4a9bfc 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -869,8 +869,9 @@ int git_open_cloexec(const char *name, int flags)
  * Note that it may point to static storage and is only valid until another
  * call to sha1_file_name(), etc.
  */
-static int stat_sha1_file(const unsigned char *sha1, struct stat *st,
-			  const char **path)
+#define stat_sha1_file(r, s, st, p) stat_sha1_file_##r(s, st, p)
+static int stat_sha1_file_the_repository(const unsigned char *sha1,
+					 struct stat *st, const char **path)
 {
 	struct alternate_object_database *alt;
 	static struct strbuf buf = STRBUF_INIT;
@@ -1176,7 +1177,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 	if (!oi->typep && !oi->typename && !oi->sizep && !oi->contentp) {
 		const char *path;
 		struct stat st;
-		if (stat_sha1_file(sha1, &st, &path) < 0)
+		if (stat_sha1_file(the_repository, sha1, &st, &path) < 0)
 			return -1;
 		if (oi->disk_sizep)
 			*oi->disk_sizep = st.st_size;
@@ -1390,7 +1391,7 @@ void *read_sha1_file_extended(const unsigned char *sha1,
 		die("replacement %s not found for %s",
 		    sha1_to_hex(repl), sha1_to_hex(sha1));
 
-	if (!stat_sha1_file(repl, &st, &path))
+	if (!stat_sha1_file(the_repository, repl, &st, &path))
 		die("loose object %s (stored in %s) is corrupt",
 		    sha1_to_hex(repl), path);
 
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 18/27] sha1_file: add repository argument to open_sha1_file
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (16 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 17/27] sha1_file: add repository argument to stat_sha1_file Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 19/27] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
                           ` (10 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow the open_sha1_file caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index b7e6e4a9bfc..7e471e0cf4b 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -898,7 +898,9 @@ static int stat_sha1_file_the_repository(const unsigned char *sha1,
  * Like stat_sha1_file(), but actually open the object and return the
  * descriptor. See the caveats on the "path" parameter above.
  */
-static int open_sha1_file(const unsigned char *sha1, const char **path)
+#define open_sha1_file(r, s, p) open_sha1_file_##r(s, p)
+static int open_sha1_file_the_repository(const unsigned char *sha1,
+					 const char **path)
 {
 	int fd;
 	struct alternate_object_database *alt;
@@ -941,7 +943,7 @@ static void *map_sha1_file_1(const char *path,
 	if (path)
 		fd = git_open(path);
 	else
-		fd = open_sha1_file(sha1, &path);
+		fd = open_sha1_file(the_repository, sha1, &path);
 	map = NULL;
 	if (fd >= 0) {
 		struct stat st;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 19/27] sha1_file: add repository argument to map_sha1_file_1
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (17 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 18/27] sha1_file: add repository argument to open_sha1_file Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 20/27] sha1_file: add repository argument to map_sha1_file Stefan Beller
                           ` (9 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow the map_sha1_file_1 caller to be
more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 7e471e0cf4b..d4af8f56814 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -933,9 +933,10 @@ static int open_sha1_file_the_repository(const unsigned char *sha1,
  * Map the loose object at "path" if it is not NULL, or the path found by
  * searching for a loose object named "sha1".
  */
-static void *map_sha1_file_1(const char *path,
-			     const unsigned char *sha1,
-			     unsigned long *size)
+#define map_sha1_file_1(r, p, s, si) map_sha1_file_1_##r(p, s, si)
+static void *map_sha1_file_1_the_repository(const char *path,
+					    const unsigned char *sha1,
+					    unsigned long *size)
 {
 	void *map;
 	int fd;
@@ -964,7 +965,7 @@ static void *map_sha1_file_1(const char *path,
 
 void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
 {
-	return map_sha1_file_1(NULL, sha1, size);
+	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
 
 static int unpack_sha1_short_header(git_zstream *stream,
@@ -2200,7 +2201,7 @@ int read_loose_object(const char *path,
 
 	*contents = NULL;
 
-	map = map_sha1_file_1(path, NULL, &mapsize);
+	map = map_sha1_file_1(the_repository, path, NULL, &mapsize);
 	if (!map) {
 		error_errno("unable to mmap %s", path);
 		goto out;
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 20/27] sha1_file: add repository argument to map_sha1_file
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (18 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 19/27] sha1_file: add repository argument to map_sha1_file_1 Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 21/27] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
                           ` (8 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow map_sha1_file callers to be more
specific about which repository to handle. This is a small mechanical
change; it doesn't change the implementation to handle repositories
other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

While at it, move the declaration to object-store.h, where it should
be easier to find.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 cache.h        | 1 -
 object-store.h | 3 +++
 sha1_file.c    | 4 ++--
 streaming.c    | 5 ++++-
 4 files changed, 9 insertions(+), 4 deletions(-)

diff --git a/cache.h b/cache.h
index 94d85a5c692..57173991839 100644
--- a/cache.h
+++ b/cache.h
@@ -1226,7 +1226,6 @@ extern int force_object_loose(const struct object_id *oid, time_t mtime);
 
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
-extern void *map_sha1_file(const unsigned char *sha1, unsigned long *size);
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
 extern int parse_sha1_header(const char *hdr, unsigned long *sizep);
 
diff --git a/object-store.h b/object-store.h
index c0d1292cfc2..dbd589e0083 100644
--- a/object-store.h
+++ b/object-store.h
@@ -120,4 +120,7 @@ void raw_object_store_clear(struct raw_object_store *o);
 #define sha1_file_name(r, b, s) sha1_file_name_##r(b, s)
 void sha1_file_name_the_repository(struct strbuf *buf, const unsigned char *sha1);
 
+#define map_sha1_file(r, s, sz) map_sha1_file_##r(s, sz)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size);
+
 #endif /* OBJECT_STORE_H */
diff --git a/sha1_file.c b/sha1_file.c
index d4af8f56814..6ec43e745d7 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -963,7 +963,7 @@ static void *map_sha1_file_1_the_repository(const char *path,
 	return map;
 }
 
-void *map_sha1_file(const unsigned char *sha1, unsigned long *size)
+void *map_sha1_file_the_repository(const unsigned char *sha1, unsigned long *size)
 {
 	return map_sha1_file_1(the_repository, NULL, sha1, size);
 }
@@ -1187,7 +1187,7 @@ static int sha1_loose_object_info(const unsigned char *sha1,
 		return 0;
 	}
 
-	map = map_sha1_file(sha1, &mapsize);
+	map = map_sha1_file(the_repository, sha1, &mapsize);
 	if (!map)
 		return -1;
 
diff --git a/streaming.c b/streaming.c
index 5892b50bd89..22d27df55eb 100644
--- a/streaming.c
+++ b/streaming.c
@@ -3,6 +3,8 @@
  */
 #include "cache.h"
 #include "streaming.h"
+#include "repository.h"
+#include "object-store.h"
 #include "packfile.h"
 
 enum input_source {
@@ -335,7 +337,8 @@ static struct stream_vtbl loose_vtbl = {
 
 static open_method_decl(loose)
 {
-	st->u.loose.mapped = map_sha1_file(sha1, &st->u.loose.mapsize);
+	st->u.loose.mapped = map_sha1_file(the_repository,
+					   sha1, &st->u.loose.mapsize);
 	if (!st->u.loose.mapped)
 		return -1;
 	if ((unpack_sha1_header(&st->z,
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 21/27] sha1_file: add repository argument to sha1_loose_object_info
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (19 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 20/27] sha1_file: add repository argument to map_sha1_file Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories Stefan Beller
                           ` (7 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Add a repository argument to allow the sha1_loose_object_info caller
to be more specific about which repository to act on. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 sha1_file.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/sha1_file.c b/sha1_file.c
index 6ec43e745d7..831e2da3572 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1154,9 +1154,10 @@ int parse_sha1_header(const char *hdr, unsigned long *sizep)
 	return parse_sha1_header_extended(hdr, &oi, 0);
 }
 
-static int sha1_loose_object_info(const unsigned char *sha1,
-				  struct object_info *oi,
-				  int flags)
+#define sha1_loose_object_info(r, s, o, f) sha1_loose_object_info_##r(s, o, f)
+static int sha1_loose_object_info_the_repository(const unsigned char *sha1,
+						 struct object_info *oi,
+						 int flags)
 {
 	int status = 0;
 	unsigned long mapsize;
@@ -1275,7 +1276,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,
 			break;
 
 		/* Most likely it's a loose object. */
-		if (!sha1_loose_object_info(real, oi, flags))
+		if (!sha1_loose_object_info(the_repository, real, oi, flags))
 			return 0;
 
 		/* Not a loose object; someone else may have just packed it. */
-- 
2.16.1.291.g4437f3f132-goog


^ permalink raw reply	[flat|nested] 239+ messages in thread

* [PATCHv4 22/27] sha1_file: allow sha1_file_name to handle arbitrary repositories
  2018-02-24  0:47       ` [PATCHv4 " Stefan Beller
                           ` (20 preceding siblings ...)
  2018-02-24  0:47         ` [PATCHv4 21/27] sha1_file: add repository argument to sha1_loose_object_info Stefan Beller
@ 2018-02-24  0:47         ` Stefan Beller
  2018-02-24  0:47         ` [PATCHv4 23/27] sha1_file: allow stat_sha1_file " Stefan Beller
                           ` (6 subsequent siblings)
  28 siblings, 0 replies; 239+ messages in thread
From: Stefan Beller @ 2018-02-24  0:47 UTC (permalink / raw)
  To: sbeller
  Cc: git, gitster, jonathantanmy, pclouds, sunshine, bmwill, Jonathan Nieder

Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
 object-store.h | 3 +--
 sha1_file.c