git@vger.kernel.org mailing list mirror (one of many)
 help / Atom feed
Search results ordered by [date|relevance]  view[summary|nested|Atom feed]
						download: 
* Re: [PATCH 5/6] submodule: fixup nested submodules after moving the submodule
      [irrelevant]   ` <20180328172449.27012-6-sbeller@google.com>
@ 2018-03-28 17:46     ` Jonathan Tan
      [irrelevant]     ` <20180328173543.GA159395@google.com>
  1 sibling, 0 replies; 200+ results
From: Jonathan Tan @ 2018-03-28 17:46 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, bmwill, git, hvoigt, seanwbehan

On Wed, 28 Mar 2018 10:24:48 -0700
Stefan Beller <sbeller@google.com> wrote:

> +static void connect_wt_gitdir_in_nested(const char *sub_worktree,
> +					const char *sub_gitdir)
> +{
> +	int i;
> +	struct repository subrepo;
> +	struct strbuf sub_wt = STRBUF_INIT;
> +	struct strbuf sub_gd = STRBUF_INIT;
> +
> +	const struct submodule *sub;
> +
> +	if (repo_init(&subrepo, sub_gitdir, sub_worktree))
> +		return;

If repo_init() fails, is it because the working tree doesn't exist on
disk, so we don't need to perform any connections on submodules? I think
a comment should be added to describe this.

> +
> +	if (repo_read_index(&subrepo) < 0)
> +		die("index file corrupt in repo %s", subrepo.gitdir);
> +
> +	for (i = 0; i < subrepo.index->cache_nr; i++) {
> +		const struct cache_entry *ce = subrepo.index->cache[i];
> +
> +		if (!S_ISGITLINK(ce->ce_mode))
> +			continue;
> +
> +		while (i + 1 < subrepo.index->cache_nr &&
> +		       !strcmp(ce->name, subrepo.index->cache[i + 1]->name))
> +			/*
> +			 * Skip entries with the same name in different stages
> +			 * to make sure an entry is returned only once.
> +			 */
> +			i++;
> +
> +		sub = submodule_from_path(&subrepo, &null_oid, ce->name);
> +		if (!sub)
> +			/* submodule not checked out? */
> +			continue;
> +
> +		if (is_submodule_active(&subrepo, ce->name)) {

Optional: This could be combined with the previous "if" block, so that
the following lines don't need to be indented.

> +			strbuf_addf(&sub_wt, "%s/%s", sub_worktree, sub->path);
> +			strbuf_addf(&sub_gd, "%s/modules/%s", sub_gitdir, sub->name);
> +
> +			connect_work_tree_and_git_dir(sub_wt.buf, sub_gd.buf, 0);
> +			connect_wt_gitdir_in_nested(sub_wt.buf, sub_gd.buf);

What's the difference between having this call to
connect_wt_gitdir_in_nested() and just passing 1 instead of 0 to
connect_work_tree_and_git_dir()? I see that the latter uses absolute
paths, but that would seem to have the same effect. (If not, I think a
comment is warranted.)

> +
> +			strbuf_reset(&sub_wt);
> +			strbuf_reset(&sub_gd);

I think we normally write the resets before the strbuf_addf(), so that
it's clearer that sub_wt and sub_gd are meant to start from scratch in
every iteration.

Overall, this version of the patch is clearer - thanks.

^ permalink raw reply	[relevance 8%]

* Re: [PATCHv2 0/6] Moving submodules with nested submodules
      [irrelevant] ` <20180328172449.27012-1-sbeller@google.com>
      [irrelevant]   ` <20180328172449.27012-6-sbeller@google.com>
@ 2018-03-28 17:54   ` Jonathan Tan
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
  1 sibling, 1 reply; 200+ results
From: Jonathan Tan @ 2018-03-28 17:54 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, bmwill, git, hvoigt, seanwbehan

On Wed, 28 Mar 2018 10:24:43 -0700
Stefan Beller <sbeller@google.com> wrote:

> * picked up Jonathans patch and added it as a nice finish of the series.
>   I did not see the need or aesthetic desire to put that patch earlier
>   in the series.

Thanks for picking up my patch. The aesthetic desire is to avoid what's
currently happening in PATCH 2/6 version 2, where we still have the
potentially confusing submodule_free() -> submodule_free(repo)
conversion, which is also later redone in PATCH 6/6 version 2
(submodule_free(repo) -> submodule_free(the_repository)). But I'll leave
the final decision to you.

Other than that, besides PATCH 5/6 (which I have already commented on),
everything looks good.

^ permalink raw reply	[relevance 5%]

* [PATCH] submodule: check for NULL return of get_submodule_ref_store()
      [irrelevant] <9c3c0161-f894-3368-ece2-500d0bb6f475@web.de>
@ 2018-03-28 18:38 ` Stefan Beller
  2018-03-28 18:57   ` Eric Sunshine
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-03-28 18:38 UTC (permalink / raw)
  To: l.s.r; +Cc: git, gitster, jeremy, pc44800, Stefan Beller

From: René Scharfe <l.s.r@web.de>

If we can't find a ref store for a submodule then assume it the latter
is not initialized (or was removed).  Print a status line accordingly
instead of causing a segmentation fault by passing NULL as the first
parameter of refs_head_ref().

Reported-by: Jeremy Feusi <jeremy@feusi.co>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

I added a test for you.

Thanks,
Stefan

 builtin/submodule--helper.c |  8 ++++++--
 t/t7400-submodule-basic.sh  | 12 ++++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index ee020d4749..ae3014ac5a 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -654,9 +654,13 @@ static void status_submodule(const char *path, const struct object_id *ce_oid,
 			     displaypath);
 	} else if (!(flags & OPT_CACHED)) {
 		struct object_id oid;
+		struct ref_store *refs = get_submodule_ref_store(path);
 
-		if (refs_head_ref(get_submodule_ref_store(path),
-				  handle_submodule_head_ref, &oid))
+		if (!refs) {
+			print_status(flags, '-', path, ce_oid, displaypath);
+			goto cleanup;
+		}
+		if (refs_head_ref(refs, handle_submodule_head_ref, &oid))
 			die(_("could not resolve HEAD ref inside the "
 			      "submodule '%s'"), path);
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index a39e69a3eb..d8aee51603 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -821,6 +821,18 @@ test_expect_success 'moving the superproject does not break submodules' '
 	)
 '
 
+test_expect_success 'moving the submodule does not break the superproject' '
+	(
+		cd addtest2 &&
+
+		mv repo repo.bak &&
+		git submodule status >actual &&
+		grep -e "^-" -e repo actual &&
+
+		mv repo.bak repo
+	)
+'
+
 test_expect_success 'submodule add --name allows to replace a submodule with another at the same path' '
 	(
 		cd addtest2 &&
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 25%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 18:38 ` [PATCH] submodule: check for NULL return of get_submodule_ref_store() Stefan Beller
@ 2018-03-28 18:57   ` Eric Sunshine
  2018-03-28 20:08     ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Eric Sunshine @ 2018-03-28 18:57 UTC (permalink / raw)
  To: Stefan Beller
  Cc: René Scharfe, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

On Wed, Mar 28, 2018 at 2:38 PM, Stefan Beller <sbeller@google.com> wrote:
> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
> @@ -821,6 +821,18 @@ test_expect_success 'moving the superproject does not break submodules' '
> +test_expect_success 'moving the submodule does not break the superproject' '
> +       (
> +               cd addtest2 &&
> +
> +               mv repo repo.bak &&
> +               git submodule status >actual &&
> +               grep -e "^-" -e repo actual &&
> +
> +               mv repo.bak repo

Should this "move back" be encapsulated in a test_when_finished?

> +       )
> +'

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 5/6] submodule: fixup nested submodules after moving the submodule
      [irrelevant]     ` <20180328173543.GA159395@google.com>
@ 2018-03-28 19:08       ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 19:08 UTC (permalink / raw)
  To: Brandon Williams
  Cc: Junio C Hamano, git, Heiko Voigt, Jonathan Tan, seanwbehan

>> +     if (repo_init(&subrepo, sub_gitdir, sub_worktree))
>> +             return;
>
> Note that in Duy's object-store series he made this function private
> (IIRC) so this will result in some clash of the two series.
>

Yes, that is the case.
I wonder if I'd rather revert to v1 where we only use the
submodule repo init, or if we revert b2f0eceecf (repository:
initialize the_repository in main(), 2018-03-03) partially to
have repo_init available.

I would think the approach with submodule init is a bit cleaner
though has some more lines of code, using just repo_init
seems easier, but we really have no use case for a separate
repo_init unless they are submodules. And here we ought to
check for the repo being a submodule.

Not yet sure which path to take.

^ permalink raw reply	[relevance 9%]

* Re: git submodule deinit resulting in BUG: builtin/submodule--helper.c:1045: module_list_compute should not choke on empty pathspec
      [irrelevant] ` <CAGZ79kYGY5bjh0WPQh7xkXQxLkB9EQ-OcJhVuGE8YUnwmvk2Fg@mail.gmail.com>
@ 2018-03-28 19:37   ` Peter Oberndorfer
  2018-03-28 21:19     ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Peter Oberndorfer @ 2018-03-28 19:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Prathamesh Chavan

On 2018-03-28 00:56, Stefan Beller wrote:
> On Tue, Mar 27, 2018 at 12:55 PM Peter Oberndorfer <kumbayo84@arcor.de>
> wrote:

Hi,

as expected your patch fixed the BUG output.
Thanks!

>> 2) Should "git submodule deinit" work on submodules that were removed by
> upstream already?
> 
> To answer the question "Is this a submodule that upstream removed
> (recently)?"
> we'd have to put in some effort, essentially checking if that was ever a
> submodule
> (and not a directory or file).
> 

Hmm, yeah looks a bit more complicated than I initially imagined
since submodules can have a name that's different from their path.
And after the rebase, the name <-> path mapping via .gitmodules is not available anymore.

Naively I think it could work the following way:
* Either iterate over all submodules in .git/modules/ and check their config
  has a worktree = "../../path" that resolves to the submodule path we want to remove.
* Or check the "gitlink:" path in submodule/.git if it points to our .git/modules/
Then if .git/config contains a [submodule "name"] entry
we should have a pretty good idea if this folder contains a stale submodule.

> When using "git pull --recurse-submodules" the submodule ought to be removed
> automatically.
> 
> When doing a fetch && merge manually, we may want to teach merge to remove
> a submodule that we have locally upon merge, too.
> 

Yeah that would be nice :-)
In my case I updated the repository via a rebase, so that would also have to be covered.

> I view the git-submodule command as a bare bones plumbing helper, that we'd
> want
> to deprecate eventually as all other higher level commands will know how to
> deal
> with submodules.
> 
> So I think we do not want to teach "git submodule deinit" to remove dormant
> repositories, that were submodules removed by upstream already.
> 

My gut feeling makes me expect the following:
* It would be nice if such stale submodules showed up in "git submodule status" or "git status"
  Now "git submodule" shows nothing related to this stale submodule
  Now "git status" shows  Untracked files: src/rt which is a bit confusing as the actual submodule is in src/rt/hoedown
  Now "Git gui" shows src/rt/hoedown as untracked git repository
* There should be an easy(and safe) way for the user to deinit such a submodule
  if if the automatic submodule updating during a merge/rebase was not enabled or somehow failed.
(Minus the problem of somebody having to actually do the work...)

>> ~/src/rust/rust$ git submodule status
> ...
>>   b87873eaceb75cf9342d5273f01ba2c020f61ca8 src/tools/lld ((null))
> 
>> -> strangely I get (null) for the current branch/commit in some
> submodules?
> 
> This sounds like (3). Looking into that.

Sorry, what do you mean by (3)?

Thanks,
Greetings Peter

^ permalink raw reply	[relevance 8%]

* [ANNOUNCE] Git v2.17.0-rc2
@ 2018-03-28 19:56 Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-03-28 19:56 UTC (permalink / raw)
  To: git; +Cc: Linux Kernel, git-packagers

A release candidate Git v2.17.0-rc2 is now available for testing
at the usual places.  It is comprised of 499 non-merge commits
since v2.16.0, contributed by 62 people, 19 of which are new faces.

I am hoping that we can have the final version tagged at the end of
coming weekend, before I fly out to Tokyo.  I expect to be offline
most of the next week after the final is tagged.

The tarballs are found at:

    https://www.kernel.org/pub/software/scm/git/testing/

The following public repositories all have a copy of the
'v2.17.0-rc2' tag and the 'master' branch that the tag points at:

  url = https://kernel.googlesource.com/pub/scm/git/git
  url = git://repo.or.cz/alt-git.git
  url = https://github.com/gitster/git

New contributors whose contributions weren't in v2.16.0 are as follows.
Welcome to the Git development community!

  Adam Borowski, Alban Gruin, Andreas G. Schacker, Bernhard
  M. Wiedemann, Christian Ludwig, Gargi Sharma, Genki Sky,
  Gregory Herrero, Jon Simons, Juan F. Codagnone, Kim Gybels,
  Lucas Werkmeister, Mathias Rav, Michele Locati, Motoki Seki,
  Stefan Moch, Stephen R Guglielmo, Tatyana Krasnukha, and Thomas
  Levesque.

Returning contributors who helped this release are as follows.
Thanks for your continued support.

  Ævar Arnfjörð Bjarmason, Alexander Shopov, Alex Bennée, Ben
  Peart, Brandon Williams, brian m. carlson, Christian Couder,
  Daniel Knittl-Frank, David Pursehouse, Derrick Stolee, Elijah
  Newren, Eric Sunshine, Eric Wong, Jason Merrill, Jeff Hostetler,
  Jeff King, Johannes Schindelin, Jonathan Nieder, Jonathan Tan,
  Junio C Hamano, Kaartic Sivaraam, Mårten Kongstad, Martin
  Ågren, Matthieu Moy, Michael Haggerty, Nathan Payre, Nguyễn
  Thái Ngọc Duy, Nicolas Morey-Chaisemartin, Olga Telezhnaya,
  Patryk Obara, Phillip Wood, Prathamesh Chavan, Ramsay Jones,
  Randall S. Becker, Rasmus Villemoes, René Scharfe, Robert
  P. J. Day, Stefan Beller, SZEDER Gábor, Thomas Gummerer,
  Todd Zullinger, Torsten Bögershausen, and Yasushi SHOJI.

----------------------------------------------------------------

Git 2.17 Release Notes (draft)
==============================

Updates since v2.16
-------------------

UI, Workflows & Features

 * "diff" family of commands learned "--find-object=<object-id>" option
   to limit the findings to changes that involve the named object.

 * "git format-patch" learned to give 72-cols to diffstat, which is
   consistent with other line length limits the subcommand uses for
   its output meant for e-mails.

 * The log from "git daemon" can be redirected with a new option; one
   relevant use case is to send the log to standard error (instead of
   syslog) when running it from inetd.

 * "git rebase" learned to take "--allow-empty-message" option.

 * "git am" has learned the "--quit" option, in addition to the
   existing "--abort" option; having the pair mirrors a few other
   commands like "rebase" and "cherry-pick".

 * "git worktree add" learned to run the post-checkout hook, just like
   "git clone" runs it upon the initial checkout.

 * "git tag" learned an explicit "--edit" option that allows the
   message given via "-m" and "-F" to be further edited.

 * "git fetch --prune-tags" may be used as a handy short-hand for
   getting rid of stale tags that are locally held.

 * The new "--show-current-patch" option gives an end-user facing way
   to get the diff being applied when "git rebase" (and "git am")
   stops with a conflict.

 * "git add -p" used to offer "/" (look for a matching hunk) as a
   choice, even there was only one hunk, which has been corrected.
   Also the single-key help is now given only for keys that are
   enabled (e.g. help for '/' won't be shown when there is only one
   hunk).

 * Since Git 1.7.9, "git merge" defaulted to --no-ff (i.e. even when
   the side branch being merged is a descendant of the current commit,
   create a merge commit instead of fast-forwarding) when merging a
   tag object.  This was appropriate default for integrators who pull
   signed tags from their downstream contributors, but caused an
   unnecessary merges when used by downstream contributors who
   habitually "catch up" their topic branches with tagged releases
   from the upstream.  Update "git merge" to default to --no-ff only
   when merging a tag object that does *not* sit at its usual place in
   refs/tags/ hierarchy, and allow fast-forwarding otherwise, to
   mitigate the problem.

 * "git status" can spend a lot of cycles to compute the relation
   between the current branch and its upstream, which can now be
   disabled with "--no-ahead-behind" option.

 * "git diff" and friends learned funcname patterns for Go language
   source files.

 * "git send-email" learned "--reply-to=<address>" option.

 * Funcname pattern used for C# now recognizes "async" keyword.

 * In a way similar to how "git tag" learned to honor the pager
   setting only in the list mode, "git config" learned to ignore the
   pager setting when it is used for setting values (i.e. when the
   purpose of the operation is not to "show").


Performance, Internal Implementation, Development Support etc.

 * More perf tests for threaded grep

 * "perf" test output can be sent to codespeed server.

 * The build procedure for perl/ part has been greatly simplified by
   weaning ourselves off of MakeMaker.

 * Perl 5.8 or greater has been required since Git 1.7.4 released in
   2010, but we continued to assume some core modules may not exist and
   used a conditional "eval { require <<module>> }"; we no longer do
   this.  Some platforms (Fedora/RedHat/CentOS, for example) ship Perl
   without all core modules by default (e.g. Digest::MD5, File::Temp,
   File::Spec, Net::Domain, Net::SMTP).  Users on such platforms may
   need to install these additional modules.

 * As a convenience, we install copies of Perl modules we require which
   are not part of the core Perl distribution (e.g. Error and
   Mail::Address).  Users and packagers whose operating system provides
   these modules can set NO_PERL_CPAN_FALLBACKS to avoid installing the
   bundled modules.

 * In preparation for implementing narrow/partial clone, the machinery
   for checking object connectivity used by gc and fsck has been
   taught that a missing object is OK when it is referenced by a
   packfile specially marked as coming from trusted repository that
   promises to make them available on-demand and lazily.

 * The machinery to clone & fetch, which in turn involves packing and
   unpacking objects, has been told how to omit certain objects using
   the filtering mechanism introduced by another topic.  It now knows
   to mark the resulting pack as a promisor pack to tolerate missing
   objects, laying foundation for "narrow" clones.

 * The first step to getting rid of mru API and using the
   doubly-linked list API directly instead.

 * Retire mru API as it does not give enough abstraction over
   underlying list API to be worth it.

 * Rewrite two more "git submodule" subcommands in C.

 * The tracing machinery learned to report tweaking of environment
   variables as well.

 * Update Coccinelle rules to catch and optimize strbuf_addf(&buf, "%s", str)

 * Prevent "clang-format" from breaking line after function return type.

 * The sequencer infrastructure is shared across "git cherry-pick",
   "git rebase -i", etc., and has always spawned "git commit" when it
   needs to create a commit.  It has been taught to do so internally,
   when able, by reusing the codepath "git commit" itself uses, which
   gives performance boost for a few tens of percents in some sample
   scenarios.

 * Push the submodule version of collision-detecting SHA-1 hash
   implementation a bit harder on builders.

 * Avoid mmapping small files while using packed refs (especially ones
   with zero size, which would cause later munmap() to fail).

 * Conversion from uchar[20] to struct object_id continues.

 * More tests for wildmatch functions.

 * The code to binary search starting from a fan-out table (which is
   how the packfile is indexed with object names) has been refactored
   into a reusable helper.

 * We now avoid using identifiers that clash with C++ keywords.  Even
   though it is not a goal to compile Git with C++ compilers, changes
   like this help use of code analysis tools that targets C++ on our
   codebase.

 * The executable is now built in 'script' phase in Travis CI integration,
   to follow the established practice, rather than during 'before_script'
   phase.  This allows the CI categorize the failures better ('failed'
   is project's fault, 'errored' is build environment's).
   (merge 3c93b82920 sg/travis-build-during-script-phase later to maint).

 * Writing out the index file when the only thing that changed in it
   is the untracked cache information is often wasteful, and this has
   been optimized out.

 * Various pieces of Perl code we have have been cleaned up.

 * Internal API clean-up to allow write_locked_index() optionally skip
   writing the in-core index when it is not modified.


Also contains various documentation updates and code clean-ups.


Fixes since v2.16
-----------------

 * An old regression in "git describe --all $annotated_tag^0" has been
   fixed.

 * "git status" after moving a path in the working tree (hence making
   it appear "removed") and then adding with the -N option (hence
   making that appear "added") detected it as a rename, but did not
   report the  old and new pathnames correctly.

 * "git svn dcommit" did not take into account the fact that a
   svn+ssh:// URL with a username@ (typically used for pushing) refers
   to the same SVN repository without the username@ and failed when
   svn.pushmergeinfo option is set.

 * API clean-up around revision traversal.

 * "git merge -Xours/-Xtheirs" learned to use our/their version when
   resolving a conflicting updates to a symbolic link.

 * "git clone $there $here" is allowed even when here directory exists
   as long as it is an empty directory, but the command incorrectly
   removed it upon a failure of the operation.

 * "git commit --fixup" did not allow "-m<message>" option to be used
   at the same time; allow it to annotate resulting commit with more
   text.

 * When resetting the working tree files recursively, the working tree
   of submodules are now also reset to match.

 * "git stash -- <pathspec>" incorrectly blew away untracked files in
   the directory that matched the pathspec, which has been corrected.

 * Instead of maintaining home-grown email address parsing code, ship
   a copy of reasonably recent Mail::Address to be used as a fallback
   in 'git send-email' when the platform lacks it.
   (merge d60be8acab mm/send-email-fallback-to-local-mail-address later to maint).

 * "git add -p" was taught to ignore local changes to submodules as
   they do not interfere with the partial addition of regular changes
   anyway.

 * Avoid showing a warning message in the middle of a line of "git
   diff" output.
   (merge 4e056c989f nd/diff-flush-before-warning later to maint).

 * The http tracing code, often used to debug connection issues,
   learned to redact potentially sensitive information from its output
   so that it can be more safely sharable.
   (merge 8ba18e6fa4 jt/http-redact-cookies later to maint).

 * Crash fix for a corner case where an error codepath tried to unlock
   what it did not acquire lock on.
   (merge 81fcb698e0 mr/packed-ref-store-fix later to maint).

 * The split-index mode had a few corner case bugs fixed.
   (merge ae59a4e44f tg/split-index-fixes later to maint).

 * Assorted fixes to "git daemon".
   (merge ed15e58efe jk/daemon-fixes later to maint).

 * Completion of "git merge -s<strategy>" (in contrib/) did not work
   well in non-C locale.
   (merge 7cc763aaa3 nd/list-merge-strategy later to maint).

 * Workaround for segfault with more recent versions of SVN.
   (merge 7f6f75e97a ew/svn-branch-segfault-fix later to maint).

 * Plug recently introduced leaks in fsck.
   (merge ba3a08ca0e jt/fsck-code-cleanup later to maint).

 * "git pull --rebase" did not pass verbosity setting down when
   recursing into a submodule.
   (merge a56771a668 sb/pull-rebase-submodule later to maint).

 * The way "git reset --hard" reports the commit the updated HEAD
   points at is made consistent with the way how the commit title is
   generated by the other parts of the system.  This matters when the
   title is spread across physically multiple lines.
   (merge 1cf823fb68 tg/reset-hard-show-head-with-pretty later to maint).

 * Test fixes.
   (merge 63b1a175ee sg/test-i18ngrep later to maint).

 * Some bugs around "untracked cache" feature have been fixed.  This
   will notice corrupt data in the untracked cache left by old and
   buggy code and issue a warning---the index can be fixed by clearing
   the untracked cache from it.
   (merge 0cacebf099 nd/fix-untracked-cache-invalidation later to maint).
   (merge 7bf0be7501 ab/untracked-cache-invalidation-docs later to maint).

 * "git blame HEAD COPYING" in a bare repository failed to run, while
   "git blame HEAD -- COPYING" run just fine.  This has been corrected.

 * "git add" files in the same directory, but spelling the directory
   path in different cases on case insensitive filesystem, corrupted
   the name hash data structure and led to unexpected results.  This
   has been corrected.
   (merge c95525e90d bp/name-hash-dirname-fix later to maint).

 * "git rebase -p" mangled log messages of a merge commit, which is
   now fixed.
   (merge ed5144d7eb js/fix-merge-arg-quoting-in-rebase-p later to maint).

 * Some low level protocol codepath could crash when they get an
   unexpected flush packet, which is now fixed.
   (merge bb1356dc64 js/packet-read-line-check-null later to maint).

 * "git check-ignore" with multiple paths got confused when one is a
   file and the other is a directory, which has been fixed.
   (merge d60771e930 rs/check-ignore-multi later to maint).

 * "git describe $garbage" stopped giving any errors when the garbage
   happens to be a string with 40 hexadecimal letters.
   (merge a8e7a2bf0f sb/describe-blob later to maint).

 * Code to unquote single-quoted string (used in the parser for
   configuration files, etc.) did not diagnose bogus input correctly
   and produced bogus results instead.
   (merge ddbbf8eb25 jk/sq-dequote-on-bogus-input later to maint).

 * Many places in "git apply" knew that "/dev/null" that signals
   "there is no such file on this side of the diff" can be followed by
   whitespace and garbage when parsing a patch, except for one, which
   made an otherwise valid patch (e.g. ones from subversion) rejected.
   (merge e454ad4bec tk/apply-dev-null-verify-name-fix later to maint).

 * We no longer create any *.spec file, so "make clean" should not
   remove it.
   (merge 4321bdcabb tz/do-not-clean-spec-file later to maint).

 * "git push" over http transport did not unquote the push-options
   correctly.
   (merge 90dce21eb0 jk/push-options-via-transport-fix later to maint).

 * "git send-email" learned to complain when the batch-size option is
   not defined when the relogin-delay option is, since these two are
   mutually required.
   (merge 9caa70697b xz/send-email-batch-size later to maint).

 * Y2k20 fix ;-) for our perl scripts.
   (merge a40e06ee33 bw/perl-timegm-timelocal-fix later to maint).

 * Threaded "git grep" has been optimized to avoid allocation in code
   section that is covered under a mutex.
   (merge 38ef24dccf rv/grep-cleanup later to maint).

 * "git subtree" script (in contrib/) scripted around "git log", whose
   output got affected by end-user configuration like log.showsignature
   (merge 8841b5222c sg/subtree-signed-commits later to maint).

 * While finding unique object name abbreviation, the code may
   accidentally have read beyond the end of the array of object names
   in a pack.
   (merge 21abed500c ds/find-unique-abbrev-optim later to maint).

 * Micro optimization in revision traversal code.
   (merge ebbed3ba04 ds/mark-parents-uninteresting-optim later to maint).

 * "git commit" used to run "gc --auto" near the end, which was lost
   when the command was reimplemented in C by mistake.
   (merge 095c741edd ab/gc-auto-in-commit later to maint).

 * Allow running a couple of tests with "sh -x".
   (merge c20bf94abc sg/cvs-tests-with-x later to maint).

 * The codepath to replace an existing entry in the index had a bug in
   updating the name hash structure, which has been fixed.
   (merge 0e267b7a24 bp/refresh-cache-ent-rehash-fix later to maint).

 * The transfer.fsckobjects configuration tells "git fetch" to
   validate the data and connected-ness of objects in the received
   pack; the code to perform this check has been taught about the
   narrow clone's convention that missing objects that are reachable
   from objects in a pack that came from a promissor remote is OK.

 * There was an unused file-scope static variable left in http.c when
   building for versions of libCURL that is older than 7.19.4, which
   has been fixed.
   (merge b8fd6008ec rj/http-code-cleanup later to maint).

 * Shell script portability fix.
   (merge 206a6ae013 ml/filter-branch-portability-fix later to maint).

 * Other minor doc, test and build updates and code cleanups.
   (merge e2a5a028c7 bw/oidmap-autoinit later to maint).
   (merge ec3b4b06f8 cl/t9001-cleanup later to maint).
   (merge e1b3f3dd38 ks/submodule-doc-updates later to maint).
   (merge fbac558a9b rs/describe-unique-abbrev later to maint).
   (merge 8462ff43e4 tb/crlf-conv-flags later to maint).
   (merge 7d68bb0766 rb/hashmap-h-compilation-fix later to maint).
   (merge 3449847168 cc/sha1-file-name later to maint).
   (merge ad622a256f ds/use-get-be64 later to maint).
   (merge f919ffebed sg/cocci-move-array later to maint).
   (merge 4e801463c7 jc/mailinfo-cleanup-fix later to maint).
   (merge ef5b3a6c5e nd/shared-index-fix later to maint).
   (merge 9f5258cbb8 tz/doc-show-defaults-to-head later to maint).
   (merge b780e4407d jc/worktree-add-short-help later to maint).
   (merge ae239fc8e5 rs/cocci-strbuf-addf-to-addstr later to maint).
   (merge 2e22a85e5c nd/ignore-glob-doc-update later to maint).
   (merge 3738031581 jk/gettext-poison later to maint).
   (merge 54360a1956 rj/sparse-updates later to maint).
   (merge 12e31a6b12 sg/doc-test-must-fail-args later to maint).
   (merge 760f1ad101 bc/doc-interpret-trailers-grammofix later to maint).
   (merge 4ccf461f56 bp/fsmonitor later to maint).
   (merge a6119f82b1 jk/test-hashmap-updates later to maint).
   (merge 5aea9fe6cc rd/typofix later to maint).
   (merge e4e5da2796 sb/status-doc-fix later to maint).
   (merge 7976e901c8 gs/test-unset-xdg-cache-home later to maint).
   (merge d023df1ee6 tg/worktree-create-tracking later to maint).
   (merge 4cbe92fd41 sm/mv-dry-run-update later to maint).
   (merge 75e5e9c3f7 sb/color-h-cleanup later to maint).
   (merge 2708ef4af6 sg/t6300-modernize later to maint).
   (merge d88e92d4e0 bw/doc-submodule-recurse-config-with-clone later to maint).
   (merge f74bbc8dd2 jk/cached-commit-buffer later to maint).
   (merge 1316416903 ms/non-ascii-ticks later to maint).
   (merge 878056005e rs/strbuf-read-file-or-whine later to maint).
   (merge 79f0ba1547 jk/strbuf-read-file-close-error later to maint).
   (merge edfb8ba068 ot/ref-filter-cleanup later to maint).
   (merge 11395a3b4b jc/test-must-be-empty later to maint).
   (merge 768b9d6db7 mk/doc-pretty-fill later to maint).
   (merge 2caa7b8d27 ab/man-sec-list later to maint).
   (merge 40c17eb184 ks/t3200-typofix later to maint).
   (merge bd9958c358 dp/merge-strategy-doc-fix later to maint).
   (merge 9ee0540a40 js/ming-strftime later to maint).
   (merge 1775e990f7 tz/complete-tag-delete-tagname later to maint).
   (merge 00a4b03501 rj/warning-uninitialized-fix later to maint).
   (merge b635ed97a0 jk/attributes-path-doc later to maint).

----------------------------------------------------------------

Changes since v2.16.0 are as follows:

Adam Borowski (1):
      hooks/pre-auto-gc-battery: allow gc to run on non-laptops

Alban Gruin (1):
      userdiff: add built-in pattern for golang

Alex Bennée (1):
      send-email: add test for Linux's get_maintainer.pl

Alexander Shopov (1):
      Mark messages for translations

Andreas G. Schacker (1):
      doc/read-tree: remove obsolete remark

Ben Peart (4):
      dir.c: don't flag the index as dirty for changes to the untracked cache
      name-hash: properly fold directory names in adjust_dirname_case()
      fsmonitor: update documentation to remove reference to invalid config settings
      Fix bugs preventing adding updated cache entries to the name hash

Bernhard M. Wiedemann (1):
      perl: call timegm and timelocal with 4-digit year

Brandon Williams (39):
      oidmap: ensure map is initialized
      object_info: change member name from 'typename' to 'type_name'
      object: rename function 'typename' to 'type_name'
      blame: rename 'this' variables
      pack-objects: rename 'this' variables
      rev-parse: rename 'this' variable
      submodule: indicate that 'submodule.recurse' doesn't apply to clone
      diff: rename 'this' variables
      apply: rename 'try' variables
      apply: rename 'new' variables
      checkout: rename 'new' variables
      help: rename 'new' variables
      pack-redundant: rename 'new' variables
      reflog: rename 'new' variables
      remote: rename 'new' variables
      combine-diff: rename 'new' variables
      commit: rename 'new' variables
      diff-lib: rename 'new' variable
      diff: rename 'new' variables
      diffcore-delta: rename 'new' variables
      entry: rename 'new' variables
      http: rename 'new' variables
      imap-send: rename 'new' variables
      line-log: rename 'new' variables
      read-cache: rename 'new' variables
      ref-filter: rename 'new' variables
      remote: rename 'new' variables
      split-index: rename 'new' variables
      submodule: rename 'new' variables
      trailer: rename 'new' variables
      unpack-trees: rename 'new' variables
      init-db: rename 'template' variables
      environment: rename 'template' variables
      diff: rename 'template' variables
      environment: rename 'namespace' variables
      wrapper: rename 'template' variables
      tempfile: rename 'template' variables
      trailer: rename 'template' variables
      replace: rename 'new' variables

Christian Couder (12):
      perf/aggregate: fix checking ENV{GIT_PERF_SUBSECTION}
      perf/aggregate: refactor printing results
      perf/aggregate: implement codespeed JSON output
      perf/run: add conf_opts argument to get_var_from_env_or_config()
      perf/run: learn about perf.codespeedOutput
      perf/run: learn to send output to codespeed server
      perf/run: read GIT_PERF_REPO_NAME from perf.repoName
      sha1_file: remove static strbuf from sha1_file_name()
      sha1_file: improve sha1_file_name() perfs
      perf/aggregate: add --subsection option
      perf/aggregate: add --reponame option
      perf/aggregate: sort JSON fields in output

Christian Ludwig (3):
      t9001: use existing helper in send-email test
      send-email: rename variable for clarity
      send-email: support separate Reply-To address

Daniel Knittl-Frank (1):
      describe: prepend "tags/" when describing tags with embedded name

David Pursehouse (1):
      Documentation/merge-strategies: typofix

Derrick Stolee (3):
      packfile: use get_be64() for large offsets
      sha1_name: fix uninitialized memory errors
      revision.c: reduce object database queries

Elijah Newren (3):
      Tighten and correct a few testcases for merging and cherry-picking
      merge-recursive: fix logic ordering issue
      merge-recursive: add explanation for src_entry and dst_entry

Eric Sunshine (5):
      t5601-clone: test case-conflicting files on case-insensitive filesystem
      worktree: add: fix 'post-checkout' not knowing new worktree location
      git-worktree.txt: fix missing ")" typo
      git-worktree.txt: fix indentation of example and text of 'add' command
      t2028: fix minor error and issues in newly-added "worktree move" tests

Eric Wong (2):
      fsck: fix leak when traversing trees
      git-svn: control destruction order to avoid segfault

Gargi Sharma (1):
      mru: Replace mru.[ch] with list.h implementation

Genki Sky (2):
      rebase: add --allow-empty-message option
      test-lib.sh: unset XDG_CACHE_HOME

Gregory Herrero (1):
      rebase -p: fix incorrect commit message when calling `git merge`.

Jason Merrill (1):
      git-svn: fix svn.pushmergeinfo handling of svn+ssh usernames.

Jeff Hostetler (12):
      upload-pack: add object filtering for partial clone
      fetch-pack, index-pack, transport: partial clone
      fetch-pack: add --no-filter
      fetch: support filters
      partial-clone: define partial clone settings in config
      t5616: end-to-end tests for partial clone
      fetch: inherit filter-spec from partial clone
      t5616: test bulk prefetch after partial fetch
      stat_tracking_info: return +1 when branches not equal
      status: add --[no-]ahead-behind to status and commit for V2 format.
      status: update short status to respect --no-ahead-behind
      status: support --no-ahead-behind in long format

Jeff King (35):
      t5600: fix outdated comment about unborn HEAD
      t5600: modernize style
      clone: factor out dir_exists() helper
      clone: do not clean up directories we didn't create
      sq_quote_argv: drop maxlen parameter
      trace: avoid unnecessary quoting
      t5570: use ls-remote instead of clone for interp tests
      t/lib-git-daemon: record daemon log
      daemon: fix off-by-one in logging extended attributes
      daemon: handle NULs in extended attribute string
      t/lib-git-daemon: add network-protocol helpers
      daemon: fix length computation in newline stripping
      t0205: drop redundant test
      git-sh-i18n: check GETTEXT_POISON before USE_GETTEXT_SCHEME
      correct error messages for NULL packet_read_line()
      CodingGuidelines: mention "static" and "extern"
      t0002: simplify error checking
      describe: confirm that blobs actually exist
      test-hashmap: use ALLOC_ARRAY rather than bare malloc
      test-hashmap: check allocation computation for overflow
      test-hashmap: use xsnprintf rather than snprintf
      test-hashmap: use strbuf_getline rather than fgets
      test-hashmap: simplify alloc_test_entry
      test-hashmap: use "unsigned int" for hash storage
      sq_dequote: fix extra consumption of source string
      t5545: factor out http repository setup
      remote-curl: unquote incoming push-options
      commit: drop uses of get_cached_commit_buffer()
      revision: drop --show-all option
      t: send verbose test-helper output to fd 4
      strbuf_read_file(): preserve errno across close() call
      smart-http: document flush after "# service" line
      t3701: add a test for interactive.diffFilter
      add--interactive: detect bogus diffFilter output
      doc/gitattributes: mention non-recursive behavior

Johannes Schindelin (3):
      sequencer: assign only free()able strings to gpg_sign
      apply: demonstrate a problem applying svn diffs
      mingw: abort on invalid strftime formats

Jon Simons (1):
      always check for NULL return from packet_read_line()

Jonathan Nieder (1):
      perl: treat PERLLIB_EXTRA as an extra path again

Jonathan Tan (23):
      extension.partialclone: introduce partial clone extension
      fsck: introduce partialclone extension
      fsck: support refs pointing to promisor objects
      fsck: support referenced promisor objects
      fsck: support promisor objects as CLI argument
      index-pack: refactor writing of .keep files
      introduce fetch-object: fetch one promisor object
      sha1_file: support lazily fetching missing objects
      rev-list: support termination at promisor objects
      gc: do not repack promisor packfiles
      fetch-pack: test support excluding large blobs
      fetch: refactor calculation of remote list
      clone: partial clone
      unpack-trees: batch fetching of missing blobs
      fetch-pack: restore save_commit_buffer after use
      http: support cookie redaction when tracing
      http: support omitting data from traces
      Docs: split out long-running subprocess handshake
      packfile: remove GIT_DEBUG_LOOKUP log statements
      packfile: refactor hash search with fanout table
      sha1_file: restore OBJECT_INFO_QUICK functionality
      index-pack: support checking objects but not links
      fetch-pack: do not check links for partial fetch

Juan F. Codagnone (1):
      mailinfo: avoid segfault when can't open files

Junio C Hamano (22):
      merge: teach -Xours/-Xtheirs to symbolic link merge
      worktree: say that "add" takes an arbitrary commit in short-help
      Start 2.17 cycle
      Git 2.16.1
      First batch after 2.16
      blame: tighten command line parser
      Second batch for 2.17
      Third batch for 2.17
      Git 2.16.2
      merge: allow fast-forward when merging a tracked tag
      Fourth batch for 2.17
      Fifth batch for 2.17
      test_must_be_empty: make sure the file exists, not just empty
      untracked cache: use git_env_bool() not getenv() for customization
      Sixth batch for 2.17
      Seventh batch for 2.17
      Eighth batch for 2.17
      Git 2.17-rc0
      Git 2.17-rc1
      Git 2.16.3
      t9902: disable test on the list of merge-strategies under GETTEXT_POISON
      Git 2.17-rc2

Kaartic Sivaraam (3):
      Doc/gitsubmodules: make some changes to improve readability and syntax
      Doc/git-submodule: improve readability and grammar of a sentence
      t/t3200: fix a typo in a test description

Kim Gybels (1):
      packed_ref_cache: don't use mmap() for small files

Lucas Werkmeister (1):
      daemon: add --log-destination=(stderr|syslog|none)

Martin Ågren (9):
      t7006: add tests for how git config paginates
      config: respect `pager.config` in list/get-mode only
      config: change default of `pager.config` to "on"
      sequencer: make lockfiles non-static
      sequencer: always roll back lock in `do_recursive_merge()`
      merge-recursive: always roll back lock in `merge_recursive_generic()`
      merge: always roll back lock in `checkout_fast_forward()`
      sequencer: do not roll back lockfile unnecessarily
      write_locked_index(): add flag to avoid writing unchanged index

Mathias Rav (1):
      files_initial_transaction_commit(): only unlock if locked

Matthieu Moy (2):
      send-email: add and use a local copy of Mail::Address
      perl/Git: remove now useless email-address parsing code

Michael Haggerty (5):
      struct snapshot: store `start` rather than `header_len`
      create_snapshot(): use `xmemdupz()` rather than a strbuf
      find_reference_location(): make function safe for empty snapshots
      packed_ref_iterator_begin(): make optimization more general
      load_contents(): don't try to mmap an empty file

Michele Locati (1):
      filter-branch: use printf instead of echo -e

Motoki Seki (1):
      Documentation/gitsubmodules.txt: avoid non-ASCII apostrophes

Mårten Kongstad (1):
      docs/pretty-formats: fix typo '% <(<N>)' -> '%<|(<N>)'

Nathan Payre (1):
      send-email: extract email-parsing code into a subroutine

Nguyễn Thái Ngọc Duy (85):
      t2203: test status output with porcelain v2 format
      Use DIFF_DETECT_RENAME for detect_rename assignments
      wt-status.c: coding style fix
      wt-status.c: catch unhandled diff status codes
      wt-status.c: rename rename-related fields in wt_status_change_data
      wt-status.c: handle worktree renames
      trace.c: move strbuf_release() out of print_trace_line()
      add--interactive: ignore submodule changes except HEAD
      read-cache.c: change type of "temp" in write_shared_index()
      read-cache.c: move tempfile creation/cleanup out of write_shared_index
      diff.c: flush stdout before printing rename warnings
      run-command.c: introduce trace_run_command()
      run-command.c: print program 'git' when tracing git_cmd mode
      run-command.c: print env vars in trace_run_command()
      run-command.c: print new cwd in trace_run_command()
      read-cache: don't write index twice if we can't write shared index
      worktree.c: add validate_worktree()
      dir.c: avoid stat() in valid_cached_dir()
      dir.c: fix missing dir invalidation in untracked code
      format-patch: keep cover-letter diffstat wrapped in 72 columns
      completion: fix completing merge strategies on non-C locales
      dir.c: stop ignoring opendir() error in open_cached_dir()
      format-patch: reduce patch diffstat width to 72
      gitignore.txt: elaborate shell glob syntax
      trace: measure where the time is spent in the index-heavy operations
      diff.c: refactor pprint_rename() to use strbuf
      dir.c: ignore paths containing .git when invalidating untracked cache
      parse-options: support --git-completion-helper
      parse-options: add OPT_xxx_F() variants
      parse-options: let OPT__FORCE take optional flags argument
      git-completion.bash: introduce __gitcomp_builtin
      completion: use __gitcomp_builtin in _git_add
      completion: use __gitcomp_builtin in _git_am
      completion: use __gitcomp_builtin in _git_apply
      completion: use __gitcomp_builtin in _git_branch
      completion: use __gitcomp_builtin in _git_checkout
      completion: use __gitcomp_builtin in _git_cherry_pick
      completion: use __gitcomp_builtin in _git_clean
      completion: use __gitcomp_builtin in _git_clone
      completion: use __gitcomp_builtin in _git_commit
      completion: use __gitcomp_builtin in _git_config
      completion: use __gitcomp_builtin in _git_describe
      completion: use __gitcomp_builtin in _git_difftool
      completion: use __gitcomp_builtin in _git_fetch
      completion: use __gitcomp_builtin in _git_fsck
      completion: use __gitcomp_builtin in _git_gc
      completion: use __gitcomp_builtin in _git_grep
      completion: use __gitcomp_builtin in _git_help
      completion: use __gitcomp_builtin in _git_init
      completion: use __gitcomp_builtin in _git_ls_files
      completion: use __gitcomp_builtin in _git_ls_remote
      completion: use __gitcomp_builtin in _git_merge
      completion: use __gitcomp_builtin in _git_merge_base
      completion: use __gitcomp_builtin in _git_mv
      completion: use __gitcomp_builtin in _git_name_rev
      completion: use __gitcomp_builtin in _git_notes
      completion: use __gitcomp_builtin in _git_pull
      completion: use __gitcomp_builtin in _git_push
      completion: use __gitcomp_builtin in _git_remote
      remote: force completing --mirror= instead of --mirror
      completion: use __gitcomp_builtin in _git_replace
      completion: use __gitcomp_builtin in _git_reset
      completion: use __gitcomp_builtin in _git_revert
      completion: use __gitcomp_builtin in _git_rm
      completion: use __gitcomp_builtin in _git_show_branch
      completion: use __gitcomp_builtin in _git_status
      completion: use __gitcomp_builtin in _git_tag
      completion: use __gitcomp_builtin in _git_worktree
      worktree.c: add update_worktree_location()
      worktree move: new command
      worktree move: accept destination as directory
      worktree move: refuse to move worktrees with submodules
      worktree remove: new command
      worktree remove: allow it when $GIT_WORK_TREE is already gone
      am: add --show-current-patch
      rebase: add --show-current-patch
      rebase: introduce and use pseudo-ref REBASE_HEAD
      am: support --quit
      diff: add --compact-summary
      object.h: update flag allocation comment
      object.h: realign object flag allocation comment
      completion: don't set PARSE_OPT_NOCOMPLETE on --rerere-autoupdate
      completion: simplify _git_notes
      completion: complete --{reuse,reedit}-message= for all notes subcmds
      completion: more subcommands in _git_notes()

Nicolas Morey-Chaisemartin (1):
      tag: add --edit option

Olga Telezhnaya (3):
      mru: use double-linked list from list.h
      ref-filter: get rid of duplicate code
      ref-filter: get rid of goto

Patryk Obara (14):
      clang-format: adjust penalty for return type line break
      http-push: improve error log
      sha1_file: convert pretend_sha1_file to object_id
      dir: convert struct sha1_stat to use object_id
      sha1_file: convert hash_sha1_file to object_id
      cache: clear whole hash buffer with oidclr
      match-trees: convert splice_tree to object_id
      commit: convert commit_tree* to object_id
      notes: convert combine_notes_* to object_id
      notes: convert write_notes_tree to object_id
      sha1_file: convert write_sha1_file to object_id
      sha1_file: convert force_object_loose to object_id
      sha1_file: convert write_loose_object to object_id
      sha1_file: rename hash_sha1_file_literally

Phillip Wood (25):
      t3404: check intermediate squash messages
      commit: move empty message checks to libgit
      Add a function to update HEAD after creating a commit
      commit: move post-rewrite code to libgit
      commit: move print_commit_summary() to libgit
      sequencer: simplify adding Signed-off-by: trailer
      sequencer: load commit related config
      sequencer: try to commit without forking 'git commit'
      t3512/t3513: remove KNOWN_FAILURE_CHERRY_PICK_SEES_EMPTY_COMMIT=1
      sequencer: improve config handling
      t7505: style fixes
      t7505: add tests for cherry-pick and rebase -i/-p
      sequencer: run 'prepare-commit-msg' hook
      add -p: only display help for active keys
      add -p: only bind search key if there's more than one hunk
      add -p: improve error messages
      add -i: add function to format hunk header
      t3701: indent here documents
      t3701: use test_write_lines and write_script
      t3701: don't hard code sha1 hash values
      t3701: add failing test for pathological context lines
      add -p: adjust offsets of subsequent hunks when one is skipped
      add -p: calculate offset delta for edited patches
      add -p: fix counting when splitting and coalescing
      add -p: don't rely on apply's '--recount' option

Prathamesh Chavan (2):
      submodule: port submodule subcommand 'sync' from shell to C
      submodule: port submodule subcommand 'deinit' from shell to C

Ramsay Jones (6):
      t4151: consolidate multiple calls to test_i18ngrep
      config.mak.uname: remove SPARSE_FLAGS setting for cygwin
      Makefile: suppress a sparse warning for pack-revindex.c
      http: fix an unused variable warning for 'curl_no_proxy'
      -Wuninitialized: remove some 'init-self' workarounds
      read-cache: fix an -Wmaybe-uninitialized warning

Randall S. Becker (1):
      hashmap.h: remove unused variable

Rasmus Villemoes (2):
      grep: move grep_source_init outside critical section
      grep: simplify grep_oid and grep_file

René Scharfe (15):
      commit: avoid allocation in clear_commit_marks_many()
      commit: use clear_commit_marks_many() in remove_redundant()
      ref-filter: use clear_commit_marks_many() in do_merge_filter()
      object: add clear_commit_marks_all()
      bisect: avoid using the rev_info flag leak_pending
      bundle: avoid using the rev_info flag leak_pending
      checkout: avoid using the rev_info flag leak_pending
      revision: remove the unused flag leak_pending
      commit: remove unused function clear_commit_marks_for_object_array()
      describe: use strbuf_add_unique_abbrev() for adding short hashes
      cocci: use format keyword instead of a literal string
      cocci: simplify check for trivial format strings
      check-ignore: fix mix of directories and other file types
      sequencer: factor out strbuf_read_file_or_whine()
      perf: use GIT_PERF_REPEAT_COUNT=3 by default even without config file

Robert P. J. Day (2):
      t/: correct obvious typo "detahced"
      Correct mispellings of ".gitmodule" to ".gitmodules"

SZEDER Gábor (34):
      travis-ci: build Git during the 'script' phase
      Use MOVE_ARRAY
      travis-ci: use 'set -x' for the commands under 'su' in the 32 bit Linux build
      travis-ci: use 'set -e' in the 32 bit Linux build job
      travis-ci: don't repeat the path of the cache directory
      travis-ci: don't run the test suite as root in the 32 bit Linux build
      travis-ci: don't fail if user already exists on 32 bit Linux build job
      t5541: add 'test_i18ngrep's missing filename parameter
      t5812: add 'test_i18ngrep's missing filename parameter
      t6022: don't run 'git merge' upstream of a pipe
      t4001: don't run 'git status' upstream of a pipe
      t5510: consolidate 'grep' and 'test_i18ngrep' patterns
      t5536: let 'test_i18ngrep' read the file without redirection
      t: move 'test_i18ncmp' and 'test_i18ngrep' to 'test-lib-functions.sh'
      t: validate 'test_i18ngrep's parameters
      t: make 'test_i18ngrep' more informative on failure
      t: document 'test_must_fail ok=<signal-name>'
      t6300-for-each-ref: fix "more than one quoting style" tests
      Makefile: generate Git(3pm) as dependency of the 'doc' and 'man' targets
      t: prevent '-x' tracing from interfering with test helpers' stderr
      t: add means to disable '-x' tracing for individual test scripts
      t1507-rev-parse-upstream: don't check the stderr of a shell function
      t5536: simplify checking of messages output to stderr
      t3030-merge-recursive: don't check the stderr of a subshell
      t5500-fetch-pack: don't check the stderr of a subshell
      t5526: use $TRASH_DIRECTORY to specify the path of GIT_TRACE log file
      t5570-git-daemon: don't check the stderr of a subshell
      t9903-bash-prompt: don't check the stderr of __git_ps1()
      t1510-repo-setup: mark as untraceable with '-x'
      t/README: add a note about don't saving stderr of compound commands
      travis-ci: run tests with '-x' tracing
      t9400-git-cvsserver-server: don't rely on the output of 'test_cmp'
      t9402-git-cvsserver-refs: don't check the stderr of a subshell
      completion: clear cached --options when sourcing the completion script

Stefan Beller (15):
      diff.h: make pickaxe_opts an unsigned bit field
      diff: migrate diff_flags.pickaxe_ignore_case to a pickaxe_opts bit
      diff: introduce DIFF_PICKAXE_KINDS_MASK
      diffcore: add a pickaxe option to find a specific blob
      diff: properly error out when combining multiple pickaxe options
      diff: use HAS_MULTI_BITS instead of counting bits manually
      t/lib-submodule-update.sh: clarify test
      t/lib-submodule-update.sh: fix test ignoring ignored files in submodules
      unpack-trees: oneway_merge to update submodules
      submodule: submodule_move_head omits old argument in forced case
      builtin/pull: respect verbosity settings in submodules
      send-email: error out when relogin delay is missing
      color.h: document and modernize header
      Documentation/git-status: clarify status table for porcelain mode
      submodule deinit: handle non existing pathspecs gracefully

Stefan Moch (2):
      t7001: add test case for --dry-run
      mv: remove unneeded 'if (!show_only)'

Stephen R Guglielmo (1):
      subtree: fix add and pull for GPG-signed commits

Tatyana Krasnukha (1):
      apply: handle Subversion diffs with /dev/null gracefully

Thomas Gummerer (6):
      stash: don't delete untracked files that match pathspec
      read-cache: fix reading the shared index for other repos
      split-index: don't write cache tree with null oid entries
      travis: run tests with GIT_TEST_SPLIT_INDEX
      reset --hard: make use of the pretty machinery
      git-stash.txt: remove extra square bracket

Thomas Levesque (1):
      userdiff.c: add C# async keyword in diff pattern

Todd Zullinger (5):
      doc: mention 'git show' defaults to HEAD
      Makefile: remove *.spec from clean target
      Makefile: add NO_PERL_CPAN_FALLBACKS knob
      RelNotes: add details on Perl module changes
      completion: complete tags with git tag --delete/--verify

Torsten Bögershausen (1):
      convert_to_git(): safe_crlf/checksafe becomes int conv_flags

Yasushi SHOJI (1):
      bisect: debug: convert struct object to object_id

brian m. carlson (15):
      repository: pre-initialize hash algo pointer
      hash: move SHA-1 macros to hash.h
      hash: create union for hash context allocation
      builtin/index-pack: improve hash function abstraction
      builtin/unpack-objects: switch uses of SHA-1 to the_hash_algo
      sha1_file: switch uses of SHA-1 to the_hash_algo
      fast-import: switch various uses of SHA-1 to the_hash_algo
      pack-check: convert various uses of SHA-1 to abstract forms
      pack-write: switch various SHA-1 values to abstract forms
      read-cache: abstract away uses of SHA-1
      csum-file: rename sha1file to hashfile
      csum-file: abstract uses of SHA-1
      bulk-checkin: abstract SHA-1 usage
      hash: update obsolete reference to SHA1_HEADER
      docs/interpret-trailers: fix agreement error

Ævar Arnfjörð Bjarmason (53):
      Makefile: don't error out under DC_SHA1_EXTERNAL if DC_SHA1_SUBMODULE=auto
      Makefile: under "make dist", include the sha1collisiondetection submodule
      sha1dc_git.h: re-arrange an ifdef chain for a subsequent change
      Makefile: replace perl/Makefile.PL with simple make rules
      commit doc: document that -c, -C, -F and --fixup with -m error
      commit: add support for --fixup <commit> -m"<extra message>"
      perl: avoid *.pmc and fix Error.pm further
      perf: amend the grep tests to test grep.threads
      cat-file doc: document that -e will return some output
      status: add a failing test showing a core.untrackedCache bug
      wildmatch test: indent with tabs, not spaces
      wildmatch test: use more standard shell style
      wildmatch test: don't try to vertically align our output
      wildmatch test: use a paranoia pattern from nul_match()
      wildmatch test: remove dead fnmatch() test code
      wildmatch test: use test_must_fail, not ! for test-wildmatch
      wildmatch test: perform all tests under all wildmatch() modes
      wildmatch test: create & test files on disk in addition to in-memory
      test-lib: add an EXPENSIVE_ON_WINDOWS prerequisite
      wildmatch test: mark test as EXPENSIVE_ON_WINDOWS
      fetch: don't redundantly NULL something calloc() gave us
      fetch: trivially refactor assignment to ref_nr
      fetch: stop accessing "remote" variable indirectly
      remote: add a macro for "refs/tags/*:refs/tags/*"
      fetch tests: refactor in preparation for testing tag pruning
      fetch tests: re-arrange arguments for future readability
      fetch tests: add a tag to be deleted to the pruning tests
      fetch tests: test --prune and refspec interaction
      fetch tests: double quote a variable for interpolation
      fetch tests: expand case/esac for later change
      fetch tests: fetch <url> <spec> as well as fetch [<remote>]
      git fetch doc: add a new section to explain the ins & outs of pruning
      git remote doc: correct dangerous lies about what prune does
      git-fetch & config doc: link to the new PRUNING section
      fetch tests: add scaffolding for the new fetch.pruneTags
      fetch: add a --prune-tags option and fetch.pruneTags config
      fetch: make the --prune-tags work with <url>
      update-index doc: note a fixed bug in the untracked cache
      update-index doc: note the caveat with "could not open..."
      perl: *.pm files should not have the executable bit
      Git.pm: remove redundant "use strict" from sub-package
      Git.pm: add the "use warnings" pragma
      commit: run git gc --auto just before the post-commit hook
      gitweb: hard-depend on the Digest::MD5 5.8 module
      Git.pm: hard-depend on the File::{Temp,Spec} modules
      git-send-email: unconditionally use Net::{SMTP,Domain}
      perl: update our ancient copy of Error.pm
      perl: update our copy of Mail::Address
      perl: move CPAN loader wrappers to another namespace
      perl: generalize the Git::LoadCPAN facility
      perl: move the perl/Git/FromCPAN tree to perl/FromCPAN
      perl Git::LoadCPAN: emit better errors under NO_PERL_CPAN_FALLBACKS
      git manpage: note git-security@googlegroups.com


^ permalink raw reply	[relevance 3%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 18:57   ` Eric Sunshine
@ 2018-03-28 20:08     ` Stefan Beller
  2018-03-28 20:21       ` Eric Sunshine
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-03-28 20:08 UTC (permalink / raw)
  To: Eric Sunshine
  Cc: René Scharfe, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

On Wed, Mar 28, 2018 at 11:57 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Wed, Mar 28, 2018 at 2:38 PM, Stefan Beller <sbeller@google.com> wrote:
>> diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
>> @@ -821,6 +821,18 @@ test_expect_success 'moving the superproject does not break submodules' '
>> +test_expect_success 'moving the submodule does not break the superproject' '
>> +       (
>> +               cd addtest2 &&
>> +
>> +               mv repo repo.bak &&
>> +               git submodule status >actual &&
>> +               grep -e "^-" -e repo actual &&
>> +
>> +               mv repo.bak repo
>
> Should this "move back" be encapsulated in a test_when_finished?

I thought about that, but decided against it for some reason as I was debating
where to put the test_when_finished. I mostly saw those at the very beginning
of a test and wondered if it can be called from within a subshell.
(I'd not want to put it at the beginning but rather adjacent to the move.)

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 20:08     ` Stefan Beller
@ 2018-03-28 20:21       ` Eric Sunshine
  2018-03-28 20:54         ` Stefan Beller
  2018-03-28 21:14         ` René Scharfe
  0 siblings, 2 replies; 200+ results
From: Eric Sunshine @ 2018-03-28 20:21 UTC (permalink / raw)
  To: Stefan Beller
  Cc: René Scharfe, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

On Wed, Mar 28, 2018 at 4:08 PM, Stefan Beller <sbeller@google.com> wrote:
> On Wed, Mar 28, 2018 at 11:57 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>> +test_expect_success 'moving the submodule does not break the superproject' '
>>> +       (
>>> +               cd addtest2 &&
>>> +
>>> +               mv repo repo.bak &&
>>> +               git submodule status >actual &&
>>> +               grep -e "^-" -e repo actual &&
>>> +
>>> +               mv repo.bak repo
>>
>> Should this "move back" be encapsulated in a test_when_finished?
>
> I thought about that, but decided against it for some reason as I was debating
> where to put the test_when_finished. I mostly saw those at the very beginning
> of a test and wondered if it can be called from within a subshell.
> (I'd not want to put it at the beginning but rather adjacent to the move.)

It looks like test_when_finished() shouldn't be used in a subshell.
However, wouldn't the following be reasonable?

    mv addtest2/repo addtest2/repo.bak &&
    test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
    (
        cd addtest2 &&
        git submodule status >actual &&
        grep -e "^-" -e repo actual
    )

Or, even simpler:

    mv addtest2/repo addtest2/repo.bak &&
    test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
    git -C addtest2 submodule status >actual &&
    grep -e "^-" -e repo actual

^ permalink raw reply	[relevance 8%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 20:21       ` Eric Sunshine
@ 2018-03-28 20:54         ` Stefan Beller
  2018-03-28 21:14         ` René Scharfe
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 20:54 UTC (permalink / raw)
  To: Eric Sunshine
  Cc: René Scharfe, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

On Wed, Mar 28, 2018 at 1:21 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:

> Or, even simpler:
>
>     mv addtest2/repo addtest2/repo.bak &&
>     test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
>     git -C addtest2 submodule status >actual &&
>     grep -e "^-" -e repo actual

I like this!
Time permitting I'll update the patch (I'll first have to take
care of some other series, so feel free to make it into a patch)

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 20:21       ` Eric Sunshine
  2018-03-28 20:54         ` Stefan Beller
@ 2018-03-28 21:14         ` René Scharfe
  2018-03-28 21:37           ` Stefan Beller
  1 sibling, 1 reply; 200+ results
From: René Scharfe @ 2018-03-28 21:14 UTC (permalink / raw)
  To: Eric Sunshine, Stefan Beller
  Cc: Git List, Junio C Hamano, jeremy, Prathamesh Chavan

Am 28.03.2018 um 22:21 schrieb Eric Sunshine:
> On Wed, Mar 28, 2018 at 4:08 PM, Stefan Beller <sbeller@google.com> wrote:
>> On Wed, Mar 28, 2018 at 11:57 AM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>>>> +test_expect_success 'moving the submodule does not break the superproject' '
>>>> +       (
>>>> +               cd addtest2 &&
>>>> +
>>>> +               mv repo repo.bak &&
>>>> +               git submodule status >actual &&
>>>> +               grep -e "^-" -e repo actual &&
>>>> +
>>>> +               mv repo.bak repo
>>>
>>> Should this "move back" be encapsulated in a test_when_finished?
>>
>> I thought about that, but decided against it for some reason as I was debating
>> where to put the test_when_finished. I mostly saw those at the very beginning
>> of a test and wondered if it can be called from within a subshell.
>> (I'd not want to put it at the beginning but rather adjacent to the move.)
> 
> It looks like test_when_finished() shouldn't be used in a subshell.
> However, wouldn't the following be reasonable?
> 
>      mv addtest2/repo addtest2/repo.bak &&
>      test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
>      (
>          cd addtest2 &&
>          git submodule status >actual &&
>          grep -e "^-" -e repo actual
>      )

I like this version, except for the grep call -- it accepts either lines
starting with a dash or containing "repo".  So if status would report
just "-" and nothing else then the test would pass. 

> Or, even simpler:
> 
>      mv addtest2/repo addtest2/repo.bak &&
>      test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
>      git -C addtest2 submodule status >actual &&
>      grep -e "^-" -e repo actual
> 

This looks nicer here in the script, but doesn't test exactly what users
type most of the time, I suppose.

So how about this?

-- >8 --
Subject: [PATCH v3] submodule: check for NULL return of get_submodule_ref_store()

If we can't find a ref store for a submodule then assume the latter
is not initialized (or was removed).  Print a status line accordingly
instead of causing a segmentation fault by passing NULL as the first
parameter of refs_head_ref().

Reported-by: Jeremy Feusi <jeremy@feusi.co>
Reviewed-by: Stefan Beller <sbeller@google.com>
Initial-Test-By: Stefan Beller <sbeller@google.com>
Helped-by: Eric Sunshine <sunshine@sunshineco.com>
Signed-off-by: Rene Scharfe <l.s.r@web.de>
---
 builtin/submodule--helper.c |  8 ++++++--
 t/t7400-submodule-basic.sh  | 15 +++++++++++++++
 2 files changed, 21 insertions(+), 2 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 6ba8587b6d..9a0fb5e784 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -654,9 +654,13 @@ static void status_submodule(const char *path, const struct object_id *ce_oid,
 			     displaypath);
 	} else if (!(flags & OPT_CACHED)) {
 		struct object_id oid;
+		struct ref_store *refs = get_submodule_ref_store(path);
 
-		if (refs_head_ref(get_submodule_ref_store(path),
-				  handle_submodule_head_ref, &oid))
+		if (!refs) {
+			print_status(flags, '-', path, ce_oid, displaypath);
+			goto cleanup;
+		}
+		if (refs_head_ref(refs, handle_submodule_head_ref, &oid))
 			die(_("could not resolve HEAD ref inside the "
 			      "submodule '%s'"), path);
 
diff --git a/t/t7400-submodule-basic.sh b/t/t7400-submodule-basic.sh
index a39e69a3eb..ef1ea8d6b0 100755
--- a/t/t7400-submodule-basic.sh
+++ b/t/t7400-submodule-basic.sh
@@ -821,6 +821,21 @@ test_expect_success 'moving the superproject does not break submodules' '
 	)
 '
 
+test_expect_success 'moving the submodule does not break the superproject' '
+	(
+		cd addtest2 &&
+		git submodule status
+	) >actual &&
+	sed -e "s/^ \([^ ]* repo\) .*/-\1/" <actual >expect &&
+	mv addtest2/repo addtest2/repo.bak &&
+	test_when_finished "mv addtest2/repo.bak addtest2/repo" &&
+	(
+		cd addtest2 &&
+		git submodule status
+	) >actual &&
+	test_cmp expect actual
+'
+
 test_expect_success 'submodule add --name allows to replace a submodule with another at the same path' '
 	(
 		cd addtest2 &&
-- 
2.16.3

^ permalink raw reply	[relevance 22%]

* Re: git submodule deinit resulting in BUG: builtin/submodule--helper.c:1045: module_list_compute should not choke on empty pathspec
  2018-03-28 19:37   ` git submodule deinit resulting in BUG: builtin/submodule--helper.c:1045: module_list_compute should not choke on empty pathspec Peter Oberndorfer
@ 2018-03-28 21:19     ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 21:19 UTC (permalink / raw)
  To: Peter Oberndorfer; +Cc: git, Prathamesh Chavan

On Wed, Mar 28, 2018 at 12:37 PM, Peter Oberndorfer <kumbayo84@arcor.de> wrote:

>>> 2) Should "git submodule deinit" work on submodules that were removed by
>> upstream already?
>>
>> To answer the question "Is this a submodule that upstream removed
>> (recently)?"
>> we'd have to put in some effort, essentially checking if that was ever a
>> submodule
>> (and not a directory or file).
>>
>
> Hmm, yeah looks a bit more complicated than I initially imagined
> since submodules can have a name that's different from their path.
> And after the rebase, the name <-> path mapping via .gitmodules is not available anymore.
>
> Naively I think it could work the following way:
> * Either iterate over all submodules in .git/modules/ and check their config
>   has a worktree = "../../path" that resolves to the submodule path we want to remove.

This would work but scales linearly with the number of submodules.


> * Or check the "gitlink:" path in submodule/.git if it points to our .git/modules/
> Then if .git/config contains a [submodule "name"] entry
> we should have a pretty good idea if this folder contains a stale submodule.

If you move a submodule a directory up or down, the relative path is not exact
any more, we'd need to check for the last part to loosely match.


>> When using "git pull --recurse-submodules" the submodule ought to be removed
>> automatically.
>>
>> When doing a fetch && merge manually, we may want to teach merge to remove
>> a submodule that we have locally upon merge, too.
>>
>
> Yeah that would be nice :-)
> In my case I updated the repository via a rebase, so that would also have to be covered.

Oh rebase itself has not yet learned about recursion into submodules.
("git pull --rebase --recurse-submodules" is a thing though)

>> I view the git-submodule command as a bare bones plumbing helper, that we'd
>> want
>> to deprecate eventually as all other higher level commands will know how to
>> deal
>> with submodules.
>>
>> So I think we do not want to teach "git submodule deinit" to remove dormant
>> repositories, that were submodules removed by upstream already.
>>
>
> My gut feeling makes me expect the following:
> * It would be nice if such stale submodules showed up in "git submodule status" or "git status"
>   Now "git submodule" shows nothing related to this stale submodule

That has currently only two ways "+" or "-" for there/not there.
Maybe we'd need to add some characters similar to "git status --porcelain"
such as "?"

>   Now "git status" shows  Untracked files: src/rt which is a bit confusing as the actual submodule is in src/rt/hoedown
>   Now "Git gui" shows src/rt/hoedown as untracked git repository

hm. The current state of affairs doesn't sound intriguing.
Though, I think we'd want to step back one more step and rather want
to ask how a dormant submodule comes into existence, instead of
just improving the reporting. Reportingthem is of course also important,
but in the long run I'd rather want to have situations like these happen
less often. When upstream deletes a file, they are also not required to be
deleted manually, but merge/checkout would take care of them.

> * There should be an easy(and safe) way for the user to deinit such a submodule
>   if if the automatic submodule updating during a merge/rebase was not enabled or somehow failed.
> (Minus the problem of somebody having to actually do the work...)
>
>>> ~/src/rust/rust$ git submodule status
>> ...
>>>   b87873eaceb75cf9342d5273f01ba2c020f61ca8 src/tools/lld ((null))
>>
>>> -> strangely I get (null) for the current branch/commit in some
>> submodules?
>>
>> This sounds like (3). Looking into that.
>
> Sorry, what do you mean by (3)?

I meant the ((null)) issue is another third thought that we can
discuss separately,
slightly unrelated to the others (that you marked as (1) and (2))

Thanks,
Stefan

^ permalink raw reply	[relevance 8%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 21:14         ` René Scharfe
@ 2018-03-28 21:37           ` Stefan Beller
  2018-03-28 22:24             ` René Scharfe
  2018-03-28 22:35             ` Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-03-28 21:37 UTC (permalink / raw)
  To: René Scharfe
  Cc: Eric Sunshine, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

> This looks nicer here in the script, but doesn't test exactly what users
> type most of the time, I suppose.
>
> So how about this?

Looks good to me, though I had a nagging feeling at first that the
regex could be made more concise.
Why do we need the optional "[^ ]" inside \1 ?

> +       sed -e "s/^ \([^ ]* repo\) .*/-\1/" <actual >expect &&

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 21:37           ` Stefan Beller
@ 2018-03-28 22:24             ` René Scharfe
  2018-03-28 22:35             ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: René Scharfe @ 2018-03-28 22:24 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Eric Sunshine, Git List, Junio C Hamano, jeremy, Prathamesh Chavan

Am 28.03.2018 um 23:37 schrieb Stefan Beller:
>> This looks nicer here in the script, but doesn't test exactly what users
>> type most of the time, I suppose.
>>
>> So how about this?
> 
> Looks good to me, though I had a nagging feeling at first that the
> regex could be made more concise.
> Why do we need the optional "[^ ]" inside \1 ?
> 
>> +       sed -e "s/^ \([^ ]* repo\) .*/-\1/" <actual >expect &&

[:xdigit:] would match the hash more precisely, but I don't know how
widely this character class is supported among sed implementations.
And being so strict may require changes once SHA1 is replaced -- new
hashes might be marked with a special character.  And it's longer
anyway.

\S could be used instead, but I also don't know how widely this is
supported.

. could be used as well, of course, but that feels a bit sloppy to
me.

Using separate s commands for beginning and end of the line would
be shorter.  This here is slightly longer, anyway:

	/ repo / {s/^ /-/; s/ (.*//}

What do you have in mind?  How to transform this:

 b2613938d15a18d9d4a504cacd9654fd1c879197 repo (heads/master)

... into this:

-b2613938d15a18d9d4a504cacd9654fd1c879197 repo

... while keeping other lines unchanged?

René

^ permalink raw reply	[relevance 4%]

* [PATCHv3 0/6] Moving submodules with nested submodules
  2018-03-28 17:54   ` [PATCHv2 0/6] Moving submodules with nested submodules Jonathan Tan
@ 2018-03-28 22:35     ` " Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 1/6] submodule.h: drop declaration of connect_work_tree_and_git_dir Stefan Beller
                         ` (6 more replies)
  0 siblings, 7 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

v3:
* reordered patches to have Jonathans patch before submodule_free
* addressed Jonathans comments on patch 5.
* rebased on origin/sb/object-store to resolve a visibility conflict
  about repo_init being exposed outside of repository.c

v2:
* addressed memleaks and messy code in patch 5
* removed the extern keyword where applicable
* extended the commit message, stating we want to rename submodule_free
  in the future.
* picked up Jonathans patch and added it as a nice finish of the series.
  I did not see the need or aesthetic desire to put that patch earlier
  in the series.
  
Thanks,
Stefan

v1:

This fixes the bug reported in [1] ("Bug: moving submodules that have submodules
inside them causes a fatal error in git status")

[1] https://public-inbox.org/git/20180306192017.GA5797@riseup.net/

Thanks,
Stefan
Jonathan Tan (1):
  grep: remove "repo" arg from non-supporting funcs

Stefan Beller (5):
  submodule.h: drop declaration of connect_work_tree_and_git_dir
  submodule-config: allow submodule_free to handle arbitrary
    repositories
  submodule-config: add repository argument to submodule_from_{name,
    path}
  submodule-config: remove submodule_from_cache
  submodule: fixup nested submodules after moving the submodule

 .../technical/api-submodule-config.txt        |  2 +-
 builtin/grep.c                                | 14 ++---
 builtin/mv.c                                  |  6 +-
 builtin/submodule--helper.c                   | 17 +++---
 dir.c                                         | 60 ++++++++++++++++++-
 dir.h                                         | 12 +++-
 repository.c                                  |  8 +--
 repository.h                                  |  3 +
 submodule-config.c                            | 29 ++++-----
 submodule-config.h                            | 15 +++--
 submodule.c                                   | 40 +++++++------
 submodule.h                                   |  1 -
 t/helper/test-submodule-config.c              |  8 ++-
 t/t7001-mv.sh                                 |  2 +-
 unpack-trees.c                                |  2 +-
 15 files changed, 140 insertions(+), 79 deletions(-)

-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 11%]

* [PATCHv3 1/6] submodule.h: drop declaration of connect_work_tree_and_git_dir
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 2/6] grep: remove "repo" arg from non-supporting funcs Stefan Beller
                         ` (5 subsequent siblings)
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

The function connect_work_tree_and_git_dir is declared in both submodule.h
and dir.h, such that one of them is redundant. As the function is
implemented in dir.c, drop the declaration from submodule.h

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 submodule.h | 1 -
 1 file changed, 1 deletion(-)

diff --git a/submodule.h b/submodule.h
index b9b7ef0030..b6130e6287 100644
--- a/submodule.h
+++ b/submodule.h
@@ -105,7 +105,6 @@ extern int push_unpushed_submodules(struct oid_array *commits,
 				    const char **refspec, int refspec_nr,
 				    const struct string_list *push_options,
 				    int dry_run);
-extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);
 /*
  * Given a submodule path (as in the index), return the repository
  * path of that submodule in 'buf'. Return -1 on error or when the
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 18%]

* [PATCHv3 2/6] grep: remove "repo" arg from non-supporting funcs
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 1/6] submodule.h: drop declaration of connect_work_tree_and_git_dir Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 3/6] submodule-config: allow submodule_free to handle arbitrary repositories Stefan Beller
                         ` (4 subsequent siblings)
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

From: Jonathan Tan <jonathantanmy@google.com>

As part of commit f9ee2fcdfa ("grep: recurse in-process using 'struct
repository'", 2017-08-02), many functions in builtin/grep.c were
converted to also take "struct repository *" arguments. Among them were
grep_object() and grep_objects().

However, at least grep_objects() was converted incompletely - it calls
gitmodules_config_oid(), which references the_repository.

But it turns out that the conversion was extraneous anyway - there has
been no user-visible effect - because grep_objects() is never invoked
except with the_repository. This is because grepping through objects
cannot be done recursively into submodules.

Revert the changes to grep_objects() and grep_object() (which conversion
is also extraneous) to show that both these functions do not support
repositories other than the_repository.

Signed-off-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/grep.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/builtin/grep.c b/builtin/grep.c
index 1e9cdbdf78..754eb6da3b 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -595,8 +595,7 @@ static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,
 }
 
 static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,
-		       struct object *obj, const char *name, const char *path,
-		       struct repository *repo)
+		       struct object *obj, const char *name, const char *path)
 {
 	if (obj->type == OBJ_BLOB)
 		return grep_oid(opt, &obj->oid, name, 0, path);
@@ -623,7 +622,7 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,
 		}
 		init_tree_desc(&tree, data, size);
 		hit = grep_tree(opt, pathspec, &tree, &base, base.len,
-				obj->type == OBJ_COMMIT, repo);
+				obj->type == OBJ_COMMIT, the_repository);
 		strbuf_release(&base);
 		free(data);
 		return hit;
@@ -632,7 +631,6 @@ static int grep_object(struct grep_opt *opt, const struct pathspec *pathspec,
 }
 
 static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,
-			struct repository *repo,
 			const struct object_array *list)
 {
 	unsigned int i;
@@ -648,8 +646,8 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,
 			submodule_free();
 			gitmodules_config_oid(&real_obj->oid);
 		}
-		if (grep_object(opt, pathspec, real_obj, list->objects[i].name, list->objects[i].path,
-				repo)) {
+		if (grep_object(opt, pathspec, real_obj, list->objects[i].name,
+				list->objects[i].path)) {
 			hit = 1;
 			if (opt->status_only)
 				break;
@@ -1098,7 +1096,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)
 		if (cached)
 			die(_("both --cached and trees are given."));
 
-		hit = grep_objects(&opt, &pathspec, the_repository, &list);
+		hit = grep_objects(&opt, &pathspec, &list);
 	}
 
 	if (num_threads)
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 6%]

* [PATCHv3 3/6] submodule-config: allow submodule_free to handle arbitrary repositories
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 1/6] submodule.h: drop declaration of connect_work_tree_and_git_dir Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 2/6] grep: remove "repo" arg from non-supporting funcs Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 4/6] submodule-config: add repository argument to submodule_from_{name, path} Stefan Beller
                         ` (3 subsequent siblings)
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

At some point we may want to rename the function so that it describes what
it actually does as 'submodule_free' doesn't quite describe that this
clears a repository's submodule cache.  But that's beyond the scope of
this series.

While at it remove the extern key word from its declaration.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/technical/api-submodule-config.txt | 2 +-
 builtin/grep.c                                   | 2 +-
 submodule-config.c                               | 6 +++---
 submodule-config.h                               | 2 +-
 t/helper/test-submodule-config.c                 | 2 +-
 unpack-trees.c                                   | 2 +-
 6 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/Documentation/technical/api-submodule-config.txt b/Documentation/technical/api-submodule-config.txt
index 3dce003fda..44a85bbb8b 100644
--- a/Documentation/technical/api-submodule-config.txt
+++ b/Documentation/technical/api-submodule-config.txt
@@ -38,7 +38,7 @@ Data Structures
 Functions
 ---------
 
-`void submodule_free()`::
+`void submodule_free(struct repository *r)`::
 
 	Use these to free the internally cached values.
 
diff --git a/builtin/grep.c b/builtin/grep.c
index 754eb6da3b..c1f22fb9fb 100644
--- a/builtin/grep.c
+++ b/builtin/grep.c
@@ -643,7 +643,7 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,
 
 		/* load the gitmodules file for this rev */
 		if (recurse_submodules) {
-			submodule_free();
+			submodule_free(the_repository);
 			gitmodules_config_oid(&real_obj->oid);
 		}
 		if (grep_object(opt, pathspec, real_obj, list->objects[i].name,
diff --git a/submodule-config.c b/submodule-config.c
index 2aa8a1747f..5b4f0baae8 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -642,8 +642,8 @@ const struct submodule *submodule_from_cache(struct repository *repo,
 			   key, lookup_path);
 }
 
-void submodule_free(void)
+void submodule_free(struct repository *r)
 {
-	if (the_repository->submodule_cache)
-		submodule_cache_clear(the_repository->submodule_cache);
+	if (r->submodule_cache)
+		submodule_cache_clear(r->submodule_cache);
 }
diff --git a/submodule-config.h b/submodule-config.h
index a5503a5d17..6b71a8cd30 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -46,6 +46,6 @@ extern const struct submodule *submodule_from_path(
 extern const struct submodule *submodule_from_cache(struct repository *repo,
 						    const struct object_id *treeish_name,
 						    const char *key);
-extern void submodule_free(void);
+void submodule_free(struct repository *r);
 
 #endif /* SUBMODULE_CONFIG_H */
diff --git a/t/helper/test-submodule-config.c b/t/helper/test-submodule-config.c
index f23db3b19a..9971c5e9dd 100644
--- a/t/helper/test-submodule-config.c
+++ b/t/helper/test-submodule-config.c
@@ -64,7 +64,7 @@ int cmd_main(int argc, const char **argv)
 		arg += 2;
 	}
 
-	submodule_free();
+	submodule_free(the_repository);
 
 	return 0;
 }
diff --git a/unpack-trees.c b/unpack-trees.c
index e6a15bbe44..3a6a28e794 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -290,7 +290,7 @@ static void load_gitmodules_file(struct index_state *index,
 		if (!state && ce->ce_flags & CE_WT_REMOVE) {
 			repo_read_gitmodules(the_repository);
 		} else if (state && (ce->ce_flags & CE_UPDATE)) {
-			submodule_free();
+			submodule_free(the_repository);
 			checkout_entry(ce, state, NULL);
 			repo_read_gitmodules(the_repository);
 		}
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 21%]

* [PATCHv3 6/6] submodule: fixup nested submodules after moving the submodule
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
                         ` (4 preceding siblings ...)
  2018-03-28 22:35       ` [PATCHv3 5/6] submodule-config: remove submodule_from_cache Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 23:17       ` [PATCHv3 0/6] Moving submodules with nested submodules Jonathan Tan
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

connect_work_tree_and_git_dir is used to connect a submodule worktree with
its git directory and vice versa after events that require a reconnection
such as moving around the working tree. As submodules can have nested
submodules themselves, we'd also want to fix the nested submodules when
asked to. Add an option to recurse into the nested submodules and connect
them as well.

As submodules are identified by their name (which determines their git
directory in relation to their superproject's git directory) internally
and by their path in the working tree of the superproject, we need to
make sure that the mapping of name <-> path is kept intact. We can do
that in the git-mv command by writing out the gitmodules file first
and then forcing a reload of the submodule config machinery.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/mv.c                |  6 ++--
 builtin/submodule--helper.c |  3 +-
 dir.c                       | 60 +++++++++++++++++++++++++++++++++++--
 dir.h                       | 12 +++++++-
 repository.c                |  6 ++--
 repository.h                |  3 ++
 submodule.c                 |  6 ++--
 t/t7001-mv.sh               |  2 +-
 8 files changed, 83 insertions(+), 15 deletions(-)

diff --git a/builtin/mv.c b/builtin/mv.c
index cf3684d907..b0c5178e0d 100644
--- a/builtin/mv.c
+++ b/builtin/mv.c
@@ -275,10 +275,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)
 			die_errno(_("renaming '%s' failed"), src);
 		}
 		if (submodule_gitfile[i]) {
-			if (submodule_gitfile[i] != SUBMODULE_WITH_GITDIR)
-				connect_work_tree_and_git_dir(dst, submodule_gitfile[i]);
 			if (!update_path_in_gitmodules(src, dst))
 				gitmodules_modified = 1;
+			if (submodule_gitfile[i] != SUBMODULE_WITH_GITDIR)
+				connect_work_tree_and_git_dir(dst,
+							      submodule_gitfile[i],
+							      1);
 		}
 
 		if (mode == WORKING_DIRECTORY)
diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 5551cf19c3..ffdc51f426 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -1260,8 +1260,7 @@ static int module_clone(int argc, const char **argv, const char *prefix)
 		strbuf_reset(&sb);
 	}
 
-	/* Connect module worktree and git dir */
-	connect_work_tree_and_git_dir(path, sm_gitdir);
+	connect_work_tree_and_git_dir(path, sm_gitdir, 0);
 
 	p = git_pathdup_submodule(path, "config");
 	if (!p)
diff --git a/dir.c b/dir.c
index ce6e50d2a2..4f401b6a3c 100644
--- a/dir.c
+++ b/dir.c
@@ -19,6 +19,7 @@
 #include "varint.h"
 #include "ewah/ewok.h"
 #include "fsmonitor.h"
+#include "submodule-config.h"
 
 /*
  * Tells read_directory_recursive how a file or directory should be treated.
@@ -2988,8 +2989,57 @@ void untracked_cache_add_to_index(struct index_state *istate,
 	untracked_cache_invalidate_path(istate, path);
 }
 
-/* Update gitfile and core.worktree setting to connect work tree and git dir */
-void connect_work_tree_and_git_dir(const char *work_tree_, const char *git_dir_)
+static void connect_wt_gitdir_in_nested(const char *sub_worktree,
+					const char *sub_gitdir)
+{
+	int i;
+	struct repository subrepo;
+	struct strbuf sub_wt = STRBUF_INIT;
+	struct strbuf sub_gd = STRBUF_INIT;
+
+	const struct submodule *sub;
+
+	/* If the submodule has no working tree, we can ignore it. */
+	if (repo_init(&subrepo, sub_gitdir, sub_worktree))
+		return;
+
+	if (repo_read_index(&subrepo) < 0)
+		die("index file corrupt in repo %s", subrepo.gitdir);
+
+	for (i = 0; i < subrepo.index->cache_nr; i++) {
+		const struct cache_entry *ce = subrepo.index->cache[i];
+
+		if (!S_ISGITLINK(ce->ce_mode))
+			continue;
+
+		while (i + 1 < subrepo.index->cache_nr &&
+		       !strcmp(ce->name, subrepo.index->cache[i + 1]->name))
+			/*
+			 * Skip entries with the same name in different stages
+			 * to make sure an entry is returned only once.
+			 */
+			i++;
+
+		sub = submodule_from_path(&subrepo, &null_oid, ce->name);
+		if (!sub || !is_submodule_active(&subrepo, ce->name))
+			/* .gitmodules broken or inactive sub */
+			continue;
+
+		strbuf_reset(&sub_wt);
+		strbuf_reset(&sub_gd);
+		strbuf_addf(&sub_wt, "%s/%s", sub_worktree, sub->path);
+		strbuf_addf(&sub_gd, "%s/modules/%s", sub_gitdir, sub->name);
+
+		connect_work_tree_and_git_dir(sub_wt.buf, sub_gd.buf, 1);
+	}
+	strbuf_release(&sub_wt);
+	strbuf_release(&sub_gd);
+	repo_clear(&subrepo);
+}
+
+void connect_work_tree_and_git_dir(const char *work_tree_,
+				   const char *git_dir_,
+				   int recurse_into_nested)
 {
 	struct strbuf gitfile_sb = STRBUF_INIT;
 	struct strbuf cfg_sb = STRBUF_INIT;
@@ -3019,6 +3069,10 @@ void connect_work_tree_and_git_dir(const char *work_tree_, const char *git_dir_)
 	strbuf_release(&gitfile_sb);
 	strbuf_release(&cfg_sb);
 	strbuf_release(&rel_path);
+
+	if (recurse_into_nested)
+		connect_wt_gitdir_in_nested(work_tree, git_dir);
+
 	free(work_tree);
 	free(git_dir);
 }
@@ -3032,5 +3086,5 @@ void relocate_gitdir(const char *path, const char *old_git_dir, const char *new_
 		die_errno(_("could not migrate git directory from '%s' to '%s'"),
 			old_git_dir, new_git_dir);
 
-	connect_work_tree_and_git_dir(path, new_git_dir);
+	connect_work_tree_and_git_dir(path, new_git_dir, 0);
 }
diff --git a/dir.h b/dir.h
index 11a047ba48..d2545a7685 100644
--- a/dir.h
+++ b/dir.h
@@ -359,7 +359,17 @@ struct untracked_cache *read_untracked_extension(const void *data, unsigned long
 void write_untracked_extension(struct strbuf *out, struct untracked_cache *untracked);
 void add_untracked_cache(struct index_state *istate);
 void remove_untracked_cache(struct index_state *istate);
-extern void connect_work_tree_and_git_dir(const char *work_tree, const char *git_dir);
+
+/*
+ * Connect a worktree to a git directory by creating (or overwriting) a
+ * '.git' file containing the location of the git directory. In the git
+ * directory set the core.worktree setting to indicate where the worktree is.
+ * When `recurse_into_nested` is set, recurse into any nested submodules,
+ * connecting them as well.
+ */
+extern void connect_work_tree_and_git_dir(const char *work_tree,
+					  const char *git_dir,
+					  int recurse_into_nested);
 extern void relocate_gitdir(const char *path,
 			    const char *old_git_dir,
 			    const char *new_git_dir);
diff --git a/repository.c b/repository.c
index eb5b8e9f5a..beff3caa9e 100644
--- a/repository.c
+++ b/repository.c
@@ -135,9 +135,9 @@ static int read_and_verify_repository_format(struct repository_format *format,
  * Initialize 'repo' based on the provided 'gitdir'.
  * Return 0 upon success and a non-zero value upon failure.
  */
-static int repo_init(struct repository *repo,
-		     const char *gitdir,
-		     const char *worktree)
+int repo_init(struct repository *repo,
+	      const char *gitdir,
+	      const char *worktree)
 {
 	struct repository_format format;
 	memset(repo, 0, sizeof(*repo));
diff --git a/repository.h b/repository.h
index 09df94a472..6041367f08 100644
--- a/repository.h
+++ b/repository.h
@@ -97,6 +97,9 @@ extern void repo_set_gitdir(struct repository *repo,
 extern void repo_set_worktree(struct repository *repo, const char *path);
 extern void repo_set_hash_algo(struct repository *repo, int algo);
 extern void initialize_the_repository(void);
+extern int repo_init(struct repository *r,
+		     const char *gitdir,
+		     const char *worktree);
 extern int repo_submodule_init(struct repository *submodule,
 			       struct repository *superproject,
 			       const char *path);
diff --git a/submodule.c b/submodule.c
index dac73d10a7..53c45e49d0 100644
--- a/submodule.c
+++ b/submodule.c
@@ -1625,7 +1625,7 @@ int submodule_move_head(const char *path,
 		} else {
 			char *gitdir = xstrfmt("%s/modules/%s",
 				    get_git_common_dir(), sub->name);
-			connect_work_tree_and_git_dir(path, gitdir);
+			connect_work_tree_and_git_dir(path, gitdir, 0);
 			free(gitdir);
 
 			/* make sure the index is clean as well */
@@ -1635,7 +1635,7 @@ int submodule_move_head(const char *path,
 		if (old && (flags & SUBMODULE_MOVE_HEAD_FORCE)) {
 			char *gitdir = xstrfmt("%s/modules/%s",
 				    get_git_common_dir(), sub->name);
-			connect_work_tree_and_git_dir(path, gitdir);
+			connect_work_tree_and_git_dir(path, gitdir, 1);
 			free(gitdir);
 		}
 	}
@@ -1948,7 +1948,7 @@ void absorb_git_dir_into_superproject(const char *prefix,
 		if (!sub)
 			die(_("could not lookup name for submodule '%s'"), path);
 		connect_work_tree_and_git_dir(path,
-			git_path("modules/%s", sub->name));
+			git_path("modules/%s", sub->name), 0);
 	} else {
 		/* Is it already absorbed into the superprojects git dir? */
 		char *real_sub_git_dir = real_pathdup(sub_git_dir, 1);
diff --git a/t/t7001-mv.sh b/t/t7001-mv.sh
index 6e5031f56f..bfe2c427f1 100755
--- a/t/t7001-mv.sh
+++ b/t/t7001-mv.sh
@@ -491,7 +491,7 @@ test_expect_success 'moving a submodule in nested directories' '
 	test_cmp expect actual
 '
 
-test_expect_failure 'moving nested submodules' '
+test_expect_success 'moving nested submodules' '
 	git commit -am "cleanup commit" &&
 	mkdir sub_nested_nested &&
 	(cd sub_nested_nested &&
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 21%]

* [PATCHv3 4/6] submodule-config: add repository argument to submodule_from_{name, path}
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
                         ` (2 preceding siblings ...)
  2018-03-28 22:35       ` [PATCHv3 3/6] submodule-config: allow submodule_free to handle arbitrary repositories Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 5/6] submodule-config: remove submodule_from_cache Stefan Beller
                         ` (2 subsequent siblings)
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

This enables submodule_from_{name, path} to handle arbitrary repositories.
All callers just pass in the_repository, a later patch will pass in other
repos.

While at it remove the extern key word from the declarations.

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c      | 14 +++++++-------
 submodule-config.c               | 14 ++++++++------
 submodule-config.h               | 10 ++++++----
 submodule.c                      | 30 ++++++++++++++++--------------
 t/helper/test-submodule-config.c |  6 ++++--
 5 files changed, 41 insertions(+), 33 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index 6d8e002be7..5551cf19c3 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -455,7 +455,7 @@ static void init_submodule(const char *path, const char *prefix,
 
 	displaypath = get_submodule_displaypath(path, prefix);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 
 	if (!sub)
 		die(_("No url found for submodule path '%s' in .gitmodules"),
@@ -622,7 +622,7 @@ static void status_submodule(const char *path, const struct object_id *ce_oid,
 	struct rev_info rev;
 	int diff_files_result;
 
-	if (!submodule_from_path(&null_oid, path))
+	if (!submodule_from_path(the_repository, &null_oid, path))
 		die(_("no submodule mapping found in .gitmodules for path '%s'"),
 		      path);
 
@@ -742,7 +742,7 @@ static int module_name(int argc, const char **argv, const char *prefix)
 	if (argc != 2)
 		usage(_("git submodule--helper name <path>"));
 
-	sub = submodule_from_path(&null_oid, argv[1]);
+	sub = submodule_from_path(the_repository, &null_oid, argv[1]);
 
 	if (!sub)
 		die(_("no submodule mapping found in .gitmodules for path '%s'"),
@@ -773,7 +773,7 @@ static void sync_submodule(const char *path, const char *prefix,
 	if (!is_submodule_active(the_repository, path))
 		return;
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 
 	if (sub && sub->url) {
 		if (starts_with_dot_dot_slash(sub->url) ||
@@ -926,7 +926,7 @@ static void deinit_submodule(const char *path, const char *prefix,
 	struct strbuf sb_config = STRBUF_INIT;
 	char *sub_git_dir = xstrfmt("%s/.git", path);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 
 	if (!sub || !sub->name)
 		goto cleanup;
@@ -1368,7 +1368,7 @@ static int prepare_to_clone_next_submodule(const struct cache_entry *ce,
 		goto cleanup;
 	}
 
-	sub = submodule_from_path(&null_oid, ce->name);
+	sub = submodule_from_path(the_repository, &null_oid, ce->name);
 
 	if (suc->recursive_prefix)
 		displaypath = relative_path(suc->recursive_prefix,
@@ -1651,7 +1651,7 @@ static const char *remote_submodule_branch(const char *path)
 	const char *branch = NULL;
 	char *key;
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 	if (!sub)
 		return NULL;
 
diff --git a/submodule-config.c b/submodule-config.c
index 5b4f0baae8..0ea1e927d2 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -619,18 +619,20 @@ static void gitmodules_read_check(struct repository *repo)
 		repo_read_gitmodules(repo);
 }
 
-const struct submodule *submodule_from_name(const struct object_id *treeish_name,
+const struct submodule *submodule_from_name(struct repository *r,
+					    const struct object_id *treeish_name,
 		const char *name)
 {
-	gitmodules_read_check(the_repository);
-	return config_from(the_repository->submodule_cache, treeish_name, name, lookup_name);
+	gitmodules_read_check(r);
+	return config_from(r->submodule_cache, treeish_name, name, lookup_name);
 }
 
-const struct submodule *submodule_from_path(const struct object_id *treeish_name,
+const struct submodule *submodule_from_path(struct repository *r,
+					    const struct object_id *treeish_name,
 		const char *path)
 {
-	gitmodules_read_check(the_repository);
-	return config_from(the_repository->submodule_cache, treeish_name, path, lookup_path);
+	gitmodules_read_check(r);
+	return config_from(r->submodule_cache, treeish_name, path, lookup_path);
 }
 
 const struct submodule *submodule_from_cache(struct repository *repo,
diff --git a/submodule-config.h b/submodule-config.h
index 6b71a8cd30..43dfe7dec0 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -39,10 +39,12 @@ extern int parse_update_recurse_submodules_arg(const char *opt, const char *arg)
 extern int parse_push_recurse_submodules_arg(const char *opt, const char *arg);
 extern void repo_read_gitmodules(struct repository *repo);
 extern void gitmodules_config_oid(const struct object_id *commit_oid);
-extern const struct submodule *submodule_from_name(
-		const struct object_id *commit_or_tree, const char *name);
-extern const struct submodule *submodule_from_path(
-		const struct object_id *commit_or_tree, const char *path);
+const struct submodule *submodule_from_name(struct repository *r,
+					    const struct object_id *commit_or_tree,
+					    const char *name);
+const struct submodule *submodule_from_path(struct repository *r,
+					    const struct object_id *commit_or_tree,
+					    const char *path);
 extern const struct submodule *submodule_from_cache(struct repository *repo,
 						    const struct object_id *treeish_name,
 						    const char *key);
diff --git a/submodule.c b/submodule.c
index b03e5f5045..9279cff2d7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -96,7 +96,7 @@ int update_path_in_gitmodules(const char *oldpath, const char *newpath)
 	if (is_gitmodules_unmerged(&the_index))
 		die(_("Cannot change unmerged .gitmodules, resolve merge conflicts first"));
 
-	submodule = submodule_from_path(&null_oid, oldpath);
+	submodule = submodule_from_path(the_repository, &null_oid, oldpath);
 	if (!submodule || !submodule->name) {
 		warning(_("Could not find section in .gitmodules where path=%s"), oldpath);
 		return -1;
@@ -130,7 +130,7 @@ int remove_path_from_gitmodules(const char *path)
 	if (is_gitmodules_unmerged(&the_index))
 		die(_("Cannot change unmerged .gitmodules, resolve merge conflicts first"));
 
-	submodule = submodule_from_path(&null_oid, path);
+	submodule = submodule_from_path(the_repository, &null_oid, path);
 	if (!submodule || !submodule->name) {
 		warning(_("Could not find section in .gitmodules where path=%s"), path);
 		return -1;
@@ -174,7 +174,8 @@ static int add_submodule_odb(const char *path)
 void set_diffopt_flags_from_submodule_config(struct diff_options *diffopt,
 					     const char *path)
 {
-	const struct submodule *submodule = submodule_from_path(&null_oid, path);
+	const struct submodule *submodule = submodule_from_path(the_repository,
+								&null_oid, path);
 	if (submodule) {
 		const char *ignore;
 		char *key;
@@ -674,7 +675,7 @@ const struct submodule *submodule_from_ce(const struct cache_entry *ce)
 	if (!should_update_submodules())
 		return NULL;
 
-	return submodule_from_path(&null_oid, ce->name);
+	return submodule_from_path(the_repository, &null_oid, ce->name);
 }
 
 static struct oid_array *submodule_commits(struct string_list *submodules,
@@ -731,13 +732,14 @@ static void collect_changed_submodules_cb(struct diff_queue_struct *q,
 		if (!S_ISGITLINK(p->two->mode))
 			continue;
 
-		submodule = submodule_from_path(commit_oid, p->two->path);
+		submodule = submodule_from_path(the_repository,
+						commit_oid, p->two->path);
 		if (submodule)
 			name = submodule->name;
 		else {
 			name = default_name_or_path(p->two->path);
 			/* make sure name does not collide with existing one */
-			submodule = submodule_from_name(commit_oid, name);
+			submodule = submodule_from_name(the_repository, commit_oid, name);
 			if (submodule) {
 				warning("Submodule in commit %s at path: "
 					"'%s' collides with a submodule named "
@@ -945,7 +947,7 @@ int find_unpushed_submodules(struct oid_array *commits,
 		const struct submodule *submodule;
 		const char *path = NULL;
 
-		submodule = submodule_from_name(&null_oid, name->string);
+		submodule = submodule_from_name(the_repository, &null_oid, name->string);
 		if (submodule)
 			path = submodule->path;
 		else
@@ -1113,7 +1115,7 @@ static void calculate_changed_submodule_paths(void)
 	const struct string_list_item *name;
 
 	/* No need to check if there are no submodules configured */
-	if (!submodule_from_path(NULL, NULL))
+	if (!submodule_from_path(the_repository, NULL, NULL))
 		return;
 
 	argv_array_push(&argv, "--"); /* argv[0] program name */
@@ -1134,7 +1136,7 @@ static void calculate_changed_submodule_paths(void)
 		const struct submodule *submodule;
 		const char *path = NULL;
 
-		submodule = submodule_from_name(&null_oid, name->string);
+		submodule = submodule_from_name(the_repository, &null_oid, name->string);
 		if (submodule)
 			path = submodule->path;
 		else
@@ -1162,7 +1164,7 @@ int submodule_touches_in_range(struct object_id *excl_oid,
 	int ret;
 
 	/* No need to check if there are no submodules configured */
-	if (!submodule_from_path(NULL, NULL))
+	if (!submodule_from_path(the_repository, NULL, NULL))
 		return 0;
 
 	argv_array_push(&args, "--"); /* args[0] program name */
@@ -1604,7 +1606,7 @@ int submodule_move_head(const char *path,
 	if (old && !is_submodule_populated_gently(path, error_code_ptr))
 		return 0;
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 
 	if (!sub)
 		die("BUG: could not get submodule information for '%s'", path);
@@ -1886,7 +1888,7 @@ static void relocate_single_git_dir_into_superproject(const char *prefix,
 
 	real_old_git_dir = real_pathdup(old_git_dir, 1);
 
-	sub = submodule_from_path(&null_oid, path);
+	sub = submodule_from_path(the_repository, &null_oid, path);
 	if (!sub)
 		die(_("could not lookup name for submodule '%s'"), path);
 
@@ -1942,7 +1944,7 @@ void absorb_git_dir_into_superproject(const char *prefix,
 		* superproject did not rewrite the git file links yet,
 		* fix it now.
 		*/
-		sub = submodule_from_path(&null_oid, path);
+		sub = submodule_from_path(the_repository, &null_oid, path);
 		if (!sub)
 			die(_("could not lookup name for submodule '%s'"), path);
 		connect_work_tree_and_git_dir(path,
@@ -2088,7 +2090,7 @@ int submodule_to_gitdir(struct strbuf *buf, const char *submodule)
 		strbuf_addstr(buf, git_dir);
 	}
 	if (!is_git_directory(buf->buf)) {
-		sub = submodule_from_path(&null_oid, submodule);
+		sub = submodule_from_path(the_repository, &null_oid, submodule);
 		if (!sub) {
 			ret = -1;
 			goto cleanup;
diff --git a/t/helper/test-submodule-config.c b/t/helper/test-submodule-config.c
index 9971c5e9dd..e044871cee 100644
--- a/t/helper/test-submodule-config.c
+++ b/t/helper/test-submodule-config.c
@@ -48,9 +48,11 @@ int cmd_main(int argc, const char **argv)
 			die_usage(argc, argv, "Commit not found.");
 
 		if (lookup_name) {
-			submodule = submodule_from_name(&commit_oid, path_or_name);
+			submodule = submodule_from_name(the_repository,
+							&commit_oid, path_or_name);
 		} else
-			submodule = submodule_from_path(&commit_oid, path_or_name);
+			submodule = submodule_from_path(the_repository,
+							&commit_oid, path_or_name);
 		if (!submodule)
 			die_usage(argc, argv, "Submodule not found.");
 
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 28%]

* Re: [PATCH] submodule: check for NULL return of get_submodule_ref_store()
  2018-03-28 21:37           ` Stefan Beller
  2018-03-28 22:24             ` René Scharfe
@ 2018-03-28 22:35             ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Junio C Hamano @ 2018-03-28 22:35 UTC (permalink / raw)
  To: Stefan Beller
  Cc: René Scharfe, Eric Sunshine, Git List, jeremy, Prathamesh Chavan

Stefan Beller <sbeller@google.com> writes:

>> This looks nicer here in the script, but doesn't test exactly what users
>> type most of the time, I suppose.
>>
>> So how about this?
>
> Looks good to me, though I had a nagging feeling at first that the
> regex could be made more concise.
> Why do we need the optional "[^ ]" inside \1 ?
>
>> +       sed -e "s/^ \([^ ]* repo\) .*/-\1/" <actual >expect &&

At that position there's 40-hex object name.  If we want to go
looser, you could say

	"s/^ \(.* repo\) .*/-\1/"

and if you want to go more strict, you could say

	"s/^ \($_x40 repo\) (heads\/master)$/-\1/"

I think "Here between the leading SP and SP before the pathname
'repo', we expect an object name which should be a run of non SP
bytes" is a reasonable mid-point that is stricter than "anything
goes" and is still concise.


^ permalink raw reply	[relevance 5%]

* [PATCHv3 5/6] submodule-config: remove submodule_from_cache
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
                         ` (3 preceding siblings ...)
  2018-03-28 22:35       ` [PATCHv3 4/6] submodule-config: add repository argument to submodule_from_{name, path} Stefan Beller
@ 2018-03-28 22:35       ` Stefan Beller
  2018-03-28 22:35       ` [PATCHv3 6/6] submodule: fixup nested submodules after moving the submodule Stefan Beller
  2018-03-28 23:17       ` [PATCHv3 0/6] Moving submodules with nested submodules Jonathan Tan
  6 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-03-28 22:35 UTC (permalink / raw)
  To: jonathantanmy; +Cc: bmwill, git, gitster, hvoigt, sbeller, seanwbehan

This continues the story of bf12fcdf5e (submodule-config: store
the_submodule_cache in the_repository, 2017-06-22).

The previous patch taught submodule_from_path to take a repository into
account, such that submodule_from_{path, cache} are the same now.
Remove submodule_from_cache, migrating all its callers to
submodule_from_path.

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 repository.c       | 2 +-
 submodule-config.c | 9 ---------
 submodule-config.h | 3 ---
 submodule.c        | 4 ++--
 4 files changed, 3 insertions(+), 15 deletions(-)

diff --git a/repository.c b/repository.c
index a4848c1bd0..eb5b8e9f5a 100644
--- a/repository.c
+++ b/repository.c
@@ -176,7 +176,7 @@ int repo_submodule_init(struct repository *submodule,
 	struct strbuf worktree = STRBUF_INIT;
 	int ret = 0;
 
-	sub = submodule_from_cache(superproject, &null_oid, path);
+	sub = submodule_from_path(superproject, &null_oid, path);
 	if (!sub) {
 		ret = -1;
 		goto out;
diff --git a/submodule-config.c b/submodule-config.c
index 0ea1e927d2..9abe9166d5 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -635,15 +635,6 @@ const struct submodule *submodule_from_path(struct repository *r,
 	return config_from(r->submodule_cache, treeish_name, path, lookup_path);
 }
 
-const struct submodule *submodule_from_cache(struct repository *repo,
-					     const struct object_id *treeish_name,
-					     const char *key)
-{
-	gitmodules_read_check(repo);
-	return config_from(repo->submodule_cache, treeish_name,
-			   key, lookup_path);
-}
-
 void submodule_free(struct repository *r)
 {
 	if (r->submodule_cache)
diff --git a/submodule-config.h b/submodule-config.h
index 43dfe7dec0..6f686184e8 100644
--- a/submodule-config.h
+++ b/submodule-config.h
@@ -45,9 +45,6 @@ const struct submodule *submodule_from_name(struct repository *r,
 const struct submodule *submodule_from_path(struct repository *r,
 					    const struct object_id *commit_or_tree,
 					    const char *path);
-extern const struct submodule *submodule_from_cache(struct repository *repo,
-						    const struct object_id *treeish_name,
-						    const char *key);
 void submodule_free(struct repository *r);
 
 #endif /* SUBMODULE_CONFIG_H */
diff --git a/submodule.c b/submodule.c
index 9279cff2d7..dac73d10a7 100644
--- a/submodule.c
+++ b/submodule.c
@@ -231,7 +231,7 @@ int is_submodule_active(struct repository *repo, const char *path)
 	const struct string_list *sl;
 	const struct submodule *module;
 
-	module = submodule_from_cache(repo, &null_oid, path);
+	module = submodule_from_path(repo, &null_oid, path);
 
 	/* early return if there isn't a path->module mapping */
 	if (!module)
@@ -1236,7 +1236,7 @@ static int get_next_submodule(struct child_process *cp,
 		if (!S_ISGITLINK(ce->ce_mode))
 			continue;
 
-		submodule = submodule_from_cache(spf->r, &null_oid, ce->name);
+		submodule = submodule_from_path(spf->r, &null_oid, ce->name);
 		if (!submodule) {
 			const char *name = default_name_or_path(ce->name);
 			if (name) {
-- 
2.17.0.rc1.321.gba9d0f2565-goog


^ permalink raw reply	[relevance 30%]

* Re: [PATCHv3 0/6] Moving submodules with nested submodules
  2018-03-28 22:35     ` [PATCHv3 " Stefan Beller
                         ` (5 preceding siblings ...)
  2018-03-28 22:35       ` [PATCHv3 6/6] submodule: fixup nested submodules after moving the submodule Stefan Beller
@ 2018-03-28 23:17       ` Jonathan Tan
  6 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-03-28 23:17 UTC (permalink / raw)
  To: Stefan Beller; +Cc: bmwill, git, gitster, hvoigt, seanwbehan

On Wed, 28 Mar 2018 15:35:25 -0700
Stefan Beller <sbeller@google.com> wrote:

> v3:
> * reordered patches to have Jonathans patch before submodule_free
> * addressed Jonathans comments on patch 5.
> * rebased on origin/sb/object-store to resolve a visibility conflict
>   about repo_init being exposed outside of repository.c

All 6 patches look good to me, thanks.

Reviewed-by: Jonathan Tan <jonathantanmy@google.com>

^ permalink raw reply	[relevance 5%]

* Fwd: [RFC PATCH] git-submodule.sh:cmd_update: if submodule branch exists, fetch that instead of default
      [irrelevant]   ` <CAK0XTWd7QGtVDwm8FDXejZfbgVH6-1NprGY0xxAnC33QH8aCCQ@mail.gmail.com>
@ 2018-03-29 20:54     ` Eddy Petrișor
  0 siblings, 0 replies; 200+ results
From: Eddy Petrișor @ 2018-03-29 20:54 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Git List, Heiko Voigt, Jacob Keller, Jonathan Nieder

mar., 27 mar. 2018, 02:07 Stefan Beller <sbeller@google.com> a scris:
>
> [snipped the cc list as well]
>
> On Tue, Mar 6, 2018 at 12:06 PM Eddy Petrișor <eddy.petrisor@gmail.com>
> wrote:
>
> > Signed-off-by: Eddy Petrișor <eddy.petrisor@gmail.com>
> > ---
>
> Did this go anywhere?
> (I just came back from a longer vacation, sorry for the delay on my site)


Not really. I am still unsure how is best to proceed. Details below.

> > There are projects such as llvm/clang which use several repositories, and
> they
> > might be forked for providing support for various features such as adding
> Redox
> > awareness to the toolchain. This typically means the superproject will use
> > another branch than master, occasionally even use an old commit from that
> > non-master branch.
>
> > Combined with the fact that when incorporating such a hierachy of
> repositories
> > usually the user is interested in just the exact commit specified in the
> > submodule info, it follows that a desireable usecase is to be also able to
> > provide '--depth 1' to avoid waiting for ages for the clone operation to
> > finish.
>
> Very sensible.


The only change is that I realized that hard coding the depth is not
necessary because the client can fetch more and more from the branch
until the commit hash is found or the entire history was fetched and
it wasn't found.

This is more robust but has a variable performance penalty and is
probably slower than single branch fetching from the start.

> > Git submodule seems to be very stubborn and cloning master, although the
> > wrapper script and the gitmodules-helper could work together to clone
> directly
> > the branch specified in the .gitmodules file, if specified.
>
> Also very sensible.
>
> So far so good, could you move these paragraphs before the triple dashed
> line
> and sign off so we record it as the commit message?


Sure, as long as the implementation and design makes sense.

> > Another wrinkle is that when the commit is not the tip of the branch, the
> depth
> > parameter should somehow be stored in the .gitmodules info, but any
> change in
> > the submodule will break the supermodule submodule depth info sooner or
> later,
> > which is definitly frigile.
>
> ... which is why I would not include that.
>
> git-fetch knows about --shallow-since or even better
> shallow-exclude which could be set to the (depth+1)-th commit
> (the boundary commit) recorded in the shallow information.


I am unsure what that means. Without yet looking in the docs, would
this --shallow-since be better than the try-until-found algorithm
explained above?

> > I tried digging into this section of the code and debugging with bashdb
> to see
> > where --depth might fit, but I got stuck on the shell-to-helper
> interaction and
> > the details of the submodule implementation, so I want to lay out this
> first
> > patch as starting point for the discussion in the hope somebody else
> picks it
> > up or can provide some inputs. I have the feeling there are multiple code
> paths
> > that are being ran, depending on the moment (initial clone, submodule
> > recursive, post-clone update etc.) and I have a gut feeling there
> shouldn't be
> > any code duplication just because the operation is different.
>
> > This first patch is only trying to use a non-master branch, I have some
> changes
> > for the --depth part, but I stopped working on it due to the "default
> depth"
> > issue above.
>
> > Does any of this sound reasonable?
> > Is this patch idea usable or did I managed to touch the part of the code
> that
> > should not be touched?
>
> This sounds reasonable. Thanks for writing the patch!


OK. Now I need to make it good, which is the hard part :)

> > diff --git a/git-submodule.sh b/git-submodule.sh
> > index 2491496..370f19e 100755
> > --- a/git-submodule.sh
> > +++ b/git-submodule.sh
> > @@ -589,8 +589,11 @@ cmd_update()
> >                          branch=$(git submodule--helper remote-branch
> "$sm_path")
> >                          if test -z "$nofetch"
> >                          then
> > +                               # non-default branch
> > +                               rbranch=$(git config -f .gitmodules
> submodule.$sm_path.branch)
> > +
> br_refspec=${rbanch:+"refs/heads/$rbranch:refs/heads/$rbranch"}
>
> Wouldn't we want to fetch into a remote tracking branch instead?
> Instead of computing all this by yourself, these two lines could be
>
>      br_refspec=$(git submodule--helper remote-branch $sm_path)
>
> I would think.


I wasn't aware of this, will implement I  the next version and see what happens.
>
>
> >                                  # Fetch remote before determining
> tracking $sha1
> > -                               fetch_in_submodule "$sm_path" $depth ||
> > +                               fetch_in_submodule "$sm_path" $depth
> $br_refspec ||
> >                                  die "$(eval_gettext "Unable to fetch in
> submodule path '\$sm_path'")"
> >                          fi
> >                          remote_name=$(sanitize_submodule_env; cd
> "$sm_path" && get_default_remote)
>
> It would be awesome if you could write a little test for this feature, too.
> Look for the tests in regarding --remote in t7406 (in the t/ directory) as
> a starting point, please.


Coming up with a test case is probably a better way to explain what I
want the behaviour to be. Thanks for pointing out the test case area
to look into.

> Thanks!
> Stefan

-- 
Eddy Petrișor

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] Support long format for log-based submodule diff
      [irrelevant]     ` <CAGZ79kZ_j3_mhk5asNEBgBe_2qD7=18foJgW=p0+p=uJa3U2nw@mail.gmail.com>
@ 2018-04-02  1:07       ` Robert Dailey
  2018-04-02 19:35         ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Robert Dailey @ 2018-04-02  1:07 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git, Stefan Beller, Brandon Williams

[-- Attachment #1: Type: text/plain, Size: 5704 bytes --]

On Tue, Mar 27, 2018 at 5:17 PM, Stefan Beller <sbeller@google.com> wrote:
>> >> $ git diff --submodule=log --submodule-log-detail=(long|short)
>> >>
>> >> I'm not sure what makes sense here. I welcome thoughts/discussion and
>> >> will provide follow-up patches.
>> >
>> > The case of merges is usually configured with --[no-]merges, or
>> > --min-parents=<n>.
>
>> But that is a knob that controls an irrelevant aspect of the detail
>> in the context of this discussion, isn't it?  This code is about "to
>> what degree the things that happened between two submodule commits
>> in an adjacent pair of commits in the superproject are summarized?"
>
> And I took it a step further and wanted to give a general solution, which
> allows giving any option that the diff machinery accepts to only apply
> to the submodule diffing part of the current diff.
>
>> The hack Robert illustrates below is to change it to stop favouring
>> such projects with "clean" histories, and show "log --oneline
>> --no-merges --left-right".  When presented that way, clean histories
>> of topic-branch based projects will suffer by losing conciseness,
>> but clean histories of totally linear projects will still be shown
>> the same way, and messy history that sometimes merges, sometimes
>> merges mergy histories, and sometimes directly builds on the trunk
>> will be shown as an enumeration of individual commits in a flat way
>> by ignoring merges and not restricting the traversal to the first
>> parent chains, which would appear more uniform than what the current
>> code shows.
>
> Oh, I realize this is in the *summary* code path, I was thinking about the
> show_submodule_inline_diff, which would benefit from more diff options.
>
>> I do not see a point in introducing --min/max-parents as a knob to
>> control how the history is summarized.
>
> For a summary a flat list of commits may be fine, ignoring
> (ideally non-evil) merges.
>
>> This is a strongly related tangent, but I wonder if we can and/or
>> want to share more code with the codepath that prepares the log
>> message for a merge.  It summarizes what happened on the side branch
>> since it forked from the history it is joining back to (I think it
>> is merge.c::shortlog() that computes this)
>
> I do not find code there. To me it looks like builtin/fmt-merge-msg.c
> is responsible for coming up with a default merge message?
> In that file there is a shortlog() function, which walks revisions
> and puts together the subject lines of commits.
>
>> and it is quite similar
>> to what Robert wants to use for submodules here.  On the other hand,
>> in a project _without_ submodule, if you are pulling history made by
>> your lieutenant whose history is full of linear merges of topic
>> branches to the mainline, it may not be a bad idea to allow
>> fmt-merge-msg to alternatively show something similar to the "diff
>> --submodule=log" gives us, i.e. summarize the history of the side
>> branch being merged by just listing the commits on the first-parent
>> chain.  So I sense some opportunity for cross pollination here.
>
> The cross pollination that I sense is the desire in both cases to freely
> specify the format as it may depend on the workflow.

First I want to apologize for having taken so long to get back with
each of you about this. I actually have a lot of work started to
expand the --submodule option to add a "full-log" option in addition
to the existing "log". This is a pretty big task for me already,
mostly because I'm unfamiliar with git and have limited personal time
to do this at home (this is part of what I am apologizing for). I kind
of get what Stefan and Junio are saying. There's a lot of opportunity
for cleanup. More specific to my use case, adding some functionality
to generate a log message (although I've developed a bash script to do
this since I wrote my original email. I'll attach it to this email for
those interested). Also I get that taking this a notch higher and
adding a new option to pass options down to submodules also addresses
my case. Before I waste anyone's time on this, I want to make sure
that my very narrow and specific implementation will be ideal. By all
means I do not want to do things the easy way which ends up adding
"cruft" you'll have to deal with later. If there's a larger effort to
generalize this and other things related to submodules maybe I can
just wait for that to happen instead? What direction would you guys
recommend?

Junio basically hit the nail on the head with the comparisons of
different mainlines. I think some repositories are more disciplined
than others. At my workplace, I deal with a lot of folks that aren't
interested in learning git beyond the required day to day
responsibilities. It's difficult to enforce very specific branching,
rebase, and merge habits. As such, the best I can do to work around
that for building release notes is to exclude merge commits (since
most of the time, people keep the default message which is generally
useless) and include all commits in the ancestry path (since often
times commits on the right side of a merge will have important
information such as JIRA issue keys, which if shown in the parent repo
will cause appropriate links back to parent repositories to show when
changes in submodules were introduced there as well).

Based on how constructive this email thread has gotten since I started
it, I'm starting to feel like my solution is too narrowly-focused and
doesn't have the long term appeal expected. Let me know, I'm happy to
do what I can but I think it will be limited due to my lack of domain
expertise in the code base and inability to invest the required time
for significant scope of work.

[-- Attachment #2: git-smcommit --]
[-- Type: application/octet-stream, Size: 1609 bytes --]

^ permalink raw reply	[relevance 6%]

* [ANNOUNCE] Git v2.17.0
@ 2018-04-02 19:34 Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-04-02 19:34 UTC (permalink / raw)
  To: git; +Cc: Linux Kernel, git-packagers

The latest feature release Git v2.17.0 is now available at the
usual places.  It is comprised of 516 non-merge commits since
v2.16.0, contributed by 71 people, 20 of which are new faces.

The tarballs are found at:

    https://www.kernel.org/pub/software/scm/git/

The following public repositories all have a copy of the 'v2.17.0'
tag and the 'master' branch that the tag points at:

  url = https://kernel.googlesource.com/pub/scm/git/git
  url = git://repo.or.cz/alt-git.git
  url = https://github.com/gitster/git

New contributors whose contributions weren't in v2.16.0 are as follows.
Welcome to the Git development community!

  Adam Borowski, Alban Gruin, Andreas G. Schacker, Bernhard
  M. Wiedemann, Christian Ludwig, Christopher Diaz Riveros,
  Gargi Sharma, Genki Sky, Gregory Herrero, Jon Simons, Juan
  F. Codagnone, Kim Gybels, Lucas Werkmeister, Mathias Rav,
  Michele Locati, Motoki Seki, Stefan Moch, Stephen R Guglielmo,
  Tatyana Krasnukha, and Thomas Levesque.

Returning contributors who helped this release are as follows.
Thanks for your continued support.

  Ævar Arnfjörð Bjarmason, Alexander Shopov, Alex Bennée,
  Ben Peart, Brandon Williams, brian m. carlson, Changwoo Ryu,
  Christian Couder, Daniel Knittl-Frank, David Pursehouse,
  Derrick Stolee, Elijah Newren, Eric Sunshine, Eric Wong, Jason
  Merrill, Jean-Noël Avila, Jeff Hostetler, Jeff King, Jiang Xin,
  Johannes Schindelin, Jonathan Nieder, Jonathan Tan, Jordi Mas,
  Junio C Hamano, Kaartic Sivaraam, Mårten Kongstad, Martin
  Ågren, Matthieu Moy, Michael Haggerty, Nathan Payre, Nguyễn
  Thái Ngọc Duy, Nicolas Morey-Chaisemartin, Olga Telezhnaya,
  Patryk Obara, Peter Krefting, Phillip Wood, Prathamesh Chavan,
  Ralf Thielow, Ramsay Jones, Randall S. Becker, Rasmus Villemoes,
  Ray Chen, René Scharfe, Robert P. J. Day, Stefan Beller, SZEDER
  Gábor, Thomas Gummerer, Todd Zullinger, Torsten Bögershausen,
  Trần Ngọc Quân, and Yasushi SHOJI.

----------------------------------------------------------------

Git 2.17 Release Notes
======================

Updates since v2.16
-------------------

UI, Workflows & Features

 * "diff" family of commands learned "--find-object=<object-id>" option
   to limit the findings to changes that involve the named object.

 * "git format-patch" learned to give 72-cols to diffstat, which is
   consistent with other line length limits the subcommand uses for
   its output meant for e-mails.

 * The log from "git daemon" can be redirected with a new option; one
   relevant use case is to send the log to standard error (instead of
   syslog) when running it from inetd.

 * "git rebase" learned to take "--allow-empty-message" option.

 * "git am" has learned the "--quit" option, in addition to the
   existing "--abort" option; having the pair mirrors a few other
   commands like "rebase" and "cherry-pick".

 * "git worktree add" learned to run the post-checkout hook, just like
   "git clone" runs it upon the initial checkout.

 * "git tag" learned an explicit "--edit" option that allows the
   message given via "-m" and "-F" to be further edited.

 * "git fetch --prune-tags" may be used as a handy short-hand for
   getting rid of stale tags that are locally held.

 * The new "--show-current-patch" option gives an end-user facing way
   to get the diff being applied when "git rebase" (and "git am")
   stops with a conflict.

 * "git add -p" used to offer "/" (look for a matching hunk) as a
   choice, even there was only one hunk, which has been corrected.
   Also the single-key help is now given only for keys that are
   enabled (e.g. help for '/' won't be shown when there is only one
   hunk).

 * Since Git 1.7.9, "git merge" defaulted to --no-ff (i.e. even when
   the side branch being merged is a descendant of the current commit,
   create a merge commit instead of fast-forwarding) when merging a
   tag object.  This was appropriate default for integrators who pull
   signed tags from their downstream contributors, but caused an
   unnecessary merges when used by downstream contributors who
   habitually "catch up" their topic branches with tagged releases
   from the upstream.  Update "git merge" to default to --no-ff only
   when merging a tag object that does *not* sit at its usual place in
   refs/tags/ hierarchy, and allow fast-forwarding otherwise, to
   mitigate the problem.

 * "git status" can spend a lot of cycles to compute the relation
   between the current branch and its upstream, which can now be
   disabled with "--no-ahead-behind" option.

 * "git diff" and friends learned funcname patterns for Go language
   source files.

 * "git send-email" learned "--reply-to=<address>" option.

 * Funcname pattern used for C# now recognizes "async" keyword.

 * In a way similar to how "git tag" learned to honor the pager
   setting only in the list mode, "git config" learned to ignore the
   pager setting when it is used for setting values (i.e. when the
   purpose of the operation is not to "show").


Performance, Internal Implementation, Development Support etc.

 * More perf tests for threaded grep

 * "perf" test output can be sent to codespeed server.

 * The build procedure for perl/ part has been greatly simplified by
   weaning ourselves off of MakeMaker.

 * Perl 5.8 or greater has been required since Git 1.7.4 released in
   2010, but we continued to assume some core modules may not exist and
   used a conditional "eval { require <<module>> }"; we no longer do
   this.  Some platforms (Fedora/RedHat/CentOS, for example) ship Perl
   without all core modules by default (e.g. Digest::MD5, File::Temp,
   File::Spec, Net::Domain, Net::SMTP).  Users on such platforms may
   need to install these additional modules.

 * As a convenience, we install copies of Perl modules we require which
   are not part of the core Perl distribution (e.g. Error and
   Mail::Address).  Users and packagers whose operating system provides
   these modules can set NO_PERL_CPAN_FALLBACKS to avoid installing the
   bundled modules.

 * In preparation for implementing narrow/partial clone, the machinery
   for checking object connectivity used by gc and fsck has been
   taught that a missing object is OK when it is referenced by a
   packfile specially marked as coming from trusted repository that
   promises to make them available on-demand and lazily.

 * The machinery to clone & fetch, which in turn involves packing and
   unpacking objects, has been told how to omit certain objects using
   the filtering mechanism introduced by another topic.  It now knows
   to mark the resulting pack as a promisor pack to tolerate missing
   objects, laying foundation for "narrow" clones.

 * The first step to getting rid of mru API and using the
   doubly-linked list API directly instead.

 * Retire mru API as it does not give enough abstraction over
   underlying list API to be worth it.

 * Rewrite two more "git submodule" subcommands in C.

 * The tracing machinery learned to report tweaking of environment
   variables as well.

 * Update Coccinelle rules to catch and optimize strbuf_addf(&buf, "%s", str)

 * Prevent "clang-format" from breaking line after function return type.

 * The sequencer infrastructure is shared across "git cherry-pick",
   "git rebase -i", etc., and has always spawned "git commit" when it
   needs to create a commit.  It has been taught to do so internally,
   when able, by reusing the codepath "git commit" itself uses, which
   gives performance boost for a few tens of percents in some sample
   scenarios.

 * Push the submodule version of collision-detecting SHA-1 hash
   implementation a bit harder on builders.

 * Avoid mmapping small files while using packed refs (especially ones
   with zero size, which would cause later munmap() to fail).

 * Conversion from uchar[20] to struct object_id continues.

 * More tests for wildmatch functions.

 * The code to binary search starting from a fan-out table (which is
   how the packfile is indexed with object names) has been refactored
   into a reusable helper.

 * We now avoid using identifiers that clash with C++ keywords.  Even
   though it is not a goal to compile Git with C++ compilers, changes
   like this help use of code analysis tools that targets C++ on our
   codebase.

 * The executable is now built in 'script' phase in Travis CI integration,
   to follow the established practice, rather than during 'before_script'
   phase.  This allows the CI categorize the failures better ('failed'
   is project's fault, 'errored' is build environment's).
   (merge 3c93b82920 sg/travis-build-during-script-phase later to maint).

 * Writing out the index file when the only thing that changed in it
   is the untracked cache information is often wasteful, and this has
   been optimized out.

 * Various pieces of Perl code we have have been cleaned up.

 * Internal API clean-up to allow write_locked_index() optionally skip
   writing the in-core index when it is not modified.


Also contains various documentation updates and code clean-ups.


Fixes since v2.16
-----------------

 * An old regression in "git describe --all $annotated_tag^0" has been
   fixed.

 * "git status" after moving a path in the working tree (hence making
   it appear "removed") and then adding with the -N option (hence
   making that appear "added") detected it as a rename, but did not
   report the  old and new pathnames correctly.

 * "git svn dcommit" did not take into account the fact that a
   svn+ssh:// URL with a username@ (typically used for pushing) refers
   to the same SVN repository without the username@ and failed when
   svn.pushmergeinfo option is set.

 * API clean-up around revision traversal.

 * "git merge -Xours/-Xtheirs" learned to use our/their version when
   resolving a conflicting updates to a symbolic link.

 * "git clone $there $here" is allowed even when here directory exists
   as long as it is an empty directory, but the command incorrectly
   removed it upon a failure of the operation.

 * "git commit --fixup" did not allow "-m<message>" option to be used
   at the same time; allow it to annotate resulting commit with more
   text.

 * When resetting the working tree files recursively, the working tree
   of submodules are now also reset to match.

 * "git stash -- <pathspec>" incorrectly blew away untracked files in
   the directory that matched the pathspec, which has been corrected.

 * Instead of maintaining home-grown email address parsing code, ship
   a copy of reasonably recent Mail::Address to be used as a fallback
   in 'git send-email' when the platform lacks it.
   (merge d60be8acab mm/send-email-fallback-to-local-mail-address later to maint).

 * "git add -p" was taught to ignore local changes to submodules as
   they do not interfere with the partial addition of regular changes
   anyway.

 * Avoid showing a warning message in the middle of a line of "git
   diff" output.
   (merge 4e056c989f nd/diff-flush-before-warning later to maint).

 * The http tracing code, often used to debug connection issues,
   learned to redact potentially sensitive information from its output
   so that it can be more safely sharable.
   (merge 8ba18e6fa4 jt/http-redact-cookies later to maint).

 * Crash fix for a corner case where an error codepath tried to unlock
   what it did not acquire lock on.
   (merge 81fcb698e0 mr/packed-ref-store-fix later to maint).

 * The split-index mode had a few corner case bugs fixed.
   (merge ae59a4e44f tg/split-index-fixes later to maint).

 * Assorted fixes to "git daemon".
   (merge ed15e58efe jk/daemon-fixes later to maint).

 * Completion of "git merge -s<strategy>" (in contrib/) did not work
   well in non-C locale.
   (merge 7cc763aaa3 nd/list-merge-strategy later to maint).

 * Workaround for segfault with more recent versions of SVN.
   (merge 7f6f75e97a ew/svn-branch-segfault-fix later to maint).

 * Plug recently introduced leaks in fsck.
   (merge ba3a08ca0e jt/fsck-code-cleanup later to maint).

 * "git pull --rebase" did not pass verbosity setting down when
   recursing into a submodule.
   (merge a56771a668 sb/pull-rebase-submodule later to maint).

 * The way "git reset --hard" reports the commit the updated HEAD
   points at is made consistent with the way how the commit title is
   generated by the other parts of the system.  This matters when the
   title is spread across physically multiple lines.
   (merge 1cf823fb68 tg/reset-hard-show-head-with-pretty later to maint).

 * Test fixes.
   (merge 63b1a175ee sg/test-i18ngrep later to maint).

 * Some bugs around "untracked cache" feature have been fixed.  This
   will notice corrupt data in the untracked cache left by old and
   buggy code and issue a warning---the index can be fixed by clearing
   the untracked cache from it.
   (merge 0cacebf099 nd/fix-untracked-cache-invalidation later to maint).
   (merge 7bf0be7501 ab/untracked-cache-invalidation-docs later to maint).

 * "git blame HEAD COPYING" in a bare repository failed to run, while
   "git blame HEAD -- COPYING" run just fine.  This has been corrected.

 * "git add" files in the same directory, but spelling the directory
   path in different cases on case insensitive filesystem, corrupted
   the name hash data structure and led to unexpected results.  This
   has been corrected.
   (merge c95525e90d bp/name-hash-dirname-fix later to maint).

 * "git rebase -p" mangled log messages of a merge commit, which is
   now fixed.
   (merge ed5144d7eb js/fix-merge-arg-quoting-in-rebase-p later to maint).

 * Some low level protocol codepath could crash when they get an
   unexpected flush packet, which is now fixed.
   (merge bb1356dc64 js/packet-read-line-check-null later to maint).

 * "git check-ignore" with multiple paths got confused when one is a
   file and the other is a directory, which has been fixed.
   (merge d60771e930 rs/check-ignore-multi later to maint).

 * "git describe $garbage" stopped giving any errors when the garbage
   happens to be a string with 40 hexadecimal letters.
   (merge a8e7a2bf0f sb/describe-blob later to maint).

 * Code to unquote single-quoted string (used in the parser for
   configuration files, etc.) did not diagnose bogus input correctly
   and produced bogus results instead.
   (merge ddbbf8eb25 jk/sq-dequote-on-bogus-input later to maint).

 * Many places in "git apply" knew that "/dev/null" that signals
   "there is no such file on this side of the diff" can be followed by
   whitespace and garbage when parsing a patch, except for one, which
   made an otherwise valid patch (e.g. ones from subversion) rejected.
   (merge e454ad4bec tk/apply-dev-null-verify-name-fix later to maint).

 * We no longer create any *.spec file, so "make clean" should not
   remove it.
   (merge 4321bdcabb tz/do-not-clean-spec-file later to maint).

 * "git push" over http transport did not unquote the push-options
   correctly.
   (merge 90dce21eb0 jk/push-options-via-transport-fix later to maint).

 * "git send-email" learned to complain when the batch-size option is
   not defined when the relogin-delay option is, since these two are
   mutually required.
   (merge 9caa70697b xz/send-email-batch-size later to maint).

 * Y2k20 fix ;-) for our perl scripts.
   (merge a40e06ee33 bw/perl-timegm-timelocal-fix later to maint).

 * Threaded "git grep" has been optimized to avoid allocation in code
   section that is covered under a mutex.
   (merge 38ef24dccf rv/grep-cleanup later to maint).

 * "git subtree" script (in contrib/) scripted around "git log", whose
   output got affected by end-user configuration like log.showsignature
   (merge 8841b5222c sg/subtree-signed-commits later to maint).

 * While finding unique object name abbreviation, the code may
   accidentally have read beyond the end of the array of object names
   in a pack.
   (merge 21abed500c ds/find-unique-abbrev-optim later to maint).

 * Micro optimization in revision traversal code.
   (merge ebbed3ba04 ds/mark-parents-uninteresting-optim later to maint).

 * "git commit" used to run "gc --auto" near the end, which was lost
   when the command was reimplemented in C by mistake.
   (merge 095c741edd ab/gc-auto-in-commit later to maint).

 * Allow running a couple of tests with "sh -x".
   (merge c20bf94abc sg/cvs-tests-with-x later to maint).

 * The codepath to replace an existing entry in the index had a bug in
   updating the name hash structure, which has been fixed.
   (merge 0e267b7a24 bp/refresh-cache-ent-rehash-fix later to maint).

 * The transfer.fsckobjects configuration tells "git fetch" to
   validate the data and connected-ness of objects in the received
   pack; the code to perform this check has been taught about the
   narrow clone's convention that missing objects that are reachable
   from objects in a pack that came from a promissor remote is OK.

 * There was an unused file-scope static variable left in http.c when
   building for versions of libCURL that is older than 7.19.4, which
   has been fixed.
   (merge b8fd6008ec rj/http-code-cleanup later to maint).

 * Shell script portability fix.
   (merge 206a6ae013 ml/filter-branch-portability-fix later to maint).

 * Other minor doc, test and build updates and code cleanups.
   (merge e2a5a028c7 bw/oidmap-autoinit later to maint).
   (merge ec3b4b06f8 cl/t9001-cleanup later to maint).
   (merge e1b3f3dd38 ks/submodule-doc-updates later to maint).
   (merge fbac558a9b rs/describe-unique-abbrev later to maint).
   (merge 8462ff43e4 tb/crlf-conv-flags later to maint).
   (merge 7d68bb0766 rb/hashmap-h-compilation-fix later to maint).
   (merge 3449847168 cc/sha1-file-name later to maint).
   (merge ad622a256f ds/use-get-be64 later to maint).
   (merge f919ffebed sg/cocci-move-array later to maint).
   (merge 4e801463c7 jc/mailinfo-cleanup-fix later to maint).
   (merge ef5b3a6c5e nd/shared-index-fix later to maint).
   (merge 9f5258cbb8 tz/doc-show-defaults-to-head later to maint).
   (merge b780e4407d jc/worktree-add-short-help later to maint).
   (merge ae239fc8e5 rs/cocci-strbuf-addf-to-addstr later to maint).
   (merge 2e22a85e5c nd/ignore-glob-doc-update later to maint).
   (merge 3738031581 jk/gettext-poison later to maint).
   (merge 54360a1956 rj/sparse-updates later to maint).
   (merge 12e31a6b12 sg/doc-test-must-fail-args later to maint).
   (merge 760f1ad101 bc/doc-interpret-trailers-grammofix later to maint).
   (merge 4ccf461f56 bp/fsmonitor later to maint).
   (merge a6119f82b1 jk/test-hashmap-updates later to maint).
   (merge 5aea9fe6cc rd/typofix later to maint).
   (merge e4e5da2796 sb/status-doc-fix later to maint).
   (merge 7976e901c8 gs/test-unset-xdg-cache-home later to maint).
   (merge d023df1ee6 tg/worktree-create-tracking later to maint).
   (merge 4cbe92fd41 sm/mv-dry-run-update later to maint).
   (merge 75e5e9c3f7 sb/color-h-cleanup later to maint).
   (merge 2708ef4af6 sg/t6300-modernize later to maint).
   (merge d88e92d4e0 bw/doc-submodule-recurse-config-with-clone later to maint).
   (merge f74bbc8dd2 jk/cached-commit-buffer later to maint).
   (merge 1316416903 ms/non-ascii-ticks later to maint).
   (merge 878056005e rs/strbuf-read-file-or-whine later to maint).
   (merge 79f0ba1547 jk/strbuf-read-file-close-error later to maint).
   (merge edfb8ba068 ot/ref-filter-cleanup later to maint).
   (merge 11395a3b4b jc/test-must-be-empty later to maint).
   (merge 768b9d6db7 mk/doc-pretty-fill later to maint).
   (merge 2caa7b8d27 ab/man-sec-list later to maint).
   (merge 40c17eb184 ks/t3200-typofix later to maint).
   (merge bd9958c358 dp/merge-strategy-doc-fix later to maint).
   (merge 9ee0540a40 js/ming-strftime later to maint).
   (merge 1775e990f7 tz/complete-tag-delete-tagname later to maint).
   (merge 00a4b03501 rj/warning-uninitialized-fix later to maint).
   (merge b635ed97a0 jk/attributes-path-doc later to maint).

^ permalink raw reply	[relevance 2%]

* Re: [PATCH] Support long format for log-based submodule diff
  2018-04-02  1:07       ` [PATCH] Support long format for log-based submodule diff Robert Dailey
@ 2018-04-02 19:35         ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-02 19:35 UTC (permalink / raw)
  To: Robert Dailey; +Cc: Junio C Hamano, git, Stefan Beller, Brandon Williams

On Sun, Apr 1, 2018 at 6:07 PM, Robert Dailey <rcdailey.lists@gmail.com> wrote:
> On Tue, Mar 27, 2018 at 5:17 PM, Stefan Beller <sbeller@google.com> wrote:
>>> >> $ git diff --submodule=log --submodule-log-detail=(long|short)
>>> >>
>>> >> I'm not sure what makes sense here. I welcome thoughts/discussion and
>>> >> will provide follow-up patches.
>>> >
>>> > The case of merges is usually configured with --[no-]merges, or
>>> > --min-parents=<n>.
>>
>>> But that is a knob that controls an irrelevant aspect of the detail
>>> in the context of this discussion, isn't it?  This code is about "to
>>> what degree the things that happened between two submodule commits
>>> in an adjacent pair of commits in the superproject are summarized?"
>>
>> And I took it a step further and wanted to give a general solution, which
>> allows giving any option that the diff machinery accepts to only apply
>> to the submodule diffing part of the current diff.
>>
>>> The hack Robert illustrates below is to change it to stop favouring
>>> such projects with "clean" histories, and show "log --oneline
>>> --no-merges --left-right".  When presented that way, clean histories
>>> of topic-branch based projects will suffer by losing conciseness,
>>> but clean histories of totally linear projects will still be shown
>>> the same way, and messy history that sometimes merges, sometimes
>>> merges mergy histories, and sometimes directly builds on the trunk
>>> will be shown as an enumeration of individual commits in a flat way
>>> by ignoring merges and not restricting the traversal to the first
>>> parent chains, which would appear more uniform than what the current
>>> code shows.
>>
>> Oh, I realize this is in the *summary* code path, I was thinking about the
>> show_submodule_inline_diff, which would benefit from more diff options.
>>
>>> I do not see a point in introducing --min/max-parents as a knob to
>>> control how the history is summarized.
>>
>> For a summary a flat list of commits may be fine, ignoring
>> (ideally non-evil) merges.
>>
>>> This is a strongly related tangent, but I wonder if we can and/or
>>> want to share more code with the codepath that prepares the log
>>> message for a merge.  It summarizes what happened on the side branch
>>> since it forked from the history it is joining back to (I think it
>>> is merge.c::shortlog() that computes this)
>>
>> I do not find code there. To me it looks like builtin/fmt-merge-msg.c
>> is responsible for coming up with a default merge message?
>> In that file there is a shortlog() function, which walks revisions
>> and puts together the subject lines of commits.
>>
>>> and it is quite similar
>>> to what Robert wants to use for submodules here.  On the other hand,
>>> in a project _without_ submodule, if you are pulling history made by
>>> your lieutenant whose history is full of linear merges of topic
>>> branches to the mainline, it may not be a bad idea to allow
>>> fmt-merge-msg to alternatively show something similar to the "diff
>>> --submodule=log" gives us, i.e. summarize the history of the side
>>> branch being merged by just listing the commits on the first-parent
>>> chain.  So I sense some opportunity for cross pollination here.
>>
>> The cross pollination that I sense is the desire in both cases to freely
>> specify the format as it may depend on the workflow.
>
> First I want to apologize for having taken so long to get back with
> each of you about this. I actually have a lot of work started to
> expand the --submodule option to add a "full-log" option in addition
> to the existing "log". This is a pretty big task for me already,
> mostly because I'm unfamiliar with git and have limited personal time
> to do this at home (this is part of what I am apologizing for).

No worries wrt. time.

> I kind
> of get what Stefan and Junio are saying. There's a lot of opportunity
> for cleanup. More specific to my use case, adding some functionality
> to generate a log message (although I've developed a bash script to do
> this since I wrote my original email. I'll attach it to this email for
> those interested).

The functionality looks very similar what Gerrit does in its
"superproject subscription mode", which would update the submodules in
the superproject automatically, when you submit on the submodule.
For example [1] is an update of the Gerrit project itself, that has some
submodules. This commit only updates the replication plugin, but
provides a summary what happened in that plugin.

[1] https://gerrit.googlesource.com/gerrit/+/db20af7123221b0b2f01d1f06e4eaac32a04cef6


I wonder if there is need for this in upstream git as well, e.g.
"git submodule update --remote" would also want to have a
switch "--commit-with-proposed-commit-message" or if the
standard commit message template would provide a submodule
summary for you. I realize that there is the config option
status.submoduleSummary already, but it is not as clear as either
your script or the Gerrit example.

> Also I get that taking this a notch higher and
> adding a new option to pass options down to submodules also addresses
> my case. Before I waste anyone's time on this, I want to make sure
> that my very narrow and specific implementation will be ideal. By all
> means I do not want to do things the easy way which ends up adding
> "cruft" you'll have to deal with later.

Sounds good. I am undecided whether to count this as cruft, as it brings in
real improvements for certain histories. And if you need this, it is not cruft
but a feature.

> If there's a larger effort to
> generalize this and other things related to submodules maybe I can
> just wait for that to happen instead? What direction would you guys
> recommend?

You could do that, though there are no timelines and you'd wait quite a
long time, potentially.

> Junio basically hit the nail on the head with the comparisons of
> different mainlines. I think some repositories are more disciplined
> than others. At my workplace, I deal with a lot of folks that aren't
> interested in learning git beyond the required day to day
> responsibilities. It's difficult to enforce very specific branching,
> rebase, and merge habits. As such, the best I can do to work around
> that for building release notes is to exclude merge commits (since
> most of the time, people keep the default message which is generally
> useless) and include all commits in the ancestry path (since often
> times commits on the right side of a merge will have important
> information such as JIRA issue keys, which if shown in the parent repo
> will cause appropriate links back to parent repositories to show when
> changes in submodules were introduced there as well).

It sounds it is fixing a real need, so don't call it cruft. ;)

> Based on how constructive this email thread has gotten since I started
> it, I'm starting to feel like my solution is too narrowly-focused and
> doesn't have the long term appeal expected. Let me know, I'm happy to
> do what I can but I think it will be limited due to my lack of domain
> expertise in the code base and inability to invest the required time
> for significant scope of work.

I guess we can have it as

  $ git diff --submodule=long-log

for now? Or instead "detailed-log" or "log-with-commits" ?

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* [PATCH 6/7] diff.c: decouple white space treatment for move detection from generic option
      [irrelevant] <20180402224854.86922-1-sbeller@google.com>
@ 2018-04-02 22:48 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-02 22:48 UTC (permalink / raw)
  To: jonathantanmy, jacob.keller, simon; +Cc: git, Stefan Beller

In the original implementation of the move detection logic we assumed that
the choice for ignoring white space changes is the same for the move
detection as it is for the generic diff.  It turns out this is wrong.

There are a couple of things where the user wants to input their
decision into the diff machinery:

* Whether or not to ignore white space for the general diff detection.
  This is covered with the -w, -b options already.
* Whether the move detection ought to pay attention to white spaces
  in general. And if so, how are white spaces handled for the block
  detection.

One example would be reviewing a current patch under discussion, that moves
code around.  In that case, you may want to have the general diff machinery
not ignore the white spaces (i.e. exact matching), as that will show you
the diff as the patch sent. The code moved however may have another
indentation level, such that you want to ignore white spaces for the move
detection. The code may be in python, such that spaces matter for program
flow, though. That is why you'd want each block to have the same change
in white space.

This patch decouples white space treatment in the general diff machinery
from the white space treatment in the move detection.

This adds all the the options for ignoring white space that the regular
diff machinery has to the move detection, such that we can cover all the
cases that were introduced in 7a55427094 (Merge branch
'jk/diff-color-moved-fix', 2017-11-06).

The example from above (different lines in the diff with -w for the regular
diff) is covered in a test. Convert the existing tests to be more explicit
on their choice of white space behavior in the move detection as the tests
should not fail if the default is changed.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/diff-options.txt |  13 +++
 diff.c                         |  17 +++-
 t/t4015-diff-whitespace.sh     | 150 +++++++++++++++++++++++++++++++--
 3 files changed, 171 insertions(+), 9 deletions(-)

diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index e3a44f03cd..4ca09c1977 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -288,6 +288,19 @@ dimmed_zebra::
 	blocks are considered interesting, the rest is uninteresting.
 --
 
+--color-moved-[no-]ignore-all-space::
+	Ignore whitespace when comparing lines when performing the move
+	detection for --color-moved.  This ignores differences even if
+	one line has whitespace where the other line has none.
+--color-moved-[no-]ignore-space-change::
+	Ignore changes in amount of whitespace when performing the move
+	detection for --color-moved.  This ignores whitespace
+	at line end, and considers all other sequences of one or
+	more whitespace characters to be equivalent.
+--color-moved-[no-]ignore-space-at-eol::
+	Ignore changes in whitespace at EOL when performing the move
+	detection for --color-moved.
+
 --word-diff[=<mode>]::
 	Show a word diff, using the <mode> to delimit changed words.
 	By default, words are delimited by whitespace; see
diff --git a/diff.c b/diff.c
index 2052a43f7a..5fe2930dca 100644
--- a/diff.c
+++ b/diff.c
@@ -720,7 +720,7 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,
 
 	return !xdiff_compare_lines(a->es->line, a->es->len,
 				    b->es->line, b->es->len,
-				    diffopt->xdl_opts);
+				    diffopt->color_moved & XDF_WHITESPACE_FLAGS);
 }
 
 static struct moved_entry *prepare_entry(struct diff_options *o,
@@ -728,8 +728,9 @@ static struct moved_entry *prepare_entry(struct diff_options *o,
 {
 	struct moved_entry *ret = xmalloc(sizeof(*ret));
 	struct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];
+	unsigned flags = o->color_moved & XDF_WHITESPACE_FLAGS;
 
-	ret->ent.hash = xdiff_hash_string(l->line, l->len, o->xdl_opts);
+	ret->ent.hash = xdiff_hash_string(l->line, l->len, flags);
 	ret->es = l;
 	ret->next_line = NULL;
 
@@ -4638,6 +4639,18 @@ int diff_opt_parse(struct diff_options *options,
 		DIFF_XDL_SET(options, IGNORE_CR_AT_EOL);
 	else if (!strcmp(arg, "--ignore-blank-lines"))
 		DIFF_XDL_SET(options, IGNORE_BLANK_LINES);
+	else if (!strcmp(arg, "--color-moved-no-ignore-all-space"))
+		options->color_moved &= ~XDF_IGNORE_WHITESPACE;
+	else if (!strcmp(arg, "--color-moved-no-ignore-space-change"))
+		options->color_moved &= ~XDF_IGNORE_WHITESPACE_CHANGE;
+	else if (!strcmp(arg, "--color-moved-no-ignore-space-at-eol"))
+		options->color_moved &= ~XDF_IGNORE_WHITESPACE_AT_EOL;
+	else if (!strcmp(arg, "--color-moved-ignore-all-space"))
+		options->color_moved |= XDF_IGNORE_WHITESPACE;
+	else if (!strcmp(arg, "--color-moved-ignore-space-change"))
+		options->color_moved |= XDF_IGNORE_WHITESPACE_CHANGE;
+	else if (!strcmp(arg, "--color-moved-ignore-space-at-eol"))
+		options->color_moved |= XDF_IGNORE_WHITESPACE_AT_EOL;
 	else if (!strcmp(arg, "--indent-heuristic"))
 		DIFF_XDL_SET(options, INDENT_HEURISTIC);
 	else if (!strcmp(arg, "--no-indent-heuristic"))
diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
index 17df491a3a..38aaf4c46c 100755
--- a/t/t4015-diff-whitespace.sh
+++ b/t/t4015-diff-whitespace.sh
@@ -1395,7 +1395,10 @@ test_expect_success 'move detection ignoring whitespace ' '
 	line 4
 	line 5
 	EOF
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1419,7 +1422,10 @@ test_expect_success 'move detection ignoring whitespace ' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames -w --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1459,7 +1465,10 @@ test_expect_success 'move detection ignoring whitespace changes' '
 	line 5
 	EOF
 
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1483,7 +1492,10 @@ test_expect_success 'move detection ignoring whitespace changes' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames -b --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-at-eol \
+		--color-moved-ignore-space-change |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1526,7 +1538,10 @@ test_expect_success 'move detection ignoring whitespace at eol' '
 	# avoid cluttering the output with complaints about our eol whitespace
 	test_config core.whitespace -blank-at-eol &&
 
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1550,7 +1565,10 @@ test_expect_success 'move detection ignoring whitespace at eol' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames --ignore-space-at-eol --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1722,7 +1740,125 @@ test_expect_success 'move detection with submodules' '
 
 	# nor did we mess with it another way
 	git diff --submodule=diff --color | test_decode_color >expect &&
-	test_cmp expect decoded_actual
+	test_cmp expect decoded_actual &&
+	rm -rf bananas &&
+	git submodule deinit bananas
+'
+
+test_expect_success 'move detection only ignores white spaces' '
+	git reset --hard &&
+	q_to_tab <<-\EOF >function.c &&
+	int func()
+	{
+	Qif (foo) {
+	QQ// this part of the function
+	QQ// function will be very long
+	QQ// indeed. We must exceed both
+	QQ// per-line and number of line
+	QQ// minimums
+	QQ;
+	Q}
+	Qbaz();
+	Qbar();
+	Q// more unrelated stuff
+	}
+	EOF
+	git add function.c &&
+	git commit -m "add function.c" &&
+	q_to_tab <<-\EOF >function.c &&
+	int do_foo()
+	{
+	Q// this part of the function
+	Q// function will be very long
+	Q// indeed. We must exceed both
+	Q// per-line and number of line
+	Q// minimums
+	Q;
+	}
+
+	int func()
+	{
+	Qif (foo)
+	QQdo_foo();
+	Qbaz();
+	Qbar();
+	Q// more unrelated stuff
+	}
+	EOF
+
+	# Make sure we get a different diff using -w ("moved function header")
+	git diff --color --color-moved -w \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
+		grep -v "index" |
+		test_decode_color >actual &&
+	q_to_tab <<-\EOF >expected &&
+	<BOLD>diff --git a/function.c b/function.c<RESET>
+	<BOLD>--- a/function.c<RESET>
+	<BOLD>+++ b/function.c<RESET>
+	<CYAN>@@ -1,6 +1,5 @@<RESET>
+	<BOLD;MAGENTA>-int func()<RESET>
+	<GREEN>+<RESET><GREEN>int do_foo()<RESET>
+	 {<RESET>
+	<RED>-	if (foo) {<RESET>
+	 Q// this part of the function<RESET>
+	 Q// function will be very long<RESET>
+	 Q// indeed. We must exceed both<RESET>
+	<CYAN>@@ -8,6 +7,11 @@<RESET> <RESET>int func()<RESET>
+	 Q// minimums<RESET>
+	 Q;<RESET>
+	 }<RESET>
+	<GREEN>+<RESET>
+	<BOLD;CYAN>+<RESET><BOLD;CYAN>int func()<RESET>
+	<GREEN>+<RESET><GREEN>{<RESET>
+	<GREEN>+<RESET>Q<GREEN>if (foo)<RESET>
+	<GREEN>+<RESET>QQ<GREEN>do_foo();<RESET>
+	 Qbaz();<RESET>
+	 Qbar();<RESET>
+	 Q// more unrelated stuff<RESET>
+	EOF
+	test_cmp expected actual &&
+
+	# And now ignoring white space only in the move detection
+	git diff --color --color-moved \
+		--color-moved-ignore-all-space \
+		--color-moved-ignore-space-change \
+		--color-moved-ignore-space-at-eol |
+		grep -v "index" |
+		test_decode_color >actual &&
+	q_to_tab <<-\EOF >expected &&
+	<BOLD>diff --git a/function.c b/function.c<RESET>
+	<BOLD>--- a/function.c<RESET>
+	<BOLD>+++ b/function.c<RESET>
+	<CYAN>@@ -1,13 +1,17 @@<RESET>
+	<GREEN>+<RESET><GREEN>int do_foo()<RESET>
+	<GREEN>+<RESET><GREEN>{<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>// this part of the function<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>// function will be very long<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>// indeed. We must exceed both<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>// per-line and number of line<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>// minimums<RESET>
+	<BOLD;CYAN>+<RESET>Q<BOLD;CYAN>;<RESET>
+	<BOLD;CYAN>+<RESET><BOLD;CYAN>}<RESET>
+	<GREEN>+<RESET>
+	 int func()<RESET>
+	 {<RESET>
+	<RED>-Qif (foo) {<RESET>
+	<BOLD;MAGENTA>-QQ// this part of the function<RESET>
+	<BOLD;MAGENTA>-QQ// function will be very long<RESET>
+	<BOLD;MAGENTA>-QQ// indeed. We must exceed both<RESET>
+	<BOLD;MAGENTA>-QQ// per-line and number of line<RESET>
+	<BOLD;MAGENTA>-QQ// minimums<RESET>
+	<BOLD;MAGENTA>-QQ;<RESET>
+	<BOLD;MAGENTA>-Q}<RESET>
+	<GREEN>+<RESET>Q<GREEN>if (foo)<RESET>
+	<GREEN>+<RESET>QQ<GREEN>do_foo();<RESET>
+	 Qbaz();<RESET>
+	 Qbar();<RESET>
+	 Q// more unrelated stuff<RESET>
+	EOF
+	test_cmp expected actual
 '
 
 test_done
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 6%]

* Re: [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule
      [irrelevant]     ` <CAK0XTWf=ubDK-4=FknLSFObZrShg=7UDgR5c6iNSjrwiDngayA@mail.gmail.com>
@ 2018-04-04 18:36       ` Stefan Beller
  2018-04-04 20:37         ` Eddy Petrișor
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-04 18:36 UTC (permalink / raw)
  To: Eddy Petrișor; +Cc: Eddy Petrișor, Jonathan Nieder, Git List

On Tue, Apr 3, 2018 at 3:26 PM, Eddy Petrișor <eddy.petrisor@gmail.com> wrote:
> Notes:
>     I am aware this test is not probably the best one and maybe I
> should have first one test that does a one level non-default, before
> trying a test with 2 levels of submodules, but I wanted to express the
> goal of the patch.

This patch only contains the test, I presume this goes on top of
https://public-inbox.org/git/20180403222053.23132-1-eddy.petrisor@codeaurora.org/
which you plan to later submit as one patch including both the change as well as
the test.

>     Currently the test fails, so I am obviously missing something.
> Help would be appreciated.
>
>
> 2018-04-04 1:20 GMT+03:00 Eddy Petrișor <eddy.petrisor@codeaurora.org>:
>> From: Eddy Petrișor <eddy.petrisor@gmail.com>
>>
>> If a submodule uses a non-default branch and the branch info is versioned, on
>> submodule update --recursive --init the correct branch should be checked out.
>>
>> Signed-off-by: Eddy Petrișor <eddy.petrisor@gmail.com>
>> ---
>>  t/t7406-submodule-update.sh | 54 +++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 54 insertions(+)
>>
>> diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
>> index 6f083c4d6..7b65f1dd1 100755
>> --- a/t/t7406-submodule-update.sh
>> +++ b/t/t7406-submodule-update.sh
>> @@ -259,6 +259,60 @@ test_expect_success 'submodule update --remote should fetch upstream changes wit
>>         )
>>  '
>>
>> +test_expect_success 'submodule update --remote --recursive --init should fetch module branch from .gitmodules' '
>> +       git clone . super5 &&

I found adding "test_pause &&"
to be a great helper as it will drop you in a shell where
you can inspect the repository.


>> +       git clone super5 submodl2b2 &&
>> +       git clone super5 submodl1b1 &&

We may want to cleanup after the test is done:

  test_when_finished "rm submodl2b2" &&
  test_when_finished "rm submodl1b2" &&

>> +       cd submodl2b2 &&

I'd think the test suite prefers subshells to operate in other dirs
instead of direct cd's, just like at the end of the test.
For short parts, we make heavy use of the -C option,
So for example the code below
       (
               cd super &&
               git submodule update --recursive --init
       ) &&

can be written as

    git -C super submodule update --recursive --init

which is shorter.

>> +       echo linel2b2 > l2b2 &&

style: The Git test suite prefers to have the redirect adjacent to the
file name:
  echo hello >world

>> +       git checkout -b b2 &&
>> +       git add l2b2 &&
>> +       test_tick &&
>> +       git commit -m "commit on b2 branch in l2" &&
>> +       git rev-parse --verify HEAD >../expectl2 &&

So until now we made a commit in a submodule on branch b2
and wrote it out to an expect file.


>> +       git checkout master &&
>> +       cd ../submodl1b1 &&
>> +       git checkout -b b1 &&
>> +       echo linel1b1 > l1b1 &&
>> +       git add l1b1 &&
>> +       test_tick &&
>> +       git commit -m "commit on b1 branch in l1" &&

very similar to above, just in another repo
instead of making a commit yourself, you may want to use
    test_commit <name>
then you don't need to call echo/add/commit yourself.

>> +       git submodule add ../submodl2b2 submodl2b2 &&
>> +       git config -f .gitmodules submodule."submodl2b2".branch b2 &&
>> +       git add .gitmodules &&
>> +       test_tick &&
>> +       git commit -m "add l2 module with branch b2 in l1 module in branch b1" &&

So one submodule is made to be a submodule of the other
with a specific branch (b2)

>> +       git submodule init submodl2b2 &&

git submodule add ought to have initialized that submodule
already, I am not sure we need the explicit init here.

>> +       git rev-parse --verify HEAD >../expectl1 &&
>> +       git checkout master &&

We go back to master, which doesn't have the nested submodule?

>> +       cd ../super5 &&
>> +       echo super_with_2_chained_modules > super5 &&
>> +       git add super5 &&
>> +       test_tick &&
>> +       git commit -m "commit on default branch in super5" &&
>> +       git submodule add ../submodl1b1 submodl1b1 &&
>> +       git config -f .gitmodules submodule."submodl1b1".branch b1 &&
>> +       git add .gitmodules &&
>> +       test_tick &&
>> +       git commit -m "add l1 module with branch b1 in super5" &&

now we do this again without the nested submodule, just one repo
as a submodule?

>> +       git submodule init submodl1b1 &&
>> +       git clone super5 super &&

does super exist here already? (I did not check, but IIRC
super and super{1-4} are there as we count upwards to
find a name that is ok.

>> +       (
>> +               cd super &&
>> +               git submodule update --recursive --init
>> +       ) &&
>> +       (
>> +               cd submodl1b1 &&
>> +               git rev-parse --verify HEAD >../../actuall1 &&
>> +               test_cmp ../../expectl1 ../../actuall1
>> +       ) &&
>> +       (
>> +               cd submodl2b2 &&
>> +               git rev-parse --verify HEAD >../../../actuall2 &&
>> +               test_cmp ../../../expectl2 ../../../actuall2
>> +       )

Stefan

^ permalink raw reply	[relevance 7%]

* Re: [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule
  2018-04-04 18:36       ` [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule Stefan Beller
@ 2018-04-04 20:37         ` Eddy Petrișor
  2018-04-04 21:41           ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Eddy Petrișor @ 2018-04-04 20:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Eddy Petrișor, Jonathan Nieder, Git List

2018-04-04 21:36 GMT+03:00 Stefan Beller <sbeller@google.com>:
> On Tue, Apr 3, 2018 at 3:26 PM, Eddy Petrișor <eddy.petrisor@gmail.com> wrote:
>> Notes:
>>     I am aware this test is not probably the best one and maybe I
>> should have first one test that does a one level non-default, before
>> trying a test with 2 levels of submodules, but I wanted to express the
>> goal of the patch.
>
> This patch only contains the test, I presume this goes on top of
> https://public-inbox.org/git/20180403222053.23132-1-eddy.petrisor@codeaurora.org/
> which you plan to later submit as one patch including both the change as well as
> the test.

Yes, not sure why the emails were not linked together, I specified the
in-reply-to mesage id when I "git format-patch"-ed the patches.

I wasn't actually planning to squash them in a single patch, will do,
but I guess for now it helps to focus the discussion around the test
since the implementation is still lacking, see below 2 lines comment.

>>     Currently the test fails, so I am obviously missing something.
>> Help would be appreciated.
>>
>>
>> 2018-04-04 1:20 GMT+03:00 Eddy Petrișor <eddy.petrisor@codeaurora.org>:
>>> From: Eddy Petrișor <eddy.petrisor@gmail.com>
[..]
>>> --- a/t/t7406-submodule-update.sh
>>> +++ b/t/t7406-submodule-update.sh
>>> @@ -259,6 +259,60 @@ test_expect_success 'submodule update --remote should fetch upstream changes wit
>>>         )
>>>  '
>>>
>>> +test_expect_success 'submodule update --remote --recursive --init should fetch module branch from .gitmodules' '
>>> +       git clone . super5 &&
>
> I found adding "test_pause &&"
> to be a great helper as it will drop you in a shell where
> you can inspect the repository.

Thanks for the pointer, I only looked at the post-failure state after
adding --debug -i --verbose, but having "test_pause" to stop and
inspect the interim stage should help a lot with debugging.

>
>>> +       git clone super5 submodl2b2 &&
>>> +       git clone super5 submodl1b1 &&
>
> We may want to cleanup after the test is done:
>
>   test_when_finished "rm submodl2b2" &&
>   test_when_finished "rm submodl1b2" &&

Sure, will do.

>>> +       cd submodl2b2 &&
>
> I'd think the test suite prefers subshells to operate in other dirs
> instead of direct cd's, just like at the end of the test.

After looking at other tests I was under the impression that the setup
phase (e.g. creating the "server" side repos) of the test was done in
the main context, while the test case (i.e. the client side, or what
the user would do) is in subshells.

> For short parts, we make heavy use of the -C option,
> So for example the code below
>        (
>                cd super &&
>                git submodule update --recursive --init
>        ) &&
>
> can be written as
>
>     git -C super submodule update --recursive --init
>
> which is shorter.

Nice.

>>> +       echo linel2b2 > l2b2 &&
>
> style: The Git test suite prefers to have the redirect adjacent to the
> file name:
>   echo hello >world

I wasn't aware of that, it seems there are some inconsistencies,
including in the modified test:

eddy@feodora:~/usr/src/git/t$ grep '> ' -c t* 2>/dev/null | grep -v
':0$' | grep 7406
t7406-submodule-update.sh:24
eddy@feodora:~/usr/src/git/t$ grep '> ' -c t* 2>/dev/null | grep -v
':0$' | wc -l
325

I suspect that a minor patch correcting these inconsistencies would be
faily fast reviewed adn accepted, of course, disconnected from this
modification.

>>> +       git checkout -b b2 &&
>>> +       git add l2b2 &&
>>> +       test_tick &&
>>> +       git commit -m "commit on b2 branch in l2" &&
>>> +       git rev-parse --verify HEAD >../expectl2 &&
>
> So until now we made a commit in a submodule on branch b2
> and wrote it out to an expect file.

Yes, I was trying to replicate the redox-os case which has this structure:

redox-os (master branch)
   + rust (@some commit on a non-default)
          + src/llvm (@some commit on a non-default branch)

This section is making sure the b2 branch has other content than the
default one and setting the expectation for level2 submodule branch,
later.

>>> +       git checkout master &&
>>> +       cd ../submodl1b1 &&
>>> +       git checkout -b b1 &&
>>> +       echo linel1b1 > l1b1 &&
>>> +       git add l1b1 &&
>>> +       test_tick &&
>>> +       git commit -m "commit on b1 branch in l1" &&
>
> very similar to above, just in another repo
> instead of making a commit yourself, you may want to use
>     test_commit <name>
> then you don't need to call echo/add/commit yourself.

thanks for the pointer, will change that since my purpose for the
commit was to make sure default branch (master) and b1 branch are
different

>>> +       git submodule add ../submodl2b2 submodl2b2 &&
>>> +       git config -f .gitmodules submodule."submodl2b2".branch b2 &&
>>> +       git add .gitmodules &&
>>> +       test_tick &&
>>> +       git commit -m "add l2 module with branch b2 in l1 module in branch b1" &&
>
> So one submodule is made to be a submodule of the other
> with a specific branch (b2)

correct

>>> +       git submodule init submodl2b2 &&
>
> git submodule add ought to have initialized that submodule
> already, I am not sure we need the explicit init here.

I think that from my tests I saw that without it the submodule wasn't
there, but I might be mistaken, will check.

>>> +       git rev-parse --verify HEAD >../expectl1 &&
>>> +       git checkout master &&
>
> We go back to master, which doesn't have the nested submodule?

exactly, since the desired behaviour is to have in the nested
submodule the b2 branch.
I guess I could have added the level2 submodule@master on l1's master
branch, but I didn't see much point in that since it should be enough
now.

It might make more sense if master and b2, and, respectively master
and l1 have diverging histories, but for now that is probably overkill
and it will make sense in the context of shallow cloning of
submodules, which I haven't yet tackled except presenting the idea.

>>> +       cd ../super5 &&
>>> +       echo super_with_2_chained_modules > super5 &&
>>> +       git add super5 &&
>>> +       test_tick &&
>>> +       git commit -m "commit on default branch in super5" &&
>>> +       git submodule add ../submodl1b1 submodl1b1 &&
>>> +       git config -f .gitmodules submodule."submodl1b1".branch b1 &&
>>> +       git add .gitmodules &&
>>> +       test_tick &&
>>> +       git commit -m "add l1 module with branch b1 in super5" &&
>
> now we do this again without the nested submodule, just one repo
> as a submodule?

My intention was to have super5 -> submodl1b1@b1 -> submodl2b2@b2 on
the "server" side.
But are you saying I just implemented super5 -> sbmodl1b1@master due
to the previous master checkout in submodl1b1?

>>> +       git submodule init submodl1b1 &&
>>> +       git clone super5 super &&
>
> does super exist here already? (I did not check, but IIRC
> super and super{1-4} are there as we count upwards to
> find a name that is ok.

I created it in the first step of the test with the intention to have
super5 as the "server" and "super" as the client clone.

>
>>> +       (
>>> +               cd super &&
>>> +               git submodule update --recursive --init
>>> +       ) &&
>>> +       (
>>> +               cd submodl1b1 &&
>>> +               git rev-parse --verify HEAD >../../actuall1 &&
>>> +               test_cmp ../../expectl1 ../../actuall1
>>> +       ) &&
>>> +       (
>>> +               cd submodl2b2 &&
>>> +               git rev-parse --verify HEAD >../../../actuall2 &&
>>> +               test_cmp ../../../expectl2 ../../../actuall2
>>> +       )

As a general idea for a test, does it look sane?

Do you think I should I start with a just one level of submodule with
a non-default branch (super -> l1@b1), or it this OK?
In my view, having 2 levels makes sure the recursive part is also
addressed and verified.

-- 
Eddy Petrișor

^ permalink raw reply	[relevance 6%]

* Re: [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule
  2018-04-04 20:37         ` Eddy Petrișor
@ 2018-04-04 21:41           ` Stefan Beller
  2018-04-18 22:35             ` [RFC PATCH v4 3/9] fixup:t7406: use test_commit instead of echo/add/commit as suggested by Stefan Beller Eddy Petrișor
                               ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Stefan Beller @ 2018-04-04 21:41 UTC (permalink / raw)
  To: Eddy Petrișor; +Cc: Eddy Petrișor, Jonathan Nieder, Git List

On Wed, Apr 4, 2018 at 1:37 PM, Eddy Petrișor <eddy.petrisor@gmail.com> wrote:

>> This patch only contains the test, I presume this goes on top of
>> https://public-inbox.org/git/20180403222053.23132-1-eddy.petrisor@codeaurora.org/
>> which you plan to later submit as one patch including both the change as well as
>> the test.
>
> Yes, not sure why the emails were not linked together, I specified the
> in-reply-to mesage id when I "git format-patch"-ed the patches.

They are linked in public-inbox, but I noticed too late. (in gmail
they were not)

> Thanks for the pointer, I only looked at the post-failure state after
> adding --debug -i --verbose, but having "test_pause" to stop and
> inspect the interim stage should help a lot with debugging.

I also like the -x flag that stops at the failing test, but as this is the
last test of this script it is of limited use here.

> After looking at other tests I was under the impression that the setup
> phase (e.g. creating the "server" side repos) of the test was done in
> the main context, while the test case (i.e. the client side, or what
> the user would do) is in subshells.

This roughly coincides, as the client is usually in a new clone. :)

>> style: The Git test suite prefers to have the redirect adjacent to the
>> file name:
>>   echo hello >world
>
> I wasn't aware of that, it seems there are some inconsistencies,
> including in the modified test:
>
> eddy@feodora:~/usr/src/git/t$ grep '> ' -c t* 2>/dev/null | grep -v
> ':0$' | grep 7406
> t7406-submodule-update.sh:24
> eddy@feodora:~/usr/src/git/t$ grep '> ' -c t* 2>/dev/null | grep -v
> ':0$' | wc -l
> 325

Thanks for digging up numbers!

> I suspect that a minor patch correcting these inconsistencies would be
> faily fast reviewed adn accepted, of course, disconnected from this
> modification.

There are two schools of thought, one of them is to fix things up
whenever you see fit. I am very sympathetic to this one.

The other is found in Documentation/CodingGuidelines:

 - Fixing style violations while working on a real change as a
   preparatory clean-up step is good, but otherwise avoid useless code
   churn for the sake of conforming to the style.

   "Once it _is_ in the tree, it's not really worth the patch noise to
   go and fix it up."
   Cf. http://lkml.iu.edu/hypermail/linux/kernel/1001.3/01069.html

> Yes, I was trying to replicate the redox-os case which has this structure:
>
> redox-os (master branch)
>    + rust (@some commit on a non-default)
>           + src/llvm (@some commit on a non-default branch)
>
> This section is making sure the b2 branch has other content than the
> default one and setting the expectation for level2 submodule branch,
> later.

Oh, cool!

>
>>>> +       git rev-parse --verify HEAD >../expectl1 &&
>>>> +       git checkout master &&
>>
>> We go back to master, which doesn't have the nested submodule?
>
> exactly, since the desired behaviour is to have in the nested
> submodule the b2 branch.

Well if the nested submodule doesn't exist at HEAD, then git should
not change branches in there, only on checking out/ updating to a state
that has the nested sub? (I may missunderstand things here)

> I guess I could have added the level2 submodule@master on l1's master
> branch, but I didn't see much point in that since it should be enough
> now.
>
> It might make more sense if master and b2, and, respectively master
> and l1 have diverging histories, but for now that is probably overkill
> and it will make sense in the context of shallow cloning of
> submodules, which I haven't yet tackled except presenting the idea.

ok.

>
>>>> +       cd ../super5 &&
>>>> +       echo super_with_2_chained_modules > super5 &&
>>>> +       git add super5 &&
>>>> +       test_tick &&
>>>> +       git commit -m "commit on default branch in super5" &&
>>>> +       git submodule add ../submodl1b1 submodl1b1 &&
>>>> +       git config -f .gitmodules submodule."submodl1b1".branch b1 &&
>>>> +       git add .gitmodules &&
>>>> +       test_tick &&
>>>> +       git commit -m "add l1 module with branch b1 in super5" &&
>>
>> now we do this again without the nested submodule, just one repo
>> as a submodule?
>
> My intention was to have super5 -> submodl1b1@b1 -> submodl2b2@b2 on
> the "server" side.
> But are you saying I just implemented super5 -> sbmodl1b1@master due
> to the previous master checkout in submodl1b1?

No. I was a little confused about the code.

>
>>>> +       git submodule init submodl1b1 &&
>>>> +       git clone super5 super &&
>>
>> does super exist here already? (I did not check, but IIRC
>> super and super{1-4} are there as we count upwards to
>> find a name that is ok.
>
> I created it in the first step of the test with the intention to have
> super5 as the "server" and "super" as the client clone.

oh, ok.

>
>>
>>>> +       (
>>>> +               cd super &&
>>>> +               git submodule update --recursive --init
>>>> +       ) &&
>>>> +       (
>>>> +               cd submodl1b1 &&
>>>> +               git rev-parse --verify HEAD >../../actuall1 &&
>>>> +               test_cmp ../../expectl1 ../../actuall1
>>>> +       ) &&
>>>> +       (
>>>> +               cd submodl2b2 &&
>>>> +               git rev-parse --verify HEAD >../../../actuall2 &&
>>>> +               test_cmp ../../../expectl2 ../../../actuall2
>>>> +       )
>
> As a general idea for a test, does it look sane?

Yes, I think it is a sane approach. Thanks for writing such a test!

> Do you think I should I start with a just one level of submodule with
> a non-default branch (super -> l1@b1), or it this OK?
> In my view, having 2 levels makes sure the recursive part is also
> addressed and verified.

I totally agree.

Stefan

^ permalink raw reply	[relevance 6%]

* [PATCH 13/19] refs: store the main ref store inside the repository struct
      [irrelevant] <20180406232136.253950-1-sbeller@google.com>
  2018-04-06 23:21 ` [PATCH 05/19] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
  2018-04-06 23:21 ` [PATCH 06/19] refs: add repository argument to get_main_ref_store Stefan Beller
@ 2018-04-06 23:21 ` Stefan Beller
  2018-04-07  9:50 ` [RFC PATCH 00/19] object-store refactoring 3 (replace objects, main ref store) Duy Nguyen
      [irrelevant] ` <20180409224533.17764-1-sbeller@google.com>
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-06 23:21 UTC (permalink / raw)
  To: jonathantanmy; +Cc: git, Stefan Beller

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 refs.c       | 13 +++++--------
 refs.h       |  4 +---
 repository.h |  5 +++++
 3 files changed, 11 insertions(+), 11 deletions(-)

diff --git a/refs.c b/refs.c
index 0352cbdcc1..793b827e19 100644
--- a/refs.c
+++ b/refs.c
@@ -1608,9 +1608,6 @@ static struct ref_store_hash_entry *alloc_ref_store_hash_entry(
 	return entry;
 }
 
-/* A pointer to the ref_store for the main repository: */
-static struct ref_store *main_ref_store;
-
 /* A hashmap of ref_stores, stored by submodule name: */
 static struct hashmap submodule_ref_stores;
 
@@ -1652,13 +1649,13 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store_the_repository(void)
+struct ref_store *get_main_ref_store(struct repository *r)
 {
-	if (main_ref_store)
-		return main_ref_store;
+	if (r->main_ref_store)
+		return r->main_ref_store;
 
-	main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
-	return main_ref_store;
+	r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
+	return r->main_ref_store;
 }
 
 /*
diff --git a/refs.h b/refs.h
index ab3d2bec2f..f5ab68c0ed 100644
--- a/refs.h
+++ b/refs.h
@@ -760,9 +760,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-#define get_main_ref_store(r) \
-	get_main_ref_store_##r()
-struct ref_store *get_main_ref_store_the_repository(void);
+struct ref_store *get_main_ref_store(struct repository *r);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/repository.h b/repository.h
index 09df94a472..2922d3a28b 100644
--- a/repository.h
+++ b/repository.h
@@ -26,6 +26,11 @@ struct repository {
 	 */
 	struct raw_object_store *objects;
 
+	/*
+	 * The store in which the refs are hold.
+	 */
+	struct ref_store *main_ref_store;
+
 	/*
 	 * Path to the repository's graft file.
 	 * Cannot be NULL after initialization.
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 9%]

* [PATCH 05/19] replace-object: check_replace_refs is safe in multi repo environment
      [irrelevant] <20180406232136.253950-1-sbeller@google.com>
@ 2018-04-06 23:21 ` Stefan Beller
  2018-04-06 23:21 ` [PATCH 06/19] refs: add repository argument to get_main_ref_store Stefan Beller
                   ` (3 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-06 23:21 UTC (permalink / raw)
  To: jonathantanmy; +Cc: git, Stefan Beller

In e1111cef23 (inline lookup_replace_object() calls, 2011-05-15) a shortcut
for checking the object replacement was added by setting check_replace_refs
to 0 once the replacements were evaluated to not exist. This works fine in
with the assumption of only one repository in existence.

The assumption won't hold true any more when we work on multiple instances
of a repository structs (e.g. one struct per submodule), as the first
repository to be inspected may have no replacements and would set the
global variable. Other repositories would then completely omit their
evaluation of replacements.

This reverts back the meaning of the flag `check_replace_refs` of
"Do we need to check with the lookup table?" to "Do we need to read
the replacement definition?", adding the bypassing logic to
lookup_replace_object after the replacement definition was read.
As with the original patch, delay the renaming of the global variable

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 environment.c    | 2 +-
 replace-object.c | 2 --
 replace-object.h | 5 ++++-
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/environment.c b/environment.c
index 93c9fbb0ba..29f677c650 100644
--- a/environment.c
+++ b/environment.c
@@ -50,7 +50,7 @@ const char *editor_program;
 const char *askpass_program;
 const char *excludes_file;
 enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;
-int check_replace_refs = 1;
+int check_replace_refs = 1; /* NEEDSWORK: rename to read_replace_refs */
 char *git_replace_ref_base;
 enum eol core_eol = EOL_UNSET;
 int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;
diff --git a/replace-object.c b/replace-object.c
index 603d11e931..c6d08c2e17 100644
--- a/replace-object.c
+++ b/replace-object.c
@@ -78,8 +78,6 @@ static void prepare_replace_object(void)
 
 	for_each_replace_ref(register_replace_ref, NULL);
 	the_repository->objects->replacements.prepared = 1;
-	if (!the_repository->objects->replacements.nr)
-		check_replace_refs = 0;
 }
 
 /* We allow "recursive" replacement. Only within reason, though */
diff --git a/replace-object.h b/replace-object.h
index 1e3e8805b9..f2555cddb9 100644
--- a/replace-object.h
+++ b/replace-object.h
@@ -3,6 +3,7 @@
 
 #include "cache.h"
 #include "repository.h"
+#include "object-store.h"
 
 struct replace_object {
 	unsigned char original[GIT_MAX_RAWSZ];
@@ -23,7 +24,9 @@ extern const unsigned char *do_lookup_replace_object(const unsigned char *sha1);
  */
 static inline const unsigned char *lookup_replace_object(const unsigned char *sha1)
 {
-	if (!check_replace_refs)
+	if (!check_replace_refs ||
+	    (the_repository->objects->replacements.prepared &&
+	     the_repository->objects->replacements.nr == 0))
 		return sha1;
 	return do_lookup_replace_object(sha1);
 }
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 6%]

* [PATCH 06/19] refs: add repository argument to get_main_ref_store
      [irrelevant] <20180406232136.253950-1-sbeller@google.com>
  2018-04-06 23:21 ` [PATCH 05/19] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
@ 2018-04-06 23:21 ` Stefan Beller
  2018-04-06 23:21 ` [PATCH 13/19] refs: store the main ref store inside the repository struct Stefan Beller
                   ` (2 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-06 23:21 UTC (permalink / raw)
  To: jonathantanmy; +Cc: git, Stefan Beller, Jonathan Nieder

Add a repository argument to allow the get_main_ref_store caller
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>

# Conflicts:
#	t/helper/test-ref-store.c
---
 builtin/pack-refs.c       |  3 +-
 refs.c                    | 67 ++++++++++++++++++++-------------------
 refs.h                    |  4 ++-
 revision.c                |  5 +--
 t/helper/test-ref-store.c |  3 +-
 5 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c
index b106a392a4..f3353564f9 100644
--- a/builtin/pack-refs.c
+++ b/builtin/pack-refs.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "refs.h"
+#include "repository.h"
 
 static char const * const pack_refs_usage[] = {
 	N_("git pack-refs [<options>]"),
@@ -17,5 +18,5 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 	};
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
-	return refs_pack_refs(get_main_ref_store(), flags);
+	return refs_pack_refs(get_main_ref_store(the_repository), flags);
 }
diff --git a/refs.c b/refs.c
index 20ba82b434..addc9d4934 100644
--- a/refs.c
+++ b/refs.c
@@ -13,6 +13,7 @@
 #include "tag.h"
 #include "submodule.h"
 #include "worktree.h"
+#include "repository.h"
 
 /*
  * List of all available backends
@@ -206,7 +207,7 @@ char *refs_resolve_refdup(struct ref_store *refs,
 char *resolve_refdup(const char *refname, int resolve_flags,
 		     struct object_id *oid, int *flags)
 {
-	return refs_resolve_refdup(get_main_ref_store(),
+	return refs_resolve_refdup(get_main_ref_store(the_repository),
 				   refname, resolve_flags,
 				   oid, flags);
 }
@@ -228,7 +229,7 @@ int refs_read_ref_full(struct ref_store *refs, const char *refname,
 
 int read_ref_full(const char *refname, int resolve_flags, struct object_id *oid, int *flags)
 {
-	return refs_read_ref_full(get_main_ref_store(), refname,
+	return refs_read_ref_full(get_main_ref_store(the_repository), refname,
 				  resolve_flags, oid, flags);
 }
 
@@ -375,7 +376,7 @@ int refs_for_each_tag_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_tag_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_tag_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_tag_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -385,7 +386,7 @@ int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_branch_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_branch_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_branch_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -395,7 +396,7 @@ int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_remote_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_remote_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_remote_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data)
@@ -730,7 +731,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 	struct strbuf err = STRBUF_INIT;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		return delete_pseudoref(refname, old_oid);
 	}
 
@@ -752,7 +753,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 int delete_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid, unsigned int flags)
 {
-	return refs_delete_ref(get_main_ref_store(), msg, refname,
+	return refs_delete_ref(get_main_ref_store(the_repository), msg, refname,
 			       old_oid, flags);
 }
 
@@ -928,7 +929,7 @@ struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,
 
 struct ref_transaction *ref_transaction_begin(struct strbuf *err)
 {
-	return ref_store_transaction_begin(get_main_ref_store(), err);
+	return ref_store_transaction_begin(get_main_ref_store(the_repository), err);
 }
 
 void ref_transaction_free(struct ref_transaction *transaction)
@@ -1060,7 +1061,7 @@ int refs_update_ref(struct ref_store *refs, const char *msg,
 	int ret = 0;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		ret = write_pseudoref(refname, new_oid, old_oid, &err);
 	} else {
 		t = ref_store_transaction_begin(refs, &err);
@@ -1099,7 +1100,7 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr)
 {
-	return refs_update_ref(get_main_ref_store(), msg, refname, new_oid,
+	return refs_update_ref(get_main_ref_store(the_repository), msg, refname, new_oid,
 			       old_oid, flags, onerr);
 }
 
@@ -1320,7 +1321,7 @@ int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int head_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_head_ref(get_main_ref_store(), fn, cb_data);
+	return refs_head_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 struct ref_iterator *refs_ref_iterator_begin(
@@ -1379,7 +1380,7 @@ int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
@@ -1390,7 +1391,7 @@ int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref_in(get_main_ref_store(), prefix, fn, cb_data);
+	return refs_for_each_ref_in(get_main_ref_store(the_repository), prefix, fn, cb_data);
 }
 
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)
@@ -1399,7 +1400,7 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig
 
 	if (broken)
 		flag = DO_FOR_EACH_INCLUDE_BROKEN;
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       prefix, fn, 0, flag, cb_data);
 }
 
@@ -1416,7 +1417,7 @@ int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       git_replace_ref_base, fn,
 			       strlen(git_replace_ref_base),
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
@@ -1427,7 +1428,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
-	ret = do_for_each_ref(get_main_ref_store(),
+	ret = do_for_each_ref(get_main_ref_store(the_repository),
 			      buf.buf, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
@@ -1441,7 +1442,7 @@ int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_rawref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_rawref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_read_raw_ref(struct ref_store *ref_store,
@@ -1547,7 +1548,7 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 /* backend functions */
 int refs_init_db(struct strbuf *err)
 {
-	struct ref_store *refs = get_main_ref_store();
+	struct ref_store *refs = get_main_ref_store(the_repository);
 
 	return refs->be->init_db(refs, err);
 }
@@ -1555,7 +1556,7 @@ int refs_init_db(struct strbuf *err)
 const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 			       struct object_id *oid, int *flags)
 {
-	return refs_resolve_ref_unsafe(get_main_ref_store(), refname,
+	return refs_resolve_ref_unsafe(get_main_ref_store(the_repository), refname,
 				       resolve_flags, oid, flags);
 }
 
@@ -1651,7 +1652,7 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store(void)
+struct ref_store *get_main_ref_store_the_repository(void)
 {
 	if (main_ref_store)
 		return main_ref_store;
@@ -1726,7 +1727,7 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)
 	const char *id;
 
 	if (wt->is_current)
-		return get_main_ref_store();
+		return get_main_ref_store(the_repository);
 
 	id = wt->id ? wt->id : "/";
 	refs = lookup_ref_store_map(&worktree_ref_stores, id);
@@ -1782,7 +1783,7 @@ int refs_peel_ref(struct ref_store *refs, const char *refname,
 
 int peel_ref(const char *refname, struct object_id *oid)
 {
-	return refs_peel_ref(get_main_ref_store(), refname, oid);
+	return refs_peel_ref(get_main_ref_store(the_repository), refname, oid);
 }
 
 int refs_create_symref(struct ref_store *refs,
@@ -1798,7 +1799,7 @@ int refs_create_symref(struct ref_store *refs,
 int create_symref(const char *ref_target, const char *refs_heads_master,
 		  const char *logmsg)
 {
-	return refs_create_symref(get_main_ref_store(), ref_target,
+	return refs_create_symref(get_main_ref_store(the_repository), ref_target,
 				  refs_heads_master, logmsg);
 }
 
@@ -2006,7 +2007,7 @@ int refs_for_each_reflog(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_reflog(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_reflog(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_reflog(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
@@ -2021,7 +2022,7 @@ int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
 int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,
 				void *cb_data)
 {
-	return refs_for_each_reflog_ent_reverse(get_main_ref_store(),
+	return refs_for_each_reflog_ent_reverse(get_main_ref_store(the_repository),
 						refname, fn, cb_data);
 }
 
@@ -2034,7 +2035,7 @@ int refs_for_each_reflog_ent(struct ref_store *refs, const char *refname,
 int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,
 			void *cb_data)
 {
-	return refs_for_each_reflog_ent(get_main_ref_store(), refname,
+	return refs_for_each_reflog_ent(get_main_ref_store(the_repository), refname,
 					fn, cb_data);
 }
 
@@ -2045,7 +2046,7 @@ int refs_reflog_exists(struct ref_store *refs, const char *refname)
 
 int reflog_exists(const char *refname)
 {
-	return refs_reflog_exists(get_main_ref_store(), refname);
+	return refs_reflog_exists(get_main_ref_store(the_repository), refname);
 }
 
 int refs_create_reflog(struct ref_store *refs, const char *refname,
@@ -2057,7 +2058,7 @@ int refs_create_reflog(struct ref_store *refs, const char *refname,
 int safe_create_reflog(const char *refname, int force_create,
 		       struct strbuf *err)
 {
-	return refs_create_reflog(get_main_ref_store(), refname,
+	return refs_create_reflog(get_main_ref_store(the_repository), refname,
 				  force_create, err);
 }
 
@@ -2068,7 +2069,7 @@ int refs_delete_reflog(struct ref_store *refs, const char *refname)
 
 int delete_reflog(const char *refname)
 {
-	return refs_delete_reflog(get_main_ref_store(), refname);
+	return refs_delete_reflog(get_main_ref_store(the_repository), refname);
 }
 
 int refs_reflog_expire(struct ref_store *refs,
@@ -2091,7 +2092,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 		  reflog_expiry_cleanup_fn cleanup_fn,
 		  void *policy_cb_data)
 {
-	return refs_reflog_expire(get_main_ref_store(),
+	return refs_reflog_expire(get_main_ref_store(the_repository),
 				  refname, oid, flags,
 				  prepare_fn, should_prune_fn,
 				  cleanup_fn, policy_cb_data);
@@ -2114,7 +2115,7 @@ int refs_delete_refs(struct ref_store *refs, const char *msg,
 int delete_refs(const char *msg, struct string_list *refnames,
 		unsigned int flags)
 {
-	return refs_delete_refs(get_main_ref_store(), msg, refnames, flags);
+	return refs_delete_refs(get_main_ref_store(the_repository), msg, refnames, flags);
 }
 
 int refs_rename_ref(struct ref_store *refs, const char *oldref,
@@ -2125,7 +2126,7 @@ int refs_rename_ref(struct ref_store *refs, const char *oldref,
 
 int rename_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_rename_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_rename_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
 
 int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
@@ -2136,5 +2137,5 @@ int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
 
 int copy_existing_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_copy_existing_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_copy_existing_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
diff --git a/refs.h b/refs.h
index 01be5ae32f..0d013377ce 100644
--- a/refs.h
+++ b/refs.h
@@ -758,7 +758,9 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-struct ref_store *get_main_ref_store(void);
+#define get_main_ref_store(r) \
+	get_main_ref_store_##r()
+struct ref_store *get_main_ref_store_the_repository(void);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/revision.c b/revision.c
index 5ce9b93baa..8fa5a3c7aa 100644
--- a/revision.c
+++ b/revision.c
@@ -6,6 +6,7 @@
 #include "diff.h"
 #include "refs.h"
 #include "revision.h"
+#include "repository.h"
 #include "graph.h"
 #include "grep.h"
 #include "reflog-walk.h"
@@ -1289,7 +1290,7 @@ void add_reflogs_to_pending(struct rev_info *revs, unsigned flags)
 
 	cb.all_revs = revs;
 	cb.all_flags = flags;
-	cb.refs = get_main_ref_store();
+	cb.refs = get_main_ref_store(the_repository);
 	for_each_reflog(handle_one_reflog, &cb);
 
 	if (!revs->single_worktree)
@@ -2182,7 +2183,7 @@ static int handle_revision_pseudo_opt(const char *submodule,
 			die("BUG: --single-worktree cannot be used together with submodule");
 		refs = get_submodule_ref_store(submodule);
 	} else
-		refs = get_main_ref_store();
+		refs = get_main_ref_store(the_repository);
 
 	/*
 	 * NOTE!
diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c
index 7314b5943e..e8c328ec0e 100644
--- a/t/helper/test-ref-store.c
+++ b/t/helper/test-ref-store.c
@@ -2,6 +2,7 @@
 #include "refs.h"
 #include "worktree.h"
 #include "object-store.h"
+#include "repository.h"
 
 static const char *notnull(const char *arg, const char *name)
 {
@@ -22,7 +23,7 @@ static const char **get_store(const char **argv, struct ref_store **refs)
 	if (!argv[0]) {
 		die("ref store required");
 	} else if (!strcmp(argv[0], "main")) {
-		*refs = get_main_ref_store();
+		*refs = get_main_ref_store(the_repository);
 	} else if (skip_prefix(argv[0], "submodule:", &gitdir)) {
 		struct strbuf sb = STRBUF_INIT;
 		int ret;
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 7%]

* Re: [RFC PATCH 00/19] object-store refactoring 3 (replace objects, main ref store)
      [irrelevant] <20180406232136.253950-1-sbeller@google.com>
                   ` (2 preceding siblings ...)
  2018-04-06 23:21 ` [PATCH 13/19] refs: store the main ref store inside the repository struct Stefan Beller
@ 2018-04-07  9:50 ` Duy Nguyen
  2018-04-09 17:39   ` Stefan Beller
      [irrelevant] ` <20180409224533.17764-1-sbeller@google.com>
  4 siblings, 1 reply; 200+ results
From: Duy Nguyen @ 2018-04-07  9:50 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Jonathan Tan, Git Mailing List

On Sat, Apr 7, 2018 at 1:21 AM, Stefan Beller <sbeller@google.com> wrote:     *
> diff --git a/repository.h b/repository.h
> index 09df94a472..2922d3a28b 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -26,6 +26,11 @@ struct repository {
>          */
>         struct raw_object_store *objects;
>
> +       /*
> +        * The store in which the refs are hold.
> +        */
> +       struct ref_store *main_ref_store;
> +

We probably should drop the main_ prefix here because this could also
be a submodule ref store (when the repository is about a submodule).
worktree ref store is a different story and I don't know how to best
present it here yet (I'm still thinking a separate struct repository).
But we can worry about that when struct repository supports multiple
worktree.
-- 
Duy

^ permalink raw reply	[relevance 5%]

* Re: What's cooking in git.git (Apr 2018, #01; Mon, 9)
      [irrelevant] ` <CACsJy8C4GUy1xv10oQyrDVkdy4nq636N24b2_pGo-nivPY-x=A@mail.gmail.com>
@ 2018-04-09 17:23   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-09 17:23 UTC (permalink / raw)
  To: Duy Nguyen; +Cc: Junio C Hamano, Git Mailing List

On Mon, Apr 9, 2018 at 8:31 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Mon, Apr 9, 2018 at 12:21 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> * sb/packfiles-in-repository (2018-03-26) 12 commits
>>   (merged to 'next' on 2018-03-30 at caa68db14d)
>>  + packfile: keep prepare_packed_git() private
>>  + packfile: allow find_pack_entry to handle arbitrary repositories
>>  + packfile: add repository argument to find_pack_entry
>>  + packfile: allow reprepare_packed_git to handle arbitrary repositories
>>  + packfile: allow prepare_packed_git to handle arbitrary repositories
>>  + packfile: allow prepare_packed_git_one to handle arbitrary repositories
>>  + packfile: add repository argument to reprepare_packed_git
>>  + packfile: add repository argument to prepare_packed_git
>>  + packfile: add repository argument to prepare_packed_git_one
>>  + packfile: allow install_packed_git to handle arbitrary repositories
>>  + packfile: allow rearrange_packed_git to handle arbitrary repositories
>>  + packfile: allow prepare_packed_git_mru to handle arbitrary repositories
>>  (this branch uses nd/remove-ignore-env-field and sb/object-store; is tangled with sb/submodule-move-nested.)
>>
>>  Refactoring of the internal global data structure continues.
>>
>>  Is this ready for 'master' by now?
>
> I think so. Things start to look much nicer.

I think so, too.

I am working on top of that series now for the third part,
assuming this is good to go.
https://public-inbox.org/git/20180406232136.253950-1-sbeller@google.com/

Thanks,
Stefan

^ permalink raw reply	[relevance 2%]

* Re: [RFC PATCH 00/19] object-store refactoring 3 (replace objects, main ref store)
  2018-04-07  9:50 ` [RFC PATCH 00/19] object-store refactoring 3 (replace objects, main ref store) Duy Nguyen
@ 2018-04-09 17:39   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-09 17:39 UTC (permalink / raw)
  To: Duy Nguyen; +Cc: Jonathan Tan, Git Mailing List

On Sat, Apr 7, 2018 at 2:50 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Sat, Apr 7, 2018 at 1:21 AM, Stefan Beller <sbeller@google.com> wrote:     *
>> diff --git a/repository.h b/repository.h
>> index 09df94a472..2922d3a28b 100644
>> --- a/repository.h
>> +++ b/repository.h
>> @@ -26,6 +26,11 @@ struct repository {
>>          */
>>         struct raw_object_store *objects;
>>
>> +       /*
>> +        * The store in which the refs are hold.
>> +        */
>> +       struct ref_store *main_ref_store;
>> +
>
> We probably should drop the main_ prefix here because this could also
> be a submodule ref store (when the repository is about a submodule).

it's still the main ref store of the submodule, then. :)
But yes, I agree we should rename it. Now or eventually later?

> worktree ref store is a different story and I don't know how to best
> present it here yet (I'm still thinking a separate struct repository).

I imagine that we'd rather want to have arrays of config/worktree path/refs
when needed and the worktree is just an index into all of these things?

> But we can worry about that when struct repository supports multiple
> worktree.

ok. Thanks for bringing this to my attention.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* [PATCH 06/16] replace-object: check_replace_refs is safe in multi repo environment
      [irrelevant] ` <20180409224533.17764-1-sbeller@google.com>
@ 2018-04-09 22:45   ` Stefan Beller
  2018-04-10  3:37     ` Junio C Hamano
  2018-04-09 22:45   ` [PATCH 07/16] refs: add repository argument to get_main_ref_store Stefan Beller
                     ` (2 subsequent siblings)
  3 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-09 22:45 UTC (permalink / raw)
  To: sbeller; +Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals

In e1111cef23 (inline lookup_replace_object() calls, 2011-05-15) a shortcut
for checking the object replacement was added by setting check_replace_refs
to 0 once the replacements were evaluated to not exist. This works fine in
with the assumption of only one repository in existence.

The assumption won't hold true any more when we work on multiple instances
of a repository structs (e.g. one struct per submodule), as the first
repository to be inspected may have no replacements and would set the
global variable. Other repositories would then completely omit their
evaluation of replacements.

This reverts back the meaning of the flag `check_replace_refs` of
"Do we need to check with the lookup table?" to "Do we need to read
the replacement definition?", adding the bypassing logic to
lookup_replace_object after the replacement definition was read.
As with the original patch, delay the renaming of the global variable

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 environment.c    | 2 +-
 replace-object.c | 3 ---
 replace-object.h | 5 ++++-
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/environment.c b/environment.c
index 39b3d906c8..b991fc0a87 100644
--- a/environment.c
+++ b/environment.c
@@ -50,7 +50,7 @@ const char *editor_program;
 const char *askpass_program;
 const char *excludes_file;
 enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;
-int check_replace_refs = 1;
+int check_replace_refs = 1; /* NEEDSWORK: rename to read_replace_refs */
 char *git_replace_ref_base;
 enum eol core_eol = EOL_UNSET;
 int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;
diff --git a/replace-object.c b/replace-object.c
index 953fa9cc40..b2405f6027 100644
--- a/replace-object.c
+++ b/replace-object.c
@@ -41,9 +41,6 @@ static void prepare_replace_object(void)
 	oidmap_init(the_repository->objects->replace_map, 0);
 
 	for_each_replace_ref(register_replace_ref, NULL);
-
-	if (!the_repository->objects->replace_map->map.tablesize)
-		check_replace_refs = 0;
 }
 
 /* We allow "recursive" replacement. Only within reason, though */
diff --git a/replace-object.h b/replace-object.h
index 15315311fb..dbc51265ec 100644
--- a/replace-object.h
+++ b/replace-object.h
@@ -3,6 +3,7 @@
 
 #include "oidmap.h"
 #include "repository.h"
+#include "object-store.h"
 
 struct replace_object {
 	struct oidmap_entry original;
@@ -23,7 +24,9 @@ extern const struct object_id *do_lookup_replace_object(const struct object_id *
  */
 static inline const struct object_id *lookup_replace_object(const struct object_id *oid)
 {
-	if (!check_replace_refs)
+	if (!check_replace_refs ||
+	    (the_repository->objects->replace_map &&
+	     the_repository->objects->replace_map->map.tablesize == 0))
 		return oid;
 	return do_lookup_replace_object(oid);
 }
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 6%]

* [PATCH 07/16] refs: add repository argument to get_main_ref_store
      [irrelevant] ` <20180409224533.17764-1-sbeller@google.com>
  2018-04-09 22:45   ` [PATCH 06/16] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
@ 2018-04-09 22:45   ` Stefan Beller
  2018-04-09 22:45   ` [PATCH 12/16] refs: store the main ref store inside the repository struct Stefan Beller
      [irrelevant]   ` <20180412002118.102976-1-sbeller@google.com>
  3 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-09 22:45 UTC (permalink / raw)
  To: sbeller
  Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals, Jonathan Nieder

Add a repository argument to allow the get_main_ref_store caller
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/pack-refs.c       |  3 +-
 refs.c                    | 67 ++++++++++++++++++++-------------------
 refs.h                    |  4 ++-
 revision.c                |  5 +--
 t/helper/test-ref-store.c |  3 +-
 5 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c
index b106a392a4..f3353564f9 100644
--- a/builtin/pack-refs.c
+++ b/builtin/pack-refs.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "refs.h"
+#include "repository.h"
 
 static char const * const pack_refs_usage[] = {
 	N_("git pack-refs [<options>]"),
@@ -17,5 +18,5 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 	};
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
-	return refs_pack_refs(get_main_ref_store(), flags);
+	return refs_pack_refs(get_main_ref_store(the_repository), flags);
 }
diff --git a/refs.c b/refs.c
index 8b7a77fe5e..74d4ed97cb 100644
--- a/refs.c
+++ b/refs.c
@@ -13,6 +13,7 @@
 #include "tag.h"
 #include "submodule.h"
 #include "worktree.h"
+#include "repository.h"
 
 /*
  * List of all available backends
@@ -206,7 +207,7 @@ char *refs_resolve_refdup(struct ref_store *refs,
 char *resolve_refdup(const char *refname, int resolve_flags,
 		     struct object_id *oid, int *flags)
 {
-	return refs_resolve_refdup(get_main_ref_store(),
+	return refs_resolve_refdup(get_main_ref_store(the_repository),
 				   refname, resolve_flags,
 				   oid, flags);
 }
@@ -228,7 +229,7 @@ int refs_read_ref_full(struct ref_store *refs, const char *refname,
 
 int read_ref_full(const char *refname, int resolve_flags, struct object_id *oid, int *flags)
 {
-	return refs_read_ref_full(get_main_ref_store(), refname,
+	return refs_read_ref_full(get_main_ref_store(the_repository), refname,
 				  resolve_flags, oid, flags);
 }
 
@@ -375,7 +376,7 @@ int refs_for_each_tag_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_tag_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_tag_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_tag_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -385,7 +386,7 @@ int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_branch_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_branch_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_branch_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -395,7 +396,7 @@ int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_remote_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_remote_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_remote_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data)
@@ -730,7 +731,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 	struct strbuf err = STRBUF_INIT;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		return delete_pseudoref(refname, old_oid);
 	}
 
@@ -752,7 +753,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 int delete_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid, unsigned int flags)
 {
-	return refs_delete_ref(get_main_ref_store(), msg, refname,
+	return refs_delete_ref(get_main_ref_store(the_repository), msg, refname,
 			       old_oid, flags);
 }
 
@@ -928,7 +929,7 @@ struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,
 
 struct ref_transaction *ref_transaction_begin(struct strbuf *err)
 {
-	return ref_store_transaction_begin(get_main_ref_store(), err);
+	return ref_store_transaction_begin(get_main_ref_store(the_repository), err);
 }
 
 void ref_transaction_free(struct ref_transaction *transaction)
@@ -1060,7 +1061,7 @@ int refs_update_ref(struct ref_store *refs, const char *msg,
 	int ret = 0;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		ret = write_pseudoref(refname, new_oid, old_oid, &err);
 	} else {
 		t = ref_store_transaction_begin(refs, &err);
@@ -1099,7 +1100,7 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr)
 {
-	return refs_update_ref(get_main_ref_store(), msg, refname, new_oid,
+	return refs_update_ref(get_main_ref_store(the_repository), msg, refname, new_oid,
 			       old_oid, flags, onerr);
 }
 
@@ -1320,7 +1321,7 @@ int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int head_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_head_ref(get_main_ref_store(), fn, cb_data);
+	return refs_head_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 struct ref_iterator *refs_ref_iterator_begin(
@@ -1379,7 +1380,7 @@ int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
@@ -1390,7 +1391,7 @@ int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref_in(get_main_ref_store(), prefix, fn, cb_data);
+	return refs_for_each_ref_in(get_main_ref_store(the_repository), prefix, fn, cb_data);
 }
 
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)
@@ -1399,7 +1400,7 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig
 
 	if (broken)
 		flag = DO_FOR_EACH_INCLUDE_BROKEN;
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       prefix, fn, 0, flag, cb_data);
 }
 
@@ -1416,7 +1417,7 @@ int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       git_replace_ref_base, fn,
 			       strlen(git_replace_ref_base),
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
@@ -1427,7 +1428,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
-	ret = do_for_each_ref(get_main_ref_store(),
+	ret = do_for_each_ref(get_main_ref_store(the_repository),
 			      buf.buf, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
@@ -1441,7 +1442,7 @@ int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_rawref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_rawref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_read_raw_ref(struct ref_store *ref_store,
@@ -1547,7 +1548,7 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 /* backend functions */
 int refs_init_db(struct strbuf *err)
 {
-	struct ref_store *refs = get_main_ref_store();
+	struct ref_store *refs = get_main_ref_store(the_repository);
 
 	return refs->be->init_db(refs, err);
 }
@@ -1555,7 +1556,7 @@ int refs_init_db(struct strbuf *err)
 const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 			       struct object_id *oid, int *flags)
 {
-	return refs_resolve_ref_unsafe(get_main_ref_store(), refname,
+	return refs_resolve_ref_unsafe(get_main_ref_store(the_repository), refname,
 				       resolve_flags, oid, flags);
 }
 
@@ -1651,7 +1652,7 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store(void)
+struct ref_store *get_main_ref_store_the_repository(void)
 {
 	if (main_ref_store)
 		return main_ref_store;
@@ -1726,7 +1727,7 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)
 	const char *id;
 
 	if (wt->is_current)
-		return get_main_ref_store();
+		return get_main_ref_store(the_repository);
 
 	id = wt->id ? wt->id : "/";
 	refs = lookup_ref_store_map(&worktree_ref_stores, id);
@@ -1782,7 +1783,7 @@ int refs_peel_ref(struct ref_store *refs, const char *refname,
 
 int peel_ref(const char *refname, struct object_id *oid)
 {
-	return refs_peel_ref(get_main_ref_store(), refname, oid);
+	return refs_peel_ref(get_main_ref_store(the_repository), refname, oid);
 }
 
 int refs_create_symref(struct ref_store *refs,
@@ -1798,7 +1799,7 @@ int refs_create_symref(struct ref_store *refs,
 int create_symref(const char *ref_target, const char *refs_heads_master,
 		  const char *logmsg)
 {
-	return refs_create_symref(get_main_ref_store(), ref_target,
+	return refs_create_symref(get_main_ref_store(the_repository), ref_target,
 				  refs_heads_master, logmsg);
 }
 
@@ -2006,7 +2007,7 @@ int refs_for_each_reflog(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_reflog(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_reflog(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_reflog(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
@@ -2021,7 +2022,7 @@ int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
 int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,
 				void *cb_data)
 {
-	return refs_for_each_reflog_ent_reverse(get_main_ref_store(),
+	return refs_for_each_reflog_ent_reverse(get_main_ref_store(the_repository),
 						refname, fn, cb_data);
 }
 
@@ -2034,7 +2035,7 @@ int refs_for_each_reflog_ent(struct ref_store *refs, const char *refname,
 int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,
 			void *cb_data)
 {
-	return refs_for_each_reflog_ent(get_main_ref_store(), refname,
+	return refs_for_each_reflog_ent(get_main_ref_store(the_repository), refname,
 					fn, cb_data);
 }
 
@@ -2045,7 +2046,7 @@ int refs_reflog_exists(struct ref_store *refs, const char *refname)
 
 int reflog_exists(const char *refname)
 {
-	return refs_reflog_exists(get_main_ref_store(), refname);
+	return refs_reflog_exists(get_main_ref_store(the_repository), refname);
 }
 
 int refs_create_reflog(struct ref_store *refs, const char *refname,
@@ -2057,7 +2058,7 @@ int refs_create_reflog(struct ref_store *refs, const char *refname,
 int safe_create_reflog(const char *refname, int force_create,
 		       struct strbuf *err)
 {
-	return refs_create_reflog(get_main_ref_store(), refname,
+	return refs_create_reflog(get_main_ref_store(the_repository), refname,
 				  force_create, err);
 }
 
@@ -2068,7 +2069,7 @@ int refs_delete_reflog(struct ref_store *refs, const char *refname)
 
 int delete_reflog(const char *refname)
 {
-	return refs_delete_reflog(get_main_ref_store(), refname);
+	return refs_delete_reflog(get_main_ref_store(the_repository), refname);
 }
 
 int refs_reflog_expire(struct ref_store *refs,
@@ -2091,7 +2092,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 		  reflog_expiry_cleanup_fn cleanup_fn,
 		  void *policy_cb_data)
 {
-	return refs_reflog_expire(get_main_ref_store(),
+	return refs_reflog_expire(get_main_ref_store(the_repository),
 				  refname, oid, flags,
 				  prepare_fn, should_prune_fn,
 				  cleanup_fn, policy_cb_data);
@@ -2114,7 +2115,7 @@ int refs_delete_refs(struct ref_store *refs, const char *msg,
 int delete_refs(const char *msg, struct string_list *refnames,
 		unsigned int flags)
 {
-	return refs_delete_refs(get_main_ref_store(), msg, refnames, flags);
+	return refs_delete_refs(get_main_ref_store(the_repository), msg, refnames, flags);
 }
 
 int refs_rename_ref(struct ref_store *refs, const char *oldref,
@@ -2125,7 +2126,7 @@ int refs_rename_ref(struct ref_store *refs, const char *oldref,
 
 int rename_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_rename_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_rename_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
 
 int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
@@ -2136,5 +2137,5 @@ int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
 
 int copy_existing_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_copy_existing_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_copy_existing_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
diff --git a/refs.h b/refs.h
index 01be5ae32f..0d013377ce 100644
--- a/refs.h
+++ b/refs.h
@@ -758,7 +758,9 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-struct ref_store *get_main_ref_store(void);
+#define get_main_ref_store(r) \
+	get_main_ref_store_##r()
+struct ref_store *get_main_ref_store_the_repository(void);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/revision.c b/revision.c
index b42c836d7a..1cff11833e 100644
--- a/revision.c
+++ b/revision.c
@@ -6,6 +6,7 @@
 #include "diff.h"
 #include "refs.h"
 #include "revision.h"
+#include "repository.h"
 #include "graph.h"
 #include "grep.h"
 #include "reflog-walk.h"
@@ -1285,7 +1286,7 @@ void add_reflogs_to_pending(struct rev_info *revs, unsigned flags)
 
 	cb.all_revs = revs;
 	cb.all_flags = flags;
-	cb.refs = get_main_ref_store();
+	cb.refs = get_main_ref_store(the_repository);
 	for_each_reflog(handle_one_reflog, &cb);
 
 	if (!revs->single_worktree)
@@ -2176,7 +2177,7 @@ static int handle_revision_pseudo_opt(const char *submodule,
 			die("BUG: --single-worktree cannot be used together with submodule");
 		refs = get_submodule_ref_store(submodule);
 	} else
-		refs = get_main_ref_store();
+		refs = get_main_ref_store(the_repository);
 
 	/*
 	 * NOTE!
diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c
index 7314b5943e..e8c328ec0e 100644
--- a/t/helper/test-ref-store.c
+++ b/t/helper/test-ref-store.c
@@ -2,6 +2,7 @@
 #include "refs.h"
 #include "worktree.h"
 #include "object-store.h"
+#include "repository.h"
 
 static const char *notnull(const char *arg, const char *name)
 {
@@ -22,7 +23,7 @@ static const char **get_store(const char **argv, struct ref_store **refs)
 	if (!argv[0]) {
 		die("ref store required");
 	} else if (!strcmp(argv[0], "main")) {
-		*refs = get_main_ref_store();
+		*refs = get_main_ref_store(the_repository);
 	} else if (skip_prefix(argv[0], "submodule:", &gitdir)) {
 		struct strbuf sb = STRBUF_INIT;
 		int ret;
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 7%]

* [PATCH 12/16] refs: store the main ref store inside the repository struct
      [irrelevant] ` <20180409224533.17764-1-sbeller@google.com>
  2018-04-09 22:45   ` [PATCH 06/16] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
  2018-04-09 22:45   ` [PATCH 07/16] refs: add repository argument to get_main_ref_store Stefan Beller
@ 2018-04-09 22:45   ` Stefan Beller
  2018-04-09 23:24     ` Brandon Williams
  2018-04-10 14:02     ` Michael Haggerty
      [irrelevant]   ` <20180412002118.102976-1-sbeller@google.com>
  3 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-04-09 22:45 UTC (permalink / raw)
  To: sbeller; +Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 refs.c       | 13 +++++--------
 refs.h       |  4 +---
 repository.h |  3 +++
 3 files changed, 9 insertions(+), 11 deletions(-)

diff --git a/refs.c b/refs.c
index f58b9fb7df..b5be754a97 100644
--- a/refs.c
+++ b/refs.c
@@ -1608,9 +1608,6 @@ static struct ref_store_hash_entry *alloc_ref_store_hash_entry(
 	return entry;
 }
 
-/* A pointer to the ref_store for the main repository: */
-static struct ref_store *main_ref_store;
-
 /* A hashmap of ref_stores, stored by submodule name: */
 static struct hashmap submodule_ref_stores;
 
@@ -1652,13 +1649,13 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store_the_repository(void)
+struct ref_store *get_main_ref_store(struct repository *r)
 {
-	if (main_ref_store)
-		return main_ref_store;
+	if (r->main_ref_store)
+		return r->main_ref_store;
 
-	main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
-	return main_ref_store;
+	r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
+	return r->main_ref_store;
 }
 
 /*
diff --git a/refs.h b/refs.h
index ab3d2bec2f..f5ab68c0ed 100644
--- a/refs.h
+++ b/refs.h
@@ -760,9 +760,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-#define get_main_ref_store(r) \
-	get_main_ref_store_##r()
-struct ref_store *get_main_ref_store_the_repository(void);
+struct ref_store *get_main_ref_store(struct repository *r);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/repository.h b/repository.h
index 09df94a472..7d0710b273 100644
--- a/repository.h
+++ b/repository.h
@@ -26,6 +26,9 @@ struct repository {
 	 */
 	struct raw_object_store *objects;
 
+	/* The store in which the refs are held. */
+	struct ref_store *main_ref_store;
+
 	/*
 	 * Path to the repository's graft file.
 	 * Cannot be NULL after initialization.
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 9%]

* Re: [PATCH 12/16] refs: store the main ref store inside the repository struct
  2018-04-09 22:45   ` [PATCH 12/16] refs: store the main ref store inside the repository struct Stefan Beller
@ 2018-04-09 23:24     ` Brandon Williams
  2018-04-09 23:29       ` Stefan Beller
  2018-04-10 14:02     ` Michael Haggerty
  1 sibling, 1 reply; 200+ results
From: Brandon Williams @ 2018-04-09 23:24 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals

On 04/09, Stefan Beller wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  refs.c       | 13 +++++--------
>  refs.h       |  4 +---
>  repository.h |  3 +++
>  3 files changed, 9 insertions(+), 11 deletions(-)
> 
> diff --git a/refs.c b/refs.c
> index f58b9fb7df..b5be754a97 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1608,9 +1608,6 @@ static struct ref_store_hash_entry *alloc_ref_store_hash_entry(
>  	return entry;
>  }
>  
> -/* A pointer to the ref_store for the main repository: */
> -static struct ref_store *main_ref_store;
> -
>  /* A hashmap of ref_stores, stored by submodule name: */
>  static struct hashmap submodule_ref_stores;
>  
> @@ -1652,13 +1649,13 @@ static struct ref_store *ref_store_init(const char *gitdir,
>  	return refs;
>  }
>  
> -struct ref_store *get_main_ref_store_the_repository(void)
> +struct ref_store *get_main_ref_store(struct repository *r)
>  {
> -	if (main_ref_store)
> -		return main_ref_store;
> +	if (r->main_ref_store)
> +		return r->main_ref_store;
>  
> -	main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
> -	return main_ref_store;
> +	r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
> +	return r->main_ref_store;

I assume that since this takes in a git dir as a parameter
that the ref-store is in a good enough place to be embedded in a
repository struct (as in this would work with an arbitrary repo)?

>  }
>  
>  /*
> diff --git a/refs.h b/refs.h
> index ab3d2bec2f..f5ab68c0ed 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -760,9 +760,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
>  
>  int ref_storage_backend_exists(const char *name);
>  
> -#define get_main_ref_store(r) \
> -	get_main_ref_store_##r()
> -struct ref_store *get_main_ref_store_the_repository(void);
> +struct ref_store *get_main_ref_store(struct repository *r);
>  /*
>   * Return the ref_store instance for the specified submodule. For the
>   * main repository, use submodule==NULL; such a call cannot fail. For
> diff --git a/repository.h b/repository.h
> index 09df94a472..7d0710b273 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -26,6 +26,9 @@ struct repository {
>  	 */
>  	struct raw_object_store *objects;
>  
> +	/* The store in which the refs are held. */
> +	struct ref_store *main_ref_store;
> +
>  	/*
>  	 * Path to the repository's graft file.
>  	 * Cannot be NULL after initialization.
> -- 
> 2.17.0.484.g0c8726318c-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[relevance 2%]

* Re: [PATCH 12/16] refs: store the main ref store inside the repository struct
  2018-04-09 23:29       ` Stefan Beller
@ 2018-04-09 23:35         ` Brandon Williams
  0 siblings, 0 replies; 200+ results
From: Brandon Williams @ 2018-04-09 23:35 UTC (permalink / raw)
  To: Stefan Beller
  Cc: git, Jonathan Tan, Eric Sunshine, Duy Nguyen, René Scharfe,
	brian m. carlson

On 04/09, Stefan Beller wrote:
> Hi Brandon,
> 
> On Mon, Apr 9, 2018 at 4:24 PM, Brandon Williams <bmwill@google.com> wrote:
> 
> >> -     main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
> >> -     return main_ref_store;
> >> +     r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
> >> +     return r->main_ref_store;
> >
> > I assume that since this takes in a git dir as a parameter
> > that the ref-store is in a good enough place to be embedded in a
> > repository struct (as in this would work with an arbitrary repo)?
> 
> That is my current understanding.
> 
> As the refs code can also take a path into a submodule and construct
> a submodule ref store for the caller, we'd want to resolve the tension
> between the ref store and the repository struct who is responsible for
> the submodule ref store eventually by removing the submodule
> functionality from the ref store and only relying on the ref stores created by
> the repository struct.

oh right, assuming it can already handle submodule ref-stores it should
work as-is then.

-- 
Brandon Williams

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 12/16] refs: store the main ref store inside the repository struct
  2018-04-09 23:24     ` Brandon Williams
@ 2018-04-09 23:29       ` Stefan Beller
  2018-04-09 23:35         ` Brandon Williams
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-09 23:29 UTC (permalink / raw)
  To: Brandon Williams
  Cc: git, Jonathan Tan, Eric Sunshine, Duy Nguyen, René Scharfe,
	brian m. carlson

Hi Brandon,

On Mon, Apr 9, 2018 at 4:24 PM, Brandon Williams <bmwill@google.com> wrote:

>> -     main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
>> -     return main_ref_store;
>> +     r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
>> +     return r->main_ref_store;
>
> I assume that since this takes in a git dir as a parameter
> that the ref-store is in a good enough place to be embedded in a
> repository struct (as in this would work with an arbitrary repo)?

That is my current understanding.

As the refs code can also take a path into a submodule and construct
a submodule ref store for the caller, we'd want to resolve the tension
between the ref store and the repository struct who is responsible for
the submodule ref store eventually by removing the submodule
functionality from the ref store and only relying on the ref stores created by
the repository struct.

^ permalink raw reply	[relevance 6%]

* Re: [PATCH 06/16] replace-object: check_replace_refs is safe in multi repo environment
  2018-04-09 22:45   ` [PATCH 06/16] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
@ 2018-04-10  3:37     ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-04-10  3:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals

Stefan Beller <sbeller@google.com> writes:

> In e1111cef23 (inline lookup_replace_object() calls, 2011-05-15) a
> shortcut for checking the object replacement was added by setting
> check_replace_refs to 0 once the replacements were evaluated to
> not exist. This works fine in with the assumption of only one
> repository in existence.

"works fine in with the..."?  I guess s/ in with/ with/?

> The assumption won't hold true any more when we work on multiple
> instances of a repository structs (e.g. one struct per submodule),
> as the first repository to be inspected may have no replacements
> and would set the global variable. Other repositories would then
> completely omit their evaluation of replacements.
>
> This reverts back the meaning of the flag `check_replace_refs` of
> "Do we need to check with the lookup table?" to "Do we need to read
> the replacement definition?", adding the bypassing logic to
> lookup_replace_object after the replacement definition was read.
> As with the original patch, delay the renaming of the global variable

Hmph, if we decided that replace database is per repository
instance, shouldn't this variable also become per repository,
instead of staying to be a system-wide global?

Perhaps that will happpen in a later stage of the series that I
haven't seen yet, I guess.  And until that happens, we disable the
optimization and always call into do_lookup_replace_object() when
lookup_replace_object() is called, which is OK.

>  static inline const struct object_id *lookup_replace_object(const struct object_id *oid)
>  {
> -	if (!check_replace_refs)
> +	if (!check_replace_refs ||
> +	    (the_repository->objects->replace_map &&
> +	     the_repository->objects->replace_map->map.tablesize == 0))

Ah, we still have the same optimization, so this looks alright.  The
variable's name and semantics do need to be updated--I didn't check
but if this variable is exposed to the end users in any way, such a
fundamental sematic change may be hard to transition, though.

This series looks good so far.


^ permalink raw reply	[relevance 2%]

* Re: [PATCH 12/16] refs: store the main ref store inside the repository struct
  2018-04-09 22:45   ` [PATCH 12/16] refs: store the main ref store inside the repository struct Stefan Beller
  2018-04-09 23:24     ` Brandon Williams
@ 2018-04-10 14:02     ` Michael Haggerty
  2018-04-10 18:38       ` Stefan Beller
  1 sibling, 1 reply; 200+ results
From: Michael Haggerty @ 2018-04-10 14:02 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, jonathantanmy, sunshine, pclouds, l.s.r, sandals

On 04/10/2018 12:45 AM, Stefan Beller wrote:
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  refs.c       | 13 +++++--------
>  refs.h       |  4 +---
>  repository.h |  3 +++
>  3 files changed, 9 insertions(+), 11 deletions(-)
> 
> diff --git a/refs.c b/refs.c
> index f58b9fb7df..b5be754a97 100644
> --- a/refs.c
> +++ b/refs.c
> @@ -1608,9 +1608,6 @@ static struct ref_store_hash_entry *alloc_ref_store_hash_entry(
>  	return entry;
>  }
>  
> -/* A pointer to the ref_store for the main repository: */
> -static struct ref_store *main_ref_store;
> -
>  /* A hashmap of ref_stores, stored by submodule name: */
>  static struct hashmap submodule_ref_stores;
>  
> @@ -1652,13 +1649,13 @@ static struct ref_store *ref_store_init(const char *gitdir,
>  	return refs;
>  }
>  
> -struct ref_store *get_main_ref_store_the_repository(void)
> +struct ref_store *get_main_ref_store(struct repository *r)
>  {
> -	if (main_ref_store)
> -		return main_ref_store;
> +	if (r->main_ref_store)
> +		return r->main_ref_store;
>  
> -	main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
> -	return main_ref_store;
> +	r->main_ref_store = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
> +	return r->main_ref_store;
>  }
>  
>  /*
> diff --git a/refs.h b/refs.h
> index ab3d2bec2f..f5ab68c0ed 100644
> --- a/refs.h
> +++ b/refs.h
> @@ -760,9 +760,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
>  
>  int ref_storage_backend_exists(const char *name);
>  
> -#define get_main_ref_store(r) \
> -	get_main_ref_store_##r()
> -struct ref_store *get_main_ref_store_the_repository(void);
> +struct ref_store *get_main_ref_store(struct repository *r);
>  /*
>   * Return the ref_store instance for the specified submodule. For the
>   * main repository, use submodule==NULL; such a call cannot fail. For
> diff --git a/repository.h b/repository.h
> index 09df94a472..7d0710b273 100644
> --- a/repository.h
> +++ b/repository.h
> @@ -26,6 +26,9 @@ struct repository {
>  	 */
>  	struct raw_object_store *objects;
>  
> +	/* The store in which the refs are held. */
> +	struct ref_store *main_ref_store;
> +
>  	/*
>  	 * Path to the repository's graft file.
>  	 * Cannot be NULL after initialization.
> 

This also makes sense to me, as far as it goes. I have a few comments
and questions:

Why do you call the new member `main_ref_store`? Is there expected to be
some other `ref_store` associated with a repository?

I think the origin of the name `main_ref_store` was to distinguish it
from submodule ref stores. But presumably those will soon become the
"main" ref stores for their respective submodule repository objects,
right? So maybe calling things `repository.ref_store` and
`get_ref_store(repository)` would be appropriate.

There are some places in the reference code that only work with the main
repository. The ones that I can think of are:

* `ref_resolves_to_object()` depends on an object store.

* `peel_object()` and `ref_iterator_peel()` also have to look up objects
(in this case, tag objects).

* Anything that calls `files_assert_main_repository()` or depends on
`REF_STORE_MAIN` isn't implemented for other reference stores (usually,
I think, these are functions that depend on the object store).

Some of these things might be easy to generalize to non-main
repositories, but I didn't bother because AFAIK currently only the main
repository store is ever mutated.

You can move a now-obsolete comment above the definition of `struct
files_ref_store` if you haven't in some other patch (search for
"libification").

Hope that helps,
Michael

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 12/16] refs: store the main ref store inside the repository struct
  2018-04-10 14:02     ` Michael Haggerty
@ 2018-04-10 18:38       ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-10 18:38 UTC (permalink / raw)
  To: Michael Haggerty
  Cc: git, Jonathan Tan, sunshine, Duy Nguyen, René Scharfe,
	brian m. carlson

Hi Michael,

On Tue, Apr 10, 2018 at 7:02 AM, Michael Haggerty <mhagger@alum.mit.edu> wrote:

> This also makes sense to me, as far as it goes. I have a few comments
> and questions:
>
> Why do you call the new member `main_ref_store`? Is there expected to be
> some other `ref_store` associated with a repository?

I'll rename it in a reroll.

>
> I think the origin of the name `main_ref_store` was to distinguish it
> from submodule ref stores. But presumably those will soon become the
> "main" ref stores for their respective submodule repository objects,
> right?

I hope so.

> So maybe calling things `repository.ref_store` and
> `get_ref_store(repository)` would be appropriate.

ok.

>
> There are some places in the reference code that only work with the main
> repository. The ones that I can think of are:
>
> * `ref_resolves_to_object()` depends on an object store.
>
> * `peel_object()` and `ref_iterator_peel()` also have to look up objects
> (in this case, tag objects).
>
> * Anything that calls `files_assert_main_repository()` or depends on
> `REF_STORE_MAIN` isn't implemented for other reference stores (usually,
> I think, these are functions that depend on the object store).
>
> Some of these things might be easy to generalize to non-main
> repositories, but I didn't bother because AFAIK currently only the main
> repository store is ever mutated.
>
> You can move a now-obsolete comment above the definition of `struct
> files_ref_store` if you haven't in some other patch (search for
> "libification").

ok, I'll have a look at that.

My plan was to remove the submodule accessors from the refs API, and
mandate the access via

    repo_submodule_init(&submodule_repo, superproject_repo, path);
    sub_ref_store = get_ref_store(submodule_repo);

instead of also having

    sub_ref_store = get_submodule_ref_store(path);

as that would ease the refs API (and its internals potentially)
as well as avoiding errors with mixing up repositories. As the
construction of a submodule repository struct requires its
superproject repo, it helps avoiding pitfalls with nested
submodules IMO.

Thanks for the comments,
Stefan

>
> Hope that helps,
> Michael

^ permalink raw reply	[relevance 5%]

* [PATCH 05/15] replace-object: check_replace_refs is safe in multi repo environment
      [irrelevant]   ` <20180412002118.102976-1-sbeller@google.com>
@ 2018-04-12  0:21     ` Stefan Beller
  2018-04-12  0:21     ` [PATCH 06/15] refs: add repository argument to get_main_ref_store Stefan Beller
  2018-04-12  0:21     ` [PATCH 11/15] refs: store the main ref store inside the repository struct Stefan Beller
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-12  0:21 UTC (permalink / raw)
  To: sbeller; +Cc: git, jonathantanmy, l.s.r, pclouds, sandals, sunshine

In e1111cef23 (inline lookup_replace_object() calls, 2011-05-15) a shortcut
for checking the object replacement was added by setting check_replace_refs
to 0 once the replacements were evaluated to not exist. This works fine in
with the assumption of only one repository in existence.

The assumption won't hold true any more when we work on multiple instances
of a repository structs (e.g. one struct per submodule), as the first
repository to be inspected may have no replacements and would set the
global variable. Other repositories would then completely omit their
evaluation of replacements.

This reverts back the meaning of the flag `check_replace_refs` of
"Do we need to check with the lookup table?" to "Do we need to read
the replacement definition?", adding the bypassing logic to
lookup_replace_object after the replacement definition was read.
As with the original patch, delay the renaming of the global variable

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 environment.c    | 2 +-
 replace-object.h | 5 ++++-
 replace_object.c | 3 ---
 3 files changed, 5 insertions(+), 5 deletions(-)

diff --git a/environment.c b/environment.c
index 39b3d906c8..b991fc0a87 100644
--- a/environment.c
+++ b/environment.c
@@ -50,7 +50,7 @@ const char *editor_program;
 const char *askpass_program;
 const char *excludes_file;
 enum auto_crlf auto_crlf = AUTO_CRLF_FALSE;
-int check_replace_refs = 1;
+int check_replace_refs = 1; /* NEEDSWORK: rename to read_replace_refs */
 char *git_replace_ref_base;
 enum eol core_eol = EOL_UNSET;
 int global_conv_flags_eol = CONV_EOL_RNDTRP_WARN;
diff --git a/replace-object.h b/replace-object.h
index 15315311fb..dbc51265ec 100644
--- a/replace-object.h
+++ b/replace-object.h
@@ -3,6 +3,7 @@
 
 #include "oidmap.h"
 #include "repository.h"
+#include "object-store.h"
 
 struct replace_object {
 	struct oidmap_entry original;
@@ -23,7 +24,9 @@ extern const struct object_id *do_lookup_replace_object(const struct object_id *
  */
 static inline const struct object_id *lookup_replace_object(const struct object_id *oid)
 {
-	if (!check_replace_refs)
+	if (!check_replace_refs ||
+	    (the_repository->objects->replace_map &&
+	     the_repository->objects->replace_map->map.tablesize == 0))
 		return oid;
 	return do_lookup_replace_object(oid);
 }
diff --git a/replace_object.c b/replace_object.c
index 953fa9cc40..b2405f6027 100644
--- a/replace_object.c
+++ b/replace_object.c
@@ -41,9 +41,6 @@ static void prepare_replace_object(void)
 	oidmap_init(the_repository->objects->replace_map, 0);
 
 	for_each_replace_ref(register_replace_ref, NULL);
-
-	if (!the_repository->objects->replace_map->map.tablesize)
-		check_replace_refs = 0;
 }
 
 /* We allow "recursive" replacement. Only within reason, though */
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 6%]

* [PATCH 06/15] refs: add repository argument to get_main_ref_store
      [irrelevant]   ` <20180412002118.102976-1-sbeller@google.com>
  2018-04-12  0:21     ` [PATCH 05/15] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
@ 2018-04-12  0:21     ` Stefan Beller
  2018-04-12  0:21     ` [PATCH 11/15] refs: store the main ref store inside the repository struct Stefan Beller
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-12  0:21 UTC (permalink / raw)
  To: sbeller
  Cc: git, jonathantanmy, l.s.r, pclouds, sandals, sunshine, Jonathan Nieder

Add a repository argument to allow the get_main_ref_store caller
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/pack-refs.c       |  3 +-
 refs.c                    | 67 ++++++++++++++++++++-------------------
 refs.h                    |  4 ++-
 revision.c                |  5 +--
 t/helper/test-ref-store.c |  3 +-
 5 files changed, 44 insertions(+), 38 deletions(-)

diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c
index b106a392a4..f3353564f9 100644
--- a/builtin/pack-refs.c
+++ b/builtin/pack-refs.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "parse-options.h"
 #include "refs.h"
+#include "repository.h"
 
 static char const * const pack_refs_usage[] = {
 	N_("git pack-refs [<options>]"),
@@ -17,5 +18,5 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix)
 	};
 	if (parse_options(argc, argv, prefix, opts, pack_refs_usage, 0))
 		usage_with_options(pack_refs_usage, opts);
-	return refs_pack_refs(get_main_ref_store(), flags);
+	return refs_pack_refs(get_main_ref_store(the_repository), flags);
 }
diff --git a/refs.c b/refs.c
index 8b7a77fe5e..74d4ed97cb 100644
--- a/refs.c
+++ b/refs.c
@@ -13,6 +13,7 @@
 #include "tag.h"
 #include "submodule.h"
 #include "worktree.h"
+#include "repository.h"
 
 /*
  * List of all available backends
@@ -206,7 +207,7 @@ char *refs_resolve_refdup(struct ref_store *refs,
 char *resolve_refdup(const char *refname, int resolve_flags,
 		     struct object_id *oid, int *flags)
 {
-	return refs_resolve_refdup(get_main_ref_store(),
+	return refs_resolve_refdup(get_main_ref_store(the_repository),
 				   refname, resolve_flags,
 				   oid, flags);
 }
@@ -228,7 +229,7 @@ int refs_read_ref_full(struct ref_store *refs, const char *refname,
 
 int read_ref_full(const char *refname, int resolve_flags, struct object_id *oid, int *flags)
 {
-	return refs_read_ref_full(get_main_ref_store(), refname,
+	return refs_read_ref_full(get_main_ref_store(the_repository), refname,
 				  resolve_flags, oid, flags);
 }
 
@@ -375,7 +376,7 @@ int refs_for_each_tag_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_tag_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_tag_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_tag_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -385,7 +386,7 @@ int refs_for_each_branch_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_branch_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_branch_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_branch_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
@@ -395,7 +396,7 @@ int refs_for_each_remote_ref(struct ref_store *refs, each_ref_fn fn, void *cb_da
 
 int for_each_remote_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_remote_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_remote_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int head_ref_namespaced(each_ref_fn fn, void *cb_data)
@@ -730,7 +731,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 	struct strbuf err = STRBUF_INIT;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		return delete_pseudoref(refname, old_oid);
 	}
 
@@ -752,7 +753,7 @@ int refs_delete_ref(struct ref_store *refs, const char *msg,
 int delete_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid, unsigned int flags)
 {
-	return refs_delete_ref(get_main_ref_store(), msg, refname,
+	return refs_delete_ref(get_main_ref_store(the_repository), msg, refname,
 			       old_oid, flags);
 }
 
@@ -928,7 +929,7 @@ struct ref_transaction *ref_store_transaction_begin(struct ref_store *refs,
 
 struct ref_transaction *ref_transaction_begin(struct strbuf *err)
 {
-	return ref_store_transaction_begin(get_main_ref_store(), err);
+	return ref_store_transaction_begin(get_main_ref_store(the_repository), err);
 }
 
 void ref_transaction_free(struct ref_transaction *transaction)
@@ -1060,7 +1061,7 @@ int refs_update_ref(struct ref_store *refs, const char *msg,
 	int ret = 0;
 
 	if (ref_type(refname) == REF_TYPE_PSEUDOREF) {
-		assert(refs == get_main_ref_store());
+		assert(refs == get_main_ref_store(the_repository));
 		ret = write_pseudoref(refname, new_oid, old_oid, &err);
 	} else {
 		t = ref_store_transaction_begin(refs, &err);
@@ -1099,7 +1100,7 @@ int update_ref(const char *msg, const char *refname,
 	       const struct object_id *old_oid,
 	       unsigned int flags, enum action_on_err onerr)
 {
-	return refs_update_ref(get_main_ref_store(), msg, refname, new_oid,
+	return refs_update_ref(get_main_ref_store(the_repository), msg, refname, new_oid,
 			       old_oid, flags, onerr);
 }
 
@@ -1320,7 +1321,7 @@ int refs_head_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int head_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_head_ref(get_main_ref_store(), fn, cb_data);
+	return refs_head_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 struct ref_iterator *refs_ref_iterator_begin(
@@ -1379,7 +1380,7 @@ int refs_for_each_ref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_ref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_ref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
@@ -1390,7 +1391,7 @@ int refs_for_each_ref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_ref_in(const char *prefix, each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_ref_in(get_main_ref_store(), prefix, fn, cb_data);
+	return refs_for_each_ref_in(get_main_ref_store(the_repository), prefix, fn, cb_data);
 }
 
 int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsigned int broken)
@@ -1399,7 +1400,7 @@ int for_each_fullref_in(const char *prefix, each_ref_fn fn, void *cb_data, unsig
 
 	if (broken)
 		flag = DO_FOR_EACH_INCLUDE_BROKEN;
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       prefix, fn, 0, flag, cb_data);
 }
 
@@ -1416,7 +1417,7 @@ int refs_for_each_fullref_in(struct ref_store *refs, const char *prefix,
 
 int for_each_replace_ref(each_ref_fn fn, void *cb_data)
 {
-	return do_for_each_ref(get_main_ref_store(),
+	return do_for_each_ref(get_main_ref_store(the_repository),
 			       git_replace_ref_base, fn,
 			       strlen(git_replace_ref_base),
 			       DO_FOR_EACH_INCLUDE_BROKEN, cb_data);
@@ -1427,7 +1428,7 @@ int for_each_namespaced_ref(each_ref_fn fn, void *cb_data)
 	struct strbuf buf = STRBUF_INIT;
 	int ret;
 	strbuf_addf(&buf, "%srefs/", get_git_namespace());
-	ret = do_for_each_ref(get_main_ref_store(),
+	ret = do_for_each_ref(get_main_ref_store(the_repository),
 			      buf.buf, fn, 0, 0, cb_data);
 	strbuf_release(&buf);
 	return ret;
@@ -1441,7 +1442,7 @@ int refs_for_each_rawref(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_rawref(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_rawref(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_rawref(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_read_raw_ref(struct ref_store *ref_store,
@@ -1547,7 +1548,7 @@ const char *refs_resolve_ref_unsafe(struct ref_store *refs,
 /* backend functions */
 int refs_init_db(struct strbuf *err)
 {
-	struct ref_store *refs = get_main_ref_store();
+	struct ref_store *refs = get_main_ref_store(the_repository);
 
 	return refs->be->init_db(refs, err);
 }
@@ -1555,7 +1556,7 @@ int refs_init_db(struct strbuf *err)
 const char *resolve_ref_unsafe(const char *refname, int resolve_flags,
 			       struct object_id *oid, int *flags)
 {
-	return refs_resolve_ref_unsafe(get_main_ref_store(), refname,
+	return refs_resolve_ref_unsafe(get_main_ref_store(the_repository), refname,
 				       resolve_flags, oid, flags);
 }
 
@@ -1651,7 +1652,7 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store(void)
+struct ref_store *get_main_ref_store_the_repository(void)
 {
 	if (main_ref_store)
 		return main_ref_store;
@@ -1726,7 +1727,7 @@ struct ref_store *get_worktree_ref_store(const struct worktree *wt)
 	const char *id;
 
 	if (wt->is_current)
-		return get_main_ref_store();
+		return get_main_ref_store(the_repository);
 
 	id = wt->id ? wt->id : "/";
 	refs = lookup_ref_store_map(&worktree_ref_stores, id);
@@ -1782,7 +1783,7 @@ int refs_peel_ref(struct ref_store *refs, const char *refname,
 
 int peel_ref(const char *refname, struct object_id *oid)
 {
-	return refs_peel_ref(get_main_ref_store(), refname, oid);
+	return refs_peel_ref(get_main_ref_store(the_repository), refname, oid);
 }
 
 int refs_create_symref(struct ref_store *refs,
@@ -1798,7 +1799,7 @@ int refs_create_symref(struct ref_store *refs,
 int create_symref(const char *ref_target, const char *refs_heads_master,
 		  const char *logmsg)
 {
-	return refs_create_symref(get_main_ref_store(), ref_target,
+	return refs_create_symref(get_main_ref_store(the_repository), ref_target,
 				  refs_heads_master, logmsg);
 }
 
@@ -2006,7 +2007,7 @@ int refs_for_each_reflog(struct ref_store *refs, each_ref_fn fn, void *cb_data)
 
 int for_each_reflog(each_ref_fn fn, void *cb_data)
 {
-	return refs_for_each_reflog(get_main_ref_store(), fn, cb_data);
+	return refs_for_each_reflog(get_main_ref_store(the_repository), fn, cb_data);
 }
 
 int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
@@ -2021,7 +2022,7 @@ int refs_for_each_reflog_ent_reverse(struct ref_store *refs,
 int for_each_reflog_ent_reverse(const char *refname, each_reflog_ent_fn fn,
 				void *cb_data)
 {
-	return refs_for_each_reflog_ent_reverse(get_main_ref_store(),
+	return refs_for_each_reflog_ent_reverse(get_main_ref_store(the_repository),
 						refname, fn, cb_data);
 }
 
@@ -2034,7 +2035,7 @@ int refs_for_each_reflog_ent(struct ref_store *refs, const char *refname,
 int for_each_reflog_ent(const char *refname, each_reflog_ent_fn fn,
 			void *cb_data)
 {
-	return refs_for_each_reflog_ent(get_main_ref_store(), refname,
+	return refs_for_each_reflog_ent(get_main_ref_store(the_repository), refname,
 					fn, cb_data);
 }
 
@@ -2045,7 +2046,7 @@ int refs_reflog_exists(struct ref_store *refs, const char *refname)
 
 int reflog_exists(const char *refname)
 {
-	return refs_reflog_exists(get_main_ref_store(), refname);
+	return refs_reflog_exists(get_main_ref_store(the_repository), refname);
 }
 
 int refs_create_reflog(struct ref_store *refs, const char *refname,
@@ -2057,7 +2058,7 @@ int refs_create_reflog(struct ref_store *refs, const char *refname,
 int safe_create_reflog(const char *refname, int force_create,
 		       struct strbuf *err)
 {
-	return refs_create_reflog(get_main_ref_store(), refname,
+	return refs_create_reflog(get_main_ref_store(the_repository), refname,
 				  force_create, err);
 }
 
@@ -2068,7 +2069,7 @@ int refs_delete_reflog(struct ref_store *refs, const char *refname)
 
 int delete_reflog(const char *refname)
 {
-	return refs_delete_reflog(get_main_ref_store(), refname);
+	return refs_delete_reflog(get_main_ref_store(the_repository), refname);
 }
 
 int refs_reflog_expire(struct ref_store *refs,
@@ -2091,7 +2092,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 		  reflog_expiry_cleanup_fn cleanup_fn,
 		  void *policy_cb_data)
 {
-	return refs_reflog_expire(get_main_ref_store(),
+	return refs_reflog_expire(get_main_ref_store(the_repository),
 				  refname, oid, flags,
 				  prepare_fn, should_prune_fn,
 				  cleanup_fn, policy_cb_data);
@@ -2114,7 +2115,7 @@ int refs_delete_refs(struct ref_store *refs, const char *msg,
 int delete_refs(const char *msg, struct string_list *refnames,
 		unsigned int flags)
 {
-	return refs_delete_refs(get_main_ref_store(), msg, refnames, flags);
+	return refs_delete_refs(get_main_ref_store(the_repository), msg, refnames, flags);
 }
 
 int refs_rename_ref(struct ref_store *refs, const char *oldref,
@@ -2125,7 +2126,7 @@ int refs_rename_ref(struct ref_store *refs, const char *oldref,
 
 int rename_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_rename_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_rename_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
 
 int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
@@ -2136,5 +2137,5 @@ int refs_copy_existing_ref(struct ref_store *refs, const char *oldref,
 
 int copy_existing_ref(const char *oldref, const char *newref, const char *logmsg)
 {
-	return refs_copy_existing_ref(get_main_ref_store(), oldref, newref, logmsg);
+	return refs_copy_existing_ref(get_main_ref_store(the_repository), oldref, newref, logmsg);
 }
diff --git a/refs.h b/refs.h
index 01be5ae32f..0d013377ce 100644
--- a/refs.h
+++ b/refs.h
@@ -758,7 +758,9 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-struct ref_store *get_main_ref_store(void);
+#define get_main_ref_store(r) \
+	get_main_ref_store_##r()
+struct ref_store *get_main_ref_store_the_repository(void);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/revision.c b/revision.c
index b42c836d7a..1cff11833e 100644
--- a/revision.c
+++ b/revision.c
@@ -6,6 +6,7 @@
 #include "diff.h"
 #include "refs.h"
 #include "revision.h"
+#include "repository.h"
 #include "graph.h"
 #include "grep.h"
 #include "reflog-walk.h"
@@ -1285,7 +1286,7 @@ void add_reflogs_to_pending(struct rev_info *revs, unsigned flags)
 
 	cb.all_revs = revs;
 	cb.all_flags = flags;
-	cb.refs = get_main_ref_store();
+	cb.refs = get_main_ref_store(the_repository);
 	for_each_reflog(handle_one_reflog, &cb);
 
 	if (!revs->single_worktree)
@@ -2176,7 +2177,7 @@ static int handle_revision_pseudo_opt(const char *submodule,
 			die("BUG: --single-worktree cannot be used together with submodule");
 		refs = get_submodule_ref_store(submodule);
 	} else
-		refs = get_main_ref_store();
+		refs = get_main_ref_store(the_repository);
 
 	/*
 	 * NOTE!
diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c
index 7314b5943e..e8c328ec0e 100644
--- a/t/helper/test-ref-store.c
+++ b/t/helper/test-ref-store.c
@@ -2,6 +2,7 @@
 #include "refs.h"
 #include "worktree.h"
 #include "object-store.h"
+#include "repository.h"
 
 static const char *notnull(const char *arg, const char *name)
 {
@@ -22,7 +23,7 @@ static const char **get_store(const char **argv, struct ref_store **refs)
 	if (!argv[0]) {
 		die("ref store required");
 	} else if (!strcmp(argv[0], "main")) {
-		*refs = get_main_ref_store();
+		*refs = get_main_ref_store(the_repository);
 	} else if (skip_prefix(argv[0], "submodule:", &gitdir)) {
 		struct strbuf sb = STRBUF_INIT;
 		int ret;
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 7%]

* [PATCH 11/15] refs: store the main ref store inside the repository struct
      [irrelevant]   ` <20180412002118.102976-1-sbeller@google.com>
  2018-04-12  0:21     ` [PATCH 05/15] replace-object: check_replace_refs is safe in multi repo environment Stefan Beller
  2018-04-12  0:21     ` [PATCH 06/15] refs: add repository argument to get_main_ref_store Stefan Beller
@ 2018-04-12  0:21     ` Stefan Beller
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-12  0:21 UTC (permalink / raw)
  To: sbeller; +Cc: git, jonathantanmy, l.s.r, pclouds, sandals, sunshine

This moves the 'main_ref_store', which was a global variable in refs.c
into the repository struct.

This patch does not deal with the parts in the refs subsystem which deal
with the submodules there. A later patch needs to get rid of the submodule
exposure in the refs API, such as 'get_submodule_ref_store(path)'.

Acked-by: Michael Haggerty <mhagger@alum.mit.edu>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 refs.c               | 13 +++++--------
 refs.h               |  4 +---
 refs/files-backend.c |  4 ----
 repository.h         |  3 +++
 4 files changed, 9 insertions(+), 15 deletions(-)

diff --git a/refs.c b/refs.c
index f58b9fb7df..36df1bc73a 100644
--- a/refs.c
+++ b/refs.c
@@ -1608,9 +1608,6 @@ static struct ref_store_hash_entry *alloc_ref_store_hash_entry(
 	return entry;
 }
 
-/* A pointer to the ref_store for the main repository: */
-static struct ref_store *main_ref_store;
-
 /* A hashmap of ref_stores, stored by submodule name: */
 static struct hashmap submodule_ref_stores;
 
@@ -1652,13 +1649,13 @@ static struct ref_store *ref_store_init(const char *gitdir,
 	return refs;
 }
 
-struct ref_store *get_main_ref_store_the_repository(void)
+struct ref_store *get_main_ref_store(struct repository *r)
 {
-	if (main_ref_store)
-		return main_ref_store;
+	if (r->refs)
+		return r->refs;
 
-	main_ref_store = ref_store_init(get_git_dir(), REF_STORE_ALL_CAPS);
-	return main_ref_store;
+	r->refs = ref_store_init(r->gitdir, REF_STORE_ALL_CAPS);
+	return r->refs;
 }
 
 /*
diff --git a/refs.h b/refs.h
index ab3d2bec2f..f5ab68c0ed 100644
--- a/refs.h
+++ b/refs.h
@@ -760,9 +760,7 @@ int reflog_expire(const char *refname, const struct object_id *oid,
 
 int ref_storage_backend_exists(const char *name);
 
-#define get_main_ref_store(r) \
-	get_main_ref_store_##r()
-struct ref_store *get_main_ref_store_the_repository(void);
+struct ref_store *get_main_ref_store(struct repository *r);
 /*
  * Return the ref_store instance for the specified submodule. For the
  * main repository, use submodule==NULL; such a call cannot fail. For
diff --git a/refs/files-backend.c b/refs/files-backend.c
index bec8e30e9e..5c76a75817 100644
--- a/refs/files-backend.c
+++ b/refs/files-backend.c
@@ -61,10 +61,6 @@ struct ref_lock {
 	struct object_id old_oid;
 };
 
-/*
- * Future: need to be in "struct repository"
- * when doing a full libification.
- */
 struct files_ref_store {
 	struct ref_store base;
 	unsigned int store_flags;
diff --git a/repository.h b/repository.h
index 09df94a472..e6e00f541b 100644
--- a/repository.h
+++ b/repository.h
@@ -26,6 +26,9 @@ struct repository {
 	 */
 	struct raw_object_store *objects;
 
+	/* The store in which the refs are held. */
+	struct ref_store *refs;
+
 	/*
 	 * Path to the repository's graft file.
 	 * Cannot be NULL after initialization.
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 11%]

* Re: [RFC 00/10] Make .the gitmodules file path configurable
      [irrelevant] <20180412222047.5716-1-ao2@ao2.it>
@ 2018-04-12 23:36 ` Stefan Beller
  2018-04-16 11:33   ` Antonio Ospite
      [irrelevant] ` <20180412222047.5716-2-ao2@ao2.it>
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-12 23:36 UTC (permalink / raw)
  To: Antonio Ospite, Brandon Williams; +Cc: git, Richard Hartmann

Hi Antonio,

On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> Hi,
>
> vcsh[1] uses bare git repositories and detached work-trees to manage
> *distinct* sets of configuration files directly into $HOME.
>
> In general, submodules have worked perfectly fine with detached
> work-trees for some time[2,3,4].
>
> However when multiple repositories take turns using the same directory
> as their work-tree, and more than one of them want to use submodules,
> there could still be conflicts about the '.gitmodules' file because git
> hardcodes this path.
>
> For comparison, in case of '.gitignore' a similar conflict might arise,
> but git has alternative ways to specify exclude files, so vcsh solves
> this by setting core.excludesFile for each repository and track ignored
> files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).
>
> This is currently not possible with submodules configuration.

So far I agree w.r.t. Gits capabilities.

> So this series proposes a mechanism to set an alternative path for the
> submodules configuration file (from now on "gitmodules file").

That sounds interesting, so let's read on.

> Patches are based on fe0a9eaf31dd0c349ae4308498c33a5c3794b293.

... which is the current master branch by Junio.

> In commit 4c0eeafe4755 (cache.h: add GITMODULES_FILE macro)[5] the
> gitmodules file path definition was centralized, AFAIU this was done
> mainly to prevent typos, as checking a symbolic constant is something
> the compiler will do for us.

+cc Brandon author of said patch.

Digging up the discussion, this was indeed only done to prevent typos.
https://public-inbox.org/git/20170802172633.GA36159@google.com/

> Expanding on that change the first patch in the series makes the path
> customizable exposing a 'core.submodulesFile' configuration setting.

I guess the similarity to core.ignoreFile is desired here. Although these
mechanisms are very different.

> The new configuration setting can be used to set an *alternative*
> location for the gitmodules file; IMHO there is no need to provide
> *additional* locations like in the case of exclude files.

I think there *may* be a need for additional files.
Currently there is only the .gitmodules file and the configuration
in .git/config overriding settings from .gitmodules.

There was some discussion on the mailing list in the past, which
presented a intermediate layer in between these two places, in
a special ref, such that:
    base is in .gitmodules
    overwritten via refs/meta/submodules:.gitmodules
    overwritten via the .git/config

The intermediate would be a config file that is tracked on another
ref. This (a) decouples main project history from submodule history
and (b) makes it easier to distribute as it is part of the repository.

For example (a) is desired if you dig up an old project and the
submodules have all moved from one git hosting provider to another.
Another example would be when you fork a project with submodules
and don't want to mess with the main history but you just want to
adjust the submodule URLs. That is possible with such an intermediate
additional place.

For (b) you can imagine the fork that you want to distribute in your
community and you don't want to tell everyone to change the
submodule URLs, but instead you can provide them with a prepared
.gitmodules file, that they have to place into that special ref (which
can be done via fetching).

I digress as these ideas seem to be orthogonal to your patch series,
just FYI. prior discussion starting at:
https://public-inbox.org/git/1462317985-640-1-git-send-email-sbeller@google.com/
I recall there was a better discussion even prior to that, but have no
link handy.


> For instance vcsh could set the location to
> '~/.gitmodules.d/$VCSH_REPO_NAME' to avoid conflicts.
>
> Since the gitmodules file is meant to be checked in into the repository,
> the overridden file path should be relative to the work-tree; is there
> a way to enforce this constraint at run time (i.e. validate the config
> value), or is it enough to have it documented?

I think we'd want to check at run time, if we need this constraint.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path
      [irrelevant] ` <20180412222047.5716-2-ao2@ao2.it>
@ 2018-04-12 23:50   ` Stefan Beller
  2018-04-16 16:37     ` Antonio Ospite
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-12 23:50 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Richard Hartmann

Hi Antonio,

On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> When multiple repositories with detached work-trees take turns using the
> same directory as their work-tree, and more than one of them want to use
> submodules, there will be conflicts about the '.gitmodules' file.

unlike other files which would not conflict?
There might be file names such as LICENSE, Readme.md etc,
which are common enough that they would produce conflicts as well?
I find this argument on its own rather weak. ("Just delete everything in
the working dir before using it with another repository"). I might be
missing a crucial bit here?

> git hardcodes this path so it's not possible to override its location on
> a per-repository basis to allow such repositories to coexists
> peacefully.
>
> Make the path of the "gitmodules file" customizable exposing
> a 'core.submodulesFile' configuration setting.
>
> The default value will still be '.gitmodules' when 'core.submodulesFile'
> is not set.

ok.


> --- a/cache.h
> +++ b/cache.h
> @@ -1774,6 +1774,7 @@ extern void prepare_pager_args(struct child_process *, const char *pager);
>  extern const char *editor_program;
>  extern const char *askpass_program;
>  extern const char *excludes_file;
> +extern const char *submodules_file;

Could you place this variable in repository.h in struct repository?
(Some developers currently try to move any global state to that place,
as that makes working with e.g. nested submodules easier in-process
and you would not need to spawn processes for submodules)

Once migrated to the repository struct mentioned above, you'd access
it via the_repository->submodules_file for the main repository.


> diff --git a/git-submodule.sh b/git-submodule.sh
> index 24914963c..610fd0dc5 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -71,7 +71,9 @@ get_submodule_config () {
>         value=$(git config submodule."$name"."$option")
>         if test -z "$value"
>         then
> -               value=$(git config -f .gitmodules submodule."$name"."$option")
> +               gitmodules_file=$(git config core.submodulesfile)
> +               : ${gitmodules_file:=.gitmodules}
> +               value=$(git config -f "$gitmodules_file" submodule."$name"."$option")

I wonder if it would be cheaper to write a special config lookup now, e.g.
in builtin/submodule--helper.c we could have a "config-from-gitmodules"
subcommand that is looking up the modules file and then running the config
on that file.

I am surprised how little access of the .gitmodules is left in git-submodule.sh
(which is partially ported to the builtin/submodule--helper.c)

> diff --git a/submodule-config.c b/submodule-config.c
> index 3f2075764..8a3396ade 100644
> --- a/submodule-config.c
> +++ b/submodule-config.c
> @@ -468,7 +468,7 @@ static int gitmodule_oid_from_commit(const struct object_id *treeish_name,
>                 return 1;
>         }
>
> -       strbuf_addf(rev, "%s:.gitmodules", oid_to_hex(treeish_name));
> +       strbuf_addf(rev, "%s:%s", oid_to_hex(treeish_name), submodules_file);
>         if (get_oid(rev->buf, gitmodules_oid) >= 0)
>                 ret = 1;
>
> @@ -583,7 +583,7 @@ void repo_read_gitmodules(struct repository *repo)
>                 if (repo_read_index(repo) < 0)
>                         return;
>
> -               gitmodules = repo_worktree_path(repo, GITMODULES_FILE);
> +               gitmodules = repo_worktree_path(repo, submodules_file);
>
>                 if (!is_gitmodules_unmerged(repo->index))
>                         git_config_from_file(gitmodules_cb, gitmodules, repo);
> diff --git a/submodule.c b/submodule.c
> index 9a50168b2..2afbdb644 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -36,13 +36,13 @@ static struct oid_array ref_tips_after_fetch;
>   */
>  int is_gitmodules_unmerged(const struct index_state *istate)
>  {
> -       int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));
> +       int pos = index_name_pos(istate, submodules_file, strlen(submodules_file));

Ah, regarding the coverletter: This clearly assumes the modules
file is in the tree. So at least here we would make an exception
for files outside the tree to either not check for un-merged-ness or
disallow that case entirely.



There are quite a few functions in submodule.c which access the new global. :/
So moving them to the_repository should be fine, but eventually (not
in this series)
all these functions would would want to take a repository argument as well
such that they work on more than the_repository.

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* Re: [RFC 02/10] submodule: fix getting custom gitmodule file in fetch command
      [irrelevant] ` <20180412222047.5716-3-ao2@ao2.it>
@ 2018-04-12 23:55   ` Stefan Beller
  2018-04-16 16:18     ` Antonio Ospite
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-12 23:55 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Richard Hartmann

Hi Antonio,

the subject line could also be
  fetch: fix custom submodule location
as I was not expecting this patch from the subject line.

On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> Import the default git configuration before accessing the gitmodules
> file.
>
> This is important when a value different from the default one is set via
> the 'core.submodulesfile' config.
>
> Signed-off-by: Antonio Ospite <ao2@ao2.it>
> ---
>  builtin/fetch.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index dcdfc66f0..d56636f74 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1428,8 +1428,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
>         for (i = 1; i < argc; i++)
>                 strbuf_addf(&default_rla, " %s", argv[i]);
>
> -       config_from_gitmodules(gitmodules_fetch_config, NULL);
>         git_config(git_fetch_config, NULL);
> +       config_from_gitmodules(gitmodules_fetch_config, NULL);

Wouldn't this break the overwriting behavior?
e.g. you configure a submodule URL in .gitmodules and then override it
in .git/config,
then we'd currently first load config from the modules file and then override it
in git_config(git_fetch_config,..), whereas swapping them might have
unintended consequences? Do the tests pass with this patch?

^ permalink raw reply	[relevance 8%]

* Re: [RFC 00/10] Make .the gitmodules file path configurable
  2018-04-12 23:36 ` [RFC 00/10] Make .the gitmodules file path configurable Stefan Beller
@ 2018-04-16 11:33   ` Antonio Ospite
  2018-04-16 19:22     ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Antonio Ospite @ 2018-04-16 11:33 UTC (permalink / raw)
  To: git; +Cc: Brandon Williams, git, Richard Hartmann

On Thu, 12 Apr 2018 16:36:32 -0700
Stefan Beller <sbeller@google.com> wrote:

> Hi Antonio,
>

Hi Stefan,

> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> > Hi,
> >
> > vcsh[1] uses bare git repositories and detached work-trees to manage
> > *distinct* sets of configuration files directly into $HOME.
> >
> > In general, submodules have worked perfectly fine with detached
> > work-trees for some time[2,3,4].
> >
> > However when multiple repositories take turns using the same directory
> > as their work-tree, and more than one of them want to use submodules,
> > there could still be conflicts about the '.gitmodules' file because git
> > hardcodes this path.
> >
[...]
> 
> > So this series proposes a mechanism to set an alternative path for the
> > submodules configuration file (from now on "gitmodules file").
> 
[...]
> > Expanding on that change the first patch in the series makes the path
> > customizable exposing a 'core.submodulesFile' configuration setting.
> 
> I guess the similarity to core.ignoreFile is desired here. Although these
> mechanisms are very different.
>

The option name is similar to 'core.excludesFile' also because, when I
started, I first looked at how multiple ignore files were used, so I
may have been influenced by that.

I acknowledge that the two mechanisms are different, in particular
because git *writes* the gitmodules file itself.

I am not opposed to change the name, something like
'core.submodulesConfigFile' might highlight that the syntax of the file
is the one of git config, but I think it's too long.

> > The new configuration setting can be used to set an *alternative*
> > location for the gitmodules file; IMHO there is no need to provide
> > *additional* locations like in the case of exclude files.
> 
> I think there *may* be a need for additional files.
> Currently there is only the .gitmodules file and the configuration
> in .git/config overriding settings from .gitmodules.
> 
> There was some discussion on the mailing list in the past, which
> presented a intermediate layer in between these two places, in
> a special ref, such that:
>     base is in .gitmodules
>     overwritten via refs/meta/submodules:.gitmodules
>     overwritten via the .git/config
> 
> The intermediate would be a config file that is tracked on another
> ref. This (a) decouples main project history from submodule history
> and (b) makes it easier to distribute as it is part of the repository.
> 
> For example (a) is desired if you dig up an old project and the
> submodules have all moved from one git hosting provider to another.
> Another example would be when you fork a project with submodules
> and don't want to mess with the main history but you just want to
> adjust the submodule URLs. That is possible with such an intermediate
> additional place.
> 
> For (b) you can imagine the fork that you want to distribute in your
> community and you don't want to tell everyone to change the
> submodule URLs, but instead you can provide them with a prepared
> .gitmodules file, that they have to place into that special ref (which
> can be done via fetching).
> 
> I digress as these ideas seem to be orthogonal to your patch series,
> just FYI. prior discussion starting at:
> https://public-inbox.org/git/1462317985-640-1-git-send-email-sbeller@google.com/
> I recall there was a better discussion even prior to that, but have no
> link handy.
>

Just to understand, is that 'refs/meta/submodules:.gitmodules' file
meant to be managed manually? Or do you imagine some options to
instruct "git submodule" to *write* to that file instead of .gitmodules?

In the latter case your idea could cover my use case too, couldn't it?

But most importantly, is this realistically going to be added?
Currently I am not that familiar with the git code base to do it
myself, and I've never explicitly dealt with special refs in git.

The approach of this patch series is a lot simpler, and something I can
work on in my spare time.

Presumably (b) could even be partially supported with my changes, by
having both '.gitmodules' and some custom '.gitmodules-alternative'
files in the repository and tell users to clone with:

  git clone --config core.submodulesFile=.gitmodules-alternative URL

Not as clean as your idea but doable.

[...]
> > Since the gitmodules file is meant to be checked in into the repository,
> > the overridden file path should be relative to the work-tree; is there
> > a way to enforce this constraint at run time (i.e. validate the config
> > value), or is it enough to have it documented?
> 
> I think we'd want to check at run time, if we need this constraint.
> 

I'll look into it once we decide which the way to go.

Thank you.

Ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 4%]

* Re: [RFC 02/10] submodule: fix getting custom gitmodule file in fetch command
  2018-04-12 23:55   ` [RFC 02/10] submodule: fix getting custom gitmodule file in fetch command Stefan Beller
@ 2018-04-16 16:18     ` Antonio Ospite
  2018-04-16 19:23       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Antonio Ospite @ 2018-04-16 16:18 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Richard Hartmann

On Thu, 12 Apr 2018 16:55:15 -0700
Stefan Beller <sbeller@google.com> wrote:

> Hi Antonio,
> 
> the subject line could also be
>   fetch: fix custom submodule location
> as I was not expecting this patch from the subject line.
>

OK.

> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> > Import the default git configuration before accessing the gitmodules
> > file.
> >
> > This is important when a value different from the default one is set via
> > the 'core.submodulesfile' config.
> >
> > Signed-off-by: Antonio Ospite <ao2@ao2.it>
> > ---
> >  builtin/fetch.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/builtin/fetch.c b/builtin/fetch.c
> > index dcdfc66f0..d56636f74 100644
> > --- a/builtin/fetch.c
> > +++ b/builtin/fetch.c
> > @@ -1428,8 +1428,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)
> >         for (i = 1; i < argc; i++)
> >                 strbuf_addf(&default_rla, " %s", argv[i]);
> >
> > -       config_from_gitmodules(gitmodules_fetch_config, NULL);
> >         git_config(git_fetch_config, NULL);
> > +       config_from_gitmodules(gitmodules_fetch_config, NULL);
> 
> Wouldn't this break the overwriting behavior?
> e.g. you configure a submodule URL in .gitmodules and then override it
> in .git/config,
> then we'd currently first load config from the modules file and then override it
> in git_config(git_fetch_config,..), whereas swapping them might have
> unintended consequences? Do the tests pass with this patch?

The patch changes the current behavior indeed, but it does not break
any of the existing tests.

Anyway, to be on the safe side, I could load only the
core.submodulesFile option from the global configuration, maybe even in
config_from_gitmodules() itself, before accessing the gitmodules file,
but I still don't know how to do that.

Is there an API to just load one config setting?

Ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 4%]

* Re: [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path
  2018-04-12 23:50   ` [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path Stefan Beller
@ 2018-04-16 16:37     ` Antonio Ospite
  2018-04-16 21:22       ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Antonio Ospite @ 2018-04-16 16:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Richard Hartmann

On Thu, 12 Apr 2018 16:50:03 -0700
Stefan Beller <sbeller@google.com> wrote:

> Hi Antonio,
> 
> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> > When multiple repositories with detached work-trees take turns using the
> > same directory as their work-tree, and more than one of them want to use
> > submodules, there will be conflicts about the '.gitmodules' file.
> 
> unlike other files which would not conflict?
> There might be file names such as LICENSE, Readme.md etc,
> which are common enough that they would produce conflicts as well?
> I find this argument on its own rather weak. ("Just delete everything in
> the working dir before using it with another repository"). I might be
> missing a crucial bit here?
>

All the vcsh repositories _share_ the same work-tree; they may control
it taking turns but, in general, all files are meant to be checked out
at all times as the basic use case is: *distinct* sets of config files.

Maybe saying that the repositories "take turns" is confusing.
It's an unnecessary information, so I will omit that part form the
commit message.

After your question I've done some research and I've seen other vcsh
users managing conflicting LICENSE and README files using git
sparse-checkouts, to have these files in the single repositories but
not checked out in the shared work-tree:
https://github.com/RichiH/vcsh/issues/120#issuecomment-42639619
https://github.com/jwhitley/vcsh-root/commit/30b0d495c2cbe47ae9617ace9c2c14720d961d78

However I guess that my point here is that the gitmodules file is
something that influences git behavior so it should not be on the user's
shoulder to manage conflicts for it, and most importantly it needs to
be checked out for git to access it, doesn't it?

> > git hardcodes this path so it's not possible to override its location on
> > a per-repository basis to allow such repositories to coexists
> > peacefully.
> >
> > Make the path of the "gitmodules file" customizable exposing
> > a 'core.submodulesFile' configuration setting.
> >
> > The default value will still be '.gitmodules' when 'core.submodulesFile'
> > is not set.
> 
> ok.
> 
> 
> > --- a/cache.h
> > +++ b/cache.h
> > @@ -1774,6 +1774,7 @@ extern void prepare_pager_args(struct child_process *, const char *pager);
> >  extern const char *editor_program;
> >  extern const char *askpass_program;
> >  extern const char *excludes_file;
> > +extern const char *submodules_file;
> 
> Could you place this variable in repository.h in struct repository?
> (Some developers currently try to move any global state to that place,
> as that makes working with e.g. nested submodules easier in-process
> and you would not need to spawn processes for submodules)
> 
> Once migrated to the repository struct mentioned above, you'd access
> it via the_repository->submodules_file for the main repository.
>

OK, thanks, I didn't like the global variable either, I was just copying
from excludes_file.

> 
> > diff --git a/git-submodule.sh b/git-submodule.sh
> > index 24914963c..610fd0dc5 100755
> > --- a/git-submodule.sh
> > +++ b/git-submodule.sh
> > @@ -71,7 +71,9 @@ get_submodule_config () {
> >         value=$(git config submodule."$name"."$option")
> >         if test -z "$value"
> >         then
> > -               value=$(git config -f .gitmodules submodule."$name"."$option")
> > +               gitmodules_file=$(git config core.submodulesfile)
> > +               : ${gitmodules_file:=.gitmodules}
> > +               value=$(git config -f "$gitmodules_file" submodule."$name"."$option")
> 
> I wonder if it would be cheaper to write a special config lookup now, e.g.
> in builtin/submodule--helper.c we could have a "config-from-gitmodules"
> subcommand that is looking up the modules file and then running the config
> on that file.
>

Can you give an example from the user point of view of such a
"config-from-gitmodules" command?

I might look into it, but that can also be a followup change. 

> I am surprised how little access of the .gitmodules is left in git-submodule.sh
> (which is partially ported to the builtin/submodule--helper.c)
> 
> > diff --git a/submodule-config.c b/submodule-config.c
> > index 3f2075764..8a3396ade 100644
> > --- a/submodule-config.c
> > +++ b/submodule-config.c
> > @@ -468,7 +468,7 @@ static int gitmodule_oid_from_commit(const struct object_id *treeish_name,
> >                 return 1;
> >         }
> >
> > -       strbuf_addf(rev, "%s:.gitmodules", oid_to_hex(treeish_name));
> > +       strbuf_addf(rev, "%s:%s", oid_to_hex(treeish_name), submodules_file);
> >         if (get_oid(rev->buf, gitmodules_oid) >= 0)
> >                 ret = 1;
> >
> > @@ -583,7 +583,7 @@ void repo_read_gitmodules(struct repository *repo)
> >                 if (repo_read_index(repo) < 0)
> >                         return;
> >
> > -               gitmodules = repo_worktree_path(repo, GITMODULES_FILE);
> > +               gitmodules = repo_worktree_path(repo, submodules_file);
> >
> >                 if (!is_gitmodules_unmerged(repo->index))
> >                         git_config_from_file(gitmodules_cb, gitmodules, repo);
> > diff --git a/submodule.c b/submodule.c
> > index 9a50168b2..2afbdb644 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -36,13 +36,13 @@ static struct oid_array ref_tips_after_fetch;
> >   */
> >  int is_gitmodules_unmerged(const struct index_state *istate)
> >  {
> > -       int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));
> > +       int pos = index_name_pos(istate, submodules_file, strlen(submodules_file));
> 
> Ah, regarding the coverletter: This clearly assumes the modules
> file is in the tree. So at least here we would make an exception
> for files outside the tree to either not check for un-merged-ness or
> disallow that case entirely.
>

Sorry I am not sure I follow what you are saying here, keep in mind
that I am new to git internals.

Do you mean that, even if we ensure (in
config.c::git_default_core_config) that only paths relative to
the work-tree are allowed, we still have to check here that the
constraint is respected? And is so, why?

> There are quite a few functions in submodule.c which access the new global. :/
> So moving them to the_repository should be fine, but eventually (not
> in this series)
> all these functions would would want to take a repository argument as well
> such that they work on more than the_repository.
> 

I will surely get rid of the global variable, but about changing the
functions signatures I don't feel like promising anything just yet.

Thanks,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 4%]

* Re: [RFC 00/10] Make .the gitmodules file path configurable
  2018-04-16 11:33   ` Antonio Ospite
@ 2018-04-16 19:22     ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-16 19:22 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Brandon Williams, Richard Hartmann

Hi Antonio,

> I acknowledge that the two mechanisms are different, in particular
> because git *writes* the gitmodules file itself.
>
> I am not opposed to change the name, something like
> 'core.submodulesConfigFile' might highlight that the syntax of the file
> is the one of git config, but I think it's too long.

I was not speculating on a better name, but on the nature of the
configuration, gitignore is additive and errors are easy to see, but in
gitmodules, there is only one "correct" path+name, so the problem
space is not additive, rather we can have a discussion where we get
the correct config from with low odds of errors.

>> > The new configuration setting can be used to set an *alternative*
>> > location for the gitmodules file; IMHO there is no need to provide
>> > *additional* locations like in the case of exclude files.
>>
>> I think there *may* be a need for additional files.
>> Currently there is only the .gitmodules file and the configuration
>> in .git/config overriding settings from .gitmodules.
>>
>> There was some discussion on the mailing list in the past, which
>> presented a intermediate layer in between these two places, in
>> a special ref, such that:
>>     base is in .gitmodules
>>     overwritten via refs/meta/submodules:.gitmodules
>>     overwritten via the .git/config
>>
>> The intermediate would be a config file that is tracked on another
>> ref. This (a) decouples main project history from submodule history
>> and (b) makes it easier to distribute as it is part of the repository.
>>
>> For example (a) is desired if you dig up an old project and the
>> submodules have all moved from one git hosting provider to another.
>> Another example would be when you fork a project with submodules
>> and don't want to mess with the main history but you just want to
>> adjust the submodule URLs. That is possible with such an intermediate
>> additional place.
>>
>> For (b) you can imagine the fork that you want to distribute in your
>> community and you don't want to tell everyone to change the
>> submodule URLs, but instead you can provide them with a prepared
>> .gitmodules file, that they have to place into that special ref (which
>> can be done via fetching).
>>
>> I digress as these ideas seem to be orthogonal to your patch series,
>> just FYI. prior discussion starting at:
>> https://public-inbox.org/git/1462317985-640-1-git-send-email-sbeller@google.com/
>> I recall there was a better discussion even prior to that, but have no
>> link handy.
>>
>
> Just to understand, is that 'refs/meta/submodules:.gitmodules' file
> meant to be managed manually? Or do you imagine some options to
> instruct "git submodule" to *write* to that file instead of .gitmodules?

I imagined it to be managed manually as it would enable some
workflows for downstream users of superproject repos.

But I'd think a convenient way to write to this location would be
super useful, so we ought to have that eventually.

> In the latter case your idea could cover my use case too, couldn't it?

I would think so, yes.

> But most importantly, is this realistically going to be added?

I plan on adding it eventually. It depends on the priorities and
schedule, no promises, though.

> Currently I am not that familiar with the git code base to do it
> myself, and I've never explicitly dealt with special refs in git.

I think core git only uses them for "actual refs", e.g. remote
tracking branches are used to know about the sha1. This new
special ref would be used to store content outside the main tree,
so we'd have too lookup the blob in that not-checked-out commit
and read and write there.

> The approach of this patch series is a lot simpler, and something I can
> work on in my spare time.
> Presumably (b) could even be partially supported with my changes, by
> having both '.gitmodules' and some custom '.gitmodules-alternative'
> files in the repository and tell users to clone with:
>
>   git clone --config core.submodulesFile=.gitmodules-alternative URL
>
> Not as clean as your idea but doable.

Traditionally Git had a strong stance on not transporting config
in the repo (except submodule URLs :/) itself, so I guess
this .gitmodules-alternative would not be a file in the tree, but
a URL or something?

> [...]
>> > Since the gitmodules file is meant to be checked in into the repository,
>> > the overridden file path should be relative to the work-tree; is there
>> > a way to enforce this constraint at run time (i.e. validate the config
>> > value), or is it enough to have it documented?
>>
>> I think we'd want to check at run time, if we need this constraint.
>>
>
> I'll look into it once we decide which the way to go.
>
> Thank you.
>

Thanks for bringing up this easier approach.
Stefan

^ permalink raw reply	[relevance 4%]

* Re: [RFC 02/10] submodule: fix getting custom gitmodule file in fetch command
  2018-04-16 16:18     ` Antonio Ospite
@ 2018-04-16 19:23       ` Stefan Beller
  2018-04-16 20:46         ` Antonio Ospite
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-16 19:23 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Richard Hartmann

On Mon, Apr 16, 2018 at 9:18 AM, Antonio Ospite <ao2@ao2.it> wrote:

>
> Is there an API to just load one config setting?

Do you mean to

  git -c key=value foo-command --options ...

^ permalink raw reply	[relevance 5%]

* Re: [RFC 02/10] submodule: fix getting custom gitmodule file in fetch command
  2018-04-16 19:23       ` Stefan Beller
@ 2018-04-16 20:46         ` Antonio Ospite
  0 siblings, 0 replies; 200+ results
From: Antonio Ospite @ 2018-04-16 20:46 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Richard Hartmann

On Mon, 16 Apr 2018 12:23:59 -0700
Stefan Beller <sbeller@google.com> wrote:

> On Mon, Apr 16, 2018 at 9:18 AM, Antonio Ospite <ao2@ao2.it> wrote:
> 
> >
> > Is there an API to just load one config setting?
> 
> Do you mean to
> 
>   git -c key=value foo-command --options ...

I meant, instead of:

  git_config(git_fetch_config, NULL);

which updates a series of config settings, something which loads
_in_the_code_ just the one value for 'core.submodulesFile' previously
set either via "git -c" or "git config".

It turns out what I was looking for is git_config_get_string_const().

The new patch will be simply this:

diff --git a/config.c b/config.c
index 6ffb1d501..1ef9801d3 100644
--- a/config.c
+++ b/config.c
@@ -2087,6 +2087,7 @@ int git_config_get_pathname(const char *key, const char **dest)
  */
 void config_from_gitmodules(config_fn_t fn, void *data)
 {
+       git_config_get_string_const("core.submodulesFile", &submodules_file);
        if (the_repository->worktree) {
                char *file = repo_worktree_path(the_repository, submodules_file);
                git_config_from_file(fn, file, data);


Which covers my use case without changing the fetch behavior for previous
users, as "core.submodulesFile" is a new option.

Thanks,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 4%]

* Re: [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path
  2018-04-16 16:37     ` Antonio Ospite
@ 2018-04-16 21:22       ` Stefan Beller
  2018-04-18 11:43         ` Antonio Ospite
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-16 21:22 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Richard Hartmann

On Mon, Apr 16, 2018 at 9:37 AM, Antonio Ospite <ao2@ao2.it> wrote:
> On Thu, 12 Apr 2018 16:50:03 -0700
> Stefan Beller <sbeller@google.com> wrote:
>
>> Hi Antonio,
>>
>> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
>> > When multiple repositories with detached work-trees take turns using the
>> > same directory as their work-tree, and more than one of them want to use
>> > submodules, there will be conflicts about the '.gitmodules' file.
>>
>> unlike other files which would not conflict?
>> There might be file names such as LICENSE, Readme.md etc,
>> which are common enough that they would produce conflicts as well?
>> I find this argument on its own rather weak. ("Just delete everything in
>> the working dir before using it with another repository"). I might be
>> missing a crucial bit here?
>>
>
> All the vcsh repositories _share_ the same work-tree; they may control
> it taking turns but, in general, all files are meant to be checked out
> at all times as the basic use case is: *distinct* sets of config files.
>
> Maybe saying that the repositories "take turns" is confusing.
> It's an unnecessary information, so I will omit that part form the
> commit message.

So they all have the same workdir, do they track the same set of files
or do they track a disjoint set of files, and ignoring the other repositories
files via the ignore mechanism?

This sounds like an interesting setup. I never though of that as something
useful (in either configuration).

> After your question I've done some research and I've seen other vcsh
> users managing conflicting LICENSE and README files using git
> sparse-checkouts, to have these files in the single repositories but
> not checked out in the shared work-tree:
> https://github.com/RichiH/vcsh/issues/120#issuecomment-42639619
> https://github.com/jwhitley/vcsh-root/commit/30b0d495c2cbe47ae9617ace9c2c14720d961d78
>
> However I guess that my point here is that the gitmodules file is
> something that influences git behavior so it should not be on the user's
> shoulder to manage conflicts for it, and most importantly it needs to
> be checked out for git to access it, doesn't it?

Good point! I wonder if the cleaner solution would be to just
tell git to use HEAD:.gitmodules and not check out the file?
then you would not need to come up with a namespace for names
of the .gitmodules files and scatter them into the worktree as well?


>> > -               value=$(git config -f .gitmodules submodule."$name"."$option")
>> > +               gitmodules_file=$(git config core.submodulesfile)
>> > +               : ${gitmodules_file:=.gitmodules}
>> > +               value=$(git config -f "$gitmodules_file" submodule."$name"."$option")
>>
>> I wonder if it would be cheaper to write a special config lookup now, e.g.
>> in builtin/submodule--helper.c we could have a "config-from-gitmodules"
>> subcommand that is looking up the modules file and then running the config
>> on that file.
>>
>
> Can you give an example from the user point of view of such a
> "config-from-gitmodules" command?
>

    git submodule config <name> <option>

as an 'alias' for

               gitmodules_file=$(git config core.submodulesfile)
               : ${gitmodules_file:=.gitmodules}
               value=$(git config -f "$gitmodules_file"
submodule."$name"."$option")

The helper would figure out which config file to load form
(.gitmodules in tree, HEAD:.gitmodules, your new proposed gitmodules file,
.git/config... or the special ref) and then return the <option> for <name>

So maybe:

    $ git clone https://gerrit.googlesource.com/gerrit && cd gerrit
    # ^ My goto-repo with submodules

    $ git submodule config "plugins/hooks" URL
    ../plugins/hooks



> I might look into it, but that can also be a followup change.


>> > diff --git a/submodule.c b/submodule.c
>> > index 9a50168b2..2afbdb644 100644
>> > --- a/submodule.c
>> > +++ b/submodule.c
>> > @@ -36,13 +36,13 @@ static struct oid_array ref_tips_after_fetch;
>> >   */
>> >  int is_gitmodules_unmerged(const struct index_state *istate)
>> >  {
>> > -       int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE));
>> > +       int pos = index_name_pos(istate, submodules_file, strlen(submodules_file));
>>
>> Ah, regarding the coverletter: This clearly assumes the modules
>> file is in the tree. So at least here we would make an exception
>> for files outside the tree to either not check for un-merged-ness or
>> disallow that case entirely.
>>
>
> Sorry I am not sure I follow what you are saying here, keep in mind
> that I am new to git internals.
>
> Do you mean that, even if we ensure (in
> config.c::git_default_core_config) that only paths relative to
> the work-tree are allowed, we still have to check here that the
> constraint is respected? And is so, why?

index_name_pos looks up a position of a file in the index,
which would fail for any file not in the index.

So if we give a path outside the tree, the lookup would fail
and we'd treat it as no .gitmodules file would be found,
which implies is_gitmodules_unmerged = false.

That sounds about right, but I think we could make the
distinction clearer, i.e. out-of-tree .gitmodule files *cannot*
be "unmerged" as that requires them to be part of a git
repository with a merge in progress.

>> There are quite a few functions in submodule.c which access the new global. :/
>> So moving them to the_repository should be fine, but eventually (not
>> in this series)
>> all these functions would would want to take a repository argument as well
>> such that they work on more than the_repository.
>>
>
> I will surely get rid of the global variable, but about changing the
> functions signatures I don't feel like promising anything just yet.

Yes, I would just use the_repository->... for now and as said by
"(not in this series)" we can do that later. It was more of a reminder
to myself, sorry for the confusion.

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* [PATCH] worktree: accept -f as short for --force for removal
@ 2018-04-16 22:16 Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-16 22:16 UTC (permalink / raw)
  To: git; +Cc: pclouds, Stefan Beller

The force flag is very common in many commands and is commonly
abbreviated with '-f', so add that to worktree removal, too

Signed-off-by: Stefan Beller <sbeller@google.com>
---

As a side note:

 $ git worktree add ../test HEAD^
 $ cd ../test
 $ make
 $ git worktree remove test
 fatal: working trees containing submodules cannot be moved or removed
 
 (This is on git.git, which does have submodules, but they should not 
 be relevant here? Instead we rather want to point out files left over.
 Not sure. I also plan on having worktrees and submodules to work well
 together eventually)


 builtin/worktree.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/builtin/worktree.c b/builtin/worktree.c
index 40a438ed6c..d734874d58 100644
--- a/builtin/worktree.c
+++ b/builtin/worktree.c
@@ -783,7 +783,7 @@ static int remove_worktree(int ac, const char **av, const char *prefix)
 {
 	int force = 0;
 	struct option options[] = {
-		OPT_BOOL(0, "force", &force,
+		OPT_BOOL('f', "force", &force,
 			 N_("force removing even if the worktree is dirty")),
 		OPT_END()
 	};
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 8%]

* Re: What's cooking in git.git (Apr 2018, #02; Tue, 17)
      [irrelevant] <xmqqzi22tlfx.fsf@gitster-ct.c.googlers.com>
@ 2018-04-17 18:05 ` Stefan Beller
  2018-04-18 13:09   ` Jonathan Tan
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-17 18:05 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git

Hi Junio,

> --------------------------------------------------
> [New Topics]

> * sb/object-store-replace (2018-04-12) 15 commits
...
>  The effort to pass the repository in-core structure throughout the
>  API continues.  This round deals with the code that implements the
>  refs/replace/ mechanism.
>
>  What's the doneness of this thing?  I didn't recall seeing any
>  response, especially ones that demonstrated the reviewer carefully
>  read and thought about the issues surrounding the code.  Not that I
>  spotted any problems in these patches myself, though.

Stolee and Brandon provided a "quick LGTM" type of review
https://public-inbox.org/git/20180409232536.GB102627@google.com/
https://public-inbox.org/git/9ddfee7e-025a-79c9-8d6b-700c65a14067@gmail.com/

I do not recall an in depth review, though Rene had some design guidance
in form of a patch, which is also the first commit of the series
https://public-inbox.org/git/38962a15-1081-bbdb-b4c4-6b46222b5f64@web.de/

My plan was to build the next series on top this week while waiting for
further review, though I wonder how much review will happen this week.
(Brandon, Jonathan Tan and Jonathan Nieder are all OOO,
Peff is on vacation, too)

I do not recall any discussion worthy design discussions left over, so
I'd lean on "cook in next for a while".

>
> --------------------------------------------------
> [Cooking]
>
> * sb/blame-color (2018-04-17) 2 commits
>  - builtin/blame: highlight recently changed lines
>  - builtin/blame: dim uninteresting metadata lines
>
>  "git blame" learns to unhighlight uninteresting metadata from the
>  originating commit on lines that are the same as the previous one,
>  and also paint lines in different colors depending on the age of
>  the commit.
>
>  The code to handle interaction between the config and command line
>  option smelled fishy.  Reviews and discussions are welcomed (not
>  just to this topic but others too ;-).

I'll look at the replies in thread there.


> * sb/submodule-move-nested (2018-03-29) 6 commits
>  - submodule: fixup nested submodules after moving the submodule
>  - submodule-config: remove submodule_from_cache
>  - submodule-config: add repository argument to submodule_from_{name, path}
>  - submodule-config: allow submodule_free to handle arbitrary repositories
>  - grep: remove "repo" arg from non-supporting funcs
>  - submodule.h: drop declaration of connect_work_tree_and_git_dir
>
>  Moving a submodule that itself has submodule in it with "git mv"
>  forgot to make necessary adjustment to the nested sub-submodules;
>  now the codepath learned to recurse into the submodules.
>
>  What's the doneness of this thing?

I considered this done a long time ago,

    "All 6 patches look good to me, thanks.
     Reviewed-by: Jonathan Tan <jonathantanmy@google.com>"

https://public-inbox.org/git/20180328161727.af10f596dffc8e01205c41dd@google.com/


Thanks,
Stefan

^ permalink raw reply	[relevance 2%]

* Re: [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path
  2018-04-16 21:22       ` Stefan Beller
@ 2018-04-18 11:43         ` Antonio Ospite
  2018-04-18 18:44           ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Antonio Ospite @ 2018-04-18 11:43 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, Richard Hartmann

On Mon, 16 Apr 2018 14:22:35 -0700
Stefan Beller <sbeller@google.com> wrote:

> On Mon, Apr 16, 2018 at 9:37 AM, Antonio Ospite <ao2@ao2.it> wrote:
> > On Thu, 12 Apr 2018 16:50:03 -0700
> > Stefan Beller <sbeller@google.com> wrote:
> >
> >> Hi Antonio,
> >>
> >> On Thu, Apr 12, 2018 at 3:20 PM, Antonio Ospite <ao2@ao2.it> wrote:
> >> > When multiple repositories with detached work-trees take turns using the
> >> > same directory as their work-tree, and more than one of them want to use
> >> > submodules, there will be conflicts about the '.gitmodules' file.
> >>
> >> unlike other files which would not conflict?
> >> There might be file names such as LICENSE, Readme.md etc,
> >> which are common enough that they would produce conflicts as well?
> >> I find this argument on its own rather weak. ("Just delete everything in
> >> the working dir before using it with another repository"). I might be
> >> missing a crucial bit here?
> >>
> >
> > All the vcsh repositories _share_ the same work-tree; they may control
> > it taking turns but, in general, all files are meant to be checked out
> > at all times as the basic use case is: *distinct* sets of config files.
> >
> > Maybe saying that the repositories "take turns" is confusing.
> > It's an unnecessary information, so I will omit that part form the
> > commit message.
> 
> So they all have the same workdir, do they track the same set of files
> or do they track a disjoint set of files, and ignoring the other repositories
> files via the ignore mechanism?
>

To recap,

vcsh[1] sets $HOME as the work-tree of multiple repositories to track
different sets of dotfiles in distinct repositories, while still having
the files directly available in $HOME. Each repository can ignore
untracked files via the ignore mechanism (namely core.excludesFile).

[1] https://github.com/RichiH/vcsh

For all this to work well, the sets of the tracked files would also need
to be disjoint, and usually they "practically" are, once a few
exceptions are taken care of.

Common intersecting items like LICENSE and README can be handled via
sparse-checkout to have "disjoint checkouts" and this solves most of
the problems, but the same mechanism cannot be used for .gitmodules as
it needs to be checked out.

And the problem cannot be worked around like done with .gitignore
(using core.excludesFile instead) because .gitmodules is unique and
hardcoded.

> This sounds like an interesting setup. I never though of that as something
> useful (in either configuration).
>

Give vcsh a try maybe.

[...]
> > However I guess that my point here is that the gitmodules file is
> > something that influences git behavior so it should not be on the user's
> > shoulder to manage conflicts for it, and most importantly it needs to
> > be checked out for git to access it, doesn't it?
> 
> Good point! I wonder if the cleaner solution would be to just
> tell git to use HEAD:.gitmodules and not check out the file?
> then you would not need to come up with a namespace for names
> of the .gitmodules files and scatter them into the worktree as well?
>

Any solution which:

  1. prevents the gitmodules file to be checked out
  2. but still tracks it in the git repository

OR
  
  1. allows to set the gitmoudles file under some namespace

would work for vcsh I guess.

> 
> >> > -               value=$(git config -f .gitmodules submodule."$name"."$option")
> >> > +               gitmodules_file=$(git config core.submodulesfile)
> >> > +               : ${gitmodules_file:=.gitmodules}
> >> > +               value=$(git config -f "$gitmodules_file" submodule."$name"."$option")
> >>
> >> I wonder if it would be cheaper to write a special config lookup now, e.g.
> >> in builtin/submodule--helper.c we could have a "config-from-gitmodules"
> >> subcommand that is looking up the modules file and then running the config
> >> on that file.
> >>
> >
> > Can you give an example from the user point of view of such a
> > "config-from-gitmodules" command?
> >
> 
>     git submodule config <name> <option>
> 
> as an 'alias' for
> 
>                gitmodules_file=$(git config core.submodulesfile)
>                : ${gitmodules_file:=.gitmodules}
>                value=$(git config -f "$gitmodules_file"
> submodule."$name"."$option")
> 
> The helper would figure out which config file to load form
> (.gitmodules in tree, HEAD:.gitmodules, your new proposed gitmodules file,
> .git/config... or the special ref) and then return the <option> for <name>
> 
> So maybe:
> 
>     $ git clone https://gerrit.googlesource.com/gerrit && cd gerrit
>     # ^ My goto-repo with submodules
> 
>     $ git submodule config "plugins/hooks" URL
>     ../plugins/hooks
> 
>

I may look into such supporting changes once you decide the approach to
take for the bigger problem.

Thank you,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 4%]

* Re: What's cooking in git.git (Apr 2018, #02; Tue, 17)
  2018-04-17 18:05 ` What's cooking in git.git (Apr 2018, #02; Tue, 17) Stefan Beller
@ 2018-04-18 13:09   ` Jonathan Tan
  0 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-04-18 13:09 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git

On Tue, Apr 17, 2018 at 11:05 AM, Stefan Beller <sbeller@google.com> wrote:
>> * sb/submodule-move-nested (2018-03-29) 6 commits
>>  - submodule: fixup nested submodules after moving the submodule
>>  - submodule-config: remove submodule_from_cache
>>  - submodule-config: add repository argument to submodule_from_{name, path}
>>  - submodule-config: allow submodule_free to handle arbitrary repositories
>>  - grep: remove "repo" arg from non-supporting funcs
>>  - submodule.h: drop declaration of connect_work_tree_and_git_dir
>>
>>  Moving a submodule that itself has submodule in it with "git mv"
>>  forgot to make necessary adjustment to the nested sub-submodules;
>>  now the codepath learned to recurse into the submodules.
>>
>>  What's the doneness of this thing?
>
> I considered this done a long time ago,
>
>     "All 6 patches look good to me, thanks.
>      Reviewed-by: Jonathan Tan <jonathantanmy@google.com>"
>
> https://public-inbox.org/git/20180328161727.af10f596dffc8e01205c41dd@google.com/

To add to this, I sent this in reply to version 3 of this patch set,
after Stefan addressed my comments. Most of my in-depth comments were
in reply to version 1 of this patch, which are the grandchild replies
to [1].

[1] https://public-inbox.org/git/20180327213918.77851-1-sbeller@google.com/

^ permalink raw reply	[relevance 2%]

* Re: [PATCH] submodule--helper: don't print null in 'submodule status'
      [irrelevant] <20180418145337.7591-1-pclouds@gmail.com>
@ 2018-04-18 17:34 ` Stefan Beller
  2018-04-18 21:24   ` Junio C Hamano
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-18 17:34 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy
  Cc: git, Prathamesh Chavan, Christian Couder

Hi Nguyễn,

On Wed, Apr 18, 2018 at 7:53 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> The function compute_rev_name() can return NULL sometimes (e.g. right
> after 'submodule init'). The current code makes 'submodule status'
> print this:
>
>  19d97bf5af05312267c2e874ee6bcf584d9e9681 sha1collisiondetection ((null))
>
> This ugly 'null' adds no value to the user using this command. More
> importantly printf() on some platform can't handle NULL as a string
> and will crash instead of printing '(null)'.
>
> Check for this and skip printing this part (the alternative is
> printing '(n/a)' or something but I think that is just noise).

This patch restores the behavior from before a9f8a37584 (submodule:
port submodule subcommand 'status' from shell to C, 2017-10-06),
so this is the right way to go instead of the alternatives you considered.

Thanks!

Reviewed-by: Stefan Beller <sbeller@google.com>

> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  builtin/submodule--helper.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index a404df3ea4..4dc7d7d29f 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -596,8 +596,12 @@ static void print_status(unsigned int flags, char state, const char *path,
>
>         printf("%c%s %s", state, oid_to_hex(oid), displaypath);
>
> -       if (state == ' ' || state == '+')
> -               printf(" (%s)", compute_rev_name(path, oid_to_hex(oid)));
> +       if (state == ' ' || state == '+') {
> +               const char *name = compute_rev_name(path, oid_to_hex(oid));
> +
> +               if (name)
> +                       printf(" (%s)", name);
> +       }
>
>         printf("\n");
>  }
> --
> 2.17.0.367.g5dd2e386c3
>

^ permalink raw reply	[relevance 11%]

* Re: [RFC 01/10] submodule: add 'core.submodulesFile' to override the '.gitmodules' path
  2018-04-18 11:43         ` Antonio Ospite
@ 2018-04-18 18:44           ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-18 18:44 UTC (permalink / raw)
  To: Antonio Ospite; +Cc: git, Richard Hartmann

Hi Antonio,

>>
>> Good point! I wonder if the cleaner solution would be to just
>> tell git to use HEAD:.gitmodules and not check out the file?
>> then you would not need to come up with a namespace for names
>> of the .gitmodules files and scatter them into the worktree as well?
>>
>
> Any solution which:
>
>   1. prevents the gitmodules file to be checked out
>   2. but still tracks it in the git repository
>
> OR
>
>   1. allows to set the gitmoudles file under some namespace
>
> would work for vcsh I guess.

I personally would tend to rather go for supporting your first solution
(prevent .gitmodules from checked-out, load from sparse HEAD),
but I do not have strong arguments or feeling about this dimension.
I am fine with a namespaced .gtimodules solution, too.

Both solutions can be implemented by either:

A) adding the code where it is (like your patch, e.g. using

> -               value=$(git config -f .gitmodules submodule."$name"."$option")
> +               gitmodules_file=$(git config core.submodulesfile)
> +               : ${gitmodules_file:=.gitmodules}
> +               value=$(git config -f "$gitmodules_file" submodule."$name"."$option")

B) adding a helper, which is a layer of indirection
to load the relevant configuration.

And when it comes to this dimension, I'd strongly favor B over A.
Having this indirection helper in place enables to add more options
later easily as only one place needs to be touched.
(These other options could include the other solution as presented above,
or the idea with the special ref as mentioned in an earlier email)


>> > Can you give an example from the user point of view of such a
>> > "config-from-gitmodules" command?
>> >
>>
>>     git submodule config <name> <option>
>>
>> as an 'alias' for
>>
>>                gitmodules_file=$(git config core.submodulesfile)
>>                : ${gitmodules_file:=.gitmodules}
>>                value=$(git config -f "$gitmodules_file"
>> submodule."$name"."$option")
>>
>> The helper would figure out which config file to load form
>> (.gitmodules in tree, HEAD:.gitmodules, your new proposed gitmodules file,
>> .git/config... or the special ref) and then return the <option> for <name>
>>
>> So maybe:
>>
>>     $ git clone https://gerrit.googlesource.com/gerrit && cd gerrit
>>     # ^ My goto-repo with submodules
>>
>>     $ git submodule config "plugins/hooks" URL
>>     ../plugins/hooks
>>
>>
>
> I may look into such supporting changes once you decide the approach to
> take for the bigger problem.

I think once we have the helper in place you can implement the solution
to the bigger problem as you like?

There are a few pros and cons for namespaced .gitmodules and
non-checked-out sparse HEAD .gitmodules:

How do you modify the .gitmodules config?
============
In the namespaced solution, you can tell users to edit that
file manually or use "git config -f $new_location" to manipulate
that file.

In the sparse solution editing becomes a little bit trickier, as you
need to edit a file in the index (or HEAD).

If you have the special ref, you could just checkout the
special ref in another worktree and make changes and
commit there


How do you change the setup?
============
In case of a sparse gitmodules file, you can just check it out
(make it non-sparse) or vice versa.

In case of a namespaced gitmodules file, you'd change the
config setting and have to move the file to the new location.
as git config is just about configuring, the user is left alone
with moving the file, or would we have a helper for that?
("git submodule relocate-gitmodules" or such)?

If you have the special ref, you could just checkout the
special ref in another worktree and make changes and
commit there.

I hope this helps instead of confusing more,

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] submodule--helper: don't print null in 'submodule status'
  2018-04-18 17:34 ` [PATCH] submodule--helper: don't print null in 'submodule status' Stefan Beller
@ 2018-04-18 21:24   ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-04-18 21:24 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Nguyễn Thái Ngọc Duy, git, Prathamesh Chavan,
	Christian Couder

Stefan Beller <sbeller@google.com> writes:

> Hi Nguyễn,
>
> On Wed, Apr 18, 2018 at 7:53 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
>> The function compute_rev_name() can return NULL sometimes (e.g. right
>> after 'submodule init'). The current code makes 'submodule status'
>> print this:
>>
>>  19d97bf5af05312267c2e874ee6bcf584d9e9681 sha1collisiondetection ((null))
>>
>> This ugly 'null' adds no value to the user using this command. More
>> importantly printf() on some platform can't handle NULL as a string
>> and will crash instead of printing '(null)'.
>>
>> Check for this and skip printing this part (the alternative is
>> printing '(n/a)' or something but I think that is just noise).
>
> This patch restores the behavior from before a9f8a37584 (submodule:
> port submodule subcommand 'status' from shell to C, 2017-10-06),
> so this is the right way to go instead of the alternatives you considered.
>
> Thanks!
>
> Reviewed-by: Stefan Beller <sbeller@google.com>

Excellent.  Thanks, both.

Will queue.

>
>> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
>> ---
>>  builtin/submodule--helper.c | 8 ++++++--
>>  1 file changed, 6 insertions(+), 2 deletions(-)
>>
>> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
>> index a404df3ea4..4dc7d7d29f 100644
>> --- a/builtin/submodule--helper.c
>> +++ b/builtin/submodule--helper.c
>> @@ -596,8 +596,12 @@ static void print_status(unsigned int flags, char state, const char *path,
>>
>>         printf("%c%s %s", state, oid_to_hex(oid), displaypath);
>>
>> -       if (state == ' ' || state == '+')
>> -               printf(" (%s)", compute_rev_name(path, oid_to_hex(oid)));
>> +       if (state == ' ' || state == '+') {
>> +               const char *name = compute_rev_name(path, oid_to_hex(oid));
>> +
>> +               if (name)
>> +                       printf(" (%s)", name);
>> +       }
>>
>>         printf("\n");
>>  }
>> --
>> 2.17.0.367.g5dd2e386c3
>>

^ permalink raw reply	[relevance 5%]

* [RFC PATCH v4 3/9] fixup:t7406: use test_commit instead of echo/add/commit as suggested by Stefan Beller
  2018-04-04 21:41           ` Stefan Beller
@ 2018-04-18 22:35             ` Eddy Petrișor
      [irrelevant]             ` <20180418223552.18345-1-eddy.petrisor@codeaurora.org>
  2018-04-19  6:07             ` [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule Eddy Petrișor
  2 siblings, 0 replies; 200+ results
From: Eddy Petrișor @ 2018-04-18 22:35 UTC (permalink / raw)
  To: sbeller, jrnieder; +Cc: Eddy Petrișor, git

From: Eddy Petrișor <eddy.petrisor@gmail.com>

Signed-off-by: Eddy Petrișor <eddy.petrisor@gmail.com>
---
 t/t7406-submodule-update.sh | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/t/t7406-submodule-update.sh b/t/t7406-submodule-update.sh
index 7b65f1dd1..7fb370991 100755
--- a/t/t7406-submodule-update.sh
+++ b/t/t7406-submodule-update.sh
@@ -264,19 +264,13 @@ test_expect_success 'submodule update --remote --recursive --init should fetch m
 	git clone super5 submodl2b2 &&
 	git clone super5 submodl1b1 &&
 	cd submodl2b2 &&
-	echo linel2b2 > l2b2 &&
 	git checkout -b b2 &&
-	git add l2b2 &&
-	test_tick &&
-	git commit -m "commit on b2 branch in l2" &&
+	test_commit "l2_on_b2" &&
 	git rev-parse --verify HEAD >../expectl2 &&
 	git checkout master &&
 	cd ../submodl1b1 &&
 	git checkout -b b1 &&
-	echo linel1b1 > l1b1 &&
-	git add l1b1 &&
-	test_tick &&
-	git commit -m "commit on b1 branch in l1" &&
+	test_commit "l1_on_b1" &&
 	git submodule add ../submodl2b2 submodl2b2 &&
 	git config -f .gitmodules submodule."submodl2b2".branch b2 &&
 	git add .gitmodules &&
@@ -286,10 +280,7 @@ test_expect_success 'submodule update --remote --recursive --init should fetch m
 	git rev-parse --verify HEAD >../expectl1 &&
 	git checkout master &&
 	cd ../super5 &&
-	echo super_with_2_chained_modules > super5 &&
-	git add super5 &&
-	test_tick &&
-	git commit -m "commit on default branch in super5" &&
+	test_commit super5_with_2_chained_modules_on_default_branch &&
 	git submodule add ../submodl1b1 submodl1b1 &&
 	git config -f .gitmodules submodule."submodl1b1".branch b1 &&
 	git add .gitmodules &&
-- 
2.16.2


^ permalink raw reply	[relevance 21%]

* Re: [RFC PATCH v4 1/9] git-submodule.sh:cmd_update: if submodule branch exists, fetch that instead of default
      [irrelevant]             ` <20180418223552.18345-1-eddy.petrisor@codeaurora.org>
@ 2018-04-18 23:53               ` Stefan Beller
  2018-04-19  5:43                 ` Eddy Petrișor
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-18 23:53 UTC (permalink / raw)
  To: Eddy Petrișor; +Cc: Jonathan Nieder, Eddy Petrișor, git

Hi Eddy,

all the following patches 3-9 are touching the test as added in patch
2, which would go best with this patch.
Could you squash all commits into one?

There are a couple ways to do it:

  git reset --soft
  git commit -a --reuse-commit-message=<...>

or using

    git rebase --interactive origin/master
    # and then marking all but the first as "fixup"

I think the end result looks good, but that is best reviewed as one
piece instead of 9 patches.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [RFC PATCH v4 1/9] git-submodule.sh:cmd_update: if submodule branch exists, fetch that instead of default
  2018-04-18 23:53               ` [RFC PATCH v4 1/9] git-submodule.sh:cmd_update: if submodule branch exists, fetch that instead of default Stefan Beller
@ 2018-04-19  5:43                 ` Eddy Petrișor
  0 siblings, 0 replies; 200+ results
From: Eddy Petrișor @ 2018-04-19  5:43 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Eddy Petrișor, Jonathan Nieder, git

2018-04-19 2:53 GMT+03:00 Stefan Beller <sbeller@google.com>:
> Hi Eddy,
>
> all the following patches 3-9 are touching the test as added in patch
> 2, which would go best with this patch.
> Could you squash all commits into one?

Yes,

I did not have time yesterday to put all changes into a single commit
with an associated note (because note managment seems to be a huge
pain), so I preferred small commits.

But I wanted to get your feedback on something, I'll reply in thread
arm where you actually suspected the problem.

> There are a couple ways to do it:
>
>   git reset --soft
>   git commit -a --reuse-commit-message=<...>
>
> or using
>
>     git rebase --interactive origin/master
>     # and then marking all but the first as "fixup"

I am aware of git rebase -i and use it regularly, that's why patches
3-9 have the 'fixup' prefix.

> I think the end result looks good, but that is best reviewed as one
> piece instead of 9 patches.


-- 
Eddy Petrișor

^ permalink raw reply	[relevance 5%]

* Re: [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule
  2018-04-04 21:41           ` Stefan Beller
  2018-04-18 22:35             ` [RFC PATCH v4 3/9] fixup:t7406: use test_commit instead of echo/add/commit as suggested by Stefan Beller Eddy Petrișor
      [irrelevant]             ` <20180418223552.18345-1-eddy.petrisor@codeaurora.org>
@ 2018-04-19  6:07             ` Eddy Petrișor
  2018-04-19 17:52               ` Stefan Beller
  2 siblings, 1 reply; 200+ results
From: Eddy Petrișor @ 2018-04-19  6:07 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Eddy Petrișor, Jonathan Nieder, Git List

2018-04-05 0:41 GMT+03:00 Stefan Beller <sbeller@google.com>:
> On Wed, Apr 4, 2018 at 1:37 PM, Eddy Petrișor <eddy.petrisor@gmail.com> wrote:
>
>>> you plan to later submit as one patch including both the change as well as
>>> the test.
>>
>> Yes,

I did not forget about having a single patch. Will do it once the
details are ironed out.

>>>>> +       cd ../super5 &&
>>>>> +       echo super_with_2_chained_modules > super5 &&
>>>>> +       git add super5 &&
>>>>> +       test_tick &&
>>>>> +       git commit -m "commit on default branch in super5" &&
>>>>> +       git submodule add ../submodl1b1 submodl1b1 &&
>>>>> +       git config -f .gitmodules submodule."submodl1b1".branch b1 &&
>>>>> +       git add .gitmodules &&
>>>>> +       test_tick &&
>>>>> +       git commit -m "add l1 module with branch b1 in super5" &&
>>>
>>> now we do this again without the nested submodule, just one repo
>>> as a submodule?
>>
>> My intention was to have super5 -> submodl1b1@b1 -> submodl2b2@b2 on
>> the "server" side.
>> But are you saying I just implemented super5 -> sbmodl1b1@master due
>> to the previous master checkout in submodl1b1?
>
> No. I was a little confused about the code.

Actually you were 100% correct. In order to link to submodl1b1@b1 I
had to move the master branch checkout after the submobl2@b2 is added.

Otherwise the submodule is added with the last commit on master, not
the last one on b1 an b2, respectively.

I suspect that in the tests, because the "server side" repos are
local, the git fetch-by-sha1/cloning by hash will be done correctly,
without the need of a branch hint, but the problem will still exist
for servers such as github which do not support fetch-by-sha1.
In case I realize that a server-side repo that doesn't support
fetch-by-sha1 is needed, is there a mechanism to set that up in the
test case, or do I have to rethink my approach?

>>>>> +       git submodule init submodl1b1 &&
>>>>> +       git clone super5 super &&
>>>
>>> does super exist here already? (I did not check, but IIRC
>>> super and super{1-4} are there as we count upwards to
>>> find a name that is ok.
>>
>> I created it in the first step of the test with the intention to have
>> super5 as the "server" and "super" as the client clone.
>
> oh, ok.

After using test_pause I realized 'super' is left over by some other
test cases, so in my v4 (unjustifibly) long series I switch to using
super_w because I was getting all sorts of issues and wanted to not
interfere with the other tests.

>> As a general idea for a test, does it look sane?
>
> Yes, I think it is a sane approach. Thanks for writing such a test!

OK, thanks for the feedback.

>> Do you think I should I start with a just one level of submodule with
>> a non-default branch (super -> l1@b1), or it this OK?
>> In my view, having 2 levels makes sure the recursive part is also
>> addressed and verified.
>
> I totally agree.


-- 
Eddy Petrișor

^ permalink raw reply	[relevance 6%]

* Re: [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule
  2018-04-19  6:07             ` [RFC PATCH v3 2/2] t7406: add test for non-default branch in submodule Eddy Petrișor
@ 2018-04-19 17:52               ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-19 17:52 UTC (permalink / raw)
  To: Eddy Petrișor; +Cc: Eddy Petrișor, Jonathan Nieder, Git List

Hi Eddy,

> I suspect that in the tests, because the "server side" repos are
> local, the git fetch-by-sha1/cloning by hash will be done correctly,
> without the need of a branch hint, but the problem will still exist
> for servers such as github which do not support fetch-by-sha1.
> In case I realize that a server-side repo that doesn't support
> fetch-by-sha1 is needed, is there a mechanism to set that up in the
> test case, or do I have to rethink my approach?

You can force a clone (at least, not sure about fetch) to use the
git protocol by --no-local, and then you can set
uploadpack.{allowTipSHA1InWant, allowReachableSHA1InWant,
allowAnySHA1InWant} as neded by the test.

From the fetch man page:
For local repositories, also supported by Git natively,
the following syntaxes may be used:

       ·   /path/to/repo.git/

       ·   file:///path/to/repo.git/

These two syntaxes are mostly equivalent, except
when cloning, when the former implies --local option.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH v1 0/5] Allocate cache entries from memory pool
      [irrelevant]     ` <3ec5999c-f81d-eafb-1597-4fa93f943a6d@gmail.com>
@ 2018-04-23 17:18       ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-23 17:18 UTC (permalink / raw)
  To: Jameson Miller
  Cc: Junio C Hamano, Jameson Miller, git, pclouds, jonathantanmy

On Mon, Apr 23, 2018 at 9:44 AM, Jameson Miller
<jameson.miller81@gmail.com> wrote:
> I would be interested to understand how the
> mem_pool would fit your needs, and if it is sufficient or needs modification
> for your use cases.
>
>> [1] proof of concept in patches nearby
>> https://public-inbox.org/git/20180206001749.218943-31-sbeller@google.com/
>>

Currenlty the parsed objects are loaded into memory and never freed.
See alloc.c which implements a specialized memory allocator for this
object loading.
When working with submodules, their objects are also just put into this
globally-namespaced object store. (See struct object **obj_hash in
object.c)

I want to make the object store a per-repository object, such that
when working with submodules, you can free up all submodule objects
once you are done with a given submodule.

To do so, the memory allocation needs to manage the whole life cycle,
while preserving the efficiency of alloc.c. See 855419f764
(Add specialized object allocator, 2006-06-19). The mem-pool that
you propose can allocate large slabs of memory and we can put
objects in there without alignment overhead, such that we preserve
the memory efficiency while being able to track all the memory.

So I would think it is sufficient as-is with this series, maybe we need
a little tweaking there, but nothing large IMHO.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: Feature Request: option for git difftool to show changes in submodule contents
      [irrelevant] <HK2PR0601MB184272A36E86014BDD988DB5F9890@HK2PR0601MB1842.apcprd06.prod.outlook.com>
@ 2018-04-23 19:39 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-23 19:39 UTC (permalink / raw)
  To: Oshaben Nicholas; +Cc: git

Hi Oshaben!

On Mon, Apr 23, 2018 at 7:49 AM, Oshaben Nicholas
<nicholas.oshaben@bwigroup.com> wrote:
> Hello,
>
> A coworker of mine was trying to use git difftool in a repository with submodules and we found that it only shows the change in the subproject commit hash. The option to show the changes in the contents of the submodules exists for git diff. Can this be added to difftool as well?

Yes, of course it can be added. Care to write a patch?

See lines 432-443 in builtin/difftool.c:
https://github.com/git/git/blob/master/builtin/difftool.c#L432

I'd think you'd want to compute some output
(Similar to the git-diff output? Then we could just
refactor the submodule diff stuff to be reusable here)
and put it into the strbuf there.

Feel free to ask away or read the docs Documentation/SubmittingPatches

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* [PATCH 0/9] object store: oid_object_info is the next contender
@ 2018-04-23 23:43 Stefan Beller
  2018-04-23 23:43 ` [PATCH 2/9] cache.h: add repository argument to oid_object_info Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-23 23:43 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

This applies on top of origin/sb/object-store-replace and is available as
https://github.com/stefanbeller/git/tree/oid_object_info

This continues the work of sb/packfiles-in-repository,
extending the layer at which we have to pass in an explicit
repository object to oid_object_info.

A test merge to next shows only a minor merge conflicit (adding
different #include lines in one c file), so this might be a good next
step for the object store series.

Notes on further object store series:
I plan on converting the "parsed object store" next,
which would be {alloc, object, tree, commit, tag}.c as that is a prerequisite
for migrating shallow (which is intermingled with grafts) information to the
object store.

There is currently work going on in allocation (mempool - Jameson Miller)
and grafts (deprecate grafts - DScho), which is why I am sending this
series first. I think it can go in parallel to the "parsed object store"
that is coming next.

Thanks,
Stefan

Jonathan Nieder (1):
  packfile: add repository argument to packed_object_info

Stefan Beller (8):
  cache.h: add repository argument to oid_object_info_extended
  cache.h: add repository argument to oid_object_info
  packfile: add repository argument to retry_bad_packed_offset
  packfile: add repository argument to packed_to_object_type
  packfile: add repository argument to read_object
  packfile: add repository argument to unpack_entry
  packfile: add repository argument to cache_or_unpack_entry
  cache.h: allow sha1_object_info to handle arbitrary repositories

 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       | 12 ++++++------
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   | 11 +++++++----
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  7 +++++--
 diff.c                   |  3 ++-
 fast-import.c            | 16 ++++++++++------
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 pack-check.c             |  3 ++-
 packfile.c               | 40 +++++++++++++++++++++++-----------------
 packfile.h               |  6 ++++--
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              | 32 ++++++++++++++++++--------------
 sha1_name.c              | 12 ++++++------
 streaming.c              |  2 +-
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 35 files changed, 121 insertions(+), 91 deletions(-)

-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 6%]

* [PATCH 2/9] cache.h: add repository argument to oid_object_info
  2018-04-23 23:43 [PATCH 0/9] object store: oid_object_info is the next contender Stefan Beller
@ 2018-04-23 23:43 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-23 23:43 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

Add a repository argument to allow the callers of oid_object_info
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

In the expanded macro the identifier `the_repository` is not actually used,
so the compiler does not catch if the repository.h header is not included
at the call site. call sites needing that #include were identified by
changing the macro to definition to

      #define sha1_object_info(r, sha1, size) \
          (r, sha1_object_info_##r(sha1, size)).

This produces a compiler warning about the left hand side of the comma
operator being unused, which can be suppressed using -Wno-unused-value.

To avoid breaking bisection, do not include this trick in the patch.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       |  6 +++---
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   |  8 +++++---
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  3 ++-
 diff.c                   |  3 ++-
 fast-import.c            | 14 +++++++++-----
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 packfile.c               |  2 +-
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              |  4 ++--
 sha1_name.c              | 12 ++++++------
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 32 files changed, 67 insertions(+), 53 deletions(-)

diff --git a/archive-tar.c b/archive-tar.c
index 3563bcb9f2..f93409324f 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -276,7 +276,7 @@ static int write_tar_entry(struct archiver_args *args,
 		memcpy(header.name, path, pathlen);
 
 	if (S_ISREG(mode) && !args->convert &&
-	    oid_object_info(oid, &size) == OBJ_BLOB &&
+	    oid_object_info(the_repository, oid, &size) == OBJ_BLOB &&
 	    size > big_file_threshold)
 		buffer = NULL;
 	else if (S_ISLNK(mode) || S_ISREG(mode)) {
diff --git a/archive-zip.c b/archive-zip.c
index 6b20bce4d1..74f3fe9103 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -325,7 +325,8 @@ static int write_zip_entry(struct archiver_args *args,
 		compressed_size = 0;
 		buffer = NULL;
 	} else if (S_ISREG(mode) || S_ISLNK(mode)) {
-		enum object_type type = oid_object_info(oid, &size);
+		enum object_type type = oid_object_info(the_repository, oid,
+							&size);
 
 		method = 0;
 		attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :
diff --git a/blame.c b/blame.c
index 78c9808bd1..dfa24473dc 100644
--- a/blame.c
+++ b/blame.c
@@ -81,7 +81,7 @@ static void verify_working_tree_path(struct commit *work_tree, const char *path)
 		unsigned mode;
 
 		if (!get_tree_entry(commit_oid, path, &blob_oid, &mode) &&
-		    oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+		    oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 			return;
 	}
 
@@ -504,7 +504,7 @@ static int fill_blob_sha1_and_mode(struct blame_origin *origin)
 		return 0;
 	if (get_tree_entry(&origin->commit->object.oid, origin->path, &origin->blob_oid, &origin->mode))
 		goto error_out;
-	if (oid_object_info(&origin->blob_oid, NULL) != OBJ_BLOB)
+	if (oid_object_info(the_repository, &origin->blob_oid, NULL) != OBJ_BLOB)
 		goto error_out;
 	return 0;
  error_out:
diff --git a/builtin/blame.c b/builtin/blame.c
index db38c0b307..bfdf7cc132 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -655,7 +655,7 @@ static int is_a_rev(const char *name)
 
 	if (get_oid(name, &oid))
 		return 0;
-	return OBJ_NONE < oid_object_info(&oid, NULL);
+	return OBJ_NONE < oid_object_info(the_repository, &oid, NULL);
 }
 
 int cmd_blame(int argc, const char **argv, const char *prefix)
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 4ecdb9ff54..b8ecbea98e 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -116,7 +116,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 		/* else fallthrough */
 
 	case 'p':
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("Not a valid object name %s", obj_name);
 
@@ -140,7 +140,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 	case 0:
 		if (type_from_string(exp_type) == OBJ_BLOB) {
 			struct object_id blob_oid;
-			if (oid_object_info(&oid, NULL) == OBJ_TAG) {
+			if (oid_object_info(the_repository, &oid, NULL) == OBJ_TAG) {
 				char *buffer = read_object_file(&oid, &type,
 								&size);
 				const char *target;
@@ -151,7 +151,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 			} else
 				oidcpy(&blob_oid, &oid);
 
-			if (oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+			if (oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 				return stream_blob_to_fd(1, &blob_oid, NULL, 0);
 			/*
 			 * we attempted to dereference a tag to a blob
diff --git a/builtin/describe.c b/builtin/describe.c
index de840f96a4..66c497f789 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -502,7 +502,7 @@ static void describe(const char *arg, int last_one)
 
 	if (cmit)
 		describe_commit(&oid, &sb);
-	else if (oid_object_info(&oid, NULL) == OBJ_BLOB)
+	else if (oid_object_info(the_repository, &oid, NULL) == OBJ_BLOB)
 		describe_blob(oid, &sb);
 	else
 		die(_("%s is neither a commit nor blob"), arg);
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index a15898d641..373c794873 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -947,7 +947,7 @@ static void import_marks(char *input_file)
 		if (last_idnum < mark)
 			last_idnum = mark;
 
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(&oid));
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index dcdfc66f09..73be393b2e 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -637,7 +637,7 @@ static int update_local_ref(struct ref *ref,
 	struct branch *current_branch = branch_get(NULL);
 	const char *pretty_ref = prettify_refname(ref->name);
 
-	type = oid_object_info(&ref->new_oid, NULL);
+	type = oid_object_info(the_repository, &ref->new_oid, NULL);
 	if (type < 0)
 		die(_("object %s not found"), oid_to_hex(&ref->new_oid));
 
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 087360a675..9d59d7d5a2 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -67,7 +67,8 @@ static const char *printable_type(struct object *obj)
 	const char *ret;
 
 	if (obj->type == OBJ_NONE) {
-		enum object_type type = oid_object_info(&obj->oid, NULL);
+		enum object_type type = oid_object_info(the_repository,
+							&obj->oid, NULL);
 		if (type > 0)
 			object_as_type(obj, type, 0);
 	}
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index d81473e722..2d04a596f5 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -223,7 +223,7 @@ static unsigned check_object(struct object *obj)
 
 	if (!(obj->flags & FLAG_CHECKED)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type <= 0)
 			die(_("did not receive expected object %s"),
 			      oid_to_hex(&obj->oid));
@@ -812,7 +812,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,
 		enum object_type has_type;
 		unsigned long has_size;
 		read_lock();
-		has_type = oid_object_info(oid, &has_size);
+		has_type = oid_object_info(the_repository, oid, &has_size);
 		if (has_type < 0)
 			die(_("cannot read existing object info %s"), oid_to_hex(oid));
 		if (has_type != type || has_size != size)
diff --git a/builtin/ls-tree.c b/builtin/ls-tree.c
index d44b4f9c27..409da4e835 100644
--- a/builtin/ls-tree.c
+++ b/builtin/ls-tree.c
@@ -94,7 +94,7 @@ static int show_tree(const struct object_id *oid, struct strbuf *base,
 			char size_text[24];
 			if (!strcmp(type, blob_type)) {
 				unsigned long size;
-				if (oid_object_info(oid, &size) == OBJ_BAD)
+				if (oid_object_info(the_repository, oid, &size) == OBJ_BAD)
 					xsnprintf(size_text, sizeof(size_text),
 						  "BAD");
 				else
diff --git a/builtin/mktree.c b/builtin/mktree.c
index 263c530315..bb76b469fd 100644
--- a/builtin/mktree.c
+++ b/builtin/mktree.c
@@ -116,7 +116,7 @@ static void mktree_line(char *buf, size_t len, int nul_term_line, int allow_miss
 	}
 
 	/* Check the type of object identified by sha1 */
-	obj_type = oid_object_info(&oid, NULL);
+	obj_type = oid_object_info(the_repository, &oid, NULL);
 	if (obj_type < 0) {
 		if (allow_missing) {
 			; /* no problem - missing objects are presumed to be of the right type */
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 4bdae5a1d8..8d4111f748 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1516,7 +1516,8 @@ static void check_object(struct object_entry *entry)
 		unuse_pack(&w_curs);
 	}
 
-	entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+	entry->type = oid_object_info(the_repository, &entry->idx.oid,
+				      &entry->size);
 	/*
 	 * The error condition is checked in prepare_pack().  This is
 	 * to permit a missing preferred base object to be ignored
@@ -1578,7 +1579,8 @@ static void drop_reused_delta(struct object_entry *entry)
 		 * And if that fails, the error will be recorded in entry->type
 		 * and dealt with in prepare_pack().
 		 */
-		entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+		entry->type = oid_object_info(the_repository, &entry->idx.oid,
+					      &entry->size);
 	}
 }
 
@@ -2706,7 +2708,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 static int add_loose_object(const struct object_id *oid, const char *path,
 			    void *data)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	if (type < 0) {
 		warning("loose object at %s could not be examined", path);
diff --git a/builtin/prune.c b/builtin/prune.c
index 38ced18dad..518ffbea13 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -50,7 +50,8 @@ static int prune_object(const struct object_id *oid, const char *fullpath,
 	if (st.st_mtime > expire)
 		return 0;
 	if (show_only || verbose) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		printf("%s %s\n", oid_to_hex(oid),
 		       (type > 0) ? type_name(type) : "unknown");
 	}
diff --git a/builtin/replace.c b/builtin/replace.c
index 237ea656cf..14e142d5a8 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -55,8 +55,9 @@ static int show_reference(const char *refname, const struct object_id *oid,
 			if (get_oid(refname, &object))
 				return error("Failed to resolve '%s' as a valid ref.", refname);
 
-			obj_type = oid_object_info(&object, NULL);
-			repl_type = oid_object_info(oid, NULL);
+			obj_type = oid_object_info(the_repository, &object,
+						   NULL);
+			repl_type = oid_object_info(the_repository, oid, NULL);
 
 			printf("%s (%s) -> %s (%s)\n", refname, type_name(obj_type),
 			       oid_to_hex(oid), type_name(repl_type));
@@ -164,8 +165,8 @@ static int replace_object_oid(const char *object_ref,
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
 
-	obj_type = oid_object_info(object, NULL);
-	repl_type = oid_object_info(repl, NULL);
+	obj_type = oid_object_info(the_repository, object, NULL);
+	repl_type = oid_object_info(the_repository, repl, NULL);
 	if (!force && obj_type != repl_type)
 		die("Objects must be of the same type.\n"
 		    "'%s' points to a replaced object of type '%s'\n"
@@ -292,7 +293,7 @@ static int edit_and_replace(const char *object_ref, int force, int raw)
 	if (get_oid(object_ref, &old_oid) < 0)
 		die("Not a valid object name: '%s'", object_ref);
 
-	type = oid_object_info(&old_oid, NULL);
+	type = oid_object_info(the_repository, &old_oid, NULL);
 	if (type < 0)
 		die("unable to get object type for %s", oid_to_hex(&old_oid));
 
diff --git a/builtin/tag.c b/builtin/tag.c
index 8cff6d0b72..26d7729f57 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -212,7 +212,7 @@ static void create_tag(const struct object_id *object, const char *tag,
 	struct strbuf header = STRBUF_INIT;
 	char *path = NULL;
 
-	type = oid_object_info(object, NULL);
+	type = oid_object_info(the_repository, object, NULL);
 	if (type <= OBJ_NONE)
 	    die(_("bad object type."));
 
@@ -298,7 +298,7 @@ static void create_reflog_msg(const struct object_id *oid, struct strbuf *sb)
 	}
 
 	strbuf_addstr(sb, " (");
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	switch (type) {
 	default:
 		strbuf_addstr(sb, "object of unknown type");
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index b7755c6cc5..cfe9019f80 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -199,7 +199,7 @@ static int check_object(struct object *obj, int type, void *data, struct fsck_op
 
 	if (!(obj->flags & FLAG_OPEN)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type != obj->type || type <= 0)
 			die("object of unexpected type");
 		obj->flags |= FLAG_WRITTEN;
diff --git a/cache.h b/cache.h
index 588c4fff9a..6340b2c572 100644
--- a/cache.h
+++ b/cache.h
@@ -1192,7 +1192,8 @@ static inline void *read_object_file(const struct object_id *oid, enum object_ty
 }
 
 /* Read and unpack an object file into memory, write memory to an object file */
-extern int oid_object_info(const struct object_id *, unsigned long *);
+#define oid_object_info(r, o, f) oid_object_info_##r(o, f)
+int oid_object_info_the_repository(const struct object_id *, unsigned long *);
 
 extern int hash_object_file(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid);
diff --git a/diff.c b/diff.c
index 1289df4b1f..4753170fe1 100644
--- a/diff.c
+++ b/diff.c
@@ -3638,7 +3638,8 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)
 	else {
 		enum object_type type;
 		if (size_only || (flags & CHECK_BINARY)) {
-			type = oid_object_info(&s->oid, &s->size);
+			type = oid_object_info(the_repository, &s->oid,
+					       &s->size);
 			if (type < 0)
 				die("unable to read %s",
 				    oid_to_hex(&s->oid));
diff --git a/fast-import.c b/fast-import.c
index 99f8f56e8c..afe06bd7c1 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1917,7 +1917,8 @@ static void read_marks(void)
 			die("corrupt mark line: %s", line);
 		e = find_object(&oid);
 		if (!e) {
-			enum object_type type = oid_object_info(&oid, NULL);
+			enum object_type type = oid_object_info(the_repository,
+								&oid, NULL);
 			if (type < 0)
 				die("object not found: %s", oid_to_hex(&oid));
 			e = insert_object(&oid);
@@ -2447,7 +2448,8 @@ static void file_change_m(const char *p, struct branch *b)
 		enum object_type expected = S_ISDIR(mode) ?
 						OBJ_TREE: OBJ_BLOB;
 		enum object_type type = oe ? oe->type :
-					oid_object_info(&oid, NULL);
+					oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("%s not found: %s",
 					S_ISDIR(mode) ?  "Tree" : "Blob",
@@ -2608,7 +2610,8 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa
 			die("Not a blob (actually a %s): %s",
 				type_name(oe->type), command_buf.buf);
 	} else if (!is_null_oid(&oid)) {
-		enum object_type type = oid_object_info(&oid, NULL);
+		enum object_type type = oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("Blob not found: %s", command_buf.buf);
 		if (type != OBJ_BLOB)
@@ -2895,7 +2898,7 @@ static void parse_new_tag(const char *arg)
 	} else if (!get_oid(from, &oid)) {
 		struct object_entry *oe = find_object(&oid);
 		if (!oe) {
-			type = oid_object_info(&oid, NULL);
+			type = oid_object_info(the_repository, &oid, NULL);
 			if (type < 0)
 				die("Not a valid object: %s", from);
 		} else
@@ -3053,7 +3056,8 @@ static struct object_entry *dereference(struct object_entry *oe,
 	unsigned long size;
 	char *buf = NULL;
 	if (!oe) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(oid));
 		/* cache it! */
diff --git a/list-objects-filter.c b/list-objects-filter.c
index 0ec83aaf18..ea94fe8af2 100644
--- a/list-objects-filter.c
+++ b/list-objects-filter.c
@@ -117,7 +117,7 @@ static enum list_objects_filter_result filter_blobs_limit(
 		assert(obj->type == OBJ_BLOB);
 		assert((obj->flags & SEEN) == 0);
 
-		t = oid_object_info(&obj->oid, &object_length);
+		t = oid_object_info(the_repository, &obj->oid, &object_length);
 		if (t != OBJ_BLOB) { /* probably OBJ_NONE */
 			/*
 			 * We DO NOT have the blob locally, so we cannot
diff --git a/object.c b/object.c
index 66cffaf6e5..5044d08e96 100644
--- a/object.c
+++ b/object.c
@@ -257,7 +257,7 @@ struct object *parse_object(const struct object_id *oid)
 
 	if ((obj && obj->type == OBJ_BLOB && has_object_file(oid)) ||
 	    (!obj && has_object_file(oid) &&
-	     oid_object_info(oid, NULL) == OBJ_BLOB)) {
+	     oid_object_info(the_repository, oid, NULL) == OBJ_BLOB)) {
 		if (check_object_signature(repl, NULL, 0, NULL) < 0) {
 			error("sha1 mismatch %s", oid_to_hex(oid));
 			return NULL;
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 41ae27fb19..cd1903e717 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -73,7 +73,8 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,
 			break;
 
 		default:
-			real_type = oid_object_info(&entry->idx.oid, NULL);
+			real_type = oid_object_info(the_repository,
+						    &entry->idx.oid, NULL);
 			break;
 		}
 
diff --git a/packfile.c b/packfile.c
index d9914ba723..80c7fa734f 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1114,7 +1114,7 @@ static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 		return OBJ_BAD;
 	nth_packed_object_oid(&oid, p, revidx->nr);
 	mark_bad_packed_object(p, oid.hash);
-	type = oid_object_info(&oid, NULL);
+	type = oid_object_info(the_repository, &oid, NULL);
 	if (type <= OBJ_NONE)
 		return OBJ_BAD;
 	return type;
diff --git a/reachable.c b/reachable.c
index a6ea33a5db..ffb976c33c 100644
--- a/reachable.c
+++ b/reachable.c
@@ -78,7 +78,7 @@ static void add_recent_object(const struct object_id *oid,
 	 * later processing, and the revision machinery expects
 	 * commits and tags to have been parsed.
 	 */
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("unable to get object info for %s", oid_to_hex(oid));
 
diff --git a/refs.c b/refs.c
index 9b56fa9b81..27c88ba768 100644
--- a/refs.c
+++ b/refs.c
@@ -302,7 +302,7 @@ enum peel_status peel_object(const struct object_id *name, struct object_id *oid
 	struct object *o = lookup_unknown_object(name->hash);
 
 	if (o->type == OBJ_NONE) {
-		int type = oid_object_info(name, NULL);
+		int type = oid_object_info(the_repository, name, NULL);
 		if (type < 0 || !object_as_type(o, type, 0))
 			return PEEL_INVALID;
 	}
diff --git a/remote.c b/remote.c
index 91eb010ca9..481bf933f3 100644
--- a/remote.c
+++ b/remote.c
@@ -1376,7 +1376,7 @@ static void add_missing_tags(struct ref *src, struct ref **dst, struct ref ***ds
 			continue; /* not a tag */
 		if (string_list_has_string(&dst_tag, ref->name))
 			continue; /* they already have it */
-		if (oid_object_info(&ref->new_oid, NULL) != OBJ_TAG)
+		if (oid_object_info(the_repository, &ref->new_oid, NULL) != OBJ_TAG)
 			continue; /* be conservative */
 		item = string_list_append(&src_tag, ref->name);
 		item->util = ref;
diff --git a/sequencer.c b/sequencer.c
index 667f35ebdf..44f0518b9c 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -2876,7 +2876,8 @@ int sequencer_pick_revisions(struct replay_opts *opts)
 
 		if (!get_oid(name, &oid)) {
 			if (!lookup_commit_reference_gently(&oid, 1)) {
-				enum object_type type = oid_object_info(&oid,
+				enum object_type type = oid_object_info(the_repository,
+									&oid,
 									NULL);
 				return error(_("%s: can't cherry-pick a %s"),
 					name, type_name(type));
diff --git a/sha1_file.c b/sha1_file.c
index 50a2dc5f0a..93f25c6c6a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1322,7 +1322,7 @@ int oid_object_info_extended_the_repository(const struct object_id *oid, struct
 }
 
 /* returns enum object_type or negative */
-int oid_object_info(const struct object_id *oid, unsigned long *sizep)
+int oid_object_info_the_repository(const struct object_id *oid, unsigned long *sizep)
 {
 	enum object_type type;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1988,7 +1988,7 @@ int read_pack_header(int fd, struct pack_header *header)
 
 void assert_oid_type(const struct object_id *oid, enum object_type expect)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("%s is not a valid object", oid_to_hex(oid));
 	if (type != expect)
diff --git a/sha1_name.c b/sha1_name.c
index 5b93bf8da3..b5406b6eb2 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -223,7 +223,7 @@ static int finish_object_disambiguation(struct disambiguate_state *ds,
 
 static int disambiguate_commit_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_COMMIT;
 }
 
@@ -232,7 +232,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -247,7 +247,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 
 static int disambiguate_tree_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_TREE;
 }
 
@@ -256,7 +256,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_TREE || kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -271,7 +271,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 
 static int disambiguate_blob_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_BLOB;
 }
 
@@ -350,7 +350,7 @@ static int show_ambiguous_object(const struct object_id *oid, void *data)
 	if (ds->fn && !ds->fn(oid, ds->cb_data))
 		return 0;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type == OBJ_COMMIT) {
 		struct commit *commit = lookup_commit(oid);
 		if (commit) {
diff --git a/submodule.c b/submodule.c
index 9a50168b23..bb133e9b93 100644
--- a/submodule.c
+++ b/submodule.c
@@ -818,7 +818,7 @@ static int check_has_commit(const struct object_id *oid, void *data)
 {
 	struct has_commit_data *cb = data;
 
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	switch (type) {
 	case OBJ_COMMIT:
diff --git a/tag.c b/tag.c
index 86b1dcbb82..3d37c1bd25 100644
--- a/tag.c
+++ b/tag.c
@@ -41,7 +41,7 @@ int gpg_verify_tag(const struct object_id *oid, const char *name_to_report,
 	unsigned long size;
 	int ret;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type != OBJ_TAG)
 		return error("%s: cannot verify a non-tag object of type %s.",
 				name_to_report ?
-- 
2.17.0.484.g0c8726318c-goog


^ permalink raw reply	[relevance 3%]

* [PATCH 6/7] diff.c: decouple white space treatment from move detection algorithm
      [irrelevant] <20180424210330.87861-1-sbeller@google.com>
@ 2018-04-24 21:03 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-24 21:03 UTC (permalink / raw)
  To: git; +Cc: jonathantanmy, simon, avarab, jacob.keller, Stefan Beller

In the original implementation of the move detection logic the choice for
ignoring white space changes is the same for the move detection as it is
for the regular diff.  Some cases came up where different treatment would
have been nice.

Allow the user to specify that whitespace should be ignored differently
during detection of moved lines than during generation of added and removed
lines. This is done by providing analogs to the --ignore-space-at-eol,
-b, and -w options (namely,
  --color-moved-[no-]ignore-space-at-eol
  --color-moved-[no-]ignore-space-change
  --color-moved-[no-]ignore-all-space) that affect only the color of the
output, and making the existing --ignore-space-at-eol, -b, and -w options
no longer affect the color of the output.

As we change the default, we'll adjust the tests.

For now we do not infer any options to treat whitespaces in the move
detection from the generic white space options given to diff.
This can be tuned later to reasonable default.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/diff-options.txt | 13 +++++
 diff.c                         | 19 ++++++-
 diff.h                         |  1 +
 t/t4015-diff-whitespace.sh     | 90 +++++++++++++++++++++++++++++++---
 4 files changed, 114 insertions(+), 9 deletions(-)

diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index bb9f1b7cd8..7b2527b9a1 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -292,6 +292,19 @@ dimmed_zebra::
 	blocks are considered interesting, the rest is uninteresting.
 --
 
+--color-moved-[no-]ignore-space-at-eol::
+	Ignore changes in whitespace at EOL when performing the move
+	detection for --color-moved.
+--color-moved-[no-]ignore-space-change::
+	Ignore changes in amount of whitespace when performing the move
+	detection for --color-moved.  This ignores whitespace
+	at line end, and considers all other sequences of one or
+	more whitespace characters to be equivalent.
+--color-moved-[no-]ignore-all-space::
+	Ignore whitespace when comparing lines when performing the move
+	detection for --color-moved.  This ignores differences even if
+	one line has whitespace where the other line has none.
+
 --word-diff[=<mode>]::
 	Show a word diff, using the <mode> to delimit changed words.
 	By default, words are delimited by whitespace; see
diff --git a/diff.c b/diff.c
index 95c51c0b7d..b5819dd538 100644
--- a/diff.c
+++ b/diff.c
@@ -717,10 +717,12 @@ static int moved_entry_cmp(const void *hashmap_cmp_fn_data,
 	const struct diff_options *diffopt = hashmap_cmp_fn_data;
 	const struct moved_entry *a = entry;
 	const struct moved_entry *b = entry_or_key;
+	unsigned flags = diffopt->color_moved_ws_handling
+			 & XDF_WHITESPACE_FLAGS;
 
 	return !xdiff_compare_lines(a->es->line, a->es->len,
 				    b->es->line, b->es->len,
-				    diffopt->xdl_opts);
+				    flags);
 }
 
 static struct moved_entry *prepare_entry(struct diff_options *o,
@@ -728,8 +730,9 @@ static struct moved_entry *prepare_entry(struct diff_options *o,
 {
 	struct moved_entry *ret = xmalloc(sizeof(*ret));
 	struct emitted_diff_symbol *l = &o->emitted_symbols->buf[line_no];
+	unsigned flags = o->color_moved_ws_handling & XDF_WHITESPACE_FLAGS;
 
-	ret->ent.hash = xdiff_hash_string(l->line, l->len, o->xdl_opts);
+	ret->ent.hash = xdiff_hash_string(l->line, l->len, flags);
 	ret->es = l;
 	ret->next_line = NULL;
 
@@ -4638,6 +4641,18 @@ int diff_opt_parse(struct diff_options *options,
 		DIFF_XDL_SET(options, IGNORE_CR_AT_EOL);
 	else if (!strcmp(arg, "--ignore-blank-lines"))
 		DIFF_XDL_SET(options, IGNORE_BLANK_LINES);
+	else if (!strcmp(arg, "--color-moved-no-ignore-all-space"))
+		options->color_moved_ws_handling &= ~XDF_IGNORE_WHITESPACE;
+	else if (!strcmp(arg, "--color-moved-no-ignore-space-change"))
+		options->color_moved_ws_handling &= ~XDF_IGNORE_WHITESPACE_CHANGE;
+	else if (!strcmp(arg, "--color-moved-no-ignore-space-at-eol"))
+		options->color_moved_ws_handling &= ~XDF_IGNORE_WHITESPACE_AT_EOL;
+	else if (!strcmp(arg, "--color-moved-ignore-all-space"))
+		options->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE;
+	else if (!strcmp(arg, "--color-moved-ignore-space-change"))
+		options->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE_CHANGE;
+	else if (!strcmp(arg, "--color-moved-ignore-space-at-eol"))
+		options->color_moved_ws_handling |= XDF_IGNORE_WHITESPACE_AT_EOL;
 	else if (!strcmp(arg, "--indent-heuristic"))
 		DIFF_XDL_SET(options, INDENT_HEURISTIC);
 	else if (!strcmp(arg, "--no-indent-heuristic"))
diff --git a/diff.h b/diff.h
index 7bd4f182c3..de5dc68005 100644
--- a/diff.h
+++ b/diff.h
@@ -214,6 +214,7 @@ struct diff_options {
 	} color_moved;
 	#define COLOR_MOVED_DEFAULT COLOR_MOVED_ZEBRA
 	#define COLOR_MOVED_MIN_ALNUM_COUNT 20
+	int color_moved_ws_handling;
 };
 
 void diff_emit_submodule_del(struct diff_options *o, const char *line);
diff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh
index 45091abb19..751fc478dd 100755
--- a/t/t4015-diff-whitespace.sh
+++ b/t/t4015-diff-whitespace.sh
@@ -1441,7 +1441,10 @@ test_expect_success 'move detection ignoring whitespace ' '
 	line 4
 	line 5
 	EOF
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1465,7 +1468,10 @@ test_expect_success 'move detection ignoring whitespace ' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames -w --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1505,7 +1511,10 @@ test_expect_success 'move detection ignoring whitespace changes' '
 	line 5
 	EOF
 
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1529,7 +1538,10 @@ test_expect_success 'move detection ignoring whitespace changes' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames -b --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-at-eol \
+		--color-moved-ignore-space-change |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1572,7 +1584,10 @@ test_expect_success 'move detection ignoring whitespace at eol' '
 	# avoid cluttering the output with complaints about our eol whitespace
 	test_config core.whitespace -blank-at-eol &&
 
-	git diff HEAD --no-renames --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1596,7 +1611,10 @@ test_expect_success 'move detection ignoring whitespace at eol' '
 	EOF
 	test_cmp expected actual &&
 
-	git diff HEAD --no-renames --ignore-space-at-eol --color-moved --color |
+	git diff HEAD --no-renames --color-moved --color \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-ignore-space-at-eol |
 		grep -v "index" |
 		test_decode_color >actual &&
 	cat <<-\EOF >expected &&
@@ -1768,7 +1786,65 @@ test_expect_success 'move detection with submodules' '
 
 	# nor did we mess with it another way
 	git diff --submodule=diff --color | test_decode_color >expect &&
-	test_cmp expect decoded_actual
+	test_cmp expect decoded_actual &&
+	rm -rf bananas &&
+	git submodule deinit bananas
+'
+
+test_expect_success 'only move detection ignores white spaces' '
+	git reset --hard &&
+	q_to_tab <<-\EOF >text.txt &&
+		a long line to exceed per-line minimum
+		another long line to exceed per-line minimum
+		original file
+	EOF
+	git add text.txt &&
+	git commit -m "add text" &&
+	q_to_tab <<-\EOF >text.txt &&
+		Qa long line to exceed per-line minimum
+		Qanother long line to exceed per-line minimum
+		new file
+	EOF
+
+	# Make sure we get a different diff using -w
+	git diff --color --color-moved -w \
+		--color-moved-no-ignore-all-space \
+		--color-moved-no-ignore-space-change \
+		--color-moved-no-ignore-space-at-eol |
+		grep -v "index" |
+		test_decode_color >actual &&
+	q_to_tab <<-\EOF >expected &&
+	<BOLD>diff --git a/text.txt b/text.txt<RESET>
+	<BOLD>--- a/text.txt<RESET>
+	<BOLD>+++ b/text.txt<RESET>
+	<CYAN>@@ -1,3 +1,3 @@<RESET>
+	 Qa long line to exceed per-line minimum<RESET>
+	 Qanother long line to exceed per-line minimum<RESET>
+	<RED>-original file<RESET>
+	<GREEN>+<RESET><GREEN>new file<RESET>
+	EOF
+	test_cmp expected actual &&
+
+	# And now ignoring white space only in the move detection
+	git diff --color --color-moved \
+		--color-moved-ignore-all-space \
+		--color-moved-ignore-space-change \
+		--color-moved-ignore-space-at-eol |
+		grep -v "index" |
+		test_decode_color >actual &&
+	q_to_tab <<-\EOF >expected &&
+	<BOLD>diff --git a/text.txt b/text.txt<RESET>
+	<BOLD>--- a/text.txt<RESET>
+	<BOLD>+++ b/text.txt<RESET>
+	<CYAN>@@ -1,3 +1,3 @@<RESET>
+	<BOLD;MAGENTA>-a long line to exceed per-line minimum<RESET>
+	<BOLD;MAGENTA>-another long line to exceed per-line minimum<RESET>
+	<RED>-original file<RESET>
+	<BOLD;YELLOW>+<RESET>Q<BOLD;YELLOW>a long line to exceed per-line minimum<RESET>
+	<BOLD;YELLOW>+<RESET>Q<BOLD;YELLOW>another long line to exceed per-line minimum<RESET>
+	<GREEN>+<RESET><GREEN>new file<RESET>
+	EOF
+	test_cmp expected actual
 '
 
 test_done
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 7%]

* [PATCHv2 0/9] object store: oid_object_info is the next contender
@ 2018-04-24 21:59 Stefan Beller
  2018-04-24 21:59 ` [PATCHv2 2/9] cache.h: add repository argument to oid_object_info Stefan Beller
                   ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Stefan Beller @ 2018-04-24 21:59 UTC (permalink / raw)
  To: git, gitster; +Cc: bmwill, jonathantanmy, sandals, Stefan Beller

v2:

* fixed the sha1/oid typo
* removed spurious new line
* Brandon and Jonthan discovered another dependency that I missed due
  to cherrypicking that commit from a tree before partial clone was a thing.
  We error out when attempting to use fetch_object for repos that are not
  the_repository.
  
Thanks,
Stefan

v1:
This applies on top of origin/sb/object-store-replace and is available as
https://github.com/stefanbeller/git/tree/oid_object_info

This continues the work of sb/packfiles-in-repository,
extending the layer at which we have to pass in an explicit
repository object to oid_object_info.

A test merge to next shows only a minor merge conflicit (adding
different #include lines in one c file), so this might be a good next
step for the object store series.

Notes on further object store series:
I plan on converting the "parsed object store" next,
which would be {alloc, object, tree, commit, tag}.c as that is a prerequisite
for migrating shallow (which is intermingled with grafts) information to the
object store.

There is currently work going on in allocation (mempool - Jameson Miller)
and grafts (deprecate grafts - DScho), which is why I am sending this
series first. I think it can go in parallel to the "parsed object store"
that is coming next.

Thanks,
Stefan

Jonathan Nieder (1):
  packfile: add repository argument to packed_object_info

Stefan Beller (8):
  cache.h: add repository argument to oid_object_info_extended
  cache.h: add repository argument to oid_object_info
  packfile: add repository argument to retry_bad_packed_offset
  packfile: add repository argument to packed_to_object_type
  packfile: add repository argument to read_object
  packfile: add repository argument to unpack_entry
  packfile: add repository argument to cache_or_unpack_entry
  cache.h: allow oid_object_info to handle arbitrary repositories

 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       | 12 ++++++------
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   | 11 +++++++----
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  7 +++++--
 diff.c                   |  3 ++-
 fast-import.c            | 16 ++++++++++------
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 pack-check.c             |  3 ++-
 packfile.c               | 40 +++++++++++++++++++++++-----------------
 packfile.h               |  6 ++++--
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              | 36 +++++++++++++++++++++---------------
 sha1_name.c              | 12 ++++++------
 streaming.c              |  2 +-
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 35 files changed, 124 insertions(+), 92 deletions(-)

-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 5%]

* [PATCHv2 2/9] cache.h: add repository argument to oid_object_info
  2018-04-24 21:59 [PATCHv2 0/9] object store: oid_object_info is the next contender Stefan Beller
@ 2018-04-24 21:59 ` Stefan Beller
      [irrelevant] ` <20180424215910.22201-10-sbeller@google.com>
  2018-04-25 18:20 ` [PATCHv3 0/9] object store: oid_object_info is the next contender Stefan Beller
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-24 21:59 UTC (permalink / raw)
  To: git, gitster; +Cc: bmwill, jonathantanmy, sandals, Stefan Beller

Add a repository argument to allow the callers of oid_object_info
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       |  6 +++---
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   |  8 +++++---
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  3 ++-
 diff.c                   |  3 ++-
 fast-import.c            | 14 +++++++++-----
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 packfile.c               |  2 +-
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              |  4 ++--
 sha1_name.c              | 12 ++++++------
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 32 files changed, 67 insertions(+), 53 deletions(-)

diff --git a/archive-tar.c b/archive-tar.c
index 3563bcb9f2..f93409324f 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -276,7 +276,7 @@ static int write_tar_entry(struct archiver_args *args,
 		memcpy(header.name, path, pathlen);
 
 	if (S_ISREG(mode) && !args->convert &&
-	    oid_object_info(oid, &size) == OBJ_BLOB &&
+	    oid_object_info(the_repository, oid, &size) == OBJ_BLOB &&
 	    size > big_file_threshold)
 		buffer = NULL;
 	else if (S_ISLNK(mode) || S_ISREG(mode)) {
diff --git a/archive-zip.c b/archive-zip.c
index 6b20bce4d1..74f3fe9103 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -325,7 +325,8 @@ static int write_zip_entry(struct archiver_args *args,
 		compressed_size = 0;
 		buffer = NULL;
 	} else if (S_ISREG(mode) || S_ISLNK(mode)) {
-		enum object_type type = oid_object_info(oid, &size);
+		enum object_type type = oid_object_info(the_repository, oid,
+							&size);
 
 		method = 0;
 		attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :
diff --git a/blame.c b/blame.c
index 78c9808bd1..dfa24473dc 100644
--- a/blame.c
+++ b/blame.c
@@ -81,7 +81,7 @@ static void verify_working_tree_path(struct commit *work_tree, const char *path)
 		unsigned mode;
 
 		if (!get_tree_entry(commit_oid, path, &blob_oid, &mode) &&
-		    oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+		    oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 			return;
 	}
 
@@ -504,7 +504,7 @@ static int fill_blob_sha1_and_mode(struct blame_origin *origin)
 		return 0;
 	if (get_tree_entry(&origin->commit->object.oid, origin->path, &origin->blob_oid, &origin->mode))
 		goto error_out;
-	if (oid_object_info(&origin->blob_oid, NULL) != OBJ_BLOB)
+	if (oid_object_info(the_repository, &origin->blob_oid, NULL) != OBJ_BLOB)
 		goto error_out;
 	return 0;
  error_out:
diff --git a/builtin/blame.c b/builtin/blame.c
index db38c0b307..bfdf7cc132 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -655,7 +655,7 @@ static int is_a_rev(const char *name)
 
 	if (get_oid(name, &oid))
 		return 0;
-	return OBJ_NONE < oid_object_info(&oid, NULL);
+	return OBJ_NONE < oid_object_info(the_repository, &oid, NULL);
 }
 
 int cmd_blame(int argc, const char **argv, const char *prefix)
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 4ecdb9ff54..b8ecbea98e 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -116,7 +116,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 		/* else fallthrough */
 
 	case 'p':
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("Not a valid object name %s", obj_name);
 
@@ -140,7 +140,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 	case 0:
 		if (type_from_string(exp_type) == OBJ_BLOB) {
 			struct object_id blob_oid;
-			if (oid_object_info(&oid, NULL) == OBJ_TAG) {
+			if (oid_object_info(the_repository, &oid, NULL) == OBJ_TAG) {
 				char *buffer = read_object_file(&oid, &type,
 								&size);
 				const char *target;
@@ -151,7 +151,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 			} else
 				oidcpy(&blob_oid, &oid);
 
-			if (oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+			if (oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 				return stream_blob_to_fd(1, &blob_oid, NULL, 0);
 			/*
 			 * we attempted to dereference a tag to a blob
diff --git a/builtin/describe.c b/builtin/describe.c
index de840f96a4..66c497f789 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -502,7 +502,7 @@ static void describe(const char *arg, int last_one)
 
 	if (cmit)
 		describe_commit(&oid, &sb);
-	else if (oid_object_info(&oid, NULL) == OBJ_BLOB)
+	else if (oid_object_info(the_repository, &oid, NULL) == OBJ_BLOB)
 		describe_blob(oid, &sb);
 	else
 		die(_("%s is neither a commit nor blob"), arg);
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index a15898d641..373c794873 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -947,7 +947,7 @@ static void import_marks(char *input_file)
 		if (last_idnum < mark)
 			last_idnum = mark;
 
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(&oid));
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index dcdfc66f09..73be393b2e 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -637,7 +637,7 @@ static int update_local_ref(struct ref *ref,
 	struct branch *current_branch = branch_get(NULL);
 	const char *pretty_ref = prettify_refname(ref->name);
 
-	type = oid_object_info(&ref->new_oid, NULL);
+	type = oid_object_info(the_repository, &ref->new_oid, NULL);
 	if (type < 0)
 		die(_("object %s not found"), oid_to_hex(&ref->new_oid));
 
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 087360a675..9d59d7d5a2 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -67,7 +67,8 @@ static const char *printable_type(struct object *obj)
 	const char *ret;
 
 	if (obj->type == OBJ_NONE) {
-		enum object_type type = oid_object_info(&obj->oid, NULL);
+		enum object_type type = oid_object_info(the_repository,
+							&obj->oid, NULL);
 		if (type > 0)
 			object_as_type(obj, type, 0);
 	}
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index d81473e722..2d04a596f5 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -223,7 +223,7 @@ static unsigned check_object(struct object *obj)
 
 	if (!(obj->flags & FLAG_CHECKED)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type <= 0)
 			die(_("did not receive expected object %s"),
 			      oid_to_hex(&obj->oid));
@@ -812,7 +812,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,
 		enum object_type has_type;
 		unsigned long has_size;
 		read_lock();
-		has_type = oid_object_info(oid, &has_size);
+		has_type = oid_object_info(the_repository, oid, &has_size);
 		if (has_type < 0)
 			die(_("cannot read existing object info %s"), oid_to_hex(oid));
 		if (has_type != type || has_size != size)
diff --git a/builtin/ls-tree.c b/builtin/ls-tree.c
index d44b4f9c27..409da4e835 100644
--- a/builtin/ls-tree.c
+++ b/builtin/ls-tree.c
@@ -94,7 +94,7 @@ static int show_tree(const struct object_id *oid, struct strbuf *base,
 			char size_text[24];
 			if (!strcmp(type, blob_type)) {
 				unsigned long size;
-				if (oid_object_info(oid, &size) == OBJ_BAD)
+				if (oid_object_info(the_repository, oid, &size) == OBJ_BAD)
 					xsnprintf(size_text, sizeof(size_text),
 						  "BAD");
 				else
diff --git a/builtin/mktree.c b/builtin/mktree.c
index 263c530315..bb76b469fd 100644
--- a/builtin/mktree.c
+++ b/builtin/mktree.c
@@ -116,7 +116,7 @@ static void mktree_line(char *buf, size_t len, int nul_term_line, int allow_miss
 	}
 
 	/* Check the type of object identified by sha1 */
-	obj_type = oid_object_info(&oid, NULL);
+	obj_type = oid_object_info(the_repository, &oid, NULL);
 	if (obj_type < 0) {
 		if (allow_missing) {
 			; /* no problem - missing objects are presumed to be of the right type */
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 4bdae5a1d8..8d4111f748 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1516,7 +1516,8 @@ static void check_object(struct object_entry *entry)
 		unuse_pack(&w_curs);
 	}
 
-	entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+	entry->type = oid_object_info(the_repository, &entry->idx.oid,
+				      &entry->size);
 	/*
 	 * The error condition is checked in prepare_pack().  This is
 	 * to permit a missing preferred base object to be ignored
@@ -1578,7 +1579,8 @@ static void drop_reused_delta(struct object_entry *entry)
 		 * And if that fails, the error will be recorded in entry->type
 		 * and dealt with in prepare_pack().
 		 */
-		entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+		entry->type = oid_object_info(the_repository, &entry->idx.oid,
+					      &entry->size);
 	}
 }
 
@@ -2706,7 +2708,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 static int add_loose_object(const struct object_id *oid, const char *path,
 			    void *data)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	if (type < 0) {
 		warning("loose object at %s could not be examined", path);
diff --git a/builtin/prune.c b/builtin/prune.c
index 38ced18dad..518ffbea13 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -50,7 +50,8 @@ static int prune_object(const struct object_id *oid, const char *fullpath,
 	if (st.st_mtime > expire)
 		return 0;
 	if (show_only || verbose) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		printf("%s %s\n", oid_to_hex(oid),
 		       (type > 0) ? type_name(type) : "unknown");
 	}
diff --git a/builtin/replace.c b/builtin/replace.c
index 237ea656cf..14e142d5a8 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -55,8 +55,9 @@ static int show_reference(const char *refname, const struct object_id *oid,
 			if (get_oid(refname, &object))
 				return error("Failed to resolve '%s' as a valid ref.", refname);
 
-			obj_type = oid_object_info(&object, NULL);
-			repl_type = oid_object_info(oid, NULL);
+			obj_type = oid_object_info(the_repository, &object,
+						   NULL);
+			repl_type = oid_object_info(the_repository, oid, NULL);
 
 			printf("%s (%s) -> %s (%s)\n", refname, type_name(obj_type),
 			       oid_to_hex(oid), type_name(repl_type));
@@ -164,8 +165,8 @@ static int replace_object_oid(const char *object_ref,
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
 
-	obj_type = oid_object_info(object, NULL);
-	repl_type = oid_object_info(repl, NULL);
+	obj_type = oid_object_info(the_repository, object, NULL);
+	repl_type = oid_object_info(the_repository, repl, NULL);
 	if (!force && obj_type != repl_type)
 		die("Objects must be of the same type.\n"
 		    "'%s' points to a replaced object of type '%s'\n"
@@ -292,7 +293,7 @@ static int edit_and_replace(const char *object_ref, int force, int raw)
 	if (get_oid(object_ref, &old_oid) < 0)
 		die("Not a valid object name: '%s'", object_ref);
 
-	type = oid_object_info(&old_oid, NULL);
+	type = oid_object_info(the_repository, &old_oid, NULL);
 	if (type < 0)
 		die("unable to get object type for %s", oid_to_hex(&old_oid));
 
diff --git a/builtin/tag.c b/builtin/tag.c
index 8cff6d0b72..26d7729f57 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -212,7 +212,7 @@ static void create_tag(const struct object_id *object, const char *tag,
 	struct strbuf header = STRBUF_INIT;
 	char *path = NULL;
 
-	type = oid_object_info(object, NULL);
+	type = oid_object_info(the_repository, object, NULL);
 	if (type <= OBJ_NONE)
 	    die(_("bad object type."));
 
@@ -298,7 +298,7 @@ static void create_reflog_msg(const struct object_id *oid, struct strbuf *sb)
 	}
 
 	strbuf_addstr(sb, " (");
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	switch (type) {
 	default:
 		strbuf_addstr(sb, "object of unknown type");
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index b7755c6cc5..cfe9019f80 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -199,7 +199,7 @@ static int check_object(struct object *obj, int type, void *data, struct fsck_op
 
 	if (!(obj->flags & FLAG_OPEN)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type != obj->type || type <= 0)
 			die("object of unexpected type");
 		obj->flags |= FLAG_WRITTEN;
diff --git a/cache.h b/cache.h
index 588c4fff9a..6340b2c572 100644
--- a/cache.h
+++ b/cache.h
@@ -1192,7 +1192,8 @@ static inline void *read_object_file(const struct object_id *oid, enum object_ty
 }
 
 /* Read and unpack an object file into memory, write memory to an object file */
-extern int oid_object_info(const struct object_id *, unsigned long *);
+#define oid_object_info(r, o, f) oid_object_info_##r(o, f)
+int oid_object_info_the_repository(const struct object_id *, unsigned long *);
 
 extern int hash_object_file(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid);
diff --git a/diff.c b/diff.c
index 1289df4b1f..4753170fe1 100644
--- a/diff.c
+++ b/diff.c
@@ -3638,7 +3638,8 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)
 	else {
 		enum object_type type;
 		if (size_only || (flags & CHECK_BINARY)) {
-			type = oid_object_info(&s->oid, &s->size);
+			type = oid_object_info(the_repository, &s->oid,
+					       &s->size);
 			if (type < 0)
 				die("unable to read %s",
 				    oid_to_hex(&s->oid));
diff --git a/fast-import.c b/fast-import.c
index 99f8f56e8c..afe06bd7c1 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1917,7 +1917,8 @@ static void read_marks(void)
 			die("corrupt mark line: %s", line);
 		e = find_object(&oid);
 		if (!e) {
-			enum object_type type = oid_object_info(&oid, NULL);
+			enum object_type type = oid_object_info(the_repository,
+								&oid, NULL);
 			if (type < 0)
 				die("object not found: %s", oid_to_hex(&oid));
 			e = insert_object(&oid);
@@ -2447,7 +2448,8 @@ static void file_change_m(const char *p, struct branch *b)
 		enum object_type expected = S_ISDIR(mode) ?
 						OBJ_TREE: OBJ_BLOB;
 		enum object_type type = oe ? oe->type :
-					oid_object_info(&oid, NULL);
+					oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("%s not found: %s",
 					S_ISDIR(mode) ?  "Tree" : "Blob",
@@ -2608,7 +2610,8 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa
 			die("Not a blob (actually a %s): %s",
 				type_name(oe->type), command_buf.buf);
 	} else if (!is_null_oid(&oid)) {
-		enum object_type type = oid_object_info(&oid, NULL);
+		enum object_type type = oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("Blob not found: %s", command_buf.buf);
 		if (type != OBJ_BLOB)
@@ -2895,7 +2898,7 @@ static void parse_new_tag(const char *arg)
 	} else if (!get_oid(from, &oid)) {
 		struct object_entry *oe = find_object(&oid);
 		if (!oe) {
-			type = oid_object_info(&oid, NULL);
+			type = oid_object_info(the_repository, &oid, NULL);
 			if (type < 0)
 				die("Not a valid object: %s", from);
 		} else
@@ -3053,7 +3056,8 @@ static struct object_entry *dereference(struct object_entry *oe,
 	unsigned long size;
 	char *buf = NULL;
 	if (!oe) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(oid));
 		/* cache it! */
diff --git a/list-objects-filter.c b/list-objects-filter.c
index 0ec83aaf18..ea94fe8af2 100644
--- a/list-objects-filter.c
+++ b/list-objects-filter.c
@@ -117,7 +117,7 @@ static enum list_objects_filter_result filter_blobs_limit(
 		assert(obj->type == OBJ_BLOB);
 		assert((obj->flags & SEEN) == 0);
 
-		t = oid_object_info(&obj->oid, &object_length);
+		t = oid_object_info(the_repository, &obj->oid, &object_length);
 		if (t != OBJ_BLOB) { /* probably OBJ_NONE */
 			/*
 			 * We DO NOT have the blob locally, so we cannot
diff --git a/object.c b/object.c
index 66cffaf6e5..5044d08e96 100644
--- a/object.c
+++ b/object.c
@@ -257,7 +257,7 @@ struct object *parse_object(const struct object_id *oid)
 
 	if ((obj && obj->type == OBJ_BLOB && has_object_file(oid)) ||
 	    (!obj && has_object_file(oid) &&
-	     oid_object_info(oid, NULL) == OBJ_BLOB)) {
+	     oid_object_info(the_repository, oid, NULL) == OBJ_BLOB)) {
 		if (check_object_signature(repl, NULL, 0, NULL) < 0) {
 			error("sha1 mismatch %s", oid_to_hex(oid));
 			return NULL;
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 41ae27fb19..cd1903e717 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -73,7 +73,8 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,
 			break;
 
 		default:
-			real_type = oid_object_info(&entry->idx.oid, NULL);
+			real_type = oid_object_info(the_repository,
+						    &entry->idx.oid, NULL);
 			break;
 		}
 
diff --git a/packfile.c b/packfile.c
index d9914ba723..80c7fa734f 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1114,7 +1114,7 @@ static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 		return OBJ_BAD;
 	nth_packed_object_oid(&oid, p, revidx->nr);
 	mark_bad_packed_object(p, oid.hash);
-	type = oid_object_info(&oid, NULL);
+	type = oid_object_info(the_repository, &oid, NULL);
 	if (type <= OBJ_NONE)
 		return OBJ_BAD;
 	return type;
diff --git a/reachable.c b/reachable.c
index a6ea33a5db..ffb976c33c 100644
--- a/reachable.c
+++ b/reachable.c
@@ -78,7 +78,7 @@ static void add_recent_object(const struct object_id *oid,
 	 * later processing, and the revision machinery expects
 	 * commits and tags to have been parsed.
 	 */
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("unable to get object info for %s", oid_to_hex(oid));
 
diff --git a/refs.c b/refs.c
index 9b56fa9b81..27c88ba768 100644
--- a/refs.c
+++ b/refs.c
@@ -302,7 +302,7 @@ enum peel_status peel_object(const struct object_id *name, struct object_id *oid
 	struct object *o = lookup_unknown_object(name->hash);
 
 	if (o->type == OBJ_NONE) {
-		int type = oid_object_info(name, NULL);
+		int type = oid_object_info(the_repository, name, NULL);
 		if (type < 0 || !object_as_type(o, type, 0))
 			return PEEL_INVALID;
 	}
diff --git a/remote.c b/remote.c
index 91eb010ca9..481bf933f3 100644
--- a/remote.c
+++ b/remote.c
@@ -1376,7 +1376,7 @@ static void add_missing_tags(struct ref *src, struct ref **dst, struct ref ***ds
 			continue; /* not a tag */
 		if (string_list_has_string(&dst_tag, ref->name))
 			continue; /* they already have it */
-		if (oid_object_info(&ref->new_oid, NULL) != OBJ_TAG)
+		if (oid_object_info(the_repository, &ref->new_oid, NULL) != OBJ_TAG)
 			continue; /* be conservative */
 		item = string_list_append(&src_tag, ref->name);
 		item->util = ref;
diff --git a/sequencer.c b/sequencer.c
index 667f35ebdf..44f0518b9c 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -2876,7 +2876,8 @@ int sequencer_pick_revisions(struct replay_opts *opts)
 
 		if (!get_oid(name, &oid)) {
 			if (!lookup_commit_reference_gently(&oid, 1)) {
-				enum object_type type = oid_object_info(&oid,
+				enum object_type type = oid_object_info(the_repository,
+									&oid,
 									NULL);
 				return error(_("%s: can't cherry-pick a %s"),
 					name, type_name(type));
diff --git a/sha1_file.c b/sha1_file.c
index 50a2dc5f0a..93f25c6c6a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1322,7 +1322,7 @@ int oid_object_info_extended_the_repository(const struct object_id *oid, struct
 }
 
 /* returns enum object_type or negative */
-int oid_object_info(const struct object_id *oid, unsigned long *sizep)
+int oid_object_info_the_repository(const struct object_id *oid, unsigned long *sizep)
 {
 	enum object_type type;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1988,7 +1988,7 @@ int read_pack_header(int fd, struct pack_header *header)
 
 void assert_oid_type(const struct object_id *oid, enum object_type expect)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("%s is not a valid object", oid_to_hex(oid));
 	if (type != expect)
diff --git a/sha1_name.c b/sha1_name.c
index 5b93bf8da3..b5406b6eb2 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -223,7 +223,7 @@ static int finish_object_disambiguation(struct disambiguate_state *ds,
 
 static int disambiguate_commit_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_COMMIT;
 }
 
@@ -232,7 +232,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -247,7 +247,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 
 static int disambiguate_tree_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_TREE;
 }
 
@@ -256,7 +256,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_TREE || kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -271,7 +271,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 
 static int disambiguate_blob_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_BLOB;
 }
 
@@ -350,7 +350,7 @@ static int show_ambiguous_object(const struct object_id *oid, void *data)
 	if (ds->fn && !ds->fn(oid, ds->cb_data))
 		return 0;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type == OBJ_COMMIT) {
 		struct commit *commit = lookup_commit(oid);
 		if (commit) {
diff --git a/submodule.c b/submodule.c
index 9a50168b23..bb133e9b93 100644
--- a/submodule.c
+++ b/submodule.c
@@ -818,7 +818,7 @@ static int check_has_commit(const struct object_id *oid, void *data)
 {
 	struct has_commit_data *cb = data;
 
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	switch (type) {
 	case OBJ_COMMIT:
diff --git a/tag.c b/tag.c
index 86b1dcbb82..3d37c1bd25 100644
--- a/tag.c
+++ b/tag.c
@@ -41,7 +41,7 @@ int gpg_verify_tag(const struct object_id *oid, const char *name_to_report,
 	unsigned long size;
 	int ret;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type != OBJ_TAG)
 		return error("%s: cannot verify a non-tag object of type %s.",
 				name_to_report ?
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 3%]

* Re: [PATCHv2 9/9] cache.h: allow oid_object_info to handle arbitrary repositories
      [irrelevant] ` <20180424215910.22201-10-sbeller@google.com>
@ 2018-04-24 22:49   ` Jonathan Tan
  0 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-04-24 22:49 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, bmwill, sandals, Jonathan Nieder

On Tue, 24 Apr 2018 14:59:09 -0700
Stefan Beller <sbeller@google.com> wrote:

> This involves also adapting oid_object_info_extended and a some
> internal functions that are used to implement these. It all has to
> happen in one patch, because of a single recursive chain of calls visits
> all these functions.

I wrote about delta_base_cache in a reply [1] to an earlier version,
which is indeed safe (as discussed), but I think that other reviewers
might have questions about that too so I think it's worth noting that in
the commit message. Maybe write something like:

  Among the functions modified to handle arbitrary repositories,
  unpack_entry() is one of them. Note that it still references the
  globals "delta_base_cache" and "delta_base_cached", but those are safe
  to be referenced (the former is indexed partly by "struct packed_git
  *", which is repo-specific, and the latter is only used to limit the
  size of the former as an optimization).

[1] https://public-inbox.org/git/20180424112332.38c0d04d96689f030e96825a@google.com/

> sha1_object_info_extended is also used in partial clones, which allow
> fetching missing objects. As this series will not add the repository
> struct to the transport code and fetch_object(), add a TODO note and
> bug out if a user tries to use a partial clone in a repository other than
> the_repository.

s/sha1_object/oid_object/ (in the 2nd paragraph)

Also, you sent 2 versions of PATCHv2 9/9.
  
> @@ -1290,9 +1291,12 @@ int oid_object_info_extended_the_repository(const struct object_id *oid, struct
>  		if (fetch_if_missing && repository_format_partial_clone &&
>  		    !already_retried) {
>  			/*
> -			 * TODO Investigate haveing fetch_object() return
> +			 * TODO Investigate having fetch_object() return
>  			 * TODO error/success and stopping the music here.
> +			 * TODO Pass a repository struct through fetch_object.
>  			 */
> +			if (r != the_repository)
> +				die(_("partial clones only supported in the_repository"));
>  			fetch_object(repository_format_partial_clone, real->hash);
>  			already_retried = 1;
>  			continue;

This most likely means that a partial clone with a submodule would
wrongly error out here. Instead, the "r == the_repository" check should
be done in the same "if" statement as repository_format_partial_clone
(and no "die"-ing occurs if it fails - just that there will be no
fetching of objects).

^ permalink raw reply	[relevance 4%]

* [PATCHv3 0/9] object store: oid_object_info is the next contender
  2018-04-24 21:59 [PATCHv2 0/9] object store: oid_object_info is the next contender Stefan Beller
  2018-04-24 21:59 ` [PATCHv2 2/9] cache.h: add repository argument to oid_object_info Stefan Beller
      [irrelevant] ` <20180424215910.22201-10-sbeller@google.com>
@ 2018-04-25 18:20 ` Stefan Beller
  2018-04-25 18:20   ` [PATCHv3 2/9] cache.h: add repository argument to oid_object_info Stefan Beller
  2018-04-26 16:30   ` [PATCHv3 0/9] object store: oid_object_info is the next contender Brandon Williams
  2 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-04-25 18:20 UTC (permalink / raw)
  To: sbeller; +Cc: bmwill, git, gitster, jonathantanmy, sandals

v3:
* fixed and extended the commit message of last commit
* fixed the last patch, as Jonathan Tan suggested, see interdiff:

    $ git diff remotes/origin/sb/oid-object-info (which is v2)
    diff --git c/sha1_file.c w/sha1_file.c
    index 94123e0299..dcd6b879ac 100644
    --- c/sha1_file.c
    +++ w/sha1_file.c
    @@ -1289,14 +1289,13 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
     
                    /* Check if it is a missing object */
                    if (fetch_if_missing && repository_format_partial_clone &&
    -                   !already_retried) {
    +                   !already_retried && r == the_repository) {
                            /*
                             * TODO Investigate having fetch_object() return
                             * TODO error/success and stopping the music here.
    -                        * TODO Pass a repository struct through fetch_object.
    +                        * TODO Pass a repository struct through fetch_object,
    +                        * such that arbitrary repositories work.
                             */
    -                       if (r != the_repository)
    -                               die(_("partial clones only supported in the_repository"));
                            fetch_object(repository_format_partial_clone, real->hash);
                            already_retried = 1;
                            continue;
    
Thanks,
Stefan

v2:

* fixed the sha1/oid typo
* removed spurious new line
* Brandon and Jonthan discovered another dependency that I missed due
  to cherrypicking that commit from a tree before partial clone was a thing.
  We error out when attempting to use fetch_object for repos that are not
  the_repository.

Thanks,
Stefan

v1:
This applies on top of origin/sb/object-store-replace and is available as
https://github.com/stefanbeller/git/tree/oid_object_info

This continues the work of sb/packfiles-in-repository,
extending the layer at which we have to pass in an explicit
repository object to oid_object_info.

A test merge to next shows only a minor merge conflicit (adding
different #include lines in one c file), so this might be a good next
step for the object store series.

Notes on further object store series:
I plan on converting the "parsed object store" next,
which would be {alloc, object, tree, commit, tag}.c as that is a prerequisite
for migrating shallow (which is intermingled with grafts) information to the
object store.

There is currently work going on in allocation (mempool - Jameson Miller)
and grafts (deprecate grafts - DScho), which is why I am sending this
series first. I think it can go in parallel to the "parsed object store"
that is coming next.

Thanks,
Stefan

Jonathan Nieder (1):
  packfile: add repository argument to packed_object_info

Stefan Beller (8):
  cache.h: add repository argument to oid_object_info_extended
  cache.h: add repository argument to oid_object_info
  packfile: add repository argument to retry_bad_packed_offset
  packfile: add repository argument to packed_to_object_type
  packfile: add repository argument to read_object
  packfile: add repository argument to unpack_entry
  packfile: add repository argument to cache_or_unpack_entry
  cache.h: allow oid_object_info to handle arbitrary repositories

 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       | 12 ++++++------
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   | 11 +++++++----
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  7 +++++--
 diff.c                   |  3 ++-
 fast-import.c            | 16 ++++++++++------
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 pack-check.c             |  3 ++-
 packfile.c               | 40 +++++++++++++++++++++++-----------------
 packfile.h               |  6 ++++--
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              | 37 +++++++++++++++++++++----------------
 sha1_name.c              | 12 ++++++------
 streaming.c              |  2 +-
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 35 files changed, 124 insertions(+), 93 deletions(-)

-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 5%]

* [PATCHv3 2/9] cache.h: add repository argument to oid_object_info
  2018-04-25 18:20 ` [PATCHv3 0/9] object store: oid_object_info is the next contender Stefan Beller
@ 2018-04-25 18:20   ` Stefan Beller
  2018-04-26 16:30   ` [PATCHv3 0/9] object store: oid_object_info is the next contender Brandon Williams
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-25 18:20 UTC (permalink / raw)
  To: sbeller; +Cc: bmwill, git, gitster, jonathantanmy, sandals

Add a repository argument to allow the callers of oid_object_info
to be more specific about which repository to handle. This is a small
mechanical change; it doesn't change the implementation to handle
repositories other than the_repository yet.

As with the previous commits, use a macro to catch callers passing a
repository other than the_repository at compile time.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 archive-tar.c            |  2 +-
 archive-zip.c            |  3 ++-
 blame.c                  |  4 ++--
 builtin/blame.c          |  2 +-
 builtin/cat-file.c       |  6 +++---
 builtin/describe.c       |  2 +-
 builtin/fast-export.c    |  2 +-
 builtin/fetch.c          |  2 +-
 builtin/fsck.c           |  3 ++-
 builtin/index-pack.c     |  4 ++--
 builtin/ls-tree.c        |  2 +-
 builtin/mktree.c         |  2 +-
 builtin/pack-objects.c   |  8 +++++---
 builtin/prune.c          |  3 ++-
 builtin/replace.c        | 11 ++++++-----
 builtin/tag.c            |  4 ++--
 builtin/unpack-objects.c |  2 +-
 cache.h                  |  3 ++-
 diff.c                   |  3 ++-
 fast-import.c            | 14 +++++++++-----
 list-objects-filter.c    |  2 +-
 object.c                 |  2 +-
 pack-bitmap-write.c      |  3 ++-
 packfile.c               |  2 +-
 reachable.c              |  2 +-
 refs.c                   |  2 +-
 remote.c                 |  2 +-
 sequencer.c              |  3 ++-
 sha1_file.c              |  4 ++--
 sha1_name.c              | 12 ++++++------
 submodule.c              |  2 +-
 tag.c                    |  2 +-
 32 files changed, 67 insertions(+), 53 deletions(-)

diff --git a/archive-tar.c b/archive-tar.c
index 3563bcb9f2..f93409324f 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -276,7 +276,7 @@ static int write_tar_entry(struct archiver_args *args,
 		memcpy(header.name, path, pathlen);
 
 	if (S_ISREG(mode) && !args->convert &&
-	    oid_object_info(oid, &size) == OBJ_BLOB &&
+	    oid_object_info(the_repository, oid, &size) == OBJ_BLOB &&
 	    size > big_file_threshold)
 		buffer = NULL;
 	else if (S_ISLNK(mode) || S_ISREG(mode)) {
diff --git a/archive-zip.c b/archive-zip.c
index 6b20bce4d1..74f3fe9103 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -325,7 +325,8 @@ static int write_zip_entry(struct archiver_args *args,
 		compressed_size = 0;
 		buffer = NULL;
 	} else if (S_ISREG(mode) || S_ISLNK(mode)) {
-		enum object_type type = oid_object_info(oid, &size);
+		enum object_type type = oid_object_info(the_repository, oid,
+							&size);
 
 		method = 0;
 		attr2 = S_ISLNK(mode) ? ((mode | 0777) << 16) :
diff --git a/blame.c b/blame.c
index 78c9808bd1..dfa24473dc 100644
--- a/blame.c
+++ b/blame.c
@@ -81,7 +81,7 @@ static void verify_working_tree_path(struct commit *work_tree, const char *path)
 		unsigned mode;
 
 		if (!get_tree_entry(commit_oid, path, &blob_oid, &mode) &&
-		    oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+		    oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 			return;
 	}
 
@@ -504,7 +504,7 @@ static int fill_blob_sha1_and_mode(struct blame_origin *origin)
 		return 0;
 	if (get_tree_entry(&origin->commit->object.oid, origin->path, &origin->blob_oid, &origin->mode))
 		goto error_out;
-	if (oid_object_info(&origin->blob_oid, NULL) != OBJ_BLOB)
+	if (oid_object_info(the_repository, &origin->blob_oid, NULL) != OBJ_BLOB)
 		goto error_out;
 	return 0;
  error_out:
diff --git a/builtin/blame.c b/builtin/blame.c
index db38c0b307..bfdf7cc132 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -655,7 +655,7 @@ static int is_a_rev(const char *name)
 
 	if (get_oid(name, &oid))
 		return 0;
-	return OBJ_NONE < oid_object_info(&oid, NULL);
+	return OBJ_NONE < oid_object_info(the_repository, &oid, NULL);
 }
 
 int cmd_blame(int argc, const char **argv, const char *prefix)
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 4ecdb9ff54..b8ecbea98e 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -116,7 +116,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 		/* else fallthrough */
 
 	case 'p':
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("Not a valid object name %s", obj_name);
 
@@ -140,7 +140,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 	case 0:
 		if (type_from_string(exp_type) == OBJ_BLOB) {
 			struct object_id blob_oid;
-			if (oid_object_info(&oid, NULL) == OBJ_TAG) {
+			if (oid_object_info(the_repository, &oid, NULL) == OBJ_TAG) {
 				char *buffer = read_object_file(&oid, &type,
 								&size);
 				const char *target;
@@ -151,7 +151,7 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name,
 			} else
 				oidcpy(&blob_oid, &oid);
 
-			if (oid_object_info(&blob_oid, NULL) == OBJ_BLOB)
+			if (oid_object_info(the_repository, &blob_oid, NULL) == OBJ_BLOB)
 				return stream_blob_to_fd(1, &blob_oid, NULL, 0);
 			/*
 			 * we attempted to dereference a tag to a blob
diff --git a/builtin/describe.c b/builtin/describe.c
index de840f96a4..66c497f789 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -502,7 +502,7 @@ static void describe(const char *arg, int last_one)
 
 	if (cmit)
 		describe_commit(&oid, &sb);
-	else if (oid_object_info(&oid, NULL) == OBJ_BLOB)
+	else if (oid_object_info(the_repository, &oid, NULL) == OBJ_BLOB)
 		describe_blob(oid, &sb);
 	else
 		die(_("%s is neither a commit nor blob"), arg);
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index a15898d641..373c794873 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -947,7 +947,7 @@ static void import_marks(char *input_file)
 		if (last_idnum < mark)
 			last_idnum = mark;
 
-		type = oid_object_info(&oid, NULL);
+		type = oid_object_info(the_repository, &oid, NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(&oid));
 
diff --git a/builtin/fetch.c b/builtin/fetch.c
index dcdfc66f09..73be393b2e 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -637,7 +637,7 @@ static int update_local_ref(struct ref *ref,
 	struct branch *current_branch = branch_get(NULL);
 	const char *pretty_ref = prettify_refname(ref->name);
 
-	type = oid_object_info(&ref->new_oid, NULL);
+	type = oid_object_info(the_repository, &ref->new_oid, NULL);
 	if (type < 0)
 		die(_("object %s not found"), oid_to_hex(&ref->new_oid));
 
diff --git a/builtin/fsck.c b/builtin/fsck.c
index 087360a675..9d59d7d5a2 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -67,7 +67,8 @@ static const char *printable_type(struct object *obj)
 	const char *ret;
 
 	if (obj->type == OBJ_NONE) {
-		enum object_type type = oid_object_info(&obj->oid, NULL);
+		enum object_type type = oid_object_info(the_repository,
+							&obj->oid, NULL);
 		if (type > 0)
 			object_as_type(obj, type, 0);
 	}
diff --git a/builtin/index-pack.c b/builtin/index-pack.c
index d81473e722..2d04a596f5 100644
--- a/builtin/index-pack.c
+++ b/builtin/index-pack.c
@@ -223,7 +223,7 @@ static unsigned check_object(struct object *obj)
 
 	if (!(obj->flags & FLAG_CHECKED)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type <= 0)
 			die(_("did not receive expected object %s"),
 			      oid_to_hex(&obj->oid));
@@ -812,7 +812,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,
 		enum object_type has_type;
 		unsigned long has_size;
 		read_lock();
-		has_type = oid_object_info(oid, &has_size);
+		has_type = oid_object_info(the_repository, oid, &has_size);
 		if (has_type < 0)
 			die(_("cannot read existing object info %s"), oid_to_hex(oid));
 		if (has_type != type || has_size != size)
diff --git a/builtin/ls-tree.c b/builtin/ls-tree.c
index d44b4f9c27..409da4e835 100644
--- a/builtin/ls-tree.c
+++ b/builtin/ls-tree.c
@@ -94,7 +94,7 @@ static int show_tree(const struct object_id *oid, struct strbuf *base,
 			char size_text[24];
 			if (!strcmp(type, blob_type)) {
 				unsigned long size;
-				if (oid_object_info(oid, &size) == OBJ_BAD)
+				if (oid_object_info(the_repository, oid, &size) == OBJ_BAD)
 					xsnprintf(size_text, sizeof(size_text),
 						  "BAD");
 				else
diff --git a/builtin/mktree.c b/builtin/mktree.c
index 263c530315..bb76b469fd 100644
--- a/builtin/mktree.c
+++ b/builtin/mktree.c
@@ -116,7 +116,7 @@ static void mktree_line(char *buf, size_t len, int nul_term_line, int allow_miss
 	}
 
 	/* Check the type of object identified by sha1 */
-	obj_type = oid_object_info(&oid, NULL);
+	obj_type = oid_object_info(the_repository, &oid, NULL);
 	if (obj_type < 0) {
 		if (allow_missing) {
 			; /* no problem - missing objects are presumed to be of the right type */
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 4bdae5a1d8..8d4111f748 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1516,7 +1516,8 @@ static void check_object(struct object_entry *entry)
 		unuse_pack(&w_curs);
 	}
 
-	entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+	entry->type = oid_object_info(the_repository, &entry->idx.oid,
+				      &entry->size);
 	/*
 	 * The error condition is checked in prepare_pack().  This is
 	 * to permit a missing preferred base object to be ignored
@@ -1578,7 +1579,8 @@ static void drop_reused_delta(struct object_entry *entry)
 		 * And if that fails, the error will be recorded in entry->type
 		 * and dealt with in prepare_pack().
 		 */
-		entry->type = oid_object_info(&entry->idx.oid, &entry->size);
+		entry->type = oid_object_info(the_repository, &entry->idx.oid,
+					      &entry->size);
 	}
 }
 
@@ -2706,7 +2708,7 @@ static void add_objects_in_unpacked_packs(struct rev_info *revs)
 static int add_loose_object(const struct object_id *oid, const char *path,
 			    void *data)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	if (type < 0) {
 		warning("loose object at %s could not be examined", path);
diff --git a/builtin/prune.c b/builtin/prune.c
index 38ced18dad..518ffbea13 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -50,7 +50,8 @@ static int prune_object(const struct object_id *oid, const char *fullpath,
 	if (st.st_mtime > expire)
 		return 0;
 	if (show_only || verbose) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		printf("%s %s\n", oid_to_hex(oid),
 		       (type > 0) ? type_name(type) : "unknown");
 	}
diff --git a/builtin/replace.c b/builtin/replace.c
index 237ea656cf..14e142d5a8 100644
--- a/builtin/replace.c
+++ b/builtin/replace.c
@@ -55,8 +55,9 @@ static int show_reference(const char *refname, const struct object_id *oid,
 			if (get_oid(refname, &object))
 				return error("Failed to resolve '%s' as a valid ref.", refname);
 
-			obj_type = oid_object_info(&object, NULL);
-			repl_type = oid_object_info(oid, NULL);
+			obj_type = oid_object_info(the_repository, &object,
+						   NULL);
+			repl_type = oid_object_info(the_repository, oid, NULL);
 
 			printf("%s (%s) -> %s (%s)\n", refname, type_name(obj_type),
 			       oid_to_hex(oid), type_name(repl_type));
@@ -164,8 +165,8 @@ static int replace_object_oid(const char *object_ref,
 	struct ref_transaction *transaction;
 	struct strbuf err = STRBUF_INIT;
 
-	obj_type = oid_object_info(object, NULL);
-	repl_type = oid_object_info(repl, NULL);
+	obj_type = oid_object_info(the_repository, object, NULL);
+	repl_type = oid_object_info(the_repository, repl, NULL);
 	if (!force && obj_type != repl_type)
 		die("Objects must be of the same type.\n"
 		    "'%s' points to a replaced object of type '%s'\n"
@@ -292,7 +293,7 @@ static int edit_and_replace(const char *object_ref, int force, int raw)
 	if (get_oid(object_ref, &old_oid) < 0)
 		die("Not a valid object name: '%s'", object_ref);
 
-	type = oid_object_info(&old_oid, NULL);
+	type = oid_object_info(the_repository, &old_oid, NULL);
 	if (type < 0)
 		die("unable to get object type for %s", oid_to_hex(&old_oid));
 
diff --git a/builtin/tag.c b/builtin/tag.c
index 8cff6d0b72..26d7729f57 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -212,7 +212,7 @@ static void create_tag(const struct object_id *object, const char *tag,
 	struct strbuf header = STRBUF_INIT;
 	char *path = NULL;
 
-	type = oid_object_info(object, NULL);
+	type = oid_object_info(the_repository, object, NULL);
 	if (type <= OBJ_NONE)
 	    die(_("bad object type."));
 
@@ -298,7 +298,7 @@ static void create_reflog_msg(const struct object_id *oid, struct strbuf *sb)
 	}
 
 	strbuf_addstr(sb, " (");
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	switch (type) {
 	default:
 		strbuf_addstr(sb, "object of unknown type");
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index b7755c6cc5..cfe9019f80 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -199,7 +199,7 @@ static int check_object(struct object *obj, int type, void *data, struct fsck_op
 
 	if (!(obj->flags & FLAG_OPEN)) {
 		unsigned long size;
-		int type = oid_object_info(&obj->oid, &size);
+		int type = oid_object_info(the_repository, &obj->oid, &size);
 		if (type != obj->type || type <= 0)
 			die("object of unexpected type");
 		obj->flags |= FLAG_WRITTEN;
diff --git a/cache.h b/cache.h
index 588c4fff9a..6340b2c572 100644
--- a/cache.h
+++ b/cache.h
@@ -1192,7 +1192,8 @@ static inline void *read_object_file(const struct object_id *oid, enum object_ty
 }
 
 /* Read and unpack an object file into memory, write memory to an object file */
-extern int oid_object_info(const struct object_id *, unsigned long *);
+#define oid_object_info(r, o, f) oid_object_info_##r(o, f)
+int oid_object_info_the_repository(const struct object_id *, unsigned long *);
 
 extern int hash_object_file(const void *buf, unsigned long len,
 			    const char *type, struct object_id *oid);
diff --git a/diff.c b/diff.c
index 1289df4b1f..4753170fe1 100644
--- a/diff.c
+++ b/diff.c
@@ -3638,7 +3638,8 @@ int diff_populate_filespec(struct diff_filespec *s, unsigned int flags)
 	else {
 		enum object_type type;
 		if (size_only || (flags & CHECK_BINARY)) {
-			type = oid_object_info(&s->oid, &s->size);
+			type = oid_object_info(the_repository, &s->oid,
+					       &s->size);
 			if (type < 0)
 				die("unable to read %s",
 				    oid_to_hex(&s->oid));
diff --git a/fast-import.c b/fast-import.c
index 99f8f56e8c..afe06bd7c1 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -1917,7 +1917,8 @@ static void read_marks(void)
 			die("corrupt mark line: %s", line);
 		e = find_object(&oid);
 		if (!e) {
-			enum object_type type = oid_object_info(&oid, NULL);
+			enum object_type type = oid_object_info(the_repository,
+								&oid, NULL);
 			if (type < 0)
 				die("object not found: %s", oid_to_hex(&oid));
 			e = insert_object(&oid);
@@ -2447,7 +2448,8 @@ static void file_change_m(const char *p, struct branch *b)
 		enum object_type expected = S_ISDIR(mode) ?
 						OBJ_TREE: OBJ_BLOB;
 		enum object_type type = oe ? oe->type :
-					oid_object_info(&oid, NULL);
+					oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("%s not found: %s",
 					S_ISDIR(mode) ?  "Tree" : "Blob",
@@ -2608,7 +2610,8 @@ static void note_change_n(const char *p, struct branch *b, unsigned char *old_fa
 			die("Not a blob (actually a %s): %s",
 				type_name(oe->type), command_buf.buf);
 	} else if (!is_null_oid(&oid)) {
-		enum object_type type = oid_object_info(&oid, NULL);
+		enum object_type type = oid_object_info(the_repository, &oid,
+							NULL);
 		if (type < 0)
 			die("Blob not found: %s", command_buf.buf);
 		if (type != OBJ_BLOB)
@@ -2895,7 +2898,7 @@ static void parse_new_tag(const char *arg)
 	} else if (!get_oid(from, &oid)) {
 		struct object_entry *oe = find_object(&oid);
 		if (!oe) {
-			type = oid_object_info(&oid, NULL);
+			type = oid_object_info(the_repository, &oid, NULL);
 			if (type < 0)
 				die("Not a valid object: %s", from);
 		} else
@@ -3053,7 +3056,8 @@ static struct object_entry *dereference(struct object_entry *oe,
 	unsigned long size;
 	char *buf = NULL;
 	if (!oe) {
-		enum object_type type = oid_object_info(oid, NULL);
+		enum object_type type = oid_object_info(the_repository, oid,
+							NULL);
 		if (type < 0)
 			die("object not found: %s", oid_to_hex(oid));
 		/* cache it! */
diff --git a/list-objects-filter.c b/list-objects-filter.c
index 0ec83aaf18..ea94fe8af2 100644
--- a/list-objects-filter.c
+++ b/list-objects-filter.c
@@ -117,7 +117,7 @@ static enum list_objects_filter_result filter_blobs_limit(
 		assert(obj->type == OBJ_BLOB);
 		assert((obj->flags & SEEN) == 0);
 
-		t = oid_object_info(&obj->oid, &object_length);
+		t = oid_object_info(the_repository, &obj->oid, &object_length);
 		if (t != OBJ_BLOB) { /* probably OBJ_NONE */
 			/*
 			 * We DO NOT have the blob locally, so we cannot
diff --git a/object.c b/object.c
index 66cffaf6e5..5044d08e96 100644
--- a/object.c
+++ b/object.c
@@ -257,7 +257,7 @@ struct object *parse_object(const struct object_id *oid)
 
 	if ((obj && obj->type == OBJ_BLOB && has_object_file(oid)) ||
 	    (!obj && has_object_file(oid) &&
-	     oid_object_info(oid, NULL) == OBJ_BLOB)) {
+	     oid_object_info(the_repository, oid, NULL) == OBJ_BLOB)) {
 		if (check_object_signature(repl, NULL, 0, NULL) < 0) {
 			error("sha1 mismatch %s", oid_to_hex(oid));
 			return NULL;
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index 41ae27fb19..cd1903e717 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -73,7 +73,8 @@ void bitmap_writer_build_type_index(struct pack_idx_entry **index,
 			break;
 
 		default:
-			real_type = oid_object_info(&entry->idx.oid, NULL);
+			real_type = oid_object_info(the_repository,
+						    &entry->idx.oid, NULL);
 			break;
 		}
 
diff --git a/packfile.c b/packfile.c
index d9914ba723..80c7fa734f 100644
--- a/packfile.c
+++ b/packfile.c
@@ -1114,7 +1114,7 @@ static int retry_bad_packed_offset(struct packed_git *p, off_t obj_offset)
 		return OBJ_BAD;
 	nth_packed_object_oid(&oid, p, revidx->nr);
 	mark_bad_packed_object(p, oid.hash);
-	type = oid_object_info(&oid, NULL);
+	type = oid_object_info(the_repository, &oid, NULL);
 	if (type <= OBJ_NONE)
 		return OBJ_BAD;
 	return type;
diff --git a/reachable.c b/reachable.c
index a6ea33a5db..ffb976c33c 100644
--- a/reachable.c
+++ b/reachable.c
@@ -78,7 +78,7 @@ static void add_recent_object(const struct object_id *oid,
 	 * later processing, and the revision machinery expects
 	 * commits and tags to have been parsed.
 	 */
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("unable to get object info for %s", oid_to_hex(oid));
 
diff --git a/refs.c b/refs.c
index 9b56fa9b81..27c88ba768 100644
--- a/refs.c
+++ b/refs.c
@@ -302,7 +302,7 @@ enum peel_status peel_object(const struct object_id *name, struct object_id *oid
 	struct object *o = lookup_unknown_object(name->hash);
 
 	if (o->type == OBJ_NONE) {
-		int type = oid_object_info(name, NULL);
+		int type = oid_object_info(the_repository, name, NULL);
 		if (type < 0 || !object_as_type(o, type, 0))
 			return PEEL_INVALID;
 	}
diff --git a/remote.c b/remote.c
index 91eb010ca9..481bf933f3 100644
--- a/remote.c
+++ b/remote.c
@@ -1376,7 +1376,7 @@ static void add_missing_tags(struct ref *src, struct ref **dst, struct ref ***ds
 			continue; /* not a tag */
 		if (string_list_has_string(&dst_tag, ref->name))
 			continue; /* they already have it */
-		if (oid_object_info(&ref->new_oid, NULL) != OBJ_TAG)
+		if (oid_object_info(the_repository, &ref->new_oid, NULL) != OBJ_TAG)
 			continue; /* be conservative */
 		item = string_list_append(&src_tag, ref->name);
 		item->util = ref;
diff --git a/sequencer.c b/sequencer.c
index 667f35ebdf..44f0518b9c 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -2876,7 +2876,8 @@ int sequencer_pick_revisions(struct replay_opts *opts)
 
 		if (!get_oid(name, &oid)) {
 			if (!lookup_commit_reference_gently(&oid, 1)) {
-				enum object_type type = oid_object_info(&oid,
+				enum object_type type = oid_object_info(the_repository,
+									&oid,
 									NULL);
 				return error(_("%s: can't cherry-pick a %s"),
 					name, type_name(type));
diff --git a/sha1_file.c b/sha1_file.c
index 50a2dc5f0a..93f25c6c6a 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1322,7 +1322,7 @@ int oid_object_info_extended_the_repository(const struct object_id *oid, struct
 }
 
 /* returns enum object_type or negative */
-int oid_object_info(const struct object_id *oid, unsigned long *sizep)
+int oid_object_info_the_repository(const struct object_id *oid, unsigned long *sizep)
 {
 	enum object_type type;
 	struct object_info oi = OBJECT_INFO_INIT;
@@ -1988,7 +1988,7 @@ int read_pack_header(int fd, struct pack_header *header)
 
 void assert_oid_type(const struct object_id *oid, enum object_type expect)
 {
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 	if (type < 0)
 		die("%s is not a valid object", oid_to_hex(oid));
 	if (type != expect)
diff --git a/sha1_name.c b/sha1_name.c
index 5b93bf8da3..b5406b6eb2 100644
--- a/sha1_name.c
+++ b/sha1_name.c
@@ -223,7 +223,7 @@ static int finish_object_disambiguation(struct disambiguate_state *ds,
 
 static int disambiguate_commit_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_COMMIT;
 }
 
@@ -232,7 +232,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -247,7 +247,7 @@ static int disambiguate_committish_only(const struct object_id *oid, void *cb_da
 
 static int disambiguate_tree_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_TREE;
 }
 
@@ -256,7 +256,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 	struct object *obj;
 	int kind;
 
-	kind = oid_object_info(oid, NULL);
+	kind = oid_object_info(the_repository, oid, NULL);
 	if (kind == OBJ_TREE || kind == OBJ_COMMIT)
 		return 1;
 	if (kind != OBJ_TAG)
@@ -271,7 +271,7 @@ static int disambiguate_treeish_only(const struct object_id *oid, void *cb_data_
 
 static int disambiguate_blob_only(const struct object_id *oid, void *cb_data_unused)
 {
-	int kind = oid_object_info(oid, NULL);
+	int kind = oid_object_info(the_repository, oid, NULL);
 	return kind == OBJ_BLOB;
 }
 
@@ -350,7 +350,7 @@ static int show_ambiguous_object(const struct object_id *oid, void *data)
 	if (ds->fn && !ds->fn(oid, ds->cb_data))
 		return 0;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type == OBJ_COMMIT) {
 		struct commit *commit = lookup_commit(oid);
 		if (commit) {
diff --git a/submodule.c b/submodule.c
index 9a50168b23..bb133e9b93 100644
--- a/submodule.c
+++ b/submodule.c
@@ -818,7 +818,7 @@ static int check_has_commit(const struct object_id *oid, void *data)
 {
 	struct has_commit_data *cb = data;
 
-	enum object_type type = oid_object_info(oid, NULL);
+	enum object_type type = oid_object_info(the_repository, oid, NULL);
 
 	switch (type) {
 	case OBJ_COMMIT:
diff --git a/tag.c b/tag.c
index 86b1dcbb82..3d37c1bd25 100644
--- a/tag.c
+++ b/tag.c
@@ -41,7 +41,7 @@ int gpg_verify_tag(const struct object_id *oid, const char *name_to_report,
 	unsigned long size;
 	int ret;
 
-	type = oid_object_info(oid, NULL);
+	type = oid_object_info(the_repository, oid, NULL);
 	if (type != OBJ_TAG)
 		return error("%s: cannot verify a non-tag object of type %s.",
 				name_to_report ?
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 3%]

* Re: [PATCHv3 0/9] object store: oid_object_info is the next contender
  2018-04-25 18:20 ` [PATCHv3 0/9] object store: oid_object_info is the next contender Stefan Beller
  2018-04-25 18:20   ` [PATCHv3 2/9] cache.h: add repository argument to oid_object_info Stefan Beller
@ 2018-04-26 16:30   ` Brandon Williams
  1 sibling, 0 replies; 200+ results
From: Brandon Williams @ 2018-04-26 16:30 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, gitster, jonathantanmy, sandals

On 04/25, Stefan Beller wrote:
> v3:
> * fixed and extended the commit message of last commit
> * fixed the last patch, as Jonathan Tan suggested, see interdiff:
> 
>     $ git diff remotes/origin/sb/oid-object-info (which is v2)
>     diff --git c/sha1_file.c w/sha1_file.c
>     index 94123e0299..dcd6b879ac 100644
>     --- c/sha1_file.c
>     +++ w/sha1_file.c
>     @@ -1289,14 +1289,13 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid,
>      
>                     /* Check if it is a missing object */
>                     if (fetch_if_missing && repository_format_partial_clone &&
>     -                   !already_retried) {
>     +                   !already_retried && r == the_repository) {
>                             /*
>                              * TODO Investigate having fetch_object() return
>                              * TODO error/success and stopping the music here.
>     -                        * TODO Pass a repository struct through fetch_object.
>     +                        * TODO Pass a repository struct through fetch_object,
>     +                        * such that arbitrary repositories work.
>                              */
>     -                       if (r != the_repository)
>     -                               die(_("partial clones only supported in the_repository"));
>                             fetch_object(repository_format_partial_clone, real->hash);
>                             already_retried = 1;
>                             continue;
>     
> Thanks,
> Stefan

v3 looks good, thanks for taking care of this.

> 
> v2:
> 
> * fixed the sha1/oid typo
> * removed spurious new line
> * Brandon and Jonthan discovered another dependency that I missed due
>   to cherrypicking that commit from a tree before partial clone was a thing.
>   We error out when attempting to use fetch_object for repos that are not
>   the_repository.
> 
> Thanks,
> Stefan
> 
> v1:
> This applies on top of origin/sb/object-store-replace and is available as
> https://github.com/stefanbeller/git/tree/oid_object_info
> 
> This continues the work of sb/packfiles-in-repository,
> extending the layer at which we have to pass in an explicit
> repository object to oid_object_info.
> 
> A test merge to next shows only a minor merge conflicit (adding
> different #include lines in one c file), so this might be a good next
> step for the object store series.
> 
> Notes on further object store series:
> I plan on converting the "parsed object store" next,
> which would be {alloc, object, tree, commit, tag}.c as that is a prerequisite
> for migrating shallow (which is intermingled with grafts) information to the
> object store.
> 
> There is currently work going on in allocation (mempool - Jameson Miller)
> and grafts (deprecate grafts - DScho), which is why I am sending this
> series first. I think it can go in parallel to the "parsed object store"
> that is coming next.
> 
> Thanks,
> Stefan
> 
> Jonathan Nieder (1):
>   packfile: add repository argument to packed_object_info
> 
> Stefan Beller (8):
>   cache.h: add repository argument to oid_object_info_extended
>   cache.h: add repository argument to oid_object_info
>   packfile: add repository argument to retry_bad_packed_offset
>   packfile: add repository argument to packed_to_object_type
>   packfile: add repository argument to read_object
>   packfile: add repository argument to unpack_entry
>   packfile: add repository argument to cache_or_unpack_entry
>   cache.h: allow oid_object_info to handle arbitrary repositories
> 
>  archive-tar.c            |  2 +-
>  archive-zip.c            |  3 ++-
>  blame.c                  |  4 ++--
>  builtin/blame.c          |  2 +-
>  builtin/cat-file.c       | 12 ++++++------
>  builtin/describe.c       |  2 +-
>  builtin/fast-export.c    |  2 +-
>  builtin/fetch.c          |  2 +-
>  builtin/fsck.c           |  3 ++-
>  builtin/index-pack.c     |  4 ++--
>  builtin/ls-tree.c        |  2 +-
>  builtin/mktree.c         |  2 +-
>  builtin/pack-objects.c   | 11 +++++++----
>  builtin/prune.c          |  3 ++-
>  builtin/replace.c        | 11 ++++++-----
>  builtin/tag.c            |  4 ++--
>  builtin/unpack-objects.c |  2 +-
>  cache.h                  |  7 +++++--
>  diff.c                   |  3 ++-
>  fast-import.c            | 16 ++++++++++------
>  list-objects-filter.c    |  2 +-
>  object.c                 |  2 +-
>  pack-bitmap-write.c      |  3 ++-
>  pack-check.c             |  3 ++-
>  packfile.c               | 40 +++++++++++++++++++++++-----------------
>  packfile.h               |  6 ++++--
>  reachable.c              |  2 +-
>  refs.c                   |  2 +-
>  remote.c                 |  2 +-
>  sequencer.c              |  3 ++-
>  sha1_file.c              | 37 +++++++++++++++++++++----------------
>  sha1_name.c              | 12 ++++++------
>  streaming.c              |  2 +-
>  submodule.c              |  2 +-
>  tag.c                    |  2 +-
>  35 files changed, 124 insertions(+), 93 deletions(-)
> 
> -- 
> 2.17.0.441.gb46fe60e1d-goog
> 

-- 
Brandon Williams

^ permalink raw reply	[relevance 2%]

* Re: git merge banch w/ different submodule revision
      [irrelevant] <1524739599.20251.17.camel@klsmartin.com>
@ 2018-04-26 17:56 ` Stefan Beller
  2018-04-26 21:46   ` Jacob Keller
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-26 17:56 UTC (permalink / raw)
  To: Middelschulte, Leif; +Cc: git

On Thu, Apr 26, 2018 at 3:49 AM, Middelschulte, Leif
<Leif.Middelschulte@klsmartin.com> wrote:
> Hi,
>
> we're using git-flow as a basic development workflow. However, doing so revealed unexpected merge-behavior by git.
>
> Assume the following setup:
>
> - Repository `S` is sourced by repository `p` as submodule `s`
> - Repository `p` has two branches: `feature_x` and `develop`
> - The revisions sourced via the submodule have a linear history
>
>
> * 1c1d38f (feature_x) update submodule revision to b17e9d9
> | * 3290e69 (HEAD -> develop) update submodule revision to 0598394
> |/
> * cd5e1a5 initial submodule revision
>
>
> Problem case: Merge either branch into the other
>
> Expected behavior: Merge conflict.
>
> Actual behavior: Auto merge without conflicts.
>
> Note 1: A merge conflict does occur, if the sourced revisions do *not* have a linear history
>
> Did I get something wrong about how git resolves merges?

We often treating a submodule as a file from the superproject, but not always.
And in case of a merge, git seems to be a bit smarter than treating it
as a textfile
with two different lines.

See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
(68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
to explain the situation you encounter. (specifically merge_submodule
at the end of the diff)

> Shouldn't git be like: "hey, you're trying to merge two different contents for the same line" (the submodule's revision)

As we have a history in the submodule we can do more than that and
resolve the conflict.

For two lines, you usually need manual intervention (which line to
pick, or craft a complete
new line out of parts of each line?), whereas for submodule commits
you can reason
about their dependencies due to their history and not just look at the
textual conflict.

Stefan

^ permalink raw reply	[relevance 8%]

* Re: git merge banch w/ different submodule revision
  2018-04-26 17:56 ` git merge banch w/ different submodule revision Stefan Beller
@ 2018-04-26 21:46   ` Jacob Keller
  2018-04-26 22:19     ` Stefan Beller
  2018-04-27  0:02     ` Elijah Newren
  0 siblings, 2 replies; 200+ results
From: Jacob Keller @ 2018-04-26 21:46 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Middelschulte, Leif, git

On Thu, Apr 26, 2018 at 10:56 AM, Stefan Beller <sbeller@google.com> wrote:
> We often treating a submodule as a file from the superproject, but not always.
> And in case of a merge, git seems to be a bit smarter than treating it
> as a textfile
> with two different lines.

Sure, but a submodule is checked out "at a commit", so if two branches
of history are merged, and they conflict over which place the
submodule is at.... shouldn't that produce a conflict??

I mean, how is the merge algorithm supposed to know which is right?
The patch you linked appears to be able to resolve it to the one which
contains both commits.. but that may not actually be true since you
can rewind submodules since they're *pointers* to commits, not commits
themselves.

I'm not against that as a possible strategy to merge submodules, but
it seems like not necessarily something you would always want...

Thanks,
Jake

^ permalink raw reply	[relevance 8%]

* Re: git merge banch w/ different submodule revision
  2018-04-26 21:46   ` Jacob Keller
@ 2018-04-26 22:19     ` Stefan Beller
  2018-04-30 17:02       ` Heiko Voigt
  2018-04-27  0:02     ` Elijah Newren
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-26 22:19 UTC (permalink / raw)
  To: Jacob Keller, Heiko Voigt; +Cc: Middelschulte, Leif, git

Stefan wrote:
> See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
> (68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
> to explain the situation you encounter. (specifically merge_submodule
> at the end of the diff)

+cc Heiko, author of that commit.

On Thu, Apr 26, 2018 at 2:46 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> On Thu, Apr 26, 2018 at 10:56 AM, Stefan Beller <sbeller@google.com> wrote:
>> We often treating a submodule as a file from the superproject, but not always.
>> And in case of a merge, git seems to be a bit smarter than treating it
>> as a textfile with two different lines.
>
> Sure, but a submodule is checked out "at a commit", so if two branches
> of history are merged, and they conflict over which place the
> submodule is at.... shouldn't that produce a conflict??

Stepping back a little bit:

When two branches developed a file differently, they can be merged
iff they do not change the same lines (plus a little bit of margin of 1
extra line)

That is the builtin merge-driver for "plain text files" and seems to be accepted
widely as "good enough" or "that is how git merges".

What if this text file happens to be the .gitmodules file and the changed lines
happen to be 2 options in there (Say one option was the path, as one branch
renamed the submodule, and the other option is submodule.branch) ?

Then we could do better as we know the structure of the file. We would not
need the extra buffer line as a cautious step, but instead could parse both
sides of the merge and merge each config in-memory and then write out
a .gitmodules file. I think David Turner proposed a custom merge driver
for .gitmodules a couple month ago.

Another example is the merge code respecting renames on one side
(even for directories) and edits in the other side. Technically the rename
of a file is a "delete of all lines in this path", which could also argued to
just conflict with the edit on the other side.

With these examples given, I think it is legit to treat submodule changes
not as "two lines of text differ at the same place, mark it as conflict",
but we are allowed to be smarter about it.

> I mean, how is the merge algorithm supposed to know which is right?

Good question. As said above, the merge algorithm for text files is just
correct for "plain text files". In source code, I can give an example
which merges fine, but doesn't compile after merging: One side changes
a function signature and the other side adds a call to the function (still using
the old signature).

Here you can see that our merge algorithm is wrong. It sucks.
The solution is a custom merge driver for C code (or whatever
language you happen to use).

For submodules, the given commit made the assumption that
progressing in history of a submodule is never bad, i.e. there are
no reverts and no bugs introduced, only perfect features are added
by new submodule commits. (I don't know which assumptions were
actually made, I made this up).

Maybe we need to revisit that decision?

> The patch you linked appears to be able to resolve it to the one which
> contains both commits.. but that may not actually be true since you
> can rewind submodules since they're *pointers* to commits, not commits
> themselves.

Right, and that is the problem, as the pointer is a small thing, which
doesn't allow for the dumb text merging strategy that is used in files.

So we could always err out and have the user make a decision.
Or we could provide a basic merge driver for submodules (which
was implemented in that commit).

If you use a different workflow this doesn't work for you, so
obviously you want a different custom merge driver for
submodules?

> I'm not against that as a possible strategy to merge submodules, but
> it seems like not necessarily something you would always want...

I agree that it is reasonable to want different things, just like
wanting a merge driver that works better with C code.
(side note: I am rebasing a large series currently and one of the
frequent conflicts were different #includes at the top of a file.
You could totally automate merging that :/)

Thanks,
Stefan

^ permalink raw reply	[relevance 6%]

* Re: git merge banch w/ different submodule revision
  2018-04-26 21:46   ` Jacob Keller
  2018-04-26 22:19     ` Stefan Beller
@ 2018-04-27  0:02     ` Elijah Newren
  1 sibling, 0 replies; 200+ results
From: Elijah Newren @ 2018-04-27  0:02 UTC (permalink / raw)
  To: Jacob Keller; +Cc: Stefan Beller, Middelschulte, Leif, git

On Thu, Apr 26, 2018 at 2:46 PM, Jacob Keller <jacob.keller@gmail.com> wrote:
> On Thu, Apr 26, 2018 at 10:56 AM, Stefan Beller <sbeller@google.com> wrote:
>> We often treating a submodule as a file from the superproject, but not always.
>> And in case of a merge, git seems to be a bit smarter than treating it
>> as a textfile
>> with two different lines.
>
> Sure, but a submodule is checked out "at a commit", so if two branches
> of history are merged, and they conflict over which place the
> submodule is at.... shouldn't that produce a conflict??

By "which place a submodule is at", do you mean the commit it points
to, or the path at which the submodule is found within the parent
repository?  Continuing on it sounds like you meant the former, but I
was unsure if you were asking mutliple different questions here.

> I mean, how is the merge algorithm supposed to know which is right?
> The patch you linked appears to be able to resolve it to the one which
> contains both commits.. but that may not actually be true since you
> can rewind submodules since they're *pointers* to commits, not commits
> themselves.

Only if both commits also contain the base; see lines 328 to 332 of
that patch.  So, if the submodules are rewound, that algorithm would
leave them as conflicted.

^ permalink raw reply	[relevance 5%]

* Re: [RFC 00/10] Make .the gitmodules file path configurable
      [irrelevant] ` <20180423174709.GA25128@aiede.svl.corp.google.com>
@ 2018-04-30 12:51   ` Antonio Ospite
  0 siblings, 0 replies; 200+ results
From: Antonio Ospite @ 2018-04-30 12:51 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: git, Richard Hartmann, Stefan Beller, Brandon Williams

On Mon, 23 Apr 2018 10:47:09 -0700
Jonathan Nieder <jrnieder@gmail.com> wrote:

> Hi,
>

Hi Jonathan, thank you for your comment.

> Antonio Ospite wrote:
> 
> > vcsh[1] uses bare git repositories and detached work-trees to manage
> > *distinct* sets of configuration files directly into $HOME.
> 
> Cool!  I like the tooling you're creating for this, though keep in mind
> that Git has some weaknesses as a tool for deployment.
>

I am not the author BTW, just a user trying to address the remaining
corner cases.

> In particular, keep in mind that when git updates a file, there is a
> period of time while it is missing from the filesystem, which can be
> problematic for dotfiles.
>

Thanks for the reminder, it may be worth mentioning this in vcsh
documentation, however I don't have knowledge of users experiencing
problems related to that.

> [...]
> > However when multiple repositories take turns using the same directory
> > as their work-tree, and more than one of them want to use submodules,
> > there could still be conflicts about the '.gitmodules' file because git
> > hardcodes this path.
> >
> > For comparison, in case of '.gitignore' a similar conflict might arise,
> > but git has alternative ways to specify exclude files, so vcsh solves
> > this by setting core.excludesFile for each repository and track ignored
> > files somewhere else (in ~/.gitignore.d/$VCSH_REPO_NAME).
> 
> For reference:
> 
> 	core.excludesFile
> 		Specifies the pathname to the file that contains
> 		patterns to describe paths that are not meant to be
> 		tracked, in addition to .gitignore (per-directory) and
> 		.git/info/exclude. Defaults to
> 		$XDG_CONFIG_HOME/git/ignore. If $XDG_CONFIG_HOME is
> 		either not set or empty, $HOME/.config/git/ignore is
> 		used instead. See gitignore(5).
> 
> Using this as a substitute for <worktree>/.gitignore is a bit of a
> hack.  It happens to work, though, so reading on. :)
> 
> [...]
> > So this series proposes a mechanism to set an alternative path for the
> > submodules configuration file (from now on "gitmodules file").
> 
> I am nervous about this.  I wonder if there is another way to
> accomplish the goal.
>
> One possibility would be to handle the case where .gitmodules is
> excluded by a sparse checkout specification and use .gitmodules from
> the index in that case.  Would that work for you?
> 

Since part of the problem is that .gitmodules *collide* between
repositories, a sparse-checkout approach make sense indeed.

As discussed[1] with Stefan Beller having git use .gitmodules from the
index without the need to have it checked out should work for us.

[1] https://www.spinics.net/lists/git/msg329153.html

Ideally git should also be able to write to that file when it's not
checked out (e.g. when running "git submodule add"), to save the
user from tedious sparse/unsparse rounds when operating with submodules.

As suggested by Stefan I'll first try to remove the hardcoded references
to .gitmodules in git-submodule.sh adding a helper sub-command to
access .gitmodules in a more robust way, and after that git could
be taught to use the file from the index, but this second part
is currently beyond my current git knowledge.

If this mechanism of using unchecked-out files from the index could be
extended to .gitignore (and .gitattributes), then vcsh might even stop
abusing core.excludesFile; sparse checkouts seem the more natural git
way to deal with colliding files in a shared-workdir scenario.

However, having users *write* to .gitignore and .gitattributes while
they are not checked out still sounds quite problematic to me, but maybe
this could be handled by vcsh itself, similarly to what is done for the
file pointed by core.excludesFile.

Ciao,
   Antonio

-- 
Antonio Ospite
https://ao2.it
https://twitter.com/ao2it

A: Because it messes up the order in which people normally read text.
   See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

^ permalink raw reply	[relevance 5%]

* Re: git merge banch w/ different submodule revision
  2018-04-26 22:19     ` Stefan Beller
@ 2018-04-30 17:02       ` Heiko Voigt
  2018-05-02  7:30         ` Middelschulte, Leif
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2018-04-30 17:02 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Jacob Keller, Middelschulte, Leif, git

On Thu, Apr 26, 2018 at 03:19:36PM -0700, Stefan Beller wrote:
> Stefan wrote:
> > See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
> > (68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
> > to explain the situation you encounter. (specifically merge_submodule
> > at the end of the diff)
> 
> +cc Heiko, author of that commit.

In that commit we tried to be very careful about. I do not understand
the situation in which the current strategy would be wrong by default.

We only merge if the following applies:

 * The changes in the superproject on both sides point forward in the
   submodule.

 * One side is contained in the other. Contained from the submodule
   perspective. Sides from the superproject merge perspective.

So in case of the mentioned rewind of a submodule: Only one side of the
3-way merge would point forward and the merge would fail.

I can imagine, that in case of a temporary revert of a commit in the
submodule that you would not want that merged into some other branch.
But that would be the same without submodules. If you merge a temporary
revert from another branch you will not get any conflict.

So maybe someone can explain the use case in which one would get the
results that seem wrong?

Cheers Heiko

^ permalink raw reply	[relevance 8%]

* Re: git-submodule is missing --dissociate option
      [irrelevant] <CAEp-SHXo2fnyUSMDqJnfOkh_R21R2FjFUtQ14u9s6-tV039tHg@mail.gmail.com>
@ 2018-04-30 18:19 ` Stefan Beller
  2018-04-30 21:39   ` Casey Fitzpatrick
  2018-05-02  4:34   ` git-submodule is missing --dissociate option Junio C Hamano
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-04-30 18:19 UTC (permalink / raw)
  To: Casey Fitzpatrick; +Cc: git

On Mon, Apr 30, 2018 at 1:29 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
> This seems to be a hole in the git feature set. I believe it is fairly
> easily worked around, but it would be best to provide the option for
> ease of use (and maybe performance?).
>
> git clone has both a --reference feature and a --dissociate option,
> with dissociate allowing for a reference to *only* speed up network
> transfers rather than have the resulting clone rely upon the reference
> always being there (creates an independent repo).

With the advent of git-worktree, I claim that --reference without further
--dissociate is dangerous, such that the combination of these two are
not the best UX, you could wish for.

> But git submodule only allows for --reference, so there isn't a an
> option to make a speedy independent submodule clone in one shot:
> https://git-scm.com/docs/git-submodule
> I checked the latest online documentation (currently at 2.16.3) and
> the documentation in the latest sources (almost 2.18):
> https://github.com/git/git/blob/next/Documentation/git-submodule.txt
>
> As far as I am aware this can be worked around with 'git repack -a'
> and manual removal of the objects/info/alternates file afterward.
> Though I don't know if this results in a less speedy clone than
> dissociate would.

That is an interesting workaround!
I agree we should have the --dissociate option available for submodules.
Care to implement it?

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 0/4] subtree: move out of contrib
      [irrelevant] <20180430095044.28492-1-avarab@gmail.com>
@ 2018-04-30 20:45 ` Avery Pennarun
  2018-04-30 21:38   ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Avery Pennarun @ 2018-04-30 20:45 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason
  Cc: Git Mailing List, Junio C Hamano, Jeff King, Stephen R Guglielmo,
	A . Wilcox, David Aguilar

On Mon, Apr 30, 2018 at 5:50 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> I think at this point git-subtree is widely used enough to move out of
> contrib/, maybe others disagree, but patches are always better for
> discussion that patch-less ML posts.

I really was hoping git-subtree would be completely obsoleted by
git-submodule by now, but... it hasn't been, so... no objections on
this end.

The gerrit team (eg. Stefan Beller) has been doing some really great
stuff to make submodules more usable by helping with relative
submodule links and by auto-updating links in supermodules at the
right times.  Unfortunately doing that requires help from the server
side, which kind of messes up decentralization and so doesn't solve
the problem in the general case.

I really wish there were a good answer, but I don't know what it is.
I do know that lots of people seem to at least be happy using
git-subtree, and would be even happier if it were installed
automatically with git.

Have fun,

Avery

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/4] subtree: move out of contrib
  2018-04-30 20:45 ` [PATCH 0/4] subtree: move out of contrib Avery Pennarun
@ 2018-04-30 21:38   ` Stefan Beller
  2018-04-30 21:53     ` Avery Pennarun
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-04-30 21:38 UTC (permalink / raw)
  To: Avery Pennarun
  Cc: Ævar Arnfjörð Bjarmason, Git Mailing List,
	Junio C Hamano, Jeff King, Stephen R Guglielmo, A . Wilcox,
	David Aguilar

On Mon, Apr 30, 2018 at 1:45 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> On Mon, Apr 30, 2018 at 5:50 AM, Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> I think at this point git-subtree is widely used enough to move out of
>> contrib/, maybe others disagree, but patches are always better for
>> discussion that patch-less ML posts.
>
> I really was hoping git-subtree would be completely obsoleted by
> git-submodule by now, but... it hasn't been, so... no objections on
> this end.

No objections from me either.

Submodules seem to serve a slightly different purpose, though?

With Subtrees the superproject always contains all the code,
even when you squash the subtree histroy when merging it in.
In the submodule world, you may not have access to one of the
submodules.

Submodules do not need to produce a synthetic project history
when splitting off again, as the history is genuine. This allows
for easier work with upstream.

Subtrees present you the whole history by default and the user
needs to be explicit about not wanting to see history from the
subtree, which is the opposite of submodules (though this
may be planned in the future to switch).

> The gerrit team (eg. Stefan Beller) has been doing some really great
> stuff to make submodules more usable by helping with relative
> submodule links and by auto-updating links in supermodules at the
> right times.  Unfortunately doing that requires help from the server
> side, which kind of messes up decentralization and so doesn't solve
> the problem in the general case.

Conceptually Gerrit is doing

  while true:
    git submodule update --remote
    if worktree is dirty:
        git commit "update the submodules"

just that Gerrit doesn't poll but does it event based.

> I really wish there were a good answer, but I don't know what it is.
> I do know that lots of people seem to at least be happy using
> git-subtree, and would be even happier if it were installed
> automatically with git.

https://trends.google.com/trends/explore?date=all&q=git%20subtree,git%20submodule

Not sure what to make of this data.

^ permalink raw reply	[relevance 6%]

* Re: git-submodule is missing --dissociate option
  2018-04-30 18:19 ` git-submodule is missing --dissociate option Stefan Beller
@ 2018-04-30 21:39   ` Casey Fitzpatrick
      [irrelevant]     ` <20180501180908.17443-1-kcghost@gmail.com>
                       ` (2 more replies)
  2018-05-02  4:34   ` git-submodule is missing --dissociate option Junio C Hamano
  1 sibling, 3 replies; 200+ results
From: Casey Fitzpatrick @ 2018-04-30 21:39 UTC (permalink / raw)
  To: Stefan Beller, avarab; +Cc: git

Sure, I'll take a crack at it and submit a patch.

On Mon, Apr 30, 2018 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:
> On Mon, Apr 30, 2018 at 1:29 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
>> This seems to be a hole in the git feature set. I believe it is fairly
>> easily worked around, but it would be best to provide the option for
>> ease of use (and maybe performance?).
>>
>> git clone has both a --reference feature and a --dissociate option,
>> with dissociate allowing for a reference to *only* speed up network
>> transfers rather than have the resulting clone rely upon the reference
>> always being there (creates an independent repo).
>
> With the advent of git-worktree, I claim that --reference without further
> --dissociate is dangerous, such that the combination of these two are
> not the best UX, you could wish for.
>
>> But git submodule only allows for --reference, so there isn't a an
>> option to make a speedy independent submodule clone in one shot:
>> https://git-scm.com/docs/git-submodule
>> I checked the latest online documentation (currently at 2.16.3) and
>> the documentation in the latest sources (almost 2.18):
>> https://github.com/git/git/blob/next/Documentation/git-submodule.txt
>>
>> As far as I am aware this can be worked around with 'git repack -a'
>> and manual removal of the objects/info/alternates file afterward.
>> Though I don't know if this results in a less speedy clone than
>> dissociate would.
>
> That is an interesting workaround!
> I agree we should have the --dissociate option available for submodules.
> Care to implement it?
>
> Thanks,
> Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/4] subtree: move out of contrib
  2018-04-30 21:38   ` Stefan Beller
@ 2018-04-30 21:53     ` Avery Pennarun
  2018-04-30 22:18       ` Stefan Beller
  2018-04-30 22:21       ` Ævar Arnfjörð Bjarmason
  0 siblings, 2 replies; 200+ results
From: Avery Pennarun @ 2018-04-30 21:53 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Ævar Arnfjörð Bjarmason, Git Mailing List,
	Junio C Hamano, Jeff King, Stephen R Guglielmo, A . Wilcox,
	David Aguilar

On Mon, Apr 30, 2018 at 5:38 PM, Stefan Beller <sbeller@google.com> wrote:
> On Mon, Apr 30, 2018 at 1:45 PM, Avery Pennarun <apenwarr@gmail.com> wrote:
> No objections from me either.
>
> Submodules seem to serve a slightly different purpose, though?

I think the purpose is actually the same - it's just the tradeoffs
that are difference.  Both sets of tradeoffs kind of suck.

> With Subtrees the superproject always contains all the code,
> even when you squash the subtree histroy when merging it in.
> In the submodule world, you may not have access to one of the
> submodules.

Right.  Personally I think it's a disadvantage of subtree that it
always contains all the code (what if some people don't want the code
for a particular build variant?).  However, it's a huge pain that
submodules *don't* contain all the code (what if I'm not online right
now, or the site supposedly containing the code goes offline, or I
want to make my own fork?).

For the best of both worlds, I've often thought that a good balance
would be to use the same data structure that submodule uses, but to
store all the code in a single git repo under different refs, which we
might or might not download (or might or might not have different
ACLs) under different circumstances.  However, when some projects get
really huge (lots of very big submodule dependencies), then repacking
one-big-repo starts becoming unwieldy; in that situation git-subtree
also fails completely.

> Submodules do not need to produce a synthetic project history
> when splitting off again, as the history is genuine. This allows
> for easier work with upstream.

Splitting for easier work upstream is great, and there really ought to
be an official version of 'git subtree split', which is good for all
sorts of purposes.

However, I suspect almost all uses of the split feature are a)
splitting a subtree that you previously merged in, or b) splitting a
subtree into a separate project that you want to maintain separately
from now on.  Repeated splits in case (a) are only necessary because
you're not using submodules, or in case (b) are only necessary because
you didn't *switch* to submodules when it finally came time to split
the projects.  (In both cases you probably didn't switch to submodules
because you didn't like one of its tradeoffs, especially the need to
track multiple repos when you fork.)

> Subtrees present you the whole history by default and the user
> needs to be explicit about not wanting to see history from the
> subtree, which is the opposite of submodules (though this
> may be planned in the future to switch).

It turns out that AFAIK, almost everyone prefers 'git subtree
--squash', which squashes into a single commit each time you merge,
much like git submodule does.  I doubt people would cry too much if
the full-history feature went away.

There's one exception, which is doing a one-time permanent merge of
two projects into one.  That's a nice feature, but is probably used
extremely rarely.  More often people get into a
merge-split-merge-split cycle that would be better served by a
slightly improved git-submodule.

>> The gerrit team (eg. Stefan Beller) has been doing some really great
>> stuff to make submodules more usable by helping with relative
>> submodule links and by auto-updating links in supermodules at the
>> right times.  Unfortunately doing that requires help from the server
>> side, which kind of messes up decentralization and so doesn't solve
>> the problem in the general case.
>
> Conceptually Gerrit is doing
>
>   while true:
>     git submodule update --remote
>     if worktree is dirty:
>         git commit "update the submodules"
>
> just that Gerrit doesn't poll but does it event based.

...and it's super handy :)  The problem is it's fundamentally
centralized: because gerrit can serialize merges into the submodule,
it also knows exactly how to update the link in the supermodule.  If
there was wild branching and merging (as there often is in git) and
you had to resolve conflicts between two submodules, I don't think it
would be obvious at all how to do it automatically when pushing a
submodule.  (This also works quite badly with git subtree --squash.)

>> I really wish there were a good answer, but I don't know what it is.
>> I do know that lots of people seem to at least be happy using
>> git-subtree, and would be even happier if it were installed
>> automatically with git.
>
> https://trends.google.com/trends/explore?date=all&q=git%20subtree,git%20submodule
>
> Not sure what to make of this data.

Clearly people need a lot more help when using submodules than when
using subtree :)

Have fun,

Avery

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/4] subtree: move out of contrib
  2018-04-30 21:53     ` Avery Pennarun
@ 2018-04-30 22:18       ` Stefan Beller
  2018-04-30 22:21       ` Ævar Arnfjörð Bjarmason
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-04-30 22:18 UTC (permalink / raw)
  To: Avery Pennarun, Jonathan Tan
  Cc: Ævar Arnfjörð Bjarmason, Git Mailing List,
	Junio C Hamano, Jeff King, Stephen R Guglielmo, A . Wilcox,
	David Aguilar

On Mon, Apr 30, 2018 at 2:53 PM, Avery Pennarun <apenwarr@gmail.com> wrote:

> For the best of both worlds, I've often thought that a good balance
> would be to use the same data structure that submodule uses, but to
> store all the code in a single git repo under different refs, which we
> might or might not download (or might or might not have different
> ACLs) under different circumstances.

There has been some experimentation with having a simpler ref
surface on the submodule side,
https://public-inbox.org/git/cover.1512168087.git.jonathantanmy@google.com/

The way you describe the future of submodules, all we'd have to do
is to teach git-clone how to select the the "interesting" refs for your use
case. Any other command would assume all submodule data to be
in the main repository.

The difference to Jonathans proposal linked above, would be the
object store to be in the main repo and the refs to be prefixed
per submodule instead of "shadowed".

>  However, when some projects get
> really huge (lots of very big submodule dependencies), then repacking
> one-big-repo starts becoming unwieldy; in that situation git-subtree
> also fails completely.

Yes, but that is a general scaling problem of Git that could be tackled,
e.g. repack into multiple packs serially instead of putting everything
into one pack.

>> Submodules do not need to produce a synthetic project history
>> when splitting off again, as the history is genuine. This allows
>> for easier work with upstream.
>
> Splitting for easier work upstream is great, and there really ought to
> be an official version of 'git subtree split', which is good for all
> sorts of purposes.
>
> However, I suspect almost all uses of the split feature are a)
> splitting a subtree that you previously merged in, or b) splitting a
> subtree into a separate project that you want to maintain separately
> from now on.  Repeated splits in case (a) are only necessary because
> you're not using submodules, or in case (b) are only necessary because
> you didn't *switch* to submodules when it finally came time to split
> the projects.  (In both cases you probably didn't switch to submodules
> because you didn't like one of its tradeoffs, especially the need to
> track multiple repos when you fork.)

That makes sense.

>
> There's one exception, which is doing a one-time permanent merge of
> two projects into one.  That's a nice feature, but is probably used
> extremely rarely.  More often people get into a
> merge-split-merge-split cycle that would be better served by a
> slightly improved git-submodule.

This rare use case is how git-subtree came into existence in gits
contrib directory AFAICT,
https://kernel.googlesource.com/pub/scm/git/git/+/634392b26275fe5436c0ea131bc89b46476aa4ae
which is interesting to view in git-show, but I think defaults could
be tweaked there, as it currently shows me mostly a license file.

>> Conceptually Gerrit is doing
>>
>>   while true:
>>     git submodule update --remote
>>     if worktree is dirty:
>>         git commit "update the submodules"
>>
>> just that Gerrit doesn't poll but does it event based.
>
> ...and it's super handy :)  The problem is it's fundamentally
> centralized: because gerrit can serialize merges into the submodule,
> it also knows exactly how to update the link in the supermodule.  If
> there was wild branching and merging (as there often is in git) and
> you had to resolve conflicts between two submodules, I don't think it
> would be obvious at all how to do it automatically when pushing a
> submodule.  (This also works quite badly with git subtree --squash.)

With the poll based solution I don't think you'd run into many more
problems than you would with Gerrits solution.

In a nearby thread, we were just discussing the submodule merging
strategies,
https://public-inbox.org/git/1524739599.20251.17.camel@klsmartin.com/
which might seem confusing, but the implementation is actually easy
as we just fastforward-only in submodules.

>>
>> https://trends.google.com/trends/explore?date=all&q=git%20subtree,git%20submodule
>>
>> Not sure what to make of this data.
>
> Clearly people need a lot more help when using submodules than when
> using subtree :)

That could be true. :)

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/4] subtree: move out of contrib
  2018-04-30 21:53     ` Avery Pennarun
  2018-04-30 22:18       ` Stefan Beller
@ 2018-04-30 22:21       ` Ævar Arnfjörð Bjarmason
  1 sibling, 0 replies; 200+ results
From: Ævar Arnfjörð Bjarmason @ 2018-04-30 22:21 UTC (permalink / raw)
  To: Avery Pennarun
  Cc: Stefan Beller, Git Mailing List, Junio C Hamano, Jeff King,
	Stephen R Guglielmo, A . Wilcox, David Aguilar


On Mon, Apr 30 2018, Avery Pennarun wrote:

> On Mon, Apr 30, 2018 at 5:38 PM, Stefan Beller <sbeller@google.com> wrote:
> There's one exception, which is doing a one-time permanent merge of
> two projects into one.  That's a nice feature, but is probably used
> extremely rarely.

FWIW this is the only thing I've used it for. I do this occasionally and
used to do this manually with format-patch + "perl -pe" before or
similar when I needed to merge some repositories together, and then some
other times I was less stupid and manually started doing something
similar to what subtree is doing with a "move everything" commit just
before the merge of the two histories.

>> https://trends.google.com/trends/explore?date=all&q=git%20subtree,git%20submodule
>>
>> Not sure what to make of this data.
>
> Clearly people need a lot more help when using submodules than when
> using subtree :)

Pretty clear it's garbage data, unless we're to believe that the
relative interest of submodules in the US, Germany and Sweden is 51, 64
& 84, but 75, 100 and 0 for subtree.

^ permalink raw reply	[relevance 2%]

* Re: [PATCH 0/2] Add --progress and --dissociate to git submodule
      [irrelevant]     ` <20180501180908.17443-1-kcghost@gmail.com>
@ 2018-05-01 18:20       ` Stefan Beller
      [irrelevant]       ` <20180501180908.17443-2-kcghost@gmail.com>
      [irrelevant]       ` <20180501180908.17443-3-kcghost@gmail.com>
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-01 18:20 UTC (permalink / raw)
  To: Casey Fitzpatrick; +Cc: git

Hi Casey,

thanks for sending those patches and improving submodules!

On Tue, May 1, 2018 at 11:09 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
> These patches add --progress and --dissociate options to git submodule.
>
> The --progress option existed beforehand, but only for the update command and
> it was left undocumented.
>
> Both add and update submodule commands supported --reference, but not its pair
> option --dissociate which allows for independent clones rather than depending
> on the reference.

This makes sense.

>
> I apologize for any errors I may have made in creation of these patches or the
> formatting of these emails. This is the first time I have submitted patches to
> a mailing list.

The formatting seems to be alright, git-format-patch/git-send-email
just works. ;)

Stefan

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 1/2] submodule: Add --progress option to add command
      [irrelevant]       ` <20180501180908.17443-2-kcghost@gmail.com>
@ 2018-05-01 18:41         ` Stefan Beller
  2018-05-01 20:48           ` Casey Fitzpatrick
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-01 18:41 UTC (permalink / raw)
  To: Casey Fitzpatrick; +Cc: git

On Tue, May 1, 2018 at 11:09 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
> --progress was missing from add command, was only in update.
> Add --progress where it makes sense (both clone helper commands).
> Add missing documentation entry.
> Add test.

Maybe:
  The '--progress' was introduced in 72c5f88311d (clone: pass --progress
  decision to recursive submodules, 2016-09-22) to fix the progress reporting
  of the clone command. Also add the progress option to the 'submodule add'
  command. The update command already support the progress flag, but it
  is not documented.

> @@ -117,6 +117,9 @@ cmd_add()
>                 -q|--quiet)
>                         GIT_QUIET=1
>                         ;;
> +               --progress)
> +                       progress="--progress"
> +                       ;;

The code looks good, though unlike the other commands progress is a
boolean decision.

If we want to support --no-progress as well, we can do so by adding
    --no-progress)
        progress="--no-progress"
        ;;
and then the submodule helper ought to cope with it.


> +test_expect_success 'setup test repo' '
> +       mkdir parent &&
> +       (cd parent && git init &&
> +        echo one >file && git add file &&
> +        git commit -m one)
> +'

Coding style:
    (
        parens open on its own line, and its contents
        are indented by a tab.

Instead of coding this yourself, you could write the
test as:

    test_create_repo parent &&
    test_create_commit -C parent one

> +test_expect_success 'clone -o' '

What are we testing here? I do not quite see the connection to
testing --progress here.

> +       git clone -o foo parent clone-o &&
> +       (cd clone-o && git rev-parse --verify refs/remotes/foo/master)


> +test_expect_success 'redirected submodule add does not show progress' '
> +       (
> +               cd addtest &&



> +               git submodule add "file://$submodurl/parent" submod-redirected \
> +                       >out 2>err &&
> +               ! grep % err &&

We're grepping for percent to see that there is no progress. At first I thought
the percent sign might be a special character, had to research it to know it's
meant literally. TiL!

> +               test_i18ngrep ! "Checking connectivity" err

Makes sense.

> +       )

We could omit the extra shell by using

    git -C addtest submodule add "file://$... \
            >../out 2>../err &&

Why do we need 'out' ?

> +test_expect_success 'redirected submodule add --progress does show progress' '
> +       (
> +               cd addtest &&
> +               git submodule add --progress "file://$submodurl/parent" \
> +                       submod-redirected-progress >out 2>err && \
> +               grep % err
> +       )
> +'

Thanks for writing these tests!
Stefan

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 2/2] submodule: Add --dissociate option to add/update commands
      [irrelevant]       ` <20180501180908.17443-3-kcghost@gmail.com>
@ 2018-05-01 20:23         ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-01 20:23 UTC (permalink / raw)
  To: Casey Fitzpatrick; +Cc: git

On Tue, May 1, 2018 at 11:09 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
>
> +test_expect_success 'submodule add --reference with --dissociate doesnt use alternates' '
> +       (
> +               cd super &&
> +               git submodule add --reference ../B --dissociate "file://$base_dir/A" sub-dissociate &&
> +               git commit -m B-super-added &&
> +               git repack -ad
> +       ) &&
> +       test_must_fail test_alternate_is_used super/.git/modules/sub-dissociate/objects/info/alternates super/sub-dissociate

We do already have the unfortunate precedent of using "test_must_fail
test_alternate_is_used"
(and it was partially included by me in 31224cbdc72 (clone: recursive
and reference option
triggers submodule alternates, 2016-08-17), further used in
bf03b790471 (submodule--helper:
set alternateLocation for cloned submodules, 2016-12-08).

I think it is the wrong approach though, as the test_must_fail only ensure that
*something* doesn't return success. Usually we use it with
"test_must_fail git ..."
to ensure we properly error out on user requests, that are impossible
to fulfill in
the current implementation.

For this test case I'd suggest we rather test that the no alternate is setup,

    test_path_is_missing
super/.git/modules/sub-dissociate/objects/info/alternates

as that tests the correctness of the --dissociate option?

Thanks,
Stefan

^ permalink raw reply	[relevance 8%]

* Re: [PATCH 1/2] submodule: Add --progress option to add command
  2018-05-01 18:41         ` [PATCH 1/2] submodule: Add --progress option to add command Stefan Beller
@ 2018-05-01 20:48           ` Casey Fitzpatrick
  0 siblings, 0 replies; 200+ results
From: Casey Fitzpatrick @ 2018-05-01 20:48 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git

Thanks, I'll clean it up based on your comments. I based the tests
from t5606-clone-options.sh; I'm not sure why now but I thought I
needed that clone -o thing from there, but it appears useless.



On Tue, May 1, 2018 at 2:41 PM, Stefan Beller <sbeller@google.com> wrote:
> On Tue, May 1, 2018 at 11:09 AM, Casey Fitzpatrick <kcghost@gmail.com> wrote:
>> --progress was missing from add command, was only in update.
>> Add --progress where it makes sense (both clone helper commands).
>> Add missing documentation entry.
>> Add test.
>
> Maybe:
>   The '--progress' was introduced in 72c5f88311d (clone: pass --progress
>   decision to recursive submodules, 2016-09-22) to fix the progress reporting
>   of the clone command. Also add the progress option to the 'submodule add'
>   command. The update command already support the progress flag, but it
>   is not documented.
>
>> @@ -117,6 +117,9 @@ cmd_add()
>>                 -q|--quiet)
>>                         GIT_QUIET=1
>>                         ;;
>> +               --progress)
>> +                       progress="--progress"
>> +                       ;;
>
> The code looks good, though unlike the other commands progress is a
> boolean decision.
>
> If we want to support --no-progress as well, we can do so by adding
>     --no-progress)
>         progress="--no-progress"
>         ;;
> and then the submodule helper ought to cope with it.
>
>
>> +test_expect_success 'setup test repo' '
>> +       mkdir parent &&
>> +       (cd parent && git init &&
>> +        echo one >file && git add file &&
>> +        git commit -m one)
>> +'
>
> Coding style:
>     (
>         parens open on its own line, and its contents
>         are indented by a tab.
>
> Instead of coding this yourself, you could write the
> test as:
>
>     test_create_repo parent &&
>     test_create_commit -C parent one
>
>> +test_expect_success 'clone -o' '
>
> What are we testing here? I do not quite see the connection to
> testing --progress here.
>
>> +       git clone -o foo parent clone-o &&
>> +       (cd clone-o && git rev-parse --verify refs/remotes/foo/master)
>
>
>> +test_expect_success 'redirected submodule add does not show progress' '
>> +       (
>> +               cd addtest &&
>
>
>
>> +               git submodule add "file://$submodurl/parent" submod-redirected \
>> +                       >out 2>err &&
>> +               ! grep % err &&
>
> We're grepping for percent to see that there is no progress. At first I thought
> the percent sign might be a special character, had to research it to know it's
> meant literally. TiL!
>
>> +               test_i18ngrep ! "Checking connectivity" err
>
> Makes sense.
>
>> +       )
>
> We could omit the extra shell by using
>
>     git -C addtest submodule add "file://$... \
>             >../out 2>../err &&
>
> Why do we need 'out' ?
>
>> +test_expect_success 'redirected submodule add --progress does show progress' '
>> +       (
>> +               cd addtest &&
>> +               git submodule add --progress "file://$submodurl/parent" \
>> +                       submod-redirected-progress >out 2>err && \
>> +               grep % err
>> +       )
>> +'
>
> Thanks for writing these tests!
> Stefan

^ permalink raw reply	[relevance 5%]

* [PATCH 01/13] repository: introduce object parser field
      [irrelevant] <20180501213403.14643-1-sbeller@google.com>
@ 2018-05-01 21:33 ` Stefan Beller
      [irrelevant]   ` <CACsJy8CgX6BME=EhiDBfMRzBOYDBNHE6Ouxv4fZC-GW7Rsi81Q@mail.gmail.com>
      [irrelevant] ` <20180507225916.155236-1-sbeller@google.com>
  1 sibling, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-01 21:33 UTC (permalink / raw)
  To: git; +Cc: jamill, Stefan Beller, Jonathan Nieder

Git's object access code can be thought of as containing two layers:
the raw object store provides access to raw object content, while the
higher level obj_hash code parses raw objects and keeps track of
parenthood and other object relationships using 'struct object'.
Keeping these layers separate should make it easier to find relevant
functions and to change the implementation of one without having to
touch the other.

Add an object_parser field to 'struct repository' to prepare obj_hash
to be handled per repository.  Callers still only use the_repository
for now --- later patches will adapt them to handle arbitrary
repositories.

Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c     | 63 +++++++++++++++++++++++++++++++++-------------------
 object.h     |  8 +++++++
 repository.c |  7 ++++++
 repository.h | 11 ++++++++-
 4 files changed, 65 insertions(+), 24 deletions(-)

diff --git a/object.c b/object.c
index 5044d08e96c..503c73e2d1f 100644
--- a/object.c
+++ b/object.c
@@ -8,17 +8,14 @@
 #include "object-store.h"
 #include "packfile.h"
 
-static struct object **obj_hash;
-static int nr_objs, obj_hash_size;
-
 unsigned int get_max_object_index(void)
 {
-	return obj_hash_size;
+	return the_repository->parsed_objects->obj_hash_size;
 }
 
 struct object *get_indexed_object(unsigned int idx)
 {
-	return obj_hash[idx];
+	return the_repository->parsed_objects->obj_hash[idx];
 }
 
 static const char *object_type_strings[] = {
@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)
 	unsigned int i, first;
 	struct object *obj;
 
-	if (!obj_hash)
+	if (!the_repository->parsed_objects->obj_hash)
 		return NULL;
 
-	first = i = hash_obj(sha1, obj_hash_size);
-	while ((obj = obj_hash[i]) != NULL) {
+	first = i = hash_obj(sha1,
+			     the_repository->parsed_objects->obj_hash_size);
+	while ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {
 		if (!hashcmp(sha1, obj->oid.hash))
 			break;
 		i++;
-		if (i == obj_hash_size)
+		if (i == the_repository->parsed_objects->obj_hash_size)
 			i = 0;
 	}
 	if (obj && i != first) {
@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)
 		 * that we do not need to walk the hash table the next
 		 * time we look for it.
 		 */
-		SWAP(obj_hash[i], obj_hash[first]);
+		SWAP(the_repository->parsed_objects->obj_hash[i],
+		     the_repository->parsed_objects->obj_hash[first]);
 	}
 	return obj;
 }
@@ -124,19 +123,19 @@ static void grow_object_hash(void)
 	 * Note that this size must always be power-of-2 to match hash_obj
 	 * above.
 	 */
-	int new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;
+	int new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;
 	struct object **new_hash;
 
 	new_hash = xcalloc(new_hash_size, sizeof(struct object *));
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (!obj)
 			continue;
 		insert_obj_hash(obj, new_hash, new_hash_size);
 	}
-	free(obj_hash);
-	obj_hash = new_hash;
-	obj_hash_size = new_hash_size;
+	free(the_repository->parsed_objects->obj_hash);
+	the_repository->parsed_objects->obj_hash = new_hash;
+	the_repository->parsed_objects->obj_hash_size = new_hash_size;
 }
 
 void *create_object(const unsigned char *sha1, void *o)
@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)
 	obj->flags = 0;
 	hashcpy(obj->oid.hash, sha1);
 
-	if (obj_hash_size - 1 <= nr_objs * 2)
+	if (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)
 		grow_object_hash();
 
-	insert_obj_hash(obj, obj_hash, obj_hash_size);
-	nr_objs++;
+	insert_obj_hash(obj, the_repository->parsed_objects->obj_hash,
+			the_repository->parsed_objects->obj_hash_size);
+	the_repository->parsed_objects->nr_objs++;
 	return obj;
 }
 
@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)
 {
 	int i;
 
-	for (i=0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj)
 			obj->flags &= ~flags;
 	}
@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)
 {
 	int i;
 
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj && obj->type == OBJ_COMMIT)
 			obj->flags &= ~flags;
 	}
 }
 
+struct object_parser *object_parser_new(void)
+{
+	struct object_parser *o = xmalloc(sizeof(*o));
+	memset(o, 0, sizeof(*o));
+	return o;
+}
+
 struct raw_object_store *raw_object_store_new(void)
 {
 	struct raw_object_store *o = xmalloc(sizeof(*o));
@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)
 	close_all_packs(o);
 	o->packed_git = NULL;
 }
+
+void object_parser_clear(struct object_parser *o)
+{
+	/*
+	 * TOOD free objects in o->obj_hash.
+	 *
+	 * As objects are allocated in slabs (see alloc.c), we do
+	 * not need to free each object, but each slab instead.
+	 */
+}
diff --git a/object.h b/object.h
index f13f85b2a94..84380b2b4d5 100644
--- a/object.h
+++ b/object.h
@@ -1,6 +1,14 @@
 #ifndef OBJECT_H
 #define OBJECT_H
 
+struct object_parser {
+	struct object **obj_hash;
+	int nr_objs, obj_hash_size;
+};
+
+struct object_parser *object_parser_new(void);
+void object_parser_clear(struct object_parser *o);
+
 struct object_list {
 	struct object *item;
 	struct object_list *next;
diff --git a/repository.c b/repository.c
index a4848c1bd05..208ee10071c 100644
--- a/repository.c
+++ b/repository.c
@@ -2,6 +2,7 @@
 #include "repository.h"
 #include "object-store.h"
 #include "config.h"
+#include "object.h"
 #include "submodule-config.h"
 
 /* The main repository */
@@ -14,6 +15,8 @@ void initialize_the_repository(void)
 
 	the_repo.index = &the_index;
 	the_repo.objects = raw_object_store_new();
+	the_repo.parsed_objects = object_parser_new();
+
 	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
 }
 
@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,
 	memset(repo, 0, sizeof(*repo));
 
 	repo->objects = raw_object_store_new();
+	repo->parsed_objects = object_parser_new();
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)
 	raw_object_store_clear(repo->objects);
 	FREE_AND_NULL(repo->objects);
 
+	object_parser_clear(repo->parsed_objects);
+	FREE_AND_NULL(repo->parsed_objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index e6e00f541bd..8d042e0fa11 100644
--- a/repository.h
+++ b/repository.h
@@ -22,10 +22,19 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Holds any information related to accessing the raw object content.
+	 * Holds any information needed to retrieve the raw content
+	 * of objects. The object_parser uses this to get object
+	 * content which it then parses.
 	 */
 	struct raw_object_store *objects;
 
+	/*
+	 * State for the object parser. This owns all parsed objects
+	 * (struct object) so callers do not have to manage their
+	 * lifetime.
+	 */
+	struct object_parser *parsed_objects;
+
 	/* The store in which the refs are held. */
 	struct ref_store *refs;
 
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 4%]

* [PATCH 0/3] Add --progress and --dissociate to git submodule
  2018-04-30 21:39   ` Casey Fitzpatrick
      [irrelevant]     ` <20180501180908.17443-1-kcghost@gmail.com>
@ 2018-05-02  0:27     ` Casey Fitzpatrick
  2018-05-02  0:55     ` Casey Fitzpatrick
  2 siblings, 0 replies; 200+ results
From: Casey Fitzpatrick @ 2018-05-02  0:27 UTC (permalink / raw)
  To: git, sbeller, sunshine; +Cc: Casey Fitzpatrick

These patches add --progress and --dissociate options to git submodule.

The --progress option existed beforehand, but only for the update command and
it was left undocumented.

Both add and update submodule commands supported --reference, but not its pair
option --dissociate which allows for independent clones rather than depending
on the reference.

This is a resubmission with comments from Stefan Beller and Eric Sunshine
addressed.

Casey Fitzpatrick (3):
  submodule: clean up subsititions in script
  submodule: add --progress option to add command
  submodule: add --dissociate option to add/update commands

 Documentation/git-submodule.txt | 17 ++++++++++++++++-
 builtin/submodule--helper.c     | 16 +++++++++++++---
 git-submodule.sh                | 21 ++++++++++++++++-----
 t/t7400-submodule-basic.sh      | 16 ++++++++++++++++
 t/t7408-submodule-reference.sh  | 17 +++++++++++++++++
 5 files changed, 78 insertions(+), 9 deletions(-)

-- 
2.17.0.1.ge0414f29c.dirty


^ permalink raw reply	[relevance 8%]

* [PATCH 0/3] Add --progress and --dissociate to git submodule
  2018-04-30 21:39   ` Casey Fitzpatrick
      [irrelevant]     ` <20180501180908.17443-1-kcghost@gmail.com>
  2018-05-02  0:27     ` [PATCH 0/3] Add --progress and --dissociate to git submodule Casey Fitzpatrick
@ 2018-05-02  0:55     ` Casey Fitzpatrick
  2018-05-02  4:37       ` Junio C Hamano
  2 siblings, 1 reply; 200+ results
From: Casey Fitzpatrick @ 2018-05-02  0:55 UTC (permalink / raw)
  To: git, sbeller, sunshine; +Cc: Casey Fitzpatrick

These patches add --progress and --dissociate options to git submodule.

The --progress option existed beforehand, but only for the update command and
it was left undocumented.

Both add and update submodule commands supported --reference, but not its pair
option --dissociate which allows for independent clones rather than depending
on the reference.

This is a resubmission with comments from Stefan Beller and Eric Sunshine
addressed.

Casey Fitzpatrick (3):
  submodule: clean up subsititions in script
  submodule: add --progress option to add command
  submodule: add --dissociate option to add/update commands

 Documentation/git-submodule.txt | 17 ++++++++++++++++-
 builtin/submodule--helper.c     | 16 +++++++++++++---
 git-submodule.sh                | 21 ++++++++++++++++-----
 t/t7400-submodule-basic.sh      | 16 ++++++++++++++++
 t/t7408-submodule-reference.sh  | 17 +++++++++++++++++
 5 files changed, 78 insertions(+), 9 deletions(-)

-- 
2.17.0.1.ge0414f29c.dirty


^ permalink raw reply	[relevance 8%]

* Re: git-submodule is missing --dissociate option
  2018-04-30 18:19 ` git-submodule is missing --dissociate option Stefan Beller
  2018-04-30 21:39   ` Casey Fitzpatrick
@ 2018-05-02  4:34   ` Junio C Hamano
  1 sibling, 0 replies; 200+ results
From: Junio C Hamano @ 2018-05-02  4:34 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Casey Fitzpatrick, git

Stefan Beller <sbeller@google.com> writes:

>> As far as I am aware this can be worked around with 'git repack -a'
>> and manual removal of the objects/info/alternates file afterward.
>> Though I don't know if this results in a less speedy clone than
>> dissociate would.
>
> That is an interesting workaround!

It's not just "workaround", but actually is an implementation of
that exact option, no?

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/3] Add --progress and --dissociate to git submodule
  2018-05-02  0:55     ` Casey Fitzpatrick
@ 2018-05-02  4:37       ` Junio C Hamano
  2018-05-02  8:54         ` Casey Fitzpatrick
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2018-05-02  4:37 UTC (permalink / raw)
  To: Casey Fitzpatrick; +Cc: git, sbeller, sunshine

Casey Fitzpatrick <kcghost@gmail.com> writes:

> These patches add --progress and --dissociate options to git submodule.
>
> The --progress option existed beforehand, but only for the update command and
> it was left undocumented.
>
> Both add and update submodule commands supported --reference, but not its pair
> option --dissociate which allows for independent clones rather than depending
> on the reference.
>
> This is a resubmission with comments from Stefan Beller and Eric Sunshine
> addressed.

Readers would really appreciate it if these are prepared with
format-patch with -v$N option.  Unless they read faster than you
post patches, they will find a few messages identically titled, all
unread in their mailbox, and it is not always easy to tell which
ones are the latest.

Thanks.

^ permalink raw reply	[relevance 2%]

* Re: git merge banch w/ different submodule revision
  2018-04-30 17:02       ` Heiko Voigt
@ 2018-05-02  7:30         ` Middelschulte, Leif
  2018-05-03 16:42           ` Heiko Voigt
  0 siblings, 1 reply; 200+ results
From: Middelschulte, Leif @ 2018-05-02  7:30 UTC (permalink / raw)
  To: hvoigt, sbeller; +Cc: git, jacob.keller

Am Montag, den 30.04.2018, 19:02 +0200 schrieb Heiko Voigt:
> On Thu, Apr 26, 2018 at 03:19:36PM -0700, Stefan Beller wrote:
> > Stefan wrote:
> > > See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
> > > (68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
> > > to explain the situation you encounter. (specifically merge_submodule
> > > at the end of the diff)
> > 
> > +cc Heiko, author of that commit.
> 
> In that commit we tried to be very careful about. I do not understand
> the situation in which the current strategy would be wrong by default.
> 
> We only merge if the following applies:
> 
>  * The changes in the superproject on both sides point forward in the
>    submodule.
> 
>  * One side is contained in the other. Contained from the submodule
>    perspective. Sides from the superproject merge perspective.
> 
> So in case of the mentioned rewind of a submodule: Only one side of the
> 3-way merge would point forward and the merge would fail.
> 
> I can imagine, that in case of a temporary revert of a commit in the
> submodule that you would not want that merged into some other branch.
> But that would be the same without submodules. If you merge a temporary
> revert from another branch you will not get any conflict.
> 
> So maybe someone can explain the use case in which one would get the
> results that seem wrong?
In an ideal world, where there are no regressions between revisions, a
fast-forward is appropriate. However, we might have regressions within
submodules.

So the usecase is the following:

Environment:
- We have a base library L that is developed by some team (Team B).
- Another team (Team A) developes a product P based on those libraries using git-flow.

Case:
The problem occurs, when a developer (D) of Team A tries to have a feature
that he developed on a branch accepted by a core developer of P:
If a core developer of P advanced the reference of L within P (linear history), he might
deem the work D insufficient. Not because of the actual work by D, but regressions
that snuck into L. The core developer will not be informed about the missmatching
revisions of L.

So it would be nice if there was some kind of switch or at least some trigger.

Cheers,

Leif


> 
> Cheers Heiko
> 
^ permalink raw reply	[relevance 4%]

* Re: [PATCH 0/3] Add --progress and --dissociate to git submodule
  2018-05-02  4:37       ` Junio C Hamano
@ 2018-05-02  8:54         ` Casey Fitzpatrick
  0 siblings, 0 replies; 200+ results
From: Casey Fitzpatrick @ 2018-05-02  8:54 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: git, Stefan Beller, Eric Sunshine

Thanks I was not aware of that option.

On Wed, May 2, 2018 at 12:37 AM, Junio C Hamano <gitster@pobox.com> wrote:
> Casey Fitzpatrick <kcghost@gmail.com> writes:
>
>> These patches add --progress and --dissociate options to git submodule.
>>
>> The --progress option existed beforehand, but only for the update command and
>> it was left undocumented.
>>
>> Both add and update submodule commands supported --reference, but not its pair
>> option --dissociate which allows for independent clones rather than depending
>> on the reference.
>>
>> This is a resubmission with comments from Stefan Beller and Eric Sunshine
>> addressed.
>
> Readers would really appreciate it if these are prepared with
> format-patch with -v$N option.  Unless they read faster than you
> post patches, they will find a few messages identically titled, all
> unread in their mailbox, and it is not always easy to tell which
> ones are the latest.
>
> Thanks.

^ permalink raw reply	[relevance 2%]

* Re: [PATCH 01/13] repository: introduce object parser field
      [irrelevant]   ` <CACsJy8CgX6BME=EhiDBfMRzBOYDBNHE6Ouxv4fZC-GW7Rsi81Q@mail.gmail.com>
@ 2018-05-02 17:26     ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-02 17:26 UTC (permalink / raw)
  To: Duy Nguyen; +Cc: Git Mailing List, Jameson Miller, Jonathan Nieder

On Wed, May 2, 2018 at 10:17 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, May 1, 2018 at 11:33 PM, Stefan Beller <sbeller@google.com> wrote:
>>         /*
>> -        * Holds any information related to accessing the raw object content.
>> +        * Holds any information needed to retrieve the raw content
>> +        * of objects. The object_parser uses this to get object
>> +        * content which it then parses.
>>          */
>>         struct raw_object_store *objects;
>>
>> +       /*
>> +        * State for the object parser. This owns all parsed objects
>> +        * (struct object) so callers do not have to manage their
>> +        * lifetime.
>> +        */
>> +       struct object_parser *parsed_objects;
>
> I like this name 'parsed_objects'. Should we rename the struct after
> it (e.g. parsed_object_store as opposed to raw_object_store above)?

I can rename it to parsed_object_store for consistency.

> Another suggestion is object_pool, if we keep 'struct object' instead
> of 'struct parsed_object' and also want to keep current allocation
> behavior: no individual deallocation. If you free, you free the whole
> pool (e.g. you could run rev-list --all --objects in a separate pool,
> once you're done, you delete the whole pool).

That is what the following patches will be about, you can
only free the whole set of parsed objects.

So if you want to do a separate rev walk, you may need to
instantiate a new repository for it (ideally you'd only need a
separate parsed object store).

I'd want to have the ability to have separate pools for submodules,
such that they can be free'd on a per-repo basis.

> --
> Duy

^ permalink raw reply	[relevance 5%]

* [PATCH 0/5] Rebooting pc/submodule-helper-foreach
@ 2018-05-03  0:53 Stefan Beller
  2018-05-03  0:53 ` [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
                   ` (5 more replies)
  0 siblings, 6 replies; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

The "What's cooking" email carried this series for some time now:
> * pc/submodule-helper-foreach (2018-02-02) 5 commits
>  - submodule: port submodule subcommand 'foreach' from shell to C
> - submodule foreach: document variable '$displaypath'
>  - submodule foreach: clarify the '$toplevel' variable documentation
>  - submodule foreach: document '$sm_path' instead of '$path'
>  - submodule foreach: correct '$path' in nested submodules from a subdirectory
> 
>  Expecting a response to review comments
>  e.g. cf. <20180206150044.1bffbb573c088d38c8e44bf5@google.com>

I reworded the commit message of the first patch and nearly confused
myself again, as "toplevel" doesn't refer to the "topmost" superproject,
just the direct superproject of the submodule.

However I think the code of the first patch is correct as originally presented.
Just the wording of the commit message was changed to explain the reasoning
more extensively.

With this series, we get
* keep the invariant of $toplevel + $path to be an absolute path that is
  correctly pointing at the submodule. "git -C $toplevel config" + $name
  allowing to ask configuration of the submodule.  
* $displaypath will have the relative path form $pwd to the submodule root.
* better documentation.

In later patches we could add $topmost, that points at the superproject
in which the command was started from, and $path_from_topmost, that would
be the relative path from $topmost to the submodule, potentially skipping
intermediate superprojects.

Thanks,
Stefan

Prathamesh Chavan (5):
  submodule foreach: correct '$path' in nested submodules from a
    subdirectory
  submodule foreach: document '$sm_path' instead of '$path'
  submodule foreach: clarify the '$toplevel' variable documentation
  submodule foreach: document variable '$displaypath'
  submodule: port submodule subcommand 'foreach' from shell to C

 Documentation/git-submodule.txt |  15 ++--
 builtin/submodule--helper.c     | 148 ++++++++++++++++++++++++++++++++
 git-submodule.sh                |  40 +--------
 t/t7407-submodule-foreach.sh    |  38 +++++++-
 4 files changed, 194 insertions(+), 47 deletions(-)

-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 8%]

* [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
@ 2018-05-03  0:53 ` Stefan Beller
  2018-05-03 13:29   ` Ramsay Jones
  2018-05-03 17:47   ` Jonathan Tan
  2018-05-03  0:53 ` [PATCH 2/5] submodule foreach: document '$sm_path' instead of '$path' Stefan Beller
                   ` (4 subsequent siblings)
  5 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

From: Prathamesh Chavan <pc44800@gmail.com>

When running 'git submodule foreach --recursive' from a subdirectory of
your repository, nested submodules get a bogus value for $path:
For a submodule 'sub' that contains a nested submodule 'nested',
running 'git -C dir submodule foreach echo $path' from the root of the
superproject would report path='../nested' for the nested submodule.
The first part '../' is derived from the logic computing the relative
path from $pwd to the root of the superproject. The second part is the
submodule path inside the submodule. This value is of little use and is
hard to document.

There are three different possible solutions that have more value:
(a) The path value is documented as the path from the toplevel of the
    superproject to the mount point of the submodule. If 'the' refers to
    the superproject holding this submodule ('sub' holding 'nested'),
    the path would be expected to be path='nested'.
(b) In case 'the' superproject is referring to the toplevel, which
    is the superproject in which the original command was invoked,
    then path is expected to be path='sub/nested'.
(c) The documentation explains $path as [...] "relative to the
    superproject", following 091a6eb0fe (submodule: drop the
    top-level requirement, 2013-06-16), such that the nested submodule
    would be expected as path='../sub/nested', when "the" superproject
    is the superproject, where the command was run from
(d) or the value of path='nested' is expected if we take the
    intermediate superproject into account. [This is the same as
    (a); it highlights that the documentation is not clear, but
    technically correct if we were to revert 091a6eb0fe.]

The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
top-level requirement, 2013-06-16) the intent for $path seemed to be
relative to $cwd to the submodule worktree, but that did not work for
nested submodules, as the intermittent submodules were not included in
the path.

If we were to fix the meaning of the $path using (a), (d) such that "path"
is "the path from the intermediate superproject to the mount point of the
submodule", we would break any existing submodule user that runs foreach
from non-root of the superproject as the non-nested submodule
'../sub' would change its path to 'sub'.

If we were to fix the meaning of $path using (b) such that "path"
is "the path from the topmost superproject to the mount point of the
submodule", then we would break any user that uses nested submodules
(even from the root directory) as the 'nested' would become 'sub/nested'.

If we were to fix the meaning of $path using (c) such that "path"
is "the display path from where the original command was invoked to the
submodule", then we would break users that rely on the assumption that
"$toplevel / $path" is the absolute path of the submodule.

All groups can be found in the wild.  The author has no data if one group
outweighs the other by large margin, and offending each one seems equally
bad at first.  However in the authors imagination it is better to go with
(a) as running from a sub directory sounds like it is carried out by a
human rather than by some automation task.  With a human on the keyboard
the feedback loop is short and the changed behavior can be adapted to
quickly unlike some automation that can break silently.

Another argument in favor of (a) is the consistency of the variables
provided, "$toplevel / $path" gives the absolute path of the submodule,
with 'toplevel' (both the variable as well as the documentation) referring
to the immediate superproject of the submodule.

Documentation of the variable is adjusted in a follow-up patch.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh             |  1 -
 t/t7407-submodule-foreach.sh | 36 ++++++++++++++++++++++++++++++++++--
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 24914963ca2..331d71c908b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -345,7 +345,6 @@ cmd_foreach()
 				prefix="$prefix$sm_path/"
 				sanitize_submodule_env
 				cd "$sm_path" &&
-				sm_path=$(git submodule--helper relative-path "$sm_path" "$wt_prefix") &&
 				# we make $path available to scripts ...
 				path=$sm_path &&
 				if test $# -eq 1
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 6ba5daf42ee..5144cc6926b 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -82,9 +82,9 @@ test_expect_success 'test basic "submodule foreach" usage' '
 
 cat >expect <<EOF
 Entering '../sub1'
-$pwd/clone-foo1-../sub1-$sub1sha1
+$pwd/clone-foo1-sub1-$sub1sha1
 Entering '../sub3'
-$pwd/clone-foo3-../sub3-$sub3sha1
+$pwd/clone-foo3-sub3-$sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach" from subdirectory' '
@@ -196,6 +196,38 @@ test_expect_success 'test messages from "foreach --recursive" from subdirectory'
 	) &&
 	test_i18ncmp expect actual
 '
+sub1sha1=$(cd clone2/sub1 && git rev-parse HEAD)
+sub2sha1=$(cd clone2/sub2 && git rev-parse HEAD)
+sub3sha1=$(cd clone2/sub3 && git rev-parse HEAD)
+nested1sha1=$(cd clone2/nested1 && git rev-parse HEAD)
+nested2sha1=$(cd clone2/nested1/nested2 && git rev-parse HEAD)
+nested3sha1=$(cd clone2/nested1/nested2/nested3 && git rev-parse HEAD)
+submodulesha1=$(cd clone2/nested1/nested2/nested3/submodule && git rev-parse HEAD)
+
+cat >expect <<EOF
+Entering '../nested1'
+toplevel: $pwd/clone2 name: nested1 path: nested1 hash: $nested1sha1
+Entering '../nested1/nested2'
+toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 hash: $nested2sha1
+Entering '../nested1/nested2/nested3'
+toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 hash: $nested3sha1
+Entering '../nested1/nested2/nested3/submodule'
+toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule hash: $submodulesha1
+Entering '../sub1'
+toplevel: $pwd/clone2 name: foo1 path: sub1 hash: $sub1sha1
+Entering '../sub2'
+toplevel: $pwd/clone2 name: foo2 path: sub2 hash: $sub2sha1
+Entering '../sub3'
+toplevel: $pwd/clone2 name: foo3 path: sub3 hash: $sub3sha1
+EOF
+
+test_expect_success 'test "submodule foreach --recursive" from subdirectory' '
+	(
+		cd clone2/untracked &&
+		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path hash: \$sha1" >../../actual
+	) &&
+	test_i18ncmp expect actual
+'
 
 cat > expect <<EOF
 nested1-nested1
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 21%]

* [PATCH 3/5] submodule foreach: clarify the '$toplevel' variable documentation
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
  2018-05-03  0:53 ` [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
  2018-05-03  0:53 ` [PATCH 2/5] submodule foreach: document '$sm_path' instead of '$path' Stefan Beller
@ 2018-05-03  0:53 ` Stefan Beller
  2018-05-03 17:51   ` Jonathan Tan
  2018-05-03  0:53 ` [PATCH 4/5] submodule foreach: document variable '$displaypath' Stefan Beller
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

From: Prathamesh Chavan <pc44800@gmail.com>

It does not contain the topmost superproject as the author assumed,
but the direct superproject, such that $toplevel/$sm_path is the
actual absolute path of the submodule.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/git-submodule.txt | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 755ed695f08..408d5a0387f 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -188,7 +188,8 @@ foreach [--recursive] <command>::
 	$name is the name of the relevant submodule section in `.gitmodules`,
 	$sm_path is the path of the submodule as recorded in the superproject,
 	$sha1 is the commit as recorded in the superproject, and
-	$toplevel is the absolute path to the top-level of the superproject.
+	$toplevel is the absolute path to its direct superproject, such that
+	$toplevel/$sm_path is the absolute path of the submodule.
 	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
 	variable is now a deprecated synonym of '$sm_path' variable.
 	Any submodules defined in the superproject but not checked out are
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 20%]

* [PATCH 2/5] submodule foreach: document '$sm_path' instead of '$path'
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
  2018-05-03  0:53 ` [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
@ 2018-05-03  0:53 ` Stefan Beller
  2018-05-03 17:50   ` Jonathan Tan
  2018-05-03  0:53 ` [PATCH 3/5] submodule foreach: clarify the '$toplevel' variable documentation Stefan Beller
                   ` (3 subsequent siblings)
  5 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

From: Prathamesh Chavan <pc44800@gmail.com>

As using a variable '$path' may be harmful to users due to
capitalization issues, see 64394e3ae9 (git-submodule.sh: Don't
use $path variable in eval_gettext string, 2012-04-17). Adjust
the documentation to advocate for using $sm_path,  which contains
the same value. We still make the 'path' variable available and
document it as a deprecated synonym of 'sm_path'.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-submodule.txt | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 71c5618e82a..755ed695f08 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -183,12 +183,14 @@ information too.
 
 foreach [--recursive] <command>::
 	Evaluates an arbitrary shell command in each checked out submodule.
-	The command has access to the variables $name, $path, $sha1 and
+	The command has access to the variables $name, $sm_path, $sha1 and
 	$toplevel:
 	$name is the name of the relevant submodule section in `.gitmodules`,
-	$path is the name of the submodule directory relative to the
-	superproject, $sha1 is the commit as recorded in the superproject,
-	and $toplevel is the absolute path to the top-level of the superproject.
+	$sm_path is the path of the submodule as recorded in the superproject,
+	$sha1 is the commit as recorded in the superproject, and
+	$toplevel is the absolute path to the top-level of the superproject.
+	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
+	variable is now a deprecated synonym of '$sm_path' variable.
 	Any submodules defined in the superproject but not checked out are
 	ignored by this command. Unless given `--quiet`, foreach prints the name
 	of each submodule before evaluating the command.
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 21%]

* [PATCH 4/5] submodule foreach: document variable '$displaypath'
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
                   ` (2 preceding siblings ...)
  2018-05-03  0:53 ` [PATCH 3/5] submodule foreach: clarify the '$toplevel' variable documentation Stefan Beller
@ 2018-05-03  0:53 ` Stefan Beller
  2018-05-03  0:53 ` [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
  5 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

From: Prathamesh Chavan <pc44800@gmail.com>

It was observed that the variable '$displaypath' was accessible but
undocumented. Hence, document it.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-submodule.txt |  6 ++++--
 t/t7407-submodule-foreach.sh    | 22 +++++++++++-----------
 2 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 408d5a0387f..4372d00c42e 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -183,10 +183,12 @@ information too.
 
 foreach [--recursive] <command>::
 	Evaluates an arbitrary shell command in each checked out submodule.
-	The command has access to the variables $name, $sm_path, $sha1 and
-	$toplevel:
+	The command has access to the variables $name, $sm_path, $displaypath,
+	$sha1 and $toplevel:
 	$name is the name of the relevant submodule section in `.gitmodules`,
 	$sm_path is the path of the submodule as recorded in the superproject,
+	$displaypath contains the relative path from the current working
+	directory to the submodules root directory,
 	$sha1 is the commit as recorded in the superproject, and
 	$toplevel is the absolute path to its direct superproject, such that
 	$toplevel/$sm_path is the absolute path of the submodule.
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 5144cc6926b..77729ac4aa1 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -82,16 +82,16 @@ test_expect_success 'test basic "submodule foreach" usage' '
 
 cat >expect <<EOF
 Entering '../sub1'
-$pwd/clone-foo1-sub1-$sub1sha1
+$pwd/clone-foo1-sub1-../sub1-$sub1sha1
 Entering '../sub3'
-$pwd/clone-foo3-sub3-$sub3sha1
+$pwd/clone-foo3-sub3-../sub3-$sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach" from subdirectory' '
 	mkdir clone/sub &&
 	(
 		cd clone/sub &&
-		git submodule foreach "echo \$toplevel-\$name-\$sm_path-\$sha1" >../../actual
+		git submodule foreach "echo \$toplevel-\$name-\$sm_path-\$displaypath-\$sha1" >../../actual
 	) &&
 	test_i18ncmp expect actual
 '
@@ -206,25 +206,25 @@ submodulesha1=$(cd clone2/nested1/nested2/nested3/submodule && git rev-parse HEA
 
 cat >expect <<EOF
 Entering '../nested1'
-toplevel: $pwd/clone2 name: nested1 path: nested1 hash: $nested1sha1
+toplevel: $pwd/clone2 name: nested1 path: nested1 displaypath: ../nested1 hash: $nested1sha1
 Entering '../nested1/nested2'
-toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 hash: $nested2sha1
+toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 displaypath: ../nested1/nested2 hash: $nested2sha1
 Entering '../nested1/nested2/nested3'
-toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 hash: $nested3sha1
+toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 displaypath: ../nested1/nested2/nested3 hash: $nested3sha1
 Entering '../nested1/nested2/nested3/submodule'
-toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule hash: $submodulesha1
+toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule displaypath: ../nested1/nested2/nested3/submodule hash: $submodulesha1
 Entering '../sub1'
-toplevel: $pwd/clone2 name: foo1 path: sub1 hash: $sub1sha1
+toplevel: $pwd/clone2 name: foo1 path: sub1 displaypath: ../sub1 hash: $sub1sha1
 Entering '../sub2'
-toplevel: $pwd/clone2 name: foo2 path: sub2 hash: $sub2sha1
+toplevel: $pwd/clone2 name: foo2 path: sub2 displaypath: ../sub2 hash: $sub2sha1
 Entering '../sub3'
-toplevel: $pwd/clone2 name: foo3 path: sub3 hash: $sub3sha1
+toplevel: $pwd/clone2 name: foo3 path: sub3 displaypath: ../sub3 hash: $sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach --recursive" from subdirectory' '
 	(
 		cd clone2/untracked &&
-		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path hash: \$sha1" >../../actual
+		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path displaypath: \$displaypath hash: \$sha1" >../../actual
 	) &&
 	test_i18ncmp expect actual
 '
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 28%]

* [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
                   ` (3 preceding siblings ...)
  2018-05-03  0:53 ` [PATCH 4/5] submodule foreach: document variable '$displaypath' Stefan Beller
@ 2018-05-03  0:53 ` Stefan Beller
  2018-05-03  1:06   ` Stefan Beller
  2018-05-03 18:05   ` Jonathan Tan
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
  5 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-05-03  0:53 UTC (permalink / raw)
  To: gitster; +Cc: git, jonathantanmy, christian.couder, pc44800, Stefan Beller

From: Prathamesh Chavan <pc44800@gmail.com>

This aims to make git-submodule foreach a builtin. 'foreach' is ported to
the submodule--helper, and submodule--helper is called from
git-submodule.sh.

Helped-by: Brandon Williams <bmwill@google.com>
Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c | 148 ++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  39 +---------
 2 files changed, 149 insertions(+), 38 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index a404df3ea49..bbbea5868de 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -439,6 +439,153 @@ static void for_each_listed_submodule(const struct module_list *list,
 		fn(list->entries[i], cb_data);
 }
 
+struct cb_foreach {
+	int argc;
+	const char **argv;
+	const char *prefix;
+	char *toplevel;
+	int quiet;
+	int recursive;
+};
+#define CB_FOREACH_INIT { 0 }
+
+static void runcommand_in_submodule_cb(const struct cache_entry *list_item,
+				       void *cb_data)
+{
+	struct cb_foreach *info = cb_data;
+	const char *path = list_item->name;
+	const struct object_id *ce_oid = &list_item->oid;
+
+	const struct submodule *sub;
+	struct child_process cp = CHILD_PROCESS_INIT;
+	char *displaypath;
+
+	displaypath = get_submodule_displaypath(path, info->prefix);
+
+	sub = submodule_from_path(&null_oid, path);
+
+	if (!sub)
+		die(_("No url found for submodule path '%s' in .gitmodules"),
+			displaypath);
+
+	if (!is_submodule_populated_gently(path, NULL))
+		goto cleanup;
+
+	prepare_submodule_repo_env(&cp.env_array);
+
+	/*
+	* For the purpose of executing <command> in the submodule,
+	* separate shell is used for the purpose of running the
+	* child process.
+	*/
+	cp.use_shell = 1;
+	cp.dir = path;
+
+	/*
+	* NEEDSWORK: the command currently has access to the variables $name,
+	* $sm_path, $displaypath, $sha1 and $toplevel only when the command
+	* contains a single argument. This is done for maintaining a faithful
+	* translation from shell script.
+	*/
+	if (info->argc == 1) {
+		char *toplevel = xgetcwd();
+
+		argv_array_pushf(&cp.env_array, "name=%s", sub->name);
+		argv_array_pushf(&cp.env_array, "sm_path=%s", path);
+		argv_array_pushf(&cp.env_array, "displaypath=%s", displaypath);
+		argv_array_pushf(&cp.env_array, "sha1=%s",
+				oid_to_hex(ce_oid));
+		argv_array_pushf(&cp.env_array, "toplevel=%s", toplevel);
+
+		/*
+		* Since the path variable was accessible from the script
+		* before porting, it is also made available after porting.
+		* The environment variable "PATH" has a very special purpose
+		* on windows. And since environment variables are
+		* case-insensitive in windows, it interferes with the
+		* existing PATH variable. Hence, to avoid that, we expose
+		* path via the args argv_array and not via env_array.
+		*/
+		argv_array_pushf(&cp.args, "path=%s; %s",
+				path, info->argv[0]);
+
+		free(toplevel);
+	} else {
+		argv_array_pushv(&cp.args, info->argv);
+	}
+
+	if (!info->quiet)
+		printf(_("Entering '%s'\n"), displaypath);
+
+	if (info->argv[0] && run_command(&cp))
+		die(_("run_command returned non-zero status for %s\n."),
+			displaypath);
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = path;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", NULL);
+		argv_array_pushf(&cpr.args, "%s/", displaypath);
+		argv_array_pushl(&cpr.args, "submodule--helper", "foreach", "--recursive",
+				NULL);
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		if (info->toplevel)
+			argv_array_pushf(&cpr.args, "--toplevel=%s", info->toplevel);
+
+		argv_array_pushv(&cpr.args, info->argv);
+
+		if (run_command(&cpr))
+			die(_("run_command returned non-zero status while"
+				"recursing in the nested submodules of %s\n."),
+				displaypath);
+	}
+
+cleanup:
+	free(displaypath);
+}
+
+static int module_foreach(int argc, const char **argv, const char *prefix)
+{
+	struct cb_foreach info = CB_FOREACH_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+
+	struct option module_foreach_options[] = {
+		OPT__QUIET(&info.quiet, N_("Suppress output of entering each submodule command")),
+		OPT_BOOL(0, "recursive", &info.recursive,
+			 N_("Recurse into nested submodules")),
+		OPT_STRING(0, "toplevel", &info.toplevel, N_("path"),
+			   N_("path from the top level of the invocation")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule--helper foreach [--quiet] [--recursive] <command>"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_foreach_options,
+			     git_submodule_helper_usage, PARSE_OPT_KEEP_UNKNOWN);
+
+	if (module_list_compute(0, NULL, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.argc = argc;
+	info.argv = argv;
+	info.prefix = prefix;
+
+	for_each_listed_submodule(&list, runcommand_in_submodule_cb, &info);
+
+	return 0;
+}
+
 struct init_cb {
 	const char *prefix;
 	unsigned int flags;
@@ -1838,6 +1985,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"foreach", module_foreach, SUPPORT_SUPER_PREFIX},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"print-default-remote", print_default_remote, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index 331d71c908b..cba585f0754 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -323,44 +323,7 @@ cmd_foreach()
 		shift
 	done
 
-	toplevel=$(pwd)
-
-	# dup stdin so that it can be restored when running the external
-	# command in the subshell (and a recursive call to this function)
-	exec 3<&0
-
-	{
-		git submodule--helper list --prefix "$wt_prefix" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		if test -e "$sm_path"/.git
-		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Entering '\$displaypath'")"
-			name=$(git submodule--helper name "$sm_path")
-			(
-				prefix="$prefix$sm_path/"
-				sanitize_submodule_env
-				cd "$sm_path" &&
-				# we make $path available to scripts ...
-				path=$sm_path &&
-				if test $# -eq 1
-				then
-					eval "$1"
-				else
-					"$@"
-				fi &&
-				if test -n "$recursive"
-				then
-					cmd_foreach "--recursive" "$@"
-				fi
-			) <&3 3<&- ||
-			die "$(eval_gettext "Stopping at '\$displaypath'; script returned non-zero status.")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper foreach ${GIT_QUIET:+--quiet} ${recursive:+--recursive} "$@"
 }
 
 #
-- 
2.17.0.441.gb46fe60e1d-goog


^ permalink raw reply	[relevance 20%]

* Re: [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-03  0:53 ` [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
@ 2018-05-03  1:06   ` Stefan Beller
  2018-05-03 18:05   ` Jonathan Tan
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-03  1:06 UTC (permalink / raw)
  To: Junio C Hamano
  Cc: git, Jonathan Tan, Christian Couder, Prathamesh Chavan, Stefan Beller

On Wed, May 2, 2018 at 5:53 PM, Stefan Beller <sbeller@google.com> wrote:

> +struct cb_foreach {
> +       char *toplevel;
...
> +               OPT_STRING(0, "toplevel", &info.toplevel, N_("path"),
> +                          N_("path from the top level of the invocation")),

This is a leftover from my experimentation that I hinted at in the cover letter
that would be used to implement $topmost.  I'll remove this in a reroll.

^ permalink raw reply	[relevance 8%]

* [PATCH v4 0/3] Add --progress and --dissociate to git submodule
@ 2018-05-03 10:53 Casey Fitzpatrick
  0 siblings, 0 replies; 200+ results
From: Casey Fitzpatrick @ 2018-05-03 10:53 UTC (permalink / raw)
  To: git, sbeller, sunshine, gitster; +Cc: Casey Fitzpatrick

These patches add --progress and --dissociate options to git submodule.

The --progress option existed beforehand, but only for the update command and
it was left undocumented.

Both add and update submodule commands supported --reference, but not its pair
option --dissociate which allows for independent clones rather than depending
on the reference.

This is a resubmission with comments from Stefan Beller, Eric Sunshine, and
Junio C Hamano addressed.


Casey Fitzpatrick (3):
  submodule: clean up subsititions in script
  submodule: add --progress option to add command
  submodule: add --dissociate option to add/update commands

 Documentation/git-submodule.txt | 17 ++++++++++++++++-
 builtin/submodule--helper.c     | 16 +++++++++++++---
 git-submodule.sh                | 21 ++++++++++++++++-----
 t/t7400-submodule-basic.sh      | 16 ++++++++++++++++
 t/t7408-submodule-reference.sh  | 17 +++++++++++++++++
 5 files changed, 78 insertions(+), 9 deletions(-)

-- 
2.17.0.1.ge0414f29c.dirty


^ permalink raw reply	[relevance 8%]

* Re: [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-03  0:53 ` [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
@ 2018-05-03 13:29   ` Ramsay Jones
  2018-05-03 17:47   ` Jonathan Tan
  1 sibling, 0 replies; 200+ results
From: Ramsay Jones @ 2018-05-03 13:29 UTC (permalink / raw)
  To: Stefan Beller, gitster; +Cc: git, jonathantanmy, christian.couder, pc44800



On 03/05/18 01:53, Stefan Beller wrote:
> From: Prathamesh Chavan <pc44800@gmail.com>
> 
> When running 'git submodule foreach --recursive' from a subdirectory of
> your repository, nested submodules get a bogus value for $path:
> For a submodule 'sub' that contains a nested submodule 'nested',
> running 'git -C dir submodule foreach echo $path' from the root of the
> superproject would report path='../nested' for the nested submodule.
> The first part '../' is derived from the logic computing the relative
> path from $pwd to the root of the superproject. The second part is the
> submodule path inside the submodule. This value is of little use and is
> hard to document.
> 
> There are three different possible solutions that have more value:
> (a) The path value is documented as the path from the toplevel of the
>     superproject to the mount point of the submodule. If 'the' refers to
>     the superproject holding this submodule ('sub' holding 'nested'),
>     the path would be expected to be path='nested'.
> (b) In case 'the' superproject is referring to the toplevel, which
>     is the superproject in which the original command was invoked,
>     then path is expected to be path='sub/nested'.
> (c) The documentation explains $path as [...] "relative to the
>     superproject", following 091a6eb0fe (submodule: drop the
>     top-level requirement, 2013-06-16), such that the nested submodule
>     would be expected as path='../sub/nested', when "the" superproject
>     is the superproject, where the command was run from
> (d) or the value of path='nested' is expected if we take the
>     intermediate superproject into account. [This is the same as
>     (a); it highlights that the documentation is not clear, but
>     technically correct if we were to revert 091a6eb0fe.]
> 
> The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
> top-level requirement, 2013-06-16) the intent for $path seemed to be
> relative to $cwd to the submodule worktree, but that did not work for
> nested submodules, as the intermittent submodules were not included in
----------------------------^^^^^^^^^^^^
intermediate

ATB,
Ramsay Jones

^ permalink raw reply	[relevance 5%]

* Re: git merge banch w/ different submodule revision
  2018-05-02  7:30         ` Middelschulte, Leif
@ 2018-05-03 16:42           ` Heiko Voigt
  2018-05-04  8:29             ` Middelschulte, Leif
  0 siblings, 1 reply; 200+ results
From: Heiko Voigt @ 2018-05-03 16:42 UTC (permalink / raw)
  To: Middelschulte, Leif; +Cc: sbeller, git, jacob.keller

Hi,

On Wed, May 02, 2018 at 07:30:25AM +0000, Middelschulte, Leif wrote:
> Am Montag, den 30.04.2018, 19:02 +0200 schrieb Heiko Voigt:
> > On Thu, Apr 26, 2018 at 03:19:36PM -0700, Stefan Beller wrote:
> > > Stefan wrote:
> > > > See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
> > > > (68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
> > > > to explain the situation you encounter. (specifically merge_submodule
> > > > at the end of the diff)
> > > 
> > > +cc Heiko, author of that commit.
> > 
> > In that commit we tried to be very careful about. I do not understand
> > the situation in which the current strategy would be wrong by default.
> > 
> > We only merge if the following applies:
> > 
> >  * The changes in the superproject on both sides point forward in the
> >    submodule.
> > 
> >  * One side is contained in the other. Contained from the submodule
> >    perspective. Sides from the superproject merge perspective.
> > 
> > So in case of the mentioned rewind of a submodule: Only one side of the
> > 3-way merge would point forward and the merge would fail.
> > 
> > I can imagine, that in case of a temporary revert of a commit in the
> > submodule that you would not want that merged into some other branch.
> > But that would be the same without submodules. If you merge a temporary
> > revert from another branch you will not get any conflict.
> > 
> > So maybe someone can explain the use case in which one would get the
> > results that seem wrong?
> In an ideal world, where there are no regressions between revisions, a
> fast-forward is appropriate. However, we might have regressions within
> submodules.
> 
> So the usecase is the following:
> 
> Environment:
> - We have a base library L that is developed by some team (Team B).
> - Another team (Team A) developes a product P based on those libraries using git-flow.
> 
> Case:
> The problem occurs, when a developer (D) of Team A tries to have a feature
> that he developed on a branch accepted by a core developer of P:
> If a core developer of P advanced the reference of L within P (linear history), he might
> deem the work D insufficient. Not because of the actual work by D, but regressions
> that snuck into L. The core developer will not be informed about the missmatching
> revisions of L.
> 
> So it would be nice if there was some kind of switch or at least some trigger.

I still do not understand how the current behaviour is mismatching with
users expectations. Let's assume that you directly tracked the files of
L in your product repository P, without any submodule boundary. How
would the behavior be different? Would it be? If D started on an older
revision and gets merged into a newer revision, there can always be
regressions even without submodules.

Why would the core developer need to be informed about mismatching
revisions if he himself advanced the submodule?

It seems to me that you do not want to mix integration testing and
testing of the feature itself. How about just testing/reviewing on the
branch then? You would still get the submodule revision D was working on
and then in a later stage check if integration with everything else
works.

Cheers Heiko

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-03  0:53 ` [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
  2018-05-03 13:29   ` Ramsay Jones
@ 2018-05-03 17:47   ` Jonathan Tan
  2018-05-03 18:12     ` Stefan Beller
  1 sibling, 1 reply; 200+ results
From: Jonathan Tan @ 2018-05-03 17:47 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, christian.couder, pc44800

On Wed,  2 May 2018 17:53:54 -0700
Stefan Beller <sbeller@google.com> wrote:

> From: Prathamesh Chavan <pc44800@gmail.com>
> 
> When running 'git submodule foreach --recursive' from a subdirectory of
> your repository, nested submodules get a bogus value for $path:

I know I said in a previous e-mail [1] that we should use $path instead
of $sm_path, but now I got confused because the test shows a difference
in $sm_path, not $path. Maybe add "(and $sm_path, which is an alias of
$path)".

[1] https://public-inbox.org/git/20180206145406.b759164cead02cd3bb3fdce0@google.com/

> There are three different possible solutions that have more value:
> (a) The path value is documented as the path from the toplevel of the
>     superproject to the mount point of the submodule. If 'the' refers to
>     the superproject holding this submodule ('sub' holding 'nested'),
>     the path would be expected to be path='nested'.

What is "the", and why is it quoted?

Also, in (c), you use the same indicative present tense as "The path
value is documented" to describe the current situation, whereas this
seems like a situation you're proposing. It would be clearer to use the
imperative here ("Document $path to be the path from the toplevel...").
Do the same for the others.

Also, whenever you mention "superproject", make it clear which
superproject you're referring to ("outermost superproject" and
"immediate superproject" seem like good terms to me).

> (b) In case 'the' superproject is referring to the toplevel, which
>     is the superproject in which the original command was invoked,
>     then path is expected to be path='sub/nested'.

Same comment about "the", and I think s/toplevel, which is the
superproject in which the original command was invoked/outermost
superproject/.

> (c) The documentation explains $path as [...] "relative to the
>     superproject", following 091a6eb0fe (submodule: drop the
>     top-level requirement, 2013-06-16), such that the nested submodule
>     would be expected as path='../sub/nested', when "the" superproject
>     is the superproject, where the command was run from

How does "relative to the superproject" result in "../" appearing in the
path? I would expect "../" to appear only if a path is relative to $pwd.

> (d) or the value of path='nested' is expected if we take the
>     intermediate superproject into account. [This is the same as
>     (a); it highlights that the documentation is not clear, but
>     technically correct if we were to revert 091a6eb0fe.]

How exactly are we taking the intermediate superproject into account?

> The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
> top-level requirement, 2013-06-16) the intent for $path seemed to be
> relative to $cwd to the submodule worktree, but that did not work for
> nested submodules, as the intermittent submodules were not included in
> the path.

I think "pwd" is more used in the Git project than "cwd", so maybe use
$pwd here.

> If we were to fix the meaning of the $path using (a), (d) such that "path"
> is "the path from the intermediate superproject to the mount point of the
> submodule", we would break any existing submodule user that runs foreach
> from non-root of the superproject as the non-nested submodule
> '../sub' would change its path to 'sub'.
> 
> If we were to fix the meaning of $path using (b) such that "path"
> is "the path from the topmost superproject to the mount point of the
> submodule", then we would break any user that uses nested submodules
> (even from the root directory) as the 'nested' would become 'sub/nested'.
> 
> If we were to fix the meaning of $path using (c) such that "path"
> is "the display path from where the original command was invoked to the
> submodule", then we would break users that rely on the assumption that
> "$toplevel / $path" is the absolute path of the submodule.
> 
> All groups can be found in the wild.  The author has no data if one group
> outweighs the other by large margin, and offending each one seems equally
> bad at first.  However in the authors imagination it is better to go with
> (a) as running from a sub directory sounds like it is carried out by a
> human rather than by some automation task.  With a human on the keyboard
> the feedback loop is short and the changed behavior can be adapted to
> quickly unlike some automation that can break silently.

As I said in my previous e-mail, this is a good analysis.

> Another argument in favor of (a) is the consistency of the variables
> provided, "$toplevel / $path" gives the absolute path of the submodule,
> with 'toplevel' (both the variable as well as the documentation) referring
> to the immediate superproject of the submodule.

It's confusing to me that $toplevel is not the path to the outermost
superproject, but the path to the immediate superproject, so I'm not
sure that the goodness of "$toplevel/$path" exists. I would omit this
paragraph.

> Documentation of the variable is adjusted in a follow-up patch.

By "the variable", I assume you mean $toplevel? If yes, this doesn't
seem relevant to this patch, since this patch does not change the
meaning of $toplevel at all.

^ permalink raw reply	[relevance 4%]

* Re: [PATCH 2/5] submodule foreach: document '$sm_path' instead of '$path'
  2018-05-03  0:53 ` [PATCH 2/5] submodule foreach: document '$sm_path' instead of '$path' Stefan Beller
@ 2018-05-03 17:50   ` Jonathan Tan
  0 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-05-03 17:50 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, christian.couder, pc44800

On Wed,  2 May 2018 17:53:55 -0700
Stefan Beller <sbeller@google.com> wrote:

>  foreach [--recursive] <command>::
>  	Evaluates an arbitrary shell command in each checked out submodule.
> -	The command has access to the variables $name, $path, $sha1 and
> +	The command has access to the variables $name, $sm_path, $sha1 and
>  	$toplevel:
>  	$name is the name of the relevant submodule section in `.gitmodules`,
> -	$path is the name of the submodule directory relative to the
> -	superproject, $sha1 is the commit as recorded in the superproject,
> -	and $toplevel is the absolute path to the top-level of the superproject.
> +	$sm_path is the path of the submodule as recorded in the superproject,
> +	$sha1 is the commit as recorded in the superproject, and
> +	$toplevel is the absolute path to the top-level of the superproject.
> +	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
> +	variable is now a deprecated synonym of '$sm_path' variable.
>  	Any submodules defined in the superproject but not checked out are
>  	ignored by this command. Unless given `--quiet`, foreach prints the name
>  	of each submodule before evaluating the command.

This patch is fine as-is. I would go further and replace all mentions of
"the superproject" to "its immediate superproject", but that is
optional.

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 3/5] submodule foreach: clarify the '$toplevel' variable documentation
  2018-05-03  0:53 ` [PATCH 3/5] submodule foreach: clarify the '$toplevel' variable documentation Stefan Beller
@ 2018-05-03 17:51   ` Jonathan Tan
  0 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-05-03 17:51 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, christian.couder, pc44800

On Wed,  2 May 2018 17:53:56 -0700
Stefan Beller <sbeller@google.com> wrote:

>  	$name is the name of the relevant submodule section in `.gitmodules`,
>  	$sm_path is the path of the submodule as recorded in the superproject,
>  	$sha1 is the commit as recorded in the superproject, and
> -	$toplevel is the absolute path to the top-level of the superproject.
> +	$toplevel is the absolute path to its direct superproject, such that
> +	$toplevel/$sm_path is the absolute path of the submodule.
>  	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
>  	variable is now a deprecated synonym of '$sm_path' variable.
>  	Any submodules defined in the superproject but not checked out are

Ah, I commented on patch 2 before reading patch 3 properly. I think that
if you follow my suggestions on patch 2, it will be obvious that
$toplevel/$sm_path is the absolute path of the submodule, so that this
patch is no longer needed.

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-03  0:53 ` [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
  2018-05-03  1:06   ` Stefan Beller
@ 2018-05-03 18:05   ` Jonathan Tan
  1 sibling, 0 replies; 200+ results
From: Jonathan Tan @ 2018-05-03 18:05 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git, christian.couder, pc44800

On Wed,  2 May 2018 17:53:58 -0700
Stefan Beller <sbeller@google.com> wrote:

> +		argv_array_pushf(&cp.args, "path=%s; %s",
> +				path, info->argv[0]);

Do we need to quote the path here? (For example, what if path has a
quotation mark?)

Also, would it be useful to have a test testing a submodule with a
"weird" character (e.g. greater-than or single quote) in the name and
the path?

^ permalink raw reply	[relevance 8%]

* Re: [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-03 17:47   ` Jonathan Tan
@ 2018-05-03 18:12     ` Stefan Beller
  2018-05-04 21:03       ` Jonathan Tan
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-03 18:12 UTC (permalink / raw)
  To: Jonathan Tan; +Cc: Junio C Hamano, git, Christian Couder, Prathamesh Chavan

On Thu, May 3, 2018 at 10:47 AM, Jonathan Tan <jonathantanmy@google.com> wrote:
> On Wed,  2 May 2018 17:53:54 -0700
> Stefan Beller <sbeller@google.com> wrote:
>
>> From: Prathamesh Chavan <pc44800@gmail.com>
>>
>> When running 'git submodule foreach --recursive' from a subdirectory of
>> your repository, nested submodules get a bogus value for $path:
>
> I know I said in a previous e-mail [1] that we should use $path instead
> of $sm_path, but now I got confused because the test shows a difference
> in $sm_path, not $path. Maybe add "(and $sm_path, which is an alias of
> $path)".
>
> [1] https://public-inbox.org/git/20180206145406.b759164cead02cd3bb3fdce0@google.com/
>
>> There are three different possible solutions that have more value:
>> (a) The path value is documented as the path from the toplevel of the
>>     superproject to the mount point of the submodule. If 'the' refers to
>>     the superproject holding this submodule ('sub' holding 'nested'),
>>     the path would be expected to be path='nested'.
>
> What is "the", and why is it quoted?

because it is unclear what is emphasizes.
It could be the intermediate (direct) superproject, or it
could be the topmost superproject (where the command was run from).

Just having "the", makes it unclear which of both it refers to.

> Also, in (c), you use the same indicative present tense as "The path
> value is documented" to describe the current situation, whereas this
> seems like a situation you're proposing. It would be clearer to use the
> imperative here ("Document $path to be the path from the toplevel...").
> Do the same for the others.
>
> Also, whenever you mention "superproject", make it clear which
> superproject you're referring to ("outermost superproject" and
> "immediate superproject" seem like good terms to me).

ok, I'll rewrite the commit message with clearer superproject
annotations.

>> (b) In case 'the' superproject is referring to the toplevel, which
>>     is the superproject in which the original command was invoked,
>>     then path is expected to be path='sub/nested'.
>
> Same comment about "the", and I think s/toplevel, which is the
> superproject in which the original command was invoked/outermost
> superproject/.

The outermost superproject may not be the one you invoke the
command in.

We have
* the direct superproject. You can access it currently via $toplevel,
  which is misleading
* the superproject, where the command was invoked from,
  Currently only the undocumented $displaypath gives hints
  at this
* the outermost superproject, which may be even further
  out than the previous superproject in this list; Consider
  a layout with 4 git repositories nested as follows:

    outmost/invoked/direct/submodule

Submodule is part of the superproject "direct", but the command
could have been invoked from outmost/invoked/dir which has "direct"
as a submodule at '../direct' and itself is a submodule of outmost.

IMHO 'outmost' is of no relevance to the discussion. If you care about
it, discover it yourself via repeated calls to
'git rev-parse --show-superproject-working-tree'

'invoked' is only interesting for $displaypath, but apart from that
there is no benefit in knowing that at the time of processing
'submodule'. (It doesn't contain information for submodule, as
all those configs are in 'direct')

'direct' is a better name than 'toplevel', which is confusing as it
could be understood as any of 'direct', 'invoked' or 'outmost'.

>> (c) The documentation explains $path as [...] "relative to the
>>     superproject", following 091a6eb0fe (submodule: drop the
>>     top-level requirement, 2013-06-16), such that the nested submodule
>>     would be expected as path='../sub/nested', when "the" superproject
>>     is the superproject, where the command was run from
>
> How does "relative to the superproject" result in "../" appearing in the
> path? I would expect "../" to appear only if a path is relative to $pwd.

Gah. I messed that up. I wanted to emphasize *relative* and not the
superproject that is merely mentioned to form a sentence there.

>
>> (d) or the value of path='nested' is expected if we take the
>>     intermediate superproject into account. [This is the same as
>>     (a); it highlights that the documentation is not clear, but
>>     technically correct if we were to revert 091a6eb0fe.]
>
> How exactly are we taking the intermediate superproject into account?

'nested' is the full in-tree path from the intermediate (direct) superproject
to that submodule.

>> The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
>> top-level requirement, 2013-06-16) the intent for $path seemed to be
>> relative to $cwd to the submodule worktree, but that did not work for
>> nested submodules, as the intermittent submodules were not included in
>> the path.
>
> I think "pwd" is more used in the Git project than "cwd", so maybe use
> $pwd here.

ok.

>
>> If we were to fix the meaning of the $path using (a), (d) such that "path"
>> is "the path from the intermediate superproject to the mount point of the
>> submodule", we would break any existing submodule user that runs foreach
>> from non-root of the superproject as the non-nested submodule
>> '../sub' would change its path to 'sub'.
>>
>> If we were to fix the meaning of $path using (b) such that "path"
>> is "the path from the topmost superproject to the mount point of the
>> submodule", then we would break any user that uses nested submodules
>> (even from the root directory) as the 'nested' would become 'sub/nested'.
>>
>> If we were to fix the meaning of $path using (c) such that "path"
>> is "the display path from where the original command was invoked to the
>> submodule", then we would break users that rely on the assumption that
>> "$toplevel / $path" is the absolute path of the submodule.
>>
>> All groups can be found in the wild.  The author has no data if one group
>> outweighs the other by large margin, and offending each one seems equally
>> bad at first.  However in the authors imagination it is better to go with
>> (a) as running from a sub directory sounds like it is carried out by a
>> human rather than by some automation task.  With a human on the keyboard
>> the feedback loop is short and the changed behavior can be adapted to
>> quickly unlike some automation that can break silently.
>
> As I said in my previous e-mail, this is a good analysis.
>
>> Another argument in favor of (a) is the consistency of the variables
>> provided, "$toplevel / $path" gives the absolute path of the submodule,
>> with 'toplevel' (both the variable as well as the documentation) referring
>> to the immediate superproject of the submodule.
>
> It's confusing to me that $toplevel is not the path to the outermost
> superproject,

yes. That confused me for a while, too. Maybe deprecate that (just like we
deprecate $path) and introduce $superproject to mean the direct
superproject that holds the submodule currently being processed?


> but the path to the immediate superproject, so I'm not
> sure that the goodness of "$toplevel/$path" exists. I would omit this
> paragraph.

This exists for all nested submodules currently as these are run
from its root tree. For non-nested submodules, you still have the
part that is relative from pwd to the submodule.


>> Documentation of the variable is adjusted in a follow-up patch.
>
> By "the variable", I assume you mean $toplevel? If yes, this doesn't
> seem relevant to this patch, since this patch does not change the
> meaning of $toplevel at all.

ok.

^ permalink raw reply	[relevance 8%]

* Re: git merge banch w/ different submodule revision
  2018-05-03 16:42           ` Heiko Voigt
@ 2018-05-04  8:29             ` Middelschulte, Leif
  0 siblings, 0 replies; 200+ results
From: Middelschulte, Leif @ 2018-05-04  8:29 UTC (permalink / raw)
  To: hvoigt; +Cc: git, sbeller, jacob.keller

Hi,
Am Donnerstag, den 03.05.2018, 18:42 +0200 schrieb Heiko Voigt:
> Hi,
> 
> On Wed, May 02, 2018 at 07:30:25AM +0000, Middelschulte, Leif wrote:
> > Am Montag, den 30.04.2018, 19:02 +0200 schrieb Heiko Voigt:
> > > On Thu, Apr 26, 2018 at 03:19:36PM -0700, Stefan Beller wrote:
> > > > Stefan wrote:
> > > > > See https://github.com/git/git/commit/68d03e4a6e448aa557f52adef92595ac4d6cd4bd
> > > > > (68d03e4a6e (Implement automatic fast-forward merge for submodules, 2010-07-07)
> > > > > to explain the situation you encounter. (specifically merge_submodule
> > > > > at the end of the diff)
> > > > 
> > > > +cc Heiko, author of that commit.
> > > 
> > > In that commit we tried to be very careful about. I do not understand
> > > the situation in which the current strategy would be wrong by default.
> > > 
> > > We only merge if the following applies:
> > > 
> > >  * The changes in the superproject on both sides point forward in the
> > >    submodule.
> > > 
> > >  * One side is contained in the other. Contained from the submodule
> > >    perspective. Sides from the superproject merge perspective.
> > > 
> > > So in case of the mentioned rewind of a submodule: Only one side of the
> > > 3-way merge would point forward and the merge would fail.
> > > 
> > > I can imagine, that in case of a temporary revert of a commit in the
> > > submodule that you would not want that merged into some other branch.
> > > But that would be the same without submodules. If you merge a temporary
> > > revert from another branch you will not get any conflict.
> > > 
> > > So maybe someone can explain the use case in which one would get the
> > > results that seem wrong?
> > 
> > In an ideal world, where there are no regressions between revisions, a
> > fast-forward is appropriate. However, we might have regressions within
> > submodules.
> > 
> > So the usecase is the following:
> > 
> > Environment:
> > - We have a base library L that is developed by some team (Team B).
> > - Another team (Team A) developes a product P based on those libraries using git-flow.
> > 
> > Case:
> > The problem occurs, when a developer (D) of Team A tries to have a feature
> > that he developed on a branch accepted by a core developer of P:
> > If a core developer of P advanced the reference of L within P (linear history), he might
> > deem the work D insufficient. Not because of the actual work by D, but regressions
> > that snuck into L. The core developer will not be informed about the missmatching
> > revisions of L.
> > 
> > So it would be nice if there was some kind of switch or at least some trigger.
> 
> I still do not understand how the current behaviour is mismatching with
> users expectations. Let's assume that you directly tracked the files of
> L in your product repository P, without any submodule boundary. How
> would the behavior be different? Would it be? If D started on an older
> revision and gets merged into a newer revision, there can always be
> regressions even without submodules.
> 
> Why would the core developer need to be informed about mismatching
> revisions if he himself advanced the submodule?
In that case you'd be right. I should have picked my example more wisely.
Assume right here that not a core developer, but another developer advanced
the submodule (also via feature branch + merge).
> 
> It seems to me that you do not want to mix integration testing and
> testing of the feature itself. 
That's on point. That's why it would be nice if git *at least* warned about the different revisions wrt submodules.

But, I guess, I learned something about submodules:
I used to think of submodules as means to pin down a specific revision like: `ver == x`.
Now I'm learning that submodules are treated as `ver >= x` during a merge.

> How about just testing/reviewing on the
> branch then? You would still get the submodule revision D was working on
> and then in a later stage check if integration with everything else
> works.
Sure. But if the behavior deviates after a merge the merging developer is currently not
aware that it *might* have to do with different submodule revisions used, not the "actual" code merged.

Like not even "beware: the (feature) branch you've merged used an 'older' revision of X"

> 
> Cheers Heiko

Cheers,

Leif

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 1/5] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-03 18:12     ` Stefan Beller
@ 2018-05-04 21:03       ` Jonathan Tan
  0 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-05-04 21:03 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git, Christian Couder, Prathamesh Chavan

On Thu, 3 May 2018 11:12:27 -0700
Stefan Beller <sbeller@google.com> wrote:

> >> There are three different possible solutions that have more value:
> >> (a) The path value is documented as the path from the toplevel of the
> >>     superproject to the mount point of the submodule. If 'the' refers to
> >>     the superproject holding this submodule ('sub' holding 'nested'),
> >>     the path would be expected to be path='nested'.
> >
> > What is "the", and why is it quoted?
> 
> because it is unclear what is emphasizes.
> It could be the intermediate (direct) superproject, or it
> could be the topmost superproject (where the command was run from).
> 
> Just having "the", makes it unclear which of both it refers to.

Ah, I see - so s/'the'/'the superproject'/

> >> (b) In case 'the' superproject is referring to the toplevel, which
> >>     is the superproject in which the original command was invoked,
> >>     then path is expected to be path='sub/nested'.

And here, s/'the' superproject/'the superproject'/

> > Same comment about "the", and I think s/toplevel, which is the
> > superproject in which the original command was invoked/outermost
> > superproject/.
> 
> The outermost superproject may not be the one you invoke the
> command in.

Good point. Maybe "the superproject the original command was run from",
but I'm open to a better name. So I would write the beginning as
follows:

  <your first paragraph starting with "When running">

  Also, in git-submodule.txt, $path is documented to be the "name of the
  submodule directory relative to the superproject", but "the
  superproject" is ambiguous.

  To resolve both these issues, we could:
  (a) Change "the superproject" to "its immediate superproject", so
      $path would be "nested" instead of "../nested".
  (b) Change "the superproject" to "the superproject the original
      command was run from", so $path would be "sub/nested" instead of
      "../nested".
  (c) Change "the superproject" to "the directory the original command
      was run from", so $path would be "../sub/nested" instead of
      "../nested".

Going back to the original patch:

> The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
> top-level requirement, 2013-06-16) the intent for $path seemed to be
> relative to $cwd to the submodule worktree, but that did not work for
> nested submodules, as the intermittent submodules were not included in
> the path.

The (c) behavior was never really introduced, right? 091a6eb0fe
attempted to introduce (c), but it didn't work when --recursive was
specified.

^ permalink raw reply	[relevance 8%]

* Re: Weak option parsing in `git submodule`
      [irrelevant] <1525630243.15782.4.camel@gmail.com>
@ 2018-05-07 19:05 ` Stefan Beller
  2018-05-08  7:56   ` Kaartic Sivaraam
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-07 19:05 UTC (permalink / raw)
  To: Kaartic Sivaraam; +Cc: Git Mailing list, Jens Lehmann

On Sun, May 6, 2018 at 11:10 AM, Kaartic Sivaraam
<kaartic.sivaraam@gmail.com> wrote:
> I recently faced the consequence of the weak option parsing done by in
> `git submodule`. Initially tried to add a new submodule using the `git
> submodule add` sub-command in the following way,
>
>     $ git submodule add ./foo/bar
>
> This cloned the submodule into a new directory named 'bar' in the
> present directory. This was initially confusing as I expected `git
> submodule` to use the actual directory itself as it resides inside a
> sub-directory the main project and thus avoiding the creation of a new
> clone. Then I came to know that `git submodule` wasn't smart enough yet
> to identify this, by reading the documentation. Then, after realizing
> that I would have to specify the path in the command to avoid the
> redundant clone, I corrected the command without consulting the
> documentation (thoroughly) to,
>
>     $ git submodule add ./foo/bar --path ./foo/bar
>
> expecting that the path needs to be specified after an argument. This
> is what triggered the weird consequence of weak option parsing. The
> output I got for the above command was:
>
>     The following path is ignored by one of your .gitignore files:
>     --path
>     Use -f if you really want to add it.

Yuck, that is bad UX.

> That really confused me pretty much (being a person who doesn't know
> much about how `git submodule` works). Instead of complaining about an
> inexistent option '--path', it considers '--path' to be the <path>
> argument which seems to be bad. It doesn't even complain about the
> extra argument. I also dug to find the reason behind this totally weird
> message. It seems to be a consequence of the following lousy check
> ($sm_path is the normalized <path> argument):
>
>     if test -z "$force" &&
>             ! git add --dry-run --ignore-missing --no-warn-embedded-repo "$sm_path" > /dev/null 2>&1
>     then
>             eval_gettextln "The following path is ignored by one of your .gitignore files:
>     \$sm_path
>     Use -f if you really want to add it." >&2
>             exit 1
>     fi
>
>     The lack of checking for the reason behind why `git add` fails seems to
>     be the reason behind that weird message.

(from the man page)
git submodule [--quiet] add [<options>] [--] <repository> [<path>]

When options are given after <repository> or <path> we can count
the arguments and know something is up. (The number of arguments
must be 1 or 2. If it is 3 or above, something fishy is going on), which
I would suggest as a first step.

>     Ways to fix this:
>
>     1. Fix this particular issue by adding a '--' after the '--no-warn-
>     embedded-repo' option in the above check. But that would also
>     require that we allow other parts of the script to accept weird
>     paths such as '--path'. Not so useful/helpful.
>
>     2. Check for the actual return value of `git add` in the check and
>     act accordingly. Also, check if there are unnecessary arguments for
>     `submodule add`.

The second part of this suggestion seems to me as the way to go.
Do you want to write a patch?

>     3. Tighten option parsing in the `git-submodule` script to ensure
>     this doesn't happen in the long term and helps users to get more
>     relevant error messages.
>
>     I find 3 to be a long term solution but not sure if it's worth trying
>     as there are efforts towards porting `git submodule` to C. So, I guess
>     we should at least do 2 as a short-term solution.

Yeah, bringing it to C, would be nice as a long term solution instead
of piling up more and more shell features. :)

Thanks for such a well written bug report with suggested bug fixes. :)
Stefan

^ permalink raw reply	[relevance 7%]

* [PATCH v2 01/13] repository: introduce parsed objects field
      [irrelevant] ` <20180507225916.155236-1-sbeller@google.com>
@ 2018-05-07 22:59   ` Stefan Beller
      [irrelevant]   ` <20180508193736.14883-1-sbeller@google.com>
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-07 22:59 UTC (permalink / raw)
  To: git; +Cc: pclouds, jonathantanmy, gitster, jamill, Stefan Beller

Convert the existing global cache for parsed objects (obj_hash) into
repository-specific parsed object caches. Existing code that uses
obj_hash are modified to use the parsed object cache of
the_repository; future patches will use the parsed object caches of
other repositories.

Another future use case for a pool of objects is ease of memory management
in revision walking: If we can free the rev-list related memory early in
pack-objects (e.g. part of repack operation) then it could lower memory
pressure significantly when running on large repos. While this has been
discussed on the mailing list lately, this series doesn't implement this.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c     | 63 +++++++++++++++++++++++++++++++++-------------------
 object.h     |  8 +++++++
 repository.c |  7 ++++++
 repository.h | 13 ++++++++++-
 4 files changed, 67 insertions(+), 24 deletions(-)

diff --git a/object.c b/object.c
index 5044d08e96c..f7c624a7ba6 100644
--- a/object.c
+++ b/object.c
@@ -8,17 +8,14 @@
 #include "object-store.h"
 #include "packfile.h"
 
-static struct object **obj_hash;
-static int nr_objs, obj_hash_size;
-
 unsigned int get_max_object_index(void)
 {
-	return obj_hash_size;
+	return the_repository->parsed_objects->obj_hash_size;
 }
 
 struct object *get_indexed_object(unsigned int idx)
 {
-	return obj_hash[idx];
+	return the_repository->parsed_objects->obj_hash[idx];
 }
 
 static const char *object_type_strings[] = {
@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)
 	unsigned int i, first;
 	struct object *obj;
 
-	if (!obj_hash)
+	if (!the_repository->parsed_objects->obj_hash)
 		return NULL;
 
-	first = i = hash_obj(sha1, obj_hash_size);
-	while ((obj = obj_hash[i]) != NULL) {
+	first = i = hash_obj(sha1,
+			     the_repository->parsed_objects->obj_hash_size);
+	while ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {
 		if (!hashcmp(sha1, obj->oid.hash))
 			break;
 		i++;
-		if (i == obj_hash_size)
+		if (i == the_repository->parsed_objects->obj_hash_size)
 			i = 0;
 	}
 	if (obj && i != first) {
@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)
 		 * that we do not need to walk the hash table the next
 		 * time we look for it.
 		 */
-		SWAP(obj_hash[i], obj_hash[first]);
+		SWAP(the_repository->parsed_objects->obj_hash[i],
+		     the_repository->parsed_objects->obj_hash[first]);
 	}
 	return obj;
 }
@@ -124,19 +123,19 @@ static void grow_object_hash(void)
 	 * Note that this size must always be power-of-2 to match hash_obj
 	 * above.
 	 */
-	int new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;
+	int new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;
 	struct object **new_hash;
 
 	new_hash = xcalloc(new_hash_size, sizeof(struct object *));
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (!obj)
 			continue;
 		insert_obj_hash(obj, new_hash, new_hash_size);
 	}
-	free(obj_hash);
-	obj_hash = new_hash;
-	obj_hash_size = new_hash_size;
+	free(the_repository->parsed_objects->obj_hash);
+	the_repository->parsed_objects->obj_hash = new_hash;
+	the_repository->parsed_objects->obj_hash_size = new_hash_size;
 }
 
 void *create_object(const unsigned char *sha1, void *o)
@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)
 	obj->flags = 0;
 	hashcpy(obj->oid.hash, sha1);
 
-	if (obj_hash_size - 1 <= nr_objs * 2)
+	if (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)
 		grow_object_hash();
 
-	insert_obj_hash(obj, obj_hash, obj_hash_size);
-	nr_objs++;
+	insert_obj_hash(obj, the_repository->parsed_objects->obj_hash,
+			the_repository->parsed_objects->obj_hash_size);
+	the_repository->parsed_objects->nr_objs++;
 	return obj;
 }
 
@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)
 {
 	int i;
 
-	for (i=0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj)
 			obj->flags &= ~flags;
 	}
@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)
 {
 	int i;
 
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj && obj->type == OBJ_COMMIT)
 			obj->flags &= ~flags;
 	}
 }
 
+struct parsed_object_pool *parsed_object_pool_new(void)
+{
+	struct parsed_object_pool *o = xmalloc(sizeof(*o));
+	memset(o, 0, sizeof(*o));
+	return o;
+}
+
 struct raw_object_store *raw_object_store_new(void)
 {
 	struct raw_object_store *o = xmalloc(sizeof(*o));
@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)
 	close_all_packs(o);
 	o->packed_git = NULL;
 }
+
+void parsed_object_pool_clear(struct parsed_object_pool *o)
+{
+	/*
+	 * TOOD free objects in o->obj_hash.
+	 *
+	 * As objects are allocated in slabs (see alloc.c), we do
+	 * not need to free each object, but each slab instead.
+	 */
+}
diff --git a/object.h b/object.h
index f13f85b2a94..cecda7da370 100644
--- a/object.h
+++ b/object.h
@@ -1,6 +1,14 @@
 #ifndef OBJECT_H
 #define OBJECT_H
 
+struct parsed_object_pool {
+	struct object **obj_hash;
+	int nr_objs, obj_hash_size;
+};
+
+struct parsed_object_pool *parsed_object_pool_new(void);
+void parsed_object_pool_clear(struct parsed_object_pool *o);
+
 struct object_list {
 	struct object *item;
 	struct object_list *next;
diff --git a/repository.c b/repository.c
index a4848c1bd05..c23404677eb 100644
--- a/repository.c
+++ b/repository.c
@@ -2,6 +2,7 @@
 #include "repository.h"
 #include "object-store.h"
 #include "config.h"
+#include "object.h"
 #include "submodule-config.h"
 
 /* The main repository */
@@ -14,6 +15,8 @@ void initialize_the_repository(void)
 
 	the_repo.index = &the_index;
 	the_repo.objects = raw_object_store_new();
+	the_repo.parsed_objects = parsed_object_pool_new();
+
 	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
 }
 
@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,
 	memset(repo, 0, sizeof(*repo));
 
 	repo->objects = raw_object_store_new();
+	repo->parsed_objects = parsed_object_pool_new();
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)
 	raw_object_store_clear(repo->objects);
 	FREE_AND_NULL(repo->objects);
 
+	parsed_object_pool_clear(repo->parsed_objects);
+	FREE_AND_NULL(repo->parsed_objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index e6e00f541bd..73389e81afd 100644
--- a/repository.h
+++ b/repository.h
@@ -22,10 +22,21 @@ struct repository {
 	char *commondir;
 
 	/*
-	 * Holds any information related to accessing the raw object content.
+	 * Holds any information needed to retrieve the raw content
+	 * of objects. The object_parser uses this to get object
+	 * content which it then parses.
 	 */
 	struct raw_object_store *objects;
 
+	/*
+	 * All objects in this repository that have been parsed. This structure
+	 * owns all objects it references, so users of "struct object *"
+	 * generally do not need to free them; instead, when a repository is no
+	 * longer used, call parsed_object_pool_clear() on this structure, which
+	 * is called by the repositories repo_clear on its desconstruction.
+	 */
+	struct parsed_object_pool *parsed_objects;
+
 	/* The store in which the refs are held. */
 	struct ref_store *refs;
 
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 4%]

* Re: Weak option parsing in `git submodule`
  2018-05-07 19:05 ` Weak option parsing in `git submodule` Stefan Beller
@ 2018-05-08  7:56   ` Kaartic Sivaraam
  0 siblings, 0 replies; 200+ results
From: Kaartic Sivaraam @ 2018-05-08  7:56 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Git Mailing list, Jens Lehmann

[-- Attachment #1.1: Type: text/plain, Size: 3996 bytes --]

On Tuesday 08 May 2018 12:35 AM, Stefan Beller wrote:
>>     The lack of checking for the reason behind why `git add` fails seems to
>>     be the reason behind that weird message.
> 
> (from the man page)
> git submodule [--quiet] add [<options>] [--] <repository> [<path>]
> 
> When options are given after <repository> or <path> we can count
> the arguments and know something is up. (The number of arguments
> must be 1 or 2. If it is 3 or above, something fishy is going on), which
> I would suggest as a first step.
> 
>>     Ways to fix this:
>>
>>     1. Fix this particular issue by adding a '--' after the '--no-warn-
>>     embedded-repo' option in the above check. But that would also
>>     require that we allow other parts of the script to accept weird
>>     paths such as '--path'. Not so useful/helpful.
>>
>>     2. Check for the actual return value of `git add` in the check and
>>     act accordingly. Also, check if there are unnecessary arguments for
>>     `submodule add`.
> 
> The second part of this suggestion seems to me as the way to go.
> Do you want to write a patch?
> 

Incidentally, I was writing a patch to check for the return value of
`git add` to fix the particular issue I noted in my initial message.
Then I was in a dilemma as to whether this was the right way to do it.
So, I thought it would be better to ask before continuing with the patch
and hence started this thread. I wasn't counting the arguments to `git
submodule add` at that time.

Now that I'm ensured that my initial approach is not the worst way to do
things and as I'm about to write a patch for this, I'll sum up what I'm
about to achieve in the short-term fix patch, for the sake of clarity.

	1. Check the return value of `git add ...` and throw an error
	   appropriately.

	2. Check the no. of arguments to `submodule add` and throw an
	   error if there are more arguments than there should be.

	   I require a little clarification with this. How should this
	   be done. Does checking whether the number of arguments after
	   <repository> is not more than one do the job? Or am I missing
	   something?


>>     3. Tighten option parsing in the `git-submodule` script to ensure
>>     this doesn't happen in the long term and helps users to get more
>>     relevant error messages.
>>
>>     I find 3 to be a long term solution but not sure if it's worth trying
>>     as there are efforts towards porting `git submodule` to C. So, I guess
>>     we should at least do 2 as a short-term solution.
> 
> Yeah, bringing it to C, would be nice as a long term solution instead
> of piling up more and more shell features. :)
> 

Hope the day it is brought into C comes soon.


> Thanks for such a well written bug report with suggested bug fixes. :)

You're welcome :-)


-- 
Sivaraam

QUOTE:

“The most valuable person on any team is the person who makes everyone
else on the team more valuable, not the person who knows the most.”

      - Joel Spolsky


Sivaraam?

You possibly might have noticed that my signature recently changed from
'Kaartic' to 'Sivaraam' both of which are parts of my name. I find the
new signature to be better for several reasons one of which is that the
former signature has a lot of ambiguities in the place I live as it is a
common name (NOTE: it's not a common spelling, just a common name). So,
I switched signatures before it's too late.

That said, I won't mind you calling me 'Kaartic' if you like it [of
course ;-)]. You can always call me using either of the names.


KIND NOTE TO THE NATIVE ENGLISH SPEAKER:

As I'm not a native English speaker myself, there might be mistaeks in
my usage of English. I apologise for any mistakes that I make.

It would be "helpful" if you take the time to point out the mistakes.

It would be "super helpful" if you could provide suggestions about how
to correct those mistakes.

Thanks in advance!


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[relevance 6%]

* Re: [PATCH 1/2] gitk: show part of submodule log instead of empty pane when listing trees
      [irrelevant] ` <20180508122229.k3n3ccpa5g3g4dxa@pflmari>
@ 2018-05-08 17:07   ` Stefan Beller
  2018-05-09  9:01     ` Alex Riesen
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-08 17:07 UTC (permalink / raw)
  To: Alex Riesen; +Cc: Paul Mackerras, Git Mailing List, Junio C Hamano

On Tue, May 8, 2018 at 5:22 AM, Alex Riesen
<alexander.riesen@cetitec.com> wrote:
> From: Alex Riesen <raa.lkml@gmail.com>
>
> Currently, the submodules either are not shown at all (if listing a
> committed tree) or a Tcl error appears (when clicking on a submodule
> from the index list).

I do not understand where this appears, yet.
Where do I have to click to see the effects of this patch?

>
> This will make it show first arbitrarily chosen number of commits,
> which might be only marginally better.
>
> Signed-off-by: Alex Riesen <raa.lkml@gmail.com>
> ---
>  gitk | 42 ++++++++++++++++++++++++++++++++----------
>  1 file changed, 32 insertions(+), 10 deletions(-)
>
> diff --git a/gitk b/gitk
> index a14d7a1..d34833f 100755
> --- a/gitk
> +++ b/gitk
> @@ -7627,9 +7627,10 @@ proc gettreeline {gtf id} {
>             if {$i < 0} continue
>             set fname [string range $line [expr {$i+1}] end]
>             set line [string range $line 0 [expr {$i-1}]]
> -           if {$diffids ne $nullid2 && [lindex $line 1] ne "blob"} continue
> +           set objtype [lindex $line 1]
> +           if {$diffids ne $nullid2 && $objtype ne "blob" && $objtype ne "commit" } { continue }
>             set sha1 [lindex $line 2]
> -           lappend treeidlist($id) $sha1
> +           lappend treeidlist($id) "$sha1 $objtype"
>         }
>         if {[string index $fname 0] eq "\""} {
>             set fname [lindex $fname 0]
> @@ -7659,21 +7660,42 @@ proc showfile {f} {
>      global ctext_file_names ctext_file_lines
>      global ctext commentend
>
> +    set submodlog "git\\ log\\ --format='%h\\ %aN:\\ %s'\\ -100"

Do we want to respect the config option diff.submodule here?
The -100 is chosen rather arbitrarily. Ideally we'd only walk to the
previous entry?

> +    set fcmt ""
>      set i [lsearch -exact $treefilelist($diffids) $f]
>      if {$i < 0} {
>         puts "oops, $f not in list for id $diffids"
>         return
>      }
>      if {$diffids eq $nullid} {
> -       if {[catch {set bf [open $f r]} err]} {
> -           puts "oops, can't read $f: $err"
> -           return
> +       if {[file isdirectory $f]} {
> +           # a submodule
> +           if {[catch {set bf [open "| sh -c cd\\ \"$f\"&&$submodlog" r]} err]} {

Can we have $submodlog use the "git -C <path> command"
option, then we could save the "cd &&" part, which might even
save us from spawning a shell?

Thanks,
Stefan

^ permalink raw reply	[relevance 7%]

* [PATCH v3 01/13] repository: introduce parsed objects field
      [irrelevant]   ` <20180508193736.14883-1-sbeller@google.com>
@ 2018-05-08 19:37     ` " Stefan Beller
      [irrelevant]     ` <20180510004024.93974-1-sbeller@google.com>
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-08 19:37 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jamill, jonathantanmy, pclouds

Convert the existing global cache for parsed objects (obj_hash) into
repository-specific parsed object caches. Existing code that uses
obj_hash are modified to use the parsed object cache of
the_repository; future patches will use the parsed object caches of
other repositories.

Another future use case for a pool of objects is ease of memory management
in revision walking: If we can free the rev-list related memory early in
pack-objects (e.g. part of repack operation) then it could lower memory
pressure significantly when running on large repos. While this has been
discussed on the mailing list lately, this series doesn't implement this.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c     | 63 +++++++++++++++++++++++++++++++++-------------------
 object.h     |  8 +++++++
 repository.c |  7 ++++++
 repository.h |  9 ++++++++
 4 files changed, 64 insertions(+), 23 deletions(-)

diff --git a/object.c b/object.c
index 5044d08e96c..f7c624a7ba6 100644
--- a/object.c
+++ b/object.c
@@ -8,17 +8,14 @@
 #include "object-store.h"
 #include "packfile.h"
 
-static struct object **obj_hash;
-static int nr_objs, obj_hash_size;
-
 unsigned int get_max_object_index(void)
 {
-	return obj_hash_size;
+	return the_repository->parsed_objects->obj_hash_size;
 }
 
 struct object *get_indexed_object(unsigned int idx)
 {
-	return obj_hash[idx];
+	return the_repository->parsed_objects->obj_hash[idx];
 }
 
 static const char *object_type_strings[] = {
@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)
 	unsigned int i, first;
 	struct object *obj;
 
-	if (!obj_hash)
+	if (!the_repository->parsed_objects->obj_hash)
 		return NULL;
 
-	first = i = hash_obj(sha1, obj_hash_size);
-	while ((obj = obj_hash[i]) != NULL) {
+	first = i = hash_obj(sha1,
+			     the_repository->parsed_objects->obj_hash_size);
+	while ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {
 		if (!hashcmp(sha1, obj->oid.hash))
 			break;
 		i++;
-		if (i == obj_hash_size)
+		if (i == the_repository->parsed_objects->obj_hash_size)
 			i = 0;
 	}
 	if (obj && i != first) {
@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)
 		 * that we do not need to walk the hash table the next
 		 * time we look for it.
 		 */
-		SWAP(obj_hash[i], obj_hash[first]);
+		SWAP(the_repository->parsed_objects->obj_hash[i],
+		     the_repository->parsed_objects->obj_hash[first]);
 	}
 	return obj;
 }
@@ -124,19 +123,19 @@ static void grow_object_hash(void)
 	 * Note that this size must always be power-of-2 to match hash_obj
 	 * above.
 	 */
-	int new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;
+	int new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;
 	struct object **new_hash;
 
 	new_hash = xcalloc(new_hash_size, sizeof(struct object *));
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (!obj)
 			continue;
 		insert_obj_hash(obj, new_hash, new_hash_size);
 	}
-	free(obj_hash);
-	obj_hash = new_hash;
-	obj_hash_size = new_hash_size;
+	free(the_repository->parsed_objects->obj_hash);
+	the_repository->parsed_objects->obj_hash = new_hash;
+	the_repository->parsed_objects->obj_hash_size = new_hash_size;
 }
 
 void *create_object(const unsigned char *sha1, void *o)
@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)
 	obj->flags = 0;
 	hashcpy(obj->oid.hash, sha1);
 
-	if (obj_hash_size - 1 <= nr_objs * 2)
+	if (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)
 		grow_object_hash();
 
-	insert_obj_hash(obj, obj_hash, obj_hash_size);
-	nr_objs++;
+	insert_obj_hash(obj, the_repository->parsed_objects->obj_hash,
+			the_repository->parsed_objects->obj_hash_size);
+	the_repository->parsed_objects->nr_objs++;
 	return obj;
 }
 
@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)
 {
 	int i;
 
-	for (i=0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj)
 			obj->flags &= ~flags;
 	}
@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)
 {
 	int i;
 
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj && obj->type == OBJ_COMMIT)
 			obj->flags &= ~flags;
 	}
 }
 
+struct parsed_object_pool *parsed_object_pool_new(void)
+{
+	struct parsed_object_pool *o = xmalloc(sizeof(*o));
+	memset(o, 0, sizeof(*o));
+	return o;
+}
+
 struct raw_object_store *raw_object_store_new(void)
 {
 	struct raw_object_store *o = xmalloc(sizeof(*o));
@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)
 	close_all_packs(o);
 	o->packed_git = NULL;
 }
+
+void parsed_object_pool_clear(struct parsed_object_pool *o)
+{
+	/*
+	 * TOOD free objects in o->obj_hash.
+	 *
+	 * As objects are allocated in slabs (see alloc.c), we do
+	 * not need to free each object, but each slab instead.
+	 */
+}
diff --git a/object.h b/object.h
index f13f85b2a94..cecda7da370 100644
--- a/object.h
+++ b/object.h
@@ -1,6 +1,14 @@
 #ifndef OBJECT_H
 #define OBJECT_H
 
+struct parsed_object_pool {
+	struct object **obj_hash;
+	int nr_objs, obj_hash_size;
+};
+
+struct parsed_object_pool *parsed_object_pool_new(void);
+void parsed_object_pool_clear(struct parsed_object_pool *o);
+
 struct object_list {
 	struct object *item;
 	struct object_list *next;
diff --git a/repository.c b/repository.c
index a4848c1bd05..c23404677eb 100644
--- a/repository.c
+++ b/repository.c
@@ -2,6 +2,7 @@
 #include "repository.h"
 #include "object-store.h"
 #include "config.h"
+#include "object.h"
 #include "submodule-config.h"
 
 /* The main repository */
@@ -14,6 +15,8 @@ void initialize_the_repository(void)
 
 	the_repo.index = &the_index;
 	the_repo.objects = raw_object_store_new();
+	the_repo.parsed_objects = parsed_object_pool_new();
+
 	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
 }
 
@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,
 	memset(repo, 0, sizeof(*repo));
 
 	repo->objects = raw_object_store_new();
+	repo->parsed_objects = parsed_object_pool_new();
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)
 	raw_object_store_clear(repo->objects);
 	FREE_AND_NULL(repo->objects);
 
+	parsed_object_pool_clear(repo->parsed_objects);
+	FREE_AND_NULL(repo->parsed_objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index e6e00f541bd..6d199819905 100644
--- a/repository.h
+++ b/repository.h
@@ -26,6 +26,15 @@ struct repository {
 	 */
 	struct raw_object_store *objects;
 
+	/*
+	 * All objects in this repository that have been parsed. This structure
+	 * owns all objects it references, so users of "struct object *"
+	 * generally do not need to free them; instead, when a repository is no
+	 * longer used, call parsed_object_pool_clear() on this structure, which
+	 * is called by the repositories repo_clear on its desconstruction.
+	 */
+	struct parsed_object_pool *parsed_objects;
+
 	/* The store in which the refs are held. */
 	struct ref_store *refs;
 
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 4%]

* [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach
  2018-05-03  0:53 [PATCH 0/5] Rebooting pc/submodule-helper-foreach Stefan Beller
                   ` (4 preceding siblings ...)
  2018-05-03  0:53 ` [PATCH 5/5] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
@ 2018-05-09  0:29 ` Stefan Beller
  2018-05-09  0:29   ` [PATCH 1/4] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
                     ` (4 more replies)
  5 siblings, 5 replies; 200+ results
From: Stefan Beller @ 2018-05-09  0:29 UTC (permalink / raw)
  To: sbeller; +Cc: christian.couder, git, gitster, jonathantanmy, pc44800

v2:
* rebased onto origin/master
* dropped leftover "toplevel" variable from experimentation
* reworded the commit message for the first patch extensively
* dropped the third patch
* see "branch-diff" below.

v1:
The "What's cooking" email carried this series for some time now:
> * pc/submodule-helper-foreach (2018-02-02) 5 commits
>  - submodule: port submodule subcommand 'foreach' from shell to C
> - submodule foreach: document variable '$displaypath'
>  - submodule foreach: clarify the '$toplevel' variable documentation
>  - submodule foreach: document '$sm_path' instead of '$path'
>  - submodule foreach: correct '$path' in nested submodules from a subdirectory
> 
>  Expecting a response to review comments
>  e.g. cf. <20180206150044.1bffbb573c088d38c8e44bf5@google.com>

I reworded the commit message of the first patch and nearly confused
myself again, as "toplevel" doesn't refer to the "topmost" superproject,
just the direct superproject of the submodule.

However I think the code of the first patch is correct as originally presented.
Just the wording of the commit message was changed to explain the reasoning
more extensively.

With this series, we get
* keep the invariant of $toplevel + $path to be an absolute path that is
  correctly pointing at the submodule. "git -C $toplevel config" + $name
  allowing to ask configuration of the submodule.  
* $displaypath will have the relative path form $pwd to the submodule root.
* better documentation.

In later patches we could add $topmost, that points at the superproject
in which the command was started from, and $path_from_topmost, that would
be the relative path from $topmost to the submodule, potentially skipping
intermediate superprojects.

Thanks,
Stefan

Prathamesh Chavan (4):
  submodule foreach: correct '$path' in nested submodules from a
    subdirectory
  submodule foreach: document '$sm_path' instead of '$path'
  submodule foreach: document variable '$displaypath'
  submodule: port submodule subcommand 'foreach' from shell to C

 Documentation/git-submodule.txt |  15 ++--
 builtin/submodule--helper.c     | 144 ++++++++++++++++++++++++++++++++
 git-submodule.sh                |  40 +--------
 t/t7407-submodule-foreach.sh    |  38 ++++++++-
 4 files changed, 190 insertions(+), 47 deletions(-)

-- 
2.17.0.255.g8bfb7c0704

1:  0c5f405db24 ! 1:  85f91b48379 submodule foreach: correct '$path' in nested submodules from a subdirectory
    @@ -12,45 +12,38 @@
         submodule path inside the submodule. This value is of little use and is
         hard to document.
     
    -    There are three different possible solutions that have more value:
    -    (a) The path value is documented as the path from the toplevel of the
    -        superproject to the mount point of the submodule. If 'the' refers to
    -        the superproject holding this submodule ('sub' holding 'nested'),
    -        the path would be expected to be path='nested'.
    -    (b) In case 'the' superproject is referring to the toplevel, which
    -        is the superproject in which the original command was invoked,
    -        then path is expected to be path='sub/nested'.
    -    (c) The documentation explains $path as [...] "relative to the
    -        superproject", following 091a6eb0fe (submodule: drop the
    -        top-level requirement, 2013-06-16), such that the nested submodule
    -        would be expected as path='../sub/nested', when "the" superproject
    -        is the superproject, where the command was run from
    -    (d) or the value of path='nested' is expected if we take the
    -        intermediate superproject into account. [This is the same as
    -        (a); it highlights that the documentation is not clear, but
    -        technically correct if we were to revert 091a6eb0fe.]
    +    Also, in git-submodule.txt, $path is documented to be the "name of the
    +    submodule directory relative to the superproject", but "the
    +    superproject" is ambiguous.
     
    -    The behavior for (c) was introduced in 091a6eb0fe (submodule: drop the
    -    top-level requirement, 2013-06-16) the intent for $path seemed to be
    -    relative to $cwd to the submodule worktree, but that did not work for
    -    nested submodules, as the intermittent submodules were not included in
    -    the path.
    +    To resolve both these issues, we could:
    +    (a) Change "the superproject" to "its immediate superproject", so
    +        $path would be "nested" instead of "../nested".
    +    (b) Change "the superproject" to "the superproject the original
    +        command was run from", so $path would be "sub/nested" instead of
    +        "../nested".
    +    (c) Change "the superproject" to "the directory the original command
    +        was run from", so $path would be "../sub/nested" instead of
    +        "../nested".
     
    -    If we were to fix the meaning of the $path using (a), (d) such that "path"
    -    is "the path from the intermediate superproject to the mount point of the
    -    submodule", we would break any existing submodule user that runs foreach
    -    from non-root of the superproject as the non-nested submodule
    -    '../sub' would change its path to 'sub'.
    +    The behavior for (c) was attempted to be introduced in 091a6eb0fe
    +    (submodule: drop the top-level requirement, 2013-06-16) with the intent
    +    for $path to be relative from $pwd to the submodule worktree, but that
    +    did not work for nested submodules, as the intermittent submodules
    +    were not included in the path.
     
    -    If we were to fix the meaning of $path using (b) such that "path"
    -    is "the path from the topmost superproject to the mount point of the
    -    submodule", then we would break any user that uses nested submodules
    -    (even from the root directory) as the 'nested' would become 'sub/nested'.
    +    If we were to fix the meaning of the $path using (a), we would break
    +    any existing submodule user that runs foreach from non-root of the
    +    superproject as the non-nested submodule '../sub' would change its
    +    path to 'sub'.
     
    -    If we were to fix the meaning of $path using (c) such that "path"
    -    is "the display path from where the original command was invoked to the
    -    submodule", then we would break users that rely on the assumption that
    -    "$toplevel / $path" is the absolute path of the submodule.
    +    If we were to fix the meaning of $path using (b), then we would break
    +    any user that uses nested submodules (even from the root directory)
    +    as the 'nested' would become 'sub/nested'.
    +
    +    If we were to fix the meaning of $path using (c), then we would break
    +    the same users as in (b) as 'nested' would become 'sub/nested' from
    +    the root directory of the superproject.
     
         All groups can be found in the wild.  The author has no data if one group
         outweighs the other by large margin, and offending each one seems equally
    @@ -60,13 +53,6 @@
         the feedback loop is short and the changed behavior can be adapted to
         quickly unlike some automation that can break silently.
     
    -    Another argument in favor of (a) is the consistency of the variables
    -    provided, "$toplevel / $path" gives the absolute path of the submodule,
    -    with 'toplevel' (both the variable as well as the documentation) referring
    -    to the immediate superproject of the submodule.
    -
    -    Documentation of the variable is adjusted in a follow-up patch.
    -
         Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
         Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
         Signed-off-by: Stefan Beller <sbeller@google.com>
2:  ae5e280a8d3 ! 2:  3a2cb4e0b01 submodule foreach: document '$sm_path' instead of '$path'
    @@ -28,9 +28,10 @@
     -	$path is the name of the submodule directory relative to the
     -	superproject, $sha1 is the commit as recorded in the superproject,
     -	and $toplevel is the absolute path to the top-level of the superproject.
    -+	$sm_path is the path of the submodule as recorded in the superproject,
    -+	$sha1 is the commit as recorded in the superproject, and
    -+	$toplevel is the absolute path to the top-level of the superproject.
    ++	$sm_path is the path of the submodule as recorded in the immediate
    ++	superproject, $sha1 is the commit as recorded in the immediate
    ++	superproject, and $toplevel is the absolute path to the top-level
    ++	of the immediate superproject.
     +	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
     +	variable is now a deprecated synonym of '$sm_path' variable.
      	Any submodules defined in the superproject but not checked out are
3:  23a5f04f846 < -:  ----------- submodule foreach: clarify the '$toplevel' variable documentation
4:  3efed1cdb80 ! 3:  3f4686e8395 submodule foreach: document variable '$displaypath'
    @@ -22,12 +22,14 @@
     +	The command has access to the variables $name, $sm_path, $displaypath,
     +	$sha1 and $toplevel:
      	$name is the name of the relevant submodule section in `.gitmodules`,
    - 	$sm_path is the path of the submodule as recorded in the superproject,
    -+	$displaypath contains the relative path from the current working
    -+	directory to the submodules root directory,
    - 	$sha1 is the commit as recorded in the superproject, and
    - 	$toplevel is the absolute path to its direct superproject, such that
    - 	$toplevel/$sm_path is the absolute path of the submodule.
    + 	$sm_path is the path of the submodule as recorded in the immediate
    +-	superproject, $sha1 is the commit as recorded in the immediate
    ++	superproject, $displaypath contains the relative path from the
    ++	current working directory to the submodules root directory,
    ++	$sha1 is the commit as recorded in the immediate
    + 	superproject, and $toplevel is the absolute path to the top-level
    + 	of the immediate superproject.
    + 	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
     
     diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
     --- a/t/t7407-submodule-foreach.sh
5:  2b17d6891b3 ! 4:  0c677680928 submodule: port submodule subcommand 'foreach' from shell to C
    @@ -23,7 +23,6 @@
     +	int argc;
     +	const char **argv;
     +	const char *prefix;
    -+	char *toplevel;
     +	int quiet;
     +	int recursive;
     +};
    @@ -42,7 +41,7 @@
     +
     +	displaypath = get_submodule_displaypath(path, info->prefix);
     +
    -+	sub = submodule_from_path(&null_oid, path);
    ++	sub = submodule_from_path(the_repository, &null_oid, path);
     +
     +	if (!sub)
     +		die(_("No url found for submodule path '%s' in .gitmodules"),
    @@ -69,6 +68,7 @@
     +	*/
     +	if (info->argc == 1) {
     +		char *toplevel = xgetcwd();
    ++		struct strbuf sb = STRBUF_INIT;
     +
     +		argv_array_pushf(&cp.env_array, "name=%s", sub->name);
     +		argv_array_pushf(&cp.env_array, "sm_path=%s", path);
    @@ -86,9 +86,10 @@
     +		* existing PATH variable. Hence, to avoid that, we expose
     +		* path via the args argv_array and not via env_array.
     +		*/
    ++		sq_quote_buf(&sb, path);
     +		argv_array_pushf(&cp.args, "path=%s; %s",
    -+				path, info->argv[0]);
    -+
    ++				 sb.buf, info->argv[0]);
    ++		strbuf_release(&sb);
     +		free(toplevel);
     +	} else {
     +		argv_array_pushv(&cp.args, info->argv);
    @@ -116,9 +117,6 @@
     +		if (info->quiet)
     +			argv_array_push(&cpr.args, "--quiet");
     +
    -+		if (info->toplevel)
    -+			argv_array_pushf(&cpr.args, "--toplevel=%s", info->toplevel);
    -+
     +		argv_array_pushv(&cpr.args, info->argv);
     +
     +		if (run_command(&cpr))
    @@ -141,8 +139,6 @@
     +		OPT__QUIET(&info.quiet, N_("Suppress output of entering each submodule command")),
     +		OPT_BOOL(0, "recursive", &info.recursive,
     +			 N_("Recurse into nested submodules")),
    -+		OPT_STRING(0, "toplevel", &info.toplevel, N_("path"),
    -+			   N_("path from the top level of the invocation")),
     +		OPT_END()
     +	};
     +

^ permalink raw reply	[relevance 6%]

* [PATCH 1/4] submodule foreach: correct '$path' in nested submodules from a subdirectory
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
@ 2018-05-09  0:29   ` Stefan Beller
  2018-05-09  0:29   ` [PATCH 2/4] submodule foreach: document '$sm_path' instead of '$path' Stefan Beller
                     ` (3 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-09  0:29 UTC (permalink / raw)
  To: sbeller; +Cc: christian.couder, git, gitster, jonathantanmy, pc44800

From: Prathamesh Chavan <pc44800@gmail.com>

When running 'git submodule foreach --recursive' from a subdirectory of
your repository, nested submodules get a bogus value for $path:
For a submodule 'sub' that contains a nested submodule 'nested',
running 'git -C dir submodule foreach echo $path' from the root of the
superproject would report path='../nested' for the nested submodule.
The first part '../' is derived from the logic computing the relative
path from $pwd to the root of the superproject. The second part is the
submodule path inside the submodule. This value is of little use and is
hard to document.

Also, in git-submodule.txt, $path is documented to be the "name of the
submodule directory relative to the superproject", but "the
superproject" is ambiguous.

To resolve both these issues, we could:
(a) Change "the superproject" to "its immediate superproject", so
    $path would be "nested" instead of "../nested".
(b) Change "the superproject" to "the superproject the original
    command was run from", so $path would be "sub/nested" instead of
    "../nested".
(c) Change "the superproject" to "the directory the original command
    was run from", so $path would be "../sub/nested" instead of
    "../nested".

The behavior for (c) was attempted to be introduced in 091a6eb0fe
(submodule: drop the top-level requirement, 2013-06-16) with the intent
for $path to be relative from $pwd to the submodule worktree, but that
did not work for nested submodules, as the intermittent submodules
were not included in the path.

If we were to fix the meaning of the $path using (a), we would break
any existing submodule user that runs foreach from non-root of the
superproject as the non-nested submodule '../sub' would change its
path to 'sub'.

If we were to fix the meaning of $path using (b), then we would break
any user that uses nested submodules (even from the root directory)
as the 'nested' would become 'sub/nested'.

If we were to fix the meaning of $path using (c), then we would break
the same users as in (b) as 'nested' would become 'sub/nested' from
the root directory of the superproject.

All groups can be found in the wild.  The author has no data if one group
outweighs the other by large margin, and offending each one seems equally
bad at first.  However in the authors imagination it is better to go with
(a) as running from a sub directory sounds like it is carried out by a
human rather than by some automation task.  With a human on the keyboard
the feedback loop is short and the changed behavior can be adapted to
quickly unlike some automation that can break silently.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh             |  1 -
 t/t7407-submodule-foreach.sh | 36 ++++++++++++++++++++++++++++++++++--
 2 files changed, 34 insertions(+), 3 deletions(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 24914963ca2..331d71c908b 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -345,7 +345,6 @@ cmd_foreach()
 				prefix="$prefix$sm_path/"
 				sanitize_submodule_env
 				cd "$sm_path" &&
-				sm_path=$(git submodule--helper relative-path "$sm_path" "$wt_prefix") &&
 				# we make $path available to scripts ...
 				path=$sm_path &&
 				if test $# -eq 1
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 6ba5daf42ee..5144cc6926b 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -82,9 +82,9 @@ test_expect_success 'test basic "submodule foreach" usage' '
 
 cat >expect <<EOF
 Entering '../sub1'
-$pwd/clone-foo1-../sub1-$sub1sha1
+$pwd/clone-foo1-sub1-$sub1sha1
 Entering '../sub3'
-$pwd/clone-foo3-../sub3-$sub3sha1
+$pwd/clone-foo3-sub3-$sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach" from subdirectory' '
@@ -196,6 +196,38 @@ test_expect_success 'test messages from "foreach --recursive" from subdirectory'
 	) &&
 	test_i18ncmp expect actual
 '
+sub1sha1=$(cd clone2/sub1 && git rev-parse HEAD)
+sub2sha1=$(cd clone2/sub2 && git rev-parse HEAD)
+sub3sha1=$(cd clone2/sub3 && git rev-parse HEAD)
+nested1sha1=$(cd clone2/nested1 && git rev-parse HEAD)
+nested2sha1=$(cd clone2/nested1/nested2 && git rev-parse HEAD)
+nested3sha1=$(cd clone2/nested1/nested2/nested3 && git rev-parse HEAD)
+submodulesha1=$(cd clone2/nested1/nested2/nested3/submodule && git rev-parse HEAD)
+
+cat >expect <<EOF
+Entering '../nested1'
+toplevel: $pwd/clone2 name: nested1 path: nested1 hash: $nested1sha1
+Entering '../nested1/nested2'
+toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 hash: $nested2sha1
+Entering '../nested1/nested2/nested3'
+toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 hash: $nested3sha1
+Entering '../nested1/nested2/nested3/submodule'
+toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule hash: $submodulesha1
+Entering '../sub1'
+toplevel: $pwd/clone2 name: foo1 path: sub1 hash: $sub1sha1
+Entering '../sub2'
+toplevel: $pwd/clone2 name: foo2 path: sub2 hash: $sub2sha1
+Entering '../sub3'
+toplevel: $pwd/clone2 name: foo3 path: sub3 hash: $sub3sha1
+EOF
+
+test_expect_success 'test "submodule foreach --recursive" from subdirectory' '
+	(
+		cd clone2/untracked &&
+		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path hash: \$sha1" >../../actual
+	) &&
+	test_i18ncmp expect actual
+'
 
 cat > expect <<EOF
 nested1-nested1
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 22%]

* [PATCH 2/4] submodule foreach: document '$sm_path' instead of '$path'
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
  2018-05-09  0:29   ` [PATCH 1/4] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
@ 2018-05-09  0:29   ` Stefan Beller
  2018-05-09  0:29   ` [PATCH 3/4] submodule foreach: document variable '$displaypath' Stefan Beller
                     ` (2 subsequent siblings)
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-09  0:29 UTC (permalink / raw)
  To: sbeller; +Cc: christian.couder, git, gitster, jonathantanmy, pc44800

From: Prathamesh Chavan <pc44800@gmail.com>

As using a variable '$path' may be harmful to users due to
capitalization issues, see 64394e3ae9 (git-submodule.sh: Don't
use $path variable in eval_gettext string, 2012-04-17). Adjust
the documentation to advocate for using $sm_path,  which contains
the same value. We still make the 'path' variable available and
document it as a deprecated synonym of 'sm_path'.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-submodule.txt | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 630999f41a9..066c7b6c18e 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -183,12 +183,15 @@ information too.
 
 foreach [--recursive] <command>::
 	Evaluates an arbitrary shell command in each checked out submodule.
-	The command has access to the variables $name, $path, $sha1 and
+	The command has access to the variables $name, $sm_path, $sha1 and
 	$toplevel:
 	$name is the name of the relevant submodule section in `.gitmodules`,
-	$path is the name of the submodule directory relative to the
-	superproject, $sha1 is the commit as recorded in the superproject,
-	and $toplevel is the absolute path to the top-level of the superproject.
+	$sm_path is the path of the submodule as recorded in the immediate
+	superproject, $sha1 is the commit as recorded in the immediate
+	superproject, and $toplevel is the absolute path to the top-level
+	of the immediate superproject.
+	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
+	variable is now a deprecated synonym of '$sm_path' variable.
 	Any submodules defined in the superproject but not checked out are
 	ignored by this command. Unless given `--quiet`, foreach prints the name
 	of each submodule before evaluating the command.
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 21%]

* [PATCH 3/4] submodule foreach: document variable '$displaypath'
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
  2018-05-09  0:29   ` [PATCH 1/4] submodule foreach: correct '$path' in nested submodules from a subdirectory Stefan Beller
  2018-05-09  0:29   ` [PATCH 2/4] submodule foreach: document '$sm_path' instead of '$path' Stefan Beller
@ 2018-05-09  0:29   ` Stefan Beller
  2018-05-09  0:29   ` [PATCH 4/4] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
  2018-05-09 17:13   ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Jonathan Tan
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-09  0:29 UTC (permalink / raw)
  To: sbeller; +Cc: christian.couder, git, gitster, jonathantanmy, pc44800

From: Prathamesh Chavan <pc44800@gmail.com>

It was observed that the variable '$displaypath' was accessible but
undocumented. Hence, document it.

Discussed-with: Ramsay Jones <ramsay@ramsayjones.plus.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 Documentation/git-submodule.txt |  8 +++++---
 t/t7407-submodule-foreach.sh    | 22 +++++++++++-----------
 2 files changed, 16 insertions(+), 14 deletions(-)

diff --git a/Documentation/git-submodule.txt b/Documentation/git-submodule.txt
index 066c7b6c18e..500dfdafd16 100644
--- a/Documentation/git-submodule.txt
+++ b/Documentation/git-submodule.txt
@@ -183,11 +183,13 @@ information too.
 
 foreach [--recursive] <command>::
 	Evaluates an arbitrary shell command in each checked out submodule.
-	The command has access to the variables $name, $sm_path, $sha1 and
-	$toplevel:
+	The command has access to the variables $name, $sm_path, $displaypath,
+	$sha1 and $toplevel:
 	$name is the name of the relevant submodule section in `.gitmodules`,
 	$sm_path is the path of the submodule as recorded in the immediate
-	superproject, $sha1 is the commit as recorded in the immediate
+	superproject, $displaypath contains the relative path from the
+	current working directory to the submodules root directory,
+	$sha1 is the commit as recorded in the immediate
 	superproject, and $toplevel is the absolute path to the top-level
 	of the immediate superproject.
 	Note that to avoid conflicts with '$PATH' on Windows, the '$path'
diff --git a/t/t7407-submodule-foreach.sh b/t/t7407-submodule-foreach.sh
index 5144cc6926b..77729ac4aa1 100755
--- a/t/t7407-submodule-foreach.sh
+++ b/t/t7407-submodule-foreach.sh
@@ -82,16 +82,16 @@ test_expect_success 'test basic "submodule foreach" usage' '
 
 cat >expect <<EOF
 Entering '../sub1'
-$pwd/clone-foo1-sub1-$sub1sha1
+$pwd/clone-foo1-sub1-../sub1-$sub1sha1
 Entering '../sub3'
-$pwd/clone-foo3-sub3-$sub3sha1
+$pwd/clone-foo3-sub3-../sub3-$sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach" from subdirectory' '
 	mkdir clone/sub &&
 	(
 		cd clone/sub &&
-		git submodule foreach "echo \$toplevel-\$name-\$sm_path-\$sha1" >../../actual
+		git submodule foreach "echo \$toplevel-\$name-\$sm_path-\$displaypath-\$sha1" >../../actual
 	) &&
 	test_i18ncmp expect actual
 '
@@ -206,25 +206,25 @@ submodulesha1=$(cd clone2/nested1/nested2/nested3/submodule && git rev-parse HEA
 
 cat >expect <<EOF
 Entering '../nested1'
-toplevel: $pwd/clone2 name: nested1 path: nested1 hash: $nested1sha1
+toplevel: $pwd/clone2 name: nested1 path: nested1 displaypath: ../nested1 hash: $nested1sha1
 Entering '../nested1/nested2'
-toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 hash: $nested2sha1
+toplevel: $pwd/clone2/nested1 name: nested2 path: nested2 displaypath: ../nested1/nested2 hash: $nested2sha1
 Entering '../nested1/nested2/nested3'
-toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 hash: $nested3sha1
+toplevel: $pwd/clone2/nested1/nested2 name: nested3 path: nested3 displaypath: ../nested1/nested2/nested3 hash: $nested3sha1
 Entering '../nested1/nested2/nested3/submodule'
-toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule hash: $submodulesha1
+toplevel: $pwd/clone2/nested1/nested2/nested3 name: submodule path: submodule displaypath: ../nested1/nested2/nested3/submodule hash: $submodulesha1
 Entering '../sub1'
-toplevel: $pwd/clone2 name: foo1 path: sub1 hash: $sub1sha1
+toplevel: $pwd/clone2 name: foo1 path: sub1 displaypath: ../sub1 hash: $sub1sha1
 Entering '../sub2'
-toplevel: $pwd/clone2 name: foo2 path: sub2 hash: $sub2sha1
+toplevel: $pwd/clone2 name: foo2 path: sub2 displaypath: ../sub2 hash: $sub2sha1
 Entering '../sub3'
-toplevel: $pwd/clone2 name: foo3 path: sub3 hash: $sub3sha1
+toplevel: $pwd/clone2 name: foo3 path: sub3 displaypath: ../sub3 hash: $sub3sha1
 EOF
 
 test_expect_success 'test "submodule foreach --recursive" from subdirectory' '
 	(
 		cd clone2/untracked &&
-		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path hash: \$sha1" >../../actual
+		git submodule foreach --recursive "echo toplevel: \$toplevel name: \$name path: \$sm_path displaypath: \$displaypath hash: \$sha1" >../../actual
 	) &&
 	test_i18ncmp expect actual
 '
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 28%]

* [PATCH 4/4] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
                     ` (2 preceding siblings ...)
  2018-05-09  0:29   ` [PATCH 3/4] submodule foreach: document variable '$displaypath' Stefan Beller
@ 2018-05-09  0:29   ` Stefan Beller
  2018-05-10  6:37     ` Junio C Hamano
  2018-05-09 17:13   ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Jonathan Tan
  4 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-09  0:29 UTC (permalink / raw)
  To: sbeller; +Cc: christian.couder, git, gitster, jonathantanmy, pc44800

From: Prathamesh Chavan <pc44800@gmail.com>

This aims to make git-submodule foreach a builtin. 'foreach' is ported to
the submodule--helper, and submodule--helper is called from
git-submodule.sh.

Helped-by: Brandon Williams <bmwill@google.com>
Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/submodule--helper.c | 144 ++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  39 +---------
 2 files changed, 145 insertions(+), 38 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index c2403a915ff..4b0b773c06b 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -439,6 +439,149 @@ static void for_each_listed_submodule(const struct module_list *list,
 		fn(list->entries[i], cb_data);
 }
 
+struct cb_foreach {
+	int argc;
+	const char **argv;
+	const char *prefix;
+	int quiet;
+	int recursive;
+};
+#define CB_FOREACH_INIT { 0 }
+
+static void runcommand_in_submodule_cb(const struct cache_entry *list_item,
+				       void *cb_data)
+{
+	struct cb_foreach *info = cb_data;
+	const char *path = list_item->name;
+	const struct object_id *ce_oid = &list_item->oid;
+
+	const struct submodule *sub;
+	struct child_process cp = CHILD_PROCESS_INIT;
+	char *displaypath;
+
+	displaypath = get_submodule_displaypath(path, info->prefix);
+
+	sub = submodule_from_path(the_repository, &null_oid, path);
+
+	if (!sub)
+		die(_("No url found for submodule path '%s' in .gitmodules"),
+			displaypath);
+
+	if (!is_submodule_populated_gently(path, NULL))
+		goto cleanup;
+
+	prepare_submodule_repo_env(&cp.env_array);
+
+	/*
+	* For the purpose of executing <command> in the submodule,
+	* separate shell is used for the purpose of running the
+	* child process.
+	*/
+	cp.use_shell = 1;
+	cp.dir = path;
+
+	/*
+	* NEEDSWORK: the command currently has access to the variables $name,
+	* $sm_path, $displaypath, $sha1 and $toplevel only when the command
+	* contains a single argument. This is done for maintaining a faithful
+	* translation from shell script.
+	*/
+	if (info->argc == 1) {
+		char *toplevel = xgetcwd();
+		struct strbuf sb = STRBUF_INIT;
+
+		argv_array_pushf(&cp.env_array, "name=%s", sub->name);
+		argv_array_pushf(&cp.env_array, "sm_path=%s", path);
+		argv_array_pushf(&cp.env_array, "displaypath=%s", displaypath);
+		argv_array_pushf(&cp.env_array, "sha1=%s",
+				oid_to_hex(ce_oid));
+		argv_array_pushf(&cp.env_array, "toplevel=%s", toplevel);
+
+		/*
+		* Since the path variable was accessible from the script
+		* before porting, it is also made available after porting.
+		* The environment variable "PATH" has a very special purpose
+		* on windows. And since environment variables are
+		* case-insensitive in windows, it interferes with the
+		* existing PATH variable. Hence, to avoid that, we expose
+		* path via the args argv_array and not via env_array.
+		*/
+		sq_quote_buf(&sb, path);
+		argv_array_pushf(&cp.args, "path=%s; %s",
+				 sb.buf, info->argv[0]);
+		strbuf_release(&sb);
+		free(toplevel);
+	} else {
+		argv_array_pushv(&cp.args, info->argv);
+	}
+
+	if (!info->quiet)
+		printf(_("Entering '%s'\n"), displaypath);
+
+	if (info->argv[0] && run_command(&cp))
+		die(_("run_command returned non-zero status for %s\n."),
+			displaypath);
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = path;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", NULL);
+		argv_array_pushf(&cpr.args, "%s/", displaypath);
+		argv_array_pushl(&cpr.args, "submodule--helper", "foreach", "--recursive",
+				NULL);
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		argv_array_pushv(&cpr.args, info->argv);
+
+		if (run_command(&cpr))
+			die(_("run_command returned non-zero status while"
+				"recursing in the nested submodules of %s\n."),
+				displaypath);
+	}
+
+cleanup:
+	free(displaypath);
+}
+
+static int module_foreach(int argc, const char **argv, const char *prefix)
+{
+	struct cb_foreach info = CB_FOREACH_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+
+	struct option module_foreach_options[] = {
+		OPT__QUIET(&info.quiet, N_("Suppress output of entering each submodule command")),
+		OPT_BOOL(0, "recursive", &info.recursive,
+			 N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule--helper foreach [--quiet] [--recursive] <command>"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_foreach_options,
+			     git_submodule_helper_usage, PARSE_OPT_KEEP_UNKNOWN);
+
+	if (module_list_compute(0, NULL, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.argc = argc;
+	info.argv = argv;
+	info.prefix = prefix;
+
+	for_each_listed_submodule(&list, runcommand_in_submodule_cb, &info);
+
+	return 0;
+}
+
 struct init_cb {
 	const char *prefix;
 	unsigned int flags;
@@ -1841,6 +1984,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"foreach", module_foreach, SUPPORT_SUPER_PREFIX},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"print-default-remote", print_default_remote, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index 331d71c908b..cba585f0754 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -323,44 +323,7 @@ cmd_foreach()
 		shift
 	done
 
-	toplevel=$(pwd)
-
-	# dup stdin so that it can be restored when running the external
-	# command in the subshell (and a recursive call to this function)
-	exec 3<&0
-
-	{
-		git submodule--helper list --prefix "$wt_prefix" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		if test -e "$sm_path"/.git
-		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Entering '\$displaypath'")"
-			name=$(git submodule--helper name "$sm_path")
-			(
-				prefix="$prefix$sm_path/"
-				sanitize_submodule_env
-				cd "$sm_path" &&
-				# we make $path available to scripts ...
-				path=$sm_path &&
-				if test $# -eq 1
-				then
-					eval "$1"
-				else
-					"$@"
-				fi &&
-				if test -n "$recursive"
-				then
-					cmd_foreach "--recursive" "$@"
-				fi
-			) <&3 3<&- ||
-			die "$(eval_gettext "Stopping at '\$displaypath'; script returned non-zero status.")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper foreach ${GIT_QUIET:+--quiet} ${recursive:+--recursive} "$@"
 }
 
 #
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 20%]

* Re: [PATCH 1/2] gitk: show part of submodule log instead of empty pane when listing trees
  2018-05-08 17:07   ` [PATCH 1/2] gitk: show part of submodule log instead of empty pane when listing trees Stefan Beller
@ 2018-05-09  9:01     ` Alex Riesen
  0 siblings, 0 replies; 200+ results
From: Alex Riesen @ 2018-05-09  9:01 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Paul Mackerras, Git Mailing List, Junio C Hamano

Stefan Beller, Tue, May 08, 2018 19:07:29 +0200:
> On Tue, May 8, 2018 at 5:22 AM, Alex Riesen
> <alexander.riesen@cetitec.com> wrote:
> > Currently, the submodules either are not shown at all (if listing a
> > committed tree) or a Tcl error appears (when clicking on a submodule
> > from the index list).
> 
> I do not understand where this appears, yet.
> Where do I have to click to see the effects of this patch?

Er. I meant to say the file list panel (bottom right panel). Sorry,
didn't come out clear. I'll reword the commit message next time.

> > @@ -7659,21 +7660,42 @@ proc showfile {f} {
> >      global ctext_file_names ctext_file_lines
> >      global ctext commentend
> >
> > +    set submodlog "git\\ log\\ --format='%h\\ %aN:\\ %s'\\ -100"
> 
> Do we want to respect the config option diff.submodule here?

Probably not. It is already done when the file list panel is in "Patch" mode.
The "Tree" mode of the panel shows the files in full, so the submodules should
be shown similarly: in a format resembling their full (referenced) contents.

> The -100 is chosen rather arbitrarily. Ideally we'd only walk to the
> previous entry?

Yes, the limit is indeed arbitrary. I'm reluctant of listing full history,
though: it might take too long a while (and does, in my line of work). Maybe
an option in the settings? Or some kind of a more natural limit (for 1 second?
Until the end of panel?)

> > -       if {[catch {set bf [open $f r]} err]} {
> > -           puts "oops, can't read $f: $err"
> > -           return
> > +       if {[file isdirectory $f]} {
> > +           # a submodule
> > +           if {[catch {set bf [open "| sh -c cd\\ \"$f\"&&$submodlog" r]} err]} {
> 
> Can we have $submodlog use the "git -C <path> command"
> option, then we could save the "cd &&" part, which might even
> save us from spawning a shell?

That's because I forgot about that option. Of course, I'll fix this.
Also need a shellquote for the path.

Thanks!
Alex

---
Diese E-Mail wurde von Avast Antivirus-Software auf Viren geprüft.
https://www.avast.com/antivirus


^ permalink raw reply	[relevance 6%]

* Re: [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach
  2018-05-09  0:29 ` [PATCHv2 0/4] Rebooting pc/submodule-helper-foreach Stefan Beller
                     ` (3 preceding siblings ...)
  2018-05-09  0:29   ` [PATCH 4/4] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
@ 2018-05-09 17:13   ` Jonathan Tan
  4 siblings, 0 replies; 200+ results
From: Jonathan Tan @ 2018-05-09 17:13 UTC (permalink / raw)
  To: Stefan Beller; +Cc: christian.couder, git, gitster, pc44800

On Tue,  8 May 2018 17:29:48 -0700
Stefan Beller <sbeller@google.com> wrote:

> v2:
> * rebased onto origin/master
> * dropped leftover "toplevel" variable from experimentation
> * reworded the commit message for the first patch extensively
> * dropped the third patch
> * see "branch-diff" below.

Patches 1-3 look good to me.

I also can't see anything wrong with patch 4, but I am not an expert at
shell and how we call it from C, so a review from another reviewer would
be appreciated.

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] repository: fix free problem with repo_clear(the_repository)
      [irrelevant] <20180509170409.13666-1-pclouds@gmail.com>
@ 2018-05-09 17:50 ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-09 17:50 UTC (permalink / raw)
  To: Nguyễn Thái Ngọc Duy; +Cc: git, Junio C Hamano

On Wed, May 9, 2018 at 10:04 AM, Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> the_repository is special. One of the special things about it is that
> it does not allocate a new index_state object like submodules but
> points to the global the_index variable instead. As a global variable,
> the_index cannot be free()'d.

ok. That is the situation we're in.

>
> Add an exception for this in repo_clear(). In the future perhaps we
> would be able to allocate the_repository's index on heap too. Then we
> can remove revert this.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  I was trying to test the new parsed_object_pool_clear() and found this.

So this would go with the latest sb/object-store-alloc ?

My impression was that we never call repo_clear() on
the_repository, which would allow us to special case
the_repository further just as I did in v2 of that series[1] by
having static allocations for certain objects in case of \
the_repository.

[1] https://public-inbox.org/git/20180501213403.14643-14-sbeller@google.com/

We could just deal with all the exceptions, but that makes repo_clear
ugly IMHO.

I would rather special case the_repository even more instead
of having it allocate all its things on the heap. (However we rather
want to profile it and argue with data....)

>  repository.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/repository.c b/repository.c
> index a4848c1bd0..f44733524a 100644
> --- a/repository.c
> +++ b/repository.c
> @@ -238,7 +238,9 @@ void repo_clear(struct repository *repo)
>
>         if (repo->index) {
>                 discard_index(repo->index);
> -               FREE_AND_NULL(repo->index);
> +               if (repo->index != &the_index)
> +                       free(repo->index);
> +               repo->index = NULL;

So after this we have a "dangling" the_index.
It is not really dangling, but it is not part of the_repository any more
and many places still use the_index, it might make up for interesting
bugs?

What is your use case of repo_clear(the_repository)?

Thanks,
Stefan

^ permalink raw reply	[relevance 2%]

* Re: [PATCH v2 0/2] gitk: improve handling of submodules in the file list panel
      [irrelevant] ` <cover.1525868167.git.raa.lkml@gmail.com>
@ 2018-05-09 18:52   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-09 18:52 UTC (permalink / raw)
  To: Alex Riesen
  Cc: Paul Mackerras, Git Mailing List, Junio C Hamano, Alex Riesen,
	Bert Wesarg

On Wed, May 9, 2018 at 5:35 AM, Alex Riesen
<alexander.riesen@cetitec.com> wrote:
> From: Alex Riesen <raa.lkml@gmail.com>
>
> Currently, the submodule entries in the file list panel are mostly ignored.
> This series attempts to improve the situation by showing part of submodule
> history when focusing it in the file list panel and by adding a menu element
> to start gitk in the submodule (similar to git gui).
>
> This iteration does not address the behaviour of file list panel in tree mode
> when gitk is started from a subdirectory (gitk strictly limits the file
> listing to the files in that repository, without a way out).
> I would like to hear some more opinions regarding changing its behaviour to
> always list the full tree.
>
> Alex Riesen (2):
>   gitk: show part of submodule log instead of empty pane when listing
>     trees
>   gitk: add an option to run gitk on an item in the file list

both patches look ok, to my untrained eye.

^ permalink raw reply	[relevance 5%]

* [PATCH v4 01/13] repository: introduce parsed objects field
      [irrelevant]     ` <20180510004024.93974-1-sbeller@google.com>
@ 2018-05-10  0:40       ` " Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-10  0:40 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jamill, jonathantanmy, pclouds

Convert the existing global cache for parsed objects (obj_hash) into
repository-specific parsed object caches. Existing code that uses
obj_hash are modified to use the parsed object cache of
the_repository; future patches will use the parsed object caches of
other repositories.

Another future use case for a pool of objects is ease of memory management
in revision walking: If we can free the rev-list related memory early in
pack-objects (e.g. part of repack operation) then it could lower memory
pressure significantly when running on large repos. While this has been
discussed on the mailing list lately, this series doesn't implement this.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 object.c     | 63 +++++++++++++++++++++++++++++++++-------------------
 object.h     |  8 +++++++
 repository.c |  7 ++++++
 repository.h |  9 ++++++++
 4 files changed, 64 insertions(+), 23 deletions(-)

diff --git a/object.c b/object.c
index 5044d08e96c..f7c624a7ba6 100644
--- a/object.c
+++ b/object.c
@@ -8,17 +8,14 @@
 #include "object-store.h"
 #include "packfile.h"
 
-static struct object **obj_hash;
-static int nr_objs, obj_hash_size;
-
 unsigned int get_max_object_index(void)
 {
-	return obj_hash_size;
+	return the_repository->parsed_objects->obj_hash_size;
 }
 
 struct object *get_indexed_object(unsigned int idx)
 {
-	return obj_hash[idx];
+	return the_repository->parsed_objects->obj_hash[idx];
 }
 
 static const char *object_type_strings[] = {
@@ -90,15 +87,16 @@ struct object *lookup_object(const unsigned char *sha1)
 	unsigned int i, first;
 	struct object *obj;
 
-	if (!obj_hash)
+	if (!the_repository->parsed_objects->obj_hash)
 		return NULL;
 
-	first = i = hash_obj(sha1, obj_hash_size);
-	while ((obj = obj_hash[i]) != NULL) {
+	first = i = hash_obj(sha1,
+			     the_repository->parsed_objects->obj_hash_size);
+	while ((obj = the_repository->parsed_objects->obj_hash[i]) != NULL) {
 		if (!hashcmp(sha1, obj->oid.hash))
 			break;
 		i++;
-		if (i == obj_hash_size)
+		if (i == the_repository->parsed_objects->obj_hash_size)
 			i = 0;
 	}
 	if (obj && i != first) {
@@ -107,7 +105,8 @@ struct object *lookup_object(const unsigned char *sha1)
 		 * that we do not need to walk the hash table the next
 		 * time we look for it.
 		 */
-		SWAP(obj_hash[i], obj_hash[first]);
+		SWAP(the_repository->parsed_objects->obj_hash[i],
+		     the_repository->parsed_objects->obj_hash[first]);
 	}
 	return obj;
 }
@@ -124,19 +123,19 @@ static void grow_object_hash(void)
 	 * Note that this size must always be power-of-2 to match hash_obj
 	 * above.
 	 */
-	int new_hash_size = obj_hash_size < 32 ? 32 : 2 * obj_hash_size;
+	int new_hash_size = the_repository->parsed_objects->obj_hash_size < 32 ? 32 : 2 * the_repository->parsed_objects->obj_hash_size;
 	struct object **new_hash;
 
 	new_hash = xcalloc(new_hash_size, sizeof(struct object *));
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (!obj)
 			continue;
 		insert_obj_hash(obj, new_hash, new_hash_size);
 	}
-	free(obj_hash);
-	obj_hash = new_hash;
-	obj_hash_size = new_hash_size;
+	free(the_repository->parsed_objects->obj_hash);
+	the_repository->parsed_objects->obj_hash = new_hash;
+	the_repository->parsed_objects->obj_hash_size = new_hash_size;
 }
 
 void *create_object(const unsigned char *sha1, void *o)
@@ -147,11 +146,12 @@ void *create_object(const unsigned char *sha1, void *o)
 	obj->flags = 0;
 	hashcpy(obj->oid.hash, sha1);
 
-	if (obj_hash_size - 1 <= nr_objs * 2)
+	if (the_repository->parsed_objects->obj_hash_size - 1 <= the_repository->parsed_objects->nr_objs * 2)
 		grow_object_hash();
 
-	insert_obj_hash(obj, obj_hash, obj_hash_size);
-	nr_objs++;
+	insert_obj_hash(obj, the_repository->parsed_objects->obj_hash,
+			the_repository->parsed_objects->obj_hash_size);
+	the_repository->parsed_objects->nr_objs++;
 	return obj;
 }
 
@@ -431,8 +431,8 @@ void clear_object_flags(unsigned flags)
 {
 	int i;
 
-	for (i=0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i=0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj)
 			obj->flags &= ~flags;
 	}
@@ -442,13 +442,20 @@ void clear_commit_marks_all(unsigned int flags)
 {
 	int i;
 
-	for (i = 0; i < obj_hash_size; i++) {
-		struct object *obj = obj_hash[i];
+	for (i = 0; i < the_repository->parsed_objects->obj_hash_size; i++) {
+		struct object *obj = the_repository->parsed_objects->obj_hash[i];
 		if (obj && obj->type == OBJ_COMMIT)
 			obj->flags &= ~flags;
 	}
 }
 
+struct parsed_object_pool *parsed_object_pool_new(void)
+{
+	struct parsed_object_pool *o = xmalloc(sizeof(*o));
+	memset(o, 0, sizeof(*o));
+	return o;
+}
+
 struct raw_object_store *raw_object_store_new(void)
 {
 	struct raw_object_store *o = xmalloc(sizeof(*o));
@@ -488,3 +495,13 @@ void raw_object_store_clear(struct raw_object_store *o)
 	close_all_packs(o);
 	o->packed_git = NULL;
 }
+
+void parsed_object_pool_clear(struct parsed_object_pool *o)
+{
+	/*
+	 * TOOD free objects in o->obj_hash.
+	 *
+	 * As objects are allocated in slabs (see alloc.c), we do
+	 * not need to free each object, but each slab instead.
+	 */
+}
diff --git a/object.h b/object.h
index f13f85b2a94..cecda7da370 100644
--- a/object.h
+++ b/object.h
@@ -1,6 +1,14 @@
 #ifndef OBJECT_H
 #define OBJECT_H
 
+struct parsed_object_pool {
+	struct object **obj_hash;
+	int nr_objs, obj_hash_size;
+};
+
+struct parsed_object_pool *parsed_object_pool_new(void);
+void parsed_object_pool_clear(struct parsed_object_pool *o);
+
 struct object_list {
 	struct object *item;
 	struct object_list *next;
diff --git a/repository.c b/repository.c
index a4848c1bd05..c23404677eb 100644
--- a/repository.c
+++ b/repository.c
@@ -2,6 +2,7 @@
 #include "repository.h"
 #include "object-store.h"
 #include "config.h"
+#include "object.h"
 #include "submodule-config.h"
 
 /* The main repository */
@@ -14,6 +15,8 @@ void initialize_the_repository(void)
 
 	the_repo.index = &the_index;
 	the_repo.objects = raw_object_store_new();
+	the_repo.parsed_objects = parsed_object_pool_new();
+
 	repo_set_hash_algo(&the_repo, GIT_HASH_SHA1);
 }
 
@@ -143,6 +146,7 @@ static int repo_init(struct repository *repo,
 	memset(repo, 0, sizeof(*repo));
 
 	repo->objects = raw_object_store_new();
+	repo->parsed_objects = parsed_object_pool_new();
 
 	if (repo_init_gitdir(repo, gitdir))
 		goto error;
@@ -226,6 +230,9 @@ void repo_clear(struct repository *repo)
 	raw_object_store_clear(repo->objects);
 	FREE_AND_NULL(repo->objects);
 
+	parsed_object_pool_clear(repo->parsed_objects);
+	FREE_AND_NULL(repo->parsed_objects);
+
 	if (repo->config) {
 		git_configset_clear(repo->config);
 		FREE_AND_NULL(repo->config);
diff --git a/repository.h b/repository.h
index e6e00f541bd..6d199819905 100644
--- a/repository.h
+++ b/repository.h
@@ -26,6 +26,15 @@ struct repository {
 	 */
 	struct raw_object_store *objects;
 
+	/*
+	 * All objects in this repository that have been parsed. This structure
+	 * owns all objects it references, so users of "struct object *"
+	 * generally do not need to free them; instead, when a repository is no
+	 * longer used, call parsed_object_pool_clear() on this structure, which
+	 * is called by the repositories repo_clear on its desconstruction.
+	 */
+	struct parsed_object_pool *parsed_objects;
+
 	/* The store in which the refs are held. */
 	struct ref_store *refs;
 
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 4%]

* Re: [PATCH 4/4] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-09  0:29   ` [PATCH 4/4] submodule: port submodule subcommand 'foreach' from shell to C Stefan Beller
@ 2018-05-10  6:37     ` Junio C Hamano
  2018-05-10 21:25       ` [PATCH] " Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Junio C Hamano @ 2018-05-10  6:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: christian.couder, git, jonathantanmy, pc44800

Stefan Beller <sbeller@google.com> writes:

> +static void runcommand_in_submodule_cb(const struct cache_entry *list_item,
> +				       void *cb_data)
> +{
> +	struct cb_foreach *info = cb_data;
> +	const char *path = list_item->name;
> +	const struct object_id *ce_oid = &list_item->oid;
> +
> +	const struct submodule *sub;
> +	struct child_process cp = CHILD_PROCESS_INIT;
> +	char *displaypath;
> +
> +	displaypath = get_submodule_displaypath(path, info->prefix);
> +
> +	sub = submodule_from_path(the_repository, &null_oid, path);
> +
> +	if (!sub)
> +		die(_("No url found for submodule path '%s' in .gitmodules"),
> +			displaypath);
> +
> +	if (!is_submodule_populated_gently(path, NULL))
> +		goto cleanup;
> +
> +	prepare_submodule_repo_env(&cp.env_array);
> +
> +	/*
> +	* For the purpose of executing <command> in the submodule,
> +	* separate shell is used for the purpose of running the
> +	* child process.
> +	*/

Micronit: this multi-line comment is indented in a funny way.

> +	cp.use_shell = 1;
> +	cp.dir = path;
> +
> +	/*
> +	* NEEDSWORK: the command currently has access to the variables $name,
> +	* $sm_path, $displaypath, $sha1 and $toplevel only when the command
> +	* contains a single argument. This is done for maintaining a faithful
> +	* translation from shell script.
> +	*/

Same micronit.

The scripted version does 'eval "$1"', so $1 could be something like 

	for-each 'echo "$name:$sm_path:$displaypath:$sha1:$toplevel"'

and it can see any variable, not just these 5 (i.e. we could have
fed e.g. $wt_prefix and $mode to the above 'echo' and with the
scripted version the script would have learned their values; with
this version it no longer does, but only these 5 are part of the
documented API, so we choose not to consider it a regression).

> +	if (info->argc == 1) {
> +		char *toplevel = xgetcwd();
> +		struct strbuf sb = STRBUF_INIT;
> +
> +		argv_array_pushf(&cp.env_array, "name=%s", sub->name);
> +		argv_array_pushf(&cp.env_array, "sm_path=%s", path);
> +		argv_array_pushf(&cp.env_array, "displaypath=%s", displaypath);
> +		argv_array_pushf(&cp.env_array, "sha1=%s",
> +				oid_to_hex(ce_oid));
> +		argv_array_pushf(&cp.env_array, "toplevel=%s", toplevel);
> +
> +		/*
> +		* Since the path variable was accessible from the script
> +		* before porting, it is also made available after porting.
> +		* The environment variable "PATH" has a very special purpose
> +		* on windows. And since environment variables are
> +		* case-insensitive in windows, it interferes with the
> +		* existing PATH variable. Hence, to avoid that, we expose
> +		* path via the args argv_array and not via env_array.
> +		*/
> +		sq_quote_buf(&sb, path);
> +		argv_array_pushf(&cp.args, "path=%s; %s",
> +				 sb.buf, info->argv[0]);

OK, so we do the equivalent of

	name=... sm_path=... displaypath=... sha1=... toplevel=... \
	sh -c 'path=...; echo "$name:$sm_path:..."'

when doing

	for-each 'echo "$name:$sm_path:..."'

with parts denoted with ... correctly quoted as necessary.  I guess
it would be the best we could do.

I myself do not know if it is true that bash ported to Windows won't
get confused with the above "we use path (all lowercase) only as a
pure shell variable without exporting it ourselves"; I'd trust those
who are more familiar with the platform to raise objections and
suggest a better alternative if it is not the case.  

Thanks for the (malformatted;-) leading comment to highlight why the
'path' variable alone is treated differently from the others.

> +		strbuf_release(&sb);
> +		free(toplevel);
> +	} else {
> +		argv_array_pushv(&cp.args, info->argv);
> +	}
> +
> +	if (!info->quiet)
> +		printf(_("Entering '%s'\n"), displaypath);
> +
> +	if (info->argv[0] && run_command(&cp))
> +		die(_("run_command returned non-zero status for %s\n."),
> +			displaypath);
> +
> +	if (info->recursive) {
> +		struct child_process cpr = CHILD_PROCESS_INIT;
> +
> +		cpr.git_cmd = 1;
> +		cpr.dir = path;
> +		prepare_submodule_repo_env(&cpr.env_array);
> +
> +		argv_array_pushl(&cpr.args, "--super-prefix", NULL);
> +		argv_array_pushf(&cpr.args, "%s/", displaypath);
> +		argv_array_pushl(&cpr.args, "submodule--helper", "foreach", "--recursive",
> +				NULL);
> +
> +		if (info->quiet)
> +			argv_array_push(&cpr.args, "--quiet");
> +
> +		argv_array_pushv(&cpr.args, info->argv);
> +
> +		if (run_command(&cpr))
> +			die(_("run_command returned non-zero status while"
> +				"recursing in the nested submodules of %s\n."),
> +				displaypath);
> +	}
> +
> +cleanup:
> +	free(displaypath);
> +}


^ permalink raw reply	[relevance 5%]

* Re: [PATCH 1/1] Warn about fast-forwarding of submodules during merge
      [irrelevant] ` <20180510182657.65095-2-leif.middelschulte@gmail.com>
@ 2018-05-10 18:49   ` Stefan Beller
  2018-05-10 20:30     ` Leif Middelschulte
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-10 18:49 UTC (permalink / raw)
  To: Leif Middelschulte; +Cc: git, Junio C Hamano, sandals

On Thu, May 10, 2018 at 11:26 AM, Leif Middelschulte
<leif.middelschulte@gmail.com> wrote:
> From: Leif Middelschulte <Leif.Middelschulte@gmail.com>

Hi Leif!

thanks for following up with a patch!

> Warn the user about an automatically fast-forwarded submodule. The silent merge
> behavior was introduced by commit 68d03e4a6e44 ("Implement automatic fast-forward
> merge for submodules", 2010-07-07)).
>
> Signed-off-by: Leif Middelschulte <Leif.Middelschulte@gmail.com>
> ---
>  submodule.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/submodule.c b/submodule.c
> index 74d35b257..0198a72e6 100644
> --- a/submodule.c
> +++ b/submodule.c
> @@ -1817,10 +1817,12 @@ int merge_submodule(struct object_id *result, const char *path,
>         /* Case #1: a is contained in b or vice versa */
>         if (in_merge_bases(commit_a, commit_b)) {
>                 oidcpy(result, b);
> +               warning("Fast-forwarding submodule %s", path);
>                 return 1;
>         }
>         if (in_merge_bases(commit_b, commit_a)) {
>                 oidcpy(result, a);
> +               warning("Fast-forwarding submodule %s", path);
>                 return 1;
>         }

The code looks correct, however I think we can improve it.
(Originally I was just wondering if stderr is the right output,
which lead me to the thoughts below:)

Looking through the code of merge-recursive.c,
all the other merge outputs are done via 'output()'
that is able to buffer up the output as well as handles
the output for different verbosity settings.

So I would think we should make the output() function available
outside of merge-recursive.c. (and rename it to a be more concise
and descriptive in the global namespace) and make use of it.

Funnily we already have MERGE_WARNING in submodule.c
which outputs information for all the other cases. I would think
we ought to convert those to the output(), too.

Thanks,
Stefan

^ permalink raw reply	[relevance 6%]

* Re: [PATCH 1/1] Warn about fast-forwarding of submodules during merge
  2018-05-10 18:49   ` [PATCH 1/1] Warn about fast-forwarding of submodules during merge Stefan Beller
@ 2018-05-10 20:30     ` Leif Middelschulte
  2018-05-10 21:19       ` [PATCH 0/2] Submodule merging: i18n, verbosity Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Leif Middelschulte @ 2018-05-10 20:30 UTC (permalink / raw)
  To: Stefan Beller; +Cc: sandals, Junio C Hamano, git

Hi Stefan,


Am 10. Mai 2018 um 20:49:39, Stefan Beller
(sbeller@google.com(mailto:sbeller@google.com)) schrieb:

> On Thu, May 10, 2018 at 11:26 AM, Leif Middelschulte
> wrote:
> > From: Leif Middelschulte
>
> Hi Leif!
>
> thanks for following up with a patch!
sure, thanks for the quick review.
>
> > Warn the user about an automatically fast-forwarded submodule. The silent merge
> > behavior was introduced by commit 68d03e4a6e44 ("Implement automatic fast-forward
> > merge for submodules", 2010-07-07)).
> >
> > Signed-off-by: Leif Middelschulte
> > ---
> > submodule.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/submodule.c b/submodule.c
> > index 74d35b257..0198a72e6 100644
> > --- a/submodule.c
> > +++ b/submodule.c
> > @@ -1817,10 +1817,12 @@ int merge_submodule(struct object_id *result, const char *path,
> > /* Case #1: a is contained in b or vice versa */
> > if (in_merge_bases(commit_a, commit_b)) {
> > oidcpy(result, b);
> > + warning("Fast-forwarding submodule %s", path);
> > return 1;
> > }
> > if (in_merge_bases(commit_b, commit_a)) {
> > oidcpy(result, a);
> > + warning("Fast-forwarding submodule %s", path);
> > return 1;
> > }
>
> The code looks correct, however I think we can improve it.
> (Originally I was just wondering if stderr is the right output,
> which lead me to the thoughts below:)
I’ve had the same thoughts about stderr. However, I thought that using a
log function named `warning` to warn the user would be the right choice.
If anything, I thought, the warning function might need refactoring.

> Looking through the code of merge-recursive.c,
> all the other merge outputs are done via 'output()'
> that is able to buffer up the output as well as handles
> the output for different verbosity settings.
>
> So I would think we should make the output() function available
> outside of merge-recursive.c. (and rename it to a be more concise
> and descriptive in the global namespace) and make use of it.
Sure, let me know what to use instead and I’ll update and resubmit the patch.

>
> Funnily we already have MERGE_WARNING in submodule.c
> which outputs information for all the other cases. I would think
> we ought to convert those to the output(), too.
Sure, but `MERGE_WARNING` prefixes all the messages with "Failed to
merge submodule“.
>
> Thanks,
> Stefan

Thank you,
Leif

^ permalink raw reply	[relevance 7%]

* [PATCH 0/2] Submodule merging: i18n, verbosity
  2018-05-10 20:30     ` Leif Middelschulte
@ 2018-05-10 21:19       ` Stefan Beller
  2018-05-10 21:19         ` [PATCH 1/2] submodule.c: move submodule merging to merge-recursive.c Stefan Beller
                           ` (2 more replies)
  0 siblings, 3 replies; 200+ results
From: Stefan Beller @ 2018-05-10 21:19 UTC (permalink / raw)
  To: leif.middelschulte; +Cc: git, gitster, sandals, sbeller

Leif wrote:
> Sure, let me know what to use instead and I’ll update and resubmit the patch.
> Sure, but `MERGE_WARNING` prefixes all the messages with "Failed to
> merge submodule“.

I thought about replying and coming up with good reasons, but I wrote some
patches instead.

They can also be found at https://github.com/stefanbeller/git/tree/submodule_i18n_verbose

I think these would be a good foundation for your patch as well, as you can use the
output() function for the desired cases.

Feel free to take these patches as part of your series or adapt
(or be inspired by) as needed.

Thanks,
Stefan


Stefan Beller (2):
  submodule.c: move submodule merging to merge-recursive.c
  merge-recursive: i18n submodule merge output and respect verbosity

 merge-recursive.c | 169 +++++++++++++++++++++++++++++++++++++++++++++-
 submodule.c       | 168 +--------------------------------------------
 submodule.h       |   6 +-
 3 files changed, 170 insertions(+), 173 deletions(-)

-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 11%]

* [PATCH 1/2] submodule.c: move submodule merging to merge-recursive.c
  2018-05-10 21:19       ` [PATCH 0/2] Submodule merging: i18n, verbosity Stefan Beller
@ 2018-05-10 21:19         ` Stefan Beller
  2018-05-10 21:19         ` [PATCH 2/2] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
  2018-05-11  0:04         ` [PATCH 0/2] Submodule merging: i18n, verbosity Elijah Newren
  2 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-10 21:19 UTC (permalink / raw)
  To: leif.middelschulte; +Cc: git, gitster, sandals, sbeller

In a later patch we want to improve submodule merging by using the output()
function in merge-recursive.c for submodule merges to deliver a consistent
UI to users.

To do so we could either make the output() function globally available
so we can use it in submodule.c#merge_submodule(), or we could integrate
the submodule merging into the merging code. Choose the later as we
generally want to move submodules closer into the core.

Therefore we move any function related to merging submodules
(merge_submodule(), find_first_merges() and print_commit) to
merge-recursive.c.  We'll keep add_submodule_odb() in submodule.c as it
is used by other submodule functions. While at it, add a TODO note that
we do not really like the function add_submodule_odb().

This commit is best viewed with --color-moved.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 merge-recursive.c | 166 +++++++++++++++++++++++++++++++++++++++++++++
 submodule.c       | 168 +---------------------------------------------
 submodule.h       |   6 +-
 3 files changed, 170 insertions(+), 170 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 0c0d48624da..700ba15bf88 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -23,6 +23,7 @@
 #include "merge-recursive.h"
 #include "dir.h"
 #include "submodule.h"
+#include "revision.h"
 
 struct path_hashmap_entry {
 	struct hashmap_entry e;
@@ -977,6 +978,171 @@ static int merge_3way(struct merge_options *o,
 	return merge_status;
 }
 
+static int find_first_merges(struct object_array *result, const char *path,
+		struct commit *a, struct commit *b)
+{
+	int i, j;
+	struct object_array merges = OBJECT_ARRAY_INIT;
+	struct commit *commit;
+	int contains_another;
+
+	char merged_revision[42];
+	const char *rev_args[] = { "rev-list", "--merges", "--ancestry-path",
+				   "--all", merged_revision, NULL };
+	struct rev_info revs;
+	struct setup_revision_opt rev_opts;
+
+	memset(result, 0, sizeof(struct object_array));
+	memset(&rev_opts, 0, sizeof(rev_opts));
+
+	/* get all revisions that merge commit a */
+	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
+			oid_to_hex(&a->object.oid));
+	init_revisions(&revs, NULL);
+	rev_opts.submodule = path;
+	/* FIXME: can't handle linked worktrees in submodules yet */
+	revs.single_worktree = path != NULL;
+	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
+
+	/* save all revisions from the above list that contain b */
+	if (prepare_revision_walk(&revs))
+		die("revision walk setup failed");
+	while ((commit = get_revision(&revs)) != NULL) {
+		struct object *o = &(commit->object);
+		if (in_merge_bases(b, commit))
+			add_object_array(o, NULL, &merges);
+	}
+	reset_revision_walk();
+
+	/* Now we've got all merges that contain a and b. Prune all
+	 * merges that contain another found merge and save them in
+	 * result.
+	 */
+	for (i = 0; i < merges.nr; i++) {
+		struct commit *m1 = (struct commit *) merges.objects[i].item;
+
+		contains_another = 0;
+		for (j = 0; j < merges.nr; j++) {
+			struct commit *m2 = (struct commit *) merges.objects[j].item;
+			if (i != j && in_merge_bases(m2, m1)) {
+				contains_another = 1;
+				break;
+			}
+		}
+
+		if (!contains_another)
+			add_object_array(merges.objects[i].item, NULL, result);
+	}
+
+	object_array_clear(&merges);
+	return result->nr;
+}
+
+static void print_commit(struct commit *commit)
+{
+	struct strbuf sb = STRBUF_INIT;
+	struct pretty_print_context ctx = {0};
+	ctx.date_mode.type = DATE_NORMAL;
+	format_commit_message(commit, " %h: %m %s", &sb, &ctx);
+	fprintf(stderr, "%s\n", sb.buf);
+	strbuf_release(&sb);
+}
+
+#define MERGE_WARNING(path, msg) \
+	warning("Failed to merge submodule %s (%s)", path, msg);
+
+static int merge_submodule(struct object_id *result, const char *path,
+			   const struct object_id *base, const struct object_id *a,
+			   const struct object_id *b, int search)
+{
+	struct commit *commit_base, *commit_a, *commit_b;
+	int parent_count;
+	struct object_array merges;
+
+	int i;
+
+	/* store a in result in case we fail */
+	oidcpy(result, a);
+
+	/* we can not handle deletion conflicts */
+	if (is_null_oid(base))
+		return 0;
+	if (is_null_oid(a))
+		return 0;
+	if (is_null_oid(b))
+		return 0;
+
+	if (add_submodule_odb(path)) {
+		MERGE_WARNING(path, "not checked out");
+		return 0;
+	}
+
+	if (!(commit_base = lookup_commit_reference(base)) ||
+	    !(commit_a = lookup_commit_reference(a)) ||
+	    !(commit_b = lookup_commit_reference(b))) {
+		MERGE_WARNING(path, "commits not present");
+		return 0;
+	}
+
+	/* check whether both changes are forward */
+	if (!in_merge_bases(commit_base, commit_a) ||
+	    !in_merge_bases(commit_base, commit_b)) {
+		MERGE_WARNING(path, "commits don't follow merge-base");
+		return 0;
+	}
+
+	/* Case #1: a is contained in b or vice versa */
+	if (in_merge_bases(commit_a, commit_b)) {
+		oidcpy(result, b);
+		return 1;
+	}
+	if (in_merge_bases(commit_b, commit_a)) {
+		oidcpy(result, a);
+		return 1;
+	}
+
+	/*
+	 * Case #2: There are one or more merges that contain a and b in
+	 * the submodule. If there is only one, then present it as a
+	 * suggestion to the user, but leave it marked unmerged so the
+	 * user needs to confirm the resolution.
+	 */
+
+	/* Skip the search if makes no sense to the calling context.  */
+	if (!search)
+		return 0;
+
+	/* find commit which merges them */
+	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
+	switch (parent_count) {
+	case 0:
+		MERGE_WARNING(path, "merge following commits not found");
+		break;
+
+	case 1:
+		MERGE_WARNING(path, "not fast-forward");
+		fprintf(stderr, "Found a possible merge resolution "
+				"for the submodule:\n");
+		print_commit((struct commit *) merges.objects[0].item);
+		fprintf(stderr,
+			"If this is correct simply add it to the index "
+			"for example\n"
+			"by using:\n\n"
+			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
+			"which will accept this suggestion.\n",
+			oid_to_hex(&merges.objects[0].item->oid), path);
+		break;
+
+	default:
+		MERGE_WARNING(path, "multiple merges found");
+		for (i = 0; i < merges.nr; i++)
+			print_commit((struct commit *) merges.objects[i].item);
+	}
+
+	object_array_clear(&merges);
+	return 0;
+}
+
 static int merge_file_1(struct merge_options *o,
 					   const struct diff_filespec *one,
 					   const struct diff_filespec *a,
diff --git a/submodule.c b/submodule.c
index 74d35b25779..654089b3647 100644
--- a/submodule.c
+++ b/submodule.c
@@ -153,7 +153,8 @@ void stage_updated_gitmodules(struct index_state *istate)
 		die(_("staging updated .gitmodules failed"));
 }
 
-static int add_submodule_odb(const char *path)
+/* TODO: remove this function, use repo_submodule_init instead. */
+int add_submodule_odb(const char *path)
 {
 	struct strbuf objects_directory = STRBUF_INIT;
 	int ret = 0;
@@ -1701,171 +1702,6 @@ int submodule_move_head(const char *path,
 	return ret;
 }
 
-static int find_first_merges(struct object_array *result, const char *path,
-		struct commit *a, struct commit *b)
-{
-	int i, j;
-	struct object_array merges = OBJECT_ARRAY_INIT;
-	struct commit *commit;
-	int contains_another;
-
-	char merged_revision[42];
-	const char *rev_args[] = { "rev-list", "--merges", "--ancestry-path",
-				   "--all", merged_revision, NULL };
-	struct rev_info revs;
-	struct setup_revision_opt rev_opts;
-
-	memset(result, 0, sizeof(struct object_array));
-	memset(&rev_opts, 0, sizeof(rev_opts));
-
-	/* get all revisions that merge commit a */
-	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
-			oid_to_hex(&a->object.oid));
-	init_revisions(&revs, NULL);
-	rev_opts.submodule = path;
-	/* FIXME: can't handle linked worktrees in submodules yet */
-	revs.single_worktree = path != NULL;
-	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
-
-	/* save all revisions from the above list that contain b */
-	if (prepare_revision_walk(&revs))
-		die("revision walk setup failed");
-	while ((commit = get_revision(&revs)) != NULL) {
-		struct object *o = &(commit->object);
-		if (in_merge_bases(b, commit))
-			add_object_array(o, NULL, &merges);
-	}
-	reset_revision_walk();
-
-	/* Now we've got all merges that contain a and b. Prune all
-	 * merges that contain another found merge and save them in
-	 * result.
-	 */
-	for (i = 0; i < merges.nr; i++) {
-		struct commit *m1 = (struct commit *) merges.objects[i].item;
-
-		contains_another = 0;
-		for (j = 0; j < merges.nr; j++) {
-			struct commit *m2 = (struct commit *) merges.objects[j].item;
-			if (i != j && in_merge_bases(m2, m1)) {
-				contains_another = 1;
-				break;
-			}
-		}
-
-		if (!contains_another)
-			add_object_array(merges.objects[i].item, NULL, result);
-	}
-
-	object_array_clear(&merges);
-	return result->nr;
-}
-
-static void print_commit(struct commit *commit)
-{
-	struct strbuf sb = STRBUF_INIT;
-	struct pretty_print_context ctx = {0};
-	ctx.date_mode.type = DATE_NORMAL;
-	format_commit_message(commit, " %h: %m %s", &sb, &ctx);
-	fprintf(stderr, "%s\n", sb.buf);
-	strbuf_release(&sb);
-}
-
-#define MERGE_WARNING(path, msg) \
-	warning("Failed to merge submodule %s (%s)", path, msg);
-
-int merge_submodule(struct object_id *result, const char *path,
-		    const struct object_id *base, const struct object_id *a,
-		    const struct object_id *b, int search)
-{
-	struct commit *commit_base, *commit_a, *commit_b;
-	int parent_count;
-	struct object_array merges;
-
-	int i;
-
-	/* store a in result in case we fail */
-	oidcpy(result, a);
-
-	/* we can not handle deletion conflicts */
-	if (is_null_oid(base))
-		return 0;
-	if (is_null_oid(a))
-		return 0;
-	if (is_null_oid(b))
-		return 0;
-
-	if (add_submodule_odb(path)) {
-		MERGE_WARNING(path, "not checked out");
-		return 0;
-	}
-
-	if (!(commit_base = lookup_commit_reference(base)) ||
-	    !(commit_a = lookup_commit_reference(a)) ||
-	    !(commit_b = lookup_commit_reference(b))) {
-		MERGE_WARNING(path, "commits not present");
-		return 0;
-	}
-
-	/* check whether both changes are forward */
-	if (!in_merge_bases(commit_base, commit_a) ||
-	    !in_merge_bases(commit_base, commit_b)) {
-		MERGE_WARNING(path, "commits don't follow merge-base");
-		return 0;
-	}
-
-	/* Case #1: a is contained in b or vice versa */
-	if (in_merge_bases(commit_a, commit_b)) {
-		oidcpy(result, b);
-		return 1;
-	}
-	if (in_merge_bases(commit_b, commit_a)) {
-		oidcpy(result, a);
-		return 1;
-	}
-
-	/*
-	 * Case #2: There are one or more merges that contain a and b in
-	 * the submodule. If there is only one, then present it as a
-	 * suggestion to the user, but leave it marked unmerged so the
-	 * user needs to confirm the resolution.
-	 */
-
-	/* Skip the search if makes no sense to the calling context.  */
-	if (!search)
-		return 0;
-
-	/* find commit which merges them */
-	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
-	switch (parent_count) {
-	case 0:
-		MERGE_WARNING(path, "merge following commits not found");
-		break;
-
-	case 1:
-		MERGE_WARNING(path, "not fast-forward");
-		fprintf(stderr, "Found a possible merge resolution "
-				"for the submodule:\n");
-		print_commit((struct commit *) merges.objects[0].item);
-		fprintf(stderr,
-			"If this is correct simply add it to the index "
-			"for example\n"
-			"by using:\n\n"
-			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
-			"which will accept this suggestion.\n",
-			oid_to_hex(&merges.objects[0].item->oid), path);
-		break;
-
-	default:
-		MERGE_WARNING(path, "multiple merges found");
-		for (i = 0; i < merges.nr; i++)
-			print_commit((struct commit *) merges.objects[i].item);
-	}
-
-	object_array_clear(&merges);
-	return 0;
-}
-
 /*
  * Embeds a single submodules git directory into the superprojects git dir,
  * non recursively.
diff --git a/submodule.h b/submodule.h
index e5526f6aaab..b96689ac0db 100644
--- a/submodule.h
+++ b/submodule.h
@@ -89,10 +89,8 @@ extern int submodule_uses_gitfile(const char *path);
 #define SUBMODULE_REMOVAL_IGNORE_UNTRACKED (1<<1)
 #define SUBMODULE_REMOVAL_IGNORE_IGNORED_UNTRACKED (1<<2)
 extern int bad_to_remove_submodule(const char *path, unsigned flags);
-extern int merge_submodule(struct object_id *result, const char *path,
-			   const struct object_id *base,
-			   const struct object_id *a,
-			   const struct object_id *b, int search);
+
+int add_submodule_odb(const char *path);
 
 /* Checks if there are submodule changes in a..b. */
 extern int submodule_touches_in_range(struct object_id *a,
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 18%]

* [PATCH 2/2] merge-recursive: i18n submodule merge output and respect verbosity
  2018-05-10 21:19       ` [PATCH 0/2] Submodule merging: i18n, verbosity Stefan Beller
  2018-05-10 21:19         ` [PATCH 1/2] submodule.c: move submodule merging to merge-recursive.c Stefan Beller
@ 2018-05-10 21:19         ` Stefan Beller
  2018-05-15  1:25           ` Elijah Newren
  2018-05-11  0:04         ` [PATCH 0/2] Submodule merging: i18n, verbosity Elijah Newren
  2 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-10 21:19 UTC (permalink / raw)
  To: leif.middelschulte; +Cc: git, gitster, sandals, sbeller

The submodule merge code now uses the output() function that is used by
all the rest of the merge-recursive-code. This allows for respecting
internationalisation as well as the verbosity setting.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 merge-recursive.c | 33 +++++++++++++++------------------
 1 file changed, 15 insertions(+), 18 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 700ba15bf88..a4b91d17f87 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1048,18 +1048,17 @@ static void print_commit(struct commit *commit)
 	strbuf_release(&sb);
 }
 
-#define MERGE_WARNING(path, msg) \
-	warning("Failed to merge submodule %s (%s)", path, msg);
-
-static int merge_submodule(struct object_id *result, const char *path,
+static int merge_submodule(struct merge_options *o,
+			   struct object_id *result, const char *path,
 			   const struct object_id *base, const struct object_id *a,
-			   const struct object_id *b, int search)
+			   const struct object_id *b)
 {
 	struct commit *commit_base, *commit_a, *commit_b;
 	int parent_count;
 	struct object_array merges;
 
 	int i;
+	int search = !o->call_depth;
 
 	/* store a in result in case we fail */
 	oidcpy(result, a);
@@ -1073,21 +1072,21 @@ static int merge_submodule(struct object_id *result, const char *path,
 		return 0;
 
 	if (add_submodule_odb(path)) {
-		MERGE_WARNING(path, "not checked out");
+		output(o, 1, _("Failed to merge submodule %s (not checked out)"), path);
 		return 0;
 	}
 
 	if (!(commit_base = lookup_commit_reference(base)) ||
 	    !(commit_a = lookup_commit_reference(a)) ||
 	    !(commit_b = lookup_commit_reference(b))) {
-		MERGE_WARNING(path, "commits not present");
+		output(o, 1, _("Failed to merge submodule %s (commits not present)"), path);
 		return 0;
 	}
 
 	/* check whether both changes are forward */
 	if (!in_merge_bases(commit_base, commit_a) ||
 	    !in_merge_bases(commit_base, commit_b)) {
-		MERGE_WARNING(path, "commits don't follow merge-base");
+		output(o, 1, _("Failed to merge submodule %s (commits don't follow merge-base)"), path);
 		return 0;
 	}
 
@@ -1116,25 +1115,24 @@ static int merge_submodule(struct object_id *result, const char *path,
 	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
 	switch (parent_count) {
 	case 0:
-		MERGE_WARNING(path, "merge following commits not found");
+		output(o, 1, _("Failed to merge submodule %s (merge following commits not found)"), path);
 		break;
 
 	case 1:
-		MERGE_WARNING(path, "not fast-forward");
-		fprintf(stderr, "Found a possible merge resolution "
-				"for the submodule:\n");
+		output(o, 1, _("Failed to merge submodule %s (not fast-forward)"), path);
+		output(o, 1, _("Found a possible merge resolution for the submodule:\n"));
 		print_commit((struct commit *) merges.objects[0].item);
-		fprintf(stderr,
+		output(o, 1, _(
 			"If this is correct simply add it to the index "
 			"for example\n"
 			"by using:\n\n"
 			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
-			"which will accept this suggestion.\n",
+			"which will accept this suggestion.\n"),
 			oid_to_hex(&merges.objects[0].item->oid), path);
 		break;
 
 	default:
-		MERGE_WARNING(path, "multiple merges found");
+		output(o, 1, _("Failed to merge submodule %s (multiple merges found)"), path);
 		for (i = 0; i < merges.nr; i++)
 			print_commit((struct commit *) merges.objects[i].item);
 	}
@@ -1205,12 +1203,11 @@ static int merge_file_1(struct merge_options *o,
 				return ret;
 			result->clean = (merge_status == 0);
 		} else if (S_ISGITLINK(a->mode)) {
-			result->clean = merge_submodule(&result->oid,
+			result->clean = merge_submodule(o, &result->oid,
 						       one->path,
 						       &one->oid,
 						       &a->oid,
-						       &b->oid,
-						       !o->call_depth);
+						       &b->oid);
 		} else if (S_ISLNK(a->mode)) {
 			switch (o->recursive_variant) {
 			case MERGE_RECURSIVE_NORMAL:
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 15%]

* [PATCH] submodule: port submodule subcommand 'foreach' from shell to C
  2018-05-10  6:37     ` Junio C Hamano
@ 2018-05-10 21:25       ` " Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-10 21:25 UTC (permalink / raw)
  To: gitster; +Cc: christian.couder, git, jonathantanmy, pc44800, sbeller

From: Prathamesh Chavan <pc44800@gmail.com>

This aims to make git-submodule foreach a builtin. 'foreach' is ported to
the submodule--helper, and submodule--helper is called from
git-submodule.sh.

Helped-by: Brandon Williams <bmwill@google.com>
Mentored-by: Christian Couder <christian.couder@gmail.com>
Mentored-by: Stefan Beller <sbeller@google.com>
Signed-off-by: Prathamesh Chavan <pc44800@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---

This is a resend of the last commit in origin/pc/submodule-helper-foreach
It addresses the micro nits of funny comment indentation.

Thanks,
Stefan

 builtin/submodule--helper.c | 144 ++++++++++++++++++++++++++++++++++++
 git-submodule.sh            |  39 +---------
 2 files changed, 145 insertions(+), 38 deletions(-)

diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
index c2403a915ff..4002026d1ac 100644
--- a/builtin/submodule--helper.c
+++ b/builtin/submodule--helper.c
@@ -439,6 +439,149 @@ static void for_each_listed_submodule(const struct module_list *list,
 		fn(list->entries[i], cb_data);
 }
 
+struct cb_foreach {
+	int argc;
+	const char **argv;
+	const char *prefix;
+	int quiet;
+	int recursive;
+};
+#define CB_FOREACH_INIT { 0 }
+
+static void runcommand_in_submodule_cb(const struct cache_entry *list_item,
+				       void *cb_data)
+{
+	struct cb_foreach *info = cb_data;
+	const char *path = list_item->name;
+	const struct object_id *ce_oid = &list_item->oid;
+
+	const struct submodule *sub;
+	struct child_process cp = CHILD_PROCESS_INIT;
+	char *displaypath;
+
+	displaypath = get_submodule_displaypath(path, info->prefix);
+
+	sub = submodule_from_path(the_repository, &null_oid, path);
+
+	if (!sub)
+		die(_("No url found for submodule path '%s' in .gitmodules"),
+			displaypath);
+
+	if (!is_submodule_populated_gently(path, NULL))
+		goto cleanup;
+
+	prepare_submodule_repo_env(&cp.env_array);
+
+	/*
+	 * For the purpose of executing <command> in the submodule,
+	 * separate shell is used for the purpose of running the
+	 * child process.
+	 */
+	cp.use_shell = 1;
+	cp.dir = path;
+
+	/*
+	 * NEEDSWORK: the command currently has access to the variables $name,
+	 * $sm_path, $displaypath, $sha1 and $toplevel only when the command
+	 * contains a single argument. This is done for maintaining a faithful
+	 * translation from shell script.
+	 */
+	if (info->argc == 1) {
+		char *toplevel = xgetcwd();
+		struct strbuf sb = STRBUF_INIT;
+
+		argv_array_pushf(&cp.env_array, "name=%s", sub->name);
+		argv_array_pushf(&cp.env_array, "sm_path=%s", path);
+		argv_array_pushf(&cp.env_array, "displaypath=%s", displaypath);
+		argv_array_pushf(&cp.env_array, "sha1=%s",
+				oid_to_hex(ce_oid));
+		argv_array_pushf(&cp.env_array, "toplevel=%s", toplevel);
+
+		/*
+		 * Since the path variable was accessible from the script
+		 * before porting, it is also made available after porting.
+		 * The environment variable "PATH" has a very special purpose
+		 * on windows. And since environment variables are
+		 * case-insensitive in windows, it interferes with the
+		 * existing PATH variable. Hence, to avoid that, we expose
+		 * path via the args argv_array and not via env_array.
+		 */
+		sq_quote_buf(&sb, path);
+		argv_array_pushf(&cp.args, "path=%s; %s",
+				 sb.buf, info->argv[0]);
+		strbuf_release(&sb);
+		free(toplevel);
+	} else {
+		argv_array_pushv(&cp.args, info->argv);
+	}
+
+	if (!info->quiet)
+		printf(_("Entering '%s'\n"), displaypath);
+
+	if (info->argv[0] && run_command(&cp))
+		die(_("run_command returned non-zero status for %s\n."),
+			displaypath);
+
+	if (info->recursive) {
+		struct child_process cpr = CHILD_PROCESS_INIT;
+
+		cpr.git_cmd = 1;
+		cpr.dir = path;
+		prepare_submodule_repo_env(&cpr.env_array);
+
+		argv_array_pushl(&cpr.args, "--super-prefix", NULL);
+		argv_array_pushf(&cpr.args, "%s/", displaypath);
+		argv_array_pushl(&cpr.args, "submodule--helper", "foreach", "--recursive",
+				NULL);
+
+		if (info->quiet)
+			argv_array_push(&cpr.args, "--quiet");
+
+		argv_array_pushv(&cpr.args, info->argv);
+
+		if (run_command(&cpr))
+			die(_("run_command returned non-zero status while"
+				"recursing in the nested submodules of %s\n."),
+				displaypath);
+	}
+
+cleanup:
+	free(displaypath);
+}
+
+static int module_foreach(int argc, const char **argv, const char *prefix)
+{
+	struct cb_foreach info = CB_FOREACH_INIT;
+	struct pathspec pathspec;
+	struct module_list list = MODULE_LIST_INIT;
+
+	struct option module_foreach_options[] = {
+		OPT__QUIET(&info.quiet, N_("Suppress output of entering each submodule command")),
+		OPT_BOOL(0, "recursive", &info.recursive,
+			 N_("Recurse into nested submodules")),
+		OPT_END()
+	};
+
+	const char *const git_submodule_helper_usage[] = {
+		N_("git submodule--helper foreach [--quiet] [--recursive] <command>"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, prefix, module_foreach_options,
+			     git_submodule_helper_usage, PARSE_OPT_KEEP_UNKNOWN);
+
+	if (module_list_compute(0, NULL, prefix, &pathspec, &list) < 0)
+		return 1;
+
+	info.argc = argc;
+	info.argv = argv;
+	info.prefix = prefix;
+
+	for_each_listed_submodule(&list, runcommand_in_submodule_cb, &info);
+
+	return 0;
+}
+
 struct init_cb {
 	const char *prefix;
 	unsigned int flags;
@@ -1841,6 +1984,7 @@ static struct cmd_struct commands[] = {
 	{"relative-path", resolve_relative_path, 0},
 	{"resolve-relative-url", resolve_relative_url, 0},
 	{"resolve-relative-url-test", resolve_relative_url_test, 0},
+	{"foreach", module_foreach, SUPPORT_SUPER_PREFIX},
 	{"init", module_init, SUPPORT_SUPER_PREFIX},
 	{"status", module_status, SUPPORT_SUPER_PREFIX},
 	{"print-default-remote", print_default_remote, 0},
diff --git a/git-submodule.sh b/git-submodule.sh
index 331d71c908b..cba585f0754 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -323,44 +323,7 @@ cmd_foreach()
 		shift
 	done
 
-	toplevel=$(pwd)
-
-	# dup stdin so that it can be restored when running the external
-	# command in the subshell (and a recursive call to this function)
-	exec 3<&0
-
-	{
-		git submodule--helper list --prefix "$wt_prefix" ||
-		echo "#unmatched" $?
-	} |
-	while read -r mode sha1 stage sm_path
-	do
-		die_if_unmatched "$mode" "$sha1"
-		if test -e "$sm_path"/.git
-		then
-			displaypath=$(git submodule--helper relative-path "$prefix$sm_path" "$wt_prefix")
-			say "$(eval_gettext "Entering '\$displaypath'")"
-			name=$(git submodule--helper name "$sm_path")
-			(
-				prefix="$prefix$sm_path/"
-				sanitize_submodule_env
-				cd "$sm_path" &&
-				# we make $path available to scripts ...
-				path=$sm_path &&
-				if test $# -eq 1
-				then
-					eval "$1"
-				else
-					"$@"
-				fi &&
-				if test -n "$recursive"
-				then
-					cmd_foreach "--recursive" "$@"
-				fi
-			) <&3 3<&- ||
-			die "$(eval_gettext "Stopping at '\$displaypath'; script returned non-zero status.")"
-		fi
-	done
+	git ${wt_prefix:+-C "$wt_prefix"} ${prefix:+--super-prefix "$prefix"} submodule--helper foreach ${GIT_QUIET:+--quiet} ${recursive:+--recursive} "$@"
 }
 
 #
-- 
2.17.0.255.g8bfb7c0704


^ permalink raw reply	[relevance 20%]

* Re: [PATCH 0/2] Submodule merging: i18n, verbosity
  2018-05-10 21:19       ` [PATCH 0/2] Submodule merging: i18n, verbosity Stefan Beller
  2018-05-10 21:19         ` [PATCH 1/2] submodule.c: move submodule merging to merge-recursive.c Stefan Beller
  2018-05-10 21:19         ` [PATCH 2/2] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
@ 2018-05-11  0:04         ` Elijah Newren
  2018-05-11  1:00           ` Stefan Beller
  2 siblings, 1 reply; 200+ results
From: Elijah Newren @ 2018-05-11  0:04 UTC (permalink / raw)
  To: Stefan Beller
  Cc: leif.middelschulte, Git Mailing List, Junio C Hamano, sandals

On Thu, May 10, 2018 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:
> Leif wrote:
>> Sure, let me know what to use instead and I’ll update and resubmit the patch.
>> Sure, but `MERGE_WARNING` prefixes all the messages with "Failed to
>> merge submodule“.
>
> I thought about replying and coming up with good reasons, but I wrote some
> patches instead.
>
> They can also be found at https://github.com/stefanbeller/git/tree/submodule_i18n_verbose
>
> I think these would be a good foundation for your patch as well, as you can use the
> output() function for the desired cases.
>
> Feel free to take these patches as part of your series or adapt
> (or be inspired by) as needed.

This is awesome.  In addition to the good reasons you gave, switching
merge_submodule() to use output() was one of several things on my todo
list since I think it'd be needed for remerge-diffs
(https://bugs.chromium.org/p/git/issues/detail?id=12) and might be
useful for merges in bare repos; thanks for tackling it.

Patches look good to me.  Having Leif's patch on top of these two
would be great.

Elijah

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 0/2] Submodule merging: i18n, verbosity
  2018-05-11  0:04         ` [PATCH 0/2] Submodule merging: i18n, verbosity Elijah Newren
@ 2018-05-11  1:00           ` Stefan Beller
  2018-05-14 20:57             ` [PATCH 0/1] rebased: inform about auto submodule ff during merge Leif Middelschulte
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-11  1:00 UTC (permalink / raw)
  To: Elijah Newren
  Cc: Leif Middelschulte, Git Mailing List, Junio C Hamano, sandals

Hi Elijah,

On Thu, May 10, 2018 at 5:04 PM, Elijah Newren <newren@gmail.com> wrote:
> On Thu, May 10, 2018 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:
>> Leif wrote:
>>> Sure, let me know what to use instead and I’ll update and resubmit the patch.
>>> Sure, but `MERGE_WARNING` prefixes all the messages with "Failed to
>>> merge submodule“.
>>
>> I thought about replying and coming up with good reasons, but I wrote some
>> patches instead.
>>
>> They can also be found at https://github.com/stefanbeller/git/tree/submodule_i18n_verbose
>>
>> I think these would be a good foundation for your patch as well, as you can use the
>> output() function for the desired cases.
>>
>> Feel free to take these patches as part of your series or adapt
>> (or be inspired by) as needed.
>
> This is awesome.  In addition to the good reasons you gave, switching
> merge_submodule() to use output() was one of several things on my todo
> list since I think it'd be needed for remerge-diffs
> (https://bugs.chromium.org/p/git/issues/detail?id=12) and might be
> useful for merges in bare repos; thanks for tackling it.

Thanks for the encouraging words!
The one nit I find on that series is that we need to rely on and export
the add_submodule_odb function as I want to get rid of that function
once the object store series has progressed far enough.

>
> Patches look good to me.  Having Leif's patch on top of these two
> would be great.

ok, Let's go with that.

Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH v2 0/4] Fix mem leaks of recent object store conversions.
      [irrelevant] <20180510195849.28023-1-sbeller@google.com>
@ 2018-05-11  8:37 ` Jeff King
  2018-05-11 18:59   ` Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Jeff King @ 2018-05-11  8:37 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git

On Thu, May 10, 2018 at 12:58:45PM -0700, Stefan Beller wrote:

> This series replaces the two commits that were queued on sb/object-store-replace,
> fixing memory leaks that were recently introduced.
> 
> Compared to v1, I merged the two independent series from yesterday,
> rewrote the commit message to clear up Junios confusion and addresses Peffs
> comments for the packfiles as well.

Mostly. :)

My one remaining complaint is that the bitmap code may hold on to a
dangling pointer to a packed_git after this series.

I think that is part of a larger problem, though, which is that the
bitmap code's globals need to be part of the struct raw_object_store.
I think this can already cause problems before your series if we were to
try to use bitmaps in both a superproject and a submodule in the same
process, though I think we'd at least hit the "ignoring extra bitmap
file" code path in open_pack_bitmap_1(). So right now it's an annoyance,
but after your series it becomes a potential segfault.

-Peff

^ permalink raw reply	[relevance 4%]

* Re: [PATCH v2 0/4] Fix mem leaks of recent object store conversions.
  2018-05-11  8:37 ` [PATCH v2 0/4] Fix mem leaks of recent object store conversions Jeff King
@ 2018-05-11 18:59   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-11 18:59 UTC (permalink / raw)
  To: Jeff King; +Cc: Junio C Hamano, git

On Fri, May 11, 2018 at 1:37 AM, Jeff King <peff@peff.net> wrote:
> On Thu, May 10, 2018 at 12:58:45PM -0700, Stefan Beller wrote:
>
>> This series replaces the two commits that were queued on sb/object-store-replace,
>> fixing memory leaks that were recently introduced.
>>
>> Compared to v1, I merged the two independent series from yesterday,
>> rewrote the commit message to clear up Junios confusion and addresses Peffs
>> comments for the packfiles as well.
>
> Mostly. :)
>
> My one remaining complaint is that the bitmap code may hold on to a
> dangling pointer to a packed_git after this series.

Ok, I'll look into that.

>
> I think that is part of a larger problem, though, which is that the
> bitmap code's globals need to be part of the struct raw_object_store.
> I think this can already cause problems before your series if we were to
> try to use bitmaps in both a superproject and a submodule in the same
> process, though I think we'd at least hit the "ignoring extra bitmap
> file" code path in open_pack_bitmap_1(). So right now it's an annoyance,
> but after your series it becomes a potential segfault.

Ok, maybe we'll need to convert bitmaps into the object store for that.

Thanks for the pointer,
Stefan

^ permalink raw reply	[relevance 2%]

* [PATCH] git-submodule.sh: try harder to fetch a submodule
@ 2018-05-11 23:17 Stefan Beller
  2018-05-11 23:28 ` Jonathan Nieder
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-11 23:17 UTC (permalink / raw)
  To: gitster; +Cc: git, Stefan Beller

This is the logical continuum of fb43e31f2b4 (submodule: try harder to
fetch needed sha1 by direct fetching sha1, 2016-02-23) and fixes it as
some assumptions were not correct.

> If $sha1 was not part of the default fetch ... fail ourselves here
assumes that the fetch_in_submodule only fails when the serverside does
not support fetching by sha1.

There are other failures, why such a fetch may fail, such as
    fatal: Couldn't find remote ref HEAD
which can happen if the remote side doesn't advertise HEAD. Not advertising
HEAD is allowed by the protocol spec and would happen, if HEAD points at a
ref, that this user cannot see (due to ACLs for example).

So do try even harder for a submodule by ignoring the exit code of the
first fetch and rather relying on the following is_tip_reachable to
see if we try fetching again.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh | 1 -
 1 file changed, 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 24914963ca2..13b378a6c8f 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -614,7 +614,6 @@ cmd_update()
 				# is not reachable from a ref.
 				is_tip_reachable "$sm_path" "$sha1" ||
 				fetch_in_submodule "$sm_path" $depth ||
-				die "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
 
 				# Now we tried the usual fetch, but $sha1 may
 				# not be reachable from any of the refs
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 17%]

* Re: [PATCH] git-submodule.sh: try harder to fetch a submodule
  2018-05-11 23:17 [PATCH] git-submodule.sh: try harder to fetch a submodule Stefan Beller
@ 2018-05-11 23:28 ` Jonathan Nieder
  2018-05-11 23:42   ` Stefan Beller
      [irrelevant]   ` <xmqqmux5g2pa.fsf@gitster-ct.c.googlers.com>
  0 siblings, 2 replies; 200+ results
From: Jonathan Nieder @ 2018-05-11 23:28 UTC (permalink / raw)
  To: Stefan Beller; +Cc: gitster, git

Hi,

Stefan Beller wrote:

> This is the logical continuum of fb43e31f2b4 (submodule: try harder to
> fetch needed sha1 by direct fetching sha1, 2016-02-23) and fixes it as
> some assumptions were not correct.

Interesting.

I think what would help most is an example set of commands I can use
to reproduce this (bonus points if in the form of a test).

> > If $sha1 was not part of the default fetch ... fail ourselves here
> assumes that the fetch_in_submodule only fails when the serverside does

I'm having some trouble with the formatting here.  Is the part
preceded by '>' a quote, and if so a quote from what?

> not support fetching by sha1.
>
> There are other failures, why such a fetch may fail, such as
>     fatal: Couldn't find remote ref HEAD
> which can happen if the remote side doesn't advertise HEAD. Not advertising

nit: it can be useful to have a blank line before and after such
example output to help both my eyes and tools like "git log
--format='%w(100)%b'" to understand the formatting.

> HEAD is allowed by the protocol spec and would happen, if HEAD points at a
> ref, that this user cannot see (due to ACLs for example).

A more typical example would be if the ref simply doesn't exist (i.e.,
is a branch yet to be born).

> So do try even harder for a submodule by ignoring the exit code of the
> first fetch and rather relying on the following is_tip_reachable to
> see if we try fetching again.
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  git-submodule.sh | 1 -
>  1 file changed, 1 deletion(-)
> 
> diff --git a/git-submodule.sh b/git-submodule.sh
> index 24914963ca2..13b378a6c8f 100755
> --- a/git-submodule.sh
> +++ b/git-submodule.sh
> @@ -614,7 +614,6 @@ cmd_update()
>  				# is not reachable from a ref.
>  				is_tip_reachable "$sm_path" "$sha1" ||
>  				fetch_in_submodule "$sm_path" $depth ||

Is keeping the '||' at the end of this line intended?

> -				die "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
>  
>  				# Now we tried the usual fetch, but $sha1 may
>  				# not be reachable from any of the refs
> 				is_tip_reachable "$sm_path" "$sha1" ||
> 				fetch_in_submodule "$sm_path" $depth "$sha1" ||
> 				die "$(eval_gettext "Fetched in submodule path '\$displaypath', but it did not contain \$sha1. Direct fetching of that commit failed.")"

Should this error message be changed?

Thanks,
Jonathan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] git-submodule.sh: try harder to fetch a submodule
  2018-05-11 23:28 ` Jonathan Nieder
@ 2018-05-11 23:42   ` Stefan Beller
      [irrelevant]   ` <xmqqmux5g2pa.fsf@gitster-ct.c.googlers.com>
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-11 23:42 UTC (permalink / raw)
  To: Jonathan Nieder; +Cc: Junio C Hamano, git

On Fri, May 11, 2018 at 4:28 PM, Jonathan Nieder <jrnieder@gmail.com> wrote:
> Hi,
>
> Stefan Beller wrote:
>
>> This is the logical continuum of fb43e31f2b4 (submodule: try harder to
>> fetch needed sha1 by direct fetching sha1, 2016-02-23) and fixes it as
>> some assumptions were not correct.
>
> Interesting.
>
> I think what would help most is an example set of commands I can use
> to reproduce this (bonus points if in the form of a test).

I tried coming up with a test in git-core that produces a remote similar to
Gerrit, and let me tell you, it's not Git that is weird here. ;)

>> > If $sha1 was not part of the default fetch ... fail ourselves here
>> assumes that the fetch_in_submodule only fails when the serverside does
>
> I'm having some trouble with the formatting here.  Is the part
> preceded by '>' a quote, and if so a quote from what?

The quote is from fb43e31f2b4.

>> There are other failures, why such a fetch may fail, such as
>>     fatal: Couldn't find remote ref HEAD
>> which can happen if the remote side doesn't advertise HEAD. Not advertising
>
> nit: it can be useful to have a blank line before and after such
> example output to help both my eyes and tools like "git log
> --format='%w(100)%b'" to understand the formatting.

Why would you use this formatting?

>
>> HEAD is allowed by the protocol spec and would happen, if HEAD points at a
>> ref, that this user cannot see (due to ACLs for example).
>
> A more typical example would be if the ref simply doesn't exist (i.e.,
> is a branch yet to be born).

Oh, I checked that but not for the submodule case, let me check that.

>> diff --git a/git-submodule.sh b/git-submodule.sh
>> index 24914963ca2..13b378a6c8f 100755
>> --- a/git-submodule.sh
>> +++ b/git-submodule.sh
>> @@ -614,7 +614,6 @@ cmd_update()
>>                               # is not reachable from a ref.
>>                               is_tip_reachable "$sm_path" "$sha1" ||
>>                               fetch_in_submodule "$sm_path" $depth ||
>
> Is keeping the '||' at the end of this line intended?

yes, as we only want to run the code below if there was some error.

>> -                             die "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
>>
>>                               # Now we tried the usual fetch, but $sha1 may
>>                               # not be reachable from any of the refs
>>                               is_tip_reachable "$sm_path" "$sha1" ||
>>                               fetch_in_submodule "$sm_path" $depth "$sha1" ||
>>                               die "$(eval_gettext "Fetched in submodule path '\$displaypath', but it did not contain \$sha1. Direct fetching of that commit failed.")"
>
> Should this error message be changed?

I don't think so?

^ permalink raw reply	[relevance 8%]

* [PATCH 0/1] rebased: inform about auto submodule ff during merge
  2018-05-11  1:00           ` Stefan Beller
@ 2018-05-14 20:57             ` Leif Middelschulte
      [irrelevant]               ` <20180514205737.21313-2-leif.middelschulte@gmail.com>
  0 siblings, 1 reply; 200+ results
From: Leif Middelschulte @ 2018-05-14 20:57 UTC (permalink / raw)
  To: git; +Cc: gitster, Leif Middelschulte

From: Leif Middelschulte <Leif.Middelschulte@gmail.com>

This patch is in response to Stefan Beller's Commit 0357af480
("merge-recursive: i18n submodule merge output and respect verbosity",
2018-05-10) and is based on the changes it provided.

Leif Middelschulte (1):
  Inform about fast-forwarding of submodules during merge

 merge-recursive.c | 4 ++++
 1 file changed, 4 insertions(+)

-- 
2.15.1 (Apple Git-101)


^ permalink raw reply	[relevance 8%]

* Re: [PATCH 1/1] Inform about fast-forwarding of submodules during merge
      [irrelevant]               ` <20180514205737.21313-2-leif.middelschulte@gmail.com>
@ 2018-05-15  0:41                 ` " Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  0:41 UTC (permalink / raw)
  To: Leif Middelschulte; +Cc: git, Junio C Hamano

On Mon, May 14, 2018 at 1:57 PM, Leif Middelschulte
<leif.middelschulte@gmail.com> wrote:
> From: Leif Middelschulte <Leif.Middelschulte@gmail.com>
>
> Inform the user about an automatically fast-forwarded submodule. The silent merge
> behavior was introduced by commit 68d03e4a6e44 ("Implement automatic fast-forward
> merge for submodules", 2010-07-07)).
>
> Signed-off-by: Leif Middelschulte <Leif.Middelschulte@gmail.com>

Thanks for following up with a patch.
This looks good to me!

Thanks,
Stefan

> ---
>  merge-recursive.c | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> index a4b91d17f..4a03044d1 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -1093,10 +1093,14 @@ static int merge_submodule(struct merge_options *o,
>         /* Case #1: a is contained in b or vice versa */
>         if (in_merge_bases(commit_a, commit_b)) {
>                 oidcpy(result, b);
> +               output(o, 1, _("Note: Fast-forwarding submodule %s to the following commit"), path);
> +               output_commit_title(o, commit_b);
>                 return 1;
>         }
>         if (in_merge_bases(commit_b, commit_a)) {
>                 oidcpy(result, a);
> +               output(o, 1, _("Note: Fast-forwarding submodule %s to the following commit:"), path);
> +               output_commit_title(o, commit_a);
>                 return 1;
>         }
>
> --
> 2.15.1 (Apple Git-101)
>

^ permalink raw reply	[relevance 5%]

* Re: [RFC PATCH 01/10] config: make config_from_gitmodules generally useful
      [irrelevant] ` <20180514105823.8378-2-ao2@ao2.it>
@ 2018-05-15  1:05   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:05 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

On Mon, May 14, 2018 at 3:58 AM, Antonio Ospite <ao2@ao2.it> wrote:
> The config_from_gitmodules() function is a good candidate for
> a centralized point where to read the gitmodules configuration file.

It is very tempting to use that function. However it was introduced
specifically to not do that. ;)

See the series that was merged at 5aa0b6c506c (Merge branch
'bw/grep-recurse-submodules', 2017-08-22), specifically
f20e7c1ea24 (submodule: remove submodule.fetchjobs from
submodule-config parsing, 2017-08-02), where both
builtin/fetch as well as the submodule helper use the pattern to
read from the .gitmodules file va this function and then overlay it
with the read from config.

> Add a repo argument to it to make the function more general, and adjust
> the current callers in cmd_fetch and update-clone.

This could be a preparatory patch, but including it here is fine, too.

> As a proof of the utility of the change, start using the function also
> in repo_read_gitmodules which was basically doing the same operations.

I think they were separated for the reason outlined above, or what Brandon
said in his reply.

I think extending 'repo_read_gitmodules' makes sense, as that is
used everywhere for the loading of submodule configuration.

^ permalink raw reply	[relevance 6%]

* Re: [RFC PATCH 00/10] Make submodules work if .gitmodules is not checked out
      [irrelevant] <20180514105823.8378-1-ao2@ao2.it>
      [irrelevant] ` <20180514105823.8378-2-ao2@ao2.it>
@ 2018-05-15  1:14 ` Stefan Beller
      [irrelevant] ` <20180514105823.8378-3-ao2@ao2.it>
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:14 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

Hi Antonio,

thanks for sending this series! Happy to review it!

>   - my git terminology may still be a little off: do "work tree" and
>     "work directory" mean the same thing?

Back in the old days, you had a "worktree" which is a directory where
things are checked out and you modify files. It sort of the opposite of the
"git directory", which git uses to store all its information.

Then quite some time later, the command git-worktree was invented.
Now we had 2 things with the same name, which is unfortunate.
But as the command git-worktree added more of these directories
that you could work in, the name collision was not apparent.

Later when people noticed the subtle difference between the
command and the thing on the file system, consensus seemed to be
that the thing on the file system should rather be called "working tree"
such that it sounds very similar but is distinguishable from the command.

However the outcome of the discussion did not yield a bi&complete
refactoring of the code base, such that there are still places with "worktree"
referring to the thing on the FS, "working trees".

I am not aware that "working directory" is an official term we use in
any documentation for Git, but it sounds like you mean a "working tree".
(From a point of view not based on the version control, "working directory"
may sound more correct, note however as the directories in Git are
named trees, the working "tree" sounds as if you can make changes to
Git trees, ... which you can. :) )

> If anyone wanted to pick up and finish the work feel free to do so,
> otherwise please comment and I'll try to address issues as time permits.

First let's review this series. :)

Thanks,
Stefan

^ permalink raw reply	[relevance 4%]

* Re: [RFC PATCH 02/10] submodule: factor out a config_gitmodules_set function
      [irrelevant] ` <20180514105823.8378-3-ao2@ao2.it>
@ 2018-05-15  1:20   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:20 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

On Mon, May 14, 2018 at 3:58 AM, Antonio Ospite <ao2@ao2.it> wrote:
> Introduce a new config_gitmodules_set function to write config values to the
> .gitmodules file.
>
> This is in preparation for a future change which will use the function
> to write to the .gitmodules file in a more controlled way instead of
> using "git config -f .gitmodules".
>
> Signed-off-by: Antonio Ospite <ao2@ao2.it>
> ---
>
> Not sure about the name, and maybe it can go in config.c for symmetry with
> config_from_gitmodules?

What is the function about (in the end state and now) ?
is it more of a
* configure_submodules_config()
  which would convey it is a generic function to configure submodules
  (i.e. it may not even write to *the* .gitmodules file but somewhere else,
  such as a helper ref)
  This doesn't sound like it as we make use of the function in
  update_path_in_gitmodules() that is used from git-mv, which would want
  to ensure that the specific .gitmodules file is changed.
* gitmodules_file_set()
  that focuses on the specific file that we want to modify?
* ...

Let's continue reading the series to see the end state for
a good name.

Stefan

^ permalink raw reply	[relevance 7%]

* Re: [RFC PATCH 03/10] t7411: be nicer to other tests and really clean things up
      [irrelevant] ` <20180514105823.8378-4-ao2@ao2.it>
@ 2018-05-15  1:23   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:23 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

On Mon, May 14, 2018 at 3:58 AM, Antonio Ospite <ao2@ao2.it> wrote:
> Tests 5 and 8 in t/t7411-submodule-config.sh add two commits with
> invalid lines in .gitmodules but then only the second commit is removed.
>
> This may affect subsequent tests if they assume that the .gitmodules
> file has no errors.
>
> Since those commits are not needed anymore remove both of them.
>
> Signed-off-by: Antonio Ospite <ao2@ao2.it>
> ---
>
> I am putting these fixups to the test-suite before the patch that actually
> needs them so that the test-suite passes after each commit.
>
>  t/t7411-submodule-config.sh | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/t/t7411-submodule-config.sh b/t/t7411-submodule-config.sh
> index 0bde5850a..a648de6a9 100755
> --- a/t/t7411-submodule-config.sh
> +++ b/t/t7411-submodule-config.sh
> @@ -135,7 +135,7 @@ test_expect_success 'error in history in fetchrecursesubmodule lets continue' '
>                         HEAD submodule \
>                                 >actual &&
>                 test_cmp expect_error actual  &&
> -               git reset --hard HEAD^
> +               git reset --hard HEAD~2
>         )
>  '

As this is the last test in this file, we do not change any subsequent
tests in a subtle way.
Good!

This is
Reviewed-by: Stefan Beller <sbeller@google.com>

FYI:
This test -- of course -- doesn't quite follow the latest coding guidelines,
as usually we'd prefer a test_when_finished "<cmd to restore>"
at the beginning of a test.

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 2/2] merge-recursive: i18n submodule merge output and respect verbosity
  2018-05-10 21:19         ` [PATCH 2/2] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
@ 2018-05-15  1:25           ` Elijah Newren
  0 siblings, 0 replies; 200+ results
From: Elijah Newren @ 2018-05-15  1:25 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Leif Middelschulte, Git Mailing List, Junio C Hamano

I know I said the patches looked okay earlier, but I just noticed something...

On Thu, May 10, 2018 at 2:19 PM, Stefan Beller <sbeller@google.com> wrote:

>         case 1:
> -               MERGE_WARNING(path, "not fast-forward");
> -               fprintf(stderr, "Found a possible merge resolution "
> -                               "for the submodule:\n");
> +               output(o, 1, _("Failed to merge submodule %s (not fast-forward)"), path);

We allow folks to set GIT_MERGE_VERBOSITY to change how much output
they get.  A setting of 1 should only show conflicts or major
warnings.  2 is the default and adds a few more messages (e.g.
"Auto-merging $PATH", "Adding $PATH" for one-sided adds, etc.), higher
levels show even more.

Anyway this output message is correct to use level 1 since this is a
conflict, but...

> +               output(o, 1, _("Found a possible merge resolution for the submodule:\n"));

I think this should use level 2.

>                 print_commit((struct commit *) merges.objects[0].item);
> -               fprintf(stderr,
> +               output(o, 1, _(
>                        "If this is correct simply add it to the index "
>                        "for example\n"
>                        "by using:\n\n"
>                        "  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
>-                       "which will accept this suggestion.\n",
>+                       "which will accept this suggestion.\n"),
>                        oid_to_hex(&merges.objects[0].item->oid), path);

and so should this one (in fact, I'm tempted to say these last two
should use level 3, but since it looks like a command users may have
difficulty finding on their own, I'm okay with going with 2).

^ permalink raw reply	[relevance 4%]

* Re: [RFC PATCH 04/10] submodule--helper: add a new 'config' subcommand
      [irrelevant] ` <20180514105823.8378-5-ao2@ao2.it>
@ 2018-05-15  1:33   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:33 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

On Mon, May 14, 2018 at 3:58 AM, Antonio Ospite <ao2@ao2.it> wrote:
> Add a new 'config' subcommand to 'submodule--helper', this extra level
> of indirection makes it possible to add some flexibility to how the
> submodules configuration is handled.
>
> Signed-off-by: Antonio Ospite <ao2@ao2.it>
> ---
>  builtin/submodule--helper.c | 39 +++++++++++++++++++++++++++++++++++++
>  t/t7411-submodule-config.sh | 26 +++++++++++++++++++++++++
>  2 files changed, 65 insertions(+)
>
> diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c
> index 9e8f2acd5..b32110e3b 100644
> --- a/builtin/submodule--helper.c
> +++ b/builtin/submodule--helper.c
> @@ -1825,6 +1825,44 @@ static int is_active(int argc, const char **argv, const char *prefix)
>         return !is_submodule_active(the_repository, argv[1]);
>  }
>
> +static int config_print_callback(const char *key_, const char *value_, void *cb_data)
> +{
> +       char *key = cb_data;
> +
> +       if (!strcmp(key, key_))
> +               printf("%s\n", value_);
> +
> +       return 0;
> +}
> +
> +static int module_config(int argc, const char **argv, const char *prefix)
> +{
> +       int ret;
> +
> +       if (argc < 2 || argc > 3)
> +               die("submodule--helper config takes 1 or 2 arguments: name [value]");
> +
> +       /* Equivalent to ACTION_GET in builtin/config.c */
> +       if (argc == 2) {
> +               char *key;
> +
> +               ret = git_config_parse_key(argv[1], &key, NULL);
> +               if (ret < 0)
> +                       return CONFIG_INVALID_KEY;
> +
> +               config_from_gitmodules(config_print_callback, the_repository, key);
> +
> +               free(key);
> +               return 0;
> +       }
> +
> +       /* Equivalent to ACTION_SET in builtin/config.c */
> +       if (argc == 3)
> +               return config_gitmodules_set(argv[1], argv[2]);

Ah, here we definitely want to set it in the .gitmodules file?
(Or does that change later in this series?)

> +
> +       return 0;
> +}
> +
>  #define SUPPORT_SUPER_PREFIX (1<<0)
>
>  struct cmd_struct {
> @@ -1850,6 +1888,7 @@ static struct cmd_struct commands[] = {
>         {"push-check", push_check, 0},
>         {"absorb-git-dirs", absorb_git_dirs, SUPPORT_SUPER_PREFIX},
>         {"is-active", is_active, 0},
> +       {"config", module_config, 0},
>  };
>
>  int cmd_submodule__helper(int argc, const char **argv, const char *prefix)
> diff --git a/t/t7411-submodule-config.sh b/t/t7411-submodule-config.sh
> index a648de6a9..dfe019f05 100755
> --- a/t/t7411-submodule-config.sh
> +++ b/t/t7411-submodule-config.sh
> @@ -139,4 +139,30 @@ test_expect_success 'error in history in fetchrecursesubmodule lets continue' '
>         )
>  '
>
> +test_expect_success 'reading submodules config with "submodule--helper config"' '
> +       (cd super &&

I think the project prefers a style
of the cd at the same level of the echo and the following commands.

However we might not need the (cd super && ...) via

  echo "../submodule"  >expected
  git -C super ubmodule--helper config submodule.submodule.url >../actual
  test_cmp expected actual

Our friends developing git on Windows will thank us for saving
to spawn a shell as spawning processes is expensive on Windows. :)
Also we have fewer lines of code.

The patch looks good to me,
Thanks,
Stefan

^ permalink raw reply	[relevance 8%]

* Re: [RFC PATCH 09/10] submodule: support reading .gitmodules even when it's not checked out
      [irrelevant] ` <20180514105823.8378-10-ao2@ao2.it>
@ 2018-05-15  1:45   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15  1:45 UTC (permalink / raw)
  To: Antonio Ospite
  Cc: git, Brandon Williams, Daniel Graña, Jonathan Nieder,
	Richard Hartmann

On Mon, May 14, 2018 at 3:58 AM, Antonio Ospite <ao2@ao2.it> wrote:
> When the .gitmodules file is not available in the working directory, try
> using HEAD:.gitmodules from the index.

I think HEAD:.gitmodules is different than the index (the former is
part of the latest commit, whereas the index could have changed via
git-add, that is not committed yet).

> This covers the case when the
> file is part of the repository but for some reason it is not checked
> out, for example because of a sparse checkout.
>
> This makes it possible to use at least the 'git submodule' commands
> which *read* the gitmodules configuration file without fully populating
> the work dir.

Instead of checking for an explicit sparse "hidden" could we just rely on
the file missing? Then I could continue using submodules if I just
"rm .gitmodules".

>
> Writing to .gitmodules wills still require that the file is checked out,
> so check for that in config_gitmodules_set.

That makes sense!

>
> Signed-off-by: Antonio Ospite <ao2@ao2.it>
> ---
>
> I am doing the is_gitmodules_hidden() check in the open for now, I am not sure
> whether it is approprate to do that inside stage_updated_gitmodules.

Why do we need that check at all?

In your use case, you want to checkout *a* .gitmodules file, not necessarily
the .gitmodules file of that repo you're currently working on. So it
sort of makes
sense to prevent cross-repo changes (i.e. committing the .gitmodules
accidentally
into the wrong repo)

Stefan

^ permalink raw reply	[relevance 7%]

* Re: [PATCH] grep: handle corrupt index files early
      [irrelevant] ` <CACsJy8AcG6zGa9vLCwm2B4ishyJVWdFQ2YV0FOZTor_0x8Q64g@mail.gmail.com>
@ 2018-05-15 16:44   ` Stefan Beller
  2018-05-16 15:24     ` Duy Nguyen
  0 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-15 16:44 UTC (permalink / raw)
  To: Duy Nguyen
  Cc: Brandon Williams, Junio C Hamano, Git Mailing List, Antonio Ospite

On Tue, May 15, 2018 at 6:13 AM, Duy Nguyen <pclouds@gmail.com> wrote:
> On Tue, May 15, 2018 at 3:04 AM, Stefan Beller <sbeller@google.com> wrote:
>> Any other caller of 'repo_read_index' dies upon a negative return of
>> it, so grep should, too.
>>
>> Signed-off-by: Stefan Beller <sbeller@google.com>
>> ---
>>
>> Found while reviewing the series
>> https://public-inbox.org/git/20180514105823.8378-1-ao2@ao2.it/
>>
>>  builtin/grep.c | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/builtin/grep.c b/builtin/grep.c
>> index 6e7bc76785a..69f0743619f 100644
>> --- a/builtin/grep.c
>> +++ b/builtin/grep.c
>> @@ -488,7 +488,8 @@ static int grep_cache(struct grep_opt *opt, struct repository *repo,
>>                 strbuf_addstr(&name, repo->submodule_prefix);
>>         }
>>
>> -       repo_read_index(repo);
>> +       if (repo_read_index(repo) < 0)
>> +               die("index file corrupt");
>
> _() the string (and maybe reuse an existing phrase if found to reduce
> workload on translators)

sbeller@sbeller:/u/git$ git grep -A 1 repo_read_index
builtin/grep.c:491:     if (repo_read_index(repo) < 0)
builtin/grep.c-492-             die("index file corrupt");
--
builtin/ls-files.c:213: if (repo_read_index(&submodule) < 0)
builtin/ls-files.c-214-         die("index file corrupt");
--
builtin/ls-files.c:582: if (repo_read_index(the_repository) < 0)
builtin/ls-files.c-583-         die("index file corrupt");
--
dir.c:3028:     if (repo_read_index(&subrepo) < 0)
dir.c-3029-             die("index file corrupt in repo %s", subrepo.gitdir);
--
repository.c:245:int repo_read_index(struct repository *repo)
repository.c-246-{
--
repository.h:70:         * 'repo_read_index()' can be used to populate 'index'.
repository.h-71-         */
--
repository.h:119:extern int repo_read_index(struct repository *repo);
repository.h-120-
--
submodule-config.c:583:         if (repo_read_index(repo) < 0)
submodule-config.c-584-                 return;
--
submodule.c:1336:       if (repo_read_index(r) < 0)
submodule.c-1337-               die("index file corrupt");

I think this is as good as it gets for using an existing phrase.
None of them are translated, which I would defer to a follow up patch
that translates all(?) of them or just the porcelains.

Thanks,
Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 35/35] submodule: convert push_unpushed_submodules to take a struct refspec
      [irrelevant]   ` <87k1s52va3.fsf@evledraar.gmail.com>
@ 2018-05-15 16:52     ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 16:52 UTC (permalink / raw)
  To: Ævar Arnfjörð Bjarmason; +Cc: Brandon Williams, git

On Tue, May 15, 2018 at 1:11 AM, Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> On Mon, May 14 2018, Brandon Williams wrote:
>
>> Convert 'push_unpushed_submodules()' to take a 'struct refspec' as a
>> parameter instead of an array of 'const char *'.
>> [...]
>> diff --git a/submodule.h b/submodule.h
>> index e5526f6aa..aae0c9c8f 100644
>> --- a/submodule.h
>> +++ b/submodule.h
>> @@ -100,9 +100,10 @@ extern int submodule_touches_in_range(struct object_id *a,
>>  extern int find_unpushed_submodules(struct oid_array *commits,
>>                                   const char *remotes_name,
>>                                   struct string_list *needs_pushing);
>> +struct refspec;
>>  extern int push_unpushed_submodules(struct oid_array *commits,
>>                                   const struct remote *remote,
>> -                                 const char **refspec, int refspec_nr,
>> +                                 const struct refspec *rs,
>>                                   const struct string_list *push_options,
>>                                   int dry_run);
>>  /*
>
> Why do you prefer doing this to having this on top?:

The fewer includes in header files the better, as then the headers
themselves don't have dependencies. (Otherwise we'd end up
multiplying cache.h ;)

In the source files we have to include all needed headers, but for
the headers, it is better if we can just get away with declaring the
existence of a struct.

This way we reduce compile time, so I'd am not keen on your patch
on top.

This is discussed a lot on stackoverflow, e.g.:
https://softwareengineering.stackexchange.com/questions/195806/forward-declaration-vs-include
https://stackoverflow.com/a/15828094

Stefan

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] git-submodule.sh: try harder to fetch a submodule
      [irrelevant]   ` <xmqqmux5g2pa.fsf@gitster-ct.c.googlers.com>
@ 2018-05-15 19:07     ` Stefan Beller
  2018-05-15 19:40       ` Stefan Beller
  2018-05-15 20:04       ` Jonathan Nieder
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-05-15 19:07 UTC (permalink / raw)
  To: Junio C Hamano; +Cc: Jonathan Nieder, git

On Fri, May 11, 2018 at 5:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
>> A more typical example would be if the ref simply doesn't exist (i.e.,
>> is a branch yet to be born).
>
> Indeed this is interesting.  At first glance I thought this was
> about underlying "git clone" failing to grab things from a
> repository with unborn HEAD, but that part works perfectly OK.

ok.

> So it probably should be more like
>
>         guard1 || action1 || warn
>         guard2 || action2 || die
>
> so that no matter what the outcome of the action1 is, the second set
> gets executed.
>

I'll resend it with a warning (using say()).

I think we have 2 bugs and this is merely fixing the second bug.

The first bug:
We had a call chain as
  git clone --recurse-submodules=<path spec>
    -> git submodule update --init --recursive $(pathspec)
      -> git submodule--helper update-clone # will clone
        -> git submodule helper clone
          -> git clone --no-checkout --separate-git-dir ...

The call to the "git clone" produces an interesting
submodule state:

  $ git init confused-head
  $ (cd confused-head && git branch test \
        $(git commit-tree $(git write-tree) -m test))
  $ git clone --no-checkout  --depth=1 \
        --separate-git-dir=test.git confused-head/.git test
Cloning into 'test'...
warning: --depth is ignored in local clones; use file:// instead.
done.

  $ git -C test.git config remote.origin.fetch
  $ echo $?
1

(A) Despite the warning of --depth having no impact, the
  omission thereof changes the repository state.
(B) There is no remote.origin.fetch configuration, which
  is weird. See builtin/clone.c:830, that states for this case:

    /*
     * otherwise, the next "git fetch" will
     * simply fetch from HEAD without updating
     * any remote-tracking branch, which is what
     * we want.
     */

I disagree as the next fetch will be confused
(HEAD is not advertised on the next ls-remote)

The patch that is under discussion here is merely
papering over the effect of having no fetch spec,
by allowing the second fetch (fetching the sha1 directly)
to run, which ignores the configuration as a refspec is
given.

However it is still a bug, as such repositories are out there,
which is why I said there are 2 bugs initially. It's just that
the first bug enables the second bug.

Thanks,
Stefan

^ permalink raw reply	[relevance 8%]

* [PATCH] git-submodule.sh: try harder to fetch a submodule
  2018-05-15 19:07     ` Stefan Beller
@ 2018-05-15 19:40       ` Stefan Beller
  2018-05-15 20:04       ` Jonathan Nieder
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 19:40 UTC (permalink / raw)
  To: sbeller; +Cc: git, gitster, jrnieder

This is the logical continuum of fb43e31f2b4 (submodule: try harder to
fetch needed sha1 by direct fetching sha1, 2016-02-23) and fixes it as
some assumptions were not correct.

The commit states:
> If $sha1 was not part of the default fetch ... fail ourselves here
> assumes that the fetch_in_submodule only fails when the serverside does
> not support fetching by sha1.

There are other failures, why such a fetch may fail, such as
    fatal: Couldn't find remote ref HEAD
which can happen if the remote side doesn't advertise HEAD and we do not
have a local fetch refspec.

Not advertising HEAD is allowed by the protocol spec and would happen,
if HEAD points at an unborn branch for example.

Not having a local fetch refspec can happen when submodules are fetched
shallowly, as then git-clone doesn't setup a fetch refspec.

So do try even harder for a submodule by ignoring the exit code of the
first fetch and rather relying on the following is_tip_reachable to
see if we try fetching again.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 24914963ca2..00fcd69138f 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -614,7 +614,7 @@ cmd_update()
 				# is not reachable from a ref.
 				is_tip_reachable "$sm_path" "$sha1" ||
 				fetch_in_submodule "$sm_path" $depth ||
-				die "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
+				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
 
 				# Now we tried the usual fetch, but $sha1 may
 				# not be reachable from any of the refs
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 20%]

* [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive
@ 2018-05-15 20:00 Stefan Beller
  2018-05-15 20:00 ` [PATCH] git-submodule.sh: try harder to fetch a submodule Stefan Beller
                   ` (5 more replies)
  0 siblings, 6 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:00 UTC (permalink / raw)
  To: git, leif.middelschulte; +Cc: gitster, newren, Stefan Beller

This rerolls the two commits found at [1] with the feedback of Eliah
and puts Leifs patch[2] on top, that I edited according to Eliahs feedback,
but kept Leifs ownership. 

This has addressed all of Eliahs feedback AFAICT.
You'll find a branch-diff below[3], which lacks
the new patch of Leif in that series, but is part of the reroll?

Leif, what do you think?

Thanks,
Stefan

[1] https://public-inbox.org/git/20180510211917.138518-1-sbeller@google.com/
[2] https://public-inbox.org/git/20180514205737.21313-2-leif.middelschulte@gmail.com/
[3] git branch-diff origin/master..origin/sb/submodule-merge-in-merge-recursive origin/master..HEAD  >>0000-cover-letter.patch

Leif Middelschulte (1):
  Inform about fast-forwarding of submodules during merge

Stefan Beller (2):
  submodule.c: move submodule merging to merge-recursive.c
  merge-recursive: i18n submodule merge output and respect verbosity

 merge-recursive.c | 185 +++++++++++++++++++++++++++++++++++++++++++++-
 submodule.c       | 168 +----------------------------------------
 submodule.h       |   6 +-
 3 files changed, 186 insertions(+), 173 deletions(-)

-- 
2.17.0.582.gccdcbd54c44.dirty



1:  e022c7976ae ! 1:  3b638ccac64 submodule.c: move submodule merging to merge-recursive.c
    @@ -20,7 +20,6 @@
         This commit is best viewed with --color-moved.
     
         Signed-off-by: Stefan Beller <sbeller@google.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
     diff --git a/merge-recursive.c b/merge-recursive.c
     --- a/merge-recursive.c
2:  2c02ece7e01 ! 2:  eb43110df9d merge-recursive: i18n submodule merge output and respect verbosity
    @@ -7,7 +7,6 @@
         internationalisation as well as the verbosity setting.
     
         Signed-off-by: Stefan Beller <sbeller@google.com>
    -    Signed-off-by: Junio C Hamano <gitster@pobox.com>
     
     diff --git a/merge-recursive.c b/merge-recursive.c
     --- a/merge-recursive.c
    @@ -73,10 +72,10 @@
     -		fprintf(stderr, "Found a possible merge resolution "
     -				"for the submodule:\n");
     +		output(o, 1, _("Failed to merge submodule %s (not fast-forward)"), path);
    -+		output(o, 1, _("Found a possible merge resolution for the submodule:\n"));
    ++		output(o, 2, _("Found a possible merge resolution for the submodule:\n"));
      		print_commit((struct commit *) merges.objects[0].item);
     -		fprintf(stderr,
    -+		output(o, 1, _(
    ++		output(o, 2, _(
      			"If this is correct simply add it to the index "
      			"for example\n"
      			"by using:\n\n"
-:  ----------- > 3:  4a3bc435023 Inform about fast-forwarding of submodules during merge

^ permalink raw reply	[relevance 11%]

* [PATCH] git-submodule.sh: try harder to fetch a submodule
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
@ 2018-05-15 20:00 ` Stefan Beller
  2018-05-15 20:00 ` [PATCH 1/3] submodule.c: move submodule merging to merge-recursive.c Stefan Beller
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:00 UTC (permalink / raw)
  To: git, leif.middelschulte; +Cc: gitster, newren, Stefan Beller

This is the logical continuum of fb43e31f2b4 (submodule: try harder to
fetch needed sha1 by direct fetching sha1, 2016-02-23) and fixes it as
some assumptions were not correct.

The commit states:
> If $sha1 was not part of the default fetch ... fail ourselves here
> assumes that the fetch_in_submodule only fails when the serverside does
> not support fetching by sha1.

There are other failures, why such a fetch may fail, such as
    fatal: Couldn't find remote ref HEAD
which can happen if the remote side doesn't advertise HEAD and we do not
have a local fetch refspec.

Not advertising HEAD is allowed by the protocol spec and would happen,
if HEAD points at an unborn branch for example.

Not having a local fetch refspec can happen when submodules are fetched
shallowly, as then git-clone doesn't setup a fetch refspec.

So do try even harder for a submodule by ignoring the exit code of the
first fetch and rather relying on the following is_tip_reachable to
see if we try fetching again.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 git-submodule.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/git-submodule.sh b/git-submodule.sh
index 24914963ca2..00fcd69138f 100755
--- a/git-submodule.sh
+++ b/git-submodule.sh
@@ -614,7 +614,7 @@ cmd_update()
 				# is not reachable from a ref.
 				is_tip_reachable "$sm_path" "$sha1" ||
 				fetch_in_submodule "$sm_path" $depth ||
-				die "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
+				say "$(eval_gettext "Unable to fetch in submodule path '\$displaypath'")"
 
 				# Now we tried the usual fetch, but $sha1 may
 				# not be reachable from any of the refs
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 19%]

* [PATCH 2/3] merge-recursive: i18n submodule merge output and respect verbosity
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
  2018-05-15 20:00 ` [PATCH] git-submodule.sh: try harder to fetch a submodule Stefan Beller
  2018-05-15 20:00 ` [PATCH 1/3] submodule.c: move submodule merging to merge-recursive.c Stefan Beller
@ 2018-05-15 20:00 ` Stefan Beller
  2018-05-16  1:17   ` Junio C Hamano
  2018-05-15 20:00 ` [PATCH 3/3] Inform about fast-forwarding of submodules during merge Stefan Beller
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 200+ results
From: Stefan Beller @ 2018-05-15 20:00 UTC (permalink / raw)
  To: git, leif.middelschulte; +Cc: gitster, newren, Stefan Beller

The submodule merge code now uses the output() function that is used by
all the rest of the merge-recursive-code. This allows for respecting
internationalisation as well as the verbosity setting.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 merge-recursive.c | 33 +++++++++++++++------------------
 1 file changed, 15 insertions(+), 18 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 700ba15bf88..0571919ee0a 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1048,18 +1048,17 @@ static void print_commit(struct commit *commit)
 	strbuf_release(&sb);
 }
 
-#define MERGE_WARNING(path, msg) \
-	warning("Failed to merge submodule %s (%s)", path, msg);
-
-static int merge_submodule(struct object_id *result, const char *path,
+static int merge_submodule(struct merge_options *o,
+			   struct object_id *result, const char *path,
 			   const struct object_id *base, const struct object_id *a,
-			   const struct object_id *b, int search)
+			   const struct object_id *b)
 {
 	struct commit *commit_base, *commit_a, *commit_b;
 	int parent_count;
 	struct object_array merges;
 
 	int i;
+	int search = !o->call_depth;
 
 	/* store a in result in case we fail */
 	oidcpy(result, a);
@@ -1073,21 +1072,21 @@ static int merge_submodule(struct object_id *result, const char *path,
 		return 0;
 
 	if (add_submodule_odb(path)) {
-		MERGE_WARNING(path, "not checked out");
+		output(o, 1, _("Failed to merge submodule %s (not checked out)"), path);
 		return 0;
 	}
 
 	if (!(commit_base = lookup_commit_reference(base)) ||
 	    !(commit_a = lookup_commit_reference(a)) ||
 	    !(commit_b = lookup_commit_reference(b))) {
-		MERGE_WARNING(path, "commits not present");
+		output(o, 1, _("Failed to merge submodule %s (commits not present)"), path);
 		return 0;
 	}
 
 	/* check whether both changes are forward */
 	if (!in_merge_bases(commit_base, commit_a) ||
 	    !in_merge_bases(commit_base, commit_b)) {
-		MERGE_WARNING(path, "commits don't follow merge-base");
+		output(o, 1, _("Failed to merge submodule %s (commits don't follow merge-base)"), path);
 		return 0;
 	}
 
@@ -1116,25 +1115,24 @@ static int merge_submodule(struct object_id *result, const char *path,
 	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
 	switch (parent_count) {
 	case 0:
-		MERGE_WARNING(path, "merge following commits not found");
+		output(o, 1, _("Failed to merge submodule %s (merge following commits not found)"), path);
 		break;
 
 	case 1:
-		MERGE_WARNING(path, "not fast-forward");
-		fprintf(stderr, "Found a possible merge resolution "
-				"for the submodule:\n");
+		output(o, 1, _("Failed to merge submodule %s (not fast-forward)"), path);
+		output(o, 2, _("Found a possible merge resolution for the submodule:\n"));
 		print_commit((struct commit *) merges.objects[0].item);
-		fprintf(stderr,
+		output(o, 2, _(
 			"If this is correct simply add it to the index "
 			"for example\n"
 			"by using:\n\n"
 			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
-			"which will accept this suggestion.\n",
+			"which will accept this suggestion.\n"),
 			oid_to_hex(&merges.objects[0].item->oid), path);
 		break;
 
 	default:
-		MERGE_WARNING(path, "multiple merges found");
+		output(o, 1, _("Failed to merge submodule %s (multiple merges found)"), path);
 		for (i = 0; i < merges.nr; i++)
 			print_commit((struct commit *) merges.objects[i].item);
 	}
@@ -1205,12 +1203,11 @@ static int merge_file_1(struct merge_options *o,
 				return ret;
 			result->clean = (merge_status == 0);
 		} else if (S_ISGITLINK(a->mode)) {
-			result->clean = merge_submodule(&result->oid,
+			result->clean = merge_submodule(o, &result->oid,
 						       one->path,
 						       &one->oid,
 						       &a->oid,
-						       &b->oid,
-						       !o->call_depth);
+						       &b->oid);
 		} else if (S_ISLNK(a->mode)) {
 			switch (o->recursive_variant) {
 			case MERGE_RECURSIVE_NORMAL:
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 15%]

* [PATCH 3/3] Inform about fast-forwarding of submodules during merge
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
                   ` (2 preceding siblings ...)
  2018-05-15 20:00 ` [PATCH 2/3] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
@ 2018-05-15 20:00 ` Stefan Beller
  2018-05-16  1:36   ` Elijah Newren
                     ` (2 more replies)
  2018-05-15 20:02 ` [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
  2018-05-15 20:15 ` Leif Middelschulte
  5 siblings, 3 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:00 UTC (permalink / raw)
  To: git, leif.middelschulte
  Cc: gitster, newren, Leif Middelschulte, Stefan Beller

From: Leif Middelschulte <Leif.Middelschulte@gmail.com>

Inform the user about an automatically fast-forwarded submodule. The
silent merge behavior was introduced by commit 68d03e4a6e44 ("Implement
automatic fast-forward merge for submodules", 2010-07-07)).

Signed-off-by: Leif Middelschulte <Leif.Middelschulte@gmail.com>
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 merge-recursive.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/merge-recursive.c b/merge-recursive.c
index 0571919ee0a..29a430c418a 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -1093,10 +1093,26 @@ static int merge_submodule(struct merge_options *o,
 	/* Case #1: a is contained in b or vice versa */
 	if (in_merge_bases(commit_a, commit_b)) {
 		oidcpy(result, b);
+		if (show(o, 3)) {
+			output(o, 1, _("Fast-forwarding submodule %s to the following commit:"), path);
+			output_commit_title(o, commit_b);
+		} else if (show(o, 2))
+			output(o, 2, _("Fast-forwarding submodule %s to %s"), path, oid_to_hex(b));
+		else
+			; /* no output */
+
 		return 1;
 	}
 	if (in_merge_bases(commit_b, commit_a)) {
 		oidcpy(result, a);
+		if (show(o, 3)) {
+			output(o, 1, _("Fast-forwarding submodule %s to the following commit:"), path);
+			output_commit_title(o, commit_a);
+		} else if (show(o, 2))
+			output(o, 2, _("Fast-forwarding submodule %s to %s"), path, oid_to_hex(a));
+		else
+			; /* no output */
+
 		return 1;
 	}
 
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 14%]

* [PATCH 1/3] submodule.c: move submodule merging to merge-recursive.c
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
  2018-05-15 20:00 ` [PATCH] git-submodule.sh: try harder to fetch a submodule Stefan Beller
@ 2018-05-15 20:00 ` Stefan Beller
  2018-05-15 20:00 ` [PATCH 2/3] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:00 UTC (permalink / raw)
  To: git, leif.middelschulte; +Cc: gitster, newren, Stefan Beller

In a later patch we want to improve submodule merging by using the output()
function in merge-recursive.c for submodule merges to deliver a consistent
UI to users.

To do so we could either make the output() function globally available
so we can use it in submodule.c#merge_submodule(), or we could integrate
the submodule merging into the merging code. Choose the later as we
generally want to move submodules closer into the core.

Therefore we move any function related to merging submodules
(merge_submodule(), find_first_merges() and print_commit) to
merge-recursive.c.  We'll keep add_submodule_odb() in submodule.c as it
is used by other submodule functions. While at it, add a TODO note that
we do not really like the function add_submodule_odb().

This commit is best viewed with --color-moved.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 merge-recursive.c | 166 +++++++++++++++++++++++++++++++++++++++++++++
 submodule.c       | 168 +---------------------------------------------
 submodule.h       |   6 +-
 3 files changed, 170 insertions(+), 170 deletions(-)

diff --git a/merge-recursive.c b/merge-recursive.c
index 0c0d48624da..700ba15bf88 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -23,6 +23,7 @@
 #include "merge-recursive.h"
 #include "dir.h"
 #include "submodule.h"
+#include "revision.h"
 
 struct path_hashmap_entry {
 	struct hashmap_entry e;
@@ -977,6 +978,171 @@ static int merge_3way(struct merge_options *o,
 	return merge_status;
 }
 
+static int find_first_merges(struct object_array *result, const char *path,
+		struct commit *a, struct commit *b)
+{
+	int i, j;
+	struct object_array merges = OBJECT_ARRAY_INIT;
+	struct commit *commit;
+	int contains_another;
+
+	char merged_revision[42];
+	const char *rev_args[] = { "rev-list", "--merges", "--ancestry-path",
+				   "--all", merged_revision, NULL };
+	struct rev_info revs;
+	struct setup_revision_opt rev_opts;
+
+	memset(result, 0, sizeof(struct object_array));
+	memset(&rev_opts, 0, sizeof(rev_opts));
+
+	/* get all revisions that merge commit a */
+	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
+			oid_to_hex(&a->object.oid));
+	init_revisions(&revs, NULL);
+	rev_opts.submodule = path;
+	/* FIXME: can't handle linked worktrees in submodules yet */
+	revs.single_worktree = path != NULL;
+	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
+
+	/* save all revisions from the above list that contain b */
+	if (prepare_revision_walk(&revs))
+		die("revision walk setup failed");
+	while ((commit = get_revision(&revs)) != NULL) {
+		struct object *o = &(commit->object);
+		if (in_merge_bases(b, commit))
+			add_object_array(o, NULL, &merges);
+	}
+	reset_revision_walk();
+
+	/* Now we've got all merges that contain a and b. Prune all
+	 * merges that contain another found merge and save them in
+	 * result.
+	 */
+	for (i = 0; i < merges.nr; i++) {
+		struct commit *m1 = (struct commit *) merges.objects[i].item;
+
+		contains_another = 0;
+		for (j = 0; j < merges.nr; j++) {
+			struct commit *m2 = (struct commit *) merges.objects[j].item;
+			if (i != j && in_merge_bases(m2, m1)) {
+				contains_another = 1;
+				break;
+			}
+		}
+
+		if (!contains_another)
+			add_object_array(merges.objects[i].item, NULL, result);
+	}
+
+	object_array_clear(&merges);
+	return result->nr;
+}
+
+static void print_commit(struct commit *commit)
+{
+	struct strbuf sb = STRBUF_INIT;
+	struct pretty_print_context ctx = {0};
+	ctx.date_mode.type = DATE_NORMAL;
+	format_commit_message(commit, " %h: %m %s", &sb, &ctx);
+	fprintf(stderr, "%s\n", sb.buf);
+	strbuf_release(&sb);
+}
+
+#define MERGE_WARNING(path, msg) \
+	warning("Failed to merge submodule %s (%s)", path, msg);
+
+static int merge_submodule(struct object_id *result, const char *path,
+			   const struct object_id *base, const struct object_id *a,
+			   const struct object_id *b, int search)
+{
+	struct commit *commit_base, *commit_a, *commit_b;
+	int parent_count;
+	struct object_array merges;
+
+	int i;
+
+	/* store a in result in case we fail */
+	oidcpy(result, a);
+
+	/* we can not handle deletion conflicts */
+	if (is_null_oid(base))
+		return 0;
+	if (is_null_oid(a))
+		return 0;
+	if (is_null_oid(b))
+		return 0;
+
+	if (add_submodule_odb(path)) {
+		MERGE_WARNING(path, "not checked out");
+		return 0;
+	}
+
+	if (!(commit_base = lookup_commit_reference(base)) ||
+	    !(commit_a = lookup_commit_reference(a)) ||
+	    !(commit_b = lookup_commit_reference(b))) {
+		MERGE_WARNING(path, "commits not present");
+		return 0;
+	}
+
+	/* check whether both changes are forward */
+	if (!in_merge_bases(commit_base, commit_a) ||
+	    !in_merge_bases(commit_base, commit_b)) {
+		MERGE_WARNING(path, "commits don't follow merge-base");
+		return 0;
+	}
+
+	/* Case #1: a is contained in b or vice versa */
+	if (in_merge_bases(commit_a, commit_b)) {
+		oidcpy(result, b);
+		return 1;
+	}
+	if (in_merge_bases(commit_b, commit_a)) {
+		oidcpy(result, a);
+		return 1;
+	}
+
+	/*
+	 * Case #2: There are one or more merges that contain a and b in
+	 * the submodule. If there is only one, then present it as a
+	 * suggestion to the user, but leave it marked unmerged so the
+	 * user needs to confirm the resolution.
+	 */
+
+	/* Skip the search if makes no sense to the calling context.  */
+	if (!search)
+		return 0;
+
+	/* find commit which merges them */
+	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
+	switch (parent_count) {
+	case 0:
+		MERGE_WARNING(path, "merge following commits not found");
+		break;
+
+	case 1:
+		MERGE_WARNING(path, "not fast-forward");
+		fprintf(stderr, "Found a possible merge resolution "
+				"for the submodule:\n");
+		print_commit((struct commit *) merges.objects[0].item);
+		fprintf(stderr,
+			"If this is correct simply add it to the index "
+			"for example\n"
+			"by using:\n\n"
+			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
+			"which will accept this suggestion.\n",
+			oid_to_hex(&merges.objects[0].item->oid), path);
+		break;
+
+	default:
+		MERGE_WARNING(path, "multiple merges found");
+		for (i = 0; i < merges.nr; i++)
+			print_commit((struct commit *) merges.objects[i].item);
+	}
+
+	object_array_clear(&merges);
+	return 0;
+}
+
 static int merge_file_1(struct merge_options *o,
 					   const struct diff_filespec *one,
 					   const struct diff_filespec *a,
diff --git a/submodule.c b/submodule.c
index 74d35b25779..654089b3647 100644
--- a/submodule.c
+++ b/submodule.c
@@ -153,7 +153,8 @@ void stage_updated_gitmodules(struct index_state *istate)
 		die(_("staging updated .gitmodules failed"));
 }
 
-static int add_submodule_odb(const char *path)
+/* TODO: remove this function, use repo_submodule_init instead. */
+int add_submodule_odb(const char *path)
 {
 	struct strbuf objects_directory = STRBUF_INIT;
 	int ret = 0;
@@ -1701,171 +1702,6 @@ int submodule_move_head(const char *path,
 	return ret;
 }
 
-static int find_first_merges(struct object_array *result, const char *path,
-		struct commit *a, struct commit *b)
-{
-	int i, j;
-	struct object_array merges = OBJECT_ARRAY_INIT;
-	struct commit *commit;
-	int contains_another;
-
-	char merged_revision[42];
-	const char *rev_args[] = { "rev-list", "--merges", "--ancestry-path",
-				   "--all", merged_revision, NULL };
-	struct rev_info revs;
-	struct setup_revision_opt rev_opts;
-
-	memset(result, 0, sizeof(struct object_array));
-	memset(&rev_opts, 0, sizeof(rev_opts));
-
-	/* get all revisions that merge commit a */
-	xsnprintf(merged_revision, sizeof(merged_revision), "^%s",
-			oid_to_hex(&a->object.oid));
-	init_revisions(&revs, NULL);
-	rev_opts.submodule = path;
-	/* FIXME: can't handle linked worktrees in submodules yet */
-	revs.single_worktree = path != NULL;
-	setup_revisions(ARRAY_SIZE(rev_args)-1, rev_args, &revs, &rev_opts);
-
-	/* save all revisions from the above list that contain b */
-	if (prepare_revision_walk(&revs))
-		die("revision walk setup failed");
-	while ((commit = get_revision(&revs)) != NULL) {
-		struct object *o = &(commit->object);
-		if (in_merge_bases(b, commit))
-			add_object_array(o, NULL, &merges);
-	}
-	reset_revision_walk();
-
-	/* Now we've got all merges that contain a and b. Prune all
-	 * merges that contain another found merge and save them in
-	 * result.
-	 */
-	for (i = 0; i < merges.nr; i++) {
-		struct commit *m1 = (struct commit *) merges.objects[i].item;
-
-		contains_another = 0;
-		for (j = 0; j < merges.nr; j++) {
-			struct commit *m2 = (struct commit *) merges.objects[j].item;
-			if (i != j && in_merge_bases(m2, m1)) {
-				contains_another = 1;
-				break;
-			}
-		}
-
-		if (!contains_another)
-			add_object_array(merges.objects[i].item, NULL, result);
-	}
-
-	object_array_clear(&merges);
-	return result->nr;
-}
-
-static void print_commit(struct commit *commit)
-{
-	struct strbuf sb = STRBUF_INIT;
-	struct pretty_print_context ctx = {0};
-	ctx.date_mode.type = DATE_NORMAL;
-	format_commit_message(commit, " %h: %m %s", &sb, &ctx);
-	fprintf(stderr, "%s\n", sb.buf);
-	strbuf_release(&sb);
-}
-
-#define MERGE_WARNING(path, msg) \
-	warning("Failed to merge submodule %s (%s)", path, msg);
-
-int merge_submodule(struct object_id *result, const char *path,
-		    const struct object_id *base, const struct object_id *a,
-		    const struct object_id *b, int search)
-{
-	struct commit *commit_base, *commit_a, *commit_b;
-	int parent_count;
-	struct object_array merges;
-
-	int i;
-
-	/* store a in result in case we fail */
-	oidcpy(result, a);
-
-	/* we can not handle deletion conflicts */
-	if (is_null_oid(base))
-		return 0;
-	if (is_null_oid(a))
-		return 0;
-	if (is_null_oid(b))
-		return 0;
-
-	if (add_submodule_odb(path)) {
-		MERGE_WARNING(path, "not checked out");
-		return 0;
-	}
-
-	if (!(commit_base = lookup_commit_reference(base)) ||
-	    !(commit_a = lookup_commit_reference(a)) ||
-	    !(commit_b = lookup_commit_reference(b))) {
-		MERGE_WARNING(path, "commits not present");
-		return 0;
-	}
-
-	/* check whether both changes are forward */
-	if (!in_merge_bases(commit_base, commit_a) ||
-	    !in_merge_bases(commit_base, commit_b)) {
-		MERGE_WARNING(path, "commits don't follow merge-base");
-		return 0;
-	}
-
-	/* Case #1: a is contained in b or vice versa */
-	if (in_merge_bases(commit_a, commit_b)) {
-		oidcpy(result, b);
-		return 1;
-	}
-	if (in_merge_bases(commit_b, commit_a)) {
-		oidcpy(result, a);
-		return 1;
-	}
-
-	/*
-	 * Case #2: There are one or more merges that contain a and b in
-	 * the submodule. If there is only one, then present it as a
-	 * suggestion to the user, but leave it marked unmerged so the
-	 * user needs to confirm the resolution.
-	 */
-
-	/* Skip the search if makes no sense to the calling context.  */
-	if (!search)
-		return 0;
-
-	/* find commit which merges them */
-	parent_count = find_first_merges(&merges, path, commit_a, commit_b);
-	switch (parent_count) {
-	case 0:
-		MERGE_WARNING(path, "merge following commits not found");
-		break;
-
-	case 1:
-		MERGE_WARNING(path, "not fast-forward");
-		fprintf(stderr, "Found a possible merge resolution "
-				"for the submodule:\n");
-		print_commit((struct commit *) merges.objects[0].item);
-		fprintf(stderr,
-			"If this is correct simply add it to the index "
-			"for example\n"
-			"by using:\n\n"
-			"  git update-index --cacheinfo 160000 %s \"%s\"\n\n"
-			"which will accept this suggestion.\n",
-			oid_to_hex(&merges.objects[0].item->oid), path);
-		break;
-
-	default:
-		MERGE_WARNING(path, "multiple merges found");
-		for (i = 0; i < merges.nr; i++)
-			print_commit((struct commit *) merges.objects[i].item);
-	}
-
-	object_array_clear(&merges);
-	return 0;
-}
-
 /*
  * Embeds a single submodules git directory into the superprojects git dir,
  * non recursively.
diff --git a/submodule.h b/submodule.h
index e5526f6aaab..b96689ac0db 100644
--- a/submodule.h
+++ b/submodule.h
@@ -89,10 +89,8 @@ extern int submodule_uses_gitfile(const char *path);
 #define SUBMODULE_REMOVAL_IGNORE_UNTRACKED (1<<1)
 #define SUBMODULE_REMOVAL_IGNORE_IGNORED_UNTRACKED (1<<2)
 extern int bad_to_remove_submodule(const char *path, unsigned flags);
-extern int merge_submodule(struct object_id *result, const char *path,
-			   const struct object_id *base,
-			   const struct object_id *a,
-			   const struct object_id *b, int search);
+
+int add_submodule_odb(const char *path);
 
 /* Checks if there are submodule changes in a..b. */
 extern int submodule_touches_in_range(struct object_id *a,
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 18%]

* Re: [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
                   ` (3 preceding siblings ...)
  2018-05-15 20:00 ` [PATCH 3/3] Inform about fast-forwarding of submodules during merge Stefan Beller
@ 2018-05-15 20:02 ` Stefan Beller
  2018-05-15 20:15 ` Leif Middelschulte
  5 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:02 UTC (permalink / raw)
  To: git, Leif Middelschulte; +Cc: Junio C Hamano, Elijah Newren, Stefan Beller

And I resent two of my earlier patches, please ignore those
(0001-grep-handle-corrupt-index-files-early.patch and
0001-git-submodule.sh-try-harder-to-fetch-a-submodule.patch)

Stefan

^ permalink raw reply	[relevance 8%]

* Re: [PATCH] git-submodule.sh: try harder to fetch a submodule
  2018-05-15 19:07     ` Stefan Beller
  2018-05-15 19:40       ` Stefan Beller
@ 2018-05-15 20:04       ` Jonathan Nieder
  1 sibling, 0 replies; 200+ results
From: Jonathan Nieder @ 2018-05-15 20:04 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Junio C Hamano, git

Stefan Beller wrote:

> I'll resend it with a warning (using say()).

Thanks, makes sense.

> I think we have 2 bugs and this is merely fixing the second bug.

I'm fearing that there are more than two.

[...]
>   $ git init confused-head
>   $ (cd confused-head && git branch test \
>         $(git commit-tree $(git write-tree) -m test))
>   $ git clone --no-checkout  --depth=1 \
>         --separate-git-dir=test.git confused-head/.git test
> Cloning into 'test'...
> warning: --depth is ignored in local clones; use file:// instead.
> done.
>
>   $ git -C test.git config remote.origin.fetch
>   $ echo $?
> 1
>
> (A) Despite the warning of --depth having no impact, the
>   omission thereof changes the repository state.
> (B) There is no remote.origin.fetch configuration, which
>   is weird. See builtin/clone.c:830, that states for this case:

I can reproduce the issue without submodules and without --local,
as follows:

	git init --bare empty.git
	git init --bare almost-empty.git
	git -C ~/src/git push $(pwd)/almost-empty HEAD:refs/heads/upstream

	git clone --single-branch file://$(pwd)/empty.git
	git clone --single-branch file://$(pwd)/almost-empty.git

	git -C almost-empty.git branch -D upstream

	git -C empty fetch
	git -C almost-empty fetch

Expected result:
Both fetches succeed.

Actual result:
First fetch succeeds, second produces
"fatal: Couldn't find remote ref HEAD".

Note that empty.git and almost-empty.git are basically identical.
The difference instead lies in the clones' .git/config files:

diff --git 1/empty/.git/config 2/almost-empty/.git/config
index b51bb0d..ee21198 100644
--- 1/empty/.git/config
+++ 2/almost-empty/.git/config
@@ -4,7 +4,4 @@
        bare = false
        logallrefupdates = true
 [remote "origin"]
-       url = file:///tmp/t/empty.git
-[branch "master"]
-       remote = origin
-       merge = refs/heads/master
+       url = file:///tmp/t/almost-empty.git

Thanks,
Jonathan

^ permalink raw reply	[relevance 7%]

* Re: [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive
  2018-05-15 20:00 [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
                   ` (4 preceding siblings ...)
  2018-05-15 20:02 ` [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive Stefan Beller
@ 2018-05-15 20:15 ` Leif Middelschulte
  2018-05-15 20:49   ` Stefan Beller
  5 siblings, 1 reply; 200+ results
From: Leif Middelschulte @ 2018-05-15 20:15 UTC (permalink / raw)
  To: Stefan Beller, git; +Cc: newren, gitster

Hello Stefan,

thank you once again for your effort.

Am 15. Mai 2018 um 22:00:34, Stefan Beller
(sbeller@google.com(mailto:sbeller@google.com)) schrieb:

> This rerolls the two commits found at [1] with the feedback of Eliah
> and puts Leifs patch[2] on top, that I edited according to Eliahs feedback,
> but kept Leifs ownership.
>
> This has addressed all of Eliahs feedback AFAICT.
> You'll find a branch-diff below[3], which lacks
> the new patch of Leif in that series, but is part of the reroll?
>
> Leif, what do you think?

Seems great to me. Thank you for picking up and improving my changes :)
One Question though: Shouldn’t an enum (like
NOTES_MERGE_VERBOSITY_DEFAULT) be used instead of numbers?


Cheers,


Leif

>
> Thanks,
> Stefan
>
> [1] https://public-inbox.org/git/20180510211917.138518-1-sbeller@google.com/
> [2] https://public-inbox.org/git/20180514205737.21313-2-leif.middelschulte@gmail.com/
> [3] git branch-diff origin/master..origin/sb/submodule-merge-in-merge-recursive origin/master..HEAD >>0000-cover-letter.patch
>
> Leif Middelschulte (1):
> Inform about fast-forwarding of submodules during merge
>
> Stefan Beller (2):
> submodule.c: move submodule merging to merge-recursive.c
> merge-recursive: i18n submodule merge output and respect verbosity
>
> merge-recursive.c | 185 +++++++++++++++++++++++++++++++++++++++++++++-
> submodule.c | 168 +----------------------------------------
> submodule.h | 6 +-
> 3 files changed, 186 insertions(+), 173 deletions(-)
>
> --
> 2.17.0.582.gccdcbd54c44.dirty
>
>
>
> 1: e022c7976ae ! 1: 3b638ccac64 submodule.c: move submodule merging to merge-recursive.c
> @@ -20,7 +20,6 @@
> This commit is best viewed with --color-moved.
>
> Signed-off-by: Stefan Beller
> - Signed-off-by: Junio C Hamano
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> --- a/merge-recursive.c
> 2: 2c02ece7e01 ! 2: eb43110df9d merge-recursive: i18n submodule merge output and respect verbosity
> @@ -7,7 +7,6 @@
> internationalisation as well as the verbosity setting.
>
> Signed-off-by: Stefan Beller
> - Signed-off-by: Junio C Hamano
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> --- a/merge-recursive.c
> @@ -73,10 +72,10 @@
> - fprintf(stderr, "Found a possible merge resolution "
> - "for the submodule:\n");
> + output(o, 1, _("Failed to merge submodule %s (not fast-forward)"), path);
> -+ output(o, 1, _("Found a possible merge resolution for the submodule:\n"));
> ++ output(o, 2, _("Found a possible merge resolution for the submodule:\n"));
> print_commit((struct commit *) merges.objects[0].item);
> - fprintf(stderr,
> -+ output(o, 1, _(
> ++ output(o, 2, _(
> "If this is correct simply add it to the index "
> "for example\n"
> "by using:\n\n"
> -: ----------- > 3: 4a3bc435023 Inform about fast-forwarding of submodules during merge

^ permalink raw reply	[relevance 5%]

* Re: [PATCHv2 0/3] Reroll of sb/submodule-merge-in-merge-recursive
  2018-05-15 20:15 ` Leif Middelschulte
@ 2018-05-15 20:49   ` Stefan Beller
  0 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 20:49 UTC (permalink / raw)
  To: Leif Middelschulte; +Cc: git, Elijah Newren, Junio C Hamano

On Tue, May 15, 2018 at 1:15 PM, Leif Middelschulte
<leif.middelschulte@gmail.com> wrote:
> Hello Stefan,
>
> thank you once again for your effort.
>
> Am 15. Mai 2018 um 22:00:34, Stefan Beller
> (sbeller@google.com(mailto:sbeller@google.com)) schrieb:
>
>> This rerolls the two commits found at [1] with the feedback of Eliah
>> and puts Leifs patch[2] on top, that I edited according to Eliahs feedback,
>> but kept Leifs ownership.
>>
>> This has addressed all of Eliahs feedback AFAICT.
>> You'll find a branch-diff below[3], which lacks
>> the new patch of Leif in that series, but is part of the reroll?
>>
>> Leif, what do you think?
>
> Seems great to me. Thank you for picking up and improving my changes :)
> One Question though: Shouldn’t an enum (like
> NOTES_MERGE_VERBOSITY_DEFAULT) be used instead of numbers?

Hah! I did not know that existed.

$ git grep NOTES_MERGE_VERBOSITY_DEFAULT
builtin/notes.c:810:    o.verbosity = verbosity + NOTES_MERGE_VERBOSITY_DEFAULT;
notes-merge.c:22:       o->verbosity = NOTES_MERGE_VERBOSITY_DEFAULT;
notes-merge.h:9:        NOTES_MERGE_VERBOSITY_DEFAULT = 2,

It doesn't seem to be used much, as opposed to numbers:

$ git grep show -- merge-recursive.c
merge-recursive.c:201:static int show(struct merge_options *o, int v)
merge-recursive.c:211:  if (!show(o, v))
merge-recursive.c:570:  opts.show_rename_progress = o->show_rename_progress;
merge-recursive.c:1096:         if (show(o, 3)) {
merge-recursive.c:1099:         } else if (show(o, 2))
merge-recursive.c:1108:         if (show(o, 3)) {
merge-recursive.c:1111:         } else if (show(o, 2))
merge-recursive.c:2178:         if (show(o, 4) || o->call_depth)
merge-recursive.c:2275: if (show(o, 4)) {
merge-recursive.c:2286: if (show(o, 5)) {
merge-recursive.c:2351: if (show(o, 2))

(The first two are the implementation of show/output, third is
somewhat unrelated to show() and all the rest is numbers).

If we'd want to use  NOTES_MERGE_VERBOSITY_DEFAULT,
I would suggest to send a followup series on top of this?

I would think numbers are fine for now.

Thanks,
Stefan

^ permalink raw reply	[relevance 4%]

* [RFC PATCH 00/19] object store: grafts and shallow.
@ 2018-05-15 23:42 Stefan Beller
  2018-05-15 23:42 ` [PATCH 01/19] object-store: move object access functions to object-store.h Stefan Beller
  2018-05-17 22:51 ` [PATCH 00/19] object store: grafts and shallow Stefan Beller
  0 siblings, 2 replies; 200+ results
From: Stefan Beller @ 2018-05-15 23:42 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

This applies on top of sb/object-store-alloc[1], and is the next part of the
object store series. 

I think we're getting close to actually being done in the object store series,
as the next series to build on top of this will convert the
lookup_{object, commit, ...} functions.

However I marked this series as RFC, as I expect heavy conflicts with the
code base. sb/object-store-alloc builds on an older base of the code base
and there have been some series that will conflict with this one.
For example the patch to migrate path functions into a repository world
will collide with one of Dschos series as he added another path helper
function there. I am happy to reroll on top of that, but for now I chose
sb/object-store-alloc to keep the momentum in the object-store series'.

There is another object store series that is not part of the critical path:
There is a mem leak in the pack files stored in the raw object store.
Upon free'ing the repository we do not free its pack files. We cannot
free the packfiles, as otherwise the bitmap code would have dangling
pointers into packfiles that no longer exists, leading to segfaults.
So we'll need to have an object store series covering the bitmap code.
I have something local, which I'll send out shortly.

Thanks,
Stefan

[1] with the latest patch replaced as in
    https://public-inbox.org/git/20180515214842.108713-1-sbeller@google.com/

Brandon Williams (3):
  commit: convert commit_graft_pos() to handle arbitrary repositories
  commit: convert register_commit_graft to handle arbitrary repositories
  commit: convert read_graft_file to handle arbitrary repositories

Jonathan Nieder (6):
  object: move grafts to object parser
  commit: add repository argument to commit_graft_pos
  commit: add repository argument to register_commit_graft
  commit: add repository argument to read_graft_file
  commit: add repository argument to prepare_commit_graft
  commit: add repository argument to lookup_commit_graft

Stefan Beller (10):
  object-store: move object access functions to object-store.h
  shallow: add repository argument to set_alternate_shallow_file
  shallow: add repository argument to register_shallow
  shallow: add repository argument to check_shallow_file_for_update
  shallow: add repository argument to is_repository_shallow
  cache: convert get_graft_file to handle arbitrary repositories
  path.c: migrate git_path_ to take a repository argument
  shallow: migrate shallow information into the object parser
  commit: allow prepare_commit_graft to handle arbitrary repositories
  commit: allow lookup_commit_graft to handle arbitrary repositories

 apply.c                  |   1 +
 archive-tar.c            |   1 +
 archive-zip.c            |   1 +
 archive.c                |   1 +
 blame.c                  |   9 ++-
 branch.c                 |  14 ++---
 builtin/blame.c          |   4 +-
 builtin/cat-file.c       |   1 +
 builtin/checkout.c       |   1 +
 builtin/clone.c          |   1 +
 builtin/commit-tree.c    |   1 +
 builtin/commit.c         |  38 ++++++-------
 builtin/describe.c       |   1 +
 builtin/difftool.c       |   1 +
 builtin/fast-export.c    |   1 +
 builtin/fetch.c          |   7 ++-
 builtin/fmt-merge-msg.c  |   1 +
 builtin/hash-object.c    |   1 +
 builtin/log.c            |   1 +
 builtin/ls-tree.c        |   1 +
 builtin/merge-tree.c     |   1 +
 builtin/merge.c          |  37 ++++++------
 builtin/mktag.c          |   1 +
 builtin/mktree.c         |   1 +
 builtin/notes.c          |   1 +
 builtin/pack-objects.c   |   6 +-
 builtin/prune.c          |   3 +-
 builtin/pull.c           |   4 +-
 builtin/receive-pack.c   |   3 +-
 builtin/reflog.c         |   1 +
 builtin/remote.c         |   1 +
 builtin/reset.c          |   2 +-
 builtin/rev-list.c       |   1 +
 builtin/rev-parse.c      |   3 +-
 builtin/show-ref.c       |   1 +
 builtin/tag.c            |   1 +
 builtin/unpack-file.c    |   1 +
 builtin/unpack-objects.c |   1 +
 builtin/verify-commit.c  |   1 +
 bulk-checkin.c           |   1 +
 bundle.c                 |   1 +
 cache-tree.c             |   1 +
 cache.h                  | 119 +--------------------------------------
 combine-diff.c           |   1 +
 commit.c                 |  77 +++++++++++++------------
 commit.h                 |  10 ++--
 config.c                 |   1 +
 convert.c                |   1 +
 diff.c                   |   1 +
 diffcore-rename.c        |   1 +
 dir.c                    |   1 +
 entry.c                  |   1 +
 environment.c            |   8 +--
 fetch-pack.c             |   9 +--
 fsck.c                   |   3 +-
 git.c                    |   2 +-
 grep.c                   |   1 +
 list-objects-filter.c    |   1 +
 list-objects.c           |   1 +
 log-tree.c               |   1 +
 mailmap.c                |   1 +
 match-trees.c            |   1 +
 merge-blobs.c            |   1 +
 merge-recursive.c        |   1 +
 notes-cache.c            |   1 +
 notes-merge.c            |   1 +
 notes.c                  |   1 +
 object-store.h           | 117 ++++++++++++++++++++++++++++++++++++++
 object.c                 |   4 ++
 object.h                 |  10 ++++
 pack-bitmap-write.c      |   1 +
 packfile.h               |   5 ++
 path.c                   |  18 +++---
 path.h                   |  40 ++++++++++---
 read-cache.c             |   1 +
 ref-filter.c             |   1 +
 refs.c                   |   1 +
 remote-testsvn.c         |   1 +
 remote.c                 |   1 +
 repository.h             |   5 ++
 rerere.c                 |   8 ++-
 revision.c               |   1 +
 send-pack.c              |   7 ++-
 sequencer.c              |  38 +++++++------
 shallow.c                |  74 ++++++++++++------------
 submodule-config.c       |   1 +
 tag.c                    |   1 +
 tree-walk.c              |   1 +
 tree.c                   |   1 +
 unpack-trees.c           |   1 +
 upload-pack.c            |  10 ++--
 walker.c                 |   1 +
 wt-status.c              |   8 +--
 xdiff-interface.c        |   1 +
 94 files changed, 450 insertions(+), 314 deletions(-)

-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 4%]

* [PATCH 01/19] object-store: move object access functions to object-store.h
  2018-05-15 23:42 [RFC PATCH 00/19] object store: grafts and shallow Stefan Beller
@ 2018-05-15 23:42 ` Stefan Beller
  2018-05-17 22:51 ` [PATCH 00/19] object store: grafts and shallow Stefan Beller
  1 sibling, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-15 23:42 UTC (permalink / raw)
  To: git; +Cc: Stefan Beller

This should make these functions easier to find and cache.h less
overwhelming to read.

In particular, this moves:
- read_object_file
- oid_object_info
- write_object_file

As a result, most of the codebase needs to #include object-store.h.
In this patch the #include is only added to files that would fail to
compile otherwise.  It would be better to #include wherever
identifiers from the header are used.  That can happen later
when we have better tooling for it.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 apply.c                  |   1 +
 archive-tar.c            |   1 +
 archive-zip.c            |   1 +
 archive.c                |   1 +
 blame.c                  |   1 +
 builtin/blame.c          |   1 +
 builtin/cat-file.c       |   1 +
 builtin/checkout.c       |   1 +
 builtin/clone.c          |   1 +
 builtin/commit-tree.c    |   1 +
 builtin/describe.c       |   1 +
 builtin/difftool.c       |   1 +
 builtin/fast-export.c    |   1 +
 builtin/fetch.c          |   1 +
 builtin/fmt-merge-msg.c  |   1 +
 builtin/hash-object.c    |   1 +
 builtin/log.c            |   1 +
 builtin/ls-tree.c        |   1 +
 builtin/merge-tree.c     |   1 +
 builtin/mktag.c          |   1 +
 builtin/mktree.c         |   1 +
 builtin/notes.c          |   1 +
 builtin/prune.c          |   1 +
 builtin/receive-pack.c   |   1 +
 builtin/reflog.c         |   1 +
 builtin/remote.c         |   1 +
 builtin/rev-list.c       |   1 +
 builtin/show-ref.c       |   1 +
 builtin/tag.c            |   1 +
 builtin/unpack-file.c    |   1 +
 builtin/unpack-objects.c |   1 +
 builtin/verify-commit.c  |   1 +
 bulk-checkin.c           |   1 +
 bundle.c                 |   1 +
 cache-tree.c             |   1 +
 cache.h                  | 117 ---------------------------------------
 combine-diff.c           |   1 +
 commit.c                 |   1 +
 config.c                 |   1 +
 convert.c                |   1 +
 diff.c                   |   1 +
 diffcore-rename.c        |   1 +
 dir.c                    |   1 +
 entry.c                  |   1 +
 fetch-pack.c             |   1 +
 fsck.c                   |   1 +
 grep.c                   |   1 +
 list-objects-filter.c    |   1 +
 list-objects.c           |   1 +
 log-tree.c               |   1 +
 mailmap.c                |   1 +
 match-trees.c            |   1 +
 merge-blobs.c            |   1 +
 merge-recursive.c        |   1 +
 notes-cache.c            |   1 +
 notes-merge.c            |   1 +
 notes.c                  |   1 +
 object-store.h           | 117 +++++++++++++++++++++++++++++++++++++++
 object.c                 |   1 +
 pack-bitmap-write.c      |   1 +
 packfile.h               |   5 ++
 read-cache.c             |   1 +
 ref-filter.c             |   1 +
 refs.c                   |   1 +
 remote-testsvn.c         |   1 +
 remote.c                 |   1 +
 rerere.c                 |   1 +
 revision.c               |   1 +
 send-pack.c              |   1 +
 sequencer.c              |   1 +
 shallow.c                |   1 +
 submodule-config.c       |   1 +
 tag.c                    |   1 +
 tree-walk.c              |   1 +
 tree.c                   |   1 +
 unpack-trees.c           |   1 +
 upload-pack.c            |   1 +
 walker.c                 |   1 +
 xdiff-interface.c        |   1 +
 79 files changed, 198 insertions(+), 117 deletions(-)

diff --git a/apply.c b/apply.c
index 7e5792c996f..cbc45fa1b0e 100644
--- a/apply.c
+++ b/apply.c
@@ -9,6 +9,7 @@
 
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "blob.h"
 #include "delta.h"
 #include "diff.h"
diff --git a/archive-tar.c b/archive-tar.c
index f93409324f9..e38435eb4ef 100644
--- a/archive-tar.c
+++ b/archive-tar.c
@@ -5,6 +5,7 @@
 #include "config.h"
 #include "tar.h"
 #include "archive.h"
+#include "object-store.h"
 #include "streaming.h"
 #include "run-command.h"
 
diff --git a/archive-zip.c b/archive-zip.c
index 74f3fe91034..abc556e5a75 100644
--- a/archive-zip.c
+++ b/archive-zip.c
@@ -6,6 +6,7 @@
 #include "archive.h"
 #include "streaming.h"
 #include "utf8.h"
+#include "object-store.h"
 #include "userdiff.h"
 #include "xdiff-interface.h"
 
diff --git a/archive.c b/archive.c
index 93ab175b0b4..9da1e3664a6 100644
--- a/archive.c
+++ b/archive.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree-walk.h"
 #include "attr.h"
diff --git a/blame.c b/blame.c
index 3a11f1ce52b..f689bde31cd 100644
--- a/blame.c
+++ b/blame.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "cache-tree.h"
 #include "mergesort.h"
 #include "diff.h"
diff --git a/builtin/blame.c b/builtin/blame.c
index bfdf7cc1325..0ffd1d443ea 100644
--- a/builtin/blame.c
+++ b/builtin/blame.c
@@ -22,6 +22,7 @@
 #include "line-log.h"
 #include "dir.h"
 #include "progress.h"
+#include "object-store.h"
 #include "blame.h"
 
 static char blame_usage[] = N_("git blame [<options>] [<rev-opts>] [<rev>] [--] <file>");
diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index b8ecbea98e9..91e7764243e 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -13,6 +13,7 @@
 #include "tree-walk.h"
 #include "sha1-array.h"
 #include "packfile.h"
+#include "object-store.h"
 
 struct batch_options {
 	int enabled;
diff --git a/builtin/checkout.c b/builtin/checkout.c
index b49b5820718..105e07981ff 100644
--- a/builtin/checkout.c
+++ b/builtin/checkout.c
@@ -4,6 +4,7 @@
 #include "lockfile.h"
 #include "parse-options.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "tree-walk.h"
diff --git a/builtin/clone.c b/builtin/clone.c
index 7df5932b855..29998c02ece 100644
--- a/builtin/clone.c
+++ b/builtin/clone.c
@@ -14,6 +14,7 @@
 #include "parse-options.h"
 #include "fetch-pack.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tree.h"
 #include "tree-walk.h"
 #include "unpack-trees.h"
diff --git a/builtin/commit-tree.c b/builtin/commit-tree.c
index ecf42191da1..9fbd3529fb1 100644
--- a/builtin/commit-tree.c
+++ b/builtin/commit-tree.c
@@ -5,6 +5,7 @@
  */
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "builtin.h"
diff --git a/builtin/describe.c b/builtin/describe.c
index 66c497f7896..65b0edc473c 100644
--- a/builtin/describe.c
+++ b/builtin/describe.c
@@ -13,6 +13,7 @@
 #include "hashmap.h"
 #include "argv-array.h"
 #include "run-command.h"
+#include "object-store.h"
 #include "revision.h"
 #include "list-objects.h"
 
diff --git a/builtin/difftool.c b/builtin/difftool.c
index ee8dce019e1..df7e75f797b 100644
--- a/builtin/difftool.c
+++ b/builtin/difftool.c
@@ -20,6 +20,7 @@
 #include "argv-array.h"
 #include "strbuf.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "dir.h"
 
 static char *diff_gui_tool;
diff --git a/builtin/fast-export.c b/builtin/fast-export.c
index 373c794873e..f593e57b9d4 100644
--- a/builtin/fast-export.c
+++ b/builtin/fast-export.c
@@ -7,6 +7,7 @@
 #include "cache.h"
 #include "config.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "object.h"
 #include "tag.h"
diff --git a/builtin/fetch.c b/builtin/fetch.c
index 73be393b2ea..c1f2df97965 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -5,6 +5,7 @@
 #include "config.h"
 #include "repository.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "builtin.h"
 #include "string-list.h"
diff --git a/builtin/fmt-merge-msg.c b/builtin/fmt-merge-msg.c
index bd680be6874..1b526adb3a9 100644
--- a/builtin/fmt-merge-msg.c
+++ b/builtin/fmt-merge-msg.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "config.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "diff.h"
 #include "revision.h"
diff --git a/builtin/hash-object.c b/builtin/hash-object.c
index 526da5c1856..d5c018eabd5 100644
--- a/builtin/hash-object.c
+++ b/builtin/hash-object.c
@@ -6,6 +6,7 @@
  */
 #include "builtin.h"
 #include "config.h"
+#include "object-store.h"
 #include "blob.h"
 #include "quote.h"
 #include "parse-options.h"
diff --git a/builtin/log.c b/builtin/log.c
index 71f68a3e4f5..9656578f58e 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -7,6 +7,7 @@
 #include "cache.h"
 #include "config.h"
 #include "refs.h"
+#include "object-store.h"
 #include "color.h"
 #include "commit.h"
 #include "diff.h"
diff --git a/builtin/ls-tree.c b/builtin/ls-tree.c
index 409da4e8351..fe3b952cb30 100644
--- a/builtin/ls-tree.c
+++ b/builtin/ls-tree.c
@@ -5,6 +5,7 @@
  */
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "blob.h"
 #include "tree.h"
 #include "commit.h"
diff --git a/builtin/merge-tree.c b/builtin/merge-tree.c
index 32736e0b101..1b702d44c9f 100644
--- a/builtin/merge-tree.c
+++ b/builtin/merge-tree.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "tree-walk.h"
 #include "xdiff-interface.h"
+#include "object-store.h"
 #include "blob.h"
 #include "exec_cmd.h"
 #include "merge-blobs.h"
diff --git a/builtin/mktag.c b/builtin/mktag.c
index 82a6e860775..6fb7dc8578d 100644
--- a/builtin/mktag.c
+++ b/builtin/mktag.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "tag.h"
 #include "replace-object.h"
+#include "object-store.h"
 
 /*
  * A signature file has a very simple fixed format: four lines
diff --git a/builtin/mktree.c b/builtin/mktree.c
index bb76b469fd1..2dc4ad6ba8f 100644
--- a/builtin/mktree.c
+++ b/builtin/mktree.c
@@ -7,6 +7,7 @@
 #include "quote.h"
 #include "tree.h"
 #include "parse-options.h"
+#include "object-store.h"
 
 static struct treeent {
 	unsigned mode;
diff --git a/builtin/notes.c b/builtin/notes.c
index 921e08d5bf5..73b680ee13a 100644
--- a/builtin/notes.c
+++ b/builtin/notes.c
@@ -11,6 +11,7 @@
 #include "config.h"
 #include "builtin.h"
 #include "notes.h"
+#include "object-store.h"
 #include "blob.h"
 #include "pretty.h"
 #include "refs.h"
diff --git a/builtin/prune.c b/builtin/prune.c
index 518ffbea139..8cc8659612f 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,6 +6,7 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
+#include "object-store.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--progress] [--expire <time>] [--] [<head>...]"),
diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c
index c4272fbc96d..36906fd5e98 100644
--- a/builtin/receive-pack.c
+++ b/builtin/receive-pack.c
@@ -25,6 +25,7 @@
 #include "tmp-objdir.h"
 #include "oidset.h"
 #include "packfile.h"
+#include "object-store.h"
 #include "protocol.h"
 
 static const char * const receive_pack_usage[] = {
diff --git a/builtin/reflog.c b/builtin/reflog.c
index a89bd1dd252..bee4795a936 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "config.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "commit.h"
 #include "refs.h"
 #include "dir.h"
diff --git a/builtin/remote.c b/builtin/remote.c
index 805ffc05cdb..9487c019092 100644
--- a/builtin/remote.c
+++ b/builtin/remote.c
@@ -7,6 +7,7 @@
 #include "strbuf.h"
 #include "run-command.h"
 #include "refs.h"
+#include "object-store.h"
 #include "argv-array.h"
 
 static const char * const builtin_remote_usage[] = {
diff --git a/builtin/rev-list.c b/builtin/rev-list.c
index fadd3ec14cb..e9bd4e378ad 100644
--- a/builtin/rev-list.c
+++ b/builtin/rev-list.c
@@ -6,6 +6,7 @@
 #include "list-objects.h"
 #include "list-objects-filter.h"
 #include "list-objects-filter-options.h"
+#include "object-store.h"
 #include "pack.h"
 #include "pack-bitmap.h"
 #include "builtin.h"
diff --git a/builtin/show-ref.c b/builtin/show-ref.c
index f2eb1a77240..2f13f1316fa 100644
--- a/builtin/show-ref.c
+++ b/builtin/show-ref.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "cache.h"
 #include "refs.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "string-list.h"
diff --git a/builtin/tag.c b/builtin/tag.c
index 26d7729f57b..f872f14f65e 100644
--- a/builtin/tag.c
+++ b/builtin/tag.c
@@ -10,6 +10,7 @@
 #include "config.h"
 #include "builtin.h"
 #include "refs.h"
+#include "object-store.h"
 #include "tag.h"
 #include "run-command.h"
 #include "parse-options.h"
diff --git a/builtin/unpack-file.c b/builtin/unpack-file.c
index 300eb59657e..58652229f27 100644
--- a/builtin/unpack-file.c
+++ b/builtin/unpack-file.c
@@ -1,5 +1,6 @@
 #include "builtin.h"
 #include "config.h"
+#include "object-store.h"
 
 static char *create_temp_file(struct object_id *oid)
 {
diff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c
index cfe9019f800..ded798b72ff 100644
--- a/builtin/unpack-objects.c
+++ b/builtin/unpack-objects.c
@@ -1,6 +1,7 @@
 #include "builtin.h"
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "object.h"
 #include "delta.h"
 #include "pack.h"
diff --git a/builtin/verify-commit.c b/builtin/verify-commit.c
index dcdaada1110..f6922da16d6 100644
--- a/builtin/verify-commit.c
+++ b/builtin/verify-commit.c
@@ -8,6 +8,7 @@
 #include "cache.h"
 #include "config.h"
 #include "builtin.h"
+#include "object-store.h"
 #include "commit.h"
 #include "run-command.h"
 #include <signal.h>
diff --git a/bulk-checkin.c b/bulk-checkin.c
index de1f4040c78..918d58a7ebf 100644
--- a/bulk-checkin.c
+++ b/bulk-checkin.c
@@ -8,6 +8,7 @@
 #include "pack.h"
 #include "strbuf.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static struct bulk_checkin_state {
 	unsigned plugged:1;
diff --git a/bundle.c b/bundle.c
index 902c9b54485..00ada05f46c 100644
--- a/bundle.c
+++ b/bundle.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "lockfile.h"
 #include "bundle.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "diff.h"
diff --git a/cache-tree.c b/cache-tree.c
index 6a555f4d431..1c338c41f3a 100644
--- a/cache-tree.c
+++ b/cache-tree.c
@@ -3,6 +3,7 @@
 #include "tree.h"
 #include "tree-walk.h"
 #include "cache-tree.h"
+#include "object-store.h"
 
 #ifndef DEBUG
 #define DEBUG 0
diff --git a/cache.h b/cache.h
index c75559b7d38..ab716011b7e 100644
--- a/cache.h
+++ b/cache.h
@@ -1183,32 +1183,6 @@ extern char *xdg_config_home(const char *filename);
  */
 extern char *xdg_cache_home(const char *filename);
 
-extern void *read_object_file_extended(const struct object_id *oid,
-				       enum object_type *type,
-				       unsigned long *size, int lookup_replace);
-static inline void *read_object_file(const struct object_id *oid, enum object_type *type, unsigned long *size)
-{
-	return read_object_file_extended(oid, type, size, 1);
-}
-
-/* Read and unpack an object file into memory, write memory to an object file */
-int oid_object_info(struct repository *r, const struct object_id *, unsigned long *);
-
-extern int hash_object_file(const void *buf, unsigned long len,
-			    const char *type, struct object_id *oid);
-
-extern int write_object_file(const void *buf, unsigned long len,
-			     const char *type, struct object_id *oid);
-
-extern int hash_object_file_literally(const void *buf, unsigned long len,
-				      const char *type, struct object_id *oid,
-				      unsigned flags);
-
-extern int pretend_object_file(void *, unsigned long, enum object_type,
-			       struct object_id *oid);
-
-extern int force_object_loose(const struct object_id *oid, time_t mtime);
-
 extern int git_open_cloexec(const char *name, int flags);
 #define git_open(name) git_open_cloexec(name, O_RDONLY)
 extern int unpack_sha1_header(git_zstream *stream, unsigned char *map, unsigned long mapsize, void *buffer, unsigned long bufsiz);
@@ -1218,43 +1192,6 @@ extern int check_object_signature(const struct object_id *oid, void *buf, unsign
 
 extern int finalize_object_file(const char *tmpfile, const char *filename);
 
-/*
- * Open the loose object at path, check its hash, and return the contents,
- * type, and size. If the object is a blob, then "contents" may return NULL,
- * to allow streaming of large blobs.
- *
- * Returns 0 on success, negative on error (details may be written to stderr).
- */
-int read_loose_object(const char *path,
-		      const struct object_id *expected_oid,
-		      enum object_type *type,
-		      unsigned long *size,
-		      void **contents);
-
-/*
- * Convenience for sha1_object_info_extended() with a NULL struct
- * object_info. OBJECT_INFO_SKIP_CACHED is automatically set; pass
- * nonzero flags to also set other flags.
- */
-extern int has_sha1_file_with_flags(const unsigned char *sha1, int flags);
-static inline int has_sha1_file(const unsigned char *sha1)
-{
-	return has_sha1_file_with_flags(sha1, 0);
-}
-
-/* Same as the above, except for struct object_id. */
-extern int has_object_file(const struct object_id *oid);
-extern int has_object_file_with_flags(const struct object_id *oid, int flags);
-
-/*
- * Return true iff an alternate object database has a loose object
- * with the specified name.  This function does not respect replace
- * references.
- */
-extern int has_loose_object_nonlocal(const unsigned char *sha1);
-
-extern void assert_oid_type(const struct object_id *oid, enum object_type expect);
-
 /* Helper to check and "touch" a file */
 extern int check_and_freshen_file(const char *fn, int freshen);
 
@@ -1624,60 +1561,6 @@ int for_each_loose_file_in_objdir_buf(struct strbuf *path,
 #define FOR_EACH_OBJECT_LOCAL_ONLY 0x1
 extern int for_each_loose_object(each_loose_object_fn, void *, unsigned flags);
 
-struct object_info {
-	/* Request */
-	enum object_type *typep;
-	unsigned long *sizep;
-	off_t *disk_sizep;
-	unsigned char *delta_base_sha1;
-	struct strbuf *type_name;
-	void **contentp;
-
-	/* Response */
-	enum {
-		OI_CACHED,
-		OI_LOOSE,
-		OI_PACKED,
-		OI_DBCACHED
-	} whence;
-	union {
-		/*
-		 * struct {
-		 * 	... Nothing to expose in this case
-		 * } cached;
-		 * struct {
-		 * 	... Nothing to expose in this case
-		 * } loose;
-		 */
-		struct {
-			struct packed_git *pack;
-			off_t offset;
-			unsigned int is_delta;
-		} packed;
-	} u;
-};
-
-/*
- * Initializer for a "struct object_info" that wants no items. You may
- * also memset() the memory to all-zeroes.
- */
-#define OBJECT_INFO_INIT {NULL}
-
-/* Invoke lookup_replace_object() on the given hash */
-#define OBJECT_INFO_LOOKUP_REPLACE 1
-/* Allow reading from a loose object file of unknown/bogus type */
-#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
-/* Do not check cached storage */
-#define OBJECT_INFO_SKIP_CACHED 4
-/* Do not retry packed storage after checking packed and loose storage */
-#define OBJECT_INFO_QUICK 8
-/* Do not check loose object */
-#define OBJECT_INFO_IGNORE_LOOSE 16
-
-int oid_object_info_extended(struct repository *r,
-			     const struct object_id *,
-			     struct object_info *, unsigned flags);
-
 /*
  * Set this to 0 to prevent sha1_object_info_extended() from fetching missing
  * blobs. This has a difference only if extensions.partialClone is set.
diff --git a/combine-diff.c b/combine-diff.c
index 2ef495963fc..de7695e7282 100644
--- a/combine-diff.c
+++ b/combine-diff.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "blob.h"
 #include "diff.h"
diff --git a/commit.c b/commit.c
index 5eb4d2f08f8..b053f07f305 100644
--- a/commit.c
+++ b/commit.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "tag.h"
 #include "commit.h"
+#include "object-store.h"
 #include "pkt-line.h"
 #include "utf8.h"
 #include "diff.h"
diff --git a/config.c b/config.c
index c698988f5e1..041db3ce7d4 100644
--- a/config.c
+++ b/config.c
@@ -14,6 +14,7 @@
 #include "quote.h"
 #include "hashmap.h"
 #include "string-list.h"
+#include "object-store.h"
 #include "utf8.h"
 #include "dir.h"
 
diff --git a/convert.c b/convert.c
index c480097a2a0..7f5afd6dd66 100644
--- a/convert.c
+++ b/convert.c
@@ -1,6 +1,7 @@
 #define NO_THE_INDEX_COMPATIBILITY_MACROS
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "attr.h"
 #include "run-command.h"
 #include "quote.h"
diff --git a/diff.c b/diff.c
index 4753170fe12..18d478c5190 100644
--- a/diff.c
+++ b/diff.c
@@ -13,6 +13,7 @@
 #include "attr.h"
 #include "run-command.h"
 #include "utf8.h"
+#include "object-store.h"
 #include "userdiff.h"
 #include "submodule-config.h"
 #include "submodule.h"
diff --git a/diffcore-rename.c b/diffcore-rename.c
index 0b7e4989a87..d775183c2fd 100644
--- a/diffcore-rename.c
+++ b/diffcore-rename.c
@@ -4,6 +4,7 @@
 #include "cache.h"
 #include "diff.h"
 #include "diffcore.h"
+#include "object-store.h"
 #include "hashmap.h"
 #include "progress.h"
 
diff --git a/dir.c b/dir.c
index 63a917be45d..9e7c90cf50b 100644
--- a/dir.c
+++ b/dir.c
@@ -11,6 +11,7 @@
 #include "cache.h"
 #include "config.h"
 #include "dir.h"
+#include "object-store.h"
 #include "attr.h"
 #include "refs.h"
 #include "wildmatch.h"
diff --git a/entry.c b/entry.c
index 2101201a111..b5d1d3cf231 100644
--- a/entry.c
+++ b/entry.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "blob.h"
+#include "object-store.h"
 #include "dir.h"
 #include "streaming.h"
 #include "submodule.h"
diff --git a/fetch-pack.c b/fetch-pack.c
index adc1b68dd3a..a1535b37b9b 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -19,6 +19,7 @@
 #include "sha1-array.h"
 #include "oidset.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;
diff --git a/fsck.c b/fsck.c
index 9218c2a643b..59b0c7d640e 100644
--- a/fsck.c
+++ b/fsck.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "object.h"
 #include "blob.h"
 #include "tree.h"
diff --git a/grep.c b/grep.c
index 65b90c10a38..438987220ea 100644
--- a/grep.c
+++ b/grep.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "grep.h"
+#include "object-store.h"
 #include "userdiff.h"
 #include "xdiff-interface.h"
 #include "diff.h"
diff --git a/list-objects-filter.c b/list-objects-filter.c
index ea94fe8af28..44b3a49bdb3 100644
--- a/list-objects-filter.c
+++ b/list-objects-filter.c
@@ -11,6 +11,7 @@
 #include "list-objects-filter.h"
 #include "list-objects-filter-options.h"
 #include "oidset.h"
+#include "object-store.h"
 
 /* Remember to update object flag allocation in object.h */
 /*
diff --git a/list-objects.c b/list-objects.c
index 168bef688a8..7ae0eb8ebc7 100644
--- a/list-objects.c
+++ b/list-objects.c
@@ -10,6 +10,7 @@
 #include "list-objects-filter.h"
 #include "list-objects-filter-options.h"
 #include "packfile.h"
+#include "object-store.h"
 
 static void process_blob(struct rev_info *revs,
 			 struct blob *blob,
diff --git a/log-tree.c b/log-tree.c
index d1c0bedf244..5660ee7c785 100644
--- a/log-tree.c
+++ b/log-tree.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "diff.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "graph.h"
diff --git a/mailmap.c b/mailmap.c
index 13f0d2884e2..962fd86d6d7 100644
--- a/mailmap.c
+++ b/mailmap.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "string-list.h"
 #include "mailmap.h"
+#include "object-store.h"
 
 #define DEBUG_MAILMAP 0
 #if DEBUG_MAILMAP
diff --git a/match-trees.c b/match-trees.c
index 72cc2baa3f9..4cdeff53e1e 100644
--- a/match-trees.c
+++ b/match-trees.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "tree.h"
 #include "tree-walk.h"
+#include "object-store.h"
 
 static int score_missing(unsigned mode, const char *path)
 {
diff --git a/merge-blobs.c b/merge-blobs.c
index fa49c17287f..fabb8c19ce9 100644
--- a/merge-blobs.c
+++ b/merge-blobs.c
@@ -4,6 +4,7 @@
 #include "ll-merge.h"
 #include "blob.h"
 #include "merge-blobs.h"
+#include "object-store.h"
 
 static int fill_mmfile_blob(mmfile_t *f, struct blob *obj)
 {
diff --git a/merge-recursive.c b/merge-recursive.c
index cbded673c28..455f59d21ac 100644
--- a/merge-recursive.c
+++ b/merge-recursive.c
@@ -8,6 +8,7 @@
 #include "advice.h"
 #include "lockfile.h"
 #include "cache-tree.h"
+#include "object-store.h"
 #include "commit.h"
 #include "blob.h"
 #include "builtin.h"
diff --git a/notes-cache.c b/notes-cache.c
index e61988e503b..d5770031776 100644
--- a/notes-cache.c
+++ b/notes-cache.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "notes-cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "refs.h"
 
diff --git a/notes-merge.c b/notes-merge.c
index 8e0726a9418..fd96c5ab3d6 100644
--- a/notes-merge.c
+++ b/notes-merge.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "commit.h"
 #include "refs.h"
+#include "object-store.h"
 #include "diff.h"
 #include "diffcore.h"
 #include "xdiff-interface.h"
diff --git a/notes.c b/notes.c
index a386d450c4c..32d3dbcc1e7 100644
--- a/notes.c
+++ b/notes.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "config.h"
 #include "notes.h"
+#include "object-store.h"
 #include "blob.h"
 #include "tree.h"
 #include "utf8.h"
diff --git a/object-store.h b/object-store.h
index 1ff862c7f93..695266891bf 100644
--- a/object-store.h
+++ b/object-store.h
@@ -137,4 +137,121 @@ void sha1_file_name(struct repository *r, struct strbuf *buf, const unsigned cha
 
 void *map_sha1_file(struct repository *r, const unsigned char *sha1, unsigned long *size);
 
+extern void *read_object_file_extended(const struct object_id *oid,
+				       enum object_type *type,
+				       unsigned long *size, int lookup_replace);
+static inline void *read_object_file(const struct object_id *oid, enum object_type *type, unsigned long *size)
+{
+	return read_object_file_extended(oid, type, size, 1);
+}
+
+/* Read and unpack an object file into memory, write memory to an object file */
+int oid_object_info(struct repository *r, const struct object_id *, unsigned long *);
+
+extern int hash_object_file(const void *buf, unsigned long len,
+			    const char *type, struct object_id *oid);
+
+extern int write_object_file(const void *buf, unsigned long len,
+			     const char *type, struct object_id *oid);
+
+extern int hash_object_file_literally(const void *buf, unsigned long len,
+				      const char *type, struct object_id *oid,
+				      unsigned flags);
+
+extern int pretend_object_file(void *, unsigned long, enum object_type,
+			       struct object_id *oid);
+
+extern int force_object_loose(const struct object_id *oid, time_t mtime);
+
+/*
+ * Open the loose object at path, check its hash, and return the contents,
+ * type, and size. If the object is a blob, then "contents" may return NULL,
+ * to allow streaming of large blobs.
+ *
+ * Returns 0 on success, negative on error (details may be written to stderr).
+ */
+int read_loose_object(const char *path,
+		      const struct object_id *expected_oid,
+		      enum object_type *type,
+		      unsigned long *size,
+		      void **contents);
+
+/*
+ * Convenience for sha1_object_info_extended() with a NULL struct
+ * object_info. OBJECT_INFO_SKIP_CACHED is automatically set; pass
+ * nonzero flags to also set other flags.
+ */
+extern int has_sha1_file_with_flags(const unsigned char *sha1, int flags);
+static inline int has_sha1_file(const unsigned char *sha1)
+{
+	return has_sha1_file_with_flags(sha1, 0);
+}
+
+/* Same as the above, except for struct object_id. */
+extern int has_object_file(const struct object_id *oid);
+extern int has_object_file_with_flags(const struct object_id *oid, int flags);
+
+/*
+ * Return true iff an alternate object database has a loose object
+ * with the specified name.  This function does not respect replace
+ * references.
+ */
+extern int has_loose_object_nonlocal(const unsigned char *sha1);
+
+extern void assert_oid_type(const struct object_id *oid, enum object_type expect);
+
+struct object_info {
+	/* Request */
+	enum object_type *typep;
+	unsigned long *sizep;
+	off_t *disk_sizep;
+	unsigned char *delta_base_sha1;
+	struct strbuf *type_name;
+	void **contentp;
+
+	/* Response */
+	enum {
+		OI_CACHED,
+		OI_LOOSE,
+		OI_PACKED,
+		OI_DBCACHED
+	} whence;
+	union {
+		/*
+		 * struct {
+		 * 	... Nothing to expose in this case
+		 * } cached;
+		 * struct {
+		 * 	... Nothing to expose in this case
+		 * } loose;
+		 */
+		struct {
+			struct packed_git *pack;
+			off_t offset;
+			unsigned int is_delta;
+		} packed;
+	} u;
+};
+
+/*
+ * Initializer for a "struct object_info" that wants no items. You may
+ * also memset() the memory to all-zeroes.
+ */
+#define OBJECT_INFO_INIT {NULL}
+
+/* Invoke lookup_replace_object() on the given hash */
+#define OBJECT_INFO_LOOKUP_REPLACE 1
+/* Allow reading from a loose object file of unknown/bogus type */
+#define OBJECT_INFO_ALLOW_UNKNOWN_TYPE 2
+/* Do not check cached storage */
+#define OBJECT_INFO_SKIP_CACHED 4
+/* Do not retry packed storage after checking packed and loose storage */
+#define OBJECT_INFO_QUICK 8
+/* Do not check loose object */
+#define OBJECT_INFO_IGNORE_LOOSE 16
+
+int oid_object_info_extended(struct repository *r,
+			     const struct object_id *,
+			     struct object_info *, unsigned flags);
+
 #endif /* OBJECT_STORE_H */
diff --git a/object.c b/object.c
index 8e29f63bf23..0116ed6529a 100644
--- a/object.c
+++ b/object.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "object.h"
 #include "replace-object.h"
+#include "object-store.h"
 #include "blob.h"
 #include "tree.h"
 #include "commit.h"
diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c
index cd1903e717a..f7fac9d0f24 100644
--- a/pack-bitmap-write.c
+++ b/pack-bitmap-write.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "diff.h"
diff --git a/packfile.h b/packfile.h
index fdfddb89b53..1abbc2c07c8 100644
--- a/packfile.h
+++ b/packfile.h
@@ -3,6 +3,11 @@
 
 #include "oidset.h"
 
+/* in object-store.h */
+struct packed_git;
+struct object_info;
+enum object_type;
+
 /*
  * Generate the filename to be used for a pack file with checksum "sha1" and
  * extension "ext". The result is written into the strbuf "buf", overwriting
diff --git a/read-cache.c b/read-cache.c
index 10f1c6bb8a3..7f10d9ffff9 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -11,6 +11,7 @@
 #include "cache-tree.h"
 #include "refs.h"
 #include "dir.h"
+#include "object-store.h"
 #include "tree.h"
 #include "commit.h"
 #include "blob.h"
diff --git a/ref-filter.c b/ref-filter.c
index 9a333e21b51..e2ae1f47fee 100644
--- a/ref-filter.c
+++ b/ref-filter.c
@@ -3,6 +3,7 @@
 #include "parse-options.h"
 #include "refs.h"
 #include "wildmatch.h"
+#include "object-store.h"
 #include "commit.h"
 #include "remote.h"
 #include "color.h"
diff --git a/refs.c b/refs.c
index 27c88ba7689..2361ba9c6ee 100644
--- a/refs.c
+++ b/refs.c
@@ -9,6 +9,7 @@
 #include "iterator.h"
 #include "refs.h"
 #include "refs/refs-internal.h"
+#include "object-store.h"
 #include "object.h"
 #include "tag.h"
 #include "submodule.h"
diff --git a/remote-testsvn.c b/remote-testsvn.c
index c4bb9a8ba92..515b922b503 100644
--- a/remote-testsvn.c
+++ b/remote-testsvn.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "refs.h"
 #include "remote.h"
+#include "object-store.h"
 #include "strbuf.h"
 #include "url.h"
 #include "exec_cmd.h"
diff --git a/remote.c b/remote.c
index 481bf933f39..db438cfbc56 100644
--- a/remote.c
+++ b/remote.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "remote.h"
 #include "refs.h"
+#include "object-store.h"
 #include "commit.h"
 #include "diff.h"
 #include "revision.h"
diff --git a/rerere.c b/rerere.c
index 18cae2d11c9..fcb99cc9542 100644
--- a/rerere.c
+++ b/rerere.c
@@ -9,6 +9,7 @@
 #include "ll-merge.h"
 #include "attr.h"
 #include "pathspec.h"
+#include "object-store.h"
 #include "sha1-lookup.h"
 
 #define RESOLVED 0
diff --git a/revision.c b/revision.c
index 1cff11833e7..260b9c276c7 100644
--- a/revision.c
+++ b/revision.c
@@ -1,4 +1,5 @@
 #include "cache.h"
+#include "object-store.h"
 #include "tag.h"
 #include "blob.h"
 #include "tree.h"
diff --git a/send-pack.c b/send-pack.c
index 19025a7aca8..71600028cdd 100644
--- a/send-pack.c
+++ b/send-pack.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "commit.h"
 #include "refs.h"
+#include "object-store.h"
 #include "pkt-line.h"
 #include "sideband.h"
 #include "run-command.h"
diff --git a/sequencer.c b/sequencer.c
index 44f0518b9c4..cf6456ab75d 100644
--- a/sequencer.c
+++ b/sequencer.c
@@ -2,6 +2,7 @@
 #include "config.h"
 #include "lockfile.h"
 #include "dir.h"
+#include "object-store.h"
 #include "object.h"
 #include "commit.h"
 #include "sequencer.h"
diff --git a/shallow.c b/shallow.c
index df4d44ea7a3..c2f81a5a5a8 100644
--- a/shallow.c
+++ b/shallow.c
@@ -1,6 +1,7 @@
 #include "cache.h"
 #include "tempfile.h"
 #include "lockfile.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tag.h"
 #include "pkt-line.h"
diff --git a/submodule-config.c b/submodule-config.c
index 3f2075764fe..a1e57c33339 100644
--- a/submodule-config.c
+++ b/submodule-config.c
@@ -4,6 +4,7 @@
 #include "submodule-config.h"
 #include "submodule.h"
 #include "strbuf.h"
+#include "object-store.h"
 #include "parse-options.h"
 
 /*
diff --git a/tag.c b/tag.c
index 7c12426b4ea..3be7206e920 100644
--- a/tag.c
+++ b/tag.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "tag.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "blob.h"
diff --git a/tree-walk.c b/tree-walk.c
index e11b3063afa..cd428a1ee53 100644
--- a/tree-walk.c
+++ b/tree-walk.c
@@ -2,6 +2,7 @@
 #include "tree-walk.h"
 #include "unpack-trees.h"
 #include "dir.h"
+#include "object-store.h"
 #include "tree.h"
 #include "pathspec.h"
 
diff --git a/tree.c b/tree.c
index 8f8ef3189af..dc20a2693fd 100644
--- a/tree.c
+++ b/tree.c
@@ -2,6 +2,7 @@
 #include "cache.h"
 #include "cache-tree.h"
 #include "tree.h"
+#include "object-store.h"
 #include "blob.h"
 #include "commit.h"
 #include "tag.h"
diff --git a/unpack-trees.c b/unpack-trees.c
index e73745051e5..e3f95cc6f95 100644
--- a/unpack-trees.c
+++ b/unpack-trees.c
@@ -15,6 +15,7 @@
 #include "submodule.h"
 #include "submodule-config.h"
 #include "fsmonitor.h"
+#include "object-store.h"
 #include "fetch-object.h"
 
 /*
diff --git a/upload-pack.c b/upload-pack.c
index 4a82602be5d..a11c6d192ce 100644
--- a/upload-pack.c
+++ b/upload-pack.c
@@ -3,6 +3,7 @@
 #include "refs.h"
 #include "pkt-line.h"
 #include "sideband.h"
+#include "object-store.h"
 #include "tag.h"
 #include "object.h"
 #include "commit.h"
diff --git a/walker.c b/walker.c
index dffb9c8e37c..3fb5fb2a0be 100644
--- a/walker.c
+++ b/walker.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "walker.h"
+#include "object-store.h"
 #include "commit.h"
 #include "tree.h"
 #include "tree-walk.h"
diff --git a/xdiff-interface.c b/xdiff-interface.c
index 9315bc0ede1..ec6e574e4aa 100644
--- a/xdiff-interface.c
+++ b/xdiff-interface.c
@@ -1,5 +1,6 @@
 #include "cache.h"
 #include "config.h"
+#include "object-store.h"
 #include "xdiff-interface.h"
 #include "xdiff/xtypes.h"
 #include "xdiff/xdiffi.h"
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 5%]

* Re: [PATCH 2/3] merge-recursive: i18n submodule merge output and respect verbosity
  2018-05-15 20:00 ` [PATCH 2/3] merge-recursive: i18n submodule merge output and respect verbosity Stefan Beller
@ 2018-05-16  1:17   ` Junio C Hamano
  0 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-05-16  1:17 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, leif.middelschulte, newren

Stefan Beller <sbeller@google.com> writes:

> +static int merge_submodule(struct merge_options *o,
> +			   struct object_id *result, const char *path,
>  			   const struct object_id *base, const struct object_id *a,
> -			   const struct object_id *b, int search)
> +			   const struct object_id *b)
>  {
>  	struct commit *commit_base, *commit_a, *commit_b;
>  	int parent_count;
>  	struct object_array merges;
>  
>  	int i;
> +	int search = !o->call_depth;

I kind of like this "while at it" change in this patch ;-)

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 3/3] Inform about fast-forwarding of submodules during merge
  2018-05-15 20:00 ` [PATCH 3/3] Inform about fast-forwarding of submodules during merge Stefan Beller
@ 2018-05-16  1:36   ` Elijah Newren
  2018-05-16  1:36   ` Junio C Hamano
  2018-05-16  1:42   ` Junio C Hamano
  2 siblings, 0 replies; 200+ results
From: Elijah Newren @ 2018-05-16  1:36 UTC (permalink / raw)
  To: Stefan Beller; +Cc: Git Mailing List, Leif Middelschulte, Junio C Hamano

On Tue, May 15, 2018 at 1:00 PM, Stefan Beller <sbeller@google.com> wrote:
> From: Leif Middelschulte <Leif.Middelschulte@gmail.com>
>
> Inform the user about an automatically fast-forwarded submodule. The
> silent merge behavior was introduced by commit 68d03e4a6e44 ("Implement
> automatic fast-forward merge for submodules", 2010-07-07)).
>
> Signed-off-by: Leif Middelschulte <Leif.Middelschulte@gmail.com>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  merge-recursive.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/merge-recursive.c b/merge-recursive.c
> index 0571919ee0a..29a430c418a 100644
> --- a/merge-recursive.c
> +++ b/merge-recursive.c
> @@ -1093,10 +1093,26 @@ static int merge_submodule(struct merge_options *o,
>         /* Case #1: a is contained in b or vice versa */
>         if (in_merge_bases(commit_a, commit_b)) {
>                 oidcpy(result, b);
> +               if (show(o, 3)) {
> +                       output(o, 1, _("Fast-forwarding submodule %s to the following commit:"), path);

Seems slightly odd to mix 3 and 1 here; although it'll work just fine,
I would have expected use of 3 in both places (much like you did with
the 2 and 2 below).

> +                       output_commit_title(o, commit_b);
> +               } else if (show(o, 2))
> +                       output(o, 2, _("Fast-forwarding submodule %s to %s"), path, oid_to_hex(b));
> +               else
> +                       ; /* no output */
> +
>                 return 1;
>         }
>         if (in_merge_bases(commit_b, commit_a)) {
>                 oidcpy(result, a);
> +               if (show(o, 3)) {
> +                       output(o, 1, _("Fast-forwarding submodule %s to the following commit:"), path);

Same.

> +                       output_commit_title(o, commit_a);
> +               } else if (show(o, 2))
> +                       output(o, 2, _("Fast-forwarding submodule %s to %s"), path, oid_to_hex(a));
> +               else
> +                       ; /* no output */
> +
>                 return 1;
>         }
>
> --
> 2.17.0.582.gccdcbd54c44.dirty

Other than that nit-pick, looks good to me.

^ permalink raw reply	[relevance 5%]

* Re: [PATCH 3/3] Inform about fast-forwarding of submodules during merge
  2018-05-15 20:00 ` [PATCH 3/3] Inform about fast-forwarding of submodules during merge Stefan Beller
  2018-05-16  1:36   ` Elijah Newren
@ 2018-05-16  1:36   ` Junio C Hamano
  2018-05-16  1:42   ` Junio C Hamano
  2 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-05-16  1:36 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, leif.middelschulte, newren

Stefan Beller <sbeller@google.com> writes:

> From: Leif Middelschulte <Leif.Middelschulte@gmail.com>
>

Subject: merge-recursive: give notice when submodule commit gets fast-forwarded

perhaps?

>  	/* Case #1: a is contained in b or vice versa */
>  	if (in_merge_bases(commit_a, commit_b)) {
>  		oidcpy(result, b);
> +		if (show(o, 3)) {
> +			output(o, 1, _("Fast-forwarding submodule %s to the following commit:"), path);
> +			output_commit_title(o, commit_b);
> +		} else if (show(o, 2))
> +			output(o, 2, _("Fast-forwarding submodule %s to %s"), path, oid_to_hex(b));
> +		else
> +			; /* no output */
> +

merge.verbosity::
	Controls the amount of output shown by the recursive merge
	strategy.  Level 0 outputs nothing except a final error
	message if conflicts were detected. Level 1 outputs only
	conflicts, 2 outputs conflicts and file changes.  Level 5 and
	above outputs debugging information.  The default is level 2.
	Can be overridden by the `GIT_MERGE_VERBOSITY` environment variable.

So, by default, we report the fact that we update submodule to a
particular commit, which is quite similar to how we report auto
merged paths using the content level 3-way merge; when you squint
your eyes, the "fast-forward" of submodules look somewhat like a
content-level 3-way merge anyway ;-)

And at level 3, which currently is used to report a non-event that
does not change the result of the merge from what was naturally
expected, we give a bit more detail by citing the commit the
submodule gets fast-forwarded to [*1*].

Sort of makes sense.


[Footnote]

*1* I wonder if that is really necessary, though---we do not give
"here is a diff" or "this is the new contents" after a path gets
merged for normal files.  And if it is needed perhaps because
submodules are so special, I wonder if we also need to give the
commit the submodule gets fast-forwarded from, i.e. the original
one, the same way.

^ permalink raw reply	[relevance 7%]

* Re: [PATCH 3/3] Inform about fast-forwarding of submodules during merge
  2018-05-15 20:00 ` [PATCH 3/3] Inform about fast-forwarding of submodules during merge Stefan Beller
  2018-05-16  1:36   ` Elijah Newren
  2018-05-16  1:36   ` Junio C Hamano
@ 2018-05-16  1:42   ` Junio C Hamano
  2 siblings, 0 replies; 200+ results
From: Junio C Hamano @ 2018-05-16  1:42 UTC (permalink / raw)
  To: Stefan Beller; +Cc: git, leif.middelschulte, newren

Stefan Beller <sbeller@google.com> writes:

> From: Leif Middelschulte <Leif.Middelschulte@gmail.com>
>
> Inform the user about an automatically fast-forwarded submodule. The
> silent merge behavior was introduced by commit 68d03e4a6e44 ("Implement
> automatic fast-forward merge for submodules", 2010-07-07)).

Oh, another thing I forgot to mention.

These three lines do not convey much useful information.  The first
sentence can be read from the patch text, and the rest can be read
from "git blame" and "git log" output.

It is correct that the silent behaviour was introduced long time
ago.  The proposed log message does not even say if that silent
behaviour is bad in any way, let alone why it is bad and need to be
changed.

Perhaps Leif can elaborate why this change is a good idea in the
first place?

Thanks.

^ permalink raw reply	[relevance 5%]

* Re: [PATCH] grep: handle corrupt index files early
  2018-05-15 16:44   ` [PATCH] grep: handle corrupt index files early Stefan Beller
@ 2018-05-16 15:24     ` Duy Nguyen
  2018-05-16 22:21       ` [PATCH 00/11] Stefan Beller
  0 siblings, 1 reply; 200+ results
From: Duy Nguyen @ 2018-05-16 15:24 UTC (permalink / raw)
  To: Stefan Beller
  Cc: Brandon Williams, Junio C Hamano, Git Mailing List, Antonio Ospite

On Tue, May 15, 2018 at 6:44 PM, Stefan Beller <sbeller@google.com> wrote:
> On Tue, May 15, 2018 at 6:13 AM, Duy Nguyen <pclouds@gmail.com> wrote:
>> On Tue, May 15, 2018 at 3:04 AM, Stefan Beller <sbeller@google.com> wrote:
>>> Any other caller of 'repo_read_index' dies upon a negative return of
>>> it, so grep should, too.
>>>
>>> Signed-off-by: Stefan Beller <sbeller@google.com>
>>> ---
>>>
>>> Found while reviewing the series
>>> https://public-inbox.org/git/20180514105823.8378-1-ao2@ao2.it/
>>>
>>>  builtin/grep.c | 3 ++-
>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/builtin/grep.c b/builtin/grep.c
>>> index 6e7bc76785a..69f0743619f 100644
>>> --- a/builtin/grep.c
>>> +++ b/builtin/grep.c
>>> @@ -488,7 +488,8 @@ static int grep_cache(struct grep_opt *opt, struct repository *repo,
>>>                 strbuf_addstr(&name, repo->submodule_prefix);
>>>         }
>>>
>>> -       repo_read_index(repo);
>>> +       if (repo_read_index(repo) < 0)
>>> +               die("index file corrupt");
>>
>> _() the string (and maybe reuse an existing phrase if found to reduce
>> workload on translators)
>
> sbeller@sbeller:/u/git$ git grep -A 1 repo_read_index
> builtin/grep.c:491:     if (repo_read_index(repo) < 0)
> builtin/grep.c-492-             die("index file corrupt");
> --
> builtin/ls-files.c:213: if (repo_read_index(&submodule) < 0)
> builtin/ls-files.c-214-         die("index file corrupt");
> --
> builtin/ls-files.c:582: if (repo_read_index(the_repository) < 0)
> builtin/ls-files.c-583-         die("index file corrupt");
> --
> dir.c:3028:     if (repo_read_index(&subrepo) < 0)
> dir.c-3029-             die("index file corrupt in repo %s", subrepo.gitdir);
> --
> repository.c:245:int repo_read_index(struct repository *repo)
> repository.c-246-{
> --
> repository.h:70:         * 'repo_read_index()' can be used to populate 'index'.
> repository.h-71-         */
> --
> repository.h:119:extern int repo_read_index(struct repository *repo);
> repository.h-120-
> --
> submodule-config.c:583:         if (repo_read_index(repo) < 0)
> submodule-config.c-584-                 return;
> --
> submodule.c:1336:       if (repo_read_index(r) < 0)
> submodule.c-1337-               die("index file corrupt");
>
> I think this is as good as it gets for using an existing phrase.
> None of them are translated, which I would defer to a follow up patch
> that translates all(?) of them or just the porcelains.

If you have time, yes translate them all. I don't see how any of these
strings are meant for script. If not, just _() the new string you
added is fine.

With a majority of call sites dying like this though, I wonder if we
should just add repo_read_index_or_die() with die() inside. Then the
next person won't likely accidentally forget _()

>
> Thanks,
> Stefan



-- 
Duy

^ permalink raw reply	[relevance 2%]

* [PATCH 06/11] read_cache: use repo_read_index_or_die with different error messages
  2018-05-16 22:21       ` [PATCH 00/11] Stefan Beller
                           ` (2 preceding siblings ...)
  2018-05-16 22:21         ` [PATCH 05/11] builtin/ls-files: " Stefan Beller
@ 2018-05-16 22:21         ` Stefan Beller
  2018-05-16 22:21         ` [PATCH 11/11] read_cache: convert most calls to repo_read_index_or_die Stefan Beller
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-16 22:21 UTC (permalink / raw)
  To: pclouds; +Cc: ao2, bmwill, git, gitster, sbeller

This replaces all patterns of "if (read_cached() < 0) die(some-msg);"
with repo_read_index_or_die; this changes the output of the error message.

However as all error messages before were translated, these are for human
consumption, so a change in error message is not bad; in fact it makes Git
more consistent.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/add.c       | 4 ++--
 builtin/commit.c    | 9 +++++----
 builtin/rev-parse.c | 4 ++--
 merge.c             | 4 ++--
 4 files changed, 11 insertions(+), 10 deletions(-)

diff --git a/builtin/add.c b/builtin/add.c
index e4751c198c1..910f619b7d5 100644
--- a/builtin/add.c
+++ b/builtin/add.c
@@ -19,6 +19,7 @@
 #include "bulk-checkin.h"
 #include "argv-array.h"
 #include "submodule.h"
+#include "repository.h"
 
 static const char * const builtin_add_usage[] = {
 	N_("git add [<options>] [--] <pathspec>..."),
@@ -229,8 +230,7 @@ static int edit_patch(int argc, const char **argv, const char *prefix)
 
 	git_config(git_diff_basic_config, NULL); /* no "diff" UI options */
 
-	if (read_cache() < 0)
-		die(_("Could not read the index"));
+	repo_read_index_or_die(the_repository);
 
 	init_revisions(&rev, prefix);
 	rev.diffopt.context = 7;
diff --git a/builtin/commit.c b/builtin/commit.c
index 5571d4a3e2b..9ebfb4db415 100644
--- a/builtin/commit.c
+++ b/builtin/commit.c
@@ -32,6 +32,8 @@
 #include "column.h"
 #include "sequencer.h"
 #include "mailmap.h"
+#include "sigchain.h"
+#include "repository.h"
 
 static const char * const builtin_commit_usage[] = {
 	N_("git commit [<options>] [--] <pathspec>..."),
@@ -428,8 +430,7 @@ static const char *prepare_index(int argc, const char **argv, const char *prefix
 		exit(1);
 
 	discard_cache();
-	if (read_cache() < 0)
-		die(_("cannot read the index"));
+	repo_read_index_or_die(the_repository);
 
 	hold_locked_index(&index_lock, LOCK_DIE_ON_ERROR);
 	add_remove_files(&partial);
@@ -853,8 +854,8 @@ static int prepare_to_commit(const char *index_file, const char *prefix,
 		struct object_id oid;
 		const char *parent = "HEAD";
 
-		if (!active_nr && read_cache() < 0)
-			die(_("Cannot read index"));
+		if (!active_nr)
+			repo_read_index_or_die(the_repository);
 
 		if (amend)
 			parent = "HEAD^1";
diff --git a/builtin/rev-parse.c b/builtin/rev-parse.c
index 36b20877828..37f29fd850d 100644
--- a/builtin/rev-parse.c
+++ b/builtin/rev-parse.c
@@ -14,6 +14,7 @@
 #include "revision.h"
 #include "split-index.h"
 #include "submodule.h"
+#include "repository.h"
 
 #define DO_REVS		1
 #define DO_NOREV	2
@@ -884,8 +885,7 @@ int cmd_rev_parse(int argc, const char **argv, const char *prefix)
 				continue;
 			}
 			if (!strcmp(arg, "--shared-index-path")) {
-				if (read_cache() < 0)
-					die(_("Could not read the index"));
+				repo_read_index_or_die(the_repository);
 				if (the_index.split_index) {
 					const unsigned char *sha1 = the_index.split_index->base_sha1;
 					const char *path = git_path("sharedindex.%s", sha1_to_hex(sha1));
diff --git a/merge.c b/merge.c
index f06a4773d4f..654d049e80d 100644
--- a/merge.c
+++ b/merge.c
@@ -8,6 +8,7 @@
 #include "tree-walk.h"
 #include "unpack-trees.h"
 #include "dir.h"
+#include "repository.h"
 
 static const char *merge_argument(struct commit *commit)
 {
@@ -70,8 +71,7 @@ int try_merge_command(const char *strategy, size_t xopts_nr,
 	argv_array_clear(&args);
 
 	discard_cache();
-	if (read_cache() < 0)
-		die(_("failed to read the cache"));
+	repo_read_index_or_die(the_repository);
 	resolve_undo_clear();
 
 	return ret;
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 7%]

* [PATCH 05/11] builtin/ls-files: use repo_read_index_or_die
  2018-05-16 22:21       ` [PATCH 00/11] Stefan Beller
  2018-05-16 22:21         ` [PATCH 02/11] repository: introduce repo_read_index_or_die Stefan Beller
  2018-05-16 22:21         ` [PATCH 04/11] submodule: use repo_read_index_or_die Stefan Beller
@ 2018-05-16 22:21         ` " Stefan Beller
  2018-05-16 22:21         ` [PATCH 06/11] read_cache: use repo_read_index_or_die with different error messages Stefan Beller
  2018-05-16 22:21         ` [PATCH 11/11] read_cache: convert most calls to repo_read_index_or_die Stefan Beller
  4 siblings, 0 replies; 200+ results
From: Stefan Beller @ 2018-05-16 22:21 UTC (permalink / raw)
  To: pclouds; +Cc: ao2, bmwill, git, gitster, sbeller

Despite ls-files being a plumbing command, which promises to not change its
output ever, and to be easy on machines (e.g. non-localized output),
it may make sense to localize the error message for a corrupt index
nevertheless:

1. that is more consistent with the rest of Git.
2. Searching for "ls-tree corrupt index file" on the web doesn't yield
   any hits, that suggest the exact string is parsed for.
   Probably the script authors rely on the exit code of ls-tree anyways.

Signed-off-by: Stefan Beller <sbeller@google.com>
---
 builtin/ls-files.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/builtin/ls-files.c b/builtin/ls-files.c
index a71f6bd088a..502f2f6db04 100644
--- a/builtin/ls-files.c
+++ b/builtin/ls-files.c
@@ -20,6 +20,7 @@
 #include "run-command.h"
 #include "submodule.h"
 #include "submodule-config.h"
+#include "repository.h"
 
 static int abbrev;
 static int show_deleted;
@@ -210,8 +211,7 @@ static void show_submodule(struct repository *superproject,
 	if (repo_submodule_init(&submodule, superproject, path))
 		return;
 
-	if (repo_read_index(&submodule) < 0)
-		die("index file corrupt");
+	repo_read_index_or_die(&submodule);
 
 	show_files(&submodule, dir);
 
@@ -579,8 +579,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)
 		prefix_len = strlen(prefix);
 	git_config(git_default_config, NULL);
 
-	if (repo_read_index(the_repository) < 0)
-		die("index file corrupt");
+	repo_read_index_or_die(the_repository);
 
 	argc = parse_options(argc, argv, prefix, builtin_ls_files_options,
 			ls_files_usage, 0);
-- 
2.17.0.582.gccdcbd54c44.dirty


^ permalink raw reply	[relevance 15%]