[PATCH 00/15] [RFC] Upstreaming the Scalar command

[PATCH 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin via GitGitGadget]

tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.


Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.

The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.


The present
===========

The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).


Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:

 1. git clone --sparse --filter=blob:none /src
 2. git -C /src sparse-checkout init --cone
 3. git -C /src config (many times)
 4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.

For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.


Epilogue
========

Now, to address some questions that I imagine every reader has who made it
this far:

 * Why not put the Scalar functionality directly into a built-in? Creating a
   Git builtin requires scrutiny over every aspect of the feature, which is
   difficult to do while also maintaining the command-line interface
   contract and expected behavior of the Scalar command (there are existing
   users, after all). By having the Scalar command in contrib/, we present a
   simple option for users to have these features in the short term while
   the Git contributor community decides which bits to absorb into Git
   built-ins.
 * Why implement the Scalar command in the Git codebase? We ported Scalar to
   the microsoft/git fork for several reasons. First, we realized it was
   possible now that the core features exist inside Git itself. Second,
   compiling Scalar directly within a version of Git allows us to remove a
   version compatibility check from each config option that might or might
   not apply based on the installed Git version. Finally, this new location
   has greatly simplified our release process and the installation process
   for users. We now have ways to install Scalar with microsoft/git via
   winget, brew, and apt-get. This has been the case since we shipped
   v2.32.0 to our users, read: this setup has served us well already.
 * Why contribute Scalar to the Git project? We are biased, of course, yet
   we do have evidence that the Scalar command is a helpful tool that offers
   an simple way to handle huge repositories with ease. By contributing it
   to the core Git project, we are able to share it with more users,
   especially some users who do not want to install the microsoft/git fork.
   We intend to include Scalar as a component in git-for-windows/git, but
   are contributing it here first. Further, we think there is benefit to the
   Git developer community as this presents an example of how to set certain
   defaults that work for large repositories.
 * Does this integrate with the built-in FSMonitor yet? No, not yet. I do
   have a couple of add-on patch series lined up, one of them being the
   integration with the built-in FSMonitor, which obviously has to wait
   until the FSMonitor patch series advances further.

Derrick Stolee (4):
  scalar: 'register' sets recommended config and starts maintenance
  scalar: 'unregister' stops background maintenance
  scalar: implement 'scalar list'
  scalar: implement the `run` command

Johannes Schindelin (10):
  scalar: create a rudimentary executable
  scalar: start documenting the command
  scalar: create test infrastructure
  scalar: let 'unregister' handle a deleted enlistment directory
    gracefully
  scalar: implement the `clone` subcommand
  scalar: teach 'clone' to support the --single-branch option
  scalar: allow reconfiguring an existing enlistment
  scalar: teach 'reconfigure' to optionally handle all registered
    enlistments
  scalar: implement the `version` command
  scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
  scalar: implement the `delete` command

 Makefile                         |   8 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 844 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 152 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1232 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh


base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1005
-- 
gitgitgadget

[PATCH 01/15] scalar: create a rudimentary executable

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.

For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.

With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

	make -C contrib/scalar/Makefile

This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.

The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.

An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Makefile                  |  8 ++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 80 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c

diff --git a/Makefile b/Makefile
index c3565fc0f8f..2d5c822f7a8 100644
--- a/Makefile
+++ b/Makefile
@@ -2447,6 +2447,10 @@ endif
 .PHONY: objects
 objects: $(OBJECTS)
 
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
 
@@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
new file mode 100644
index 00000000000..ff3d47e84d0
--- /dev/null
+++ b/contrib/scalar/.gitignore
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
new file mode 100644
index 00000000000..40c03ad10e1
--- /dev/null
+++ b/contrib/scalar/Makefile
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$X
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: all clean FORCE
diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
new file mode 100644
index 00000000000..7cff29e0fcd
--- /dev/null
+++ b/contrib/scalar/scalar.c
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
-- 
gitgitgadget

[PATCH 02/15] scalar: start documenting the command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit establishes the infrastructure to build the manual page for
the `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt

diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
index ff3d47e84d0..00441073f59 100644
--- a/contrib/scalar/.gitignore
+++ b/contrib/scalar/.gitignore
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 40c03ad10e1..85c186634e9 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: all clean docs FORCE
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
new file mode 100644
index 00000000000..5f7131861a5
--- /dev/null
+++ b/contrib/scalar/scalar.txt
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
-- 
gitgitgadget

[PATCH 03/15] scalar: create test infrastructure

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.

Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 85c186634e9..8620042f281 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$X
+all: scalar$X ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: all clean docs FORCE
+.PHONY: all clean docs test FORCE
diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile
new file mode 100644
index 00000000000..6170672bb37
--- /dev/null
+++ b/contrib/scalar/t/Makefile
@@ -0,0 +1,78 @@
+# Run scalar tests
+#
+# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
+#
+
+-include ../../../config.mak.autogen
+-include ../../../config.mak
+
+SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
+RM ?= rm -f
+PROVE ?= prove
+DEFAULT_TEST_TARGET ?= test
+TEST_LINT ?= test-lint
+
+ifdef TEST_OUTPUT_DIRECTORY
+TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
+else
+TEST_RESULTS_DIRECTORY = ../../../t/test-results
+endif
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
+TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
+
+T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
+
+all: $(DEFAULT_TEST_TARGET)
+
+test: $(TEST_LINT)
+	$(MAKE) aggregate-results-and-cleanup
+
+prove: $(TEST_LINT)
+	@echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
+	$(MAKE) clean-except-prove-cache
+
+$(T):
+	@echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+
+clean-except-prove-cache:
+	$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+	$(RM) -r valgrind/bin
+
+clean: clean-except-prove-cache
+	$(RM) .prove
+
+test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax
+
+test-lint-duplicates:
+	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+		test -z "$$dups" || { \
+		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
+
+test-lint-executable:
+	@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
+		test -z "$$bad" || { \
+		echo >&2 "non-executable tests:" $$bad; exit 1; }
+
+test-lint-shell-syntax:
+	@'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T)
+
+aggregate-results-and-cleanup: $(T)
+	$(MAKE) aggregate-results
+	$(MAKE) clean
+
+aggregate-results:
+	for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
+		echo "$$f"; \
+	done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh
+
+valgrind:
+	$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
+
+test-results:
+	mkdir -p test-results
+
+.PHONY: $(T) aggregate-results clean valgrind
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
new file mode 100755
index 00000000000..16f2b72b126
--- /dev/null
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
-- 
gitgitgadget

[PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.

The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 256 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  14 ++-
 2 files changed, 269 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7cff29e0fcd..7660327c27b 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -5,11 +5,267 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_strip_suffix(&path, "/.git");
+
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		strbuf_addstr(&path, "/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_setlen(&path, len);
+
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+
+		strbuf_setlen(&path, len);
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.bare", "false" },
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 5f7131861a5..41429db7990 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,18 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+    Adds the enlistment's repository to the list of registered repositories
+    and starts background maintenance. If `<enlistment>` is not provided,
+    then the enlistment associated with the current working directory is
+    registered.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH 05/15] scalar: 'unregister' stops background maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.

The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7660327c27b..ef91a1af38b 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -205,12 +205,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -221,24 +221,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -261,11 +276,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 41429db7990..cddaa969403 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -41,6 +42,13 @@ register [<enlistment>]::
     then the enlistment associated with the current working directory is
     registered.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+    Remove the specified repository from the list of repositories
+    registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH 06/15] scalar: let 'unregister' handle a deleted enlistment directory gracefully

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index ef91a1af38b..14e688f44a9 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -276,6 +276,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -289,6 +307,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 16f2b72b126..ef0e8d680d5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
-- 
gitgitgadget

[PATCH 07/15] scalar: implement 'scalar list'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 12 +++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 14e688f44a9..91ceb97e552 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -258,6 +258,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -344,6 +354,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index cddaa969403..e1f629fddad 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,20 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+    To see which repositories are currently registered by the service, run
+    `scalar list`. This subcommand does not need to be run inside a Scalar
+    enlistment.
+
 Register
 ~~~~~~~~
 
-- 
gitgitgadget

[PATCH 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.

Note: We intentionally use a slightly wasteful `set_config()` function
(which does not reuse a single `strbuf`, for example, though performance
_really_ does not matter here) for convenience and readability.

Also note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 200 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  35 +++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 262 insertions(+), 5 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 91ceb97e552..13cdfa94d16 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -258,6 +259,204 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	strbuf_addstr(&out, "-\n");
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		char *ref = out.buf;
+
+		while ((ref = strstr(ref + 1, "\nref: "))) {
+			const char *p;
+			char *head, *branch;
+
+			ref += strlen("\nref: ");
+			head = strstr(ref, "\tHEAD");
+
+			if (!head || memchr(ref, '\n', head - ref))
+				continue;
+
+			if (skip_prefix(ref, "refs/heads/", &p)) {
+				branch = xstrndup(p, head - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(head - ref), ref);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -354,6 +553,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index e1f629fddad..90d59f1d79f 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,19 +30,43 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+    Clones the specified repository, similar to linkgit:git-clone[1]. By
+    default, only commit and tree objects are cloned. Once finished, the
+    worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
+
+-b <name>::
+--branch <name>::
+    Instead of checking out the branch pointed to by the cloned repository's
+    HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+    A sparse-checkout is initialized by default. This behavior can be turned
+    off via `--full-clone`.
+
 List
 ~~~~
 
 list::
     To see which repositories are currently registered by the service, run
-    `scalar list`. This subcommand does not need to be run inside a Scalar
-    enlistment.
+    `scalar list`. This subcommand, like `clone`, does not need to be run
+    inside a Scalar enlistment.
 
 Register
 ~~~~~~~~
@@ -61,7 +86,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index ef0e8d680d5..295398f62cc 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
-- 
gitgitgadget

[PATCH 09/15] scalar: teach 'clone' to support the --single-branch option

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 13cdfa94d16..908eaa84df1 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 90d59f1d79f..bb9411b38cb 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
     Instead of checking out the branch pointed to by the cloned repository's
     HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+    Clone only the history leading to the tip of a single branch,
+    either specified by the `--branch` option or the primary
+    branch remote's `HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
     A sparse-checkout is initialized by default. This behavior can be turned
     off via `--full-clone`.
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 295398f62cc..9a35ab4fde6 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
-- 
gitgitgadget

[PATCH 10/15] scalar: implement the `run` command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.

The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 908eaa84df1..d5d38a1afeb 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (argc == 0)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0]))
+		i = -1;
+	else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -562,6 +625,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index bb9411b38cb..9aadaf6323f 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -94,6 +95,24 @@ unregister [<enlistment>]::
     Remove the specified repository from the list of repositories
     registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+    Run the given maintenance task (or all tasks, if `all` was specified).
+    Except for `all` and `config`, this subcommand simply hands off to
+    linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+    `pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 81 ++++++++++++++++++++------------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 68 insertions(+), 29 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index d5d38a1afeb..4eff3464a13 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,29 +142,30 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.bare", "false" },
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.bare", "false", 1 },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -173,7 +176,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -238,7 +242,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -425,7 +429,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -490,6 +494,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -626,6 +648,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 9aadaf6323f..227e3542a07 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -113,6 +114,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 9a35ab4fde6..e6d74a06ca0 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
-- 
gitgitgadget

[PATCH 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        | 10 ++++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 68 insertions(+), 6 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 4eff3464a13..1f8778cbb39 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 227e3542a07..2a1a0695b4d 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. This option is meant to to be run every time Scalar
+was upgraded.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index e6d74a06ca0..5fe7fabd0e5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
-- 
gitgitgadget

[PATCH 13/15] scalar: implement the `delete` command

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).

On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 1f8778cbb39..c616e91231a 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	res = delete_enlistment(&enlistment);
+	strbuf_release(&enlistment);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 2a1a0695b4d..00923023243 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -125,6 +126,13 @@ With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. This option is meant to to be run every time Scalar
 was upgraded.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+    This subcommand lets you delete an existing Scalar enlistment from your
+    local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 5fe7fabd0e5..7e8771d0eff 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
-- 
gitgitgadget

[PATCH 14/15] scalar: implement the `version` command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git versions`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index c616e91231a..be0a49b0d75 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -9,6 +9,7 @@
 #include "run-command.h"
 #include "refs.h"
 #include "dir.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -362,6 +363,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -748,6 +758,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -759,6 +797,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
-- 
gitgitgadget

[PATCH 15/15] scalar: accept -C and -c options before the subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git` executable has these two very useful options:

-C <directory>:
	switch to the specified directory before performing any actions

-c <key>=<value>:
	temporarily configure this setting for the duration of the
	specified scalar subcommand

With this commit, we teach the `scalar` executable the same trick.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index be0a49b0d75..41ba4b2f3b1 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -816,7 +835,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 00923023243..6d85640ef42 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+    Before running the subcommand, change the working directory. This
+    option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+    For the duration of running the specified subcommand, configure this
+    setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
-- 
gitgitgadget

Re: [PATCH 00/15] [RFC] Upstreaming the Scalar command

[Derrick Stolee]

On 8/30/21 5:34 PM, Johannes Schindelin via GitGitGadget wrote:
> tl;dr: This series contributes the Scalar command to the Git project. This
> command provides an opinionated way to create and configure repositories
> with a focus on very large repositories.

I want to give Johannes a big thanks for organizing this RFC. As you
can see from the authorship of the patches, this was an amazingly
collaborative effort, but Johannes led the way by creating a base that
the rest of us could work with, then finally he brought in all of the
gritty details to finish the effort.

> Background
> ==========

...

> The Scalar project
> was created to make that separation, refine the key concepts, and then
> extract those features into the new Scalar command.

When people have asked me how Scalar fits with the core Git client, I
point them to our "Philosophy of Scalar" document [1]. The most concise
summary of our goals since starting Scalar has been that Scalar aligns
with features already within Git that enable scale. I've said several
times that we are constantly making Scalar do less by making Git do more.

[1] https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/philosophy.md

Here is an example: when our large, internal customer told us that they
required Linux support for Scalar, we looked at what it would take. We
could have done the necessary platform-specific things to convince .NET
Core to create a long-running process that launched Git maintenance tasks
at different intervals, creating a similar mechanism to the Windows and
macOS services that did those operations. But we also knew that the
existing system was stuck with architectural decisions from VFS for Git
that were not actually in service of how Scalar worked. Instead, we
decided to build background maintenance into Git itself and had our Linux
port of Scalar run "git maintenance start".

Once the Linux port was proven out with Git's background maintenance, we
realized that the window where a user actually interacts with Scalar instead
of Git is extremely narrow: users run "scalar clone" or "scalar register"
and otherwise only run Git commands. The Scalar process does not need to
exist outside of that. (There are some other helpers that can be used in
a pinch to diagnose and fix problems, but they are rarely used. These
commands, such as 'scalar diagnose' can be contributed separately.)

It became clear that for our own needs it would be easier to ship one
installer that included the microsoft/git fork and the Scalar CLI, and
it would be simple to rewrite the Scalar CLI with all of the Git helper
APIs. We organized the code in a way that we thought would be amenable
to an upstream contribution (by placing in contrib/ and using Git code
style).

The thing about these commands is that they are _opinionated_. We rely
on these opinions for important internal users, but we realize that they
are not necessarily optimal for all users. Hence, we did not think it
wise to push those opinions onto the 'git' executable. Having 'scalar'
continue to live as a separate executable made sense to us.

I believe that by contributing Scalar to the full community, that we
create opportunities for Git in the future. For one, users and Git
distributors can opt into compiling Scalar so it is more available
to users who are interested. Another hopeful idea is that maybe this
reinvigorates ideas of how to streamline Git clones for large repos
without users needing to learn each and every knob to twist to get
things working. Since the Scalar CLI is contributed in the full
license of the Git project, pieces of it can be adapted into Git
proper as needed.

I look forward to hearing your thoughts.

Thanks,
-Stolee

Re: [PATCH 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Eric Sunshine]

On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
> After a Scalar upgrade, it can come in really handy if there is an easy
> way to reconfigure all Scalar enlistments. This new option offers this
> functionality.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
> diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
> @@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
> +With the `--all` option, all enlistments currently registered with Scalar
> +will be reconfigured. This option is meant to to be run every time Scalar
> +was upgraded.

s/was/is/

Re: [PATCH 14/15] scalar: implement the `version` command

[Eric Sunshine]

On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
> The .NET version of Scalar has a `version` command. This was necessary
> because it was versioned independently of Git.
>
> Since Scalar is now tightly coupled with Git, it does not make sense for
> them to show different versions. Therefore, it shows the same output as
> `git versions`. For backwards-compatibility with the .NET version,

s/versions/version/

> `scalar version` prints to `stderr`, though (`git version` prints to
> `stdout` instead).
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Re: [PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:

> [...]
> +#ifndef WIN32
> +		{ "core.untrackedCache", "true" },
> +#else
> +		/*
> +		 * Unfortunately, Scalar's Functional Tests demonstrated
> +		 * that the untracked cache feature is unreliable on Windows
> +		 * (which is a bummer because that platform would benefit the
> +		 * most from it). For some reason, freshly created files seem
> +		 * not to update the directory's `lastModified` time
> +		 * immediately, but the untracked cache would need to rely on
> +		 * that.
> +		 *
> +		 * Therefore, with a sad heart, we disable this very useful
> +		 * feature on Windows.
> +		 */
> +		{ "core.untrackedCache", "false" },
> +#endif
> [...]

Ok, but why the need to set it to "false" explicitly? Does it need to be
so opinionated as to overwrite existing user-set config in these cases?

> +		{ "core.bare", "false" },

Shouldn't this be set by "git init" already?

> [...]
> +		{ "core.logAllRefUpdates", "true" },

An opinionated thing unrelated to performance?

> [...]
> +		{ "feature.manyFiles", "false" },
> +		{ "feature.experimental", "false" },

Ditto the question about the need to set this, these are false by
default, right?

> [...]
> +		if (git_config_get_string(config[i].key, &value)) {
> +			trace2_data_string("scalar", the_repository, config[i].key, "created");
> +			if (git_config_set_gently(config[i].key,
> +						  config[i].value) < 0)
> +				return error(_("could not configure %s=%s"),
> +					     config[i].key, config[i].value);
> +		} else {
> +			trace2_data_string("scalar", the_repository, config[i].key, "exists");
> +			free(value);
> +		}

The commit message doesn't discuss these trace2 additions, these in
particular seem like they might be useful, but better done as as some
more general trace2 intergration in config.c, i.e. if the functions
being called here did the same logging on config set/get.

Re: [PATCH 03/15] scalar: create test infrastructure

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:

> To test the Scalar command, create a test script in contrib/scalar/t
> that is executed as `make -C contrib/scalar test`. Since Scalar has no
> meaningful capabilities yet, the only test is rather simple. We will add
> more tests in subsequent commits that introduce corresponding, new
> functionality.

As a comment on 01..03/15: I'd really prefer if we stop using this
pattern of sub-Makefile, the dependencies are a pain to manage, and we
end up copy/pasting large sets of functionality.

That would mean just adding the build of this command to the top-level
Makefile behind some "CONTRIB_SCALAR" flag or whatever, but I find that
much cleaner than....

> @@ -21,7 +22,7 @@ include ../../config.mak.uname
>  TARGETS = scalar$(X) scalar.o
>  GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
>  
> -all: scalar$X
> +all: scalar$X ../../bin-wrappers/scalar
>  
> [...]
> +../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
> [...]
>  scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel

...things like this, which refer to assets built by other Makefiles, and
need to plaster over the dependency issues...

> +++ b/contrib/scalar/t/Makefile
> @@ -0,0 +1,78 @@
> +# Run scalar tests
> +#
> +# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
> +#
> +
> +-include ../../../config.mak.autogen
> +-include ../../../config.mak
> +
> +SHELL_PATH ?= $(SHELL)
> +PERL_PATH ?= /usr/bin/perl
> +RM ?= rm -f
> +PROVE ?= prove
> +DEFAULT_TEST_TARGET ?= test
> +TEST_LINT ?= test-lint
> +
> +ifdef TEST_OUTPUT_DIRECTORY
> +TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
> +else
> +TEST_RESULTS_DIRECTORY = ../../../t/test-results
> +endif
> +
> +# Shell quote;
> +SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
> +PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
> +TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
> +
> +T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
> +
> +all: $(DEFAULT_TEST_TARGET)
> +
> +test: $(TEST_LINT)
> +	$(MAKE) aggregate-results-and-cleanup
> +
> +prove: $(TEST_LINT)
> +	@echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
> +	$(MAKE) clean-except-prove-cache
> +
> +$(T):
> +	@echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
> +
> +clean-except-prove-cache:
> +	$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
> +	$(RM) -r valgrind/bin
> +
> +clean: clean-except-prove-cache
> +	$(RM) .prove
> +
> +test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax
> +
> +test-lint-duplicates:
> +	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
> +		test -z "$$dups" || { \
> +		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
> +
> +test-lint-executable:
> +	@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
> +		test -z "$$bad" || { \
> +		echo >&2 "non-executable tests:" $$bad; exit 1; }
> +
> +test-lint-shell-syntax:
> +	@'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T)
> +
> +aggregate-results-and-cleanup: $(T)
> +	$(MAKE) aggregate-results
> +	$(MAKE) clean
> +
> +aggregate-results:
> +	for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
> +		echo "$$f"; \
> +	done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh
> +
> +valgrind:
> +	$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
> +
> +test-results:
> +	mkdir -p test-results
> +
> +.PHONY: $(T) aggregate-results clean valgrind

...and this entire copy/pasting & adjusting of t/Makefile.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:

> This implements Scalar's opinionated `clone` command: it tries to use a
> partial clone and sets up a sparse checkout by default. In contrast to
> `git clone`, `scalar clone` sets up the worktree in the `src/`
> subdirectory, to encourage a separation between the source files and the
> build output (which helps Git tremendously because it avoids untracked
> files that have to be specifically ignored when refreshing the index).

Perhaps nobody else wondered this while reading this, but I thought this
might be some sparse/worktree magic where cloning into "foo" would have
"foo/.git", but the worktree was somehow magically mapped at foo/src/".

But no, it just takes your "scalar clone <url> foo" and translates it to
"foo/src", so you'll get a directory at "foo".

> Note: We intentionally use a slightly wasteful `set_config()` function
> (which does not reuse a single `strbuf`, for example, though performance
> _really_ does not matter here) for convenience and readability.

FWIW I think the commit message could do without this, that part of the
code is obviously not performance sensitive at all. But maybe an
explicit note helps anyway...

Re: [PATCH 10/15] scalar: implement the `run` command

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:

> +	const char *usagestr[] = { NULL, NULL };

Missing usage strings?

> +	if (argc == 0)

Style nit (per style guide): s/argc == 0/!argc/g.

> +	if (!strcmp("all", argv[0]))
> +		i = -1;

Style nit (per style guide): missing braces here.

(Just noting the style nits once, but more in this patch, and presumably
the rest of the series...)

Re: [PATCH 11/15] scalar: allow reconfiguring an existing enlistment

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:

> This comes in handy during Scalar upgrades, or when config settings were
> messed up by mistake.

> [...]
>  		const char *key;
>  		const char *value;
> +		int overwrite_on_reconfigure;

If you make this a "keep_on_reconfigure", then ...

>  	} config[] = {
> -		{ "am.keepCR", "true" },
> -		{ "core.FSCache", "true" },
> -		{ "core.multiPackIndex", "true" },
> -		{ "core.preloadIndex", "true" },
> +		/* Required */
> +		{ "am.keepCR", "true", 1 },
> +		{ "core.FSCache", "true", 1 },
> +		{ "core.multiPackIndex", "true", 1 },
> +		{ "core.preloadIndex", "true", 1 },

You won't need the churn/boilerplate of adding "1" to everything here,
but can just change the initial patch to use designated initializers.

That along with a throwaway macro like:

#define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
#define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")

Might (or might not) make this even easier to eyeball...

Re: [PATCH 15/15] scalar: accept -C and -c options before the subcommand

[Ævar Arnfjörð Bjarmason]

On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:

> The `git` executable has these two very useful options:
>
> -C <directory>:
> 	switch to the specified directory before performing any actions
>
> -c <key>=<value>:
> 	temporarily configure this setting for the duration of the
> 	specified scalar subcommand
>
> With this commit, we teach the `scalar` executable the same trick.
> [...]
> +	while (argc > 1 && *argv[1] == '-') {
> +		if (!strcmp(argv[1], "-C")) {
> +			if (argc < 3)
> +				die(_("-C requires a <directory>"));
> +			if (chdir(argv[2]) < 0)
> +				die_errno(_("could not change to '%s'"),
> +					  argv[2]);
> +			argc -= 2;
> +			argv += 2;
> +		} else if (!strcmp(argv[1], "-c")) {
> +			if (argc < 3)
> +				die(_("-c requires a <key>=<value> argument"));
> +			git_config_push_parameter(argv[2]);
> +			argc -= 2;
> +			argv += 2;
> +		} else
> +			break;
> +	}

This along with my earlier comment about the Makefile copy/pasting makes
me wonder if an easier way to integrate this wouldn't be to refactor
git.c a bit to have it understand either "git" or "scalar", then instead
of "ls-tree" etc. as "git" the subcommands would become "built-ins".

Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
elimante the need for the inevetable future divergence of wanting -p,
-P, --exec-path etc. in both.

Re: [PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Derrick Stolee]

On 8/31/2021 4:11 AM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
> 
>> [...]
>> +#ifndef WIN32
>> +		{ "core.untrackedCache", "true" },
>> +#else
>> +		/*
>> +		 * Unfortunately, Scalar's Functional Tests demonstrated
>> +		 * that the untracked cache feature is unreliable on Windows
>> +		 * (which is a bummer because that platform would benefit the
>> +		 * most from it). For some reason, freshly created files seem
>> +		 * not to update the directory's `lastModified` time
>> +		 * immediately, but the untracked cache would need to rely on
>> +		 * that.
>> +		 *
>> +		 * Therefore, with a sad heart, we disable this very useful
>> +		 * feature on Windows.
>> +		 */
>> +		{ "core.untrackedCache", "false" },
>> +#endif
>> [...]
> 
> Ok, but why the need to set it to "false" explicitly? Does it need to be
> so opinionated as to overwrite existing user-set config in these cases?

Users can overwrite this local config value, but this is placed to avoid
a global config value from applying specifically within Scalar-created
repos.
 
>> +		{ "core.bare", "false" },
> 
> Shouldn't this be set by "git init" already?

This one is probably a bit _too_ defensive. It can be removed.

>> [...]
>> +		{ "core.logAllRefUpdates", "true" },
> 
> An opinionated thing unrelated to performance?

It's an opinionated thing related to supporting monorepo users. It helps
us diagnose issues they have by recreating a sequence of events.

>> [...]
>> +		{ "feature.manyFiles", "false" },
>> +		{ "feature.experimental", "false" },
> 
> Ditto the question about the need to set this, these are false by
> default, right?

But if a user has them on globally, then we don't want them to apply
locally (in favor of the settings that we set explicitly).

>> [...]
>> +		if (git_config_get_string(config[i].key, &value)) {
>> +			trace2_data_string("scalar", the_repository, config[i].key, "created");
>> +			if (git_config_set_gently(config[i].key,
>> +						  config[i].value) < 0)
>> +				return error(_("could not configure %s=%s"),
>> +					     config[i].key, config[i].value);
>> +		} else {
>> +			trace2_data_string("scalar", the_repository, config[i].key, "exists");
>> +			free(value);
>> +		}
> 
> The commit message doesn't discuss these trace2 additions, these in
> particular seem like they might be useful, but better done as as some
> more general trace2 intergration in config.c, i.e. if the functions
> being called here did the same logging on config set/get.

If we want to do such a tracing change within git_config_set*(), then
that would be an appropriate replacement. The biggest reason to include
them here is to trace that an existing value already exists, for the
case of running 'scalar reconfigure' during an upgrade. That part
doesn't make much sense to put into config.c.

Thanks,
-Stolee

Re: [PATCH 15/15] scalar: accept -C and -c options before the subcommand

[Derrick Stolee]

On 8/31/2021 4:32 AM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
> 
>> The `git` executable has these two very useful options:
>>
>> -C <directory>:
>> 	switch to the specified directory before performing any actions
>>
>> -c <key>=<value>:
>> 	temporarily configure this setting for the duration of the
>> 	specified scalar subcommand
>>
>> With this commit, we teach the `scalar` executable the same trick.
>> [...]
>> +	while (argc > 1 && *argv[1] == '-') {
>> +		if (!strcmp(argv[1], "-C")) {
>> +			if (argc < 3)
>> +				die(_("-C requires a <directory>"));
>> +			if (chdir(argv[2]) < 0)
>> +				die_errno(_("could not change to '%s'"),
>> +					  argv[2]);
>> +			argc -= 2;
>> +			argv += 2;
>> +		} else if (!strcmp(argv[1], "-c")) {
>> +			if (argc < 3)
>> +				die(_("-c requires a <key>=<value> argument"));
>> +			git_config_push_parameter(argv[2]);
>> +			argc -= 2;
>> +			argv += 2;
>> +		} else
>> +			break;
>> +	}
> 
> This along with my earlier comment about the Makefile copy/pasting makes
> me wonder if an easier way to integrate this wouldn't be to refactor
> git.c a bit to have it understand either "git" or "scalar", then instead
> of "ls-tree" etc. as "git" the subcommands would become "built-ins".
> 
> Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
> elimante the need for the inevetable future divergence of wanting -p,
> -P, --exec-path etc. in both.
 
Such a change would likely eliminate the ability to not include Scalar
when building the Git codebase, which we tried to avoid by keeping it
within contrib and have it be compiled via an opt-in flag.

If we want to talk about integrating Scalar into Git in a deeper way,
then that is an interesting discussion to have, but it lives at a much
higher level than Makefile details.

The questions we are really looking to answer in this RFC are:

1. Will the Git project accept Scalar into its codebase?

2. What is the best place for Scalar to live in the Git codebase?

We erred on the side of keeping Scalar as optional as possible. If
the community is more interested in a deeper integration, then that
could be an interesting direction.

In my opinion, I think the current tactic is safest. We could always
decide on a deeper integration later by moving the code around. It
seems harder to do the reverse.

Thanks,
-Stolee

Re: [PATCH 15/15] scalar: accept -C and -c options before the subcommand

[Ævar Arnfjörð Bjarmason]

On Tue, Aug 31 2021, Derrick Stolee wrote:

> On 8/31/2021 4:32 AM, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
>> 
>>> The `git` executable has these two very useful options:
>>>
>>> -C <directory>:
>>> 	switch to the specified directory before performing any actions
>>>
>>> -c <key>=<value>:
>>> 	temporarily configure this setting for the duration of the
>>> 	specified scalar subcommand
>>>
>>> With this commit, we teach the `scalar` executable the same trick.
>>> [...]
>>> +	while (argc > 1 && *argv[1] == '-') {
>>> +		if (!strcmp(argv[1], "-C")) {
>>> +			if (argc < 3)
>>> +				die(_("-C requires a <directory>"));
>>> +			if (chdir(argv[2]) < 0)
>>> +				die_errno(_("could not change to '%s'"),
>>> +					  argv[2]);
>>> +			argc -= 2;
>>> +			argv += 2;
>>> +		} else if (!strcmp(argv[1], "-c")) {
>>> +			if (argc < 3)
>>> +				die(_("-c requires a <key>=<value> argument"));
>>> +			git_config_push_parameter(argv[2]);
>>> +			argc -= 2;
>>> +			argv += 2;
>>> +		} else
>>> +			break;
>>> +	}
>> 
>> This along with my earlier comment about the Makefile copy/pasting makes
>> me wonder if an easier way to integrate this wouldn't be to refactor
>> git.c a bit to have it understand either "git" or "scalar", then instead
>> of "ls-tree" etc. as "git" the subcommands would become "built-ins".
>> 
>> Which would give us both "[git|scalar] [-c ...] <cmd>" for free, and
>> elimante the need for the inevetable future divergence of wanting -p,
>> -P, --exec-path etc. in both.
>  
> Such a change would likely eliminate the ability to not include Scalar
> when building the Git codebase, which we tried to avoid by keeping it
> within contrib and have it be compiled via an opt-in flag.

I mean to still have it behind a flag, but to handle it similar to how
we handle NO_CURL, EXCLUDED_PROGRAMS and the like, i.e. not requiring
parallel maintenance of copy/pasted Makefile logic in contrib/.

> If we want to talk about integrating Scalar into Git in a deeper way,
> then that is an interesting discussion to have, but it lives at a much
> higher level than Makefile details.

To be clear I'm proposing no change at all in term of what happens when
you run "make install", just commenting on the implementation details of
how we arrange for things to be built and configured before that step.

I realize that this is following some prior art of
e.g. contrib/subtree/Makefile, but IMNSHO that approach is a historical
mistake we should be backing out of. There was some recent discussion of
this here:
https://lore.kernel.org/git/87pmz4ig4o.fsf@evledraar.gmail.com/

E.g. now we have some painful management of the depencency graph between
/Makefile and Documentation/Makefile requiring fixes like 56550ea7180
(Makefile: add missing dependencies of 'config-list.h', 2021-04-08),
adding yet another Makefile into the mix which (to take one example)
depends on doc.dep, which in turn depends on ...; It's all a bunch of
needless complexity we can avoid.

> The questions we are really looking to answer in this RFC are:
>
> 1. Will the Git project accept Scalar into its codebase?
>
> 2. What is the best place for Scalar to live in the Git codebase?
>
> We erred on the side of keeping Scalar as optional as possible. If
> the community is more interested in a deeper integration, then that
> could be an interesting direction.

Indeed, to be clear I realize I'm entirely punting on the real questions
you're interested in. I just gave this an initial cursory skimming for
now, I have not formed an informed opinion on your #1, but just a little
bit of #2.

My initial reaction to #1 without having looked into it deeply is some
combination of "sure, why not?", and that the people/group contributing
major scalability work to git.git should be given the benefit of the
doubt. Maybe we won't keep "scalar" long-term, or change its UI etc.,
all of that can be handled in some carefully worded documentation
somewhere.

Of course all these suggestions I'm making about Makefile arrangement
are rather pointless if there isn't consensus to get past the hurdle of
your #1.

> In my opinion, I think the current tactic is safest. We could always
> decide on a deeper integration later by moving the code around. It
> seems harder to do the reverse.

I think "deeper integration" is the reverse of what you think it is.

I.e. if I'm patching or maintaining part of the Makefile logic to it's
deeper (or perhaps "gnarlier" is the righ word?) integration to need to
duplicate that work in two places, or always take into account that some
not-built-by-default-but-quite-common command's *.txt docs and *.sh
tests live in some unusual place for the purposes of CI, lint, tooling
etc.

In other words, it's a question of how much net complexity is being
added to the (build) system. That complexity doesn't automatically
reduce just because some files live in another directory, sometimes
that's an increase in complexity.

Whereas just conditionally adding it to some list in the top-level
Makefile (or Documentation/Makefile) is relatively maintenance-free, and
to our users / packagers the result should be the same or near enough.
It won't matter to them if building the optional thing is another "make"
command or just a flag to the existing "make" command.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Eric Sunshine]

On Tue, Aug 31, 2021 at 8:04 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
> > Note: We intentionally use a slightly wasteful `set_config()` function
> > (which does not reuse a single `strbuf`, for example, though performance
> > _really_ does not matter here) for convenience and readability.
>
> FWIW I think the commit message could do without this, that part of the
> code is obviously not performance sensitive at all. But maybe an
> explicit note helps anyway...

FWIW, I also found this distracting; it takes the reader's attention
away from more important aspects of the patch. (But it alone is not
worth a re-roll; it was just a minor hiccup.)

Re: [PATCH 00/15] [RFC] Upstreaming the Scalar command

[Elijah Newren]

On Mon, Aug 30, 2021 at 5:52 PM Derrick Stolee <stolee@gmail.com> wrote:
>
> On 8/30/21 5:34 PM, Johannes Schindelin via GitGitGadget wrote:
> > tl;dr: This series contributes the Scalar command to the Git project. This
> > command provides an opinionated way to create and configure repositories
> > with a focus on very large repositories.
>
> I want to give Johannes a big thanks for organizing this RFC. As you
> can see from the authorship of the patches, this was an amazingly
> collaborative effort, but Johannes led the way by creating a base that
> the rest of us could work with, then finally he brought in all of the
> gritty details to finish the effort.
>
> > Background
> > ==========
>
> ...
>
> > The Scalar project
> > was created to make that separation, refine the key concepts, and then
> > extract those features into the new Scalar command.
>
> When people have asked me how Scalar fits with the core Git client, I
> point them to our "Philosophy of Scalar" document [1]. The most concise
> summary of our goals since starting Scalar has been that Scalar aligns
> with features already within Git that enable scale. I've said several
> times that we are constantly making Scalar do less by making Git do more.
>
> [1] https://github.com/microsoft/git/blob/HEAD/contrib/scalar/docs/philosophy.md
>
> Here is an example: when our large, internal customer told us that they
> required Linux support for Scalar, we looked at what it would take. We
> could have done the necessary platform-specific things to convince .NET
> Core to create a long-running process that launched Git maintenance tasks
> at different intervals, creating a similar mechanism to the Windows and
> macOS services that did those operations. But we also knew that the
> existing system was stuck with architectural decisions from VFS for Git
> that were not actually in service of how Scalar worked. Instead, we
> decided to build background maintenance into Git itself and had our Linux
> port of Scalar run "git maintenance start".
>
> Once the Linux port was proven out with Git's background maintenance, we
> realized that the window where a user actually interacts with Scalar instead
> of Git is extremely narrow: users run "scalar clone" or "scalar register"
> and otherwise only run Git commands. The Scalar process does not need to
> exist outside of that. (There are some other helpers that can be used in
> a pinch to diagnose and fix problems, but they are rarely used. These
> commands, such as 'scalar diagnose' can be contributed separately.)
>
> It became clear that for our own needs it would be easier to ship one
> installer that included the microsoft/git fork and the Scalar CLI, and
> it would be simple to rewrite the Scalar CLI with all of the Git helper
> APIs. We organized the code in a way that we thought would be amenable
> to an upstream contribution (by placing in contrib/ and using Git code
> style).
>
> The thing about these commands is that they are _opinionated_. We rely
> on these opinions for important internal users, but we realize that they
> are not necessarily optimal for all users. Hence, we did not think it
> wise to push those opinions onto the 'git' executable. Having 'scalar'
> continue to live as a separate executable made sense to us.
>
> I believe that by contributing Scalar to the full community, that we
> create opportunities for Git in the future. For one, users and Git
> distributors can opt into compiling Scalar so it is more available
> to users who are interested. Another hopeful idea is that maybe this
> reinvigorates ideas of how to streamline Git clones for large repos
> without users needing to learn each and every knob to twist to get
> things working. Since the Scalar CLI is contributed in the full
> license of the Git project, pieces of it can be adapted into Git
> proper as needed.
>
> I look forward to hearing your thoughts.
>
> Thanks,
> -Stolee

Looks like exciting stuff, you two.  I'm behind on review as it is; I
still need to get back to Stolee's sparse-index add/rm/mv series, but
I'll try to circle back and take a look.

Re: [PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Junio C Hamano]

"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +static void setup_enlistment_directory(int argc, const char **argv,
> +				       const char * const *usagestr,
> +				       const struct option *options,
> +				       struct strbuf *enlistment_root)
> +{
> +	struct strbuf path = STRBUF_INIT;
> +	char *root;
> +	int enlistment_found = 0;
> +
> +	if (startup_info->have_repository)
> +		BUG("gitdir already set up?!?");
> +
> +	if (argc > 1)
> +		usage_with_options(usagestr, options);
> +
> +	/* find the worktree, determine its corresponding root */
> +	if (argc == 1)
> +		strbuf_add_absolute_path(&path, argv[0]);
> +	else if (strbuf_getcwd(&path) < 0)
> +		die(_("need a working directory"));
> +
> +	strbuf_trim_trailing_dir_sep(&path);
> +	do {
> +		const size_t len = path.len;
> +
> +		/* check if currently in enlistment root with src/ workdir */
> +		strbuf_addstr(&path, "/src/.git");
> +		if (is_git_directory(path.buf)) {
> +			strbuf_strip_suffix(&path, "/.git");
> +
> +			if (enlistment_root)
> +				strbuf_add(enlistment_root, path.buf, len);
> +
> +			enlistment_found = 1;
> +			break;
> +		}

This special casing of "normally the top of the working tree is
enlisted, but if the repository is called src/, then we enslist
one level up" is a bit of eyesore because

 (1) it is unclear why such a directory with 'src/' subdirectory is
     so special, and

 (2) it fails to serve those who has the same need but named their
     source subdirectory differently (like 'source/').

"The design decisions we made are all part of being opinionated" can
all explain it away, but at least we should let the users know where
the opinionated choices scalar makes want to lead them to, and this
"src/" stuff needs a bit of clarification.  Perhaps a documentation
will be added in later steps?

> +	for (i = 0; config[i].key; i++) {
> +		if (git_config_get_string(config[i].key, &value)) {
> +			trace2_data_string("scalar", the_repository, config[i].key, "created");
> +			if (git_config_set_gently(config[i].key,
> +						  config[i].value) < 0)
> +				return error(_("could not configure %s=%s"),
> +					     config[i].key, config[i].value);
> +		} else {
> +			trace2_data_string("scalar", the_repository, config[i].key, "exists");
> +			free(value);
> +		}

I wonder if we should have a table of configuration variables and
their default values.  The above code implements a skewed "we only
avoid overriding what is explicitly configured".  A variable that
the user left unconfigured because the user found its default
satisfactory will be overridden, and if the value scalar wants to
use happens to be the default value, we leave an explicit
configuration to that default value in the resulting configuration
file.

But I think the above is the best we can do without such a central
registry of configuration variables.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Junio C Hamano]

"Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
writes:

> +static char *remote_default_branch(const char *url)
> +{
> +	struct child_process cp = CHILD_PROCESS_INIT;
> +	struct strbuf out = STRBUF_INIT;
> +
> +	cp.git_cmd = 1;
> +	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
> +	strbuf_addstr(&out, "-\n");

Is this a workaround for the problem that the first "ref:" line
won't be found by looking for "\nref: " in the loop?  Cute, but the
extra "-" is a bit misleading.

> +	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> +		char *ref = out.buf;
> +
> +		while ((ref = strstr(ref + 1, "\nref: "))) {
> +			const char *p;
> +			char *head, *branch;
> +
> +			ref += strlen("\nref: ");
> +			head = strstr(ref, "\tHEAD");
> +
> +			if (!head || memchr(ref, '\n', head - ref))
> +				continue;

OK.  We expect "ref: " <refname> "\t" <head> "\n" where <head> is
"HEAD" for their .git/HEAD and refs/remotes/<nick>/HEAD for their
remote-tracking branch for the remote they call <nick>, on a single
line.  We reject a line that is not of that shape, and we reject a
line that is about remote-tracking branch by only looking for
"\tHEAD". Makes sense.

The strstr() goes from "ref + 1", which feels sloppy.  When we
reject the line we found that begins with "ref :", I would have
expected that the next scan would start at the beginning of the next
line, not from the middle of this line at the first letter 'e' in
'refs/heads/' on the current line "ref: refs/heads/.....".  As long
as the current line is long enough, strstr() would not miss the
beginning of the next line, so it might be OK.

> +			if (skip_prefix(ref, "refs/heads/", &p)) {
> +				branch = xstrndup(p, head - p);
> +				strbuf_release(&out);
> +				return branch;
> +			}
> +
> +			error(_("remote HEAD is not a branch: '%.*s'"),
> +			      (int)(head - ref), ref);
> +			strbuf_release(&out);
> +			return NULL;

OK.  Any symref whose basename is HEAD in their remote-tracking
hierarchy would have been rejected earlier in the loop.

Is there a particular reason why we return early here, instead of
breaking out of hte loop and let the generic "failed to get" code
path below to handle this case?

> +		}
> +	}
> +	warning(_("failed to get default branch name from remote; "
> +		  "using local default"));
> +	strbuf_reset(&out);
> +
> +	child_process_init(&cp);
> +	cp.git_cmd = 1;
> +	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
> +	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> +		strbuf_trim(&out);
> +		return strbuf_detach(&out, NULL);
> +	}
> +
> +	strbuf_release(&out);
> +	error(_("failed to get default branch name"));
> +	return NULL;
> +}

> +static int cmd_clone(int argc, const char **argv)
> +{
> +	const char *branch = NULL;
> +	int full_clone = 0;
> +	struct option clone_options[] = {
> +		OPT_STRING('b', "branch", &branch, N_("<branch>"),
> +			   N_("branch to checkout after clone")),
> +		OPT_BOOL(0, "full-clone", &full_clone,
> +			 N_("when cloning, create full working directory")),
> +		OPT_END(),
> +	};
> +	const char * const clone_usage[] = {
> +		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
> +		NULL
> +	};
> +	const char *url;
> +	char *enlistment = NULL, *dir = NULL;
> +	struct strbuf buf = STRBUF_INIT;
> +	int res;
> +
> +	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
> +
> +	if (argc == 2) {
> +		url = argv[0];
> +		enlistment = xstrdup(argv[1]);
> +	} else if (argc == 1) {
> +		url = argv[0];
> +
> +		strbuf_addstr(&buf, url);
> +		/* Strip trailing slashes, if any */
> +		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
> +			strbuf_setlen(&buf, buf.len - 1);
> +		/* Strip suffix `.git`, if any */
> +		strbuf_strip_suffix(&buf, ".git");
> +
> +		enlistment = find_last_dir_sep(buf.buf);
> +		if (!enlistment) {
> +			die(_("cannot deduce worktree name from '%s'"), url);
> +		}
> +		enlistment = xstrdup(enlistment + 1);
> +	} else {
> +		usage_msg_opt(_("You must specify a repository to clone."),
> +			      clone_usage, clone_options);
> +	}
> +
> +	if (is_directory(enlistment))
> +		die(_("directory '%s' exists already"), enlistment);
> +
> +	dir = xstrfmt("%s/src", enlistment);
> +
> +	strbuf_reset(&buf);
> +	if (branch)
> +		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
> +	else {
> +		char *b = repo_default_branch_name(the_repository, 1);
> +		strbuf_addf(&buf, "init.defaultBranch=%s", b);
> +		free(b);

Doesn't "git clone" already use their HEAD without having to make an
extra "git ls-remote" roundtrip?

Ahh, you do not do "git clone"; you do "git init", set things up,
and then "git fetch" and checkout, all manually.

Which is kind of shame.

I wonder if it is a cleaner implementation to give a new option to
"git clone" that gives a command sequence (not necessarily have to
be implemented as a shell script) that specifies necessary
pre-configuration steps to be done before the command starts the
transfer step.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Derrick Stolee]

On 9/1/2021 12:45 PM, Junio C Hamano wrote:
> "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
> 
...
>> +	dir = xstrfmt("%s/src", enlistment);
>> +
>> +	strbuf_reset(&buf);
>> +	if (branch)
>> +		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
>> +	else {
>> +		char *b = repo_default_branch_name(the_repository, 1);
>> +		strbuf_addf(&buf, "init.defaultBranch=%s", b);
>> +		free(b);
> 
> Doesn't "git clone" already use their HEAD without having to make an
> extra "git ls-remote" roundtrip?
> 
> Ahh, you do not do "git clone"; you do "git init", set things up,
> and then "git fetch" and checkout, all manually.
> 
> Which is kind of shame.
> 
> I wonder if it is a cleaner implementation to give a new option to
> "git clone" that gives a command sequence (not necessarily have to
> be implemented as a shell script) that specifies necessary
> pre-configuration steps to be done before the command starts the
> transfer step.

I agree that 'git clone' plus maybe some more improvements like
'--sparse=cone' to set up cone-mode sparse-checkout would be good.
And also the implementation being contributed here is cleaner if
we can use 'git clone'.

We are trying to balance a clean upstream implementation with some
custom things that we still need in our microsoft/git fork to
handle the integration with the GVFS Protocol (i.e. partial clone
on Azure Repos). That customization is cleaner to keep here in the
scalar code instead of adding an option to 'git clone'. It is
difficult to justify code patterns here due to choices we have made
in our fork, so I _could_ see a way to replace those custom bits
with new, custom flags to 'git clone'. It just requires additional
investment during our integration when we incorporate these upstream
changes. Naturally, I'm motivated to avoid that extra work.

If your opinion to switch to 'git clone' is a strong one, then I
could see us doing that change. I just want you to be aware of the
hidden reasons for choices like these.

Thanks,
-Stolee

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin]

Hi Junio,

On Wed, 1 Sep 2021, Junio C Hamano wrote:

> "Johannes Schindelin via GitGitGadget" <gitgitgadget@gmail.com>
> writes:
>
> > +static char *remote_default_branch(const char *url)
> > +{
> > +	struct child_process cp = CHILD_PROCESS_INIT;
> > +	struct strbuf out = STRBUF_INIT;
> > +
> > +	cp.git_cmd = 1;
> > +	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
> > +	strbuf_addstr(&out, "-\n");
>
> Is this a workaround for the problem that the first "ref:" line
> won't be found by looking for "\nref: " in the loop?  Cute, but the
> extra "-" is a bit misleading.

The `-` is actually needed because of the `ref + 1` below, over which you
stumbled.

>
> > +	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> > +		char *ref = out.buf;
> > +
> > +		while ((ref = strstr(ref + 1, "\nref: "))) {
> > +			const char *p;
> > +			char *head, *branch;
> > +
> > +			ref += strlen("\nref: ");
> > +			head = strstr(ref, "\tHEAD");
> > +
> > +			if (!head || memchr(ref, '\n', head - ref))
> > +				continue;
>
> OK.  We expect "ref: " <refname> "\t" <head> "\n" where <head> is
> "HEAD" for their .git/HEAD and refs/remotes/<nick>/HEAD for their
> remote-tracking branch for the remote they call <nick>, on a single
> line.  We reject a line that is not of that shape, and we reject a
> line that is about remote-tracking branch by only looking for
> "\tHEAD". Makes sense.
>
> The strstr() goes from "ref + 1", which feels sloppy.

I would use a different adjective, one that is less judgemental in nature,
but then, you were talking about your feelings.

> When we reject the line we found that begins with "ref :", I would have
> expected that the next scan would start at the beginning of the next
> line, not from the middle of this line at the first letter 'e' in
> 'refs/heads/' on the current line "ref: refs/heads/.....".  As long as
> the current line is long enough, strstr() would not miss the beginning
> of the next line, so it might be OK.

It would even work if the current line is shorter, but as you point out:
it is wasteful. And it could be improved to be more readable. I reworked
it, and it now looks like this:

	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
		const char *line = out.buf;

		while (*line) {
			const char *eol = strchrnul(line, '\n'), *p;
			size_t len = eol - line;
			char *branch;

			if (!skip_prefix(line, "ref: ", &p) ||
			    !strip_suffix_mem(line, &len, "\tHEAD")) {
				line = eol + (*eol == '\n');
				continue;
			}

			eol = line + len;
			if (skip_prefix(p, "refs/heads/", &p)) {
				branch = xstrndup(p, eol - p);
				strbuf_release(&out);
				return branch;
			}

			error(_("remote HEAD is not a branch: '%.*s'"),
			      (int)(eol - p), p);
			strbuf_release(&out);
			return NULL;
		}
	}

It now parses the output line by line, looking for the expected prefix and
suffix, then verifies the ref name format, and either returns the short
branch name or errors out with the message that this is not a branch.

>
> > +			if (skip_prefix(ref, "refs/heads/", &p)) {
> > +				branch = xstrndup(p, head - p);
> > +				strbuf_release(&out);
> > +				return branch;
> > +			}
> > +
> > +			error(_("remote HEAD is not a branch: '%.*s'"),
> > +			      (int)(head - ref), ref);
> > +			strbuf_release(&out);
> > +			return NULL;
>
> OK.  Any symref whose basename is HEAD in their remote-tracking
> hierarchy would have been rejected earlier in the loop.
>
> Is there a particular reason why we return early here, instead of
> breaking out of hte loop and let the generic "failed to get" code
> path below to handle this case?

Yes, the reason is that I wanted to err on the side of caution. If the
remote repository reports a default branch that is not a default branch at
all, I do not want to pretend that things are fine and then run into
trouble later when we set up a non-branch as remote-tracking target or
something like that.

>
> > +		}
> > +	}
> > +	warning(_("failed to get default branch name from remote; "
> > +		  "using local default"));
> > +	strbuf_reset(&out);
> > +
> > +	child_process_init(&cp);
> > +	cp.git_cmd = 1;
> > +	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
> > +	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> > +		strbuf_trim(&out);
> > +		return strbuf_detach(&out, NULL);
> > +	}
> > +
> > +	strbuf_release(&out);
> > +	error(_("failed to get default branch name"));
> > +	return NULL;
> > +}
>
> > +static int cmd_clone(int argc, const char **argv)
> > +{
> > +	const char *branch = NULL;
> > +	int full_clone = 0;
> > +	struct option clone_options[] = {
> > +		OPT_STRING('b', "branch", &branch, N_("<branch>"),
> > +			   N_("branch to checkout after clone")),
> > +		OPT_BOOL(0, "full-clone", &full_clone,
> > +			 N_("when cloning, create full working directory")),
> > +		OPT_END(),
> > +	};
> > +	const char * const clone_usage[] = {
> > +		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
> > +		NULL
> > +	};
> > +	const char *url;
> > +	char *enlistment = NULL, *dir = NULL;
> > +	struct strbuf buf = STRBUF_INIT;
> > +	int res;
> > +
> > +	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
> > +
> > +	if (argc == 2) {
> > +		url = argv[0];
> > +		enlistment = xstrdup(argv[1]);
> > +	} else if (argc == 1) {
> > +		url = argv[0];
> > +
> > +		strbuf_addstr(&buf, url);
> > +		/* Strip trailing slashes, if any */
> > +		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
> > +			strbuf_setlen(&buf, buf.len - 1);
> > +		/* Strip suffix `.git`, if any */
> > +		strbuf_strip_suffix(&buf, ".git");
> > +
> > +		enlistment = find_last_dir_sep(buf.buf);
> > +		if (!enlistment) {
> > +			die(_("cannot deduce worktree name from '%s'"), url);
> > +		}
> > +		enlistment = xstrdup(enlistment + 1);
> > +	} else {
> > +		usage_msg_opt(_("You must specify a repository to clone."),
> > +			      clone_usage, clone_options);
> > +	}
> > +
> > +	if (is_directory(enlistment))
> > +		die(_("directory '%s' exists already"), enlistment);
> > +
> > +	dir = xstrfmt("%s/src", enlistment);
> > +
> > +	strbuf_reset(&buf);
> > +	if (branch)
> > +		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
> > +	else {
> > +		char *b = repo_default_branch_name(the_repository, 1);
> > +		strbuf_addf(&buf, "init.defaultBranch=%s", b);
> > +		free(b);
>
> Doesn't "git clone" already use their HEAD without having to make an
> extra "git ls-remote" roundtrip?
>
> Ahh, you do not do "git clone"; you do "git init", set things up,
> and then "git fetch" and checkout, all manually.
>
> Which is kind of shame.
>
> I wonder if it is a cleaner implementation to give a new option to
> "git clone" that gives a command sequence (not necessarily have to
> be implemented as a shell script) that specifies necessary
> pre-configuration steps to be done before the command starts the
> transfer step.

Right. It is a shame, I agree. And it is one of the things I want to work
on, after the Scalar patch series made it into Git.

The reason why I don't want to work on this now is that I expect this
effort to result in new options for `git clone`, new options that need to
be designed well, and where I fully expect a long discussion until we
reach a consensus how these options should look like, especially since we
will need to maintain backwards-compatibility of Scalar's functionality.

Therefore, in the interest to keep the patch series relatively easy to
review, I left this in the "for later" pile, for now.

Ciao,
Dscho

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 958 bytes --]

Hi Eric,

On Tue, 31 Aug 2021, Eric Sunshine wrote:

> On Tue, Aug 31, 2021 at 8:04 AM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
> > On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
> > > Note: We intentionally use a slightly wasteful `set_config()` function
> > > (which does not reuse a single `strbuf`, for example, though performance
> > > _really_ does not matter here) for convenience and readability.
> >
> > FWIW I think the commit message could do without this, that part of the
> > code is obviously not performance sensitive at all. But maybe an
> > explicit note helps anyway...
>
> FWIW, I also found this distracting; it takes the reader's attention
> away from more important aspects of the patch. (But it alone is not
> worth a re-roll; it was just a minor hiccup.)

Since I reworked the remote default branch parsing anyway, I removed this
paragraph from the commit message.

Ciao,
Dscho

Re: [PATCH 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin]

Hi Eric,

On Tue, 31 Aug 2021, Eric Sunshine wrote:

> On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> > After a Scalar upgrade, it can come in really handy if there is an easy
> > way to reconfigure all Scalar enlistments. This new option offers this
> > functionality.
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> > ---
> > diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
> > @@ -121,6 +121,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
> > +With the `--all` option, all enlistments currently registered with Scalar
> > +will be reconfigured. This option is meant to to be run every time Scalar
> > +was upgraded.
>
> s/was/is/

I wanted to convey a temporal order, so I changed it to "every time after
Scalar is upgraded". Okay?

Ciao,
Dscho

Re: [PATCH 14/15] scalar: implement the `version` command

[Johannes Schindelin]

Hi Eric,

On Tue, 31 Aug 2021, Eric Sunshine wrote:

> On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> > The .NET version of Scalar has a `version` command. This was necessary
> > because it was versioned independently of Git.
> >
> > Since Scalar is now tightly coupled with Git, it does not make sense for
> > them to show different versions. Therefore, it shows the same output as
> > `git versions`. For backwards-compatibility with the .NET version,
>
> s/versions/version/

Thank you!
Dscho

>
> > `scalar version` prints to `stderr`, though (`git version` prints to
> > `stdout` instead).
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>

Re: [PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Johannes Schindelin]

Hi Junio,

On Wed, 1 Sep 2021, Junio C Hamano wrote:

> "Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
> > +static void setup_enlistment_directory(int argc, const char **argv,
> > +				       const char * const *usagestr,
> > +				       const struct option *options,
> > +				       struct strbuf *enlistment_root)
> > +{
> > +	struct strbuf path = STRBUF_INIT;
> > +	char *root;
> > +	int enlistment_found = 0;
> > +
> > +	if (startup_info->have_repository)
> > +		BUG("gitdir already set up?!?");
> > +
> > +	if (argc > 1)
> > +		usage_with_options(usagestr, options);
> > +
> > +	/* find the worktree, determine its corresponding root */
> > +	if (argc == 1)
> > +		strbuf_add_absolute_path(&path, argv[0]);
> > +	else if (strbuf_getcwd(&path) < 0)
> > +		die(_("need a working directory"));
> > +
> > +	strbuf_trim_trailing_dir_sep(&path);
> > +	do {
> > +		const size_t len = path.len;
> > +
> > +		/* check if currently in enlistment root with src/ workdir */
> > +		strbuf_addstr(&path, "/src/.git");
> > +		if (is_git_directory(path.buf)) {
> > +			strbuf_strip_suffix(&path, "/.git");
> > +
> > +			if (enlistment_root)
> > +				strbuf_add(enlistment_root, path.buf, len);
> > +
> > +			enlistment_found = 1;
> > +			break;
> > +		}
>
> This special casing of "normally the top of the working tree is
> enlisted, but if the repository is called src/, then we enslist
> one level up" is a bit of eyesore because
>
>  (1) it is unclear why such a directory with 'src/' subdirectory is
>      so special, and
>
>  (2) it fails to serve those who has the same need but named their
>      source subdirectory differently (like 'source/').

All true. I wish we had come up with a better way, or with a way to
override this via an option.

Unfortunately, we are now bound by the fact that there are already users
out there...

> "The design decisions we made are all part of being opinionated" can
> all explain it away, but at least we should let the users know where
> the opinionated choices scalar makes want to lead them to, and this
> "src/" stuff needs a bit of clarification.  Perhaps a documentation
> will be added in later steps?

I had hoped that the initial blurb of the manual page was sufficient, but
you're right, the `register` subcommand is particular in that it allows to
force Scalar to consider the worktree to be identical to the Scalar
enlistment. I added this:

	diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
	index 1593da45eae..568987064b2 100644
	--- a/contrib/scalar/scalar.txt
	+++ b/contrib/scalar/scalar.txt
	@@ -40,6 +40,10 @@ register [<enlistment>]::
		and starts background maintenance. If `<enlistment>` is not provided,
		then the enlistment associated with the current working directory is
		registered.
	++
	+Note: when this subcommand is called in a worktree that is called `src/`, its
	+parent directory is considered to be the Scalar enlistment. If the worktree is
	+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.

> > +	for (i = 0; config[i].key; i++) {
> > +		if (git_config_get_string(config[i].key, &value)) {
> > +			trace2_data_string("scalar", the_repository, config[i].key, "created");
> > +			if (git_config_set_gently(config[i].key,
> > +						  config[i].value) < 0)
> > +				return error(_("could not configure %s=%s"),
> > +					     config[i].key, config[i].value);
> > +		} else {
> > +			trace2_data_string("scalar", the_repository, config[i].key, "exists");
> > +			free(value);
> > +		}
>
> I wonder if we should have a table of configuration variables and
> their default values.  The above code implements a skewed "we only
> avoid overriding what is explicitly configured".  A variable that
> the user left unconfigured because the user found its default
> satisfactory will be overridden, and if the value scalar wants to
> use happens to be the default value, we leave an explicit
> configuration to that default value in the resulting configuration
> file.
>
> But I think the above is the best we can do without such a central
> registry of configuration variables.

Even with such a central registry, there would still be the question
whether the user, by staying with the default, wanted Git (or in this
instance, Scalar) to keep using the old default. The intention is
unfortunately not clear just from setting the variable.

So I think this is the best we can do.

Ciao,
Dscho

Re: [PATCH 10/15] scalar: implement the `run` command

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

Hi Ævar,

On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
>
> > +	const char *usagestr[] = { NULL, NULL };
>
> Missing usage strings?

This command will show a generated usage, i.e. a non-static string. It
therefore cannot be specified here already. See the `strbuf_*()` calls
populating `buf` and the `usagestr[0] = buf.buf;` assignment.

> > +	if (argc == 0)
>
> Style nit (per style guide): s/argc == 0/!argc/g.

It is true that we often do this, but in this instance it would be
misleading: `argc` is a counter, not a Boolean.

> > +	if (!strcmp("all", argv[0]))
> > +		i = -1;
>
> Style nit (per style guide): missing braces here.

The style guide specifically allows my preference to leave single-line
blocks without curlies.

Ciao,
Johannes

Re: [PATCH 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 1470 bytes --]

Hi Ævar,

On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:

>
> On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
>
> > This comes in handy during Scalar upgrades, or when config settings were
> > messed up by mistake.
>
> > [...]
> >  		const char *key;
> >  		const char *value;
> > +		int overwrite_on_reconfigure;
>
> If you make this a "keep_on_reconfigure", then ...

I do not think that this would be a better name, or that renaming this
field would do anything except cause more work for me.

>
> >  	} config[] = {
> > -		{ "am.keepCR", "true" },
> > -		{ "core.FSCache", "true" },
> > -		{ "core.multiPackIndex", "true" },
> > -		{ "core.preloadIndex", "true" },
> > +		/* Required */
> > +		{ "am.keepCR", "true", 1 },
> > +		{ "core.FSCache", "true", 1 },
> > +		{ "core.multiPackIndex", "true", 1 },
> > +		{ "core.preloadIndex", "true", 1 },
>
> You won't need the churn/boilerplate of adding "1" to everything here,
> but can just change the initial patch to use designated initializers.
>
> That along with a throwaway macro like:
>
> #define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
> #define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")
>
> Might (or might not) make this even easier to eyeball...

To me, it makes things less readable. There is an entire section with the
header `/* Optional */` below, and I want this list to stay as readable as
it is now.

Ciao,
Dscho

Re: [PATCH 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Eric Sunshine]

On Fri, Sep 3, 2021 at 11:23 AM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> On Tue, 31 Aug 2021, Eric Sunshine wrote:
> > On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
> > > +With the `--all` option, all enlistments currently registered with Scalar
> > > +will be reconfigured. This option is meant to to be run every time Scalar
> > > +was upgraded.
> >
> > s/was/is/
>
> I wanted to convey a temporal order, so I changed it to "every time after
> Scalar is upgraded". Okay?

I think I understood the intent of the original, but it causes a
grammatical hiccup. Your revised version can work, although I might
write it this way:

    This option is meant to be run each time Scalar is upgraded.

However, perhaps that is too ambiguous and some users may think that
the process of upgrading Scalar will automatically run this command,
and you'd like to make it clear that it is the user's responsibility.
So, perhaps:

    Use this option after each Scalar upgrade.

or something.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Junio C Hamano]

Derrick Stolee <stolee@gmail.com> writes:

>> Ahh, you do not do "git clone"; you do "git init", set things up,
>> and then "git fetch" and checkout, all manually.
>> 
>> Which is kind of shame.
>> 
>> I wonder if it is a cleaner implementation to give a new option to
>> "git clone" that gives a command sequence (not necessarily have to
>> be implemented as a shell script) that specifies necessary
>> pre-configuration steps to be done before the command starts the
>> transfer step.
>
> I agree that 'git clone' plus maybe some more improvements like
> '--sparse=cone' to set up cone-mode sparse-checkout would be good.
> And also the implementation being contributed here is cleaner if
> we can use 'git clone'.
>
> We are trying to balance a clean upstream implementation with some
> custom things that we still need in our microsoft/git fork to
> handle the integration with the GVFS Protocol (i.e. partial clone
> on Azure Repos). That customization is cleaner to keep here in the
> scalar code instead of adding an option to 'git clone'.

Oh, there is no disagreement on that point, at least in the short
term.  I was wondering why "clone" subcommand needs a duplicated
logic that should be unnecessary, before realizing that this was
not implemented as a wrapper to (possibly updated) "clone", and
I agree that starting with a looser coupling like this step does
is easier to everybody.

"Kind of shame" is just that I wished we had already prepared "git
clone" side to accept customization more easily before its various
distinct phases (new repository creation, where a custom logic may
want to affect the name and location of it and how "git init" is
driven, initial "fetch", where a custom logic may want to affect the
fetch refspec and its parameters like depths and cones, and initial
"checkout") do their things.  If we allowed such plug-in of logic to
affect how "git clone" worked already, it would have been possible
to do "scalar clone" with much less code.  It also would allow us to
reorganize the "clone --local" hack in a way that is easier to
reason about (I think even in today's code, the way I hooked it up
can be seen which is quite messy).  It may even help folks who want
to extend "git clone" to clone a repository recursively its
submodules with project-specific customizations (like which ones to
clone by default, etc.).

I suspect that learning from the way "scalar clone" is done on top
of "init" + "fetch" + "checkout" in this initial series may help us
extend "git clone" later to fill such needs.

> If your opinion to switch to 'git clone' is a strong one, then I
> could see us doing that change. I just want you to be aware of the
> hidden reasons for choices like these.

Not at all at this moment.

It is mostly that the way "init" + "fetch" + "checkout" was done in
this step reminded me of a much longer-term wish I have had for a
while.

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> It would even work if the current line is shorter, but as you point out:
> it is wasteful. And it could be improved to be more readable. I reworked
> it, and it now looks like this:
>
> 	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> 		const char *line = out.buf;
>
> 		while (*line) {
> 			const char *eol = strchrnul(line, '\n'), *p;
> 			size_t len = eol - line;
> 			char *branch;
>
> 			if (!skip_prefix(line, "ref: ", &p) ||
> 			    !strip_suffix_mem(line, &len, "\tHEAD")) {
> 				line = eol + (*eol == '\n');
> 				continue;
> 			}
>
> 			eol = line + len;
> 			if (skip_prefix(p, "refs/heads/", &p)) {
> 				branch = xstrndup(p, eol - p);
> 				strbuf_release(&out);
> 				return branch;
> 			}
>
> 			error(_("remote HEAD is not a branch: '%.*s'"),
> 			      (int)(eol - p), p);
> 			strbuf_release(&out);
> 			return NULL;
> 		}
> 	}
>
> It now parses the output line by line, looking for the expected prefix and
> suffix, then verifies the ref name format, and either returns the short
> branch name or errors out with the message that this is not a branch.

It is much easier to read and understand how the loop works with
above.

>> > +			error(_("remote HEAD is not a branch: '%.*s'"),
>> > +			      (int)(head - ref), ref);
>> > +			strbuf_release(&out);
>> > +			return NULL;
>>
>> OK.  Any symref whose basename is HEAD in their remote-tracking
>> hierarchy would have been rejected earlier in the loop.
>>
>> Is there a particular reason why we return early here, instead of
>> breaking out of hte loop and let the generic "failed to get" code
>> path below to handle this case?
>
> Yes, the reason is that I wanted to err on the side of caution. If the
> remote repository reports a default branch that is not a default branch at
> all, I do not want to pretend that things are fine and then run into
> trouble later when we set up a non-branch as remote-tracking target or
> something like that.

Wouldn't we have the same problem when the remote end does not
advertise HEAD and we fall back to "local default", though?  We'd
run into trouble later as we use "local default" that may correspond
to a non-branch there as remote-tracking target, or something like
that.

Not that I care too deeply in the error case, though.  I just felt
that this early return was an uneven way to follow the principle to
err on the side of caution, as we continue with the local default
when the other side fails to tell us what their HEAD points at.

Thanks.

Re: [PATCH 04/15] scalar: 'register' sets recommended config and starts maintenance

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

>> "The design decisions we made are all part of being opinionated" can
>> all explain it away, but at least we should let the users know where
>> the opinionated choices scalar makes want to lead them to, and this
>> "src/" stuff needs a bit of clarification.  Perhaps a documentation
>> will be added in later steps?
>
> I had hoped that the initial blurb of the manual page was sufficient, but
> you're right, the `register` subcommand is particular in that it allows to
> force Scalar to consider the worktree to be identical to the Scalar
> enlistment. I added this:

Sorry, if it weren't clear that I was commenting on each step as I
read along without peeking later steps.  I think I saw it was
written somewhere that this was to encourage use of read-only
directory that keeps the sources with build artifacts and crufts
created outside it (so forests of projects will not have the source
directories, each of which has its own .git/, next to each other---
instead we would have shell directories, each with its own src/ and
src/.git, next to each other).  The additional documentation below
is a good thing to have handy when readers learn how to use
"register" (or more generally, what an "enlistment" is).  As long as
the motivation behind that design is given somewhere (not necessarily
here) for readers to discover, I am OK with the design.

> 	diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
> 	index 1593da45eae..568987064b2 100644
> 	--- a/contrib/scalar/scalar.txt
> 	+++ b/contrib/scalar/scalar.txt
> 	@@ -40,6 +40,10 @@ register [<enlistment>]::
> 		and starts background maintenance. If `<enlistment>` is not provided,
> 		then the enlistment associated with the current working directory is
> 		registered.
> 	++
> 	+Note: when this subcommand is called in a worktree that is called `src/`, its
> 	+parent directory is considered to be the Scalar enlistment. If the worktree is
> 	+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.

Thanks.

Re: [PATCH 10/15] scalar: implement the `run` command

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi Ævar,
>
> On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:
>
>> On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
>>
>> > +	const char *usagestr[] = { NULL, NULL };
>>
>> Missing usage strings?
>
> This command will show a generated usage, i.e. a non-static string. It
> therefore cannot be specified here already. See the `strbuf_*()` calls
> populating `buf` and the `usagestr[0] = buf.buf;` assignment.
>
>> > +	if (argc == 0)
>>
>> Style nit (per style guide): s/argc == 0/!argc/g.
>
> It is true that we often do this, but in this instance it would be
> misleading: `argc` is a counter, not a Boolean.

That argument could be a plausible excuse to deviate from the style
if it were

	if (argc == 0)
		do no args case;
	else if (argc == 1)
		do one arg case;
	else if (argc == 2)
		do two args case;
	...

Replacing the first one with "if (!argc)" may make it less readable.

But I do not think the reasoning applies here

	if (argc == 0)
		do a thing that applies only to no args case;

without "else".  This is talking about "do we have any argument? Yes
or no?" Boolean here.

>> > +	if (!strcmp("all", argv[0]))
>> > +		i = -1;
>>
>> Style nit (per style guide): missing braces here.
>
> The style guide specifically allows my preference to leave single-line
> blocks without curlies.

Actually, the exception goes the other way, no?

We generally want to avoid such an unnecessary braces around a
single statement block.  But when we have an else clause that has a
block with multiple statements (hence braces are required), as an
exception, the guide asks you to write braces around the body of the
if side for consistency.

When you only have just a couple of lines on the "else {}" side, I
do not think it matters too much either way for readability, though.
I cannot see the "else" side in the above clause, but IIRC it wasn't
just a few lines, was it?

Thanks.

[PATCH v2 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin via GitGitGadget]

tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.


Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.

The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.


The present
===========

The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).


Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:

 1. git clone --sparse --filter=blob:none /src
 2. git -C /src sparse-checkout init --cone
 3. git -C /src config (many times)
 4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.

For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.


Epilogue
========

Now, to address some questions that I imagine every reader has who made it
this far:

 * Why not put the Scalar functionality directly into a built-in? Creating a
   Git builtin requires scrutiny over every aspect of the feature, which is
   difficult to do while also maintaining the command-line interface
   contract and expected behavior of the Scalar command (there are existing
   users, after all). By having the Scalar command in contrib/, we present a
   simple option for users to have these features in the short term while
   the Git contributor community decides which bits to absorb into Git
   built-ins.
 * Why implement the Scalar command in the Git codebase? We ported Scalar to
   the microsoft/git fork for several reasons. First, we realized it was
   possible now that the core features exist inside Git itself. Second,
   compiling Scalar directly within a version of Git allows us to remove a
   version compatibility check from each config option that might or might
   not apply based on the installed Git version. Finally, this new location
   has greatly simplified our release process and the installation process
   for users. We now have ways to install Scalar with microsoft/git via
   winget, brew, and apt-get. This has been the case since we shipped
   v2.32.0 to our users, read: this setup has served us well already.
 * Why contribute Scalar to the Git project? We are biased, of course, yet
   we do have evidence that the Scalar command is a helpful tool that offers
   an simple way to handle huge repositories with ease. By contributing it
   to the core Git project, we are able to share it with more users,
   especially some users who do not want to install the microsoft/git fork.
   We intend to include Scalar as a component in git-for-windows/git, but
   are contributing it here first. Further, we think there is benefit to the
   Git developer community as this presents an example of how to set certain
   defaults that work for large repositories.
 * Does this integrate with the built-in FSMonitor yet? No, not yet. I do
   have a couple of add-on patch series lined up, one of them being the
   integration with the built-in FSMonitor, which obviously has to wait
   until the FSMonitor patch series advances further.

Changes since v1:

 * A couple typos were fixed
 * The code parsing the output of ls-remote was made more readable
 * The indentation used in scalar.txt now consistently uses tabs
 * We no longer hard-code core.bare = false when registering with Scalar

Derrick Stolee (4):
  scalar: 'register' sets recommended config and starts maintenance
  scalar: 'unregister' stops background maintenance
  scalar: implement 'scalar list'
  scalar: implement the `run` command

Johannes Schindelin (10):
  scalar: create a rudimentary executable
  scalar: start documenting the command
  scalar: create test infrastructure
  scalar: let 'unregister' handle a deleted enlistment directory
    gracefully
  scalar: implement the `clone` subcommand
  scalar: teach 'clone' to support the --single-branch option
  scalar: allow reconfiguring an existing enlistment
  scalar: teach 'reconfigure' to optionally handle all registered
    enlistments
  scalar: implement the `version` command
  scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
  scalar: implement the `delete` command

 Makefile                         |   8 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 844 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 156 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1236 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh


base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v1:

  1:  b8c7d3f8450 =  1:  b8c7d3f8450 scalar: create a rudimentary executable
  2:  4f886575dcf =  2:  4f886575dcf scalar: start documenting the command
  3:  bcfde9bc765 =  3:  bcfde9bc765 scalar: create test infrastructure
  4:  3786f4c597f !  4:  ee3e26a0c4e scalar: 'register' sets recommended config and starts maintenance
     @@ contrib/scalar/scalar.c
      +		 */
      +		{ "core.untrackedCache", "false" },
      +#endif
     -+		{ "core.bare", "false" },
      +		{ "core.logAllRefUpdates", "true" },
      +		{ "credential.https://dev.azure.com.useHttpPath", "true" },
      +		{ "credential.validate", "false" }, /* GCM4W-only */
     @@ contrib/scalar/scalar.txt: will be identical to the worktree.
      +~~~~~~~~
      +
      +register [<enlistment>]::
     -+    Adds the enlistment's repository to the list of registered repositories
     -+    and starts background maintenance. If `<enlistment>` is not provided,
     -+    then the enlistment associated with the current working directory is
     -+    registered.
     ++	Adds the enlistment's repository to the list of registered repositories
     ++	and starts background maintenance. If `<enlistment>` is not provided,
     ++	then the enlistment associated with the current working directory is
     ++	registered.
     +++
     ++Note: when this subcommand is called in a worktree that is called `src/`, its
     ++parent directory is considered to be the Scalar enlistment. If the worktree is
     ++_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
      +
       SEE ALSO
       --------
  5:  2a6ac170e6b !  5:  6142f75875b scalar: 'unregister' stops background maintenance
     @@ contrib/scalar/scalar.txt: SYNOPSIS
       
       DESCRIPTION
       -----------
     -@@ contrib/scalar/scalar.txt: register [<enlistment>]::
     -     then the enlistment associated with the current working directory is
     -     registered.
     +@@ contrib/scalar/scalar.txt: Note: when this subcommand is called in a worktree that is called `src/`, its
     + parent directory is considered to be the Scalar enlistment. If the worktree is
     + _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
       
      +Unregister
      +~~~~~~~~~~
      +
      +unregister [<enlistment>]::
     -+    Remove the specified repository from the list of repositories
     -+    registered with Scalar and stop the scheduled background maintenance.
     ++	Remove the specified repository from the list of repositories
     ++	registered with Scalar and stop the scheduled background maintenance.
      +
       SEE ALSO
       --------
  6:  087fc9be194 =  6:  82dd253154f scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  c272ff4069d !  7:  fb7c931ddb3 scalar: implement 'scalar list'
     @@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
      +~~~~
      +
      +list::
     -+    To see which repositories are currently registered by the service, run
     -+    `scalar list`. This subcommand does not need to be run inside a Scalar
     -+    enlistment.
     ++	To see which repositories are currently registered by the service, run
     ++	`scalar list`. This subcommand does not need to be run inside a Scalar
     ++	enlistment.
      +
       Register
       ~~~~~~~~
  8:  2cbf0b61113 !  8:  f3223c10788 scalar: implement the `clone` subcommand
     @@ Commit message
          experience and experiments of the Microsoft Windows and the Microsoft
          Office development teams.
      
     -    Note: We intentionally use a slightly wasteful `set_config()` function
     -    (which does not reuse a single `strbuf`, for example, though performance
     -    _really_ does not matter here) for convenience and readability.
     -
     -    Also note: since the `scalar clone` command is by far the most commonly
     +    Note: since the `scalar clone` command is by far the most commonly
          called `scalar` subcommand, we document it at the top of the manual
          page.
      
     @@ contrib/scalar/scalar.c: static int unregister_dir(void)
      +
      +	cp.git_cmd = 1;
      +	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
     -+	strbuf_addstr(&out, "-\n");
      +	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
     -+		char *ref = out.buf;
     -+
     -+		while ((ref = strstr(ref + 1, "\nref: "))) {
     -+			const char *p;
     -+			char *head, *branch;
     ++		const char *line = out.buf;
      +
     -+			ref += strlen("\nref: ");
     -+			head = strstr(ref, "\tHEAD");
     ++		while (*line) {
     ++			const char *eol = strchrnul(line, '\n'), *p;
     ++			size_t len = eol - line;
     ++			char *branch;
      +
     -+			if (!head || memchr(ref, '\n', head - ref))
     ++			if (!skip_prefix(line, "ref: ", &p) ||
     ++			    !strip_suffix_mem(line, &len, "\tHEAD")) {
     ++				line = eol + (*eol == '\n');
      +				continue;
     ++			}
      +
     -+			if (skip_prefix(ref, "refs/heads/", &p)) {
     -+				branch = xstrndup(p, head - p);
     ++			eol = line + len;
     ++			if (skip_prefix(p, "refs/heads/", &p)) {
     ++				branch = xstrndup(p, eol - p);
      +				strbuf_release(&out);
      +				return branch;
      +			}
      +
      +			error(_("remote HEAD is not a branch: '%.*s'"),
     -+			      (int)(head - ref), ref);
     ++			      (int)(eol - p), p);
      +			strbuf_release(&out);
      +			return NULL;
      +		}
     @@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
      +~~~~~
      +
      +clone [<options>] <url> [<enlistment>]::
     -+    Clones the specified repository, similar to linkgit:git-clone[1]. By
     -+    default, only commit and tree objects are cloned. Once finished, the
     -+    worktree is located at `<enlistment>/src`.
     ++	Clones the specified repository, similar to linkgit:git-clone[1]. By
     ++	default, only commit and tree objects are cloned. Once finished, the
     ++	worktree is located at `<enlistment>/src`.
      ++
      +The sparse-checkout feature is enabled (except when run with `--full-clone`)
      +and the only files present are those in the top-level directory. Use
     @@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
      +
      +-b <name>::
      +--branch <name>::
     -+    Instead of checking out the branch pointed to by the cloned repository's
     -+    HEAD, check out the `<name>` branch instead.
     ++	Instead of checking out the branch pointed to by the cloned
     ++	repository's HEAD, check out the `<name>` branch instead.
      +
      +--[no-]full-clone::
     -+    A sparse-checkout is initialized by default. This behavior can be turned
     -+    off via `--full-clone`.
     ++	A sparse-checkout is initialized by default. This behavior can be
     ++	turned off via `--full-clone`.
      +
       List
       ~~~~
       
       list::
     -     To see which repositories are currently registered by the service, run
     --    `scalar list`. This subcommand does not need to be run inside a Scalar
     --    enlistment.
     -+    `scalar list`. This subcommand, like `clone`, does not need to be run
     -+    inside a Scalar enlistment.
     + 	To see which repositories are currently registered by the service, run
     +-	`scalar list`. This subcommand does not need to be run inside a Scalar
     +-	enlistment.
     ++	`scalar list`. This subcommand, like `clone`, does not need to be run
     ++	inside a Scalar enlistment.
       
       Register
       ~~~~~~~~
  9:  9af1c37c2ea !  9:  b3c4b3dccc6 scalar: teach 'clone' to support the --single-branch option
     @@ contrib/scalar/scalar.txt: scalar - an opinionated repository management tool
       scalar register [<enlistment>]
       scalar unregister [<enlistment>]
      @@ contrib/scalar/scalar.txt: subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
     -     Instead of checking out the branch pointed to by the cloned repository's
     -     HEAD, check out the `<name>` branch instead.
     + 	Instead of checking out the branch pointed to by the cloned
     + 	repository's HEAD, check out the `<name>` branch instead.
       
      +--[no-]single-branch::
     -+    Clone only the history leading to the tip of a single branch,
     -+    either specified by the `--branch` option or the primary
     -+    branch remote's `HEAD` points at.
     ++	Clone only the history leading to the tip of a single branch, either
     ++	specified by the `--branch` option or the primary branch remote's
     ++	`HEAD` points at.
      ++
      +Further fetches into the resulting repository will only update the
      +remote-tracking branch for the branch this option was used for the initial
     @@ contrib/scalar/scalar.txt: subdirectories outside your sparse-checkout by using
      +`--single-branch` clone was made, no remote-tracking branch is created.
      +
       --[no-]full-clone::
     -     A sparse-checkout is initialized by default. This behavior can be turned
     -     off via `--full-clone`.
     + 	A sparse-checkout is initialized by default. This behavior can be
     + 	turned off via `--full-clone`.
      
       ## contrib/scalar/t/t9099-scalar.sh ##
      @@ contrib/scalar/t/t9099-scalar.sh: test_expect_success 'set up repository to clone' '
 10:  c3f16bccd02 ! 10:  b7fc2dc29c8 scalar: implement the `run` command
     @@ contrib/scalar/scalar.txt: scalar clone [--single-branch] [--branch <main-branch
       DESCRIPTION
       -----------
      @@ contrib/scalar/scalar.txt: unregister [<enlistment>]::
     -     Remove the specified repository from the list of repositories
     -     registered with Scalar and stop the scheduled background maintenance.
     + 	Remove the specified repository from the list of repositories
     + 	registered with Scalar and stop the scheduled background maintenance.
       
      +Run
      +~~~
      +
      +scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
     -+    Run the given maintenance task (or all tasks, if `all` was specified).
     -+    Except for `all` and `config`, this subcommand simply hands off to
     -+    linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
     -+    `pack-files` to `incremental-repack`).
     ++	Run the given maintenance task (or all tasks, if `all` was specified).
     ++	Except for `all` and `config`, this subcommand simply hands off to
     ++	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
     ++	`pack-files` to `incremental-repack`).
      ++
      +These tasks are run automatically as part of the scheduled maintenance,
      +as soon as the repository is registered with Scalar. It should therefore
 11:  13056f02018 ! 11:  9a834c23d08 scalar: allow reconfiguring an existing enlistment
     @@ contrib/scalar/scalar.c: static int set_recommended_config(void)
      -		{ "core.untrackedCache", "false" },
      +		{ "core.untrackedCache", "false", 1 },
       #endif
     --		{ "core.bare", "false" },
      -		{ "core.logAllRefUpdates", "true" },
      -		{ "credential.https://dev.azure.com.useHttpPath", "true" },
      -		{ "credential.validate", "false" }, /* GCM4W-only */
     @@ contrib/scalar/scalar.c: static int set_recommended_config(void)
      -		{ "feature.experimental", "false" },
      -		{ "fetch.unpackLimit", "1" },
      -		{ "fetch.writeCommitGraph", "false" },
     -+		{ "core.bare", "false", 1 },
      +		{ "core.logAllRefUpdates", "true", 1 },
      +		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
      +		{ "credential.validate", "false", 1 }, /* GCM4W-only */
 12:  732a28c22fc ! 12:  79e9f5d203a scalar: teach 'reconfigure' to optionally handle all registered enlistments
     @@ contrib/scalar/scalar.txt: After a Scalar upgrade, or when the configuration of
       reconfigure the enlistment.
       
      +With the `--all` option, all enlistments currently registered with Scalar
     -+will be reconfigured. This option is meant to to be run every time Scalar
     -+was upgraded.
     ++will be reconfigured. This option is meant to to be run every time after
     ++Scalar is upgraded.
      +
       SEE ALSO
       --------
 13:  13afbd68812 ! 13:  94a21982652 scalar: implement the `delete` command
     @@ contrib/scalar/scalar.txt: scalar register [<enlistment>]
       DESCRIPTION
       -----------
      @@ contrib/scalar/scalar.txt: With the `--all` option, all enlistments currently registered with Scalar
     - will be reconfigured. This option is meant to to be run every time Scalar
     - was upgraded.
     + will be reconfigured. This option is meant to to be run every time after
     + Scalar is upgraded.
       
      +Delete
      +~~~~~~
      +
      +delete <enlistment>::
     -+    This subcommand lets you delete an existing Scalar enlistment from your
     -+    local file system, unregistering the repository.
     ++	This subcommand lets you delete an existing Scalar enlistment from your
     ++	local file system, unregistering the repository.
      +
       SEE ALSO
       --------
 14:  73d08c0c894 ! 14:  707d8e19683 scalar: implement the `version` command
     @@ Commit message
      
          Since Scalar is now tightly coupled with Git, it does not make sense for
          them to show different versions. Therefore, it shows the same output as
     -    `git versions`. For backwards-compatibility with the .NET version,
     +    `git version`. For backwards-compatibility with the .NET version,
          `scalar version` prints to `stderr`, though (`git version` prints to
          `stdout` instead).
      
 15:  6455b18f1b6 ! 15:  26e23b5c5e5 scalar: accept -C and -c options before the subcommand
     @@ contrib/scalar/scalar.txt: The `scalar` command implements various subcommands,
      +The following options can be specified _before_ the subcommand:
      +
      +-C <directory>::
     -+    Before running the subcommand, change the working directory. This
     -+    option imitates the same option of linkgit:git[1].
     ++	Before running the subcommand, change the working directory. This
     ++	option imitates the same option of linkgit:git[1].
      +
      +-c <key>=<value>::
     -+    For the duration of running the specified subcommand, configure this
     -+    setting. This option imitates the same option of linkgit:git[1].
     ++	For the duration of running the specified subcommand, configure this
     ++	setting. This option imitates the same option of linkgit:git[1].
      +
       COMMANDS
       --------

-- 
gitgitgadget

[PATCH v2 01/15] scalar: create a rudimentary executable

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.

For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.

With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

	make -C contrib/scalar/Makefile

This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.

The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.

An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Makefile                  |  8 ++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 80 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c

diff --git a/Makefile b/Makefile
index c3565fc0f8f..2d5c822f7a8 100644
--- a/Makefile
+++ b/Makefile
@@ -2447,6 +2447,10 @@ endif
 .PHONY: objects
 objects: $(OBJECTS)
 
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
 
@@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
new file mode 100644
index 00000000000..ff3d47e84d0
--- /dev/null
+++ b/contrib/scalar/.gitignore
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
new file mode 100644
index 00000000000..40c03ad10e1
--- /dev/null
+++ b/contrib/scalar/Makefile
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$X
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: all clean FORCE
diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
new file mode 100644
index 00000000000..7cff29e0fcd
--- /dev/null
+++ b/contrib/scalar/scalar.c
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
-- 
gitgitgadget

[PATCH v2 02/15] scalar: start documenting the command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit establishes the infrastructure to build the manual page for
the `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt

diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
index ff3d47e84d0..00441073f59 100644
--- a/contrib/scalar/.gitignore
+++ b/contrib/scalar/.gitignore
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 40c03ad10e1..85c186634e9 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: all clean docs FORCE
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
new file mode 100644
index 00000000000..5f7131861a5
--- /dev/null
+++ b/contrib/scalar/scalar.txt
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
-- 
gitgitgadget

[PATCH v2 03/15] scalar: create test infrastructure

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.

Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 85c186634e9..8620042f281 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$X
+all: scalar$X ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: all clean docs FORCE
+.PHONY: all clean docs test FORCE
diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile
new file mode 100644
index 00000000000..6170672bb37
--- /dev/null
+++ b/contrib/scalar/t/Makefile
@@ -0,0 +1,78 @@
+# Run scalar tests
+#
+# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
+#
+
+-include ../../../config.mak.autogen
+-include ../../../config.mak
+
+SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
+RM ?= rm -f
+PROVE ?= prove
+DEFAULT_TEST_TARGET ?= test
+TEST_LINT ?= test-lint
+
+ifdef TEST_OUTPUT_DIRECTORY
+TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
+else
+TEST_RESULTS_DIRECTORY = ../../../t/test-results
+endif
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
+TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
+
+T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
+
+all: $(DEFAULT_TEST_TARGET)
+
+test: $(TEST_LINT)
+	$(MAKE) aggregate-results-and-cleanup
+
+prove: $(TEST_LINT)
+	@echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
+	$(MAKE) clean-except-prove-cache
+
+$(T):
+	@echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+
+clean-except-prove-cache:
+	$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+	$(RM) -r valgrind/bin
+
+clean: clean-except-prove-cache
+	$(RM) .prove
+
+test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax
+
+test-lint-duplicates:
+	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+		test -z "$$dups" || { \
+		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
+
+test-lint-executable:
+	@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
+		test -z "$$bad" || { \
+		echo >&2 "non-executable tests:" $$bad; exit 1; }
+
+test-lint-shell-syntax:
+	@'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T)
+
+aggregate-results-and-cleanup: $(T)
+	$(MAKE) aggregate-results
+	$(MAKE) clean
+
+aggregate-results:
+	for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
+		echo "$$f"; \
+	done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh
+
+valgrind:
+	$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
+
+test-results:
+	mkdir -p test-results
+
+.PHONY: $(T) aggregate-results clean valgrind
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
new file mode 100755
index 00000000000..16f2b72b126
--- /dev/null
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
-- 
gitgitgadget

[PATCH v2 04/15] scalar: 'register' sets recommended config and starts maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.

The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 255 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 272 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7cff29e0fcd..0e627bb100e 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -5,11 +5,266 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_strip_suffix(&path, "/.git");
+
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		strbuf_addstr(&path, "/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_setlen(&path, len);
+
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+
+		strbuf_setlen(&path, len);
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 5f7131861a5..568987064b2 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v2 05/15] scalar: 'unregister' stops background maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.

The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 0e627bb100e..2b5c52a25f5 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -204,12 +204,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -220,24 +220,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -260,11 +275,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 568987064b2..d9a79984492 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v2 06/15] scalar: let 'unregister' handle a deleted enlistment directory gracefully

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 2b5c52a25f5..d114c038b64 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -275,6 +275,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -288,6 +306,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 16f2b72b126..ef0e8d680d5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
-- 
gitgitgadget

[PATCH v2 07/15] scalar: implement 'scalar list'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 12 +++++++++++-
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index d114c038b64..7f5436399da 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -257,6 +257,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -343,6 +353,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d9a79984492..f2528557a0c 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,20 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	To see which repositories are currently registered by the service, run
+	`scalar list`. This subcommand does not need to be run inside a Scalar
+	enlistment.
+
 Register
 ~~~~~~~~
 
-- 
gitgitgadget

[PATCH v2 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.

Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  35 +++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 263 insertions(+), 5 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7f5436399da..bf18003b297 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -257,6 +258,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -353,6 +553,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f2528557a0c..d4e3cb73fda 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,19 +30,43 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
 list::
 	To see which repositories are currently registered by the service, run
-	`scalar list`. This subcommand does not need to be run inside a Scalar
-	enlistment.
+	`scalar list`. This subcommand, like `clone`, does not need to be run
+	inside a Scalar enlistment.
 
 Register
 ~~~~~~~~
@@ -65,7 +90,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index ef0e8d680d5..295398f62cc 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
-- 
gitgitgadget

[PATCH v2 09/15] scalar: teach 'clone' to support the --single-branch option

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index bf18003b297..7dd1f28948f 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d4e3cb73fda..6ceb528f347 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 295398f62cc..9a35ab4fde6 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
-- 
gitgitgadget

[PATCH v2 10/15] scalar: implement the `run` command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.

The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7dd1f28948f..0452dfce915 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (argc == 0)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0]))
+		i = -1;
+	else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -562,6 +625,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 6ceb528f347..ff8792e5a64 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -98,6 +99,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v2 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 0452dfce915..bfbf58f7a91 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,28 +142,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -172,7 +175,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -237,7 +241,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -425,7 +429,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -490,6 +494,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -626,6 +648,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index ff8792e5a64..df6e961ca37 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -117,6 +118,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 9a35ab4fde6..e6d74a06ca0 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
-- 
gitgitgadget

[PATCH v2 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        | 10 ++++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 68 insertions(+), 6 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index bfbf58f7a91..7e98a1d6b06 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index df6e961ca37..8637e207860 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -125,6 +125,10 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. This option is meant to to be run every time after
+Scalar is upgraded.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index e6d74a06ca0..5fe7fabd0e5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
-- 
gitgitgadget

[PATCH v2 13/15] scalar: implement the `delete` command

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).

On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7e98a1d6b06..822d7c39903 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	res = delete_enlistment(&enlistment);
+	strbuf_release(&enlistment);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 8637e207860..b7ace8b9f1f 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -129,6 +130,13 @@ With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. This option is meant to to be run every time after
 Scalar is upgraded.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 5fe7fabd0e5..7e8771d0eff 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
-- 
gitgitgadget

[PATCH v2 14/15] scalar: implement the `version` command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 822d7c39903..e3349dce47b 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -9,6 +9,7 @@
 #include "run-command.h"
 #include "refs.h"
 #include "dir.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -362,6 +363,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -748,6 +758,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -759,6 +797,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
-- 
gitgitgadget

[PATCH v2 15/15] scalar: accept -C and -c options before the subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git` executable has these two very useful options:

-C <directory>:
	switch to the specified directory before performing any actions

-c <key>=<value>:
	temporarily configure this setting for the duration of the
	specified scalar subcommand

With this commit, we teach the `scalar` executable the same trick.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index e3349dce47b..670a259a12c 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -816,7 +835,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index b7ace8b9f1f..5804baf7d70 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
-- 
gitgitgadget

Re: [PATCH v2 07/15] scalar: implement 'scalar list'

[Bagas Sanjaya]

On 04/09/21 00.54, Derrick Stolee via GitGitGadget wrote:
> +List
> +~~~~
> +
> +list::
> +	To see which repositories are currently registered by the service, run
> +	`scalar list`. This subcommand does not need to be run inside a Scalar
> +	enlistment.
> +

I think the man-page-style wording should be:

> list::
> 	List enlistments that are currently registered by Scalar. This
> 	subcommand does not need to be run inside an enlistment.

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH v2 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:

> Changes since v1:
>
>  * A couple typos were fixed
>  * The code parsing the output of ls-remote was made more readable
>  * The indentation used in scalar.txt now consistently uses tabs
>  * We no longer hard-code core.bare = false when registering with Scalar

A summary of outstanding but unaddressed things would be useful during
re-rolls. In this case at least:

 - My point that the build system part of this is more complex than it
   probably needs to be:
   https://lore.kernel.org/git/87mtoxwt63.fsf@evledraar.gmail.com/

 - My point & Junio concurring with some style suggestions:
   https://lore.kernel.org/git/xmqqk0jxft1p.fsf@gitster.g

 - My minor style comment about the ", 1". Just saying it's
   "outstanding" because I think what you had & I just replied to in
   https://lore.kernel.org/git/87bl56plbi.fsf@evledraar.gmail.com/ might
   have been a dismissal of the macro suggestion (which I don't like
   either), not just the simpler "get rid of init verbosity to all 1".

Re: [PATCH 11/15] scalar: allow reconfiguring an existing enlistment

[Ævar Arnfjörð Bjarmason]

On Fri, Sep 03 2021, Johannes Schindelin wrote:

> Hi Ævar,
>
> On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:
>
>>
>> On Mon, Aug 30 2021, Johannes Schindelin via GitGitGadget wrote:
>>
>> > This comes in handy during Scalar upgrades, or when config settings were
>> > messed up by mistake.
>>
>> > [...]
>> >  		const char *key;
>> >  		const char *value;
>> > +		int overwrite_on_reconfigure;
>>
>> If you make this a "keep_on_reconfigure", then ...
>
> I do not think that this would be a better name, or that renaming this
> field would do anything except cause more work for me.

It would also result in more readable code, i.e. why add boilerplate ",
1" to a boolean field in this case if every single setting is set to
"1"? Doesn't it make more sense to invert the variable name & save on
the verbosity?

>>
>> >  	} config[] = {
>> > -		{ "am.keepCR", "true" },
>> > -		{ "core.FSCache", "true" },
>> > -		{ "core.multiPackIndex", "true" },
>> > -		{ "core.preloadIndex", "true" },
>> > +		/* Required */
>> > +		{ "am.keepCR", "true", 1 },
>> > +		{ "core.FSCache", "true", 1 },
>> > +		{ "core.multiPackIndex", "true", 1 },
>> > +		{ "core.preloadIndex", "true", 1 },
>>
>> You won't need the churn/boilerplate of adding "1" to everything here,
>> but can just change the initial patch to use designated initializers.
>>
>> That along with a throwaway macro like:
>>
>> #define SCALAR_CFG_TRUE(k) (.key = k, .value = "true")
>> #define SCALAR_CFG_FALSE(k) (.key = k, .value = "false")
>>
>> Might (or might not) make this even easier to eyeball...
>
> To me, it makes things less readable. There is an entire section with the
> header `/* Optional */` below, and I want this list to stay as readable as
> it is now.

Yeah, I think those macros are probably less readable too. I should have
phrased that as a "one could even...", but just the smaller change of
avoiding the ", 1" everywhere seems worthwhile.

Re: [PATCH v2 08/15] scalar: implement the `clone` subcommand

[Ævar Arnfjörð Bjarmason]

On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:

> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> This implements Scalar's opinionated `clone` command: it tries to use a
> partial clone and sets up a sparse checkout by default. In contrast to
> `git clone`, `scalar clone` sets up the worktree in the `src/`
> subdirectory, to encourage a separation between the source files and the
> build output (which helps Git tremendously because it avoids untracked
> files that have to be specifically ignored when refreshing the index).

Perhaps it's simpler to just say about that /src/ injection:

    `scalar clone` adds an implicit "/src" subdirectory to whatever
    directory the user provides, with the added stricture on top of
    doing that with "git clone" that the "src" cannot exist already.

...

> +	if (is_directory(enlistment))
> +		die(_("directory '%s' exists already"), enlistment);
> +
> +	dir = xstrfmt("%s/src", enlistment);

Which also seems to suggest a bug here. I.e. if I "git clone <repo>
/tmp/xyz/abc" and Ctrl+C it we'll remove "abc", but leave "xyz"
behind. Since we're creating that "xyz" (or "src") implicitly here an
abort/ctrl+C followed by a retry is going to run into this error, isn't
it?

I.e. it seems what's missing in this state machine is checking if the
directory was there already, and if it isn't add it to the existing
atexit() removals.

Which may be tricky seeing as this is shelling out to "init" then
"fetch" etc, i.e. who removes it? But maybe not.

Re: [PATCH 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin]

Hi Eric,

On Fri, 3 Sep 2021, Eric Sunshine wrote:

> On Fri, Sep 3, 2021 at 11:23 AM Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> > On Tue, 31 Aug 2021, Eric Sunshine wrote:
> > > On Mon, Aug 30, 2021 at 5:35 PM Johannes Schindelin via GitGitGadget
> > > > +With the `--all` option, all enlistments currently registered with Scalar
> > > > +will be reconfigured. This option is meant to to be run every time Scalar
> > > > +was upgraded.
> > >
> > > s/was/is/
> >
> > I wanted to convey a temporal order, so I changed it to "every time after
> > Scalar is upgraded". Okay?
>
> I think I understood the intent of the original, but it causes a
> grammatical hiccup. Your revised version can work, although I might
> write it this way:
>
>     This option is meant to be run each time Scalar is upgraded.
>
> However, perhaps that is too ambiguous and some users may think that
> the process of upgrading Scalar will automatically run this command,
> and you'd like to make it clear that it is the user's responsibility.
> So, perhaps:
>
>     Use this option after each Scalar upgrade.
>
> or something.

I like the last one best, too.

Thank you,
Dscho

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin]

Hi Junio,

On Fri, 3 Sep 2021, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > It would even work if the current line is shorter, but as you point out:
> > it is wasteful. And it could be improved to be more readable. I reworked
> > it, and it now looks like this:
> >
> > 	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
> > 		const char *line = out.buf;
> >
> > 		while (*line) {
> > 			const char *eol = strchrnul(line, '\n'), *p;
> > 			size_t len = eol - line;
> > 			char *branch;
> >
> > 			if (!skip_prefix(line, "ref: ", &p) ||
> > 			    !strip_suffix_mem(line, &len, "\tHEAD")) {
> > 				line = eol + (*eol == '\n');
> > 				continue;
> > 			}
> >
> > 			eol = line + len;
> > 			if (skip_prefix(p, "refs/heads/", &p)) {
> > 				branch = xstrndup(p, eol - p);
> > 				strbuf_release(&out);
> > 				return branch;
> > 			}
> >
> > 			error(_("remote HEAD is not a branch: '%.*s'"),
> > 			      (int)(eol - p), p);
> > 			strbuf_release(&out);
> > 			return NULL;
> > 		}
> > 	}
> >
> > It now parses the output line by line, looking for the expected prefix and
> > suffix, then verifies the ref name format, and either returns the short
> > branch name or errors out with the message that this is not a branch.
>
> It is much easier to read and understand how the loop works with
> above.

Excellent.

> >> > +			error(_("remote HEAD is not a branch: '%.*s'"),
> >> > +			      (int)(head - ref), ref);
> >> > +			strbuf_release(&out);
> >> > +			return NULL;
> >>
> >> OK.  Any symref whose basename is HEAD in their remote-tracking
> >> hierarchy would have been rejected earlier in the loop.
> >>
> >> Is there a particular reason why we return early here, instead of
> >> breaking out of hte loop and let the generic "failed to get" code
> >> path below to handle this case?
> >
> > Yes, the reason is that I wanted to err on the side of caution. If the
> > remote repository reports a default branch that is not a default branch at
> > all, I do not want to pretend that things are fine and then run into
> > trouble later when we set up a non-branch as remote-tracking target or
> > something like that.
>
> Wouldn't we have the same problem when the remote end does not
> advertise HEAD and we fall back to "local default", though?  We'd
> run into trouble later as we use "local default" that may correspond
> to a non-branch there as remote-tracking target, or something like
> that.

All true, there will be trouble at some stage.

The difference between the two cases, in my mind, is that cloning an empty
repository would run into the latter code path and might still have a
chance to work as intended by using the local default branch name.

In any case, as I indicated earlier, I am _very_ interested in moving as
much functionality as possible from Scalar to Git proper. In this
instance, I hope to move most of the code from `scalar.c` to `clone.c`
(guarded by one or more new options). And as soon as that happens, the
discussion we're having right now will be moot ;-)

Which means that I want to weigh how much effort to put into polishing an
unlikely code path on one side, and on the other side how much effort to
put into moving the functionality away from `contrib/` and deleting that
unlikely code path.

In the same vein, while this patch series contains (mostly) code in
`contrib/` (and therefore technically does not need to adhere strictly to
Git's code style), it is probably wise to pay closer attention to the code
style particularly in those parts that are prone to be moved verbatim (or
close to verbatim) to Git proper.

Thanks,
Dscho

> Not that I care too deeply in the error case, though.  I just felt
> that this early return was an uneven way to follow the principle to
> err on the side of caution, as we continue with the local default
> when the other side fails to tell us what their HEAD points at.
>
> Thanks.
>

Re: [PATCH 10/15] scalar: implement the `run` command

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 3097 bytes --]

Hi Junio,

On Fri, 3 Sep 2021, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
> > Hi Ævar,
> >
> > On Tue, 31 Aug 2021, Ævar Arnfjörð Bjarmason wrote:
> >
> >> On Mon, Aug 30 2021, Derrick Stolee via GitGitGadget wrote:
> >>
> >> > +	const char *usagestr[] = { NULL, NULL };
> >>
> >> Missing usage strings?
> >
> > This command will show a generated usage, i.e. a non-static string. It
> > therefore cannot be specified here already. See the `strbuf_*()` calls
> > populating `buf` and the `usagestr[0] = buf.buf;` assignment.
> >
> >> > +	if (argc == 0)
> >>
> >> Style nit (per style guide): s/argc == 0/!argc/g.
> >
> > It is true that we often do this, but in this instance it would be
> > misleading: `argc` is a counter, not a Boolean.
>
> That argument could be a plausible excuse to deviate from the style
> if it were
>
> 	if (argc == 0)
> 		do no args case;
> 	else if (argc == 1)
> 		do one arg case;
> 	else if (argc == 2)
> 		do two args case;
> 	...
>
> Replacing the first one with "if (!argc)" may make it less readable.
>
> But I do not think the reasoning applies here
>
> 	if (argc == 0)
> 		do a thing that applies only to no args case;
>
> without "else".  This is talking about "do we have any argument? Yes
> or no?" Boolean here.

Well, I offer a differing opinion. But you're right, we are at least
consistent in Git's source code in using `!i` where other projects would
use `i == 0`, and consistency is definitely something I'd like to see more
in Git, not less.

So I changed it as you suggested.

>
> >> > +	if (!strcmp("all", argv[0]))
> >> > +		i = -1;
> >>
> >> Style nit (per style guide): missing braces here.
> >
> > The style guide specifically allows my preference to leave single-line
> > blocks without curlies.
>
> Actually, the exception goes the other way, no?
>
> We generally want to avoid such an unnecessary braces around a
> single statement block.  But when we have an else clause that has a
> block with multiple statements (hence braces are required), as an
> exception, the guide asks you to write braces around the body of the
> if side for consistency.

You're right. I am somehow still using the previous style where we
_required_ single-line blocks _not_ to have curly brackets (see e.g.
aa1c48df817 ([PATCH] ls-tree enhancements, 2005-04-15), the `else` part of
the added `if (! eltbuf)` block).

>
> When you only have just a couple of lines on the "else {}" side, I
> do not think it matters too much either way for readability, though.
> I cannot see the "else" side in the above clause, but IIRC it wasn't
> just a few lines, was it?

It depends what you count as "just a few lines". There are seven lines
enclosed within the curly brackets of the `else` block.

But as much as I enjoy thorough reviews of the Scalar code, I am failing
at getting excited about code style discussions, therefore I simply went
with your suggestion to enclose even the single-line block in curly
brackets.

Thanks,
Dscho

Re: [PATCH v2 07/15] scalar: implement 'scalar list'

[Johannes Schindelin]

Hi Bagas,

On Sat, 4 Sep 2021, Bagas Sanjaya wrote:

> On 04/09/21 00.54, Derrick Stolee via GitGitGadget wrote:
> > +List
> > +~~~~
> > +
> > +list::
> > +	To see which repositories are currently registered by the service, run
> > +	`scalar list`. This subcommand does not need to be run inside a Scalar
> > +	enlistment.
> > +
>
> I think the man-page-style wording should be:
>
> > list::
> >  List enlistments that are currently registered by Scalar. This
> >  subcommand does not need to be run inside an enlistment.

Thank you, I adopted that wording.

Ciao,
Dscho

Re: [PATCH v2 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 2367 bytes --]

Hi Ævar,

On Mon, 6 Sep 2021, Ævar Arnfjörð Bjarmason wrote:

>
> On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:
>
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > This implements Scalar's opinionated `clone` command: it tries to use a
> > partial clone and sets up a sparse checkout by default. In contrast to
> > `git clone`, `scalar clone` sets up the worktree in the `src/`
> > subdirectory, to encourage a separation between the source files and the
> > build output (which helps Git tremendously because it avoids untracked
> > files that have to be specifically ignored when refreshing the index).
>
> Perhaps it's simpler to just say about that /src/ injection:
>
>     `scalar clone` adds an implicit "/src" subdirectory to whatever
>     directory the user provides, with the added stricture on top of
>     doing that with "git clone" that the "src" cannot exist already.

It would not only be simpler, it would also skip an important point I
tried to make, namely how this _differs_ from `git clone`. Rather crucial,
really.

> ...
>
> > +	if (is_directory(enlistment))
> > +		die(_("directory '%s' exists already"), enlistment);
> > +
> > +	dir = xstrfmt("%s/src", enlistment);
>
> Which also seems to suggest a bug here. I.e. if I "git clone <repo>
> /tmp/xyz/abc" and Ctrl+C it we'll remove "abc", but leave "xyz"
> behind. Since we're creating that "xyz" (or "src") implicitly here an
> abort/ctrl+C followed by a retry is going to run into this error, isn't
> it?

Sure, it's just like calling `git clone <url> a/b/c` and upon failure
seeing only the `c` directory removed, while `a/b` is left behind.

> I.e. it seems what's missing in this state machine is checking if the
> directory was there already, and if it isn't add it to the existing
> atexit() removals.
>
> Which may be tricky seeing as this is shelling out to "init" then
> "fetch" etc, i.e. who removes it? But maybe not.

I would rather spend time (after this patch series landed, of course) on
teaching `git clone` to handle what Scalar needs, and upon Ctrl+C to
optionally remove _all_ the directories it created, not just the innermost
one.

Then we get that functionality "for free", without spending a lot of time
on code that will be obsolete soon enough anyway.

Ciao,
Johannes

[PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin via GitGitGadget]

tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.


Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.

The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.


The present
===========

The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).


Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:

 1. git clone --sparse --filter=blob:none /src
 2. git -C /src sparse-checkout init --cone
 3. git -C /src config (many times)
 4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.

For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.


Epilogue
========

Now, to address some questions that I imagine every reader has who made it
this far:

 * Why not put the Scalar functionality directly into a built-in? Creating a
   Git builtin requires scrutiny over every aspect of the feature, which is
   difficult to do while also maintaining the command-line interface
   contract and expected behavior of the Scalar command (there are existing
   users, after all). By having the Scalar command in contrib/, we present a
   simple option for users to have these features in the short term while
   the Git contributor community decides which bits to absorb into Git
   built-ins.
 * Why implement the Scalar command in the Git codebase? We ported Scalar to
   the microsoft/git fork for several reasons. First, we realized it was
   possible now that the core features exist inside Git itself. Second,
   compiling Scalar directly within a version of Git allows us to remove a
   version compatibility check from each config option that might or might
   not apply based on the installed Git version. Finally, this new location
   has greatly simplified our release process and the installation process
   for users. We now have ways to install Scalar with microsoft/git via
   winget, brew, and apt-get. This has been the case since we shipped
   v2.32.0 to our users, read: this setup has served us well already.
 * Why contribute Scalar to the Git project? We are biased, of course, yet
   we do have evidence that the Scalar command is a helpful tool that offers
   an simple way to handle huge repositories with ease. By contributing it
   to the core Git project, we are able to share it with more users,
   especially some users who do not want to install the microsoft/git fork.
   We intend to include Scalar as a component in git-for-windows/git, but
   are contributing it here first. Further, we think there is benefit to the
   Git developer community as this presents an example of how to set certain
   defaults that work for large repositories.
 * Does this integrate with the built-in FSMonitor yet? No, not yet. I do
   have a couple of add-on patch series lined up, one of them being the
   integration with the built-in FSMonitor, which obviously has to wait
   until the FSMonitor patch series advances further.

Changes since v2:

 * Adjusted the description of the list command in the manual page , as
   suggested by Bagas.
 * Addressed two style nits in cmd_run().
 * The documentation of git reconfigure -a was improved.

Changes since v1:

 * A couple typos were fixed
 * The code parsing the output of ls-remote was made more readable
 * The indentation used in scalar.txt now consistently uses tabs
 * We no longer hard-code core.bare = false when registering with Scalar

Derrick Stolee (4):
  scalar: 'register' sets recommended config and starts maintenance
  scalar: 'unregister' stops background maintenance
  scalar: implement 'scalar list'
  scalar: implement the `run` command

Johannes Schindelin (10):
  scalar: create a rudimentary executable
  scalar: start documenting the command
  scalar: create test infrastructure
  scalar: let 'unregister' handle a deleted enlistment directory
    gracefully
  scalar: implement the `clone` subcommand
  scalar: teach 'clone' to support the --single-branch option
  scalar: allow reconfiguring an existing enlistment
  scalar: teach 'reconfigure' to optionally handle all registered
    enlistments
  scalar: implement the `version` command
  scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
  scalar: implement the `delete` command

 Makefile                         |   8 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 844 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 154 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1234 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh


base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v2:

  1:  b8c7d3f8450 =  1:  b8c7d3f8450 scalar: create a rudimentary executable
  2:  4f886575dcf =  2:  4f886575dcf scalar: start documenting the command
  3:  bcfde9bc765 =  3:  bcfde9bc765 scalar: create test infrastructure
  4:  ee3e26a0c4e =  4:  ee3e26a0c4e scalar: 'register' sets recommended config and starts maintenance
  5:  6142f75875b =  5:  6142f75875b scalar: 'unregister' stops background maintenance
  6:  82dd253154f =  6:  82dd253154f scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  fb7c931ddb3 !  7:  d291d3723a6 scalar: implement 'scalar list'
     @@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
      +~~~~
      +
      +list::
     -+	To see which repositories are currently registered by the service, run
     -+	`scalar list`. This subcommand does not need to be run inside a Scalar
     -+	enlistment.
     ++	List enlistments that are currently registered by Scalar. This
     ++	subcommand does not need to be run inside an enlistment.
      +
       Register
       ~~~~~~~~
  8:  f3223c10788 !  8:  40dbf61771e scalar: implement the `clone` subcommand
     @@ contrib/scalar/scalar.txt: an existing Git worktree with Scalar whose name is no
       List
       ~~~~
       
     - list::
     - 	To see which repositories are currently registered by the service, run
     --	`scalar list`. This subcommand does not need to be run inside a Scalar
     --	enlistment.
     -+	`scalar list`. This subcommand, like `clone`, does not need to be run
     -+	inside a Scalar enlistment.
     - 
     - Register
     - ~~~~~~~~
      @@ contrib/scalar/scalar.txt: unregister [<enlistment>]::
       
       SEE ALSO
  9:  b3c4b3dccc6 =  9:  414dbe7d859 scalar: teach 'clone' to support the --single-branch option
 10:  b7fc2dc29c8 ! 10:  76de416a643 scalar: implement the `run` command
     @@ contrib/scalar/scalar.c: static int cmd_register(int argc, const char **argv)
      +	argc = parse_options(argc, argv, NULL, options,
      +			     usagestr, 0);
      +
     -+	if (argc == 0)
     ++	if (!argc)
      +		usage_with_options(usagestr, options);
      +
     -+	if (!strcmp("all", argv[0]))
     ++	if (!strcmp("all", argv[0])) {
      +		i = -1;
     -+	else {
     ++	} else {
      +		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
      +			; /* keep looking for the task */
      +
 11:  9a834c23d08 = 11:  655a902b9df scalar: allow reconfiguring an existing enlistment
 12:  79e9f5d203a ! 12:  2d1987bfcda scalar: teach 'reconfigure' to optionally handle all registered enlistments
     @@ contrib/scalar/scalar.txt: After a Scalar upgrade, or when the configuration of
       reconfigure the enlistment.
       
      +With the `--all` option, all enlistments currently registered with Scalar
     -+will be reconfigured. This option is meant to to be run every time after
     -+Scalar is upgraded.
     ++will be reconfigured. Use this option after each Scalar upgrade.
      +
       SEE ALSO
       --------
 13:  94a21982652 ! 13:  c67938299ee scalar: implement the `delete` command
     @@ contrib/scalar/scalar.txt: scalar register [<enlistment>]
       
       DESCRIPTION
       -----------
     -@@ contrib/scalar/scalar.txt: With the `--all` option, all enlistments currently registered with Scalar
     - will be reconfigured. This option is meant to to be run every time after
     - Scalar is upgraded.
     +@@ contrib/scalar/scalar.txt: reconfigure the enlistment.
     + With the `--all` option, all enlistments currently registered with Scalar
     + will be reconfigured. Use this option after each Scalar upgrade.
       
      +Delete
      +~~~~~~
 14:  707d8e19683 = 14:  d2cd2b7094b scalar: implement the `version` command
 15:  26e23b5c5e5 = 15:  7ccc4f8b9b0 scalar: accept -C and -c options before the subcommand

-- 
gitgitgadget

[PATCH v3 01/15] scalar: create a rudimentary executable

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It was always to plan to contribute all of the proven strategies back to
core Git.

For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.

With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does that which
`git.exe` cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

	make -C contrib/scalar/Makefile

This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.

The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.

An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Makefile                  |  8 ++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 80 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c

diff --git a/Makefile b/Makefile
index c3565fc0f8f..2d5c822f7a8 100644
--- a/Makefile
+++ b/Makefile
@@ -2447,6 +2447,10 @@ endif
 .PHONY: objects
 objects: $(OBJECTS)
 
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
 
@@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
new file mode 100644
index 00000000000..ff3d47e84d0
--- /dev/null
+++ b/contrib/scalar/.gitignore
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
new file mode 100644
index 00000000000..40c03ad10e1
--- /dev/null
+++ b/contrib/scalar/Makefile
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$X
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: all clean FORCE
diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
new file mode 100644
index 00000000000..7cff29e0fcd
--- /dev/null
+++ b/contrib/scalar/scalar.c
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
-- 
gitgitgadget

[PATCH v3 02/15] scalar: start documenting the command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit establishes the infrastructure to build the manual page for
the `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt

diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
index ff3d47e84d0..00441073f59 100644
--- a/contrib/scalar/.gitignore
+++ b/contrib/scalar/.gitignore
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 40c03ad10e1..85c186634e9 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: all clean docs FORCE
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
new file mode 100644
index 00000000000..5f7131861a5
--- /dev/null
+++ b/contrib/scalar/scalar.txt
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
-- 
gitgitgadget

[PATCH v3 03/15] scalar: create test infrastructure

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.

Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 85c186634e9..8620042f281 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$X
+all: scalar$X ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: all clean docs FORCE
+.PHONY: all clean docs test FORCE
diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile
new file mode 100644
index 00000000000..6170672bb37
--- /dev/null
+++ b/contrib/scalar/t/Makefile
@@ -0,0 +1,78 @@
+# Run scalar tests
+#
+# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
+#
+
+-include ../../../config.mak.autogen
+-include ../../../config.mak
+
+SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
+RM ?= rm -f
+PROVE ?= prove
+DEFAULT_TEST_TARGET ?= test
+TEST_LINT ?= test-lint
+
+ifdef TEST_OUTPUT_DIRECTORY
+TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
+else
+TEST_RESULTS_DIRECTORY = ../../../t/test-results
+endif
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
+TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
+
+T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
+
+all: $(DEFAULT_TEST_TARGET)
+
+test: $(TEST_LINT)
+	$(MAKE) aggregate-results-and-cleanup
+
+prove: $(TEST_LINT)
+	@echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
+	$(MAKE) clean-except-prove-cache
+
+$(T):
+	@echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+
+clean-except-prove-cache:
+	$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+	$(RM) -r valgrind/bin
+
+clean: clean-except-prove-cache
+	$(RM) .prove
+
+test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax
+
+test-lint-duplicates:
+	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+		test -z "$$dups" || { \
+		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
+
+test-lint-executable:
+	@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
+		test -z "$$bad" || { \
+		echo >&2 "non-executable tests:" $$bad; exit 1; }
+
+test-lint-shell-syntax:
+	@'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T)
+
+aggregate-results-and-cleanup: $(T)
+	$(MAKE) aggregate-results
+	$(MAKE) clean
+
+aggregate-results:
+	for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
+		echo "$$f"; \
+	done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh
+
+valgrind:
+	$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
+
+test-results:
+	mkdir -p test-results
+
+.PHONY: $(T) aggregate-results clean valgrind
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
new file mode 100755
index 00000000000..16f2b72b126
--- /dev/null
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
-- 
gitgitgadget

[PATCH v3 04/15] scalar: 'register' sets recommended config and starts maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.

The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 255 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 272 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7cff29e0fcd..0e627bb100e 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -5,11 +5,266 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_strip_suffix(&path, "/.git");
+
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		strbuf_addstr(&path, "/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_setlen(&path, len);
+
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+
+		strbuf_setlen(&path, len);
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 5f7131861a5..568987064b2 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v3 05/15] scalar: 'unregister' stops background maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.

The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 0e627bb100e..2b5c52a25f5 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -204,12 +204,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -220,24 +220,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -260,11 +275,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 568987064b2..d9a79984492 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v3 06/15] scalar: let 'unregister' handle a deleted enlistment directory gracefully

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 2b5c52a25f5..d114c038b64 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -275,6 +275,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -288,6 +306,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 16f2b72b126..ef0e8d680d5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
-- 
gitgitgadget

[PATCH v3 07/15] scalar: implement 'scalar list'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index d114c038b64..7f5436399da 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -257,6 +257,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -343,6 +353,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d9a79984492..f93e3d00efd 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
-- 
gitgitgadget

[PATCH v3 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.

Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  31 ++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 261 insertions(+), 3 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7f5436399da..bf18003b297 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -257,6 +258,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -353,6 +553,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f93e3d00efd..d65fb5f1491 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,12 +30,36 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
@@ -64,7 +89,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index ef0e8d680d5..295398f62cc 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
-- 
gitgitgadget

[PATCH v3 09/15] scalar: teach 'clone' to support the --single-branch option

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index bf18003b297..7dd1f28948f 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d65fb5f1491..46999cf7c84 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 295398f62cc..9a35ab4fde6 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
-- 
gitgitgadget

[PATCH v3 10/15] scalar: implement the `run` command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.

The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7dd1f28948f..8a11f390251 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (!argc)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0])) {
+		i = -1;
+	} else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -562,6 +625,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 46999cf7c84..f139a14445d 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -97,6 +98,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v3 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 8a11f390251..1fff7eb7c12 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,28 +142,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -172,7 +175,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -237,7 +241,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -425,7 +429,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -490,6 +494,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -626,6 +648,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f139a14445d..f4e4686e8c8 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -116,6 +117,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 9a35ab4fde6..e6d74a06ca0 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
-- 
gitgitgadget

[PATCH v3 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 1fff7eb7c12..67fa5305225 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f4e4686e8c8..2fa96fcabc6 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -124,6 +124,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index e6d74a06ca0..5fe7fabd0e5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
-- 
gitgitgadget

[PATCH v3 13/15] scalar: implement the `delete` command

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).

On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 67fa5305225..00bedb0bf66 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	res = delete_enlistment(&enlistment);
+	strbuf_release(&enlistment);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 2fa96fcabc6..6fc57707718 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -127,6 +128,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 5fe7fabd0e5..7e8771d0eff 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
-- 
gitgitgadget

[PATCH v3 14/15] scalar: implement the `version` command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 00bedb0bf66..728166aa97a 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -9,6 +9,7 @@
 #include "run-command.h"
 #include "refs.h"
 #include "dir.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -362,6 +363,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -748,6 +758,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -759,6 +797,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
-- 
gitgitgadget

[PATCH v3 15/15] scalar: accept -C and -c options before the subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git` executable has these two very useful options:

-C <directory>:
	switch to the specified directory before performing any actions

-c <key>=<value>:
	temporarily configure this setting for the duration of the
	specified scalar subcommand

With this commit, we teach the `scalar` executable the same trick.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 728166aa97a..76a77ca1ed0 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -816,7 +835,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 6fc57707718..3a80f829edc 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
-- 
gitgitgadget

Re: [PATCH v3 07/15] scalar: implement 'scalar list'

[Bagas Sanjaya]

On 09/09/21 02.24, Derrick Stolee via GitGitGadget wrote:
> +List
> +~~~~
> +
> +list::
> +	List enlistments that are currently registered by Scalar. This
> +	subcommand does not need to be run inside an enlistment.
> +

Looks OK.

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Wed, Sep 08 2021, Johannes Schindelin via GitGitGadget wrote:

> Changes since v2:
>
>  * Adjusted the description of the list command in the manual page , as
>    suggested by Bagas.
>  * Addressed two style nits in cmd_run().
>  * The documentation of git reconfigure -a was improved.
>
> Changes since v1:
>
>  * A couple typos were fixed
>  * The code parsing the output of ls-remote was made more readable
>  * The indentation used in scalar.txt now consistently uses tabs
>  * We no longer hard-code core.bare = false when registering with Scalar

In the summary I had on v1->v2 points 1-3 are for v2->v3, respectively,
outstanding, addressed, outstanding:

    https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/

In addition the discussion ending here:
https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/

For that point: I think it's fair enough not to properly handle the
cleanup case in "scalar clone", but perhaps add a note in the commit
message that unlike "git clone" this is known not to clean after itself
properly on ctrl+c?

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Ævar Arnfjörð Bjarmason]

On Wed, Sep 08 2021, Johannes Schindelin wrote:

> [...]
> Which means that I want to weigh how much effort to put into polishing an
> unlikely code path on one side, and on the other side how much effort to
> put into moving the functionality away from `contrib/` and deleting that
> unlikely code path.
>
> In the same vein, while this patch series contains (mostly) code in
> `contrib/` (and therefore technically does not need to adhere strictly to
> Git's code style), it is probably wise to pay closer attention to the code
> style particularly in those parts that are prone to be moved verbatim (or
> close to verbatim) to Git proper.

I don't think we have any such exception to our usual style & preferred
code patterns in contrib/* or compat/* in general. In the latter case we
have e.g. compat/regex/ and other externally-imported codebases, which
we've tried to stylistically modify as little as possible to make
subsequent imports easier, ditto sha1dc/ etc.

I think the general (but unwritten) rule has been to draw the
distinction on whether or not code is still externally maintained and
expected to be imported, or if it's expected to be maintained in git.git
going forward.

I think that this proposed series falls thoroughly in the latter
category, but maybe I've misunderstood it..

Also re my [1] I had some (still relevant, but unaddressed) points on v1
about how placing this in contrib/* made certain aspects of integrating
it into our build system harder. I was imagining that distinction as
purely an internal implementation detail to git.git (make install
etc. would behave the same), but per the above it seems to come with
deeper connotations than that at least in your mind.

1. https://lore.kernel.org/git/87r1dydp4m.fsf@evledraar.gmail.com

Re: [PATCH v3 01/15] scalar: create a rudimentary executable

[Elijah Newren]

On Wed, Sep 8, 2021 at 12:24 PM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> The idea of Scalar (https://github.com/microsoft/scalar), and before
> that, of VFS for Git, has always been to prove that Git _can_ scale, and
> to upstream whatever strategies have been demonstrated to help.
>
> With this patch, we start the journey from that C# project to move what
> is left to Git's own `contrib/` directory, reimplementing it in pure C,
> with the intention to facilitate integrating the functionality into core
> Git all while maintaining backwards-compatibility for existing Scalar
> users (which will be much easier when both live in the same worktree).
> It was always to plan to contribute all of the proven strategies back to
> core Git.

s/always to plan/always the plan/

> For example, while the virtual filesystem provided by VFS for Git helped
> the team developing the Windows operating system to move onto Git, while
> trying to upstream it we realized that it cannot be done: getting the
> virtual filesystem to work (which we only managed to implement fully on
> Windows, but not on, say, macOS or Linux), and the required server-side
> support for the GVFS protocol, made this not quite feasible.
>
> The Scalar project learned from that and tackled the problem with
> different tactics: instead of pretending to Git that the working
> directory is fully populated, it _specifically_ teaches Git about
> partial clone (which is based on VFS for Git's cache server), about
> sparse checkout (which VFS for Git tried to do transparently, in the
> file system layer), and regularly runs maintenance tasks to keep the
> repository in a healthy state.
>
> With partial clone, sparse checkout and `git maintenance` having been
> upstreamed, there is little left that `scalar.exe` does that which
> `git.exe` cannot do. One such thing is that `scalar clone <url>` will
> automatically set up a partial, sparse clone, and configure
> known-helpful settings from the start.

s/does that which/does which/

> So let's bring this convenience into Git's tree.
>
> The idea here is that you can (optionally) build Scalar via
>
>         make -C contrib/scalar/Makefile
>
> This will build the `scalar` executable and put it into the
> contrib/scalar/ subdirectory.
>
> The slightly awkward addition of the `contrib/scalar/*` bits to the
> top-level `Makefile` are actually really required: we want to link to
> `libgit.a`, which means that we will need to use the very same `CFLAGS`
> and `LDFLAGS` as the rest of Git.
>
> An early development version of this patch tried to replicate all the
> conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
> `contrib/svn-fe/Makefile` used to do before it was retired. It turned
> out to be quite the whack-a-mole game: the SHA-1-related flags, the
> flags enabling/disabling `compat/poll/`, `compat/regex/`,
> `compat/win32mmap.c` & friends depending on the current platform... To
> put it mildly: it was a major mess.
>
> Instead, this patch makes minimal changes to the top-level `Makefile` so
> that the bits in `contrib/scalar/` can be compiled and linked, and
> adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
> most minimal way to do the actual compiling.
>
> Note: With this commit, we only establish the infrastructure, no
> Scalar functionality is implemented yet; We will do that incrementally
> over the next few commits.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  Makefile                  |  8 ++++++++
>  contrib/scalar/.gitignore |  2 ++
>  contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
>  contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
>  4 files changed, 80 insertions(+)
>  create mode 100644 contrib/scalar/.gitignore
>  create mode 100644 contrib/scalar/Makefile
>  create mode 100644 contrib/scalar/scalar.c
>
> diff --git a/Makefile b/Makefile
> index c3565fc0f8f..2d5c822f7a8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2447,6 +2447,10 @@ endif
>  .PHONY: objects
>  objects: $(OBJECTS)
>
> +SCALAR_SOURCES := contrib/scalar/scalar.c
> +SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
> +OBJECTS += $(SCALAR_OBJECTS)
> +
>  dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
>  dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
>
> @@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
>         $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
>                 $(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
>
> +contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
> +       $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
> +               $(filter %.o,$^) $(LIBS)
> +
>  $(LIB_FILE): $(LIB_OBJS)
>         $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
>
> diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
> new file mode 100644
> index 00000000000..ff3d47e84d0
> --- /dev/null
> +++ b/contrib/scalar/.gitignore
> @@ -0,0 +1,2 @@
> +/*.exe
> +/scalar
> diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
> new file mode 100644
> index 00000000000..40c03ad10e1
> --- /dev/null
> +++ b/contrib/scalar/Makefile
> @@ -0,0 +1,34 @@
> +QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
> +QUIET_SUBDIR1  =
> +
> +ifneq ($(findstring s,$(MAKEFLAGS)),s)
> +ifndef V
> +       QUIET_SUBDIR0  = +@subdir=
> +       QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
> +                        $(MAKE) $(PRINT_DIR) -C $$subdir
> +else
> +       export V
> +endif
> +endif
> +
> +all:
> +
> +include ../../config.mak.uname
> +-include ../../config.mak.autogen
> +-include ../../config.mak
> +
> +TARGETS = scalar$(X) scalar.o
> +GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
> +
> +all: scalar$X
> +
> +$(GITLIBS):
> +       $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
> +
> +$(TARGETS): $(GITLIBS) scalar.c
> +       $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
> +
> +clean:
> +       $(RM) $(TARGETS)
> +
> +.PHONY: all clean FORCE
> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
> new file mode 100644
> index 00000000000..7cff29e0fcd
> --- /dev/null
> +++ b/contrib/scalar/scalar.c
> @@ -0,0 +1,36 @@
> +/*
> + * The Scalar command-line interface.
> + */
> +
> +#include "cache.h"
> +#include "gettext.h"
> +#include "parse-options.h"
> +
> +static struct {
> +       const char *name;
> +       int (*fn)(int, const char **);
> +} builtins[] = {
> +       { NULL, NULL},
> +};
> +
> +int cmd_main(int argc, const char **argv)
> +{
> +       struct strbuf scalar_usage = STRBUF_INIT;
> +       int i;
> +
> +       if (argc > 1) {
> +               argv++;
> +               argc--;
> +
> +               for (i = 0; builtins[i].name; i++)
> +                       if (!strcmp(builtins[i].name, argv[0]))
> +                               return !!builtins[i].fn(argc, argv);
> +       }
> +
> +       strbuf_addstr(&scalar_usage,
> +                     N_("scalar <command> [<options>]\n\nCommands:\n"));
> +       for (i = 0; builtins[i].name; i++)
> +               strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
> +
> +       usage(scalar_usage.buf);
> +}
> --
> gitgitgadget
>

Re: [PATCH v3 01/15] scalar: create a rudimentary executable

[Johannes Schindelin]

Hi Elijah,

On Thu, 9 Sep 2021, Elijah Newren wrote:

> On Wed, Sep 8, 2021 at 12:24 PM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> >
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > The idea of Scalar (https://github.com/microsoft/scalar), and before
> > that, of VFS for Git, has always been to prove that Git _can_ scale, and
> > to upstream whatever strategies have been demonstrated to help.
> >
> > With this patch, we start the journey from that C# project to move what
> > is left to Git's own `contrib/` directory, reimplementing it in pure C,
> > with the intention to facilitate integrating the functionality into core
> > Git all while maintaining backwards-compatibility for existing Scalar
> > users (which will be much easier when both live in the same worktree).
> > It was always to plan to contribute all of the proven strategies back to
> > core Git.
>
> s/always to plan/always the plan/
>
> > For example, while the virtual filesystem provided by VFS for Git helped
> > the team developing the Windows operating system to move onto Git, while
> > trying to upstream it we realized that it cannot be done: getting the
> > virtual filesystem to work (which we only managed to implement fully on
> > Windows, but not on, say, macOS or Linux), and the required server-side
> > support for the GVFS protocol, made this not quite feasible.
> >
> > The Scalar project learned from that and tackled the problem with
> > different tactics: instead of pretending to Git that the working
> > directory is fully populated, it _specifically_ teaches Git about
> > partial clone (which is based on VFS for Git's cache server), about
> > sparse checkout (which VFS for Git tried to do transparently, in the
> > file system layer), and regularly runs maintenance tasks to keep the
> > repository in a healthy state.
> >
> > With partial clone, sparse checkout and `git maintenance` having been
> > upstreamed, there is little left that `scalar.exe` does that which
> > `git.exe` cannot do. One such thing is that `scalar clone <url>` will
> > automatically set up a partial, sparse clone, and configure
> > known-helpful settings from the start.
>
> s/does that which/does which/

Thank you!

I am holding off from sending a new iteration (with the suggested fixes)
until tomorrow, waiting for more suggestions to trickle in.

Thanks,
Dscho

>
> > So let's bring this convenience into Git's tree.
> >
> > The idea here is that you can (optionally) build Scalar via
> >
> >         make -C contrib/scalar/Makefile
> >
> > This will build the `scalar` executable and put it into the
> > contrib/scalar/ subdirectory.
> >
> > The slightly awkward addition of the `contrib/scalar/*` bits to the
> > top-level `Makefile` are actually really required: we want to link to
> > `libgit.a`, which means that we will need to use the very same `CFLAGS`
> > and `LDFLAGS` as the rest of Git.
> >
> > An early development version of this patch tried to replicate all the
> > conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
> > `contrib/svn-fe/Makefile` used to do before it was retired. It turned
> > out to be quite the whack-a-mole game: the SHA-1-related flags, the
> > flags enabling/disabling `compat/poll/`, `compat/regex/`,
> > `compat/win32mmap.c` & friends depending on the current platform... To
> > put it mildly: it was a major mess.
> >
> > Instead, this patch makes minimal changes to the top-level `Makefile` so
> > that the bits in `contrib/scalar/` can be compiled and linked, and
> > adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
> > most minimal way to do the actual compiling.
> >
> > Note: With this commit, we only establish the infrastructure, no
> > Scalar functionality is implemented yet; We will do that incrementally
> > over the next few commits.
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> > ---
> >  Makefile                  |  8 ++++++++
> >  contrib/scalar/.gitignore |  2 ++
> >  contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
> >  contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
> >  4 files changed, 80 insertions(+)
> >  create mode 100644 contrib/scalar/.gitignore
> >  create mode 100644 contrib/scalar/Makefile
> >  create mode 100644 contrib/scalar/scalar.c
> >
> > diff --git a/Makefile b/Makefile
> > index c3565fc0f8f..2d5c822f7a8 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -2447,6 +2447,10 @@ endif
> >  .PHONY: objects
> >  objects: $(OBJECTS)
> >
> > +SCALAR_SOURCES := contrib/scalar/scalar.c
> > +SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
> > +OBJECTS += $(SCALAR_OBJECTS)
> > +
> >  dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
> >  dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
> >
> > @@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
> >         $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
> >                 $(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
> >
> > +contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
> > +       $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
> > +               $(filter %.o,$^) $(LIBS)
> > +
> >  $(LIB_FILE): $(LIB_OBJS)
> >         $(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
> >
> > diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
> > new file mode 100644
> > index 00000000000..ff3d47e84d0
> > --- /dev/null
> > +++ b/contrib/scalar/.gitignore
> > @@ -0,0 +1,2 @@
> > +/*.exe
> > +/scalar
> > diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
> > new file mode 100644
> > index 00000000000..40c03ad10e1
> > --- /dev/null
> > +++ b/contrib/scalar/Makefile
> > @@ -0,0 +1,34 @@
> > +QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
> > +QUIET_SUBDIR1  =
> > +
> > +ifneq ($(findstring s,$(MAKEFLAGS)),s)
> > +ifndef V
> > +       QUIET_SUBDIR0  = +@subdir=
> > +       QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
> > +                        $(MAKE) $(PRINT_DIR) -C $$subdir
> > +else
> > +       export V
> > +endif
> > +endif
> > +
> > +all:
> > +
> > +include ../../config.mak.uname
> > +-include ../../config.mak.autogen
> > +-include ../../config.mak
> > +
> > +TARGETS = scalar$(X) scalar.o
> > +GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
> > +
> > +all: scalar$X
> > +
> > +$(GITLIBS):
> > +       $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
> > +
> > +$(TARGETS): $(GITLIBS) scalar.c
> > +       $(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
> > +
> > +clean:
> > +       $(RM) $(TARGETS)
> > +
> > +.PHONY: all clean FORCE
> > diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
> > new file mode 100644
> > index 00000000000..7cff29e0fcd
> > --- /dev/null
> > +++ b/contrib/scalar/scalar.c
> > @@ -0,0 +1,36 @@
> > +/*
> > + * The Scalar command-line interface.
> > + */
> > +
> > +#include "cache.h"
> > +#include "gettext.h"
> > +#include "parse-options.h"
> > +
> > +static struct {
> > +       const char *name;
> > +       int (*fn)(int, const char **);
> > +} builtins[] = {
> > +       { NULL, NULL},
> > +};
> > +
> > +int cmd_main(int argc, const char **argv)
> > +{
> > +       struct strbuf scalar_usage = STRBUF_INIT;
> > +       int i;
> > +
> > +       if (argc > 1) {
> > +               argv++;
> > +               argc--;
> > +
> > +               for (i = 0; builtins[i].name; i++)
> > +                       if (!strcmp(builtins[i].name, argv[0]))
> > +                               return !!builtins[i].fn(argc, argv);
> > +       }
> > +
> > +       strbuf_addstr(&scalar_usage,
> > +                     N_("scalar <command> [<options>]\n\nCommands:\n"));
> > +       for (i = 0; builtins[i].name; i++)
> > +               strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
> > +
> > +       usage(scalar_usage.buf);
> > +}
> > --
> > gitgitgadget
> >
>

Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:

> On Wed, Sep 08 2021, Johannes Schindelin via GitGitGadget wrote:
>
>> Changes since v2:
>>
>>  * Adjusted the description of the list command in the manual page , as
>>    suggested by Bagas.
>>  * Addressed two style nits in cmd_run().
>>  * The documentation of git reconfigure -a was improved.
>>
>> Changes since v1:
>>
>>  * A couple typos were fixed
>>  * The code parsing the output of ls-remote was made more readable
>>  * The indentation used in scalar.txt now consistently uses tabs
>>  * We no longer hard-code core.bare = false when registering with Scalar
>
> In the summary I had on v1->v2 points 1-3 are for v2->v3, respectively,
> outstanding, addressed, outstanding:
>
>     https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
>
> In addition the discussion ending here:
> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/
>
> For that point: I think it's fair enough not to properly handle the
> cleanup case in "scalar clone", but perhaps add a note in the commit
> message that unlike "git clone" this is known not to clean after itself
> properly on ctrl+c?

Seeing [1] about the planned re-roll I have the above a shot a few days
ago, see the original discussion at [2] (indirectly linked above).

The dependency graph isn't quite there yet, but I basically had it
working (some twiddling around adding a new top-level command name
needed). Result:
    
     .gitignore             |  1 +
     Documentation/Makefile |  4 ++++
     Makefile               | 27 +++++++++++++++++++++++++++
     3 files changed, 32 insertions(+)

And by comparison, your v3:
    
     Makefile                  |  8 +++++
     contrib/scalar/.gitignore |  5 +++
     contrib/scalar/Makefile   | 57 ++++++++++++++++++++++++++++++++++
     contrib/scalar/t/Makefile | 78 +++++++++++++++++++++++++++++++++++++++++++++++
     4 files changed, 148 insertions(+)

So that's a very pleasing reduction in complexity.

The WIP change for that is below, some oddities like the
s/scalarscalar/scalar/ (couldn't find the bit that generated that
built-in blurb at the time).

But for one the automatic lint integration in Documentation/ found one
nit, and "make test" etc. all work with this automatically.

All guarded behind a CONTRIB_SCALAR flag, so the end result isn't in any
way different.

Again, as pointed out in [2] the proposal isn't in any way to change
what an end user sees, just to make our build system less
complex. Stretching dependency graphs across Makefiles is a pain.

So just as an example I was improving the "sparse" and "hdr-check"
targets today, with this dropped into the main Makefile under a flag
stuff like that will Just Work, if it's under its own Makefile
infrastructure it'll need to be treated specially for every such change,
ditto "make TAGS" etc.

1. http://lore.kernel.org/git/nycvar.QRO.7.76.6.2109131531210.55@tvgsbejvaqbjf.bet;
2. https://lore.kernel.org/git/87mtoxwt63.fsf@evledraar.gmail.com/

diff --git a/.gitignore b/.gitignore
index 311841f9bed..491cb2177af 100644
--- a/.gitignore
+++ b/.gitignore
@@ -216,6 +216,7 @@
 /configure
 /.vscode/
 /tags
+/scalar
 /TAGS
 /cscope*
 /compile_commands.json
diff --git a/Documentation/Makefile b/Documentation/Makefile
index f5605b7767f..f0a03faf40f 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -19,6 +19,10 @@ MAN1_TXT += git.txt
 MAN1_TXT += gitk.txt
 MAN1_TXT += gitweb.txt
 
+ifdef CONTRIB_SCALAR
+MAN1_TXT += scalar.txt
+endif
+
 # man5 / man7 guides (note: new guides should also be added to command-list.txt)
 MAN5_TXT += gitattributes.txt
 MAN5_TXT += githooks.txt
diff --git a/Makefile b/Makefile
index 429c276058d..7407df45b2a 100644
--- a/Makefile
+++ b/Makefile
@@ -584,6 +584,7 @@ FUZZ_OBJS =
 FUZZ_PROGRAMS =
 GIT_OBJS =
 LIB_OBJS =
+NONGIT_PROGRAM_OBJS =
 OBJECTS =
 PROGRAM_OBJS =
 PROGRAMS =
@@ -691,10 +692,17 @@ PROGRAM_OBJS += shell.o
 .PHONY: program-objs
 program-objs: $(PROGRAM_OBJS)
 
+ifdef CONTRIB_SCALAR
+NONGIT_PROGRAM_OBJS += scalar.o
+endif
+.PHONY: nongit-program-objs
+nongit-program-objs: $(NONGIT_PROGRAM_OBJS)
+
 # Binary suffix, set to .exe for Windows builds
 X =
 
 PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS))
+PROGRAMS += $(patsubst %.o,%$X,$(NONGIT_PROGRAM_OBJS))
 
 TEST_BUILTINS_OBJS += test-advise.o
 TEST_BUILTINS_OBJS += test-bitmap.o
@@ -800,6 +808,9 @@ BINDIR_PROGRAMS_NEED_X += git-receive-pack
 BINDIR_PROGRAMS_NEED_X += git-shell
 BINDIR_PROGRAMS_NEED_X += git-upload-archive
 BINDIR_PROGRAMS_NEED_X += git-upload-pack
+ifdef CONTRIB_SCALAR
+BINDIR_PROGRAMS_NEED_X += scalar
+endif
 
 BINDIR_PROGRAMS_NO_X += git-cvsserver
 
@@ -1247,6 +1258,10 @@ else
 ALL_COMMANDS_TO_INSTALL += git-receive-pack$(X)
 ALL_COMMANDS_TO_INSTALL += git-upload-archive$(X)
 ALL_COMMANDS_TO_INSTALL += git-upload-pack$(X)
+
+ifdef CONTRIB_SCALAR
+ALL_COMMANDS_TO_INSTALL += scalar$(X)
+endif
 endif
 
 ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
@@ -2459,6 +2474,7 @@ git-objs: $(GIT_OBJS)
 
 OBJECTS += $(GIT_OBJS)
 OBJECTS += $(PROGRAM_OBJS)
+OBJECTS += $(NONGIT_PROGRAM_OBJS)
 OBJECTS += $(TEST_OBJS)
 OBJECTS += $(XDIFF_OBJS)
 OBJECTS += $(FUZZ_OBJS)
@@ -2582,6 +2598,9 @@ compat/nedmalloc/nedmalloc.sp compat/nedmalloc/nedmalloc.o: EXTRA_CPPFLAGS = \
 compat/nedmalloc/nedmalloc.sp: SP_EXTRA_FLAGS += -Wno-non-pointer-null
 endif
 
+ifdef CONTRIB_SCALAR
+# TODO: Implicit rule for git-scalar here
+endif
 git-%$X: %.o GIT-LDFLAGS $(GITLIBS)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
 
@@ -2596,6 +2615,11 @@ git-http-push$X: http.o http-push.o GIT-LDFLAGS $(GITLIBS)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+ifdef CONTRIB_SCALAR
+scalar: scalar.o $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
+endif
+
 $(REMOTE_CURL_ALIASES): $(REMOTE_CURL_PRIMARY)
 	$(QUIET_LNCP)$(RM) $@ && \
 	ln $< $@ 2>/dev/null || \
@@ -2800,6 +2824,7 @@ GIT-BUILD-OPTIONS: FORCE
 	@echo SHELL_PATH=\''$(subst ','\'',$(SHELL_PATH_SQ))'\' >$@+
 	@echo TEST_SHELL_PATH=\''$(subst ','\'',$(TEST_SHELL_PATH_SQ))'\' >>$@+
 	@echo PERL_PATH=\''$(subst ','\'',$(PERL_PATH_SQ))'\' >>$@+
+	@echo CONTRIB_SCALAR=\''$(subst ','\'',$(subst ','\'',$(CONTRIB_SCALAR)))'\' >>$@+
 	@echo DIFF=\''$(subst ','\'',$(subst ','\'',$(DIFF)))'\' >>$@+
 	@echo PYTHON_PATH=\''$(subst ','\'',$(PYTHON_PATH_SQ))'\' >>$@+
 	@echo TAR=\''$(subst ','\'',$(subst ','\'',$(TAR)))'\' >>$@+
@@ -2881,6 +2906,8 @@ bin-wrappers/%: wrap-for-bin.sh
 	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
 	     -e 's|@@BUILD_DIR@@|$(shell pwd)|' \
 	     -e 's|@@PROG@@|$(patsubst test-%,t/helper/test-%$(X),$(@F))$(patsubst git%,$(X),$(filter $(@F),$(BINDIR_PROGRAMS_NEED_X)))|' < $< > $@ && \
+	sed -e 's|scalarscalar|scalar|' <$@ >$@+ && \
+	mv $@+ $@ && \
 	chmod +x $@

Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 2690 bytes --]

Hi Ævar,

On Mon, 13 Sep 2021, Ævar Arnfjörð Bjarmason wrote:

>
> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:
>
> > On Wed, Sep 08 2021, Johannes Schindelin via GitGitGadget wrote:
> >
> >> Changes since v2:
> >>
> >>  * Adjusted the description of the list command in the manual page , as
> >>    suggested by Bagas.
> >>  * Addressed two style nits in cmd_run().
> >>  * The documentation of git reconfigure -a was improved.
> >>
> >> Changes since v1:
> >>
> >>  * A couple typos were fixed
> >>  * The code parsing the output of ls-remote was made more readable
> >>  * The indentation used in scalar.txt now consistently uses tabs
> >>  * We no longer hard-code core.bare = false when registering with Scalar
> >
> > In the summary I had on v1->v2 points 1-3 are for v2->v3, respectively,
> > outstanding, addressed, outstanding:
> >
> >     https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
> >
> > In addition the discussion ending here:
> > https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/
> >
> > For that point: I think it's fair enough not to properly handle the
> > cleanup case in "scalar clone", but perhaps add a note in the commit
> > message that unlike "git clone" this is known not to clean after itself
> > properly on ctrl+c?
>
> Seeing [1] about the planned re-roll I have the above a shot a few days
> ago, see the original discussion at [2] (indirectly linked above).

There is a good reason why I did not engage in that tangent about
deviating from the established `contrib/*/Makefile` paradigm: I find it
particularly unrelated to what this here patch series is trying to
accomplish, and I cannot bring myself to be interested in the proposed
build system changes, either, because I do not see any benefit in the
changes, only downsides.

I find the distraction unnecessary.

Besides, the way I designed it, the code in `contrib/scalar/` intrudes as
little as possible on the core Git build system. The impact on the
top-level `Makefile` is quite minimal, which is just the way I firmly
believe it should be.

In short: I do not want those intrusive changes to the top-level
`Makefile`, not in this patch series, and not as a follow-up, either.

We have much bigger fries to fry: namely, how to migrate the improvements
for large-scale operations from Scalar to core Git, so that all Git users
can benefit. Granted, it will take a lot effort, and it would be easier to
move around `Makefile` rules instead. But ultimately, the benefit of
allowing users to handle larger repositories with ease will be worth that
effort.

Ciao,
Johannes

Re: [PATCH v2 01/15] scalar: create a rudimentary executable

[Ævar Arnfjörð Bjarmason]

On Fri, Sep 03 2021, Johannes Schindelin via GitGitGadget wrote:

> diff --git a/Makefile b/Makefile
> index c3565fc0f8f..2d5c822f7a8 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2447,6 +2447,10 @@ endif
>  .PHONY: objects
>  objects: $(OBJECTS)
>  
> +SCALAR_SOURCES := contrib/scalar/scalar.c
> +SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
> +OBJECTS += $(SCALAR_OBJECTS)
> +
>  dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
>  dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
>  

Aside from anything else we may disagree with about the build system
integration, this breaks the "objects" target. You're adding things to
it, but it won't be reflected by those changes.

I have this fixup for it on top locally, the reference to 852ec00310 is
to a commit in your repo:

-- >8 --
Makefile: fix scalar "make objects" regression

In 852ec00310 (scalar: create a rudimentary executable, 2021-04-10)
the contrib/scalar/scalar.o file was added to OBJECTS, but was added
below the "objects" target.

That target was added in 029bac01a8 (Makefile: add
{program,xdiff,test,git,fuzz}-objs & objects targets, 2021-02-23)
along with others (git-objs, test-objs) to serve as ad-hoc targets for
e.g. compiling during interactive the targets need to come after we
fully declare the variable.

Before this change we'd still do /some/ scalar things on "make
objects", namely create the contrib/scalar/.depend/ directory (see
dep_dirs in the context), we just wouldn't create the object itself.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 6e14f626e3..6ace4be141 100644
--- a/Makefile
+++ b/Makefile
@@ -2464,12 +2464,11 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
-.PHONY: objects
-objects: $(OBJECTS)
-
 SCALAR_SOURCES := contrib/scalar/scalar.c
 SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
 OBJECTS += $(SCALAR_OBJECTS)
+.PHONY: objects
+objects: $(OBJECTS)
 
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
-- 
2.33.0.1013.ge8323766266

Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Mon, Sep 13 2021, Johannes Schindelin wrote:

> Hi Ævar,
>
> On Mon, 13 Sep 2021, Ævar Arnfjörð Bjarmason wrote:
>
>>
>> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:
>>
>> > On Wed, Sep 08 2021, Johannes Schindelin via GitGitGadget wrote:
>> >
>> >> Changes since v2:
>> >>
>> >>  * Adjusted the description of the list command in the manual page , as
>> >>    suggested by Bagas.
>> >>  * Addressed two style nits in cmd_run().
>> >>  * The documentation of git reconfigure -a was improved.
>> >>
>> >> Changes since v1:
>> >>
>> >>  * A couple typos were fixed
>> >>  * The code parsing the output of ls-remote was made more readable
>> >>  * The indentation used in scalar.txt now consistently uses tabs
>> >>  * We no longer hard-code core.bare = false when registering with Scalar
>> >
>> > In the summary I had on v1->v2 points 1-3 are for v2->v3, respectively,
>> > outstanding, addressed, outstanding:
>> >
>> >     https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
>> >
>> > In addition the discussion ending here:
>> > https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/
>> >
>> > For that point: I think it's fair enough not to properly handle the
>> > cleanup case in "scalar clone", but perhaps add a note in the commit
>> > message that unlike "git clone" this is known not to clean after itself
>> > properly on ctrl+c?
>>
>> Seeing [1] about the planned re-roll I have the above a shot a few days
>> ago, see the original discussion at [2] (indirectly linked above).
>
> There is a good reason why I did not engage in that tangent about
> deviating from the established `contrib/*/Makefile` paradigm: I find it
> particularly unrelated to what this here patch series is trying to
> accomplish, and I cannot bring myself to be interested in the proposed
> build system changes, either, because I do not see any benefit in the
> changes, only downsides.
>
> I find the distraction unnecessary.

Perhaps I'm reading too much between the lines here, so forgive any
undue knee-jerk reaction.

But aside from any technical disagreement we may have I find this way of
handling reviews quite disrespectful of other people's time.

Sure, maybe I'm wrong, and maybe you either don't see any value in the
proposed changes or maybe they're just bad suggestions.

I still took the time to review and comment on a series you're
submitting. I think the least you can do is to include some comment in a
re-roll like:

    Skimmed Ævar's proposal about an alternate build system
    implementation, sorry, I just don't think it's worth it, going to
    not change anything there.

Which would be fair enough, and would leave the ball in my court in
terms of either dropping it, submitting any patch on top etc.

As opposed to just ignoring that whole thread, leaving both me wondering
if it's even been seen (and sending a few reminders like the linked
upthread), as well as others trying to track the state of the series.

> Besides, the way I designed it, the code in `contrib/scalar/` intrudes as
> little as possible on the core Git build system. The impact on the
> top-level `Makefile` is quite minimal, which is just the way I firmly
> believe it should be.
>
> In short: I do not want those intrusive changes to the top-level
> `Makefile`, not in this patch series, and not as a follow-up, either.

Communication grievances aside:

 * Saying that we have an "established `contrib/*/Makefile` paradigm"
   doesn't really follow in this case. Most of that isn't C code, and
   the bits that are C code are not using libgit.a.

   And as I've argued elsewhere I think that whole pattern was a mistake
   in the first place, it makes inter-Makefile dependencies a pain to
   manage, has resulted in bitrot of things like git-subtree.sh and
   mw-to-git, all because we conflate whether we want to build/test
   things with what we'd like in a default installation.

 * Because of that any number of targets / workflows in the Makefile
   aren't going to work by default, e.g. try checking it out and doing
   "make TAGS".

   That specific one happens to because we exclude contrib explicitly,
   that could be fixed, but there's going to be any number of things
   like that, but current and future ones.

 * One target that seems missing (maybe I've somehow missed it) is any
   support for installing the build command, its docs etc.

 * I hacked a bit more on this today and came up with a not-quite-ready
   change that both for the Makefile and Documentation/Makefile will
   build scalar.c like any other top-level command, and we'll always get
   a bin-wrapper/scalar, we'll only do something different at "install"
   time.

   This means that just like the "all:: $(FUZZ_OBJS)" we'll always build
   a scalar.o, so that'll prevent others from e.g. breaking a library
   function you rely on (which'll only be annoying caught in CI, or
   integration or whatever).

 * I think it's fair to say that whatever one thinks of my argument,
   your [1] leaves the reader hanging about *why* you went for this
   "make -C contrib/scalar/Makefile".

   As summarized there you tried to have it completely separate, but
   failed.

   So now there's a bit of integration in the top-level Makefile
   already. But why try in the first place? Just slavish conformance to
   existing "contrib/" convention?

   The dependency back on assets in subdirs there is deep, so surely
   it's not to build this independently somehow.

> We have much bigger fries to fry: namely, how to migrate the improvements
> for large-scale operations from Scalar to core Git, so that all Git users
> can benefit. Granted, it will take a lot effort, and it would be easier to
> move around `Makefile` rules instead. But ultimately, the benefit of
> allowing users to handle larger repositories with ease will be worth that
> effort.

It's your implementation that requires mostly moving around / copying
Makefile rules, if you piggy-back on the existing Makefile rules you'll
get most of the behavior you've duplicated for free without any moving
or copying.

But in any case, re the last bullet point above and your "[...]and not
as a follow-up, either" comments it's not clear whether you're saying
that you don't have time to work on this, or that you wouldn't want it
at all in any shape or form, even if someone else did it.

Which is not to say that I'm promising to do so even if that's the case,
I do think the onus is on the person proposing the change, and to take
productive feedback about things that are introducing unnecessary
complexity, and won't saddle others with undue technical debt going
forward.

1. https://lore.kernel.org/git/b8c7d3f84508ae0fb300f47c726764f4cbf46be9.1631129086.git.gitgitgadget@gmail.com/

Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin]

[-- Attachment #1: Type: text/plain, Size: 4312 bytes --]

Hi Ævar,

On Tue, 14 Sep 2021, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Sep 13 2021, Johannes Schindelin wrote:
>
> > On Mon, 13 Sep 2021, Ævar Arnfjörð Bjarmason wrote:
> >
> >> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:
> >>
> >> > In the summary I had on v1->v2 points 1-3 are for v2->v3,
> >> > respectively, outstanding, addressed, outstanding:
> >> >
> >> >     https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
> >> >
> >> > In addition the discussion ending here:
> >> > https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/
> >> >
> >> > For that point: I think it's fair enough not to properly handle the
> >> > cleanup case in "scalar clone", but perhaps add a note in the
> >> > commit message that unlike "git clone" this is known not to clean
> >> > after itself properly on ctrl+c?
> >>
> >> Seeing [1] about the planned re-roll I have the above a shot a few
> >> days ago, see the original discussion at [2] (indirectly linked
> >> above).
> >
> > There is a good reason why I did not engage in that tangent about
> > deviating from the established `contrib/*/Makefile` paradigm: I find
> > it particularly unrelated to what this here patch series is trying to
> > accomplish, and I cannot bring myself to be interested in the proposed
> > build system changes, either, because I do not see any benefit in the
> > changes, only downsides.
> >
> > I find the distraction unnecessary.
>
> Perhaps I'm reading too much between the lines here, so forgive any
> undue knee-jerk reaction.

Okay, let's try an analogy.

Imagine that a person is asking for directions to the train station. And
the other person is replying by asking "did you know that this train
station was built in 1878? It is actually quite interesting a story...
[and then goes on to describe the history and what excites them about
it]". Now, the first person tries again to ask for directions, again does
not get an answer to that question, and is slowly starting to look at
their watch. The second person, being completely oblivious to all of this,
goes on with their wonderful story about the train station and its
cultural heritage. So the first person walks a bit further to ask a third
person, but the second person is not done yet and says "but you haven't
heard me out! That's disrespectful!".

Just imagine for a minute how you would feel if you were the first person.

And that is how I feel asking for reviews about the Scalar patch series
and then being forcefully dragged into that tangent about the build
process.

I find the well-established paradigm to keep contrib/'s build procedures
as confined to their own directory as possible the most reasonable way to
handle the build by virtue of _not_ polluting the top-level Makefile
unnecessarily. All of your objections strike me simply as personal
viewpoints, not as technical arguments, and they fail to address this
"pollution of the top-level Makefile" problem. I therefore strongly
disagree with your suggestion that the build system should be changed, I
would even argue that your suggestion should been dismissed on purely
technical grounds, and I wish you hadn't forced me to say this as
forcefully.

And even if I looked more favorably on your suggestion to change the build
procedure, I find this distraction about the build as little constructive
as the explanations about the train station's history above. Those
suggestions do succeed in derailing the conversation about how Git could
scale better, how Scalar _does_ teach Git how to scale better, and about
how to teach Git itself more and more of Scalar's tricks.

If you have ideas how to teach, say, `git clone` to perform a couple of
Scalar's tricks, by all means, let's hear them, or even better, let's see
those patches. If you want to change the build system, still, I cannot
stop you from sending patches to that end to the Git mailing list, but
please expect me to be uninterested in them in any way, and to prefer to
spend my efforts to improve Git elsewhere. If you have other ideas how to
improve on Scalar in a user-perceptible way, however, I am all ears again.

I hope this clarifies it, without the need to read between the lines,
Johannes

[PATCH v4 00/15] Upstreaming the Scalar command

[Johannes Schindelin via GitGitGadget]

tl;dr: This series contributes the Scalar command to the Git project. This
command provides an opinionated way to create and configure repositories
with a focus on very large repositories.

Changes since v3:

 * Moved the "Changes since" section to the top, to make it easier to see
   what changed.
 * Reworded the commit message of the first patch.
 * Removed the [RFC] prefix because I did not hear any objections against
   putting this into contrib/.

Changes since v2:

 * Adjusted the description of the list command in the manual page , as
   suggested by Bagas.
 * Addressed two style nits in cmd_run().
 * The documentation of git reconfigure -a was improved.

Changes since v1:

 * A couple typos were fixed
 * The code parsing the output of ls-remote was made more readable
 * The indentation used in scalar.txt now consistently uses tabs
 * We no longer hard-code core.bare = false when registering with Scalar


Background
==========

Years ago, Microsoft wanted to move the source code of the Windows operating
system to Git. The challenge there was to prove that Git could scale to
massive monorepos. The VFS for Git (formerly GVFS) project was born to take
up that challenge.

The final solution included a virtual filesystem (with both user-mode and
kernel components) and a customized fork of Git for Windows. This solution
contained several key concepts, such as only populating a portion of the
working directory, demand-fetching blobs, and performing periodic repo
maintenance in the background. However, the required kernel drivers made it
difficult to port the solution to other platforms.

But it was realized that many of these key concepts were independent of the
actual VFS and its projection of the working directory. The Scalar project
was created to make that separation, refine the key concepts, and then
extract those features into the new Scalar command.


The present
===========

The Scalar project provides a completely functional non-virtual experience
for monorepos. But why stop there. The Scalar project was designed to be a
self-destructing vehicle to allow those key concepts to be moved into core
Git itself for the benefit of all. For example, partial clone,
sparse-checkout, and background maintenance have already been upstreamed and
removed from Scalar proper. This patch series provides a C-based
implementation of the final remaining portions of the Scalar command. This
will make it easier for users to experiment with the Scalar command. It will
also make it substantially easier to experiment with moving functionality
from Scalar into core Git, while maintaining backwards-compatibility for
existing Scalar users.

The C-based Scalar has been shipped to Scalar users, and can be tested by
any interested reader:
https://github.com/microsoft/git/releases/tag/v2.33.0.vfs.0.0 (it offers a
Git for Windows installer, a macOS package and an Ubuntu package).


Opportunities
=============

Apart from providing the Scalar command, this contribution is intended to
serve as a basis for further mailing list discussions on moving (some of)
these key concepts into the main Git commands.

For example, we previously discussed the idea of a "git big-clone" that does
much of what "scalar clone" is doing. This patch series is a step to make
such functionality exist in the Git code base while we simmer on what such a
"git big-clone" command-line interface would look like.

This is one of many possible ways to do this. Creating a 'git big-clone'
could lock Git into backwards compatibility concerns so it is necessary to
approach such an endeavor with caution. As a discussion starter, the scalar
clone <url> command does roughly this:

 1. git clone --sparse --filter=blob:none /src
 2. git -C /src sparse-checkout init --cone
 3. git -C /src config (many times)
 4. git -C /src maintenance start

It is my hope inspire discussions about what parts of Scalar could go into
core Git, and where, and in which form. While we wish to maintain
backwards-compatibility of Scalar's command-line interface (because it is
already in use), by having the Scalar code in the same code base as Git's,
it will be much easier to move functionality without having to maintain
loose version coupling between independently-versioned Scalar and Git. The
tight version-coupling, along with having access to libgit.a also allows the
C-based implementation of Scalar to be much smaller than the original .NET
version.

For example, we might choose in the future to implement, say, git clone
--scale=partial,cone to initialize a partial clone with a cone-sparse
checkout, that would not only be totally doable, and not only would we
already have precedent and data to prove that this actually makes engineers
happy who have to work on ginormous repositories, but we could then also
implement it by moving parts of contrib/scalar/ to builtin/ (where
contrib/scalar/ would then call the built-ins accordingly rather than
hard-coding the defaults itself).

We now also have the opportunity to discuss the merits of Scalar's clone
caching, which is not actually part of this patch series because it is a bit
coupled with the GVFS parts of microsoft/git for the moment, where clones
automatically get registered with a populated alternate repository that is
identified by the URL, meaning: subsequent clones of the same repository are
vastly faster than the first one because they do not actually download the
already-received objects again, they access the cache instead.

Another thing that I could imagine to be discussed at length is the
distinction between enlistment and worktree (where the latter is the actual
Git worktree and usually lives in the src/ subdirectory of the former). This
encourages untracked and ignored files to be placed outside the worktree,
making Git's job much easier. This idea, too, might find its way in one way
or another into Git proper.

These are just a few concepts in Scalar that do not yet have equivalents in
Git. By putting this initial implementation into contrib/, we create a
foundation for future discussions of these concepts.

We plan on updating the recommended config settings in scalar register as
new Git features are available (such as builtin FSMonitor and sparse-index,
when ready). To facilitate upgrading existing Scalar enlistments, their
paths are automatically added to the [scalar] section of the global Git
config, and the scalar reconfigure --all command will process all of them.


Epilogue
========

Now, to address some questions that I imagine every reader has who made it
this far:

 * Why not put the Scalar functionality directly into a built-in? Creating a
   Git builtin requires scrutiny over every aspect of the feature, which is
   difficult to do while also maintaining the command-line interface
   contract and expected behavior of the Scalar command (there are existing
   users, after all). By having the Scalar command in contrib/, we present a
   simple option for users to have these features in the short term while
   the Git contributor community decides which bits to absorb into Git
   built-ins.
 * Why implement the Scalar command in the Git codebase? We ported Scalar to
   the microsoft/git fork for several reasons. First, we realized it was
   possible now that the core features exist inside Git itself. Second,
   compiling Scalar directly within a version of Git allows us to remove a
   version compatibility check from each config option that might or might
   not apply based on the installed Git version. Finally, this new location
   has greatly simplified our release process and the installation process
   for users. We now have ways to install Scalar with microsoft/git via
   winget, brew, and apt-get. This has been the case since we shipped
   v2.32.0 to our users, read: this setup has served us well already.
 * Why contribute Scalar to the Git project? We are biased, of course, yet
   we do have evidence that the Scalar command is a helpful tool that offers
   an simple way to handle huge repositories with ease. By contributing it
   to the core Git project, we are able to share it with more users,
   especially some users who do not want to install the microsoft/git fork.
   We intend to include Scalar as a component in git-for-windows/git, but
   are contributing it here first. Further, we think there is benefit to the
   Git developer community as this presents an example of how to set certain
   defaults that work for large repositories.
 * Does this integrate with the built-in FSMonitor yet? No, not yet. I do
   have a couple of add-on patch series lined up, one of them being the
   integration with the built-in FSMonitor, which obviously has to wait
   until the FSMonitor patch series advances further.

Derrick Stolee (4):
  scalar: 'register' sets recommended config and starts maintenance
  scalar: 'unregister' stops background maintenance
  scalar: implement 'scalar list'
  scalar: implement the `run` command

Johannes Schindelin (10):
  scalar: create a rudimentary executable
  scalar: start documenting the command
  scalar: create test infrastructure
  scalar: let 'unregister' handle a deleted enlistment directory
    gracefully
  scalar: implement the `clone` subcommand
  scalar: teach 'clone' to support the --single-branch option
  scalar: allow reconfiguring an existing enlistment
  scalar: teach 'reconfigure' to optionally handle all registered
    enlistments
  scalar: implement the `version` command
  scalar: accept -C and -c options before the subcommand

Matthew John Cheetham (1):
  scalar: implement the `delete` command

 Makefile                         |   8 +
 contrib/scalar/.gitignore        |   5 +
 contrib/scalar/Makefile          |  57 +++
 contrib/scalar/scalar.c          | 844 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        | 154 ++++++
 contrib/scalar/t/Makefile        |  78 +++
 contrib/scalar/t/t9099-scalar.sh |  88 ++++
 7 files changed, 1234 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c
 create mode 100644 contrib/scalar/scalar.txt
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh


base-commit: ebf3c04b262aa27fbb97f8a0156c2347fecafafb
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1005%2Fdscho%2Fscalar-the-beginning-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1005/dscho/scalar-the-beginning-v4
Pull-Request: https://github.com/gitgitgadget/git/pull/1005

Range-diff vs v3:

  1:  b8c7d3f8450 !  1:  852ec003109 scalar: create a rudimentary executable
     @@ Commit message
          with the intention to facilitate integrating the functionality into core
          Git all while maintaining backwards-compatibility for existing Scalar
          users (which will be much easier when both live in the same worktree).
     -    It was always to plan to contribute all of the proven strategies back to
     -    core Git.
     +    It has always been the plan to contribute all of the proven strategies
     +    back to core Git.
      
          For example, while the virtual filesystem provided by VFS for Git helped
          the team developing the Windows operating system to move onto Git, while
     @@ Commit message
          repository in a healthy state.
      
          With partial clone, sparse checkout and `git maintenance` having been
     -    upstreamed, there is little left that `scalar.exe` does that which
     -    `git.exe` cannot do. One such thing is that `scalar clone <url>` will
     +    upstreamed, there is little left that `scalar.exe` does which `git.exe`
     +    cannot do. One such thing is that `scalar clone <url>` will
          automatically set up a partial, sparse clone, and configure
          known-helpful settings from the start.
      
  2:  4f886575dcf =  2:  6ab9c7195da scalar: start documenting the command
  3:  bcfde9bc765 =  3:  14992033d7c scalar: create test infrastructure
  4:  ee3e26a0c4e =  4:  bbbc4c33390 scalar: 'register' sets recommended config and starts maintenance
  5:  6142f75875b =  5:  eadcddb2a9b scalar: 'unregister' stops background maintenance
  6:  82dd253154f =  6:  c3c2f3a4971 scalar: let 'unregister' handle a deleted enlistment directory gracefully
  7:  d291d3723a6 =  7:  90ef9b826b2 scalar: implement 'scalar list'
  8:  40dbf61771e =  8:  79cde4417d8 scalar: implement the `clone` subcommand
  9:  414dbe7d859 =  9:  0acdaeb7396 scalar: teach 'clone' to support the --single-branch option
 10:  76de416a643 = 10:  64e3403ac12 scalar: implement the `run` command
 11:  655a902b9df = 11:  ada242c7c8c scalar: allow reconfiguring an existing enlistment
 12:  2d1987bfcda = 12:  5c11117da51 scalar: teach 'reconfigure' to optionally handle all registered enlistments
 13:  c67938299ee = 13:  914c16c7fcd scalar: implement the `delete` command
 14:  d2cd2b7094b = 14:  06995770120 scalar: implement the `version` command
 15:  7ccc4f8b9b0 = 15:  7539725bb4f scalar: accept -C and -c options before the subcommand

-- 
gitgitgadget

[PATCH v4 01/15] scalar: create a rudimentary executable

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The idea of Scalar (https://github.com/microsoft/scalar), and before
that, of VFS for Git, has always been to prove that Git _can_ scale, and
to upstream whatever strategies have been demonstrated to help.

With this patch, we start the journey from that C# project to move what
is left to Git's own `contrib/` directory, reimplementing it in pure C,
with the intention to facilitate integrating the functionality into core
Git all while maintaining backwards-compatibility for existing Scalar
users (which will be much easier when both live in the same worktree).
It has always been the plan to contribute all of the proven strategies
back to core Git.

For example, while the virtual filesystem provided by VFS for Git helped
the team developing the Windows operating system to move onto Git, while
trying to upstream it we realized that it cannot be done: getting the
virtual filesystem to work (which we only managed to implement fully on
Windows, but not on, say, macOS or Linux), and the required server-side
support for the GVFS protocol, made this not quite feasible.

The Scalar project learned from that and tackled the problem with
different tactics: instead of pretending to Git that the working
directory is fully populated, it _specifically_ teaches Git about
partial clone (which is based on VFS for Git's cache server), about
sparse checkout (which VFS for Git tried to do transparently, in the
file system layer), and regularly runs maintenance tasks to keep the
repository in a healthy state.

With partial clone, sparse checkout and `git maintenance` having been
upstreamed, there is little left that `scalar.exe` does which `git.exe`
cannot do. One such thing is that `scalar clone <url>` will
automatically set up a partial, sparse clone, and configure
known-helpful settings from the start.

So let's bring this convenience into Git's tree.

The idea here is that you can (optionally) build Scalar via

	make -C contrib/scalar/Makefile

This will build the `scalar` executable and put it into the
contrib/scalar/ subdirectory.

The slightly awkward addition of the `contrib/scalar/*` bits to the
top-level `Makefile` are actually really required: we want to link to
`libgit.a`, which means that we will need to use the very same `CFLAGS`
and `LDFLAGS` as the rest of Git.

An early development version of this patch tried to replicate all the
conditional code in `contrib/scalar/Makefile` (e.g. `NO_POLL`) just like
`contrib/svn-fe/Makefile` used to do before it was retired. It turned
out to be quite the whack-a-mole game: the SHA-1-related flags, the
flags enabling/disabling `compat/poll/`, `compat/regex/`,
`compat/win32mmap.c` & friends depending on the current platform... To
put it mildly: it was a major mess.

Instead, this patch makes minimal changes to the top-level `Makefile` so
that the bits in `contrib/scalar/` can be compiled and linked, and
adds a `contrib/scalar/Makefile` that uses the top-level `Makefile` in a
most minimal way to do the actual compiling.

Note: With this commit, we only establish the infrastructure, no
Scalar functionality is implemented yet; We will do that incrementally
over the next few commits.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 Makefile                  |  8 ++++++++
 contrib/scalar/.gitignore |  2 ++
 contrib/scalar/Makefile   | 34 ++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.c   | 36 ++++++++++++++++++++++++++++++++++++
 4 files changed, 80 insertions(+)
 create mode 100644 contrib/scalar/.gitignore
 create mode 100644 contrib/scalar/Makefile
 create mode 100644 contrib/scalar/scalar.c

diff --git a/Makefile b/Makefile
index c3565fc0f8f..2d5c822f7a8 100644
--- a/Makefile
+++ b/Makefile
@@ -2447,6 +2447,10 @@ endif
 .PHONY: objects
 objects: $(OBJECTS)
 
+SCALAR_SOURCES := contrib/scalar/scalar.c
+SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
+OBJECTS += $(SCALAR_OBJECTS)
+
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
 
@@ -2586,6 +2590,10 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
+contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 $(LIB_FILE): $(LIB_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
new file mode 100644
index 00000000000..ff3d47e84d0
--- /dev/null
+++ b/contrib/scalar/.gitignore
@@ -0,0 +1,2 @@
+/*.exe
+/scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
new file mode 100644
index 00000000000..40c03ad10e1
--- /dev/null
+++ b/contrib/scalar/Makefile
@@ -0,0 +1,34 @@
+QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
+QUIET_SUBDIR1  =
+
+ifneq ($(findstring s,$(MAKEFLAGS)),s)
+ifndef V
+	QUIET_SUBDIR0  = +@subdir=
+	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
+			 $(MAKE) $(PRINT_DIR) -C $$subdir
+else
+	export V
+endif
+endif
+
+all:
+
+include ../../config.mak.uname
+-include ../../config.mak.autogen
+-include ../../config.mak
+
+TARGETS = scalar$(X) scalar.o
+GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
+
+all: scalar$X
+
+$(GITLIBS):
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
+
+$(TARGETS): $(GITLIBS) scalar.c
+	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
+
+clean:
+	$(RM) $(TARGETS)
+
+.PHONY: all clean FORCE
diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
new file mode 100644
index 00000000000..7cff29e0fcd
--- /dev/null
+++ b/contrib/scalar/scalar.c
@@ -0,0 +1,36 @@
+/*
+ * The Scalar command-line interface.
+ */
+
+#include "cache.h"
+#include "gettext.h"
+#include "parse-options.h"
+
+static struct {
+	const char *name;
+	int (*fn)(int, const char **);
+} builtins[] = {
+	{ NULL, NULL},
+};
+
+int cmd_main(int argc, const char **argv)
+{
+	struct strbuf scalar_usage = STRBUF_INIT;
+	int i;
+
+	if (argc > 1) {
+		argv++;
+		argc--;
+
+		for (i = 0; builtins[i].name; i++)
+			if (!strcmp(builtins[i].name, argv[0]))
+				return !!builtins[i].fn(argc, argv);
+	}
+
+	strbuf_addstr(&scalar_usage,
+		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+	for (i = 0; builtins[i].name; i++)
+		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
+
+	usage(scalar_usage.buf);
+}
-- 
gitgitgadget

[PATCH v4 02/15] scalar: start documenting the command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This commit establishes the infrastructure to build the manual page for
the `scalar` command.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/.gitignore |  3 +++
 contrib/scalar/Makefile   | 14 +++++++++++++-
 contrib/scalar/scalar.txt | 38 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 contrib/scalar/scalar.txt

diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
index ff3d47e84d0..00441073f59 100644
--- a/contrib/scalar/.gitignore
+++ b/contrib/scalar/.gitignore
@@ -1,2 +1,5 @@
+/*.xml
+/*.1
+/*.html
 /*.exe
 /scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 40c03ad10e1..85c186634e9 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -6,6 +6,7 @@ ifndef V
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
+	QUIET          = @
 else
 	export V
 endif
@@ -30,5 +31,16 @@ $(TARGETS): $(GITLIBS) scalar.c
 
 clean:
 	$(RM) $(TARGETS)
+	$(RM) scalar.1 scalar.html scalar.xml
 
-.PHONY: all clean FORCE
+docs: scalar.html scalar.1
+
+scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
+
+scalar.html scalar.1: scalar.txt
+	$(QUIET_SUBDIR0)../../Documentation$(QUIET_SUBDIR1) \
+		MAN_TXT=../contrib/scalar/scalar.txt \
+		../contrib/scalar/$@
+	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
+
+.PHONY: all clean docs FORCE
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
new file mode 100644
index 00000000000..5f7131861a5
--- /dev/null
+++ b/contrib/scalar/scalar.txt
@@ -0,0 +1,38 @@
+scalar(1)
+=========
+
+NAME
+----
+scalar - an opinionated repository management tool
+
+SYNOPSIS
+--------
+[verse]
+scalar <command> [<options>]
+
+DESCRIPTION
+-----------
+
+Scalar is an opinionated repository management tool. By creating new
+repositories or registering existing repositories with Scalar, your Git
+experience will speed up. Scalar sets advanced Git config settings,
+maintains your repositories in the background, and helps reduce data sent
+across the network.
+
+An important Scalar concept is the enlistment: this is the top-level directory
+of the project. It usually contains the subdirectory `src/` which is a Git
+worktree. This encourages the separation between tracked files (inside `src/`)
+and untracked files, such as build artifacts (outside `src/`). When registering
+an existing Git worktree with Scalar whose name is not `src`, the enlistment
+will be identical to the worktree.
+
+The `scalar` command implements various subcommands, and different options
+depending on the subcommand.
+
+SEE ALSO
+--------
+linkgit:git-maintenance[1].
+
+Scalar
+---
+Associated with the linkgit:git[1] suite
-- 
gitgitgadget

[PATCH v4 03/15] scalar: create test infrastructure

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

To test the Scalar command, create a test script in contrib/scalar/t
that is executed as `make -C contrib/scalar test`. Since Scalar has no
meaningful capabilities yet, the only test is rather simple. We will add
more tests in subsequent commits that introduce corresponding, new
functionality.

Note: this test script is intended to test `scalar` only lightly, even
after all of the functionality is implemented.

A more comprehensive functional (or: integration) test suite can be
found at https://github.com/microsoft/scalar; It is used in the workflow
https://github.com/microsoft/git/blob/HEAD/.github/workflows/scalar-functional-tests.yml
in Microsoft's Git fork. This test suite performs end-to-end tests with
a real remote repository, and is run as part of the regular CI builds.
Since those tests require some functionality supported only by
Microsoft's Git fork ("GVFS protocol"), there is no intention to port
that fuller test suite to `contrib/scalar/`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/Makefile          | 17 +++++--
 contrib/scalar/t/Makefile        | 78 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 17 +++++++
 3 files changed, 109 insertions(+), 3 deletions(-)
 create mode 100644 contrib/scalar/t/Makefile
 create mode 100755 contrib/scalar/t/t9099-scalar.sh

diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
index 85c186634e9..8620042f281 100644
--- a/contrib/scalar/Makefile
+++ b/contrib/scalar/Makefile
@@ -3,6 +3,7 @@ QUIET_SUBDIR1  =
 
 ifneq ($(findstring s,$(MAKEFLAGS)),s)
 ifndef V
+	QUIET_GEN      = @echo '   ' GEN $@;
 	QUIET_SUBDIR0  = +@subdir=
 	QUIET_SUBDIR1  = ;$(NO_SUBDIR) echo '   ' SUBDIR $$subdir; \
 			 $(MAKE) $(PRINT_DIR) -C $$subdir
@@ -21,7 +22,7 @@ include ../../config.mak.uname
 TARGETS = scalar$(X) scalar.o
 GITLIBS = ../../common-main.o ../../libgit.a ../../xdiff/lib.a
 
-all: scalar$X
+all: scalar$X ../../bin-wrappers/scalar
 
 $(GITLIBS):
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(subst ../../,,$@)
@@ -30,9 +31,19 @@ $(TARGETS): $(GITLIBS) scalar.c
 	$(QUIET_SUBDIR0)../.. $(QUIET_SUBDIR1) $(patsubst %,contrib/scalar/%,$@)
 
 clean:
-	$(RM) $(TARGETS)
+	$(RM) $(TARGETS) ../../bin-wrappers/scalar
 	$(RM) scalar.1 scalar.html scalar.xml
 
+../../bin-wrappers/scalar: ../../wrap-for-bin.sh Makefile
+	@mkdir -p ../../bin-wrappers
+	$(QUIET_GEN)sed -e '1s|#!.*/sh|#!$(SHELL_PATH_SQ)|' \
+	     -e 's|@@BUILD_DIR@@|$(shell cd ../.. && pwd)|' \
+	     -e 's|@@PROG@@|contrib/scalar/scalar$(X)|' < $< > $@ && \
+	chmod +x $@
+
+test: all
+	$(MAKE) -C t
+
 docs: scalar.html scalar.1
 
 scalar.html: | scalar.1 # prevent them from trying to build `doc.dep` in parallel
@@ -43,4 +54,4 @@ scalar.html scalar.1: scalar.txt
 		../contrib/scalar/$@
 	$(QUIET)test scalar.1 != "$@" || mv ../../Documentation/$@ .
 
-.PHONY: all clean docs FORCE
+.PHONY: all clean docs test FORCE
diff --git a/contrib/scalar/t/Makefile b/contrib/scalar/t/Makefile
new file mode 100644
index 00000000000..6170672bb37
--- /dev/null
+++ b/contrib/scalar/t/Makefile
@@ -0,0 +1,78 @@
+# Run scalar tests
+#
+# Copyright (c) 2005,2021 Junio C Hamano, Johannes Schindelin
+#
+
+-include ../../../config.mak.autogen
+-include ../../../config.mak
+
+SHELL_PATH ?= $(SHELL)
+PERL_PATH ?= /usr/bin/perl
+RM ?= rm -f
+PROVE ?= prove
+DEFAULT_TEST_TARGET ?= test
+TEST_LINT ?= test-lint
+
+ifdef TEST_OUTPUT_DIRECTORY
+TEST_RESULTS_DIRECTORY = $(TEST_OUTPUT_DIRECTORY)/test-results
+else
+TEST_RESULTS_DIRECTORY = ../../../t/test-results
+endif
+
+# Shell quote;
+SHELL_PATH_SQ = $(subst ','\'',$(SHELL_PATH))
+PERL_PATH_SQ = $(subst ','\'',$(PERL_PATH))
+TEST_RESULTS_DIRECTORY_SQ = $(subst ','\'',$(TEST_RESULTS_DIRECTORY))
+
+T = $(sort $(wildcard t[0-9][0-9][0-9][0-9]-*.sh))
+
+all: $(DEFAULT_TEST_TARGET)
+
+test: $(TEST_LINT)
+	$(MAKE) aggregate-results-and-cleanup
+
+prove: $(TEST_LINT)
+	@echo "*** prove ***"; GIT_CONFIG=.git/config $(PROVE) --exec '$(SHELL_PATH_SQ)' $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
+	$(MAKE) clean-except-prove-cache
+
+$(T):
+	@echo "*** $@ ***"; GIT_CONFIG=.git/config '$(SHELL_PATH_SQ)' $@ $(GIT_TEST_OPTS)
+
+clean-except-prove-cache:
+	$(RM) -r 'trash directory'.* '$(TEST_RESULTS_DIRECTORY_SQ)'
+	$(RM) -r valgrind/bin
+
+clean: clean-except-prove-cache
+	$(RM) .prove
+
+test-lint: test-lint-duplicates test-lint-executable test-lint-shell-syntax
+
+test-lint-duplicates:
+	@dups=`echo $(T) | tr ' ' '\n' | sed 's/-.*//' | sort | uniq -d` && \
+		test -z "$$dups" || { \
+		echo >&2 "duplicate test numbers:" $$dups; exit 1; }
+
+test-lint-executable:
+	@bad=`for i in $(T); do test -x "$$i" || echo $$i; done` && \
+		test -z "$$bad" || { \
+		echo >&2 "non-executable tests:" $$bad; exit 1; }
+
+test-lint-shell-syntax:
+	@'$(PERL_PATH_SQ)' ../../../t/check-non-portable-shell.pl $(T)
+
+aggregate-results-and-cleanup: $(T)
+	$(MAKE) aggregate-results
+	$(MAKE) clean
+
+aggregate-results:
+	for f in '$(TEST_RESULTS_DIRECTORY_SQ)'/t*-*.counts; do \
+		echo "$$f"; \
+	done | '$(SHELL_PATH_SQ)' ../../../t/aggregate-results.sh
+
+valgrind:
+	$(MAKE) GIT_TEST_OPTS="$(GIT_TEST_OPTS) --valgrind"
+
+test-results:
+	mkdir -p test-results
+
+.PHONY: $(T) aggregate-results clean valgrind
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
new file mode 100755
index 00000000000..16f2b72b126
--- /dev/null
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -0,0 +1,17 @@
+#!/bin/sh
+
+test_description='test the `scalar` command'
+
+TEST_DIRECTORY=$PWD/../../../t
+export TEST_DIRECTORY
+
+# Make it work with --no-bin-wrappers
+PATH=$PWD/..:$PATH
+
+. ../../../t/test-lib.sh
+
+test_expect_success 'scalar shows a usage' '
+	test_expect_code 129 scalar -h
+'
+
+test_done
-- 
gitgitgadget

[PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Let's start implementing the `register` command. With this commit,
recommended settings are configured upon `scalar register`, and Git's
background maintenance is started.

The recommended config settings may very well change in the future. For
example, once the built-in FSMonitor is available, we will want to
enable it upon `scalar register`. For that reason, we explicitly support
running `scalar register` in an already-registered enlistment.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 255 ++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt |  18 ++-
 2 files changed, 272 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7cff29e0fcd..0e627bb100e 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -5,11 +5,266 @@
 #include "cache.h"
 #include "gettext.h"
 #include "parse-options.h"
+#include "config.h"
+#include "run-command.h"
+
+/*
+ * Remove the deepest subdirectory in the provided path string. Path must not
+ * include a trailing path separator. Returns 1 if parent directory found,
+ * otherwise 0.
+ */
+static int strbuf_parent_directory(struct strbuf *buf)
+{
+	size_t len = buf->len;
+	size_t offset = offset_1st_component(buf->buf);
+	char *path_sep = find_last_dir_sep(buf->buf + offset);
+	strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset);
+
+	return buf->len < len;
+}
+
+static void setup_enlistment_directory(int argc, const char **argv,
+				       const char * const *usagestr,
+				       const struct option *options,
+				       struct strbuf *enlistment_root)
+{
+	struct strbuf path = STRBUF_INIT;
+	char *root;
+	int enlistment_found = 0;
+
+	if (startup_info->have_repository)
+		BUG("gitdir already set up?!?");
+
+	if (argc > 1)
+		usage_with_options(usagestr, options);
+
+	/* find the worktree, determine its corresponding root */
+	if (argc == 1)
+		strbuf_add_absolute_path(&path, argv[0]);
+	else if (strbuf_getcwd(&path) < 0)
+		die(_("need a working directory"));
+
+	strbuf_trim_trailing_dir_sep(&path);
+	do {
+		const size_t len = path.len;
+
+		/* check if currently in enlistment root with src/ workdir */
+		strbuf_addstr(&path, "/src/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_strip_suffix(&path, "/.git");
+
+			if (enlistment_root)
+				strbuf_add(enlistment_root, path.buf, len);
+
+			enlistment_found = 1;
+			break;
+		}
+
+		/* reset to original path */
+		strbuf_setlen(&path, len);
+
+		/* check if currently in workdir */
+		strbuf_addstr(&path, "/.git");
+		if (is_git_directory(path.buf)) {
+			strbuf_setlen(&path, len);
+
+			if (enlistment_root) {
+				/*
+				 * If the worktree's directory's name is `src`, the enlistment is the
+				 * parent directory, otherwise it is identical to the worktree.
+				 */
+				root = strip_path_suffix(path.buf, "src");
+				strbuf_addstr(enlistment_root, root ? root : path.buf);
+				free(root);
+			}
+
+			enlistment_found = 1;
+			break;
+		}
+
+		strbuf_setlen(&path, len);
+	} while (strbuf_parent_directory(&path));
+
+	if (!enlistment_found)
+		die(_("could not find enlistment root"));
+
+	if (chdir(path.buf) < 0)
+		die_errno(_("could not switch to '%s'"), path.buf);
+
+	strbuf_release(&path);
+	setup_git_directory();
+}
+
+static int run_git(const char *arg, ...)
+{
+	struct strvec argv = STRVEC_INIT;
+	va_list args;
+	const char *p;
+	int res;
+
+	va_start(args, arg);
+	strvec_push(&argv, arg);
+	while ((p = va_arg(args, const char *)))
+		strvec_push(&argv, p);
+	va_end(args);
+
+	res = run_command_v_opt(argv.v, RUN_GIT_CMD);
+
+	strvec_clear(&argv);
+	return res;
+}
+
+static int set_recommended_config(void)
+{
+	struct {
+		const char *key;
+		const char *value;
+	} config[] = {
+		{ "am.keepCR", "true" },
+		{ "core.FSCache", "true" },
+		{ "core.multiPackIndex", "true" },
+		{ "core.preloadIndex", "true" },
+#ifndef WIN32
+		{ "core.untrackedCache", "true" },
+#else
+		/*
+		 * Unfortunately, Scalar's Functional Tests demonstrated
+		 * that the untracked cache feature is unreliable on Windows
+		 * (which is a bummer because that platform would benefit the
+		 * most from it). For some reason, freshly created files seem
+		 * not to update the directory's `lastModified` time
+		 * immediately, but the untracked cache would need to rely on
+		 * that.
+		 *
+		 * Therefore, with a sad heart, we disable this very useful
+		 * feature on Windows.
+		 */
+		{ "core.untrackedCache", "false" },
+#endif
+		{ "core.logAllRefUpdates", "true" },
+		{ "credential.https://dev.azure.com.useHttpPath", "true" },
+		{ "credential.validate", "false" }, /* GCM4W-only */
+		{ "gc.auto", "0" },
+		{ "gui.GCWarning", "false" },
+		{ "index.threads", "true" },
+		{ "index.version", "4" },
+		{ "merge.stat", "false" },
+		{ "merge.renames", "false" },
+		{ "pack.useBitmaps", "false" },
+		{ "pack.useSparse", "true" },
+		{ "receive.autoGC", "false" },
+		{ "reset.quiet", "true" },
+		{ "feature.manyFiles", "false" },
+		{ "feature.experimental", "false" },
+		{ "fetch.unpackLimit", "1" },
+		{ "fetch.writeCommitGraph", "false" },
+#ifdef WIN32
+		{ "http.sslBackend", "schannel" },
+#endif
+		{ "status.aheadBehind", "false" },
+		{ "commitGraph.generationVersion", "1" },
+		{ "core.autoCRLF", "false" },
+		{ "core.safeCRLF", "false" },
+		{ NULL, NULL },
+	};
+	int i;
+	char *value;
+
+	for (i = 0; config[i].key; i++) {
+		if (git_config_get_string(config[i].key, &value)) {
+			trace2_data_string("scalar", the_repository, config[i].key, "created");
+			if (git_config_set_gently(config[i].key,
+						  config[i].value) < 0)
+				return error(_("could not configure %s=%s"),
+					     config[i].key, config[i].value);
+		} else {
+			trace2_data_string("scalar", the_repository, config[i].key, "exists");
+			free(value);
+		}
+	}
+
+	/*
+	 * The `log.excludeDecoration` setting is special because it allows
+	 * for multiple values.
+	 */
+	if (git_config_get_string("log.excludeDecoration", &value)) {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "created");
+		if (git_config_set_multivar_gently("log.excludeDecoration",
+						   "refs/prefetch/*",
+						   CONFIG_REGEX_NONE, 0))
+			return error(_("could not configure "
+				       "log.excludeDecoration"));
+	} else {
+		trace2_data_string("scalar", the_repository,
+				   "log.excludeDecoration", "exists");
+		free(value);
+	}
+
+	return 0;
+}
+
+static int start_maintenance(void)
+{
+	return run_git("maintenance", "start", NULL);
+}
+
+static int add_enlistment(void)
+{
+	int res;
+
+	if (!the_repository->worktree)
+		die(_("Scalar enlistments require a worktree"));
+
+	res = run_git("config", "--global", "--get", "--fixed-value",
+		      "scalar.repo", the_repository->worktree, NULL);
+
+	/*
+	 * If the setting is already there, then do nothing.
+	 */
+	if (!res)
+		return 0;
+
+	return run_git("config", "--global", "--add",
+		       "scalar.repo", the_repository->worktree, NULL);
+}
+
+static int register_dir(void)
+{
+	int res = add_enlistment();
+
+	if (!res)
+		res = set_recommended_config();
+
+	if (!res)
+		res = start_maintenance();
+
+	return res;
+}
+
+static int cmd_register(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar register [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return register_dir();
+}
 
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "register", cmd_register },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 5f7131861a5..568987064b2 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar <command> [<options>]
+scalar register [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -29,6 +29,22 @@ will be identical to the worktree.
 The `scalar` command implements various subcommands, and different options
 depending on the subcommand.
 
+COMMANDS
+--------
+
+Register
+~~~~~~~~
+
+register [<enlistment>]::
+	Adds the enlistment's repository to the list of registered repositories
+	and starts background maintenance. If `<enlistment>` is not provided,
+	then the enlistment associated with the current working directory is
+	registered.
++
+Note: when this subcommand is called in a worktree that is called `src/`, its
+parent directory is considered to be the Scalar enlistment. If the worktree is
+_not_ called `src/`, it itself will be considered to be the Scalar enlistment.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v4 05/15] scalar: 'unregister' stops background maintenance

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Just like `scalar register` starts the scheduled background maintenance,
`scalar unregister` stops it. Note that we use `git maintenance start`
in `scalar register`, but we do not use `git maintenance stop` in
`scalar unregister`: this would stop maintenance for _all_ repositories,
not just for the one we want to unregister.

The `unregister` command also removes the corresponding entry from the
`[scalar]` section in the global Git config.

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 50 ++++++++++++++++++++++++++++++++-------
 contrib/scalar/scalar.txt |  8 +++++++
 2 files changed, 50 insertions(+), 8 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 0e627bb100e..2b5c52a25f5 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -204,12 +204,12 @@ static int set_recommended_config(void)
 	return 0;
 }
 
-static int start_maintenance(void)
+static int toggle_maintenance(int enable)
 {
-	return run_git("maintenance", "start", NULL);
+	return run_git("maintenance", enable ? "start" : "unregister", NULL);
 }
 
-static int add_enlistment(void)
+static int add_or_remove_enlistment(int add)
 {
 	int res;
 
@@ -220,24 +220,39 @@ static int add_enlistment(void)
 		      "scalar.repo", the_repository->worktree, NULL);
 
 	/*
-	 * If the setting is already there, then do nothing.
+	 * If we want to add and the setting is already there, then do nothing.
+	 * If we want to remove and the setting is not there, then do nothing.
 	 */
-	if (!res)
+	if ((add && !res) || (!add && res))
 		return 0;
 
-	return run_git("config", "--global", "--add",
+	return run_git("config", "--global", add ? "--add" : "--unset",
+		       add ? "--no-fixed-value" : "--fixed-value",
 		       "scalar.repo", the_repository->worktree, NULL);
 }
 
 static int register_dir(void)
 {
-	int res = add_enlistment();
+	int res = add_or_remove_enlistment(1);
 
 	if (!res)
 		res = set_recommended_config();
 
 	if (!res)
-		res = start_maintenance();
+		res = toggle_maintenance(1);
+
+	return res;
+}
+
+static int unregister_dir(void)
+{
+	int res = 0;
+
+	if (toggle_maintenance(0) < 0)
+		res = -1;
+
+	if (add_or_remove_enlistment(0) < 0)
+		res = -1;
 
 	return res;
 }
@@ -260,11 +275,30 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_unregister(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar unregister [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return unregister_dir();
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
 	{ "register", cmd_register },
+	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 568987064b2..d9a79984492 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -9,6 +9,7 @@ SYNOPSIS
 --------
 [verse]
 scalar register [<enlistment>]
+scalar unregister [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -45,6 +46,13 @@ Note: when this subcommand is called in a worktree that is called `src/`, its
 parent directory is considered to be the Scalar enlistment. If the worktree is
 _not_ called `src/`, it itself will be considered to be the Scalar enlistment.
 
+Unregister
+~~~~~~~~~~
+
+unregister [<enlistment>]::
+	Remove the specified repository from the list of repositories
+	registered with Scalar and stop the scheduled background maintenance.
+
 SEE ALSO
 --------
 linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v4 06/15] scalar: let 'unregister' handle a deleted enlistment directory gracefully

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

When a user deleted an enlistment manually, let's be generous and
_still_ unregister it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 46 ++++++++++++++++++++++++++++++++
 contrib/scalar/t/t9099-scalar.sh | 15 +++++++++++
 2 files changed, 61 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 2b5c52a25f5..d114c038b64 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -275,6 +275,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int remove_deleted_enlistment(struct strbuf *path)
+{
+	int res = 0;
+	strbuf_realpath_forgiving(path, path->buf, 1);
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "scalar.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	if (run_git("config", "--global",
+		    "--unset", "--fixed-value",
+		    "maintenance.repo", path->buf, NULL) < 0)
+		res = -1;
+
+	return res;
+}
+
 static int cmd_unregister(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -288,6 +306,34 @@ static int cmd_unregister(int argc, const char **argv)
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
+	/*
+	 * Be forgiving when the enlistment or worktree does not even exist any
+	 * longer; This can be the case if a user deleted the worktree by
+	 * mistake and _still_ wants to unregister the thing.
+	 */
+	if (argc == 1) {
+		struct strbuf src_path = STRBUF_INIT, workdir_path = STRBUF_INIT;
+
+		strbuf_addf(&src_path, "%s/src/.git", argv[0]);
+		strbuf_addf(&workdir_path, "%s/.git", argv[0]);
+		if (!is_directory(src_path.buf) && !is_directory(workdir_path.buf)) {
+			/* remove possible matching registrations */
+			int res = -1;
+
+			strbuf_strip_suffix(&src_path, "/.git");
+			res = remove_deleted_enlistment(&src_path) && res;
+
+			strbuf_strip_suffix(&workdir_path, "/.git");
+			res = remove_deleted_enlistment(&workdir_path) && res;
+
+			strbuf_release(&src_path);
+			strbuf_release(&workdir_path);
+			return res;
+		}
+		strbuf_release(&src_path);
+		strbuf_release(&workdir_path);
+	}
+
 	setup_enlistment_directory(argc, argv, usage, options, NULL);
 
 	return unregister_dir();
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 16f2b72b126..ef0e8d680d5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -14,4 +14,19 @@ test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
 
+test_expect_success 'scalar unregister' '
+	git init vanish/src &&
+	scalar register vanish/src &&
+	git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	grep -F "$(pwd)/vanish/src" scalar.repos &&
+	rm -rf vanish/src/.git &&
+	scalar unregister vanish &&
+	test_must_fail git config --get --global --fixed-value \
+		maintenance.repo "$(pwd)/vanish/src" &&
+	scalar list >scalar.repos &&
+	! grep -F "$(pwd)/vanish/src" scalar.repos
+'
+
 test_done
-- 
gitgitgadget

[PATCH v4 07/15] scalar: implement 'scalar list'

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

The produced list simply consists of those repositories registered under
the multi-valued `scalar.repo` config setting in the user's Git config.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 11 +++++++++++
 contrib/scalar/scalar.txt | 11 ++++++++++-
 2 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index d114c038b64..7f5436399da 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -257,6 +257,16 @@ static int unregister_dir(void)
 	return res;
 }
 
+static int cmd_list(int argc, const char **argv)
+{
+	if (argc != 1)
+		die(_("`scalar list` does not take arguments"));
+
+	if (run_git("config", "--global", "--get-all", "scalar.repo", NULL) < 0)
+		return -1;
+	return 0;
+}
+
 static int cmd_register(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -343,6 +353,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ NULL, NULL},
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d9a79984492..f93e3d00efd 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 
@@ -28,11 +29,19 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand.
+depending on the subcommand. With the exception of `list`, all subcommands
+expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+List
+~~~~
+
+list::
+	List enlistments that are currently registered by Scalar. This
+	subcommand does not need to be run inside an enlistment.
+
 Register
 ~~~~~~~~
 
-- 
gitgitgadget

[PATCH v4 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This implements Scalar's opinionated `clone` command: it tries to use a
partial clone and sets up a sparse checkout by default. In contrast to
`git clone`, `scalar clone` sets up the worktree in the `src/`
subdirectory, to encourage a separation between the source files and the
build output (which helps Git tremendously because it avoids untracked
files that have to be specifically ignored when refreshing the index).

Also, it registers the repository for regular, scheduled maintenance,
and configures a flurry of configuration settings based on the
experience and experiments of the Microsoft Windows and the Microsoft
Office development teams.

Note: since the `scalar clone` command is by far the most commonly
called `scalar` subcommand, we document it at the top of the manual
page.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 201 +++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  31 ++++-
 contrib/scalar/t/t9099-scalar.sh |  32 +++++
 3 files changed, 261 insertions(+), 3 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7f5436399da..bf18003b297 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -7,6 +7,7 @@
 #include "parse-options.h"
 #include "config.h"
 #include "run-command.h"
+#include "refs.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -257,6 +258,205 @@ static int unregister_dir(void)
 	return res;
 }
 
+/* printf-style interface, expects `<key>=<value>` argument */
+static int set_config(const char *fmt, ...)
+{
+	struct strbuf buf = STRBUF_INIT;
+	char *value;
+	int res;
+	va_list args;
+
+	va_start(args, fmt);
+	strbuf_vaddf(&buf, fmt, args);
+	va_end(args);
+
+	value = strchr(buf.buf, '=');
+	if (value)
+		*(value++) = '\0';
+	res = git_config_set_gently(buf.buf, value);
+	strbuf_release(&buf);
+
+	return res;
+}
+
+static char *remote_default_branch(const char *url)
+{
+	struct child_process cp = CHILD_PROCESS_INIT;
+	struct strbuf out = STRBUF_INIT;
+
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "ls-remote", "--symref", url, "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		const char *line = out.buf;
+
+		while (*line) {
+			const char *eol = strchrnul(line, '\n'), *p;
+			size_t len = eol - line;
+			char *branch;
+
+			if (!skip_prefix(line, "ref: ", &p) ||
+			    !strip_suffix_mem(line, &len, "\tHEAD")) {
+				line = eol + (*eol == '\n');
+				continue;
+			}
+
+			eol = line + len;
+			if (skip_prefix(p, "refs/heads/", &p)) {
+				branch = xstrndup(p, eol - p);
+				strbuf_release(&out);
+				return branch;
+			}
+
+			error(_("remote HEAD is not a branch: '%.*s'"),
+			      (int)(eol - p), p);
+			strbuf_release(&out);
+			return NULL;
+		}
+	}
+	warning(_("failed to get default branch name from remote; "
+		  "using local default"));
+	strbuf_reset(&out);
+
+	child_process_init(&cp);
+	cp.git_cmd = 1;
+	strvec_pushl(&cp.args, "symbolic-ref", "--short", "HEAD", NULL);
+	if (!pipe_command(&cp, NULL, 0, &out, 0, NULL, 0)) {
+		strbuf_trim(&out);
+		return strbuf_detach(&out, NULL);
+	}
+
+	strbuf_release(&out);
+	error(_("failed to get default branch name"));
+	return NULL;
+}
+
+static int cmd_clone(int argc, const char **argv)
+{
+	const char *branch = NULL;
+	int full_clone = 0;
+	struct option clone_options[] = {
+		OPT_STRING('b', "branch", &branch, N_("<branch>"),
+			   N_("branch to checkout after clone")),
+		OPT_BOOL(0, "full-clone", &full_clone,
+			 N_("when cloning, create full working directory")),
+		OPT_END(),
+	};
+	const char * const clone_usage[] = {
+		N_("scalar clone [<options>] [--] <repo> [<dir>]"),
+		NULL
+	};
+	const char *url;
+	char *enlistment = NULL, *dir = NULL;
+	struct strbuf buf = STRBUF_INIT;
+	int res;
+
+	argc = parse_options(argc, argv, NULL, clone_options, clone_usage, 0);
+
+	if (argc == 2) {
+		url = argv[0];
+		enlistment = xstrdup(argv[1]);
+	} else if (argc == 1) {
+		url = argv[0];
+
+		strbuf_addstr(&buf, url);
+		/* Strip trailing slashes, if any */
+		while (buf.len > 0 && is_dir_sep(buf.buf[buf.len - 1]))
+			strbuf_setlen(&buf, buf.len - 1);
+		/* Strip suffix `.git`, if any */
+		strbuf_strip_suffix(&buf, ".git");
+
+		enlistment = find_last_dir_sep(buf.buf);
+		if (!enlistment) {
+			die(_("cannot deduce worktree name from '%s'"), url);
+		}
+		enlistment = xstrdup(enlistment + 1);
+	} else {
+		usage_msg_opt(_("You must specify a repository to clone."),
+			      clone_usage, clone_options);
+	}
+
+	if (is_directory(enlistment))
+		die(_("directory '%s' exists already"), enlistment);
+
+	dir = xstrfmt("%s/src", enlistment);
+
+	strbuf_reset(&buf);
+	if (branch)
+		strbuf_addf(&buf, "init.defaultBranch=%s", branch);
+	else {
+		char *b = repo_default_branch_name(the_repository, 1);
+		strbuf_addf(&buf, "init.defaultBranch=%s", b);
+		free(b);
+	}
+
+	if ((res = run_git("-c", buf.buf, "init", "--", dir, NULL)))
+		goto cleanup;
+
+	if (chdir(dir) < 0) {
+		res = error_errno(_("could not switch to '%s'"), dir);
+		goto cleanup;
+	}
+
+	setup_git_directory();
+
+	/* common-main already logs `argv` */
+	trace2_def_repo(the_repository);
+
+	if (!branch && !(branch = remote_default_branch(url))) {
+		res = error(_("failed to get default branch for '%s'"), url);
+		goto cleanup;
+	}
+
+	if (set_config("remote.origin.url=%s", url) ||
+	    set_config("remote.origin.fetch="
+		       "+refs/heads/*:refs/remotes/origin/*") ||
+	    set_config("remote.origin.promisor=true") ||
+	    set_config("remote.origin.partialCloneFilter=blob:none")) {
+		res = error(_("could not configure remote in '%s'"), dir);
+		goto cleanup;
+	}
+
+	if (!full_clone &&
+	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
+		goto cleanup;
+
+	if (set_recommended_config())
+		return error(_("could not configure '%s'"), dir);
+
+	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
+		warning(_("partial clone failed; attempting full clone"));
+
+		if (set_config("remote.origin.promisor") ||
+		    set_config("remote.origin.partialCloneFilter")) {
+			res = error(_("could not configure for full clone"));
+			goto cleanup;
+		}
+
+		if ((res = run_git("fetch", "--quiet", "origin", NULL)))
+			goto cleanup;
+	}
+
+	if ((res = set_config("branch.%s.remote=origin", branch)))
+		goto cleanup;
+	if ((res = set_config("branch.%s.merge=refs/heads/%s",
+			      branch, branch)))
+		goto cleanup;
+
+	strbuf_reset(&buf);
+	strbuf_addf(&buf, "origin/%s", branch);
+	res = run_git("checkout", "-f", "-t", buf.buf, NULL);
+	if (res)
+		goto cleanup;
+
+	res = register_dir();
+
+cleanup:
+	free(enlistment);
+	free(dir);
+	strbuf_release(&buf);
+	return res;
+}
+
 static int cmd_list(int argc, const char **argv)
 {
 	if (argc != 1)
@@ -353,6 +553,7 @@ static struct {
 	const char *name;
 	int (*fn)(int, const char **);
 } builtins[] = {
+	{ "clone", cmd_clone },
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f93e3d00efd..d65fb5f1491 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,6 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
+scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -29,12 +30,36 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `list`, all subcommands
-expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone` and `list`, all
+subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
 
+Clone
+~~~~~
+
+clone [<options>] <url> [<enlistment>]::
+	Clones the specified repository, similar to linkgit:git-clone[1]. By
+	default, only commit and tree objects are cloned. Once finished, the
+	worktree is located at `<enlistment>/src`.
++
+The sparse-checkout feature is enabled (except when run with `--full-clone`)
+and the only files present are those in the top-level directory. Use
+`git sparse-checkout set` to expand the set of directories you want to see,
+or `git sparse-checkout disable` to expand to all files (see
+linkgit:git-sparse-checkout[1] for more details). You can explore the
+subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
+
+-b <name>::
+--branch <name>::
+	Instead of checking out the branch pointed to by the cloned
+	repository's HEAD, check out the `<name>` branch instead.
+
+--[no-]full-clone::
+	A sparse-checkout is initialized by default. This behavior can be
+	turned off via `--full-clone`.
+
 List
 ~~~~
 
@@ -64,7 +89,7 @@ unregister [<enlistment>]::
 
 SEE ALSO
 --------
-linkgit:git-maintenance[1].
+linkgit:git-clone[1], linkgit:git-maintenance[1].
 
 Scalar
 ---
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index ef0e8d680d5..295398f62cc 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -10,6 +10,9 @@ PATH=$PWD/..:$PATH
 
 . ../../../t/test-lib.sh
 
+GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab ../cron.txt"
+export GIT_TEST_MAINT_SCHEDULER
+
 test_expect_success 'scalar shows a usage' '
 	test_expect_code 129 scalar -h
 '
@@ -29,4 +32,33 @@ test_expect_success 'scalar unregister' '
 	! grep -F "$(pwd)/vanish/src" scalar.repos
 '
 
+test_expect_success 'set up repository to clone' '
+	test_commit first &&
+	test_commit second &&
+	test_commit third &&
+	git switch -c parallel first &&
+	mkdir -p 1/2 &&
+	test_commit 1/2/3 &&
+	git config uploadPack.allowFilter true &&
+	git config uploadPack.allowAnySHA1InWant true
+'
+
+test_expect_success 'scalar clone' '
+	second=$(git rev-parse --verify second:second.t) &&
+	scalar clone "file://$(pwd)" cloned &&
+	(
+		cd cloned/src &&
+
+		git config --get --global --fixed-value maintenance.repo \
+			"$(pwd)" &&
+
+		test_path_is_missing 1/2 &&
+		test_must_fail git rev-list --missing=print $second &&
+		git rev-list $second &&
+		git cat-file blob $second >actual &&
+		echo "second" >expect &&
+		test_cmp expect actual
+	)
+'
+
 test_done
-- 
gitgitgadget

[PATCH v4 09/15] scalar: teach 'clone' to support the --single-branch option

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

Just like `git clone`, the `scalar clone` command now also offers to
restrict the clone to a single branch.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          |  9 +++++++--
 contrib/scalar/scalar.txt        | 12 +++++++++++-
 contrib/scalar/t/t9099-scalar.sh |  6 +++++-
 3 files changed, 23 insertions(+), 4 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index bf18003b297..7dd1f28948f 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -333,12 +333,15 @@ static char *remote_default_branch(const char *url)
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
-	int full_clone = 0;
+	int full_clone = 0, single_branch = 0;
 	struct option clone_options[] = {
 		OPT_STRING('b', "branch", &branch, N_("<branch>"),
 			   N_("branch to checkout after clone")),
 		OPT_BOOL(0, "full-clone", &full_clone,
 			 N_("when cloning, create full working directory")),
+		OPT_BOOL(0, "single-branch", &single_branch,
+			 N_("only download metadata for the branch that will "
+			    "be checked out")),
 		OPT_END(),
 	};
 	const char * const clone_usage[] = {
@@ -409,7 +412,9 @@ static int cmd_clone(int argc, const char **argv)
 
 	if (set_config("remote.origin.url=%s", url) ||
 	    set_config("remote.origin.fetch="
-		       "+refs/heads/*:refs/remotes/origin/*") ||
+		       "+refs/heads/%s:refs/remotes/origin/%s",
+		       single_branch ? branch : "*",
+		       single_branch ? branch : "*") ||
 	    set_config("remote.origin.promisor=true") ||
 	    set_config("remote.origin.partialCloneFilter=blob:none")) {
 		res = error(_("could not configure remote in '%s'"), dir);
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index d65fb5f1491..46999cf7c84 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -8,7 +8,7 @@ scalar - an opinionated repository management tool
 SYNOPSIS
 --------
 [verse]
-scalar clone [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
+scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<enlistment>]
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
@@ -56,6 +56,16 @@ subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
 	Instead of checking out the branch pointed to by the cloned
 	repository's HEAD, check out the `<name>` branch instead.
 
+--[no-]single-branch::
+	Clone only the history leading to the tip of a single branch, either
+	specified by the `--branch` option or the primary branch remote's
+	`HEAD` points at.
++
+Further fetches into the resulting repository will only update the
+remote-tracking branch for the branch this option was used for the initial
+cloning. If the HEAD at the remote did not point at any branch when
+`--single-branch` clone was made, no remote-tracking branch is created.
+
 --[no-]full-clone::
 	A sparse-checkout is initialized by default. This behavior can be
 	turned off via `--full-clone`.
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 295398f62cc..9a35ab4fde6 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -45,13 +45,17 @@ test_expect_success 'set up repository to clone' '
 
 test_expect_success 'scalar clone' '
 	second=$(git rev-parse --verify second:second.t) &&
-	scalar clone "file://$(pwd)" cloned &&
+	scalar clone "file://$(pwd)" cloned --single-branch &&
 	(
 		cd cloned/src &&
 
 		git config --get --global --fixed-value maintenance.repo \
 			"$(pwd)" &&
 
+		git for-each-ref --format="%(refname)" refs/remotes/origin/ >actual &&
+		echo "refs/remotes/origin/parallel" >expect &&
+		test_cmp expect actual &&
+
 		test_path_is_missing 1/2 &&
 		test_must_fail git rev-list --missing=print $second &&
 		git rev-list $second &&
-- 
gitgitgadget

[PATCH v4 10/15] scalar: implement the `run` command

[Derrick Stolee via GitGitGadget]

From: Derrick Stolee <dstolee@microsoft.com>

Note: this subcommand is provided primarily for backwards-compatibility,
for existing Scalar uses. It is mostly just a shim for `git
maintenance`, mapping task names from the way Scalar called them to the
way Git calls them.

The reason why those names differ? The background maintenance was first
implemented in Scalar, and when it was contributed as a patch series
implementing the `git maintenance` command, reviewers suggested better
names, those suggestions were accepted before the patches were
integrated into core Git.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 64 +++++++++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt | 19 ++++++++++++
 2 files changed, 83 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 7dd1f28948f..8a11f390251 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -490,6 +490,69 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_run(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	struct {
+		const char *arg, *task;
+	} tasks[] = {
+		{ "config", NULL },
+		{ "commit-graph", "commit-graph" },
+		{ "fetch", "prefetch" },
+		{ "loose-objects", "loose-objects" },
+		{ "pack-files", "incremental-repack" },
+		{ NULL, NULL }
+	};
+	struct strbuf buf = STRBUF_INIT;
+	const char *usagestr[] = { NULL, NULL };
+	int i;
+
+	strbuf_addstr(&buf, N_("scalar run <task> [<enlistment>]\nTasks:\n"));
+	for (i = 0; tasks[i].arg; i++)
+		strbuf_addf(&buf, "\t%s\n", tasks[i].arg);
+	usagestr[0] = buf.buf;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usagestr, 0);
+
+	if (!argc)
+		usage_with_options(usagestr, options);
+
+	if (!strcmp("all", argv[0])) {
+		i = -1;
+	} else {
+		for (i = 0; tasks[i].arg && strcmp(tasks[i].arg, argv[0]); i++)
+			; /* keep looking for the task */
+
+		if (i > 0 && !tasks[i].arg) {
+			error(_("no such task: '%s'"), argv[0]);
+			usage_with_options(usagestr, options);
+		}
+	}
+
+	argc--;
+	argv++;
+	setup_enlistment_directory(argc, argv, usagestr, options, NULL);
+	strbuf_release(&buf);
+
+	if (i == 0)
+		return register_dir();
+
+	if (i > 0)
+		return run_git("maintenance", "run",
+			       "--task", tasks[i].task, NULL);
+
+	if (register_dir())
+		return -1;
+	for (i = 1; tasks[i].arg; i++)
+		if (run_git("maintenance", "run",
+			    "--task", tasks[i].task, NULL))
+			return -1;
+	return 0;
+}
+
 static int remove_deleted_enlistment(struct strbuf *path)
 {
 	int res = 0;
@@ -562,6 +625,7 @@ static struct {
 	{ "list", cmd_list },
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
+	{ "run", cmd_run },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 46999cf7c84..f139a14445d 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -12,6 +12,7 @@ scalar clone [--single-branch] [--branch <main-branch>] [--full-clone] <url> [<e
 scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 
 DESCRIPTION
 -----------
@@ -97,6 +98,24 @@ unregister [<enlistment>]::
 	Remove the specified repository from the list of repositories
 	registered with Scalar and stop the scheduled background maintenance.
 
+Run
+~~~
+
+scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]::
+	Run the given maintenance task (or all tasks, if `all` was specified).
+	Except for `all` and `config`, this subcommand simply hands off to
+	linkgit:git-maintenance[1] (mapping `fetch` to `prefetch` and
+	`pack-files` to `incremental-repack`).
++
+These tasks are run automatically as part of the scheduled maintenance,
+as soon as the repository is registered with Scalar. It should therefore
+not be necessary to run this subcommand manually.
++
+The `config` task is specific to Scalar and configures all those
+opinionated default settings that make Git work more efficiently with
+large repositories. As this task is run as part of `scalar clone`
+automatically, explicit invocations of this task are rarely needed.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
-- 
gitgitgadget

[PATCH v4 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

This comes in handy during Scalar upgrades, or when config settings were
messed up by mistake.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
 contrib/scalar/scalar.txt        |  8 ++++
 contrib/scalar/t/t9099-scalar.sh |  8 ++++
 3 files changed, 67 insertions(+), 28 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 8a11f390251..1fff7eb7c12 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
 	return res;
 }
 
-static int set_recommended_config(void)
+static int set_recommended_config(int reconfigure)
 {
 	struct {
 		const char *key;
 		const char *value;
+		int overwrite_on_reconfigure;
 	} config[] = {
-		{ "am.keepCR", "true" },
-		{ "core.FSCache", "true" },
-		{ "core.multiPackIndex", "true" },
-		{ "core.preloadIndex", "true" },
+		/* Required */
+		{ "am.keepCR", "true", 1 },
+		{ "core.FSCache", "true", 1 },
+		{ "core.multiPackIndex", "true", 1 },
+		{ "core.preloadIndex", "true", 1 },
 #ifndef WIN32
-		{ "core.untrackedCache", "true" },
+		{ "core.untrackedCache", "true", 1 },
 #else
 		/*
 		 * Unfortunately, Scalar's Functional Tests demonstrated
@@ -140,28 +142,29 @@ static int set_recommended_config(void)
 		 * Therefore, with a sad heart, we disable this very useful
 		 * feature on Windows.
 		 */
-		{ "core.untrackedCache", "false" },
+		{ "core.untrackedCache", "false", 1 },
 #endif
-		{ "core.logAllRefUpdates", "true" },
-		{ "credential.https://dev.azure.com.useHttpPath", "true" },
-		{ "credential.validate", "false" }, /* GCM4W-only */
-		{ "gc.auto", "0" },
-		{ "gui.GCWarning", "false" },
-		{ "index.threads", "true" },
-		{ "index.version", "4" },
-		{ "merge.stat", "false" },
-		{ "merge.renames", "false" },
-		{ "pack.useBitmaps", "false" },
-		{ "pack.useSparse", "true" },
-		{ "receive.autoGC", "false" },
-		{ "reset.quiet", "true" },
-		{ "feature.manyFiles", "false" },
-		{ "feature.experimental", "false" },
-		{ "fetch.unpackLimit", "1" },
-		{ "fetch.writeCommitGraph", "false" },
+		{ "core.logAllRefUpdates", "true", 1 },
+		{ "credential.https://dev.azure.com.useHttpPath", "true", 1 },
+		{ "credential.validate", "false", 1 }, /* GCM4W-only */
+		{ "gc.auto", "0", 1 },
+		{ "gui.GCWarning", "false", 1 },
+		{ "index.threads", "true", 1 },
+		{ "index.version", "4", 1 },
+		{ "merge.stat", "false", 1 },
+		{ "merge.renames", "false", 1 },
+		{ "pack.useBitmaps", "false", 1 },
+		{ "pack.useSparse", "true", 1 },
+		{ "receive.autoGC", "false", 1 },
+		{ "reset.quiet", "true", 1 },
+		{ "feature.manyFiles", "false", 1 },
+		{ "feature.experimental", "false", 1 },
+		{ "fetch.unpackLimit", "1", 1 },
+		{ "fetch.writeCommitGraph", "false", 1 },
 #ifdef WIN32
-		{ "http.sslBackend", "schannel" },
+		{ "http.sslBackend", "schannel", 1 },
 #endif
+		/* Optional */
 		{ "status.aheadBehind", "false" },
 		{ "commitGraph.generationVersion", "1" },
 		{ "core.autoCRLF", "false" },
@@ -172,7 +175,8 @@ static int set_recommended_config(void)
 	char *value;
 
 	for (i = 0; config[i].key; i++) {
-		if (git_config_get_string(config[i].key, &value)) {
+		if ((reconfigure && config[i].overwrite_on_reconfigure) ||
+		    git_config_get_string(config[i].key, &value)) {
 			trace2_data_string("scalar", the_repository, config[i].key, "created");
 			if (git_config_set_gently(config[i].key,
 						  config[i].value) < 0)
@@ -237,7 +241,7 @@ static int register_dir(void)
 	int res = add_or_remove_enlistment(1);
 
 	if (!res)
-		res = set_recommended_config();
+		res = set_recommended_config(0);
 
 	if (!res)
 		res = toggle_maintenance(1);
@@ -425,7 +429,7 @@ static int cmd_clone(int argc, const char **argv)
 	    (res = run_git("sparse-checkout", "init", "--cone", NULL)))
 		goto cleanup;
 
-	if (set_recommended_config())
+	if (set_recommended_config(0))
 		return error(_("could not configure '%s'"), dir);
 
 	if ((res = run_git("fetch", "--quiet", "origin", NULL))) {
@@ -490,6 +494,24 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int cmd_reconfigure(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar reconfigure [<enlistment>]"),
+		NULL
+	};
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+	return set_recommended_config(1);
+}
+
 static int cmd_run(int argc, const char **argv)
 {
 	struct option options[] = {
@@ -626,6 +648,7 @@ static struct {
 	{ "register", cmd_register },
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
+	{ "reconfigure", cmd_reconfigure },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f139a14445d..f4e4686e8c8 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,6 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
+scalar reconfigure <enlistment>
 
 DESCRIPTION
 -----------
@@ -116,6 +117,13 @@ opinionated default settings that make Git work more efficiently with
 large repositories. As this task is run as part of `scalar clone`
 automatically, explicit invocations of this task are rarely needed.
 
+Reconfigure
+~~~~~~~~~~~
+
+After a Scalar upgrade, or when the configuration of a Scalar enlistment
+was somehow corrupted or changed by mistake, this subcommand allows to
+reconfigure the enlistment.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 9a35ab4fde6..e6d74a06ca0 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -65,4 +65,12 @@ test_expect_success 'scalar clone' '
 	)
 '
 
+test_expect_success 'scalar reconfigure' '
+	git init one/src &&
+	scalar register one &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)"
+'
+
 test_done
-- 
gitgitgadget

[PATCH v4 12/15] scalar: teach 'reconfigure' to optionally handle all registered enlistments

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

After a Scalar upgrade, it can come in really handy if there is an easy
way to reconfigure all Scalar enlistments. This new option offers this
functionality.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 61 ++++++++++++++++++++++++++++++--
 contrib/scalar/scalar.txt        |  9 +++--
 contrib/scalar/t/t9099-scalar.sh |  3 ++
 3 files changed, 67 insertions(+), 6 deletions(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 1fff7eb7c12..67fa5305225 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -494,22 +494,77 @@ static int cmd_register(int argc, const char **argv)
 	return register_dir();
 }
 
+static int get_scalar_repos(const char *key, const char *value, void *data)
+{
+	struct string_list *list = data;
+
+	if (!strcmp(key, "scalar.repo"))
+		string_list_append(list, value);
+
+	return 0;
+}
+
 static int cmd_reconfigure(int argc, const char **argv)
 {
+	int all = 0;
 	struct option options[] = {
+		OPT_BOOL('a', "all", &all,
+			 N_("reconfigure all registered enlistments")),
 		OPT_END(),
 	};
 	const char * const usage[] = {
-		N_("scalar reconfigure [<enlistment>]"),
+		N_("scalar reconfigure [--all | <enlistment>]"),
 		NULL
 	};
+	struct string_list scalar_repos = STRING_LIST_INIT_DUP;
+	int i, res = 0;
+	struct repository r = { NULL };
+	struct strbuf commondir = STRBUF_INIT, gitdir = STRBUF_INIT;
 
 	argc = parse_options(argc, argv, NULL, options,
 			     usage, 0);
 
-	setup_enlistment_directory(argc, argv, usage, options, NULL);
+	if (!all) {
+		setup_enlistment_directory(argc, argv, usage, options, NULL);
+
+		return set_recommended_config(1);
+	}
+
+	if (argc > 0)
+		usage_msg_opt(_("--all or <enlistment>, but not both"),
+			      usage, options);
+
+	git_config(get_scalar_repos, &scalar_repos);
 
-	return set_recommended_config(1);
+	for (i = 0; i < scalar_repos.nr; i++) {
+		const char *dir = scalar_repos.items[i].string;
+
+		strbuf_reset(&commondir);
+		strbuf_reset(&gitdir);
+
+		if (chdir(dir) < 0) {
+			warning_errno(_("could not switch to '%s'"), dir);
+			res = -1;
+		} else if (discover_git_directory(&commondir, &gitdir) < 0) {
+			warning_errno(_("git repository gone in '%s'"), dir);
+			res = -1;
+		} else {
+			git_config_clear();
+
+			the_repository = &r;
+			r.commondir = commondir.buf;
+			r.gitdir = gitdir.buf;
+
+			if (set_recommended_config(1) < 0)
+				res = -1;
+		}
+	}
+
+	string_list_clear(&scalar_repos, 1);
+	strbuf_release(&commondir);
+	strbuf_release(&gitdir);
+
+	return res;
 }
 
 static int cmd_run(int argc, const char **argv)
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index f4e4686e8c8..2fa96fcabc6 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -13,7 +13,7 @@ scalar list
 scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
-scalar reconfigure <enlistment>
+scalar reconfigure [ --all | <enlistment> ]
 
 DESCRIPTION
 -----------
@@ -32,8 +32,8 @@ an existing Git worktree with Scalar whose name is not `src`, the enlistment
 will be identical to the worktree.
 
 The `scalar` command implements various subcommands, and different options
-depending on the subcommand. With the exception of `clone` and `list`, all
-subcommands expect to be run in an enlistment.
+depending on the subcommand. With the exception of `clone`, `list` and
+`reconfigure --all`, all subcommands expect to be run in an enlistment.
 
 COMMANDS
 --------
@@ -124,6 +124,9 @@ After a Scalar upgrade, or when the configuration of a Scalar enlistment
 was somehow corrupted or changed by mistake, this subcommand allows to
 reconfigure the enlistment.
 
+With the `--all` option, all enlistments currently registered with Scalar
+will be reconfigured. Use this option after each Scalar upgrade.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index e6d74a06ca0..5fe7fabd0e5 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -70,6 +70,9 @@ test_expect_success 'scalar reconfigure' '
 	scalar register one &&
 	git -C one/src config core.preloadIndex false &&
 	scalar reconfigure one &&
+	test true = "$(git -C one/src config core.preloadIndex)" &&
+	git -C one/src config core.preloadIndex false &&
+	scalar reconfigure -a &&
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
-- 
gitgitgadget

[PATCH v4 13/15] scalar: implement the `delete` command

[Matthew John Cheetham via GitGitGadget]

From: Matthew John Cheetham <mjcheetham@outlook.com>

Delete an enlistment by first unregistering the repository and then
deleting the enlistment directory (usually the directory containing the
worktree `src/` directory).

On Windows, if the current directory is inside the enlistment's
directory, change to the parent of the enlistment directory, to allow us
to delete the enlistment (directories used by processes e.g. as current
working directories cannot be deleted on Windows).

Co-authored-by: Victoria Dye <vdye@github.com>
Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
 contrib/scalar/scalar.txt        |  8 +++++
 contrib/scalar/t/t9099-scalar.sh |  9 ++++++
 3 files changed, 72 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 67fa5305225..00bedb0bf66 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -8,6 +8,7 @@
 #include "config.h"
 #include "run-command.h"
 #include "refs.h"
+#include "dir.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
 	return NULL;
 }
 
+static int delete_enlistment(struct strbuf *enlistment)
+{
+#ifdef WIN32
+	struct strbuf parent = STRBUF_INIT;
+#endif
+
+	if (unregister_dir())
+		die(_("failed to unregister repository"));
+
+#ifdef WIN32
+	/*
+	 * Change the current directory to one outside of the enlistment so
+	 * that we may delete everything underneath it.
+	 */
+	strbuf_addbuf(&parent, enlistment);
+	strbuf_parent_directory(&parent);
+	if (chdir(parent.buf) < 0)
+		die_errno(_("could not switch to '%s'"), parent.buf);
+	strbuf_release(&parent);
+#endif
+
+	if (remove_dir_recursively(enlistment, 0))
+		die(_("failed to delete enlistment directory"));
+
+	return 0;
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
 	return unregister_dir();
 }
 
+static int cmd_delete(int argc, const char **argv)
+{
+	struct option options[] = {
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar delete <enlistment>"),
+		NULL
+	};
+	struct strbuf enlistment = STRBUF_INIT;
+	int res = 0;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 1)
+		usage_with_options(usage, options);
+
+	setup_enlistment_directory(argc, argv, usage, options, &enlistment);
+
+	res = delete_enlistment(&enlistment);
+	strbuf_release(&enlistment);
+
+	return res;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -704,6 +758,7 @@ static struct {
 	{ "unregister", cmd_unregister },
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
+	{ "delete", cmd_delete },
 	{ NULL, NULL},
 };
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 2fa96fcabc6..6fc57707718 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -14,6 +14,7 @@ scalar register [<enlistment>]
 scalar unregister [<enlistment>]
 scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
 scalar reconfigure [ --all | <enlistment> ]
+scalar delete <enlistment>
 
 DESCRIPTION
 -----------
@@ -127,6 +128,13 @@ reconfigure the enlistment.
 With the `--all` option, all enlistments currently registered with Scalar
 will be reconfigured. Use this option after each Scalar upgrade.
 
+Delete
+~~~~~~
+
+delete <enlistment>::
+	This subcommand lets you delete an existing Scalar enlistment from your
+	local file system, unregistering the repository.
+
 SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
index 5fe7fabd0e5..7e8771d0eff 100755
--- a/contrib/scalar/t/t9099-scalar.sh
+++ b/contrib/scalar/t/t9099-scalar.sh
@@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
 	test true = "$(git -C one/src config core.preloadIndex)"
 '
 
+test_expect_success 'scalar delete without enlistment shows a usage' '
+	test_expect_code 129 scalar delete
+'
+
+test_expect_success 'scalar delete with enlistment' '
+	scalar delete cloned &&
+	test_path_is_missing cloned
+'
+
 test_done
-- 
gitgitgadget

[PATCH v4 14/15] scalar: implement the `version` command

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The .NET version of Scalar has a `version` command. This was necessary
because it was versioned independently of Git.

Since Scalar is now tightly coupled with Git, it does not make sense for
them to show different versions. Therefore, it shows the same output as
`git version`. For backwards-compatibility with the .NET version,
`scalar version` prints to `stderr`, though (`git version` prints to
`stdout` instead).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 00bedb0bf66..728166aa97a 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -9,6 +9,7 @@
 #include "run-command.h"
 #include "refs.h"
 #include "dir.h"
+#include "help.h"
 
 /*
  * Remove the deepest subdirectory in the provided path string. Path must not
@@ -362,6 +363,15 @@ static int delete_enlistment(struct strbuf *enlistment)
 	return 0;
 }
 
+/*
+ * Dummy implementation; Using `get_version_info()` would cause a link error
+ * without this.
+ */
+void load_builtin_commands(const char *prefix, struct cmdnames *cmds)
+{
+	die("not implemented");
+}
+
 static int cmd_clone(int argc, const char **argv)
 {
 	const char *branch = NULL;
@@ -748,6 +758,34 @@ static int cmd_delete(int argc, const char **argv)
 	return res;
 }
 
+static int cmd_version(int argc, const char **argv)
+{
+	int verbose = 0, build_options = 0;
+	struct option options[] = {
+		OPT__VERBOSE(&verbose, N_("include Git version")),
+		OPT_BOOL(0, "build-options", &build_options,
+			 N_("include Git's build options")),
+		OPT_END(),
+	};
+	const char * const usage[] = {
+		N_("scalar verbose [-v | --verbose] [--build-options]"),
+		NULL
+	};
+	struct strbuf buf = STRBUF_INIT;
+
+	argc = parse_options(argc, argv, NULL, options,
+			     usage, 0);
+
+	if (argc != 0)
+		usage_with_options(usage, options);
+
+	get_version_info(&buf, build_options);
+	fprintf(stderr, "%s\n", buf.buf);
+	strbuf_release(&buf);
+
+	return 0;
+}
+
 static struct {
 	const char *name;
 	int (*fn)(int, const char **);
@@ -759,6 +797,7 @@ static struct {
 	{ "run", cmd_run },
 	{ "reconfigure", cmd_reconfigure },
 	{ "delete", cmd_delete },
+	{ "version", cmd_version },
 	{ NULL, NULL},
 };
 
-- 
gitgitgadget

[PATCH v4 15/15] scalar: accept -C and -c options before the subcommand

[Johannes Schindelin via GitGitGadget]

From: Johannes Schindelin <johannes.schindelin@gmx.de>

The `git` executable has these two very useful options:

-C <directory>:
	switch to the specified directory before performing any actions

-c <key>=<value>:
	temporarily configure this setting for the duration of the
	specified scalar subcommand

With this commit, we teach the `scalar` executable the same trick.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 contrib/scalar/scalar.c   | 22 +++++++++++++++++++++-
 contrib/scalar/scalar.txt | 10 ++++++++++
 2 files changed, 31 insertions(+), 1 deletion(-)

diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
index 728166aa97a..76a77ca1ed0 100644
--- a/contrib/scalar/scalar.c
+++ b/contrib/scalar/scalar.c
@@ -806,6 +806,25 @@ int cmd_main(int argc, const char **argv)
 	struct strbuf scalar_usage = STRBUF_INIT;
 	int i;
 
+	while (argc > 1 && *argv[1] == '-') {
+		if (!strcmp(argv[1], "-C")) {
+			if (argc < 3)
+				die(_("-C requires a <directory>"));
+			if (chdir(argv[2]) < 0)
+				die_errno(_("could not change to '%s'"),
+					  argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else if (!strcmp(argv[1], "-c")) {
+			if (argc < 3)
+				die(_("-c requires a <key>=<value> argument"));
+			git_config_push_parameter(argv[2]);
+			argc -= 2;
+			argv += 2;
+		} else
+			break;
+	}
+
 	if (argc > 1) {
 		argv++;
 		argc--;
@@ -816,7 +835,8 @@ int cmd_main(int argc, const char **argv)
 	}
 
 	strbuf_addstr(&scalar_usage,
-		      N_("scalar <command> [<options>]\n\nCommands:\n"));
+		      N_("scalar [-C <directory>] [-c <key>=<value>] "
+			 "<command> [<options>]\n\nCommands:\n"));
 	for (i = 0; builtins[i].name; i++)
 		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
 
diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
index 6fc57707718..3a80f829edc 100644
--- a/contrib/scalar/scalar.txt
+++ b/contrib/scalar/scalar.txt
@@ -36,6 +36,16 @@ The `scalar` command implements various subcommands, and different options
 depending on the subcommand. With the exception of `clone`, `list` and
 `reconfigure --all`, all subcommands expect to be run in an enlistment.
 
+The following options can be specified _before_ the subcommand:
+
+-C <directory>::
+	Before running the subcommand, change the working directory. This
+	option imitates the same option of linkgit:git[1].
+
+-c <key>=<value>::
+	For the duration of running the specified subcommand, configure this
+	setting. This option imitates the same option of linkgit:git[1].
+
 COMMANDS
 --------
 
-- 
gitgitgadget

Re: [PATCH v4 00/15] Upstreaming the Scalar command

[Johannes Schindelin]

Hi,

On Tue, 14 Sep 2021, Johannes Schindelin via GitGitGadget wrote:

> tl;dr: This series contributes the Scalar command to the Git project. This
> command provides an opinionated way to create and configure repositories
> with a focus on very large repositories.
>
> Changes since v3:
>
>  * Moved the "Changes since" section to the top, to make it easier to see
>    what changed.
>  * Reworded the commit message of the first patch.
>  * Removed the [RFC] prefix because I did not hear any objections against
>    putting this into contrib/.

Forgot to say:

  * Sent this out to reflect my latest state before taking off for two
    weeks.

Ciao,
Dscho

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Okay, let's try an analogy.
>
> Imagine that a person is asking for directions to the train station. And
> the other person is replying by asking "did you know that this train
> station was built in 1878? It is actually quite interesting a story...
> [and then goes on to describe the history and what excites them about
> it]". Now, the first person tries again to ask for directions, again does
> not get an answer to that question, and is slowly starting to look at
> their watch. The second person, being completely oblivious to all of this,
> goes on with their wonderful story about the train station and its
> cultural heritage. So the first person walks a bit further to ask a third
> person, but the second person is not done yet and says "but you haven't
> heard me out! That's disrespectful!".
>
> Just imagine for a minute how you would feel if you were the first person.
>
> And that is how I feel asking for reviews about the Scalar patch series
> and then being forcefully dragged into that tangent about the build
> process.

At least to me, how this Makefile for Scalar should interact with
the overall build process does not mesh well with the story about
hwo direction to and history of the station are unrelated.  If we
plan to start from contrib/ and eventually want to make it a part
of the core Git (i.e. "git scalar <subcmd> ..." becomes just like
"git bisect <subcmd> ..."), we would eventually need to see the
recipe needed for including "bisect" and "scalar" work the same
way, no?

I am getting the impression that such a unified build process is
Ævar wants to see at the end, I am not even sure if you do from
the above "analogy".  Cool down a bit, perhaps?

The following assumes that you share the goal of making "git
scalar" just like "git bisect"---another first class citizen of
Git toolbox, the user can choose to use it or the user may not
have a need to interact with it, but it exists there by default
and is not an opt-in add-on component.

I would understand it if your plan is to convert to a unified
build procedure at the very end of the upstreaming process, and
not while you populate contrib/ with more and more scalar stuff,
because the Makefile bits for the entire scalar, while not yet
upstreamed, has already been written as a separate procedure and
having to convert the whole thing upfront before you can start
trickle parts would mean you need to (re)start the process.  And
I would even be sympathetic if you felt it like a distraction.

But at least I view it as a step that needs to happen sometime
between now and at the end.  I do not yet have an opinion on
which one is more pleasant, between (1) having to deal with a
single Makefile that needs to be aware of two different locations
*.[ch] lives in, and (2) having to deal with two Makefiles that
duplicates definitions and risks them needlessly diverging.

I also would understand it if the reason why you want to keep the
top-level Makefile as intact as possible because you sense a high
probability that scalar will stay in contrib/ and even turn out
to be a failure.  Keeping the build procedure separated certainly
will keep it easier to yank it out later.  But I do not think
such a case is quite likely.

Thanks.

Re: [PATCH v4 00/15] Upstreaming the Scalar command

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> Hi,
>
> On Tue, 14 Sep 2021, Johannes Schindelin via GitGitGadget wrote:
>
>> tl;dr: This series contributes the Scalar command to the Git project. This
>> command provides an opinionated way to create and configure repositories
>> with a focus on very large repositories.
>>
>> Changes since v3:
>>
>>  * Moved the "Changes since" section to the top, to make it easier to see
>>    what changed.
>>  * Reworded the commit message of the first patch.
>>  * Removed the [RFC] prefix because I did not hear any objections against
>>    putting this into contrib/.
>
> Forgot to say:
>
>   * Sent this out to reflect my latest state before taking off for two
>     weeks.

Thanks, will queue.

Have fun.

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Tue, Sep 14 2021, Junio C Hamano wrote:

> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>
>> Okay, let's try an analogy.
>>
>> Imagine that a person is asking for directions to the train station. And
>> the other person is replying by asking "did you know that this train
>> station was built in 1878? It is actually quite interesting a story...
>> [and then goes on to describe the history and what excites them about
>> it]". Now, the first person tries again to ask for directions, again does
>> not get an answer to that question, and is slowly starting to look at
>> their watch. The second person, being completely oblivious to all of this,
>> goes on with their wonderful story about the train station and its
>> cultural heritage. So the first person walks a bit further to ask a third
>> person, but the second person is not done yet and says "but you haven't
>> heard me out! That's disrespectful!".
>>
>> Just imagine for a minute how you would feel if you were the first person.
>>
>> And that is how I feel asking for reviews about the Scalar patch series
>> and then being forcefully dragged into that tangent about the build
>> process.
>
> At least to me, how this Makefile for Scalar should interact with
> the overall build process does not mesh well with the story about
> hwo direction to and history of the station are unrelated.  If we
> plan to start from contrib/ and eventually want to make it a part
> of the core Git (i.e. "git scalar <subcmd> ..." becomes just like
> "git bisect <subcmd> ..."), we would eventually need to see the
> recipe needed for including "bisect" and "scalar" work the same
> way, no?
>
> I am getting the impression that such a unified build process is
> Ævar wants to see at the end, I am not even sure if you do from
> the above "analogy".  Cool down a bit, perhaps?
>
> The following assumes that you share the goal of making "git
> scalar" just like "git bisect"---another first class citizen of
> Git toolbox, the user can choose to use it or the user may not
> have a need to interact with it, but it exists there by default
> and is not an opt-in add-on component.
>
> I would understand it if your plan is to convert to a unified
> build procedure at the very end of the upstreaming process, and
> not while you populate contrib/ with more and more scalar stuff,
> because the Makefile bits for the entire scalar, while not yet
> upstreamed, has already been written as a separate procedure and
> having to convert the whole thing upfront before you can start
> trickle parts would mean you need to (re)start the process.  And
> I would even be sympathetic if you felt it like a distraction.
>
> But at least I view it as a step that needs to happen sometime
> between now and at the end.  I do not yet have an opinion on
> which one is more pleasant, between (1) having to deal with a
> single Makefile that needs to be aware of two different locations
> *.[ch] lives in, and (2) having to deal with two Makefiles that
> duplicates definitions and risks them needlessly diverging.

For what it's worth what I had on top of this is not (1) or (2), but a
(0): I.e. there isn't a contrib/scalar anymore, I moved:

    contrib/scalar/scalar.c -> scalar
    contrib/scalar/scalar.txt -> Documentation/scalar.txt
    contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh

We build, test, and otherwise check (e.g. "make check-docs") it by
default, what we don't do is install it unless you ask. You need to run:

    # Or any other install* target
    make install install-doc INSTALL_SCALAR=YesPlease

It could be be kept in contrib/scalar/ even with that sort of approach,
and it would still be simpler than the two-Makefile approach.

But just moving the code, tests and documentation where everything else
lives cuts down an all sorts of special cases, file globs in various
places (e.g. doc lints) will just work and won't need adjustment.

> I also would understand it if the reason why you want to keep the
> top-level Makefile as intact as possible because you sense a high
> probability that scalar will stay in contrib/ and even turn out
> to be a failure.  Keeping the build procedure separated certainly
> will keep it easier to yank it out later.  But I do not think
> such a case is quite likely.

For what it's worth the WIP patch(es) I have on top of it will probably
make such a thing even easier, not that removing it from the tree would
be much of a problem in either case. It's mostly a few lines added to
lists in various places in the Makfile.

If I were to clean this up properly most of the changes would be
teaching the Makefile that it can build N number of named top-level
"special" commands that get dropped into bin/, not just the "git" we
hardcode now.

If you're interested I could try to clean that up and send something on
top, but given the tense-ness of the discussion & unrelated but relevant
patch queue I've got outstanding wouldn't do so otherwise...

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Tue, Sep 14 2021, Johannes Schindelin wrote:

> Hi Ævar,
>
> On Tue, 14 Sep 2021, Ævar Arnfjörð Bjarmason wrote:
>
>> On Mon, Sep 13 2021, Johannes Schindelin wrote:
>>
>> > On Mon, 13 Sep 2021, Ævar Arnfjörð Bjarmason wrote:
>> >
>> >> On Thu, Sep 09 2021, Ævar Arnfjörð Bjarmason wrote:
>> >>
>> >> > In the summary I had on v1->v2 points 1-3 are for v2->v3,
>> >> > respectively, outstanding, addressed, outstanding:
>> >> >
>> >> >     https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
>> >> >
>> >> > In addition the discussion ending here:
>> >> > https://lore.kernel.org/git/nycvar.QRO.7.76.6.2109082112270.55@tvgsbejvaqbjf.bet/
>> >> >
>> >> > For that point: I think it's fair enough not to properly handle the
>> >> > cleanup case in "scalar clone", but perhaps add a note in the
>> >> > commit message that unlike "git clone" this is known not to clean
>> >> > after itself properly on ctrl+c?
>> >>
>> >> Seeing [1] about the planned re-roll I have the above a shot a few
>> >> days ago, see the original discussion at [2] (indirectly linked
>> >> above).
>> >
>> > There is a good reason why I did not engage in that tangent about
>> > deviating from the established `contrib/*/Makefile` paradigm: I find
>> > it particularly unrelated to what this here patch series is trying to
>> > accomplish, and I cannot bring myself to be interested in the proposed
>> > build system changes, either, because I do not see any benefit in the
>> > changes, only downsides.
>> >
>> > I find the distraction unnecessary.
>>
>> Perhaps I'm reading too much between the lines here, so forgive any
>> undue knee-jerk reaction.
>
> Okay, let's try an analogy.
>
> Imagine that a person is asking for directions to the train station. And
> the other person is replying by asking "did you know that this train
> station was built in 1878? It is actually quite interesting a story...
> [and then goes on to describe the history and what excites them about
> it]". Now, the first person tries again to ask for directions, again does
> not get an answer to that question, and is slowly starting to look at
> their watch. The second person, being completely oblivious to all of this,
> goes on with their wonderful story about the train station and its
> cultural heritage. So the first person walks a bit further to ask a third
> person, but the second person is not done yet and says "but you haven't
> heard me out! That's disrespectful!".

To be clear that's not what I said or meant. I wasn't saying you had to
exhaustively hear out some argument or participate in a discussion
you're not interested in.

I am saying that if you're soliciting feedback and you get some, and the
person giving you the feedback sends you pings back, as I did here:

    https://lore.kernel.org/git/871r6axban.fsf@evledraar.gmail.com/
    https://lore.kernel.org/git/87mtoxwt63.fsf@evledraar.gmail.com/
    https://lore.kernel.org/git/877dfupl7o.fsf@evledraar.gmail.com/
    https://lore.kernel.org/git/87r1dydp4m.fsf@evledraar.gmail.com/

That it would be nice to at least reply with some brief comment to the
effect that you're not personally interested in improving this area.

Now, shortly after you send this you re-rolled the v3
(https://lore.kernel.org/git/pull.1005.v4.git.1631630356.gitgitgadget@gmail.com/)
with a note of:

 * Removed the [RFC] prefix because I did not hear any objections against
   putting this into contrib/.

I don't know if you wrote that after this reply, or really didn't see
any of above, but that's a really inaccurate/misleading comment
considering that context.

> Just imagine for a minute how you would feel if you were the first person.
>
> And that is how I feel asking for reviews about the Scalar patch series
> and then being forcefully dragged into that tangent about the build
> process.
>
> I find the well-established paradigm to keep contrib/'s build procedures
> as confined to their own directory as possible the most reasonable way to
> handle the build by virtue of _not_ polluting the top-level Makefile
> unnecessarily. All of your objections strike me simply as personal
> viewpoints, not as technical arguments, and they fail to address this
> "pollution of the top-level Makefile" problem. I therefore strongly
> disagree with your suggestion that the build system should be changed, I
> would even argue that your suggestion should been dismissed on purely
> technical grounds, and I wish you hadn't forced me to say this as
> forcefully.
>
> And even if I looked more favorably on your suggestion to change the build
> procedure, I find this distraction about the build as little constructive
> as the explanations about the train station's history above.

If we're indulging each other, here's my version of that train analogy:

We're in the business of selling a sugary drink that comes in a red can.

We've got a big factory with an attached train station, and all the
trains that move our product are also red.

Now, some of our customers want the new purple colored sugary drink
we've cooked up. A new purple factory's all set up for making them.

But for some reason the part of the business and distribution plans call
for building a new purple train station parallel to our existing one.

All purple sugary drinks are expected to be moved on purple trains, all
our conductors will need some slight re-training and switchover time to
flip-flop between red and purple trains. Issues with purple production
floors and purple workflows will need to be ironed out.

We did a survey of our customers and most of them weren't even aware
that there was such a thing as train. Or perhaps they've seen other
trains, but most haven't seen our trains.

Trains occasionally need field servicing, luckily our fleet of red
trains is set up to carry its own spare parts. Purple trains can only be
serviced with the assistance of a red train.

A product survey asking customers whether their enjoyment of the red or
purple sugary drinks might be impacted by the color of the train they
were shipped on only resulted in puzzled blank looks from the
participants.

The current state of the purple train station and its fleet of purple
trains is that it somewhat works with some hiccups. The currently built
purple train station omitted any sort of train track for leaving the
station though. It should be easy to add one, but...

The manager of the purple train project has been asked whether we can't
just have our red machines in our red factory make the purple sugary
drink, which we can then load on our fleet of red trains. Why do we need
a new parallel rail system when we really care about customers drinking
our delicious sugary drinks?

In case it's not obvious:

   {red,purple} sugary drink  = /usr/bin/git & /usr/bin/scalar
   {red,purple} train station = ./Makefile & ./contrib/scalar/Makefile
   train track for leaving the station = make install (AFAICT your
                                         current patches have no
                                         installation mechanism)

> Those
> suggestions do succeed in derailing the conversation about how Git could
> scale better, how Scalar _does_ teach Git how to scale better, and about
> how to teach Git itself more and more of Scalar's tricks.
>
> If you have ideas how to teach, say, `git clone` to perform a couple of
> Scalar's tricks, by all means, let's hear them, or even better, let's see
> those patches. If you want to change the build system, still, I cannot
> stop you from sending patches to that end to the Git mailing list, but
> please expect me to be uninterested in them in any way, and to prefer to
> spend my efforts to improve Git elsewhere. If you have other ideas how to
> improve on Scalar in a user-perceptible way, however, I am all ears again.
>
> I hope this clarifies it, without the need to read between the lines,
> Johannes

Whatever the end goal of the patches you're sending part of them is
proposing a given approach to go from A to B.

I find it odd to be claiming that the end-state is so important that we
can't spare time to discuss some of the implementation
details.

Particularly in this case, where I've sent a mostly working patch-on-top
which I think should be clear from context I'd be willing to finish up,
so it's not like it's just pointless nitpicking with no end-goal of a
code improvement in sight.

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Derrick Stolee]

On 9/14/2021 2:09 PM, Ævar Arnfjörð Bjarmason wrote:
> 
> On Tue, Sep 14 2021, Junio C Hamano wrote:
> 
>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>>
>> At least to me, how this Makefile for Scalar should interact with
>> the overall build process does not mesh well with the story about
>> hwo direction to and history of the station are unrelated.  If we
>> plan to start from contrib/ and eventually want to make it a part
>> of the core Git (i.e. "git scalar <subcmd> ..." becomes just like
>> "git bisect <subcmd> ..."), we would eventually need to see the
>> recipe needed for including "bisect" and "scalar" work the same
>> way, no?

We should definitely work to find a better way to describe our
vision for how _the ideas in Scalar_ can be adopted into Git proper.

Before this series, we were adding functionality to Git that allowed
Scalar to simplify to just a CLI that configures Git features. This
submission allows that CLI to be available via an opt-in compile flag.
This should allow more users to try out the ideas and perhaps we find
the things that really work for people (and almost more importantly,
the ideas that are _too_ opinionated).

But the way I see Scalar being fully incorporated into Git is not as
a "git scalar <foo>" command or even having "scalar" be included by
default. Instead, perhaps a new builtin would need to be created and
its CLI would need to be presented and reviewed with significant
attention to long-term support in the Git project. Having Scalar as
a testing ground for these ideas seems like a positive way forward.

This is a big reason why we think that contrib/ is a good place for
it to exist.

>> I am getting the impression that such a unified build process is
>> Ævar wants to see at the end, I am not even sure if you do from
>> the above "analogy".  Cool down a bit, perhaps?

I agree that the temperature of this thread has gotten a bit
heated. I think there is something valuable to be gained from
each perspective, but not in a way that either has presented it.

>> The following assumes that you share the goal of making "git
>> scalar" just like "git bisect"---another first class citizen of
>> Git toolbox, the user can choose to use it or the user may not
>> have a need to interact with it, but it exists there by default
>> and is not an opt-in add-on component.
>>
>> I would understand it if your plan is to convert to a unified
>> build procedure at the very end of the upstreaming process, and
>> not while you populate contrib/ with more and more scalar stuff,
>> because the Makefile bits for the entire scalar, while not yet
>> upstreamed, has already been written as a separate procedure and
>> having to convert the whole thing upfront before you can start
>> trickle parts would mean you need to (re)start the process.  And
>> I would even be sympathetic if you felt it like a distraction.
>>
>> But at least I view it as a step that needs to happen sometime
>> between now and at the end.  I do not yet have an opinion on
>> which one is more pleasant, between (1) having to deal with a
>> single Makefile that needs to be aware of two different locations
>> *.[ch] lives in, and (2) having to deal with two Makefiles that
>> duplicates definitions and risks them needlessly diverging.

Since we already need to modify the root Makefile, I think having
the root Makefile add the files from contrib/scalar from an
optional flag is a great way to reduce duplication across multiple
Makefiles while also maintaining the Scalar is compiled optionally.

One big goal is to minimize how often we need to update Scalar. I
can see things like adjusting the recommended config once per
release cycle based on which new features are available. I don't
really want to be spending time updating the Makefile to match a
contribution that was already carefully reviewed and tested. I
also don't want to put the burden of updating contrib/scalar upon
those contributors.

> For what it's worth what I had on top of this is not (1) or (2), but a
> (0): I.e. there isn't a contrib/scalar anymore, I moved:
> 
>     contrib/scalar/scalar.c -> scalar>     contrib/scalar/scalar.txt -> Documentation/scalar.txt
>     contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh
> 
> We build, test, and otherwise check (e.g. "make check-docs") it by
> default, what we don't do is install it unless you ask. You need to run:
> 
>     # Or any other install* target
>     make install install-doc INSTALL_SCALAR=YesPlease
> 
> It could be be kept in contrib/scalar/ even with that sort of approach,
> and it would still be simpler than the two-Makefile approach.

I think keeping it in contrib/scalar is best for now. But I do
agree that a single Makefile has benefits.

One early suggestion from a while back was to modify git.c to
handle the "scalar" executable as well as the "git" executable,
specifically to reduce duplication handling options such as

  -c config.key=value
  -C worktree
  --exec-path

and similar commands. While our duplication of the "-c" option
does add similar code in a second place, these other options
are less critical for Scalar, especially in its current version.
I think refactoring the code in git.c to cater to the "scalar"
executable is at least premature. If we want to pursue these
other options in the future, then that refactoring could happen
as a separate discussion after the rest of the build system and
CLI have been figured out.

_Perhaps_ Johannes still had that level of integration in his
head when responding to the single-Makefile recommendations.

> But just moving the code, tests and documentation where everything else
> lives cuts down an all sorts of special cases, file globs in various
> places (e.g. doc lints) will just work and won't need adjustment.
> 
>> I also would understand it if the reason why you want to keep the
>> top-level Makefile as intact as possible because you sense a high
>> probability that scalar will stay in contrib/ and even turn out
>> to be a failure.  Keeping the build procedure separated certainly
>> will keep it easier to yank it out later.  But I do not think
>> such a case is quite likely.
> 
> For what it's worth the WIP patch(es) I have on top of it will probably
> make such a thing even easier, not that removing it from the tree would
> be much of a problem in either case. It's mostly a few lines added to
> lists in various places in the Makfile.

Do you have a version of these patches available for adaptation
into this series? I'd like to take a look and see what it would
look like to squash them into this series. Forgive me if I just
missed the link. (I see the diff you posted earlier in this thread.)

> If I were to clean this up properly most of the changes would be
> teaching the Makefile that it can build N number of named top-level
> "special" commands that get dropped into bin/, not just the "git" we
> hardcode now.

This is an interesting idea for revamping how adjacent tools are
compiled and shipped with Git from contrib/ (or possibly elsewhere
if we decided to start including more things as "blessed helpers".

As a complete aside: I'm interested in using the sparse-checkout
feature as I work on the Git codebase, just to make sure I hit
pain points before any other user.

This is the best that I could do for my purposes:

$ git sparse-checkout list
.github
Documentation
builtin
compat
contrib/scalar
ewah
git-gui
gitk-git
gitk-gui
gitweb
mergetools
negotiator
perl
po
refs
sha1dc
sha256
t
templates
trace2
xdiff

And 'git status' reports that this includes 97% of the tracked
files. Perhaps there are ways to make this be smaller by having
make skip building things like git-gui if the directory doesn't
exist. Another idea would be to skip any logic around translating
messages if the 'po' directory is missing.

The reason I bring this up is that I'm interested in finding
ways to make our build system be streamlined a bit using the
presence of directories as a way to opt in/out of certain build
outputs. Since Scalar is being added as a new component, this is
a good opportunity to establish a pattern that works for this
effort, too.

Thanks,
-Stolee

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> But at least I view it as a step that needs to happen sometime
> between now and at the end.  I do not yet have an opinion on
> which one is more pleasant, between (1) having to deal with a
> single Makefile that needs to be aware of two different locations
> *.[ch] lives in, and (2) having to deal with two Makefiles that
> duplicates definitions and risks them needlessly diverging.

FWIW, I am leaning towards the latter, with the assumption that this
may take more than a cycle to cook in contrib/.

Adding the Makefile bits to the top-level and keeping the topic in
'next' means all the topics that pass the pipeline will need to be
written in such a way that its Makefile change works well with and
without the unified Makefile bits from this topic, an additional
burden on other topics this topic would impose.

So it is understandable to want to keep the changes to the top-level
Makefile to the minimum, even if it may mean that it requires more
effort in the end to clean things up when the topic graduates.

An alternative would be to bypass the contrib/ phase and start as a
new subcommand that is first-class citizen from day one and let it
spend as much time as it needs to mature.  It would burden the
topics that pass the pipeline while this is cooking the same way as
having a unified build procedure in the top-level Makefile, of
course, though.

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Theodore Ts'o]

On Tue, Sep 14, 2021 at 04:35:42PM -0400, Derrick Stolee wrote:
> 
> We should definitely work to find a better way to describe our
> vision for how _the ideas in Scalar_ can be adopted into Git proper.
> ..
> 
> But the way I see Scalar being fully incorporated into Git is not as
> a "git scalar <foo>" command or even having "scalar" be included by
> default.

I agree 1000%.  It may be convenient for existing scalar users to have
a way they can stick with the CLI that they are used to, but before we
add this functionality to git proper, let's please make sure git users
get a CLI which makes sense and not dictated by history.

We already have enough people who complain that the git interface
design is hard to understand but which can't be changed because we
bias our UI design in favor of existing users as opposed new users.
Given that I'm an existing user, I'm actually not complaining, but I
do recognize the validity of the complaints that for example, git
reset does three different, mostly unrelated things.

Given that the existing scalar users are used to "scalar <foo>" and
not "git scalar <foo">, I'd gently suggest that it's better that there
be an existing compatibility program which translates "scalar <foo>"
to whatever git command(s) makes sense, as opposed to optimizing for
the simplicity of said program so that all forms of "scalar <foo>"
should get translated to "git scalar <foo>".

So for example, if we are inside a mono repro, it would seem to me
that "git commit" should automatically do the right thing, as opposed
to imposing cognitive load on the user to know when they are supposed
to type "git commit ..." versus "git scalar commit ..." versus "scalar
commit".

Please, let's design the UI for scalar integration into git with deep
sympathy for the users, and *not* the convenience of the compatibility
script for the existing scalar CLI users.

						- Ted

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Tue, Sep 14 2021, Derrick Stolee wrote:

> On 9/14/2021 2:09 PM, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Tue, Sep 14 2021, Junio C Hamano wrote:
>> 
>>> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>>>
>>> At least to me, how this Makefile for Scalar should interact with
>>> the overall build process does not mesh well with the story about
>>> hwo direction to and history of the station are unrelated.  If we
>>> plan to start from contrib/ and eventually want to make it a part
>>> of the core Git (i.e. "git scalar <subcmd> ..." becomes just like
>>> "git bisect <subcmd> ..."), we would eventually need to see the
>>> recipe needed for including "bisect" and "scalar" work the same
>>> way, no?
>
> We should definitely work to find a better way to describe our
> vision for how _the ideas in Scalar_ can be adopted into Git proper.
>
> Before this series, we were adding functionality to Git that allowed
> Scalar to simplify to just a CLI that configures Git features. This
> submission allows that CLI to be available via an opt-in compile flag.
> This should allow more users to try out the ideas and perhaps we find
> the things that really work for people (and almost more importantly,
> the ideas that are _too_ opinionated).

Yeah that makes sense. I think it looks like a useful command & it's
already useful to users.

I haven't been suggesting any changes to what gets installed here, FWIW
I think we could be even more aggressive on that front, e.g. shipping it
unconditionally in libexec, maybe with an optional switch for
/usr/bin/scalar, or to ship "scalar" symlinked to "git" and have it
route to the top-level scalar command depending on argv.

I dabbled in that a bit locally, FWIW it seems if anything even easier
to do than the approaches we've discussed so far, but I wanted to focus
on providing the same behavior in terms of build system maintenance.

> But the way I see Scalar being fully incorporated into Git is not as
> a "git scalar <foo>" command or even having "scalar" be included by
> default. Instead, perhaps a new builtin would need to be created and
> its CLI would need to be presented and reviewed with significant
> attention to long-term support in the Git project. Having Scalar as
> a testing ground for these ideas seems like a positive way forward.

*Nod*

> This is a big reason why we think that contrib/ is a good place for
> it to exist.

Here's where you and Johannes lose me. There's some rationale in your
minds for why sticking it in contrib is the obvious way to go. So far
you've been describing how it'll look to users etc, how it's arranged in
our source tree only matters to git.git developers.

I think the actual reason is to carve in advance some subjective
ownership/apartness or whatever for this thing, if that's the case I
think documentation/commit messages would also work.

I really don't care much if something lives in contrib or not in the
abstract, but various integration around builds in Makefile makes that
much easier in practice, and if "make install" looks the same...

>>> I am getting the impression that such a unified build process is
>>> Ævar wants to see at the end, I am not even sure if you do from
>>> the above "analogy".  Cool down a bit, perhaps?
>
> I agree that the temperature of this thread has gotten a bit
> heated. I think there is something valuable to be gained from
> each perspective, but not in a way that either has presented it.

Thanks, hopefully we can keep it jovial going forward. If you've got any
(either on-list or off-list) feedback about how I can improve my side of
that it would be most welcome.

>>> The following assumes that you share the goal of making "git
>>> scalar" just like "git bisect"---another first class citizen of
>>> Git toolbox, the user can choose to use it or the user may not
>>> have a need to interact with it, but it exists there by default
>>> and is not an opt-in add-on component.
>>>
>>> I would understand it if your plan is to convert to a unified
>>> build procedure at the very end of the upstreaming process, and
>>> not while you populate contrib/ with more and more scalar stuff,
>>> because the Makefile bits for the entire scalar, while not yet
>>> upstreamed, has already been written as a separate procedure and
>>> having to convert the whole thing upfront before you can start
>>> trickle parts would mean you need to (re)start the process.  And
>>> I would even be sympathetic if you felt it like a distraction.
>>>
>>> But at least I view it as a step that needs to happen sometime
>>> between now and at the end.  I do not yet have an opinion on
>>> which one is more pleasant, between (1) having to deal with a
>>> single Makefile that needs to be aware of two different locations
>>> *.[ch] lives in, and (2) having to deal with two Makefiles that
>>> duplicates definitions and risks them needlessly diverging.
>
> Since we already need to modify the root Makefile, I think having
> the root Makefile add the files from contrib/scalar from an
> optional flag is a great way to reduce duplication across multiple
> Makefiles while also maintaining the Scalar is compiled optionally.
>
> One big goal is to minimize how often we need to update Scalar. I
> can see things like adjusting the recommended config once per
> release cycle based on which new features are available. I don't
> really want to be spending time updating the Makefile to match a
> contribution that was already carefully reviewed and tested. I
> also don't want to put the burden of updating contrib/scalar upon
> those contributors.

I'd think not having large parts of t/Makefile & Makefile should ease
that maintenance burden for you & others.

>> For what it's worth what I had on top of this is not (1) or (2), but a
>> (0): I.e. there isn't a contrib/scalar anymore, I moved:
>> 
>>     contrib/scalar/scalar.c -> scalar>     contrib/scalar/scalar.txt -> Documentation/scalar.txt
>>     contrib/scalar/t9099-scalar.sh -> t/t9099-scalar.sh
>> 
>> We build, test, and otherwise check (e.g. "make check-docs") it by
>> default, what we don't do is install it unless you ask. You need to run:
>> 
>>     # Or any other install* target
>>     make install install-doc INSTALL_SCALAR=YesPlease
>> 
>> It could be be kept in contrib/scalar/ even with that sort of approach,
>> and it would still be simpler than the two-Makefile approach.
>
> I think keeping it in contrib/scalar is best for now. But I do
> agree that a single Makefile has benefits.

I noted the "why contrib" above.

> One early suggestion from a while back was to modify git.c to
> handle the "scalar" executable as well as the "git" executable,
> specifically to reduce duplication handling options such as
>
>   -c config.key=value
>   -C worktree
>   --exec-path
>
> and similar commands. While our duplication of the "-c" option
> does add similar code in a second place, these other options
> are less critical for Scalar, especially in its current version.
> I think refactoring the code in git.c to cater to the "scalar"
> executable is at least premature. If we want to pursue these
> other options in the future, then that refactoring could happen
> as a separate discussion after the rest of the build system and
> CLI have been figured out.

As noted above that seems like a sensible way forward, I hadn't noticed
how much of git.c's setup was copied into scalar.c.

It seems to me that it wouldn't be that hard, on the order of the
existing setup code or less. I.e. just make "git.c" learn that it may be
running some arbitrary command name, and do some options parsing, but
and finally dispatch to a cmd_scalar(). IOW mostly like a built-in.

> _Perhaps_ Johannes still had that level of integration in his
> head when responding to the single-Makefile recommendations.
>
>> But just moving the code, tests and documentation where everything else
>> lives cuts down an all sorts of special cases, file globs in various
>> places (e.g. doc lints) will just work and won't need adjustment.
>> 
>>> I also would understand it if the reason why you want to keep the
>>> top-level Makefile as intact as possible because you sense a high
>>> probability that scalar will stay in contrib/ and even turn out
>>> to be a failure.  Keeping the build procedure separated certainly
>>> will keep it easier to yank it out later.  But I do not think
>>> such a case is quite likely.
>> 
>> For what it's worth the WIP patch(es) I have on top of it will probably
>> make such a thing even easier, not that removing it from the tree would
>> be much of a problem in either case. It's mostly a few lines added to
>> lists in various places in the Makfile.
>
> Do you have a version of these patches available for adaptation
> into this series? I'd like to take a look and see what it would
> look like to squash them into this series. Forgive me if I just
> missed the link. (I see the diff you posted earlier in this thread.)

I've got it at
https://github.com/avar/git/tree/avar-dscho/scalar-the-beginning-normalize-Makefile

Not very ML-ready, and soft-depends on some other Makefile cleanups I
thought I'd do & still haven't untangled and submitted. Soft-depends as
in this can easily be done on master, but some of the variable names
etc. are quite confusing there.

But you should be able to check it out, it'll build, test and install if
you run "install" with "INSTALL_SCALAR=Y".

You may run into on everly eager new (but unrelated to this, I just
merged it on top) Makefile assertion I'm experimenting with, just
comment out the relevant line in the Makefile if that happens,
i.e. something like this error:

    Makefile:3608: *** "please sort and de-duplicate BUILT_INS_EXTRA!".  Stop.

>> If I were to clean this up properly most of the changes would be
>> teaching the Makefile that it can build N number of named top-level
>> "special" commands that get dropped into bin/, not just the "git" we
>> hardcode now.
>
> This is an interesting idea for revamping how adjacent tools are
> compiled and shipped with Git from contrib/ (or possibly elsewhere
> if we decided to start including more things as "blessed helpers".
>
> As a complete aside: I'm interested in using the sparse-checkout
> feature as I work on the Git codebase, just to make sure I hit
> pain points before any other user.
>
> This is the best that I could do for my purposes:
>
> $ git sparse-checkout list
> .github
> Documentation
> builtin
> compat
> contrib/scalar
> ewah
> git-gui
> gitk-git
> gitk-gui
> gitweb
> mergetools
> negotiator
> perl
> po
> refs
> sha1dc
> sha256
> t
> templates
> trace2
> xdiff
>
> And 'git status' reports that this includes 97% of the tracked
> files. Perhaps there are ways to make this be smaller by having
> make skip building things like git-gui if the directory doesn't
> exist. Another idea would be to skip any logic around translating
> messages if the 'po' directory is missing.

For git-gui in particular NO_TCLTK=Y should do it.

> The reason I bring this up is that I'm interested in finding
> ways to make our build system be streamlined a bit using the
> presence of directories as a way to opt in/out of certain build
> outputs. Since Scalar is being added as a new component, this is
> a good opportunity to establish a pattern that works for this
> effort, too.

Sure, the hard part isn't that you can't grep out nonexisting files or
directories when building where we now use a glob.

It's that everything downstream of that, i.e. tests, installation
etc. is going to have to work properly in the face of arbitrary parts of
what the developer who tested the code expected going missing.

Which is why we've generally carved out very specific things, usually
along the boundaries of installed dependencies.

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

[Ævar Arnfjörð Bjarmason]

On Tue, Sep 14 2021, Johannes Schindelin via GitGitGadget wrote:


> +int cmd_main(int argc, const char **argv)
> +{
> +	struct strbuf scalar_usage = STRBUF_INIT;
> +	int i;
> +
> +	if (argc > 1) {
> +		argv++;
> +		argc--;
> +
> +		for (i = 0; builtins[i].name; i++)
> +			if (!strcmp(builtins[i].name, argv[0]))
> +				return !!builtins[i].fn(argc, argv);
> +	}
> +
> +	strbuf_addstr(&scalar_usage,
> +		      N_("scalar <command> [<options>]\n\nCommands:\n"));
> +	for (i = 0; builtins[i].name; i++)
> +		strbuf_addf(&scalar_usage, "\t%s\n", builtins[i].name);
> +
> +	usage(scalar_usage.buf);
> +}

In 04/15 you continue and use the parse-options.c API, but here it's the
usage.c API, which is generally being phased out. Any reason for the
difference? It's preferrable not to add new usage() users if we can help
it, I think we'll eventually want to remove it.

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> ... the
> usage.c API, which is generally being phased out.

That is news to me.  Any reason why you think so?

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

[Ævar Arnfjörð Bjarmason]

On Fri, Sep 24 2021, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> ... the
>> usage.c API, which is generally being phased out.
>
> That is news to me.  Any reason why you think so?

Perhaps better phrased as "generally going unused where we're using
parse-options.c", although some quick historical trends I ran as an
ad-hoc show usage() standing still since v1.6.0 in number of builtin*.c
files[1], v.s. a 3x growth in usage_with_options() since then[2].

But in this case we're using an ad-hoc parser in cmd_main(), seemingly
because it ends up copy/pasting a very small part of git.ci
functionality over there, which is something that's been discussed as
something we should move over to parse_options() sooner than later.

It looks like a better approach to just use parse_options() consistently
in scalar.c, as e.g. commit-graph.c, stash.c, multi-pack-index.c
etc. that all implement a similar cmd/subcommand pattern do.

The end-state of duplicating the "-C" and "-c" options from git.c can
then easily be handled by parse-options.c, IIRC the stumbling point in
migrating over git.c was some of the statefulness of other parts
potentially needing incremenatl parsing (i.e. via parse_options_step()
and friends).

1. $ parallel "printf "%s: " {} && git grep -l '\busage\(' {} -- 'builtin*.c' | wc -l" ::: v1.{1..9}.0 v2.{0..32}.0
v1.1.0:0
v1.2.0:0
v1.3.0:0
v1.4.0:20
v1.5.0:51
v1.7.0:35
v1.6.0:40
v1.8.0:34
v2.1.0:32
v1.9.0:33
v2.0.0:33
v2.2.0:32
v2.3.0:32
v2.4.0:32
v2.5.0:32
v2.6.0:31
v2.7.0:31
v2.9.0:29
v2.8.0:30
v2.11.0:29
v2.10.0:29
v2.12.0:29
v2.13.0:29
v2.14.0:31
v2.15.0:31
v2.16.0:31
v2.17.0:31
v2.19.0:32
v2.18.0:31
v2.22.0:31
v2.21.0:32
v2.24.0:31
v2.23.0:31
v2.20.0:32
v2.25.0:30
v2.26.0:30
v2.27.0:30
v2.32.0:29
v2.31.0:30
v2.29.0:31
v2.28.0:29
v2.30.0:31

2. $ parallel "printf "%s: " {} && git grep -l '\busage_with_options\(' {} -- 'builtin*.c' | wc -l" ::: v1.{1..9}.0 v2.{0..32}.0
v1.1.0:0
v1.2.0:0
v1.3.0:0
v1.4.0:0
v1.5.0:0
v1.6.0:19
v1.7.0:33
v1.8.0:42
v2.0.0:42
v1.9.0:42
v2.1.0:43
v2.2.0:43
v2.3.0:43
v2.4.0:43
v2.5.0:44
v2.6.0:45
v2.8.0:44
v2.10.0:45
v2.7.0:44
v2.9.0:45
v2.11.0:45
v2.12.0:45
v2.13.0:46
v2.14.0:47
v2.15.0:47
v2.16.0:47
v2.17.0:47
v2.19.0:50
v2.18.0:49
v2.20.0:52
v2.21.0:52
v2.22.0:53
v2.24.0:54
v2.23.0:54
v2.25.0:56
v2.26.0:55
v2.27.0:55
v2.28.0:55
v2.30.0:58
v2.32.0:60
v2.29.0:58
v2.31.0:58

Re: [PATCH v4 01/15] scalar: create a rudimentary executable

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Fri, Sep 24 2021, Junio C Hamano wrote:
>
>> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>>
>>> ... the
>>> usage.c API, which is generally being phased out.
>>
>> That is news to me.  Any reason why you think so?
>
> Perhaps better phrased as "generally going unused where we're using
> parse-options.c", ...

Ah, if it was merely an observation of the general trend, then I
agree.  My reaction was primarily "Oh, why is somebody all of a
sudden setting a project decision unilaterally, especially when even
I do not do so very often myself?  Did I miss recent discussion that
resulted in an update to Documentation/ meant for developers?"

Thanks.

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Elijah Newren]

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
...
> +static int set_recommended_config(void)
> +{
> +       struct {
> +               const char *key;
> +               const char *value;
> +       } config[] = {
> +               { "am.keepCR", "true" },
> +               { "core.FSCache", "true" },
> +               { "core.multiPackIndex", "true" },
> +               { "core.preloadIndex", "true" },
> +#ifndef WIN32
> +               { "core.untrackedCache", "true" },
> +#else
> +               /*
> +                * Unfortunately, Scalar's Functional Tests demonstrated
> +                * that the untracked cache feature is unreliable on Windows
> +                * (which is a bummer because that platform would benefit the
> +                * most from it). For some reason, freshly created files seem
> +                * not to update the directory's `lastModified` time
> +                * immediately, but the untracked cache would need to rely on
> +                * that.
> +                *
> +                * Therefore, with a sad heart, we disable this very useful
> +                * feature on Windows.
> +                */
> +               { "core.untrackedCache", "false" },
> +#endif

Interesting.  (I'm somewhat leery of the untrackedCache just knowing
that it used to operate despite an exponential number of visits to
files (exponential in depth of directories) and getting different
answers with different visits, making me feel like it was black magic
that it ever worked and wondering what kind of corner case issues
still lurk with it.  See e.g.
https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/)

> +               { "core.logAllRefUpdates", "true" },
> +               { "credential.https://dev.azure.com.useHttpPath", "true" },

Not only opinionated, but special configuration for certain sites?
I'm not complaining, just slightly surprised.

> +               { "credential.validate", "false" }, /* GCM4W-only */
> +               { "gc.auto", "0" },
> +               { "gui.GCWarning", "false" },
> +               { "index.threads", "true" },
> +               { "index.version", "4" },

I take it your users don't make use of jgit?  (Users aren't using jgit
directly here, at least not to my knowledge, but multiple gradle
plugins do.)  I tried turning this on a while back, and quickly got
multiple reports of problems because jgit didn't understand the index.
I had to turn it off and send out various PSAs on how to recover.

> +               { "merge.stat", "false" },
> +               { "merge.renames", "false" },

Is this just historical and not needed anymore, is it here just for a
little longer and you are planning on transitioning away from this, or
are you still set on this setting?

> +               { "pack.useBitmaps", "false" },

I don't understand anything bitmap related, but I thought they were
performance related, so I'm surprised by this one.  Is there a reason
for this one?  (Is it handled by maintenance instead?)

> +               { "pack.useSparse", "true" },
> +               { "receive.autoGC", "false" },
> +               { "reset.quiet", "true" },
> +               { "feature.manyFiles", "false" },

If you simply set core.untrackedCache to false _after_ setting
feature.manyFiles to true, would it make sense to switch this?  (Or
does it matter, since you've already individually set all the config
settings that this one would set?)

> +               { "feature.experimental", "false" },
> +               { "fetch.unpackLimit", "1" },
> +               { "fetch.writeCommitGraph", "false" },
> +#ifdef WIN32
> +               { "http.sslBackend", "schannel" },
> +#endif
> +               { "status.aheadBehind", "false" },
> +               { "commitGraph.generationVersion", "1" },
> +               { "core.autoCRLF", "false" },
> +               { "core.safeCRLF", "false" },
> +               { NULL, NULL },
> +       };

Are there easy-ish ways for other groups of users to adopt scalar but
change the list of config settings (e.g. index.version and
merge.renames) in some common way for all those users?

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Elijah Newren]

Sorry, one more thing...

On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget
<gitgitgadget@gmail.com> wrote:
...
> +               /* check if currently in enlistment root with src/ workdir */
> +               strbuf_addstr(&path, "/src/.git");
> +               if (is_git_directory(path.buf)) {

...and...

> +               /* check if currently in workdir */
> +               strbuf_addstr(&path, "/.git");
> +               if (is_git_directory(path.buf)) {

Do these two checks suggest that only a primary worktree can be
enlisted with scalar?  (Is git-worktree generally incompatible?)

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Elijah Newren]

On Mon, Aug 30, 2021 at 2:36 PM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
...
>  COMMANDS
>  --------
>
> +Clone
> +~~~~~
> +
> +clone [<options>] <url> [<enlistment>]::
> +    Clones the specified repository, similar to linkgit:git-clone[1]. By
> +    default, only commit and tree objects are cloned. Once finished, the
> +    worktree is located at `<enlistment>/src`.
> ++
> +The sparse-checkout feature is enabled (except when run with `--full-clone`)
> +and the only files present are those in the top-level directory. Use
> +`git sparse-checkout set` to expand the set of directories you want to see,
> +or `git sparse-checkout disable` to expand to all files (see
> +linkgit:git-sparse-checkout[1] for more details). You can explore the
> +subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.

Should this be `git ls-tree [-r] HEAD`?  Do you expect people to just
add directories that are found immediately under the toplevel, rather
than some that are a bit deeper?

Re: [PATCH v4 11/15] scalar: allow reconfiguring an existing enlistment

[Elijah Newren]

On Tue, Sep 14, 2021 at 7:39 AM Johannes Schindelin via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Johannes Schindelin <johannes.schindelin@gmx.de>
>
> This comes in handy during Scalar upgrades, or when config settings were
> messed up by mistake.
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
>  contrib/scalar/scalar.txt        |  8 ++++
>  contrib/scalar/t/t9099-scalar.sh |  8 ++++
>  3 files changed, 67 insertions(+), 28 deletions(-)
>
> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
> index 8a11f390251..1fff7eb7c12 100644
> --- a/contrib/scalar/scalar.c
> +++ b/contrib/scalar/scalar.c
> @@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
>         return res;
>  }
>
> -static int set_recommended_config(void)
> +static int set_recommended_config(int reconfigure)
>  {
>         struct {
>                 const char *key;
>                 const char *value;
> +               int overwrite_on_reconfigure;
>         } config[] = {
> -               { "am.keepCR", "true" },
> -               { "core.FSCache", "true" },
> -               { "core.multiPackIndex", "true" },
> -               { "core.preloadIndex", "true" },
> +               /* Required */
> +               { "am.keepCR", "true", 1 },
> +               { "core.FSCache", "true", 1 },
> +               { "core.multiPackIndex", "true", 1 },
> +               { "core.preloadIndex", "true", 1 },
>  #ifndef WIN32
> -               { "core.untrackedCache", "true" },
> +               { "core.untrackedCache", "true", 1 },
>  #else
>                 /*
>                  * Unfortunately, Scalar's Functional Tests demonstrated
> @@ -140,28 +142,29 @@ static int set_recommended_config(void)
>                  * Therefore, with a sad heart, we disable this very useful
>                  * feature on Windows.
>                  */
> -               { "core.untrackedCache", "false" },
> +               { "core.untrackedCache", "false", 1 },
>  #endif
> -               { "core.logAllRefUpdates", "true" },
> -               { "credential.https://dev.azure.com.useHttpPath", "true" },
> -               { "credential.validate", "false" }, /* GCM4W-only */
> -               { "gc.auto", "0" },
> -               { "gui.GCWarning", "false" },
> -               { "index.threads", "true" },
> -               { "index.version", "4" },
> -               { "merge.stat", "false" },
> -               { "merge.renames", "false" },
> -               { "pack.useBitmaps", "false" },
> -               { "pack.useSparse", "true" },
> -               { "receive.autoGC", "false" },
> -               { "reset.quiet", "true" },
> -               { "feature.manyFiles", "false" },
> -               { "feature.experimental", "false" },
> -               { "fetch.unpackLimit", "1" },
> -               { "fetch.writeCommitGraph", "false" },
> +               { "core.logAllRefUpdates", "true", 1 },
> +               { "credential.https://dev.azure.com.useHttpPath", "true", 1 },
> +               { "credential.validate", "false", 1 }, /* GCM4W-only */
> +               { "gc.auto", "0", 1 },
> +               { "gui.GCWarning", "false", 1 },
> +               { "index.threads", "true", 1 },
> +               { "index.version", "4", 1 },
> +               { "merge.stat", "false", 1 },
> +               { "merge.renames", "false", 1 },
> +               { "pack.useBitmaps", "false", 1 },
> +               { "pack.useSparse", "true", 1 },
> +               { "receive.autoGC", "false", 1 },
> +               { "reset.quiet", "true", 1 },
> +               { "feature.manyFiles", "false", 1 },
> +               { "feature.experimental", "false", 1 },
> +               { "fetch.unpackLimit", "1", 1 },
> +               { "fetch.writeCommitGraph", "false", 1 },
>  #ifdef WIN32
> -               { "http.sslBackend", "schannel" },
> +               { "http.sslBackend", "schannel", 1 },
>  #endif
> +               /* Optional */
>                 { "status.aheadBehind", "false" },
>                 { "commitGraph.generationVersion", "1" },
>                 { "core.autoCRLF", "false" },

Now you have optional settings...but index.version and merge.renames
aren't among them??  Why are those required?  (...and to go a step
further; should merge.renames even be off in a merge-ort world?)

Re: [PATCH v4 13/15] scalar: implement the `delete` command

[Elijah Newren]

On Tue, Sep 14, 2021 at 7:39 AM Matthew John Cheetham via GitGitGadget
<gitgitgadget@gmail.com> wrote:
>
> From: Matthew John Cheetham <mjcheetham@outlook.com>
>
> Delete an enlistment by first unregistering the repository and then
> deleting the enlistment directory (usually the directory containing the
> worktree `src/` directory).
>
> On Windows, if the current directory is inside the enlistment's
> directory, change to the parent of the enlistment directory, to allow us
> to delete the enlistment (directories used by processes e.g. as current
> working directories cannot be deleted on Windows).

But if the current directory is inside the enlistment's directory,
didn't that happen because the parent process' current directory was
inside the enlistment directory?  Or was there some kind of directory
switching that scalar itself was doing causing it to be inside the
enlistment directory?

If the the current directory was inside the enlistment's directory
because it inherited a parent process' current directory, wouldn't
that also prevent deleting it?  If so, should there be a special check
for that case and pre-emptively returning an error rather than
attempting the recursive directory deletion and just spitting out an
error when it fails?

(Also seems slightly related to
https://github.com/gitgitgadget/git/pull/1037, which I'll submit as
soon as en/removing_untracked_fixes hits next.)

>
> Co-authored-by: Victoria Dye <vdye@github.com>
> Signed-off-by: Matthew John Cheetham <mjcheetham@outlook.com>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  contrib/scalar/scalar.c          | 55 ++++++++++++++++++++++++++++++++
>  contrib/scalar/scalar.txt        |  8 +++++
>  contrib/scalar/t/t9099-scalar.sh |  9 ++++++
>  3 files changed, 72 insertions(+)
>
> diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
> index 67fa5305225..00bedb0bf66 100644
> --- a/contrib/scalar/scalar.c
> +++ b/contrib/scalar/scalar.c
> @@ -8,6 +8,7 @@
>  #include "config.h"
>  #include "run-command.h"
>  #include "refs.h"
> +#include "dir.h"
>
>  /*
>   * Remove the deepest subdirectory in the provided path string. Path must not
> @@ -334,6 +335,33 @@ static char *remote_default_branch(const char *url)
>         return NULL;
>  }
>
> +static int delete_enlistment(struct strbuf *enlistment)
> +{
> +#ifdef WIN32
> +       struct strbuf parent = STRBUF_INIT;
> +#endif
> +
> +       if (unregister_dir())
> +               die(_("failed to unregister repository"));
> +
> +#ifdef WIN32
> +       /*
> +        * Change the current directory to one outside of the enlistment so
> +        * that we may delete everything underneath it.
> +        */
> +       strbuf_addbuf(&parent, enlistment);
> +       strbuf_parent_directory(&parent);
> +       if (chdir(parent.buf) < 0)
> +               die_errno(_("could not switch to '%s'"), parent.buf);
> +       strbuf_release(&parent);
> +#endif
> +
> +       if (remove_dir_recursively(enlistment, 0))
> +               die(_("failed to delete enlistment directory"));
> +
> +       return 0;
> +}
> +
>  static int cmd_clone(int argc, const char **argv)
>  {
>         const char *branch = NULL;
> @@ -694,6 +722,32 @@ static int cmd_unregister(int argc, const char **argv)
>         return unregister_dir();
>  }
>
> +static int cmd_delete(int argc, const char **argv)
> +{
> +       struct option options[] = {
> +               OPT_END(),
> +       };
> +       const char * const usage[] = {
> +               N_("scalar delete <enlistment>"),
> +               NULL
> +       };
> +       struct strbuf enlistment = STRBUF_INIT;
> +       int res = 0;
> +
> +       argc = parse_options(argc, argv, NULL, options,
> +                            usage, 0);
> +
> +       if (argc != 1)
> +               usage_with_options(usage, options);
> +
> +       setup_enlistment_directory(argc, argv, usage, options, &enlistment);
> +
> +       res = delete_enlistment(&enlistment);
> +       strbuf_release(&enlistment);
> +
> +       return res;
> +}
> +
>  static struct {
>         const char *name;
>         int (*fn)(int, const char **);
> @@ -704,6 +758,7 @@ static struct {
>         { "unregister", cmd_unregister },
>         { "run", cmd_run },
>         { "reconfigure", cmd_reconfigure },
> +       { "delete", cmd_delete },
>         { NULL, NULL},
>  };
>
> diff --git a/contrib/scalar/scalar.txt b/contrib/scalar/scalar.txt
> index 2fa96fcabc6..6fc57707718 100644
> --- a/contrib/scalar/scalar.txt
> +++ b/contrib/scalar/scalar.txt
> @@ -14,6 +14,7 @@ scalar register [<enlistment>]
>  scalar unregister [<enlistment>]
>  scalar run ( all | config | commit-graph | fetch | loose-objects | pack-files ) [<enlistment>]
>  scalar reconfigure [ --all | <enlistment> ]
> +scalar delete <enlistment>
>
>  DESCRIPTION
>  -----------
> @@ -127,6 +128,13 @@ reconfigure the enlistment.
>  With the `--all` option, all enlistments currently registered with Scalar
>  will be reconfigured. Use this option after each Scalar upgrade.
>
> +Delete
> +~~~~~~
> +
> +delete <enlistment>::
> +       This subcommand lets you delete an existing Scalar enlistment from your
> +       local file system, unregistering the repository.
> +
>  SEE ALSO
>  --------
>  linkgit:git-clone[1], linkgit:git-maintenance[1].
> diff --git a/contrib/scalar/t/t9099-scalar.sh b/contrib/scalar/t/t9099-scalar.sh
> index 5fe7fabd0e5..7e8771d0eff 100755
> --- a/contrib/scalar/t/t9099-scalar.sh
> +++ b/contrib/scalar/t/t9099-scalar.sh
> @@ -76,4 +76,13 @@ test_expect_success 'scalar reconfigure' '
>         test true = "$(git -C one/src config core.preloadIndex)"
>  '
>
> +test_expect_success 'scalar delete without enlistment shows a usage' '
> +       test_expect_code 129 scalar delete
> +'
> +
> +test_expect_success 'scalar delete with enlistment' '
> +       scalar delete cloned &&
> +       test_path_is_missing cloned
> +'
> +
>  test_done
> --
> gitgitgadget

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Ævar Arnfjörð Bjarmason]

On Mon, Sep 27 2021, Elijah Newren wrote:

> On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> ...
>> +               { "pack.useBitmaps", "false" },
>
> I don't understand anything bitmap related, but I thought they were
> performance related, so I'm surprised by this one.  Is there a reason
> for this one?  (Is it handled by maintenance instead?)

I don't know why Derrick did this, but there's still AFAIK cases where
bitmaps are worse than not in the context of a client (the scalar
use-case), see the rabbit hole starting at:
https://lore.kernel.org/git/878s6nfq54.fsf@evledraar.gmail.com/

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Johannes Schindelin]

Hi Junio,

On Tue, 14 Sep 2021, Junio C Hamano wrote:

> An alternative would be to bypass the contrib/ phase and start as a
> new subcommand that is first-class citizen from day one and let it
> spend as much time as it needs to mature.

I don't think that there is a lot of sense in that. The main benefits of
`scalar` are in the `register` and the `clone` part, and the most natural
end game would hence be for `git init` and `git clone` to sprout new
options to support Scalar's features, in a Git-native way.

As I have explained earlier, the `scalar` command has existing users, and
therefore its command-line interface is not up for discussion (for
example, turning `scalar` into `git scalar` would be a usability
disaster). Scalar's _functionality_, however, should make it into Git
proper. Into existing built-ins, that is.

So I don't think that the contrib/ phase can be by-passed. It would not
make sense to port Scalar to a new builtin. To the contrary,
contrib/scalar/ should be the final destination for the `scalar` command.
And you can't bypass a final destination. That simply makes no sense.

So why bother with contrib/ at all? you may ask. The reason is that it
makes it substantially easier for me to move the features into core Git,
as I can incrementally implement those new options for Git's built-ins,
use them in `contrib/scalar/` instead of duplicating the functionality,
and then make use of Scalar's Functional Test suite for a much more
comprehensive testing (which has served us already really well in the
past). It also doesn't hurt that this way, my day job will be very happy
because Scalar users directly benefit from that work.

Of course, these suggestions to integrate Scalar more into the core part
of Git (missing the point that the final destination for the functionality
is not a new built-in, but rather new options for existing built-ins) made
everything much more cumbersome for me instead, for no gain that would be
apparent to me, impeding on aforementioned ease to move the features into
core Git (which has not happened yet, as a consequence), but hopefully
this will soon be a thing of the past.

So I would like to request that we close the discussion about the question
whether to integrate Scalar more into the top-level Makefile or into
git.c, and instead go ahead with keeping the `scalar` command in
contrib/scalar/. The freed-up time can then be used to focus on the much
more rewarding project of upstreaming Scalar's functionality such as
teaching `git clone` a short-and-sweet option that Just Makes Sense for
large monorepos (i.e. that imitates at least a large part of what `scalar
clone` does right now).

Ciao,
Dscho

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Junio C Hamano]

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:

> I don't think that there is a lot of sense in that. The main benefits of
> `scalar` are in the `register` and the `clone` part, and the most natural
> end game would hence be for `git init` and `git clone` to sprout new
> options to support Scalar's features, in a Git-native way.

Yes, that is even better.  An endgame where everybody benefits
natively would be highly desirable.

Now you are back, do you think we can have the "no more preserve
merges backend" topic graduate to 'master', or do you prefer to cook
it over the cycle (or even two)?

Thanks.

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Johannes Schindelin]

Hi Elijah,

On Mon, 27 Sep 2021, Elijah Newren wrote:

> On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> ...
> > +static int set_recommended_config(void)
> > +{
> > +       struct {
> > +               const char *key;
> > +               const char *value;
> > +       } config[] = {
> > +               { "am.keepCR", "true" },
> > +               { "core.FSCache", "true" },
> > +               { "core.multiPackIndex", "true" },
> > +               { "core.preloadIndex", "true" },
> > +#ifndef WIN32
> > +               { "core.untrackedCache", "true" },
> > +#else
> > +               /*
> > +                * Unfortunately, Scalar's Functional Tests demonstrated
> > +                * that the untracked cache feature is unreliable on Windows
> > +                * (which is a bummer because that platform would benefit the
> > +                * most from it). For some reason, freshly created files seem
> > +                * not to update the directory's `lastModified` time
> > +                * immediately, but the untracked cache would need to rely on
> > +                * that.
> > +                *
> > +                * Therefore, with a sad heart, we disable this very useful
> > +                * feature on Windows.
> > +                */
> > +               { "core.untrackedCache", "false" },
> > +#endif
>
> Interesting.  (I'm somewhat leery of the untrackedCache just knowing
> that it used to operate despite an exponential number of visits to
> files (exponential in depth of directories) and getting different
> answers with different visits, making me feel like it was black magic
> that it ever worked and wondering what kind of corner case issues
> still lurk with it.  See e.g.
> https://lore.kernel.org/git/CABPp-BFiwzzUgiTj_zu+vF5x20L0=1cf25cHwk7KZQj2YkVzXw@mail.gmail.com/)

The implementation of the untracked cache certainly is quite a challenge
to wrap one's head around, for sure. However, it does manage to speed up
operations substantially (when it works).

The real fun starts when you turn on the FSMonitor, though. Then it is
reliable, all of a sudden! The reason seems to be some sort of delayed
lastModified (AKA mtime) evaluation which is somehow triggered by
FSMonitor ;-)

So in microsoft/git, where we include FSMonitor and turn it on as part of
`scalar clone`, we also enable the untracked cache, for noticeably happier
users.

> > +               { "core.logAllRefUpdates", "true" },
> > +               { "credential.https://dev.azure.com.useHttpPath", "true" },
>
> Not only opinionated, but special configuration for certain sites?
> I'm not complaining, just slightly surprised.

Yes. I am not aware of other sites where you would want to use different
credentials depending on the URL path, but Azure DevOps definitely is such
a site, and therefore needs `useHttpPath`. Rather than requiring users to
know this, we set it for them.

> > +               { "credential.validate", "false" }, /* GCM4W-only */
> > +               { "gc.auto", "0" },
> > +               { "gui.GCWarning", "false" },
> > +               { "index.threads", "true" },
> > +               { "index.version", "4" },
>
> I take it your users don't make use of jgit?

Nope ;-) I doubt that the features we use to make Git scalable are
implemented in JGit.

> (Users aren't using jgit directly here, at least not to my knowledge,
> but multiple gradle plugins do.)  I tried turning this on a while back,
> and quickly got multiple reports of problems because jgit didn't
> understand the index. I had to turn it off and send out various PSAs on
> how to recover.

TBH it gives me shivers of dread thinking about large
repositories/worktrees being handled within a Java VM. The amount of,
let's call it "non-canonical" code, required by JGit to make it somewhat
performant, is staggering. Just think about the way you have to emulate
mmap()ing part of a packfile and interpreting it as a packed C struct. I
forgot the details, of course, and I am quite glad that I did.

> > +               { "merge.stat", "false" },
> > +               { "merge.renames", "false" },
>
> Is this just historical and not needed anymore, is it here just for a
> little longer and you are planning on transitioning away from this, or
> are you still set on this setting?

It is here mostly for historical reasons.

> > +               { "pack.useBitmaps", "false" },
>
> I don't understand anything bitmap related, but I thought they were
> performance related, so I'm surprised by this one.  Is there a reason
> for this one?  (Is it handled by maintenance instead?)

Again, this is here for historical reasons. Scalar sets this, and my goal
with this patch series is to port it from .NET to C. So I did not question
the reasoning.

My _guess_ however is that bitmaps really only work well when everything
is in one single pack. Which is rather not the case with Scalar
enlistments: they are way too large to be repacked all the time.

> > +               { "pack.useSparse", "true" },
> > +               { "receive.autoGC", "false" },
> > +               { "reset.quiet", "true" },
> > +               { "feature.manyFiles", "false" },
>
> If you simply set core.untrackedCache to false _after_ setting
> feature.manyFiles to true, would it make sense to switch this?  (Or
> does it matter, since you've already individually set all the config
> settings that this one would set?)

Frankly, I was a bit puzzled why `feature.manyFiles` was set to `false`.
The rationale is explained in
https://github.com/microsoft/scalar/commit/2fc84dba9c95:

	The feature.* config settings change the defaults for some other
	config settings. We already monitor config settings pretty carefully,
	so let's disable these.

As to switching this, it shouldn't matter. The idea of `feature.*` is to
set defaults, but not override any explicitly configured settings.

> > +               { "feature.experimental", "false" },
> > +               { "fetch.unpackLimit", "1" },
> > +               { "fetch.writeCommitGraph", "false" },
> > +#ifdef WIN32
> > +               { "http.sslBackend", "schannel" },
> > +#endif
> > +               { "status.aheadBehind", "false" },
> > +               { "commitGraph.generationVersion", "1" },
> > +               { "core.autoCRLF", "false" },
> > +               { "core.safeCRLF", "false" },
> > +               { NULL, NULL },
> > +       };
>
> Are there easy-ish ways for other groups of users to adopt scalar but
> change the list of config settings (e.g. index.version and
> merge.renames) in some common way for all those users?

Not in Scalar.

I would hope, however, that we could figure out ways to make this more
configurable when re-implementing this functionality in core Git. I have a
couple ideas, but nothing fleshed out, and besides, I do not want to think
too far ahead, I already made that mistake and then got bogged down in
discussions about minimal vs non-minimal changes in the top-level Makefile
;-)

So yeah, good point, but it's probably not a good time yet to discuss this
tangent.

Thank you for reviewing,
Dscho

Re: [PATCH v4 04/15] scalar: 'register' sets recommended config and starts maintenance

[Johannes Schindelin]

Hi Elijah,

On Mon, 27 Sep 2021, Elijah Newren wrote:

> Sorry, one more thing...
>
> On Tue, Sep 14, 2021 at 7:39 AM Derrick Stolee via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> ...
> > +               /* check if currently in enlistment root with src/ workdir */
> > +               strbuf_addstr(&path, "/src/.git");
> > +               if (is_git_directory(path.buf)) {
>
> ...and...
>
> > +               /* check if currently in workdir */
> > +               strbuf_addstr(&path, "/.git");
> > +               if (is_git_directory(path.buf)) {
>
> Do these two checks suggest that only a primary worktree can be
> enlisted with scalar?  (Is git-worktree generally incompatible?)

Good point! I think we'll need to use `is_nonbare_repository_dir()`
instead.

This also has the additional benefit of doing away with quite a bit of
`/.git` appending and undoing it. I.e. it simplifies the code
dramatically.

Ciao,
Dscho

Re: [PATCH 08/15] scalar: implement the `clone` subcommand

[Johannes Schindelin]

Hi Elijah,

On Mon, 27 Sep 2021, Elijah Newren wrote:

> On Mon, Aug 30, 2021 at 2:36 PM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> >
> ...
> >  COMMANDS
> >  --------
> >
> > +Clone
> > +~~~~~
> > +
> > +clone [<options>] <url> [<enlistment>]::
> > +    Clones the specified repository, similar to linkgit:git-clone[1]. By
> > +    default, only commit and tree objects are cloned. Once finished, the
> > +    worktree is located at `<enlistment>/src`.
> > ++
> > +The sparse-checkout feature is enabled (except when run with `--full-clone`)
> > +and the only files present are those in the top-level directory. Use
> > +`git sparse-checkout set` to expand the set of directories you want to see,
> > +or `git sparse-checkout disable` to expand to all files (see
> > +linkgit:git-sparse-checkout[1] for more details). You can explore the
> > +subdirectories outside your sparse-checkout by using `git ls-tree HEAD`.
>
> Should this be `git ls-tree [-r] HEAD`?  Do you expect people to just
> add directories that are found immediately under the toplevel, rather
> than some that are a bit deeper?

I fear that `git ls-tree -r HEAD` in any monorepo might be a bit too
overwhelming for any reader.

But I agree that just looking at HEAD is probably not enough. Maybe we
should use `git ls-tree HEAD[:<dir>]`?

Ciao,
Dscho

Re: [PATCH v4 11/15] scalar: allow reconfiguring an existing enlistment

[Johannes Schindelin]

Hi Elijah,

On Mon, 27 Sep 2021, Elijah Newren wrote:

> On Tue, Sep 14, 2021 at 7:39 AM Johannes Schindelin via GitGitGadget
> <gitgitgadget@gmail.com> wrote:
> >
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > This comes in handy during Scalar upgrades, or when config settings were
> > messed up by mistake.
> >
> > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> > ---
> >  contrib/scalar/scalar.c          | 79 +++++++++++++++++++++-----------
> >  contrib/scalar/scalar.txt        |  8 ++++
> >  contrib/scalar/t/t9099-scalar.sh |  8 ++++
> >  3 files changed, 67 insertions(+), 28 deletions(-)
> >
> > diff --git a/contrib/scalar/scalar.c b/contrib/scalar/scalar.c
> > index 8a11f390251..1fff7eb7c12 100644
> > --- a/contrib/scalar/scalar.c
> > +++ b/contrib/scalar/scalar.c
> > @@ -115,18 +115,20 @@ static int run_git(const char *arg, ...)
> >         return res;
> >  }
> >
> > -static int set_recommended_config(void)
> > +static int set_recommended_config(int reconfigure)
> >  {
> >         struct {
> >                 const char *key;
> >                 const char *value;
> > +               int overwrite_on_reconfigure;
> >         } config[] = {
> > -               { "am.keepCR", "true" },
> > -               { "core.FSCache", "true" },
> > -               { "core.multiPackIndex", "true" },
> > -               { "core.preloadIndex", "true" },
> > +               /* Required */
> > +               { "am.keepCR", "true", 1 },
> > +               { "core.FSCache", "true", 1 },
> > +               { "core.multiPackIndex", "true", 1 },
> > +               { "core.preloadIndex", "true", 1 },
> >  #ifndef WIN32
> > -               { "core.untrackedCache", "true" },
> > +               { "core.untrackedCache", "true", 1 },
> >  #else
> >                 /*
> >                  * Unfortunately, Scalar's Functional Tests demonstrated
> > @@ -140,28 +142,29 @@ static int set_recommended_config(void)
> >                  * Therefore, with a sad heart, we disable this very useful
> >                  * feature on Windows.
> >                  */
> > -               { "core.untrackedCache", "false" },
> > +               { "core.untrackedCache", "false", 1 },
> >  #endif
> > -               { "core.logAllRefUpdates", "true" },
> > -               { "credential.https://dev.azure.com.useHttpPath", "true" },
> > -               { "credential.validate", "false" }, /* GCM4W-only */
> > -               { "gc.auto", "0" },
> > -               { "gui.GCWarning", "false" },
> > -               { "index.threads", "true" },
> > -               { "index.version", "4" },
> > -               { "merge.stat", "false" },
> > -               { "merge.renames", "false" },
> > -               { "pack.useBitmaps", "false" },
> > -               { "pack.useSparse", "true" },
> > -               { "receive.autoGC", "false" },
> > -               { "reset.quiet", "true" },
> > -               { "feature.manyFiles", "false" },
> > -               { "feature.experimental", "false" },
> > -               { "fetch.unpackLimit", "1" },
> > -               { "fetch.writeCommitGraph", "false" },
> > +               { "core.logAllRefUpdates", "true", 1 },
> > +               { "credential.https://dev.azure.com.useHttpPath", "true", 1 },
> > +               { "credential.validate", "false", 1 }, /* GCM4W-only */
> > +               { "gc.auto", "0", 1 },
> > +               { "gui.GCWarning", "false", 1 },
> > +               { "index.threads", "true", 1 },
> > +               { "index.version", "4", 1 },
> > +               { "merge.stat", "false", 1 },
> > +               { "merge.renames", "false", 1 },
> > +               { "pack.useBitmaps", "false", 1 },
> > +               { "pack.useSparse", "true", 1 },
> > +               { "receive.autoGC", "false", 1 },
> > +               { "reset.quiet", "true", 1 },
> > +               { "feature.manyFiles", "false", 1 },
> > +               { "feature.experimental", "false", 1 },
> > +               { "fetch.unpackLimit", "1", 1 },
> > +               { "fetch.writeCommitGraph", "false", 1 },
> >  #ifdef WIN32
> > -               { "http.sslBackend", "schannel" },
> > +               { "http.sslBackend", "schannel", 1 },
> >  #endif
> > +               /* Optional */
> >                 { "status.aheadBehind", "false" },
> >                 { "commitGraph.generationVersion", "1" },
> >                 { "core.autoCRLF", "false" },
>
> Now you have optional settings...but index.version and merge.renames
> aren't among them??  Why are those required?  (...and to go a step
> further; should merge.renames even be off in a merge-ort world?)

I think the idea here is that they are required so that a `scalar
reconfigure` will set them, even if the current enlistment had been
created by a previous Scalar version that had _not_ set those.

And yes, in a merge-ort world, `merge.renames` should probably be forced
to `true`, again because it is in the "Required" section.

Ciao,
Dscho

Re: [PATCH v4 13/15] scalar: implement the `delete` command

[Johannes Schindelin]

Hi Matthew and Elijah,

On Mon, 4 Oct 2021, Matthew Cheetham wrote:

> On 28 Sep 2021, at 7:24 am, Elijah Newren <newren@gmail.com> wrote:
>
> > But if the current directory is inside the enlistment's directory,
> > didn't that happen because the parent process' current directory was
> > inside the enlistment directory?  Or was there some kind of directory
> > switching that scalar itself was doing causing it to be inside the
> > enlistment directory?
>
> Yes to the latter. `setup_enlistment_directory` changes the current
> directory much like `setup_git_directory`.
>
> > If the the current directory was inside the enlistment's directory
> > because it inherited a parent process' current directory, wouldn't
> > that also prevent deleting it?  If so, should there be a special check
> > for that case and pre-emptively returning an error rather than
> > attempting the recursive directory deletion and just spitting out an
> > error when it fails?
>
> You are correct. Speaking to Johannes about this I believe he is looking
> to add a check/error in a new patch series revision.

Indeed, I did notice GGG#1037, and I changed the code so that it detects
whether `scalar delete` was called from within the enlistment, and refuses
to run in that case. Users will have to call `scalar delete <path>` from
outside the enlistment.

Ciao,
Dscho

Re: Train station analogy, was Re: [PATCH v3 00/15] [RFC] Upstreaming the Scalar command

[Ævar Arnfjörð Bjarmason]

On Wed, Oct 06 2021, Johannes Schindelin wrote:

> On Tue, 14 Sep 2021, Junio C Hamano wrote:
>
>> An alternative would be to bypass the contrib/ phase and start as a
>> new subcommand that is first-class citizen from day one and let it
>> spend as much time as it needs to mature.
>
> I don't think that there is a lot of sense in that. The main benefits of
> `scalar` are in the `register` and the `clone` part, and the most natural
> end game would hence be for `git init` and `git clone` to sprout new
> options to support Scalar's features, in a Git-native way.
>
> As I have explained earlier, the `scalar` command has existing users,
> and therefore its command-line interface is not up for discussion (for
> example, turning `scalar` into `git scalar` would be a usability
> disaster). Scalar's _functionality_, however, should make it into Git
> proper. Into existing built-ins, that is.

Given the sub-thread this seems like it's trying to be an indirect reply
to me, but I haven't been advocating changing the UX of "scalar" in any
way, or that we should have a "git scalar".

I have in fact been advocating exactly what you're advocating here. So
we're in violent agreement. Yes the CLI UI shouldn't change, that's the
whole point of having a "scalar" in git.git, not a "git scalar" or
whatever.

I've been suggesting we can simplify the *build system* git.git uses,
something no user will ever see.

> So I don't think that the contrib/ phase can be by-passed. It would not
> make sense to port Scalar to a new builtin. To the contrary,
> contrib/scalar/ should be the final destination for the `scalar` command.
> And you can't bypass a final destination. That simply makes no sense.

I'm right now using a "scalar" installed on my system based on the diff
at the end of this E-Mail that changes the *build system* without any
user-facing changes (see
https://lore.kernel.org/git/87k0jhn0p9.fsf@evledraar.gmail.com/) for a
previous reference.
    
    $ which scalar
    /home/avar/local/bin/scalar
    $ scalar -h 2>&1 | head -n 1
    usage: scalar [-C <directory>] [-c <key>=<value>] <command> [<options>]
    $ man scalar
    [...]
    NAME
           scalar - an opinionated repository management tool
    $ git help scalar
    No manual entry for gitscalar

Except that I think your patches as they stand (correct me if I'm wrong)
don't have any way to install it or its documentation, just to build it
in-place.

> So why bother with contrib/ at all? you may ask. The reason is that it
> makes it substantially easier for me to move the features into core Git,
> as I can incrementally implement those new options for Git's built-ins,
> use them in `contrib/scalar/` instead of duplicating the functionality,
> and then make use of Scalar's Functional Test suite for a much more
> comprehensive testing (which has served us already really well in the
> past). It also doesn't hurt that this way, my day job will be very happy
> because Scalar users directly benefit from that work.

I think all of that summarizes "why have this live in git.git", which I
100% agree with. It's not a counter-argument to a working solution for
simplifying your proposed build system integration.

> Of course, these suggestions to integrate Scalar more into the core part
> of Git (missing the point that the final destination for the functionality
> is not a new built-in, but rather new options for existing built-ins) made
> everything much more cumbersome for me instead, for no gain that would be
> apparent to me, impeding on aforementioned ease to move the features into
> core Git (which has not happened yet, as a consequence), but hopefully
> this will soon be a thing of the past.

The diff below doesn't make scalar a built-in.

> So I would like to request that we close the discussion about the question
> whether to integrate Scalar more into the top-level Makefile or into
> git.c, and instead go ahead with keeping the `scalar` command in
> contrib/scalar/. The freed-up time can then be used to focus on the much
> more rewarding project of upstreaming Scalar's functionality such as
> teaching `git clone` a short-and-sweet option that Just Makes Sense for
> large monorepos (i.e. that imitates at least a large part of what `scalar
> clone` does right now).

The reason I care about this is because duplicating this as a one-off
may be easier for you now, but it creates a lot of maintenance burden
for others down the line.

For example, I've been fixing memory leaks recently and have a
linux-leaks CI job that's now running on GitHub. Scalar would benefit
from having a t/t9099-scalar.sh run as part of that.

Yes you can special-case it somehow and teach ci/, or even "make test"
to do the same thing. But there's a lot of little things both current
and future that are going to be like that. Here's another one:

    $ make -C contrib/scalar scalar
    $ echo bad >cache.h
    $ make -C contrib/scalar scalar
    make: 'scalar' is up to date.

I.e. because it's using its own Makefile it's not getting the very
basics of the dependency graph right. My version?

    $ make scalar
    $ touch cache.h
    $ make scalar
    [... Works exactly as well as doing the same with "git"...]

Yes we could fix that and other things etc, but why not just get all
that for free in around 1/4 lines of Makefile boilerplate (and less than
that in complexity)?

 .gitignore                                   |  1 +
 Documentation/Makefile                       |  3 ++
 {contrib/scalar => Documentation}/scalar.txt |  4 +-
 Makefile                                     | 44 +++++++++++-----
 contrib/scalar/.gitignore                    |  5 --
 contrib/scalar/Makefile                      | 57 --------------------
 contrib/scalar/t/Makefile                    | 78 ----------------------------
 contrib/scalar/scalar.c => scalar.c          |  0
 {contrib/scalar/t => t}/t9099-scalar.sh      |  8 +--
 9 files changed, 37 insertions(+), 163 deletions(-)

[Note that this probably doesn't fully apply to your series/master, not
because of any great textual/semantic conflict, but just because I find
it convenient to stack my WIP/unsubmitted serieses in related areas on
top of one another. This is on top of other Makefile changes I've got
unsubmitted, but it would be easy/trivial to re-apply on master+your
series].

diff --git a/.gitignore b/.gitignore
index 68ecb7f7e9c..ec9771e3532 100644
--- a/.gitignore
+++ b/.gitignore
@@ -217,6 +217,7 @@
 /configure
 /.vscode/
 /tags
+/scalar
 /TAGS
 /cscope*
 /compile_commands.json
diff --git a/Documentation/Makefile b/Documentation/Makefile
index f5605b7767f..57b72e1999a 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -18,6 +18,9 @@ MAN1_TXT += $(filter-out \
 MAN1_TXT += git.txt
 MAN1_TXT += gitk.txt
 MAN1_TXT += gitweb.txt
+ifndef NO_INSTALL_SCALAR_DOC
+MAN1_TXT += scalar.txt
+endif
 
 # man5 / man7 guides (note: new guides should also be added to command-list.txt)
 MAN5_TXT += gitattributes.txt
diff --git a/contrib/scalar/scalar.txt b/Documentation/scalar.txt
similarity index 99%
rename from contrib/scalar/scalar.txt
rename to Documentation/scalar.txt
index 3a80f829edc..f287ab18c31 100644
--- a/contrib/scalar/scalar.txt
+++ b/Documentation/scalar.txt
@@ -149,6 +149,6 @@ SEE ALSO
 --------
 linkgit:git-clone[1], linkgit:git-maintenance[1].
 
-Scalar
+GIT
 ---
-Associated with the linkgit:git[1] suite
+Part of the linkgit:git[1] suite
diff --git a/Makefile b/Makefile
index fd617f8c775..536028922e1 100644
--- a/Makefile
+++ b/Makefile
@@ -602,6 +602,7 @@ LIB_OBJS =
 LIB_OBJS_NO_COMPAT_OBJS =
 OBJECTS =
 PROGRAM_OBJS =
+SCALAR_OBJS =
 SCRIPT_LIB =
 SCRIPT_PERL =
 SCRIPT_PROGRAMS =
@@ -807,6 +808,7 @@ BUILT_INS += $(BUILT_INS_EXTRA)
 
 # what 'all' will build but not install in gitexecdir
 OTHER_PROGRAMS = git$X
+OTHER_PROGRAMS += scalar$X
 ARTIFACTS_TAR += $(OTHER_PROGRAMS)
 
 # what test wrappers are needed and 'install' will install, in bindir
@@ -818,12 +820,18 @@ BINDIR_PROGRAMS_NEED_X += git-upload-pack
 
 BINDIR_PROGRAMS_NO_X += git-cvsserver
 
+ifdef INSTALL_SCALAR
+BINDIR_PROGRAMS_NEED_X += scalar
+endif
 INSTALL_BINDIR_XPROGRAMS += $(patsubst %,%$X,$(BINDIR_PROGRAMS_NEED_X))
 INSTALL_BINDIR_PROGRAMS += $(INSTALL_BINDIR_XPROGRAMS) $(BINDIR_PROGRAMS_NO_X)
 
 # We have bin-wrappers for programs that we don't install
 TEST_BINDIR_PROGRAMS_NEED_X += $(BINDIR_PROGRAMS_NEED_X)
 TEST_BINDIR_PROGRAMS_NEED_X += $(TEST_PROGRAMS_NEED_X)
+ifndef INSTALL_SCALAR
+TEST_BINDIR_PROGRAMS_NEED_X += scalar$X
+endif
 
 TEST_BINDIR_PROGRAMS += $(TEST_BINDIR_PROGRAMS_NEED_X)
 TEST_BINDIR_PROGRAMS += $(BINDIR_PROGRAMS_NO_X)
@@ -2230,7 +2238,7 @@ please_set_SHELL_PATH_to_a_more_modern_shell:
 
 shell_compatibility_test: please_set_SHELL_PATH_to_a_more_modern_shell
 
-strip: $(BIN_PROGRAMS) git$X
+strip: $(BIN_PROGRAMS) git$X scalar$X
 	$(STRIP) $(STRIP_OPTS) $^
 
 ### Flags affecting all rules
@@ -2286,6 +2294,10 @@ git$X: git.o GIT-LDFLAGS $(BUILTIN_OBJS) $(GITLIBS)
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
 		$(filter %.o,$^) $(LIBS)
 
+scalar$X: scalar.o GIT-LDFLAGS $(GITLIBS)
+	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
+		$(filter %.o,$^) $(LIBS)
+
 help.sp help.s help.o: command-list.h
 
 builtin/help.sp builtin/help.s builtin/help.o: config-list.h GIT-PREFIX
@@ -2557,7 +2569,12 @@ GIT_OBJS += git.o
 .PHONY: git-objs
 git-objs: $(GIT_OBJS)
 
+SCALAR_OBJS += scalar.o
+.PHONY: scalar-objs
+scalar-objs: $(SCALAR_OBJS)
+
 OBJECTS += $(GIT_OBJS)
+OBJECTS += $(SCALAR_OBJS)
 OBJECTS += $(PROGRAM_OBJS)
 OBJECTS += $(TEST_OBJS)
 OBJECTS += $(XDIFF_OBJS)
@@ -2565,13 +2582,10 @@ OBJECTS += $(FUZZ_OBJS)
 ifndef NO_CURL
 	OBJECTS += http.o http-walker.o remote-curl.o
 endif
+OBJECTS += $(SCALAR_OBJECTS)
 .PHONY: objects
 objects: $(OBJECTS)
 
-SCALAR_SOURCES := contrib/scalar/scalar.c
-SCALAR_OBJECTS := $(SCALAR_SOURCES:c=o)
-OBJECTS += $(SCALAR_OBJECTS)
-
 dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
 dep_dirs := $(addsuffix .depend,$(sort $(dir $(OBJECTS))))
 
@@ -2710,10 +2724,6 @@ $(REMOTE_CURL_PRIMARY): remote-curl.o http.o http-walker.o GIT-LDFLAGS $(GITLIBS
 	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) \
 		$(CURL_LIBCURL) $(EXPAT_LIBEXPAT) $(LIBS)
 
-contrib/scalar/scalar$X: $(SCALAR_OBJECTS) GIT-LDFLAGS $(GITLIBS)
-	$(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) \
-		$(filter %.o,$^) $(LIBS)
-
 $(LIB_FILE): $(LIB_OBJS) $(LIB_COMPAT_OBJS)
 	$(QUIET_AR)$(RM) $@ && $(AR) $(ARFLAGS) $@ $^
 
@@ -3260,11 +3270,17 @@ endif
 install-gitweb:
 	$(MAKE) -C gitweb install
 
+ifdef INSTALL_SCALAR
+NO_INSTALL_SCALAR_DOC =
+else
+NO_INSTALL_SCALAR_DOC = NoScalarPlease
+endif
+
 install-doc: install-man-perl
-	$(MAKE) -C Documentation install
+	$(MAKE) -C Documentation install NO_INSTALL_SCALAR_DOC=$(NO_INSTALL_SCALAR_DOC)
 
 install-man: install-man-perl
-	$(MAKE) -C Documentation install-man
+	$(MAKE) -C Documentation install-man NO_INSTALL_SCALAR_DOC=$(NO_INSTALL_SCALAR_DOC)
 
 install-man-perl: man-perl
 	$(INSTALL) -d -m 755 '$(DESTDIR_SQ)$(mandir_SQ)/man3'
@@ -3272,13 +3288,13 @@ install-man-perl: man-perl
 	(cd '$(DESTDIR_SQ)$(mandir_SQ)/man3' && umask 022 && $(TAR) xof -)
 
 install-html:
-	$(MAKE) -C Documentation install-html
+	$(MAKE) -C Documentation install-html NO_INSTALL_SCALAR_DOC=$(NO_INSTALL_SCALAR_DOC)
 
 install-info:
-	$(MAKE) -C Documentation install-info
+	$(MAKE) -C Documentation install-info NO_INSTALL_SCALAR_DOC=$(NO_INSTALL_SCALAR_DOC)
 
 install-pdf:
-	$(MAKE) -C Documentation install-pdf
+	$(MAKE) -C Documentation install-pdf NO_INSTALL_SCALAR_DOC=$(NO_INSTALL_SCALAR_DOC)
 
 quick-install-doc:
 	$(MAKE) -C Documentation quick-install
diff --git a/contrib/scalar/.gitignore b/contrib/scalar/.gitignore
deleted file mode 100644
index 00441073f59..00000000000
--- a/contrib/scalar/.gitignore
+++ /dev/null
@@ -1,5 +0,0 @@
-/*.xml
-/*.1
-/*.html
-/*.exe
-/scalar
diff --git a/contrib/scalar/Makefile b/contrib/scalar/Makefile
deleted file mode 100644
index 8620042f281..00000000000
--- a/contrib/scalar/Makefile
+++ /dev/null
@@ -1,57 +0,0 @@
-QUIET_SUBDIR0  = +$(MAKE) -C # space to separate -C and subdir
-QUIET_SUBDIR1  =
-
-ifneq ($(findstring s,$(MAKEFLAGS)),s)
-ifndef V
-	QUIET_GEN      = @echo '   ' GEN $@;
-	QUIET_SUBDIR0