[PATCH 0/4] Additional FAQ entries

[PATCH 0/4] Additional FAQ entries

[brian m. carlson]

This series introduces some additional Git FAQ entries on various
topics.  They are all things I've seen in my professional life or on
Stack Overflow, and I tire of repeating myself, so I've written
documentation.

The first covers monorepos.  It tries to strike a neutral tone and
indicate that we're improving support for them, but it does note that
many large users still run into scale problems, especially at a point
where it's inconvenient to change[0].  I do frequently see these problems
on some scale, so my hope is that we can make users consider this case
carefully.  I have mentioned only scalability features which are
released and reasonably mature, but if I've missed any that I should
have mentioned, please let me know, since I don't always follow the list
as closely as I should.

The second covers proxies, in the generic sense of that word.  It
instructs users how to use an HTTP proxy intentionally, and it also
notes that any proxy which buffers or tampers with the data will break
Git.  I frequently see users with TLS middleboxes or antivirus software
other than the Windows default with breakage on Stack Overflow[1], and
we see them occasionally on the list as well.  Because these tools are
known to cause both functionality and security problems, both for Git
and in general[2][3][4], we explicitly recommend against them here.  I
am happy to provide additional citations to Project Zero, various bug
reports, security experts, and the cryptographic literature if folks
have doubts about whether this is the right approach.

The third should be relatively uncontroversial.  We discuss using eol
attributes in gitattributes, which we had neglected to cover before, and
illustrate their utility.  In addition, we fix a reference.

The fourth and final patch covers syncing working trees.  An earlier
version of this patch has come up on the list before and I had never
sent a rerolled version.  I realize that we don't love that people want
to do this, but users are currently using cloud syncing services and
experiencing data loss, so I figured we'd better tell them how to do it
as safely as they can so they stop losing data.

I realize that folks are busy with the Contributor Summit and other
things, and so of course reviews may be delayed.  I've tried to pick
people who I think will have relevant interest in each of these areas,
but of course feedback from anyone is welcome.

[0] For instance, the Windows repo.
[1] In addition to users for whom SSH "magically" works when HTTPS does
    not, many of which are likely the same issue.
[2] https://robert.ocallahan.org/2017/01/disable-your-antivirus-software-except.html
[3] https://twitter.com/parityzero/status/826489070310207489
[4] https://groups.google.com/g/mozilla.dev.platform/c/Bh8U0DLHrCc

brian m. carlson (4):
  gitfaq: add advice on monorepos
  gitfaq: add documentation on proxies
  gitfaq: give advice on using eol attribute in gitattributes
  gitfaq: add entry about syncing working trees

 Documentation/gitfaq.txt | 107 ++++++++++++++++++++++++++++++++++++---
 1 file changed, 101 insertions(+), 6 deletions(-)

[PATCH 1/4] gitfaq: add advice on monorepos

[brian m. carlson]

Many projects around the world have chosen monorepos, and active
development on Git is ongoing to support them better.  However, as
projects using monorepos grow, they often find various performance
and scalability problems that are unpleasant to deal with.

Add a FAQ entry to note that while Git is attempting improvements in
this area, it is not uncommon to see performance problems that
necessitate the use of partial or shallow clone, sparse checkout, or the
like, and that if users wish to avoid these problems, avoiding a
monorepo may be best.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 8c1f2d5675..946691c153 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+Design
+------
+
+[[monorepos]]
+Should we use a monorepo or many individual repos?::
+	This is a decision that is typically made based on an organization's needs and
+	desires for their projects.  Git has several features, such as shallow clone,
+	partial clone, and sparse checkout to make working with large repositories
+	easier, and there is active development on making the monorepo experience
+	better.
++
+However, at a certain size, the performance of a monorepo will likely become
+unacceptable _unless_ you use these features.  If you choose to start with a
+monorepo and continue to grow, you may end up unhappy with the performance
+characteristics at a point where making a change is difficult.  The performance
+of using many smaller repositories will almost always be much better and will
+generally not necessitate the use of these more advanced features.  If you are
+concerned about future performance of your repository and related tools, you may
+wish to avoid a monorepo.
++
+Ultimately, you should make a decision fully informed about the potential
+benefits and downsides, including the capabilities, performance, and future
+requirements for your repository and related tools, including your hosting
+platform, build tools, and other programs you typically use as part of your
+workflow.
+
 Merging and Rebasing
 --------------------
 

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[Bagas Sanjaya]

On 20/10/21 08.06, brian m. carlson wrote:
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.
> +
>   Merging and Rebasing
>   --------------------
>   
> 

It seems like recommending to split repo, right?

Ultimately, if people choose split repo instead of monorepo, it will 
only delay the necessity to use advanced features (partial/shallow 
clones, sparse checkout, etc.) when the repos become large.

For balanced view, we should describe benefits and drawbacks of both 
monorepo and split repos.

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[Ævar Arnfjörð Bjarmason]

On Wed, Oct 20 2021, brian m. carlson wrote:

> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.

In the context of git development we're typically talking about really
big repos when we're talking about monorepos, saying "monorepo"
communicates among other things that the user of that pattern is
unwilling to use splitting up as a way to address any scalability issues
they may have.

But a monorepo doesn't really say anything about size per-se, and it
would be confusing to conflate the two in a FAQ. I may be wrong, perhaps
the term has really come to exclusively refer to colossal size, but I
haven't seen or heard it exclusively (or even mainly) used like that

My understanding of what a monorepo in the context of software
development is a collection of all your "main" code and all its
dependencies in one repository, such that a person working on it rarely
or never has to worry about N=N dependencies between different pieces of
that collection, they'll all move in unison. You are able to atomically
change a function and all its users.

I think people have a different understanding of "all its
dependencies". Some monorepo users really mean it and try to e.g. fold
their system configuration system that might manage files in /etc in
with their monorepo, others might have "another monorepo" for that
software, etc.

I bet that the vast majority of monorepo users are never going to
experience scaling problems, e.g. having your laptop dotfiles and
automation of /etc in one repo is a "monorepo", and most companies/teams
that use monorepos I'd bet are in the long tail of size
distribution. They're not going to grow to the size of a MS's, FB's
etc. monorepo, but they might benefit (or not) from the monorepo
/workflow/.

Anyway, all of the above can be read as a suggestion that we should
split any discussion of "large repo [that runs into scaling issues]"
from "monorepo", the latter should of course make a passing reference to
scaling (as the pattern will lead to that sooner than not), but IMO not
conflate the two.

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 3902 bytes --]

On 2021-10-20 at 10:54:47, Ævar Arnfjörð Bjarmason wrote:
> 
> On Wed, Oct 20 2021, brian m. carlson wrote:
> 
> > +[[monorepos]]
> > +Should we use a monorepo or many individual repos?::
> > +	This is a decision that is typically made based on an organization's needs and
> > +	desires for their projects.  Git has several features, such as shallow clone,
> > +	partial clone, and sparse checkout to make working with large repositories
> > +	easier, and there is active development on making the monorepo experience
> > +	better.
> > ++
> > +However, at a certain size, the performance of a monorepo will likely become
> > +unacceptable _unless_ you use these features.  If you choose to start with a
> > +monorepo and continue to grow, you may end up unhappy with the performance
> > +characteristics at a point where making a change is difficult.  The performance
> > +of using many smaller repositories will almost always be much better and will
> > +generally not necessitate the use of these more advanced features.  If you are
> > +concerned about future performance of your repository and related tools, you may
> > +wish to avoid a monorepo.
> > ++
> > +Ultimately, you should make a decision fully informed about the potential
> > +benefits and downsides, including the capabilities, performance, and future
> > +requirements for your repository and related tools, including your hosting
> > +platform, build tools, and other programs you typically use as part of your
> > +workflow.
> 
> In the context of git development we're typically talking about really
> big repos when we're talking about monorepos, saying "monorepo"
> communicates among other things that the user of that pattern is
> unwilling to use splitting up as a way to address any scalability issues
> they may have.
> 
> But a monorepo doesn't really say anything about size per-se, and it
> would be confusing to conflate the two in a FAQ. I may be wrong, perhaps
> the term has really come to exclusively refer to colossal size, but I
> haven't seen or heard it exclusively (or even mainly) used like that

I routinely hear "monorepo" used to imply repositories of specifically
large size.  However, I'm happy to rephrase to make it clearer.

> I bet that the vast majority of monorepo users are never going to
> experience scaling problems, e.g. having your laptop dotfiles and
> automation of /etc in one repo is a "monorepo", and most companies/teams
> that use monorepos I'd bet are in the long tail of size
> distribution. They're not going to grow to the size of a MS's, FB's
> etc. monorepo, but they might benefit (or not) from the monorepo
> /workflow/.

I almost never hear individuals refer to such a configuration as a
monorepo.  Technically, it is one, yes, but I almost always hear it in
the context of an organization's repository covering all of their
services or the entirety of one major project.

I will point out that I personally would run into scaling issues if I
put all of my projects in the same repository.  I have many projects,
and that would quickly become unsustainable, since the resources I have
at my disposal are more limited than most organizations.

> Anyway, all of the above can be read as a suggestion that we should
> split any discussion of "large repo [that runs into scaling issues]"
> from "monorepo", the latter should of course make a passing reference to
> scaling (as the pattern will lead to that sooner than not), but IMO not
> conflate the two.

I'm happy to clarify, but I think we need to mention the word "monorepo"
specifically because (a) that's the term that's commonly used for this
approach and (b) that approach is one that tends to lead to
significantly greater growth in a single repository leading to scale
problems.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[Johannes Schindelin]

Hi brian,

thank you for this excellent idea to talk about monorepos.

On Wed, 20 Oct 2021, brian m. carlson wrote:

> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 8c1f2d5675..946691c153 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>
> +Design
> +------
> +
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories

May I request taking out shallow clones? A user new to Git might think
that shallow clones are a sane way to clone a large repository. In
practice, this only makes sense for "throw-away" clones, though. As soon
as you fetch in such clones, performance will be so horrible that it is
frequently a better idea to start with a partial clone instead.

At the same time, I would like to swap in "sparse index" for "shallow
clone" because it _does_ have the best potential of all currently
discussed new features to improve performance with monorepos.

> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a

I would like to add a plug for Scalar here. Maybe we can link to this
"opinionated tool based on Git" here? I wouldn't ask if I didn't _know_
that it helps monorepo users out there.

> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.

I wish we had a good article to link to, here. Yes, it is a decision that
should be fully informed, and yes, this FAQ entry is not the place for a
thorough discussion of monorepos and how Git can be asked to handle them
more efficiently.

Do you know of any good resource that we could use here?

Thanks,
Dscho

> +
>  Merging and Rebasing
>  --------------------
>
>

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[Philip Oakley]

On 20/10/2021 02:06, brian m. carlson wrote:
> Many projects around the world have chosen monorepos, and active
> development on Git is ongoing to support them better.  However, as
> projects using monorepos grow, they often find various performance
> and scalability problems that are unpleasant to deal with.
>
> Add a FAQ entry to note that while Git is attempting improvements in
> this area, it is not uncommon to see performance problems that
> necessitate the use of partial or shallow clone, sparse checkout, or the
> like, and that if users wish to avoid these problems, avoiding a
> monorepo may be best.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 26 ++++++++++++++++++++++++++
>  1 file changed, 26 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 8c1f2d5675..946691c153 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,32 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>  
> +Design
> +------
> +
> +[[monorepos]]
> +Should we use a monorepo or many individual repos?::
> +	This is a decision that is typically made based on an organization's needs and
> +	desires for their projects.  Git has several features, such as shallow clone,
> +	partial clone, and sparse checkout to make working with large repositories
> +	easier, and there is active development on making the monorepo experience
> +	better.
> ++
> +However, at a certain size, the performance of a monorepo will likely become
> +unacceptable _unless_ you use these features.  If you choose to start with a
> +monorepo and continue to grow, you may end up unhappy with the performance
> +characteristics at a point where making a change is difficult.  The performance
> +of using many smaller repositories will almost always be much better and will
> +generally not necessitate the use of these more advanced features.  If you are
> +concerned about future performance of your repository and related tools, you may
> +wish to avoid a monorepo.
> ++
> +Ultimately, you should make a decision fully informed about the potential
> +benefits and downsides, including the capabilities, performance, and future
> +requirements for your repository and related tools, including your hosting
> +platform, build tools, and other programs you typically use as part of your
> +workflow.
> +

Does this need some comparison, or link, with sub-module methods and
issues? Such as the nested sub-module problem, the distinction between
active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?

As an aside, I don't think we provide any background to the Git
philosophy that frames some of these issues.

-- 
Philip

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On 2021-10-20 at 14:11:09, Philip Oakley wrote:
> Does this need some comparison, or link, with sub-module methods and
> issues? Such as the nested sub-module problem, the distinction between
> active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?

I don't think it does.  Some projects choose to use many repositories
with submodules, and some use many repositories without submodules.  At
work, we do the latter, and it tends to work just fine.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 1/4] gitfaq: add advice on monorepos

[Philip Oakley]

Hi Brian,
On 20/10/2021 23:22, brian m. carlson wrote:
> On 2021-10-20 at 14:11:09, Philip Oakley wrote:
>> Does this need some comparison, or link, with sub-module methods and
>> issues? Such as the nested sub-module problem, the distinction between
>> active sub-modules and quiescent sub-modules (e.g. libraries Vx.y.z)?
> I don't think it does.  Some projects choose to use many repositories
> with submodules, and some use many repositories without submodules.  At
> work, we do the latter, and it tends to work just fine.

To clarify, my comment was with regard to the complementary discussions
about _choice_ of repo types, rather than mono-repo and other
post-choice issues. Possibly, part of such a discussion on choice of
repo-type, could include the potential slippery slope between the
different uses of sub-modules.

I feel a lot of the difficult sub-module discussion are because we don't
have a common terminology for the different (mental) models of
sub-module use, their benefits and problems.
--
Philip

[PATCH 2/4] gitfaq: add documentation on proxies

[brian m. carlson]

Many corporate environments and local systems have proxies in use.  Note
the situations in which proxies can be used and how to configure them.
At the same time, note what standards a proxy must follow to work with
Git.  Explicitly call out certain classes that are known to routinely
have problems reported various places online, including in the Git for
Windows issue tracker and on Stack Overflow, and recommend against the
use of such software.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 946691c153..abc0f62e6c 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -241,6 +241,24 @@ How do I know if I want to do a fetch or a pull?::
 	ignore the upstream changes.  A pull consists of a fetch followed
 	immediately by either a merge or rebase.  See linkgit:git-pull[1].
 
+[[proxy]]
+Can I use a proxy with Git?::
+	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
+	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
+	it also can be configured with `http.proxy` and similar options for HTTPS (see
+	linkgit:git-config[1]).  The `http.proxy` and related options can be
+	customized on a per-URL pattern basis.  In addition, Git can in theory
+	function normally with transparent proxies that exist on the network.
++
+However, note that for Git to work properly, the proxy must be completely
+transparent.  The proxy cannot modify, tamper with, change, or buffer the
+connection in any way, or Git will almost certainly fail to work.  Note that
+many proxies, including many TLS middleboxes, Windows antivirus and firewall
+programs other than Windows Defender and Windows Firewall, and filtering proxies
+fail to meet this standard, and as a result end up breaking Git.  Because of the
+many reports of problems, we recommend against the use of these classes of
+software and devices.
+
 Design
 ------
 

Re: [PATCH 2/4] gitfaq: add documentation on proxies

[Johannes Schindelin]

Hi brian,

On Wed, 20 Oct 2021, brian m. carlson wrote:

> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.

I fully agree with this patch, therefore: ACK!

Ciao,
Dscho

Re: [PATCH 2/4] gitfaq: add documentation on proxies

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 867 bytes --]

On 2021-10-20 at 11:57:42, Johannes Schindelin wrote:
> Hi brian,
> 
> On Wed, 20 Oct 2021, brian m. carlson wrote:
> 
> > Many corporate environments and local systems have proxies in use.  Note
> > the situations in which proxies can be used and how to configure them.
> > At the same time, note what standards a proxy must follow to work with
> > Git.  Explicitly call out certain classes that are known to routinely
> > have problems reported various places online, including in the Git for
> > Windows issue tracker and on Stack Overflow, and recommend against the
> > use of such software.
> 
> I fully agree with this patch, therefore: ACK!

I CC'd you specifically because of this patch, and I'm glad to hear you
agree, even if I'm sorry you have to deal with these reports.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 2/4] gitfaq: add documentation on proxies

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> Many corporate environments and local systems have proxies in use.  Note
> the situations in which proxies can be used and how to configure them.
> At the same time, note what standards a proxy must follow to work with
> Git.  Explicitly call out certain classes that are known to routinely
> have problems reported various places online, including in the Git for
> Windows issue tracker and on Stack Overflow, and recommend against the
> use of such software.

The new section only talks about http(s), but I recall Peff recently
helped folks with broken invocation of socat, a relay that does not
handle severing connection only in one direction very well.

I wonder if it fits in this section.

> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
>  Documentation/gitfaq.txt | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> index 946691c153..abc0f62e6c 100644
> --- a/Documentation/gitfaq.txt
> +++ b/Documentation/gitfaq.txt
> @@ -241,6 +241,24 @@ How do I know if I want to do a fetch or a pull?::
>  	ignore the upstream changes.  A pull consists of a fetch followed
>  	immediately by either a merge or rebase.  See linkgit:git-pull[1].
>  
> +[[proxy]]
> +Can I use a proxy with Git?::
> +	Yes, Git supports the use of proxies.  Git honors the standard `http_proxy`,
> +	`https_proxy`, and `no_proxy` environment variables commonly used on Unix, and
> +	it also can be configured with `http.proxy` and similar options for HTTPS (see
> +	linkgit:git-config[1]).  The `http.proxy` and related options can be
> +	customized on a per-URL pattern basis.  In addition, Git can in theory
> +	function normally with transparent proxies that exist on the network.
> ++
> +However, note that for Git to work properly, the proxy must be completely
> +transparent.  The proxy cannot modify, tamper with, change, or buffer the
> +connection in any way, or Git will almost certainly fail to work.  Note that
> +many proxies, including many TLS middleboxes, Windows antivirus and firewall
> +programs other than Windows Defender and Windows Firewall, and filtering proxies
> +fail to meet this standard, and as a result end up breaking Git.  Because of the
> +many reports of problems, we recommend against the use of these classes of
> +software and devices.
> +
>  Design
>  ------
>  

Re: [PATCH 2/4] gitfaq: add documentation on proxies

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On 2021-10-20 at 14:48:04, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > Many corporate environments and local systems have proxies in use.  Note
> > the situations in which proxies can be used and how to configure them.
> > At the same time, note what standards a proxy must follow to work with
> > Git.  Explicitly call out certain classes that are known to routinely
> > have problems reported various places online, including in the Git for
> > Windows issue tracker and on Stack Overflow, and recommend against the
> > use of such software.
> 
> The new section only talks about http(s), but I recall Peff recently
> helped folks with broken invocation of socat, a relay that does not
> handle severing connection only in one direction very well.
> 
> I wonder if it fits in this section.

I think it does, and I'll add it in a reroll.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

[PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[brian m. carlson]

In the FAQ, we tell people how to use the text attribute, but we fail to
explain what to do with the eol attribute.  As we ourselves have
noticed, most shell implementations do not care for carriage returns,
and as such, people will practically always want them to use LF endings.
Similar things can be said for batch files on Windows, except with CRLF
endings.

Since these are common things to have in a repository, let's help users
make a good decision by recommending that they use the gitattributes
file to correctly check out the endings.

In addition, let's correct the cross-reference to this question, which
originally referred to "the following entry", even though a new entry
has been inserted in between.  The cross-reference notation should
prevent this from occurring and provide a link in formats, such as HTML,
which support that.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index abc0f62e6c..85ac99c7b2 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -401,8 +401,9 @@ I'm on Windows and git diff shows my files as having a `^M` at the end.::
 +
 You can store the files in the repository with Unix line endings and convert
 them automatically to your platform's line endings.  To do that, set the
-configuration option `core.eol` to `native` and see the following entry for
-information about how to configure files as text or binary.
+configuration option `core.eol` to `native` and see
+<<recommended-storage-settings,the question on recommended storage settings>>
+for information about how to configure files as text or binary.
 +
 You can also control this behavior with the `core.whitespace` setting if you
 don't wish to remove the carriage returns from your line endings.
@@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
 +
 We also recommend setting a linkgit:gitattributes[5] file to explicitly mark
 which files are text and which are binary.  If you want Git to guess, you can
-set the attribute `text=auto`.  For example, the following might be appropriate
-in some projects:
+set the attribute `text=auto`.
++
+With text files, Git will generally the repository contains LF endings in the
+repository, and will honor `core.autocrlf` and `core.eol` to decide what options
+to use when checking files out.  You can also override this by specifying a
+particular line ending such as `eol=lf` or `eol=crlf` if those files must always
+have that ending (e.g., for functionality reasons).
++
+For example, the following might be appropriate in some projects:
 +
 ----
 # By default, guess.
 *	text=auto
 # Mark all C files as text.
 *.c	text
+# Ensure all shell files end up with LF endings and all batch files end up
+# with CRLF endings in the working tree and both end up with LF in the repo.
+*.sh text eol=lf
+*.bat text eol=crlf
 # Mark all JPEG files as binary.
 *.jpg	binary
 ----

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[Eric Sunshine]

On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> In the FAQ, we tell people how to use the text attribute, but we fail to
> explain what to do with the eol attribute.  As we ourselves have
> noticed, most shell implementations do not care for carriage returns,
> and as such, people will practically always want them to use LF endings.
> Similar things can be said for batch files on Windows, except with CRLF
> endings.
>
> Since these are common things to have in a repository, let's help users
> make a good decision by recommending that they use the gitattributes
> file to correctly check out the endings.
>
> In addition, let's correct the cross-reference to this question, which
> originally referred to "the following entry", even though a new entry
> has been inserted in between.  The cross-reference notation should
> prevent this from occurring and provide a link in formats, such as HTML,
> which support that.
>
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> +With text files, Git will generally the repository contains LF endings in the
> +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> +to use when checking files out.  You can also override this by specifying a
> +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> +have that ending (e.g., for functionality reasons).

The first sentence in the paragraph is unparseable.

> +# Ensure all shell files end up with LF endings and all batch files end up
> +# with CRLF endings in the working tree and both end up with LF in the repo.
> +*.sh text eol=lf
> +*.bat text eol=crlf

Maybe: s/end up with/have/g

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 2334 bytes --]

On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > In the FAQ, we tell people how to use the text attribute, but we fail to
> > explain what to do with the eol attribute.  As we ourselves have
> > noticed, most shell implementations do not care for carriage returns,
> > and as such, people will practically always want them to use LF endings.
> > Similar things can be said for batch files on Windows, except with CRLF
> > endings.
> >
> > Since these are common things to have in a repository, let's help users
> > make a good decision by recommending that they use the gitattributes
> > file to correctly check out the endings.
> >
> > In addition, let's correct the cross-reference to this question, which
> > originally referred to "the following entry", even though a new entry
> > has been inserted in between.  The cross-reference notation should
> > prevent this from occurring and provide a link in formats, such as HTML,
> > which support that.
> >
> > Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> > ---
> > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > +With text files, Git will generally the repository contains LF endings in the
> > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > +to use when checking files out.  You can also override this by specifying a
> > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > +have that ending (e.g., for functionality reasons).
> 
> The first sentence in the paragraph is unparseable.

Yes, I think perhaps I omitted the word "ensure".

And I should reflect that they should have that ending in the working
tree, which I neglected to mention.

> > +# Ensure all shell files end up with LF endings and all batch files end up
> > +# with CRLF endings in the working tree and both end up with LF in the repo.
> > +*.sh text eol=lf
> > +*.bat text eol=crlf
> 
> Maybe: s/end up with/have/g

Great.  That does read more clearly and succinctly.

I'll include those fixes in a reroll.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[Johannes Schindelin]

Hi brian,

On Wed, 20 Oct 2021, brian m. carlson wrote:

> On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > <sandals@crustytoothpaste.net> wrote:
> >
> > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > +With text files, Git will generally the repository contains LF endings in the
> > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > +to use when checking files out.  You can also override this by specifying a
> > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > +have that ending (e.g., for functionality reasons).
> >
> > The first sentence in the paragraph is unparseable.
>
> Yes, I think perhaps I omitted the word "ensure".
>
> And I should reflect that they should have that ending in the working
> tree, which I neglected to mention.

Please note that Git for Windows defaults to `core.autoCRLF=true`,
therefore this sentence is not completely correct. Maybe something as
short as "(except in Git for Windows, which defaults to CRLF endings)"
would suffice?

Ciao,
Dscho

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1735 bytes --]

On 2021-10-20 at 12:02:02, Johannes Schindelin wrote:
> Hi brian,
> 
> On Wed, 20 Oct 2021, brian m. carlson wrote:
> 
> > On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > > <sandals@crustytoothpaste.net> wrote:
> > >
> > > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > > +With text files, Git will generally the repository contains LF endings in the
> > > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > > +to use when checking files out.  You can also override this by specifying a
> > > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > > +have that ending (e.g., for functionality reasons).
> > >
> > > The first sentence in the paragraph is unparseable.
> >
> > Yes, I think perhaps I omitted the word "ensure".
> >
> > And I should reflect that they should have that ending in the working
> > tree, which I neglected to mention.
> 
> Please note that Git for Windows defaults to `core.autoCRLF=true`,
> therefore this sentence is not completely correct. Maybe something as
> short as "(except in Git for Windows, which defaults to CRLF endings)"
> would suffice?

What I meant by that sentence was that I should add, "in the working
tree" to the sentence ending "if those files must always have that
ending".  I believe that is still the case for Git for Windows, since
otherwise our shell files in the repository would be broken there, and
I'm fairly confident they are not.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 3/4] gitfaq: give advice on using eol attribute in gitattributes

[Johannes Schindelin]

Hi brian,

On Wed, 20 Oct 2021, brian m. carlson wrote:

> On 2021-10-20 at 12:02:02, Johannes Schindelin wrote:
> > Hi brian,
> >
> > On Wed, 20 Oct 2021, brian m. carlson wrote:
> >
> > > On 2021-10-20 at 01:21:40, Eric Sunshine wrote:
> > > > On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> > > > <sandals@crustytoothpaste.net> wrote:
> > > >
> > > > > diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> > > > > @@ -464,14 +465,25 @@ references, URLs, and hashes stored in the repository.
> > > > > +With text files, Git will generally the repository contains LF endings in the
> > > > > +repository, and will honor `core.autocrlf` and `core.eol` to decide what options
> > > > > +to use when checking files out.  You can also override this by specifying a
> > > > > +particular line ending such as `eol=lf` or `eol=crlf` if those files must always
> > > > > +have that ending (e.g., for functionality reasons).
> > > >
> > > > The first sentence in the paragraph is unparseable.
> > >
> > > Yes, I think perhaps I omitted the word "ensure".
> > >
> > > And I should reflect that they should have that ending in the working
> > > tree, which I neglected to mention.
> >
> > Please note that Git for Windows defaults to `core.autoCRLF=true`,
> > therefore this sentence is not completely correct. Maybe something as
> > short as "(except in Git for Windows, which defaults to CRLF endings)"
> > would suffice?
>
> What I meant by that sentence was that I should add, "in the working
> tree" to the sentence ending "if those files must always have that
> ending".  I believe that is still the case for Git for Windows, since
> otherwise our shell files in the repository would be broken there, and
> I'm fairly confident they are not.

I swear I read this sentence multiple times yesterday, and still managed
to miss that you were talking about LF endings _in the repository_.

FWIW I think that `core.autoCRLF` defaults to `off` everywhere but on
Windows, so _technically_ Git would not ensure LF endings e.g. on Linux.
In practice, it is highly unlikely that a Linux user would generate text
files with CR/LF endings, so it would not matter in practice.

I feel a bit bad for arguing subtleties that realistically won't matter,
so please feel free to leave the sentence as you have intended it.

Ciao,
Dscho

[PATCH 4/4] doc: add a FAQ entry about syncing working trees

[brian m. carlson]

Users very commonly want to sync their working tree across machines,
often to carry across in-progress work or stashes.  Despite this not
being a recommended approach, users want to do it and are not dissuaded
by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing
service, which is a bad idea.  Users have reported problems where they
end up with duplicate files that won't go away (with names like "file.c
2"), broken references, oddly named references that have date stamps
appended to them, missing objects, and general corruption and data loss.
That's because almost all of these tools sync file by file, which is a
great technique if your project is a single word processing document or
spreadsheet, but is utterly abysmal for Git repositories because they
don't necessarily snapshot the entire repository correctly.  They also
tend to sync the files immediately instead of when the repository is
quiescent, so writing multiple files, as occurs during a commit or a gc,
can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that
the repository is quiescent, so let's suggest that and dissuade people
from using cloud syncing tools.  Let's tell people about common things
they should be aware of before doing this and that this is still
potentially risky.  Additionally, let's tell people that Git's security
model does not permit sharing working trees across users in case they
planned to do that.  While we'd still prefer users didn't try to do
this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 43 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 85ac99c7b2..4a8a46f980 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  Doing so
+	can cause `git status` to need to re-read every file in the working tree.
+	Additionally, Git's security model does not permit sharing a working tree
+	across untrusted users, so it is only safe to sync a working tree if it will
+	only be used by a single user across all machines.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+The recommended approach is to use `rsync -a --delete-after` (ideally with an
+encrypted connection such as with `ssh`) on the root of repository.  You should
+ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc`).
++
+Be aware that even with these recommendations, syncing in this way is
+potentially risky since it bypasses Git's normal integrity checking for
+repositories, so having backups is advised.
+
 Common Issues
 -------------
 

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

[Bagas Sanjaya]

On 20/10/21 08.06, brian m. carlson wrote:
> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.
> +

This raises question: how users can backup their repos? The answer is 
same as this FAQ for syncing working trees: have your repo to be pushed 
to the spare server at disposal, or use rsync (as long as the 
preconditions are met).

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

[Philip Oakley]

On 20/10/2021 05:58, Bagas Sanjaya wrote:
> On 20/10/21 08.06, brian m. carlson wrote:
>> +Be aware that even with these recommendations, syncing in this way is
>> +potentially risky since it bypasses Git's normal integrity checking for
>> +repositories, so having backups is advised.
>> +
>
> This raises question: how users can backup their repos? The answer is
> same as this FAQ for syncing working trees: have your repo to be
> pushed to the spare server at disposal, or use rsync (as long as the
> preconditions are met).
>
Perhaps, for backing up a repo, users might be pointed toward `git
bundle` (with caveats about `--all` being over complete..)

-- 
Philip

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

[Ævar Arnfjörð Bjarmason]

On Wed, Oct 20 2021, brian m. carlson wrote:

> +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> +encrypted connection such as with `ssh`) on the root of repository.  You should
> +ensure several things when you do this:

What's the reason to recommend --delete-after in particular? I realize
that e.g. in the .git directory not using *A* delete option *will* cause
corruption, e.g. if you can leave behind stale loose refs with an
up-to-date pack-refs file.

But isn't that equally covered by --delete and --delete-before? I'm not
very well worsed in rsync, but aren't the two equivalent as far as the
end-state goes?

If the intention with --delete-after over --delete or --delete-before is
to somehow make the repository useful during the transfer, doesn't that
go against the later advice of:

> +* The repository is in a quiescent state for the duration of the transfer (that
> +	is, no operations of any sort are taking place on it, including background
> +	operations like `git gc`).

Also for this:

> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.

Perhaps we should recommend running a "git gc" or other integrity check
after (or "git fsck"), although those don't cover some cases, e.g. the
pack-refs v.s. loose refs problem in the case of a missing
--delete-whatever.

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1768 bytes --]

On 2021-10-20 at 23:35:43, Ævar Arnfjörð Bjarmason wrote:
> 
> On Wed, Oct 20 2021, brian m. carlson wrote:
> 
> > +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> > +encrypted connection such as with `ssh`) on the root of repository.  You should
> > +ensure several things when you do this:
> 
> What's the reason to recommend --delete-after in particular? I realize
> that e.g. in the .git directory not using *A* delete option *will* cause
> corruption, e.g. if you can leave behind stale loose refs with an
> up-to-date pack-refs file.
> 
> But isn't that equally covered by --delete and --delete-before? I'm not
> very well worsed in rsync, but aren't the two equivalent as far as the
> end-state goes?

Yes.  The goal is that if something goes wrong, you have all the objects
you did before, even if you have some potentially invalid refs.  The
goal is to make it a little less risky if you interrupt it with a Ctrl-C
because you realize the destination contained data you wanted.  I always
prefer --delete-after for that reason, assuming the destination has
sufficient disk space.

It shouldn't make a difference in a successful end state, however.
> 
> > +Be aware that even with these recommendations, syncing in this way is
> > +potentially risky since it bypasses Git's normal integrity checking for
> > +repositories, so having backups is advised.
> 
> Perhaps we should recommend running a "git gc" or other integrity check
> after (or "git fsck"), although those don't cover some cases, e.g. the
> pack-refs v.s. loose refs problem in the case of a missing
> --delete-whatever.

I can recommend something like that.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 4/4] doc: add a FAQ entry about syncing working trees

[Ævar Arnfjörð Bjarmason]

On Thu, Oct 21 2021, brian m. carlson wrote:

> [[PGP Signed Part:Undecided]]
> On 2021-10-20 at 23:35:43, Ævar Arnfjörð Bjarmason wrote:
>> 
>> On Wed, Oct 20 2021, brian m. carlson wrote:
>> 
>> > +The recommended approach is to use `rsync -a --delete-after` (ideally with an
>> > +encrypted connection such as with `ssh`) on the root of repository.  You should
>> > +ensure several things when you do this:
>> 
>> What's the reason to recommend --delete-after in particular? I realize
>> that e.g. in the .git directory not using *A* delete option *will* cause
>> corruption, e.g. if you can leave behind stale loose refs with an
>> up-to-date pack-refs file.
>> 
>> But isn't that equally covered by --delete and --delete-before? I'm not
>> very well worsed in rsync, but aren't the two equivalent as far as the
>> end-state goes?
>
> Yes.  The goal is that if something goes wrong, you have all the objects
> you did before, even if you have some potentially invalid refs.  The
> goal is to make it a little less risky if you interrupt it with a Ctrl-C
> because you realize the destination contained data you wanted.  I always
> prefer --delete-after for that reason, assuming the destination has
> sufficient disk space.

Isn't it preferable to recommend --delete-before for that reason?
I.e. --delete-after will produce subtle corruption of e.g. refs
potentially pointing to the wrong thing.

But if you recommend --delete-before I think (but maybe I'm missing some
cases) that it will be more likely to produce obvious corruption,
e.g. git dying due to missing objects.

Anyway, I'm also happy to just leave this as-is, it just stood out to me
as od..

> It shouldn't make a difference in a successful end state, however.
>> 
>> > +Be aware that even with these recommendations, syncing in this way is
>> > +potentially risky since it bypasses Git's normal integrity checking for
>> > +repositories, so having backups is advised.
>> 
>> Perhaps we should recommend running a "git gc" or other integrity check
>> after (or "git fsck"), although those don't cover some cases, e.g. the
>> pack-refs v.s. loose refs problem in the case of a missing
>> --delete-whatever.
>
> I can recommend something like that.

...or just leave it as-is is also fine with me, whatever you think is
best.

[PATCH 4/4] gitfaq: add entry about syncing working trees

[brian m. carlson]

Users very commonly want to sync their working tree across machines,
often to carry across in-progress work or stashes.  Despite this not
being a recommended approach, users want to do it and are not dissuaded
by suggestions not to, so let's recommend a sensible technique.

The technique that many users are using is their preferred cloud syncing
service, which is a bad idea.  Users have reported problems where they
end up with duplicate files that won't go away (with names like "file.c
2"), broken references, oddly named references that have date stamps
appended to them, missing objects, and general corruption and data loss.
That's because almost all of these tools sync file by file, which is a
great technique if your project is a single word processing document or
spreadsheet, but is utterly abysmal for Git repositories because they
don't necessarily snapshot the entire repository correctly.  They also
tend to sync the files immediately instead of when the repository is
quiescent, so writing multiple files, as occurs during a commit or a gc,
can confuse the tools and lead to corruption.

We know that the old standby, rsync, is up to the task, provided that
the repository is quiescent, so let's suggest that and dissuade people
from using cloud syncing tools.  Let's tell people about common things
they should be aware of before doing this and that this is still
potentially risky.  Additionally, let's tell people that Git's security
model does not permit sharing working trees across users in case they
planned to do that.  While we'd still prefer users didn't try to do
this, hopefully this will lead them in a safer direction.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/gitfaq.txt | 43 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 41 insertions(+), 2 deletions(-)

diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
index 85ac99c7b2..4a8a46f980 100644
--- a/Documentation/gitfaq.txt
+++ b/Documentation/gitfaq.txt
@@ -83,8 +83,8 @@ Windows would be the configuration `"C:\Program Files\Vim\gvim.exe" --nofork`,
 which quotes the filename with spaces and specifies the `--nofork` option to
 avoid backgrounding the process.
 
-Credentials
------------
+Credentials and Transfers
+-------------------------
 
 [[http-credentials]]
 How do I specify my credentials when pushing over HTTP?::
@@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
 `git@example_committer` instead of `git@example.org` (e.g., `git remote set-url
 git@example_author:org1/project1.git`).
 
+[[sync-working-tree]]
+How do I sync a working tree across systems?::
+	First, decide whether you want to do this at all.  Git usually works better
+	when you push or pull your work using the typical `git push` and `git fetch`
+	commands and isn't designed to share a working tree across systems.  Doing so
+	can cause `git status` to need to re-read every file in the working tree.
+	Additionally, Git's security model does not permit sharing a working tree
+	across untrusted users, so it is only safe to sync a working tree if it will
+	only be used by a single user across all machines.
++
+Therefore, it's better to push your work to either the other system or a central
+server using the normal push and pull mechanism.  However, this doesn't always
+preserve important data, like stashes, so some people prefer to share a working
+tree across systems.
++
+It is important not to use a cloud syncing service to sync any portion of a Git
+repository, since this can cause corruption, such as missing objects, changed
+or added files, broken refs, and a wide variety of other corruption.  These
+services tend to sync file by file and don't understand the structure of a Git
+repository.  This is especially bad if they sync the repository in the middle of
+it being updated, since that is very likely to cause incomplete or partial
+updates and therefore data loss.
++
+The recommended approach is to use `rsync -a --delete-after` (ideally with an
+encrypted connection such as with `ssh`) on the root of repository.  You should
+ensure several things when you do this:
++
+* There are no additional worktrees enabled for your repository.
+* You are not using a separate Git directory outside of your repository root.
+* You are comfortable with the destination directory being an exact copy of the
+	source directory, _deleting any data that is already there_.
+* The repository is in a quiescent state for the duration of the transfer (that
+	is, no operations of any sort are taking place on it, including background
+	operations like `git gc`).
++
+Be aware that even with these recommendations, syncing in this way is
+potentially risky since it bypasses Git's normal integrity checking for
+repositories, so having backups is advised.
+
 Common Issues
 -------------
 

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

[Eric Sunshine]

On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
<sandals@crustytoothpaste.net> wrote:
> gitfaq: add entry about syncing working trees

You sent two [4/4] patches. I'm guessing the one prefixed by "gitfaq:"
is the correct one.

> Users very commonly want to sync their working tree across machines,
> often to carry across in-progress work or stashes.  Despite this not
> being a recommended approach, users want to do it and are not dissuaded
> by suggestions not to, so let's recommend a sensible technique.
> [...]
> Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
> ---
> diff --git a/Documentation/gitfaq.txt b/Documentation/gitfaq.txt
> @@ -185,6 +185,45 @@ Then, you can adjust your push URL to use `git@example_author` or
> +[[sync-working-tree]]
> +How do I sync a working tree across systems?::
> +       First, decide whether you want to do this at all.  Git usually works better
> +       when you push or pull your work using the typical `git push` and `git fetch`
> +       commands and isn't designed to share a working tree across systems.  Doing so
> +       can cause `git status` to need to re-read every file in the working tree.
> +       Additionally, Git's security model does not permit sharing a working tree
> +       across untrusted users, so it is only safe to sync a working tree if it will
> +       only be used by a single user across all machines.
> ++
> +Therefore, it's better to push your work to either the other system or a central
> +server using the normal push and pull mechanism.  However, this doesn't always
> +preserve important data, like stashes, so some people prefer to share a working
> +tree across systems.
> ++
> +It is important not to use a cloud syncing service to sync any portion of a Git
> +repository, since this can cause corruption, such as missing objects, changed
> +or added files, broken refs, and a wide variety of other corruption.  These
> +services tend to sync file by file and don't understand the structure of a Git
> +repository.  This is especially bad if they sync the repository in the middle of
> +it being updated, since that is very likely to cause incomplete or partial
> +updates and therefore data loss.
> ++
> +The recommended approach is to use `rsync -a --delete-after` (ideally with an
> +encrypted connection such as with `ssh`) on the root of repository.  You should
> +ensure several things when you do this:
> [...]
> +Be aware that even with these recommendations, syncing in this way is
> +potentially risky since it bypasses Git's normal integrity checking for
> +repositories, so having backups is advised.

Considering the potential damage which can result from this sort of
synching, this entire section seems too gentle. My knee-jerk reaction
is that it would be better to strongly dissuade upfront rather than
saying that it's okay to do this if you really want to. As such, I'm
wondering if organizing the section like this would be better:

(1) Make a strong statement against doing this: "<strong>Don't do it.</strong>"

(2) Explain why users shouldn't do it; in particular, the final
paragraph above which talks about integrity checks and whatnot should
be right up near the top along with discussion of corruption.

(3) Say that cloud-synching services must _not_ be used and explain why.

(4) Relent a tiny bit and explain that the only slightly acceptable
mechanism is `rsync` when used in a very strict fashion (quiescent
repository, etc.)

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1818 bytes --]

On 2021-10-20 at 01:38:47, Eric Sunshine wrote:
> On Tue, Oct 19, 2021 at 9:06 PM brian m. carlson
> <sandals@crustytoothpaste.net> wrote:
> > gitfaq: add entry about syncing working trees
> 
> You sent two [4/4] patches. I'm guessing the one prefixed by "gitfaq:"
> is the correct one.

Yes, I appear to have done a bad cherry-pick.  Will fix in v2.

> Considering the potential damage which can result from this sort of
> synching, this entire section seems too gentle. My knee-jerk reaction
> is that it would be better to strongly dissuade upfront rather than
> saying that it's okay to do this if you really want to. As such, I'm
> wondering if organizing the section like this would be better:
> 
> (1) Make a strong statement against doing this: "<strong>Don't do it.</strong>"

I agree this is dangerous.  The reason this is so painful is why I long
ago stopped having a desktop: I needed to sync in-progress work
frequently, and having multiple machines is too much of a hassle for
that case.  The laptop is more portable and can be used everywhere, even
if less powerful.

However, some people do legitimately need to work on the same project
across machines, and the current tooling for syncing stashes and other
in-progress work is insufficient.  Therefore, if we just tell people,
"Don't do this," they're going to stop reading and ignore us, because
we've neglected their needs and they have a job to do.  That would be
worse, because instead of using something like rsync, they might use a
cloud syncing service, and then they'll be really in a bad place.

However, I'm happy to try the reorganization you proposed, even if I
don't necessarily adopt the strength of the proposal, and see how it works.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [PATCH 4/4] gitfaq: add entry about syncing working trees

[Johannes Schindelin]

Hi brian,

On Wed, 20 Oct 2021, brian m. carlson wrote:

> Users very commonly want to sync their working tree across machines,

I was confused at first, because I do sync my working trees frequently. I
do this via `git push` and `git pull`, though.

Maybe clarify that you mean rsync, or sync services like DropBox and
OneDrive? I see that you mention "cloud syncing service" below, but I
believe that it might be better to lead with the examples.

> often to carry across in-progress work or stashes.  Despite this not
> being a recommended approach, users want to do it and are not dissuaded
> by suggestions not to, so let's recommend a sensible technique.
>
> The technique that many users are using is their preferred cloud syncing
> service, which is a bad idea.  Users have reported problems where they
> end up with duplicate files that won't go away (with names like "file.c
> 2"), broken references, oddly named references that have date stamps
> appended to them, missing objects, and general corruption and data loss.
> That's because almost all of these tools sync file by file, which is a
> great technique if your project is a single word processing document or
> spreadsheet, but is utterly abysmal for Git repositories because they
> don't necessarily snapshot the entire repository correctly.  They also
> tend to sync the files immediately instead of when the repository is
> quiescent, so writing multiple files, as occurs during a commit or a gc,
> can confuse the tools and lead to corruption.
>
> We know that the old standby, rsync, is up to the task, provided that
> the repository is quiescent, so let's suggest that and dissuade people
> from using cloud syncing tools.  Let's tell people about common things
> they should be aware of before doing this and that this is still
> potentially risky.  Additionally, let's tell people that Git's security
> model does not permit sharing working trees across users in case they
> planned to do that.  While we'd still prefer users didn't try to do
> this, hopefully this will lead them in a safer direction.

The remainder of the commit message is very clear.

Thank you,
Dscho