"Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

"Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Konstantin Khomoutov]

The "It Will Never Work in Theory" blog has just posted a summary of a
study which tried to identify shortcomings in the design of Git.

In the hope it might be interesting, I post this summary here.
URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html

The except from that resource written by Greg Wilson, the blog author:
---------------->8----------------
Santiago Perez De Rosso and Daniel Jackson: "[Purposes, Concepts,
Misfits, and a Redesign of Git]
(http://people.csail.mit.edu/sperezde/pre-print-oopsla16.pdf)", _SPLASH
2016_. 

> Git is a widely used version control system that is powerful but
> complicated. Its complexity may not be an inevitable consequence of
> its power but rather evidence of flaws in its design. To explore this
> hypothesis, we analyzed the design of Git using a theory that
> identifies concepts, purposes, and misfits. Some well-known
> difficulties with Git are described, and explained as misfits in
> which underlying concepts fail to meet their intended purpose. Based
> on this analysis, we designed a reworking of Git (called Gitless)
> that attempts to remedy these flaws. 
> 
> To correlate misfits with issues reported by users, we conducted a
> study of Stack Overflow questions. And to determine whether users
> experienced fewer complications using Gitless in place of Git, we
> conducted a small user study. Results suggest our approach can be
> profitable in identifying, analyzing, and fixing design problems. 

This paper presents a detailed, well-founded critique of one of the
most powerful, but frustrating, tools in widespread use today. A
follow-up to earlier work published in 2013, it is distinguished from
most other discussion of software design by three things: 

  1. It clearly describes its design paradigm, which comprises
_concepts_ (the major elements of the user's mental model of the
system), _purposes_ (which motivate the concepts), and _misfits_ (which
are instances where concepts do not satisfy purposes, or contradict one
another). 

  2. It lays out Git's concepts and purposes, analyzes its main
features in terms of them, and uses that analysis to identify
mis-matches. 

  3. Crucially, it then analyzes independent discussion of Git (on
Stack Overflow) to see if users are stumbling over the misfits
identified in step 2. 

That would count as a major contribution on its own, but the authors go
further. They have designed a tool called Gitless that directly
addresses the shortcomings they have identified, and the penultimate
section of this paper presents a usability study that compares it to
standard Git. Overall, subjects found Gitles more satisfying and less
frustrating than Git, even though there was no big difference in
efficiency, difficulty, or confusion. Quoting the paper, "This apparent
contradiction might be due to the fact that all of the participants had
used Git before but were encountering Gitless for the first time
without any substantive training. Some participants (2 regular, 1
expert) commented that indeed their problems with Gitless were mostly
due to their lack of practice using it." 

This paper is one of the best examples I have ever seen of how software
designs ought to be critiqued. It combines an explicit, coherent
conceptual base, detailed analysis of a specific system, design
grounded in that analysis, and an empirical check of that design.
Sadly, nothing shows the actual state of our profession more clearly
than the way this work has been greeted: 

> In some respects, this project has been a fool's errand. We picked a
> product that was popular and widely used so as not to be investing
> effort in analyzing a strawman design; we thought that its popularity
> would mean that a larger audience would be interested in our
> experiment. In sharing our research with colleagues, however, we have
> discovered a significant polarization. Experts, who are deeply
> familiar with the product, have learned its many intricacies,
> developed complex, customized workflows, and regularly exploit its
> most elaborate features, are often defensive and resistant to the
> suggestion that the design has flaws. In contrast, less intensive
> users, who have given up on understanding the product, and rely on
> only a handful of memorized commands, are so frustrated by their
> experience that an analysis like ours seems to them belaboring the
> obvious.
---------------->8----------------
(This text is Copyright © Never Work in Theory, under the CC license.)

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Jakub Narębski]

W dniu 30.09.2016 o 18:14, Konstantin Khomoutov pisze:

> The "It Will Never Work in Theory" blog has just posted a summary of a
> study which tried to identify shortcomings in the design of Git.
> 
> In the hope it might be interesting, I post this summary here.
> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html

I will comment on the article itself, not just on the summary.

| 2.2 Git
[...]
| But tracked files cannot be ignored; to ignore a tracked file
| one has to mark it as “assume unchanged.” This “assume
| unchanged” file will not be recognized by add; to make it
| tracked again this marking has to be removed.

WRONG!  Git has tracked files, untracked unignored files, and
untracked ignored files (mostly considered unimportant).

The "assume unchanged" bit is _performance_ optimization. It is not,
and cannot be a 'ignore tracked files' bit - here lies lost work!!!
You can use (imperfectly) "prefer worktree" bit hack instead.

You can say, if 'ignoring change to tracked files' is motivation,
or purpose, it lacks direct concept.

[...]
| As a result, when a user switches branches, files may be
| unexpectedly overwritten. 

This is possible _only_ if there are uncommitted changes. If they
are there, and they do not conflict with switching a branch, they
are "floated" to a newly checked out branch.

| Git fails with an error if there are any conflicting changes,
| effectively preventing the user from switching in this case.
| To mitigate this problem, Git provides a way to save versions
| of files to another storage area, called the “stash,” using
| a special command issued prior to the branch switch.

Or you can try to merge uncommitted changes with changes between
two branches: current and switched to.

Or you can forcibly discard your changes.

Or (with modern Git), you can put each branch in a separate
working area (with "git worktree"), though the article may predate
this feature.

[...]
| *Syncing with Other Repositories* Crucial to the understanding
| of how syncing with other repositories work is the notion
| of a “remote branch.” This is a branch (pointer to a commit)
| that (asynchronously) reflects the state of a branch in another
| repository. It is updated whenever there is some network
| communication (e.g., a push or fetch).

It is called "remote-tracking branch", rather than "remote branch".
At least in Git documentation. This branch is in local repository,
not in remote one. The remote-tracking branch for example
`origin/master` follows (tracks) branch `master` in remote
repository `origin`.

| The notion of a “remote branch” must not be confused
| with that of an “upstream branch.” An upstream branch is
| just a convenience for users: after the user assigns it to some
| branch, commands like pull and push default to use that
| branch for fetching and pushing changes if no branch is given
| as input.

Actually "upstream branch" (and related "upstream repository")
are a concept, not only a convenience. They denote a branch
(usually in remote repository) which is intended to ultimately
include changes in given branch. Note that "upstream branch"
can be set separately for any given local branch.

One thing that can enormously help recovering from errors, and
is not covered in the list of concepts is REFLOG.


[...]
| 3. Operational Misfits
[...]
| *Saving Changes* Suppose you are in the middle of a long
| task and want to save your changes, so that they can be later
| retrieved in case of failure. How would you do that?

You would use `git stash` or `git stash --include-untracked`!
In more complicated situations (during long-running operation
like resolving merge conflicts, interactive rebase, or finding
bugs with bisect) with modern Git you can create a new separate
working area with `git worktree`.

This also applies to the "*Switching branches*" problem (which
is more involved, as `git stash` would often not work, and if
it does it is harder to restore state - note however that stash
description includes the branch it was on).

So I would say that *Saving Changes* is solved with stash,
while *Switching Branches* remains a misfit.

| *Detached Head* Suppose you are working on some branch
| and realize that the last few commits you did are wrong, so
| you decide to go back to an old commit to start over again.
| You checkout that old commit and keep working creating
| commits. You might be surprised to discover that these new
| commits you’ve been working on belong to no branch at all.
| To avoid losing them you need to create a new branch or reset
| an existing one to point to the last commit.

It would be hard to be surprised unless one is in habit of
disregarding multi-line warning from Git... ;-)

I think it might be more of an UX problem, namely that the
`git checkout` command does too many things, which include
checking out revision (detaching HEAD, or landing on unnamed
branch, unless we create a new branch at the same time with
the '-b <newbranch>' option), and checking out a branch,
that is switching to other branch.

| *File Rename* Suppose you rename a file and make some
| changes to it. If you changed a significant portion of the file,
| then, as far as Git is concerned, you didn’t rename the file,
| but it is instead as if you deleted the old file and created a new
| one (which means that the file history is now lost). To work
| around this, you have to be diligent about creating a commit
| with the rename only, and only then creating a new commit
| with the modifications. This, however, likely creates a bogus
| commit that doesn’t correspond to a logical group of changes.

First, I think it might be inherent problem. The version control
system may implement *rename tracking* (store information about
renames), or *rename detection*. The latter is what Git does,
and it is what allows to do for example detecting copying and
movement of contents across files and within the same file
for `git blame` (and `git gui blame`) - which is impossible with
only rename tracking.

Nb. that handling file renames and other source reorganization
is important thing that modern version control systems should
be able to handle.

Second, the need for handling renames can be found in two
different operations. One is following history of a single
file across renames, or seeing a rename when looking at changes
in a single revision. The other is merging two lines of
development where one did a rename. The trick of splitting
large rename+change into pure rename and change without rename
can help only the former... the less important one. Merging
in Git is done using [recursive] 3-way merge strategy, which
takes into account only the endpoint state, and not history
between them - so if it was rename then change doesn't matter.

Third, (which is a bit of victim^W user blaming), large changes
are process smell. The cases when change accompanying rename
is so large that it screws heuristic based rename detection
legitimately are (or are supposed to be) rare. And for the
legitimate cases, that is end-of-line changes, there are
solutions to help (-w for diff, -Xrenormalize for merge).

| *File Tracking* Suppose you create a new file and then you
| add the file to start tracking changes to it. You keep working
| on the file making new modifications and then you make
| a vanilla commit. You might be surprised to find out that
| what actually got committed is the old version of the file
| (representing its state the last time the file was staged), and
| not the most recent one.

Unless one uses `git commit -a`, like most people (I think)
in most cases do.

Hopefully `git add -N`, aka. "intent to add", would help here...
when people switch to it to adding new files, and when the
feature gets improved (as it is now) to make it better and
easier to use (e.g. so "git diff" shows new i-t-a files).

| *Untracking File* Suppose there’s a database configuration
| file committed in the repository and you now want to edit
| this file to do some local testing. This new version of the
| file should not be committed. You could always leave out
| the file from the commit every time, but this is laborious and
| error-prone. You might think that you could make it ignored
| by modifying the `.gitignore` file but this doesn’t work for
| committed files.

The name "Untracking File" is misleading. You untrack file
(that is remove it from future commits) with `git rm --cached`.
No problem here. The name is "Ignore changes to tracked files",
or "Stopping tracking changes to file".

If you think of ignored files as unimportant, not precious,
then mismatch between understanding and what Git does would
lessen.

| The way to ignore this file is to mark it as
| “assume unchanged,” but this marking will be cleared when
| you switch to another branch.

s/“assume unchanged,”/“skip worktree,” (see earlier comment).

Right. Also, currently you need to use low-level commands
(`git update-index`) to mark file in this way.

[...]
| 4. Purposes for Version Control
[...]
| *Collaboration* To this point, all the purposes might apply in
| the context of a single user. Collaboration needs arise when
| multiple users work together on a single code base.
| /* ..........................................................
| /* _Purpose 4. Synchronize changes of collaborators_

One very important purpose that must be fulfilled before even
trying to synchronize changes is *isolation of changes*. Each
developer needs its own working area, so his or her changes do
not interfere with work of other developers.

Though "Disconnected operations" a bit overlaps (and is partial
superset) of this purpose.

See also intro to "Version Control by Example", by Eric Sink
http://ericsink.com/vcbe/html/intro.html

[...]
| 5.1 Stashing: An Example
|
| We consider the motivating purpose of stashing not to be
| a subpurpose of any of the high-level purposes for version
| control (§4). This section elaborates on our rationale for this.
|
| Take (what seem to be) the motivating use cases for stashing
| [9, Chapter 7.3]: (1) to pull into a dirty working directory
| and (2) to deal with an interruption in your workflow

Actually (1) is not a separate use case, but a subset 
(specialization) of (2) - and interruption in your workflow,
where the interruption is pull.

Note that to synchronize with remote repository one should use
fetch, not pull.  The latter is more involved operation, and
should be considered interruption.  Also, rarer if using
feature branches workflow, and not working on long-lived
stabilization branches directly.

[...]
| The problem
| is the lack of connection between this purpose and the highlevel
| purposes for version control, which suggests that the
| introduction of stashing might be to patch flaws in the design
| of Git and not to satisfy a requirement of version control.

Or the problem might be that you are missing some (maybe minor)
requirement of version control system. Just saying...

| 6. Analysis
[...]
| *Divided Ignored and Assumed Unchanged*

This ignores the fact that (unstated) assumption is that ignored
files are considered not important (or at least less important;
Git cares less about changes in those files, and may in some
cases make you loose changes to them).

Ignore file != ignore changes, though it may look like it is.

[...]
| 7. Gitless
|
| 7.1 Overview
|
| Gitless has no staging area, and the only file classifications
| are “tracked,” “untracked,” “ignored,” and “in conflict.”

Without staging area, I wonder how you would be able to handle
well different types of integration conflicts, which are not
limited to CONFLICT(content).

You also loose the ability to select subset of *changes* to
be committed, not only files.  This is often very useful, see
http://tomayko.com/writings/the-thing-about-git
http://2ndscale.com/rtomayko/2008/the-thing-about-git

[...]
| A branch in Gitless is a completely independent line of
| development: each branch includes the working version of
| files [...]

If I understand your model correctly, you would get rid of
one problem / mismatch, but get into other. What would you
do if you started work on some branch, and then realized
that you should have been working on a new topic branch?
Or you realized that you are on wrong branch, and want to
move changes?

Ah, I haven't realized that it is described later (well,
at least the first case):

: In regard to branching, to address situations in which the
: user wants changes made in the current branch to be moved
: onto the destination branch (e.g., the user realizes that she
: has been working in the wrong branch), the Gitless branch
: command has a `move-over` flag


[...]
|  Also, there
| is no possible way of getting in a “detached head” state; at
| any time, the user is always working on some branch (the
| “current” branch). Head is a per-branch reference to the last
| commit of the branch.

How do you solve the problem of checking out the state of
the tag, that is the state of repository at given revision?

Also during some long lived multi-step operations, like bisect
or interactive rebase, you are not really on any branch,

| 7.2.1 Discussion
[...]
| There could be other use cases for the
| staging area that Gitless doesn’t handle well but we expect
| these to be fairly infrequent.

Like handling merge conflict...??? Infrequent doesn't mean
unimportant.


That's all my comments for now.


This is a very interesting research.  Those problems (misfits)
are legitimate concern.  Even if it would not result in changes
to how Git works, it should improve how we are talking about
version control, and how we design them (that includes GUIs
that work above version control systems).

Best regards,
-- 
Jakub Narębski

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Kevin Daudt]

On Sat, Oct 01, 2016 at 12:24:57AM +0200, Jakub Narębski wrote:
> 
> | 7.2.1 Discussion
> [...]
> | There could be other use cases for the
> | staging area that Gitless doesn’t handle well but we expect
> | these to be fairly infrequent.
> 
> Like handling merge conflict...??? Infrequent doesn't mean
> unimportant.
> 

For me the most important thing is that the lack of staging area leads
on commits that have no clear goal, people just commit everything they
have changed at some point, as a sort of checkpoint.

Although lots of people still do this with git currently, they don't
even have the possibility[1] to improve on this.

This makes history and features like git bisect less useful.


[1] At most they can specify the files they want to commit, but this is
still a very crude way to group together changes.

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Jakub Narębski]

W dniu 01.10.2016 o 22:13, Kevin Daudt pisze:
> On Sat, Oct 01, 2016 at 12:24:57AM +0200, Jakub Narębski wrote:
>>
>> | 7.2.1 Discussion
>> [...]
>> | There could be other use cases for the
>> | staging area that Gitless doesn’t handle well but we expect
>> | these to be fairly infrequent.

I'd like to point to the word "other" in the above sentence...

>>
>> Like handling merge conflict...??? Infrequent doesn't mean
>> unimportant.
> 
> For me the most important thing is that the lack of staging area leads
> on commits that have no clear goal, people just commit everything they
> have changed at some point, as a sort of checkpoint.
> 
> Although lots of people still do this with git currently, they don't
> even have the possibility[1] to improve on this.
> 
> This makes history and features like git bisect less useful.
> 
> 
> [1] At most they can specify the files they want to commit, but this is
> still a very crude way to group together changes.

... If you had read the original discussed research paper 
http://people.csail.mit.edu/sperezde/pre-print-oopsla16.pdf
you would notice that this use case is covered by Gitless,
though imperfectly.

  Common use cases for the staging area in Git are to select
  files to commit, split up a large change into multiple commits,
  and review the changes selected to be committed. We address
  the first by providing a more flexible `commit` command that
  lets the user easily customize the set of files to commit (with
  `only`, `include` and `exclude` flags). For the second use case
  we have a `partial` flag in `commit` that allows the user to
  interactively select segments of files to commit (like Git’s
  `commit --patch`).

First imperfection is that only equivalent of `git commit --patch`,
that is additive selection of changes during commit.  There are,
from what I understand, no equivalents of `git add --interactive`
(crafting additively and subtractively part by part, not all at
once), `git reset --patch` (crafting additively from any commit,
defaults to HEAD), `git checkout --patch` (crafting subtractively).
It is a good thing to have those possibilities when disentangling
working area, or splitting commit during interactive rebase.

Though all of those could be added (if they are not present
already) to the interactive interface of `partial` flag in
`gl commit`.


Second imperfection is that you cannot test the crafted state
without creating a commit.  In Git you have `git stash --keep-index`
for this; go to the state crafted in the staging area (the index),
and you can test it.

Though you can always create a [temporary] commit, run tests, and
then if necessary amend this commit.


On the other hand the index / the staging area is important and
useful thing in helping to resolve merge conflicts, for example
the one where one side changed file and other deleted it, or
where one side renamed file and other changed it, etc.

-- 
Jakub Narębski

mailto:jnareb@gmail.com
mailto:jnareb@mat.umk.pl

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Jakub Narębski]

[git@vger.kernel.org does not accept HTML emails]

I just hope that this email don't get mangled too much...

On 5 October 2016 at 04:55, Santiago Perez De Rosso
<sperezde@csail.mit.edu> wrote:
> On Fri, Sep 30, 2016 at 6:25 PM Jakub Narębski <jnareb@gmail.com> wrote:
>> W dniu 30.09.2016 o 18:14, Konstantin Khomoutov pisze:
>>
>>> The "It Will Never Work in Theory" blog has just posted a summary of a
>>> study which tried to identify shortcomings in the design of Git.
>>>
>>> In the hope it might be interesting, I post this summary here.
>>> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html
>>
>> I will comment on the article itself, not just on the summary.
>>
>> | 2.2 Git
>> [...]
>> | But tracked files cannot be ignored; to ignore a tracked file
>> | one has to mark it as “assume unchanged.” This “assume
>> | unchanged” file will not be recognized by add; to make it
>> | tracked again this marking has to be removed.
>>
>> WRONG!  Git has tracked files, untracked unignored files, and
>> untracked ignored files (mostly considered unimportant).
>>
>> The "assume unchanged" bit is _performance_ optimization. It is not,
>> and cannot be a 'ignore tracked files' bit - here lies lost work!!!
>> You can use (imperfectly) "prefer worktree" bit hack instead.
>>
>> You can say, if 'ignoring change to tracked files' is motivation,
>> or purpose, it lacks direct concept.
>
>
> I don't see what's wrong with the paragraph you mention. I am aware of the
> fact that assumed unchanged is intended to be used as a performance
> optimization but that doesn't seem to be the way it is used in practice.
> Users have appropriated the optimization and effectively turned into a
> concept that serves the purpose of preventing the commit of a file. For
> example:
>
> from http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html
>
>  So, to temporarily ignore changes in a certain file, run:
>  git update-index --assume-unchanged <file>
>  ...
>
> from http://stackoverflow.com/questions/17195861/undo-git-update-index-assume-unchanged-file
>  The way you git ignore watching/tracking a particular dir/file.
>  you just run this:
>  git update-index --assume-unchanged <file>
> ...
>
>
> btw, this appropriation suggests that users want to be able to ignore
> tracked files and they do what they can with what they are given (which
> in this case means abusing the assumed unchanged bit).

Yes, this is true that users may want to be able to ignore changes to
tracked files (commit with dirty tree), but using `assume-unchanged` is
wrong and dangerous solution.  Unfortunately the advice to use it is
surprisingly pervasive.  I would thank you to not further this error.
(Well, `skip-worktree` is newer, that's why it is lesser known, perhaps)

To ignore tracked files you need to use `skip-worktree` bit.

You can find the difference between `assume-unchanged` and
`skip-worktree`, and when use which in:
http://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree
http://fallengamer.livejournal.com/93321.html
http://blog.stephan-partzsch.de/how-to-ignore-changes-in-tracked-files-with-git/

The difference is that skip-worktree will not overwrite a file that is
different from the version in the index, but assume-unchanged can.  This
means that the latter can OVERWRITE YOUR PRECIOUS CHANGES!

Some people started to recommend it
http://stackoverflow.com/questions/32251037/ignore-changes-to-a-tracked-file
http://www.virtuouscode.com/2011/05/20/keep-local-modifications-in-git-tracked-files/


>> | The notion of a “remote branch” must not be confused
>> | with that of an “upstream branch.” An upstream branch is
>> | just a convenience for users: after the user assigns it to some
>> | branch, commands like pull and push default to use that
>> | branch for fetching and pushing changes if no branch is given
>> | as input.
>>
>> Actually "upstream branch" (and related "upstream repository")
>> are a concept, not only a convenience. They denote a branch
>> (usually in remote repository) which is intended to ultimately
>> include changes in given branch. Note that "upstream branch"
>> can be set separately for any given local branch.
>
>
> We never say upstream branch is not a concept. It even appears in the
> table.

Hmmm... I got onfused by words "is just a convenience", which for me
implies that it is not a concept.

>>
>>
>> One thing that can enormously help recovering from errors, and
>> is not covered in the list of concepts is REFLOG.
>
>
> Yes, the analysis is not exhaustive, there are other concepts missing
> too (e.g., "submodule")

I think reflog is something that every user should know about, and
useful for all.  Submodules and subtrees is something situational,
needed only for a subset of users.

On the other hand the concept of "submodules" is something that it is
under active development, and it is assumed to be immature.  So coming
up with a good (re)design for this feature, and/or good set of
concept, would be a very good thing.


Another thing that it is not present is the concept of "immutable
history", and how to deal with it (`git-notes`, `git-replace`), and
how it conflict with other concepts (interactive rebase and the
concept of "rewriting history").  But I agree with focusing on most
commonly known, used and encountered concepts and misfits.

>> [...]
>> | 3. Operational Misfits
>> [...]
>> | *Saving Changes* Suppose you are in the middle of a long
>> | task and want to save your changes, so that they can be later
>> | retrieved in case of failure. How would you do that?
>>
>> You would use `git stash` or `git stash --include-untracked`!
>> In more complicated situations (during long-running operation
>> like resolving merge conflicts, interactive rebase, or finding
>> bugs with bisect) with modern Git you can create a new separate
>> working area with `git worktree`.
>
> A general comment regarding misfits: misfits correspond to scenarios
> in which Git behaves in a way that is unpredictable or inconvenient.
> It doesn't mean that a task has to be impossible to do in order for
> something to be a misfit.

If there were one-command solution, and the problem was that users
don't know about it, I would say that it is not a misfit.  But,
admittedly, it is not the case here...

>> | *Detached Head* Suppose you are working on some branch
>> | and realize that the last few commits you did are wrong, so
>> | you decide to go back to an old commit to start over again.
>> | You checkout that old commit and keep working creating
>> | commits. You might be surprised to discover that these new
>> | commits you’ve been working on belong to no branch at all.
>> | To avoid losing them you need to create a new branch or reset
>> | an existing one to point to the last commit.
>>
>> It would be hard to be surprised unless one is in habit of
>> disregarding multi-line warning from Git... ;-)
>
> True if you are an expert user, but I can assure you novices will
> find that situation baffling, even with the multi-line warnings.

True, the "detached HEAD" case (aka "unnamed branch") can be puzzling
for Git users, and it has few uses (e.g. checking out the state of
tag temporarily, to test it).

I wonder if `git status` should be enhanced to tell user how to get
out of "detached HEAD" situation -- it has lots of advices in it.

>> [...]
>> | The problem
>> | is the lack of connection between this purpose and the highlevel
>> | purposes for version control, which suggests that the
>> | introduction of stashing might be to patch flaws in the design
>> | of Git and not to satisfy a requirement of version control.
>>
>> Or the problem might be that you are missing some (maybe minor)
>> requirement of version control system. Just saying...
>
> What would that purpose be? and why would you say that's a
> high-level purpose for version control and not one that's
> git-specific?

The stash (or rather its equivalent) is not something Git specific.
It is present also in other version control systems, among others:

* Mercurial: as 'shelve' extension (in core since 1.8)
* Bazaar: as 'bzr shelve' command
* Fossil: as 'fossil stash' command (with subcommands)
* Subversion: Shelve planned for 1.10 (2017?)

I would say that 'stash' could be considered about isolating work on
different features, different sub-branch sized parallel work.

But it might be that stash doesn't have connection with highlevel
purposes for version control, and that it is purely convenience
feature.  Just playing the role of Advocatus Diaboli (important in
scientific works, isn' it?)...

>> [...]
>> | 7. Gitless
>> |
>> | 7.1 Overview
>> |
>> | Gitless has no staging area, and the only file classifications
>> | are “tracked,” “untracked,” “ignored,” and “in conflict.”
>>
>> Without staging area, I wonder how you would be able to handle
>> well different types of integration conflicts, which are not
>> limited to CONFLICT(content).
>>
>> You also loose the ability to select subset of *changes* to
>> be committed, not only files.  This is often very useful, see
>> http://tomayko.com/writings/the-thing-about-git
>> http://2ndscale.com/rtomayko/2008/the-thing-about-git
>
> This is described later:

Right, my mistake, I have missed this.  I should have read article in
full, then respond, rather than reply as I go...

> Common use cases for the staging area in Git are to select files to commit,
> split up a large change into multiple commits, and review the changes
> selected to be committed. We address the first by providing a more flexible
> commit command that lets the user easily customize the set of files to
> commit (with only, include and exclude flags). For the second use case we
> have a partial flag in commit that allows the user to interactively select
> segments of files to commit (like Git’s commit --patch). Finally, our diff
> command accepts the same only, include and exclude flags to customize the
> set of files to be diffed. There could be other use cases for the staging
> area that Gitless doesn’t handle well but we expect these to be fairly
> infrequent.

I wrote later that Git offers more flexibility with respect of
interactive edition of the staging area (additive, substractive, from
HEAD, from arbitrary commit) and checking the prepared state (add <->
diff <-> stash --keep-index + test + unstash)... but I think this is a
matter of implementing interface for those fetures to 'partial'
commit.  It might be less flexible, and less powerfull, but it would
be there.

> Note that Gitless is built on top of Git so we do have a staging area, the
> difference is that unlike Git, in Gitless the index is hidden from the user.

All right.

I have just an idea of using the index (or rather its extensions) to
implement sub-commit history (forgotten after committing), as a way to
resolve misfit about "commit" concept and (P1) vs (P2) purposes...

>> [...]
>> |  Also, there
>> | is no possible way of getting in a “detached head” state; at
>> | any time, the user is always working on some branch (the
>> | “current” branch). Head is a per-branch reference to the last
>> | commit of the branch.
>>
>> How do you solve the problem of checking out the state of
>> the tag, that is the state of repository at given revision?
>
> You can't checkout a tag, you would have to create a new branch with
> its head equal to the tag, and switch to that branch.

For the purpose of testing a state at a tag (for example as a
prerequisite to bisection), it fills like unnecessarily complicated
solution, a new misfit.  But I guess that you consider "detached HEAD"
misfit to be more important to get rid of.

>> Also during some long lived multi-step operations, like bisect
>> or interactive rebase, you are not really on any branch,
>
> In Gitless we don't have bisect but for rebase (fuse in Gitless) we
> record the current branch.

No bisect?  This is very useful feature.  Though it might be done
without detached HEAD, but with specialized pseudo-branch 'bisect' (as
it was done in earlier versions of Git, or maybe even now).

Anyway, for [interactive] rebase / transplant / graft / fuse you need
to be able to abort an operation and return to the state before
staring rebase.  Though you can or do solve this by remembering
the starting position.

>> | 7.2.1 Discussion
>> [...]
>> | There could be other use cases for the
>> | staging area that Gitless doesn’t handle well but we expect
>> | these to be fairly infrequent.
>>
>> Like handling merge conflict...??? Infrequent doesn't mean
>> unimportant.
>
> I regard handling merge conflicts as very important. Why do you need
> an explicit staging area for this? Note that Gitless has a staging
> area, it's just hidden from the user. The `gl diff` or `gl status`
> command should be able to show the same conflict information.

Well, the staging area was created (also?) to handle merges.  Though
perhaps it doesn't need to be explicit to be useful...

Best regards,
-- 
Jakub Narębski

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Duy Nguyen]

On Wed, Oct 5, 2016 at 5:14 PM, Jakub Narębski <jnareb@gmail.com> wrote:
> [git@vger.kernel.org does not accept HTML emails]
>
> I just hope that this email don't get mangled too much...
>
> On 5 October 2016 at 04:55, Santiago Perez De Rosso
> <sperezde@csail.mit.edu> wrote:
>> On Fri, Sep 30, 2016 at 6:25 PM Jakub Narębski <jnareb@gmail.com> wrote:
>>> W dniu 30.09.2016 o 18:14, Konstantin Khomoutov pisze:
>>>
>>>> The "It Will Never Work in Theory" blog has just posted a summary of a
>>>> study which tried to identify shortcomings in the design of Git.
>>>>
>>>> In the hope it might be interesting, I post this summary here.
>>>> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html
>>>
>>> I will comment on the article itself, not just on the summary.
>>>
>>> | 2.2 Git
>>> [...]
>>> | But tracked files cannot be ignored; to ignore a tracked file
>>> | one has to mark it as “assume unchanged.” This “assume
>>> | unchanged” file will not be recognized by add; to make it
>>> | tracked again this marking has to be removed.
>>>
>>> WRONG!  Git has tracked files, untracked unignored files, and
>>> untracked ignored files (mostly considered unimportant).
>>>
>>> The "assume unchanged" bit is _performance_ optimization. It is not,
>>> and cannot be a 'ignore tracked files' bit - here lies lost work!!!
>>> You can use (imperfectly) "prefer worktree" bit hack instead.
>>>
>>> You can say, if 'ignoring change to tracked files' is motivation,
>>> or purpose, it lacks direct concept.
>>
>>
>> I don't see what's wrong with the paragraph you mention. I am aware of the
>> fact that assumed unchanged is intended to be used as a performance
>> optimization but that doesn't seem to be the way it is used in practice.
>> Users have appropriated the optimization and effectively turned into a
>> concept that serves the purpose of preventing the commit of a file. For
>> example:
>>
>> from http://gitready.com/intermediate/2009/02/18/temporarily-ignoring-files.html
>>
>>  So, to temporarily ignore changes in a certain file, run:
>>  git update-index --assume-unchanged <file>
>>  ...
>>
>> from http://stackoverflow.com/questions/17195861/undo-git-update-index-assume-unchanged-file
>>  The way you git ignore watching/tracking a particular dir/file.
>>  you just run this:
>>  git update-index --assume-unchanged <file>
>> ...
>>
>>
>> btw, this appropriation suggests that users want to be able to ignore
>> tracked files and they do what they can with what they are given (which
>> in this case means abusing the assumed unchanged bit).
>
> Yes, this is true that users may want to be able to ignore changes to
> tracked files (commit with dirty tree), but using `assume-unchanged` is
> wrong and dangerous solution.  Unfortunately the advice to use it is
> surprisingly pervasive.  I would thank you to not further this error.
> (Well, `skip-worktree` is newer, that's why it is lesser known, perhaps)
>
> To ignore tracked files you need to use `skip-worktree` bit.
>
> You can find the difference between `assume-unchanged` and
> `skip-worktree`, and when use which in:
> http://stackoverflow.com/questions/13630849/git-difference-between-assume-unchanged-and-skip-worktree
> http://fallengamer.livejournal.com/93321.html
> http://blog.stephan-partzsch.de/how-to-ignore-changes-in-tracked-files-with-git/
>
> The difference is that skip-worktree will not overwrite a file that is
> different from the version in the index, but assume-unchanged can.  This
> means that the latter can OVERWRITE YOUR PRECIOUS CHANGES!
>
> Some people started to recommend it
> http://stackoverflow.com/questions/32251037/ignore-changes-to-a-tracked-file
> http://www.virtuouscode.com/2011/05/20/keep-local-modifications-in-git-tracked-files/

And since skip-worktree bits may be set/cleared freely when sparse
checkout mode is on, you should never manipulate these bits directly
if you also use sparse checkout.

>>> | *Detached Head* Suppose you are working on some branch
>>> | and realize that the last few commits you did are wrong, so
>>> | you decide to go back to an old commit to start over again.
>>> | You checkout that old commit and keep working creating
>>> | commits. You might be surprised to discover that these new
>>> | commits you’ve been working on belong to no branch at all.
>>> | To avoid losing them you need to create a new branch or reset
>>> | an existing one to point to the last commit.
>>>
>>> It would be hard to be surprised unless one is in habit of
>>> disregarding multi-line warning from Git... ;-)
>>
>> True if you are an expert user, but I can assure you novices will
>> find that situation baffling, even with the multi-line warnings.

Hmm...  when you switch away from a detached HEAD, you are advised to
do "git branch <new-name> blah blah". How is it baffling? Genuine
question, maybe I have been using git for too long I just fail to see
it.

> True, the "detached HEAD" case (aka "unnamed branch") can be puzzling
> for Git users, and it has few uses (e.g. checking out the state of
> tag temporarily, to test it).
>
> I wonder if `git status` should be enhanced to tell user how to get
> out of "detached HEAD" situation -- it has lots of advices in it.

Detached HEAD is also present in interactive rebase or any command
that has --abort/--continue options. I don't think we need to tell the
user to get out of detached HEAD in that case. Just two cents if
someone is going to add this advice to git-status.
-- 
Duy

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Jakub Narębski]

On 7 October 2016 at 18:55, Santiago Perez De Rosso
<sperezde@csail.mit.edu> wrote:
> On Wed, Oct 5, 2016 at 6:15 AM Jakub Narębski <jnareb@gmail.com> wrote:
>> On 5 October 2016 at 04:55, Santiago Perez De Rosso
>> <sperezde@csail.mit.edu> wrote:
>>> On Fri, Sep 30, 2016 at 6:25 PM Jakub Narębski <jnareb@gmail.com> wrote:
>>>> W dniu 30.09.2016 o 18:14, Konstantin Khomoutov pisze:
>>>>
>>>>> The "It Will Never Work in Theory" blog has just posted a summary of a
>>>>> study which tried to identify shortcomings in the design of Git.
>>>>>
>>>>> In the hope it might be interesting, I post this summary here.
>>>>> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html
>>>>
>>>> I will comment on the article itself, not just on the summary.
>>>>
>>>> | 2.2 Git
>>>> [...]
>>>> | But tracked files cannot be ignored; to ignore a tracked file
>>>> | one has to mark it as “assume unchanged.” This “assume
>>>> | unchanged” file will not be recognized by add; to make it
>>>> | tracked again this marking has to be removed.
[...]
>> Yes, this is true that users may want to be able to ignore changes to
>> tracked files (commit with dirty tree), but using `assume-unchanged` is
>> wrong and dangerous solution.  Unfortunately the advice to use it is
>> surprisingly pervasive.  I would thank you to not further this error.
>
> Ok, I added a footnote in the paper when we first mention assume unchanged
> that says:
>
> Assume unchanged was intended to be used as a performance optimization but
> has since been appropriated by users as a way to ignore tracked files. The
> current advice is to use the “skip worktree” marking instead
>
> This should prompt readers to look into skip worktree next time they want to
> ignore tracked files. I don't think people reading the paper are doing so to
> learn Git but at least it should contribute to not furthering the error.

Thank you very much.

The problem with "assume-unchanged" is that by using it to ignore
changes to tracked files you are lying to Git (telling it 'assume this
is unchanged' while changing it), and can lead to DATA LOSS, that
is to losing those changes.

[...]
>>>> [...]
>>>> | The problem
>>>> | is the lack of connection between this purpose and the highlevel
>>>> | purposes for version control, which suggests that the
>>>> | introduction of stashing might be to patch flaws in the design
>>>> | of Git and not to satisfy a requirement of version control.
>>>>
>>>> Or the problem might be that you are missing some (maybe minor)
>>>> requirement of version control system. Just saying...
>>>
>>> What would that purpose be? and why would you say that's a
>>> high-level purpose for version control and not one that's
>>> git-specific?
>>
>> The stash (or rather its equivalent) is not something Git specific.
>> It is present also in other version control systems, among others:
>>
>> * Mercurial: as 'shelve' extension (in core since 1.8)
>> * Bazaar: as 'bzr shelve' command
>> * Fossil: as 'fossil stash' command (with subcommands)
>> * Subversion: Shelve planned for 1.10 (2017?)
>
> Do these other VCSs have the same "Switching branches" misfit? Do you
> usually need to stash if you want to switch with uncommitted changes? I know
> Mercurial has the same problem since ``bookmarks'' are like Git branches, so
> it makes sense for them to have added something like stashing (if otherwise
> switching with uncommitted changes would be very difficult).

I suspect that all those are inspired by each other, and that
they all use 'uncommitted changes are not tied to a branch'
paradigm, which allows for creating a branch for changes
after a fact (when it turns out that it would take more than
one commit to implement the feature) quite easy.

>> I would say that 'stash' could be considered about isolating work on
>> different features, different sub-branch sized parallel work.

Note that 'isolating work' is missing from your list of purposes
of a version control system; though it is fairly obvious.

>
> That sounds a lot like having independent lines of development, which is
> what branches are supposed to be for

Those are sub-commit changes. Branches are composed of
commits. But I agree that this may be a bit of a stretch in
trying to find a high-level purpose for stash (rather than it being
a convenience feature). As I said below...

>> But it might be that stash doesn't have connection with highlevel
>> purposes for version control, and that it is purely convenience
>> feature.  Just playing the role of Advocatus Diaboli (important in
>> scientific works, isn' it?)...

[...]

>>>> | 7. Gitless
[...]
>>>> |  Also, there
>>>> | is no possible way of getting in a “detached head” state; at
>>>> | any time, the user is always working on some branch (the
>>>> | “current” branch). Head is a per-branch reference to the last
>>>> | commit of the branch.
[...]
>>>> [...] during some long lived multi-step operations, like bisect
>>>> or interactive rebase, you are not really on any branch,
>>>
>>> In Gitless we don't have bisect but for rebase (fuse in Gitless) we
>>> record the current branch.
>>
>> No bisect?  This is very useful feature.  Though it might be done
>> without detached HEAD, but with specialized pseudo-branch 'bisect' (as
>> it was done in earlier versions of Git, or maybe even now).
>>
>> Anyway, for [interactive] rebase / transplant / graft / fuse you need
>> to be able to abort an operation and return to the state before
>> staring rebase.  Though you can or do solve this by remembering
>> the starting position.
>
> Yes, Gitless remembers the starting position. We should be able to get
> bisect working too in the same way. Internally, the head is detached but
> that's irrelevant to the user. As far as the user is concerned she's still
> working on the current branch.

There are quite a few problems with "remember the starting position"
approach. For one, the rebase / fuse operation should be recorded as
whole in the branch reflog (assuming that you implement this feature).
Working on a branch would ordinarily mean that all those intermediate
steps would be recorded (well, they are, but in separate reflog, namely
HEAD reflog).

The second issue is that you wouldn't want for your partially done rebase
to be visible; for example, you would want for 'git push' to not include
partial work (which might get abandoned).

I suppose all this can be solved without user-visible detached HEAD...


I have one more comment and one more issue about the article in
general.

First, while the entry into a list of version control systems (or even
interfaces to them) is hard, among others because of network effects,
it should be much easier to try to come up with a GUI or IDE plugin
starting from the same principles. Also with GUI there is not much
problem if you don;t implement everything; users would just fall back
on command line of underlying version control system.

Second, I think at least some of the concepts phase would not be
possible when Git was starting to be created. At the beginning, we
didn't know much about how distributed version control systems would
be used. For example, the very useful "topic branch" workflow was
not even imagined. Mercurial, which was created in parallel and at
the same time as Git, started with "clone to create a new branch"
paradigm!  Unfortunately the curse of "worse is better" is often many
misfits in paid for lots of power.

Best regards,
-- 
Jakub Narębski

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Konstantin Khomoutov]

On Fri, 30 Sep 2016 19:14:13 +0300
Konstantin Khomoutov <kostix+git@007spb.ru> wrote:

> The "It Will Never Work in Theory" blog has just posted a summary of a
> study which tried to identify shortcomings in the design of Git.
> 
> In the hope it might be interesting, I post this summary here.
> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html

I think it would be cool if someone among subscribers could post a link
to our discussion thread [2] back onto that page.  Unfortunatrly that
requires a Disqus login which I don't have, so may be someone in
possession of such login could do that? ;-)

2. https://public-inbox.org/git/ce42f934-4a94-fa29-cff0-5ebb0f004eb5@gmail.com/T/#e95875b7940512b432ab2e29b3dd50ca448df9720

Re: "Purposes, Concepts,Misfits, and a Redesign of Git" (a research paper)

[Jakub Narębski]

W dniu 07.10.2016 o 19:52, Konstantin Khomoutov pisze:
> On Fri, 30 Sep 2016 19:14:13 +0300
> Konstantin Khomoutov <kostix+git@007spb.ru> wrote:
> 
>> The "It Will Never Work in Theory" blog has just posted a summary of a
>> study which tried to identify shortcomings in the design of Git.
>>
>> In the hope it might be interesting, I post this summary here.
>> URL: http://neverworkintheory.org/2016/09/30/rethinking-git.html
> 
> I think it would be cool if someone among subscribers could post a link
> to our discussion thread [2] back onto that page.  Unfortunatrly that
> requires a Disqus login which I don't have, so may be someone in
> possession of such login could do that? ;-)
> 
> 2. https://public-inbox.org/git/ce42f934-4a94-fa29-cff0-5ebb0f004eb5@gmail.com/T/#e95875b7940512b432ab2e29b3dd50ca448df9720

Posted.  Thanks for the idea.

-- 
Jakub Narębski