tying files to git repository

tying files to git repository

[Jim Edwards]

Hi,

I am a developer of scientific software, as science software we expect
and encourage users to modify source code in order to customize their
experiments.   The mechanism which has been developed to do this
predates having source code in git and we are trying to figure out a
way to minimize changes to the scientists workflow, while leveraging
the power of git to improve the process.  In the workflow the
scientist creates a 'case' using script in the repository to create a
directory structure from which they will conduct their experiment.
Part of that directory structure is a SourceMods directory where the
user can drop modified source files that will be compiled in place of
a file of the same name in the source tree.  These files are sometimes
long lived and passed from case to case and even user to user and it
is not hard to have the files get out of sync with the source tree.
We have discussed at length removing the SourceMods capability and
requiring scientists to create branches in git, but there is a lot of
resistance to this in the community.   What I would like to explore is
allowing scientists to keep the method that they are used to but at
the same time tying these modified files to their history in git.   Is
there a way to get the git metadata associated with an individual file
so that we can treat that file as if it were in the repo?

-- 
Jim Edwards

CESM Software Engineer
National Center for Atmospheric Research
Boulder, CO

Re: tying files to git repository

[Philip Oakley]

Hi Jim

This 
(https://public-inbox.org/git/CAPuR+ZhwnHCp8j76PscuBqG2rCLkgG0+6Y3WwLgNRhaoj4OR9A@mail.gmail.com/) 
has been lying around for a long while..

On 11/11/2019 15:28, Jim Edwards wrote:
> Hi,
>
> I am a developer of scientific software, as science software we expect
> and encourage users to modify source code in order to customize their
> experiments.
Sounds an ideal usage for git - giving back control to the users so they 
can easily maintain their versions.
> The mechanism which has been developed to do this
> predates having source code in git and we are trying to figure out a
> way to minimize changes to the scientists workflow, while leveraging
> the power of git to improve the process.
Good, though sounds like you will need some terminology mapping.. Is 
there a public reference to the method?
> In the workflow the
> scientist creates a 'case' using script in the repository to create a
> directory structure from which they will conduct their experiment.
In what sense do you use "case"?
  e.g. a [single] suitcase that holds a choice of clothes to wear / 
experimental methods;
Or, a long list of experimental setups, each with a name, selecting a 
'case' statement (like a software 'case' statement);
Or, a use-case that gives a half complete suggestion about how it may 
work, but with some details still to be filled in due to lack of space 
on the post-it note..?

Given that the scientists create a 'directory structure' (containing 
files?) for each experiment, this sounds very much like creating a 
'branch' (line of development) from the initial template of that 
structure, and as they develop their experiment's directory structure, 
they record their development in 'commits' on that branch (and sub 
branches if they are looking at alternatives). Finally, when they have a 
good structure ready, they can 'tag' that commit so it's easy to find.


> Part of that directory structure is a SourceMods directory where the
> user can drop modified source files that will be compiled in place of
> a file of the same name in the source tree.
This 'dropping' has a strong _conceptual_ similarity to the staging area 
or 'index', where git users 'add' files that they feel are ready to an 
area that is used (like an outbox awaiting collection) as a temporary 
holding area waiting till all the bits are ready and waiting before they 
commit the ensemble.

So this "SourceMods" is very similar to the staging area, except that in 
your case it sounds like it is a specific place, while in git it is more 
conceptual as the user will 'git add <file>', and that change is 
registered in "the index" (a local file in a hidden .git directory), 
ready for the big commit.

Behind the scenes, a copy of the file is saved (in the object store) and 
hashed ready for inclusion in the commit hierarchy . Later the files 
(objects) stored in the object store are 'packed' resulting in a very 
compact storage, particularly for source files. The git repository can 
be 'pushed' to other servers, and other repositories 'fetched' from 
servers (and mixed together if they have common ancestry).


> These files are sometimes
> long lived and passed from case to case and even user to user and it
> is not hard to have the files get out of sync with the source tree.
In Git, because the current files stay 'in place', you can start a 
branch (new experiment definition) from anywhere in any line of 
development. You/they simply checkout that particular commit and, voila, 
all the files are back as they were exactly.
> We have discussed at length removing the SourceMods capability and
> requiring scientists to create branches in git, but there is a lot of
> resistance to this in the community.
Most of that will be fear of the unknown and the unfamiliarity of the 
git terminology. There can also be confusion about how Git has changed 
the old ways of working. Because you get 100% verification and 
validation you no longer need to worry about requiring a central golden 
reference store (though usually the "organisation" will want to have a 
_copy_ ;-)
> What I would like to explore is
> allowing scientists to keep the method that they are used to but at
> the same time tying these modified files to their history in git.
The key part will be in how you map what they already do and know to the 
git commands and structure, and how you show them that it will remove a 
lot of the pain points and bottle necks.
> Is
> there a way to get the git metadata associated with an individual file
> so that we can treat that file as if it were in the repo?
>
That [mental model view] way is a Sisyphean task, a never ending up-hill 
struggle. With a few careful words (mapping out solutions to their pain 
points) you should be able to get the scientists to pester you to 
implement git sooner rather than later. Choose the lead experimenters of 
git carefully.

-- 
Philip