Move some files, with all history, from one project into a new one

Move some files, with all history, from one project into a new one

[Kerry, Richard]

I would like to move some files, from the project in which they have always resided into  a new project.  I would like to keep all their history.
I don't want to waste space by also moving the rest of the old project's history, or historical file contents.

We have a long-standing project, main-system.  A group of files within it are designated as demo-system (sometimes whole folders, sometimes files within folders with other files).
All development is done on the master branch.
It has now transpired that the demo-system files are necessary, but no longer want to be within main-system but in a new project of their own.  So I would like to move them from the main-system repo into a new repo of their own.  I do want all their history but I don't want to take any contents from other files from main-system, which is quite big.
Please can someone advise if there is a particular method I should best use for this.
Do I create a new branch, then delete the rest of main-system leaving only what I want?  Surely if I do that then I would end up with all the main-system file data within the new repo, which would expand its size.
Is there a recommended way to extract certain files with their histories?

Regards,
Richard.

Re: Move some files, with all history, from one project into a new one

[Jeff King]

On Wed, Apr 15, 2020 at 08:31:35AM +0000, Kerry, Richard wrote:

> I would like to move some files, from the project in which they have
> always resided into  a new project.  I would like to keep all their
> history.  I don't want to waste space by also moving the rest of the
> old project's history, or historical file contents.

Try git-filter-branch's --subdirectory repository, which is designed to
do exactly this.

Or the much newer (and faster) git-filter-repo:

  https://github.com/newren/git-filter-repo

> Do I create a new branch, then delete the rest of main-system leaving
> only what I want?  Surely if I do that then I would end up with all
> the main-system file data within the new repo, which would expand its
> size.  Is there a recommended way to extract certain files with their
> histories?

Correct; just deleting files and creating a new commit will still carry
them in the history. You have to rewrite the old commits.

-Peff

Re: Move some files, with all history, from one project into a new one

[Taylor Blau]

On Wed, Apr 15, 2020 at 11:11:28AM -0400, Jeff King wrote:
> On Wed, Apr 15, 2020 at 08:31:35AM +0000, Kerry, Richard wrote:
>
> > I would like to move some files, from the project in which they have
> > always resided into  a new project.  I would like to keep all their
> > history.  I don't want to waste space by also moving the rest of the
> > old project's history, or historical file contents.
>
> Try git-filter-branch's --subdirectory repository, which is designed to
> do exactly this.
>
> Or the much newer (and faster) git-filter-repo:
>
>   https://github.com/newren/git-filter-repo

For what it's worth, Elijah has provided some excellent documentation on
how to use git-filter-repo to do exactly this here:

  https://github.com/newren/git-filter-repo#solving-this-with-filter-repo

Thanks,
Taylor

Re: Move some files, with all history, from one project into a new one

[Elijah Newren]

On Wed, Apr 15, 2020 at 8:49 AM Taylor Blau <me@ttaylorr.com> wrote:
>
> On Wed, Apr 15, 2020 at 11:11:28AM -0400, Jeff King wrote:
> > On Wed, Apr 15, 2020 at 08:31:35AM +0000, Kerry, Richard wrote:
> >
> > > I would like to move some files, from the project in which they have
> > > always resided into  a new project.  I would like to keep all their
> > > history.  I don't want to waste space by also moving the rest of the
> > > old project's history, or historical file contents.
> >
> > Try git-filter-branch's --subdirectory repository, which is designed to
> > do exactly this.
> >
> > Or the much newer (and faster) git-filter-repo:
> >
> >   https://github.com/newren/git-filter-repo
>
> For what it's worth, Elijah has provided some excellent documentation on
> how to use git-filter-repo to do exactly this here:
>
>   https://github.com/newren/git-filter-repo#solving-this-with-filter-repo

That particular example might be for a different case than what
Richard requested, though.  Let's say the original repo had a file
structure like the following:

   module/
      foo.c
      bar.c
   otherDir/
      blah.config
      stuff.txt
   zebra.jpg

If the request is to e.g. take module/ and all files within it with
their history and make a new repository out of it, with module/ being
remapped to the root of the repository, then you would want:
   git filtrer-repo --subdirectory-filter module
and yes, this looks exactly like filter-branch; that's the one flag I
copied from it.  So this one usecase maps directly between the two
tools.

In contrast, if you wanted to keep all files from the original repo
but move everything into a subdirectory named "myProject" (so that
e.g. module/foo.c became myProject/module/foo.c), possibly in
preparation for merging your repo into some larger monorepo, then
you'd want to pass `--to-subdirectory-filter myProject` as in the link
you pointed out.  filter-branch doesn't have an equivalent.


Hope that helps,
Elijah

RE: Move some files, with all history, from one project into a new one

[Kerry, Richard]

Thanks All.
I've cloned git-filter-repo and I shall shortly give it a try.

First step is to get Python 3 installed (I'm on Windows).

Regards,
Richard.



-----Original Message-----
From: Elijah Newren <newren@gmail.com> 
Sent: 16 April 2020 07:06
To: Taylor Blau <me@ttaylorr.com>
Cc: Jeff King <peff@peff.net>; Kerry, Richard <richard.kerry@atos.net>; git@vger.kernel.org
Subject: Re: Move some files, with all history, from one project into a new one

Caution! External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.

On Wed, Apr 15, 2020 at 8:49 AM Taylor Blau <me@ttaylorr.com> wrote:
>
> On Wed, Apr 15, 2020 at 11:11:28AM -0400, Jeff King wrote:
> > On Wed, Apr 15, 2020 at 08:31:35AM +0000, Kerry, Richard wrote:
> >
> > > I would like to move some files, from the project in which they 
> > > have always resided into  a new project.  I would like to keep all 
> > > their history.  I don't want to waste space by also moving the 
> > > rest of the old project's history, or historical file contents.
> >
> > Try git-filter-branch's --subdirectory repository, which is designed 
> > to do exactly this.
> >
> > Or the much newer (and faster) git-filter-repo:
> >
> >   
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgi
> > thub.com%2Fnewren%2Fgit-filter-repo&amp;data=02%7C01%7Crichard.kerry
> > %40atos.net%7Cf28b944624e04084395f08d7e1cc7b35%7C33440fc6b7c7412cbb7
> > 30e70b0198d5a%7C0%7C0%7C637226140664962267&amp;sdata=pb2EnEADuC9Bdi5
> > JU2So3DK6VyEcsEH5X5kybFAw9%2FM%3D&amp;reserved=0
>
> For what it's worth, Elijah has provided some excellent documentation 
> on how to use git-filter-repo to do exactly this here:
>
>   
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> ub.com%2Fnewren%2Fgit-filter-repo%23solving-this-with-filter-repo&amp;
> data=02%7C01%7Crichard.kerry%40atos.net%7Cf28b944624e04084395f08d7e1cc
> 7b35%7C33440fc6b7c7412cbb730e70b0198d5a%7C0%7C0%7C637226140664962267&a
> mp;sdata=h03FogXX0iuqHB9d49CltOmzNkwGK66chC5ZvhOyWfs%3D&amp;reserved=0

That particular example might be for a different case than what Richard requested, though.  Let's say the original repo had a file structure like the following:

   module/
      foo.c
      bar.c
   otherDir/
      blah.config
      stuff.txt
   zebra.jpg

If the request is to e.g. take module/ and all files within it with their history and make a new repository out of it, with module/ being remapped to the root of the repository, then you would want:
   git filtrer-repo --subdirectory-filter module and yes, this looks exactly like filter-branch; that's the one flag I copied from it.  So this one usecase maps directly between the two tools.

In contrast, if you wanted to keep all files from the original repo but move everything into a subdirectory named "myProject" (so that e.g. module/foo.c became myProject/module/foo.c), possibly in preparation for merging your repo into some larger monorepo, then you'd want to pass `--to-subdirectory-filter myProject` as in the link you pointed out.  filter-branch doesn't have an equivalent.


Hope that helps,
Elijah