Removing data from repository

Removing data from repository

[Naum Derzhi]

Greetings,

I have this problem: years ago one of our developers committed a large (Gigabytes) piece of binary data into our project repository. This should not have been done, but it happened. (Honest, it was not me). We never needed this data in the repository.

Using git rm removes these files from the working tree, but they are still somewhere in the repository, so when we clone, we transfer gigabytes of unneeded data. 

Is it possible to fix this problem?

Thank you
 
 
Naum Derzhi
Chief Scientific Adviser, Physics
 
3000 N Sam Houston Pkwy E, Technology Center, Office T1241H
Houston, TX 77032

Office: +1 (281) 871 3278

________________________________________
This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.

Re: Removing data from repository

[Jeff King]

On Thu, Jan 24, 2019 at 02:51:50PM +0000, Naum Derzhi wrote:

> I have this problem: years ago one of our developers committed a large
> (Gigabytes) piece of binary data into our project repository. This
> should not have been done, but it happened. (Honest, it was not me).
> We never needed this data in the repository.
> 
> Using git rm removes these files from the working tree, but they are
> still somewhere in the repository, so when we clone, we transfer
> gigabytes of unneeded data. 
> 
> Is it possible to fix this problem?

You'll have to rewrite the offending history. You can use
git-filter-branch. See especially these sections of the manpage:

  https://git-scm.com/docs/git-filter-branch#_examples

  https://git-scm.com/docs/git-filter-branch#_checklist_for_shrinking_a_repository

as well as the warning in the DESCRIPTION section:

  WARNING! The rewritten history will have different object names for
  all the objects and will not converge with the original branch. You
  will not be able to easily push and distribute the rewritten branch on
  top of the original branch. Please do not use this command if you do
  not know the full implications, and avoid using it anyway, if a simple
  single commit would suffice to fix your problem. (See the "RECOVERING
  FROM UPSTREAM REBASE" section in git-rebase(1) for further information
  about rewriting published history.)

You may also want to check out the BFG repo cleaner[1], as separate
project that handles this case a little more simply (it doesn't save you
from dealing with rewritten history, but it does avoid having to learn
filter-branch's flexible but confusing syntax).

-Peff

[1] https://rtyley.github.io/bfg-repo-cleaner/

Re: Removing data from repository

[Paul Smith]

On Thu, 2019-01-24 at 14:51 +0000, Naum Derzhi wrote:
> Greetings,
> 
> I have this problem: years ago one of our developers committed a
> large (Gigabytes) piece of binary data into our project repository.
> This should not have been done, but it happened. (Honest, it was not
> me). We never needed this data in the repository.
> 
> Using git rm removes these files from the working tree, but they are
> still somewhere in the repository, so when we clone, we transfer
> gigabytes of unneeded data. 
> 
> Is it possible to fix this problem?

It is possible, but it isn't pretty.  By waiting so long to try to fix
it you've compounded the impacts dramatically unfortunately.  By
removing that file you will in effect be rewriting the history of your
repository starting with the commit that introduced the problematic
file, that will be removed, all the way forward.

That means that every clone of the repository will have to be, at the
very least, "hard-reset" and preferably (to avoid accidentally re-
introducing the bad file) re-cloned from scratch.

As for actually removing the file, you can find information on how to
do this all over the Google.  Here's some reasonable advice from
StackOverflow:


https://stackoverflow.com/questions/2100907/how-to-remove-delete-a-large-file-from-commit-history-in-git-repository

RE: [EXTERNAL] Re: Removing data from repository

[Naum Derzhi]

Thank you very much. We'll try

 
 
Naum Derzhi
Chief Scientific Adviser, Physics
 
3000 N Sam Houston Pkwy E, Technology Center, Office T1241H
Houston, TX 77032

Office: +1 (281) 871 3278

Follow Halliburton: LinkedIn | Facebook | Twitter | YouTube | Blog

 

                                                                              




This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.

-----Original Message-----
From: Jeff King <peff@peff.net> 
Sent: Thursday, January 24, 2019 9:18 AM
To: Naum Derzhi <Naum.Derzhi@halliburton.com>
Cc: git@vger.kernel.org
Subject: [EXTERNAL] Re: Removing data from repository

External Sender: Use caution with links/attachments.



On Thu, Jan 24, 2019 at 02:51:50PM +0000, Naum Derzhi wrote:

> I have this problem: years ago one of our developers committed a large
> (Gigabytes) piece of binary data into our project repository. This 
> should not have been done, but it happened. (Honest, it was not me).
> We never needed this data in the repository.
>
> Using git rm removes these files from the working tree, but they are 
> still somewhere in the repository, so when we clone, we transfer 
> gigabytes of unneeded data.
>
> Is it possible to fix this problem?

You'll have to rewrite the offending history. You can use git-filter-branch. See especially these sections of the manpage:

  https://urldefense.proofpoint.com/v2/url?u=https-3A__git-2Dscm.com_docs_git-2Dfilter-2Dbranch-23-5Fexamples&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=_cdkm_E9dmM7yuNx0_5fpQKrV9wtnG4LPr8uooiV3cI&e=

  https://urldefense.proofpoint.com/v2/url?u=https-3A__git-2Dscm.com_docs_git-2Dfilter-2Dbranch-23-5Fchecklist-5Ffor-5Fshrinking-5Fa-5Frepository&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=iBeuLkNNZbLqcAmKNw6JI7TnSt6KDObIDaBaEgQFcDQ&e=

as well as the warning in the DESCRIPTION section:

  WARNING! The rewritten history will have different object names for
  all the objects and will not converge with the original branch. You
  will not be able to easily push and distribute the rewritten branch on
  top of the original branch. Please do not use this command if you do
  not know the full implications, and avoid using it anyway, if a simple
  single commit would suffice to fix your problem. (See the "RECOVERING
  FROM UPSTREAM REBASE" section in git-rebase(1) for further information
  about rewriting published history.)

You may also want to check out the BFG repo cleaner[1], as separate project that handles this case a little more simply (it doesn't save you from dealing with rewritten history, but it does avoid having to learn filter-branch's flexible but confusing syntax).

-Peff

[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__rtyley.github.io_bfg-2Drepo-2Dcleaner_&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=ADIQXtQrHUv3xXudfCN94eL00x0w0vr8uK2FQMpA2dI&e=

RE: [EXTERNAL] Re: Removing data from repository

[Naum Derzhi]

Thank you

 
 
Naum Derzhi
Chief Scientific Adviser, Physics
 
3000 N Sam Houston Pkwy E, Technology Center, Office T1241H
Houston, TX 77032

Office: +1 (281) 871 3278

Follow Halliburton: LinkedIn | Facebook | Twitter | YouTube | Blog

 

                                                                              




This e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.


-----Original Message-----
From: Paul Smith <paul@mad-scientist.net> 
Sent: Thursday, January 24, 2019 9:23 AM
To: Naum Derzhi <Naum.Derzhi@halliburton.com>; git@vger.kernel.org
Subject: [EXTERNAL] Re: Removing data from repository

External Sender: Use caution with links/attachments.



On Thu, 2019-01-24 at 14:51 +0000, Naum Derzhi wrote:
> Greetings,
>
> I have this problem: years ago one of our developers committed a large 
> (Gigabytes) piece of binary data into our project repository.
> This should not have been done, but it happened. (Honest, it was not 
> me). We never needed this data in the repository.
>
> Using git rm removes these files from the working tree, but they are 
> still somewhere in the repository, so when we clone, we transfer 
> gigabytes of unneeded data.
>
> Is it possible to fix this problem?

It is possible, but it isn't pretty.  By waiting so long to try to fix it you've compounded the impacts dramatically unfortunately.  By removing that file you will in effect be rewriting the history of your repository starting with the commit that introduced the problematic file, that will be removed, all the way forward.

That means that every clone of the repository will have to be, at the very least, "hard-reset" and preferably (to avoid accidentally re- introducing the bad file) re-cloned from scratch.

As for actually removing the file, you can find information on how to do this all over the Google.  Here's some reasonable advice from
StackOverflow:


https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_2100907_how-2Dto-2Dremove-2Ddelete-2Da-2Dlarge-2Dfile-2Dfrom-2Dcommit-2Dhistory-2Din-2Dgit-2Drepository&d=DwICaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=dmUTLEwyrdm6hRWOzlTFqBaVv4_WJoUWDed9sG4uDQ0&s=QrNGuuWuNZQqkSksRUK9sfcG5iwVZIxmE6UZcneRxHg&e=