From: Lars Schneider <larsxschneider@gmail.com>
To: Eric Sunshine <sunshine@sunshineco.com>
Cc: Git List <git@vger.kernel.org>, Jeff King <peff@peff.net>,
Taylor Blau <me@ttaylorr.com>,
"brian m. carlson" <sandals@crustytoothpaste.net>
Subject: Re: Import/Export as a fast way to purge files from Git?
Date: Sun, 23 Sep 2018 17:58:14 +0200 [thread overview]
Message-ID: <DE9CF60B-7DCB-4B82-9C96-663E145CAD56@gmail.com> (raw)
In-Reply-To: <CAPig+cTLjThK4CVzfgV=Uk5OumpjhaQD_YNXmg7pNtkkUFiiyQ@mail.gmail.com>
> On Sep 23, 2018, at 4:55 PM, Eric Sunshine <sunshine@sunshineco.com> wrote:
>
> On Sun, Sep 23, 2018 at 9:05 AM Lars Schneider <larsxschneider@gmail.com> wrote:
>> I recently had to purge files from large Git repos (many files, many commits).
>> The usual recommendation is to use `git filter-branch --index-filter` to purge
>> files. However, this is *very* slow for large repos (e.g. it takes 45min to
>> remove the `builtin` directory from git core). I realized that I can remove
>> files *way* faster by exporting the repo, removing the file references,
>> and then importing the repo (see Perl script below, it takes ~30sec to remove
>> the `builtin` directory from git core). Do you see any problem with this
>> approach?
>
> A couple comments:
>
> For purging files from a history, take a look at BFG[1] which bills
> itself as "a simpler, faster alternative to git-filter-branch for
> cleansing bad data out of your Git repository history".
Yes, BFG is great. Unfortunately, it requires Java which is not available
on every system I have to work with. I required a solution that would work
in every Git environment. Hence the Perl script :-)
> The approach of exporting to a fast-import stream, modifying the
> stream, and re-importing is quite reasonable.
Thanks for the confirmation!
> However, rather than
> re-inventing, take a look at reposurgeon[2], which allows you to do
> major surgery on fast-import streams. Not only can it purge files from
> a repository, but it can slice, dice, puree, and saute pretty much any
> attribute of a repository.
Wow. Reposurgeon looks very interesting. Thanks a lot for the pointer!
Cheers,
Lars
> [1]: https://rtyley.github.io/bfg-repo-cleaner/
> [2]: http://www.catb.org/esr/reposurgeon/
next prev parent reply other threads:[~2018-09-23 15:58 UTC|newest]
Thread overview: 90+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-23 13:04 Import/Export as a fast way to purge files from Git? Lars Schneider
2018-09-23 14:55 ` Eric Sunshine
2018-09-23 15:58 ` Lars Schneider [this message]
2018-09-23 15:53 ` brian m. carlson
2018-09-23 17:04 ` Jeff King
2018-09-24 17:24 ` Elijah Newren
2018-10-31 19:15 ` Lars Schneider
2018-11-01 7:12 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 00/10] fast export and import fixes and features Elijah Newren
2018-11-11 6:23 ` [PATCH 01/10] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-11 6:33 ` Jeff King
2018-11-11 6:23 ` [PATCH 02/10] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-11 6:36 ` Jeff King
2018-11-11 7:17 ` Elijah Newren
2018-11-13 23:25 ` Elijah Newren
2018-11-13 23:39 ` Jonathan Nieder
2018-11-14 0:02 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 03/10] fast-export: use value from correct enum Elijah Newren
2018-11-11 6:36 ` Jeff King
2018-11-11 20:10 ` Ævar Arnfjörð Bjarmason
2018-11-12 9:12 ` Ævar Arnfjörð Bjarmason
2018-11-12 11:31 ` Jeff King
2018-11-11 6:23 ` [PATCH 04/10] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-11 6:44 ` Jeff King
2018-11-11 7:38 ` Elijah Newren
2018-11-12 12:32 ` Jeff King
2018-11-12 22:50 ` brian m. carlson
2018-11-13 14:38 ` Jeff King
2018-11-11 6:23 ` [PATCH 05/10] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-11 6:47 ` Jeff King
2018-11-11 6:23 ` [PATCH 06/10] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-11 6:53 ` Jeff King
2018-11-11 8:01 ` Elijah Newren
2018-11-12 12:45 ` Jeff King
2018-11-12 15:36 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 07/10] fast-export: ensure we export requested refs Elijah Newren
2018-11-11 7:02 ` Jeff King
2018-11-11 8:20 ` Elijah Newren
2018-11-11 6:23 ` [PATCH 08/10] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-11 7:11 ` Jeff King
2018-11-11 6:23 ` [PATCH 09/10] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-11 7:20 ` Jeff King
2018-11-11 8:32 ` Elijah Newren
2018-11-12 12:53 ` Jeff King
2018-11-12 15:46 ` Elijah Newren
2018-11-12 16:31 ` Jeff King
2018-11-11 6:23 ` [PATCH 10/10] fast-export: add --always-show-modify-after-rename Elijah Newren
2018-11-11 7:23 ` Jeff King
2018-11-11 8:42 ` Elijah Newren
2018-11-12 12:58 ` Jeff King
2018-11-12 18:08 ` Elijah Newren
2018-11-13 14:45 ` Jeff King
2018-11-13 17:10 ` Elijah Newren
2018-11-14 7:14 ` Jeff King
2018-11-11 7:27 ` [PATCH 00/10] fast export and import fixes and features Jeff King
2018-11-11 8:44 ` Elijah Newren
2018-11-12 13:00 ` Jeff King
2018-11-14 0:25 ` [PATCH v2 00/11] " Elijah Newren
2018-11-14 0:25 ` [PATCH v2 01/11] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-14 0:25 ` [PATCH v2 02/11] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-14 0:25 ` [PATCH v2 03/11] fast-export: use value from correct enum Elijah Newren
2018-11-14 0:25 ` [PATCH v2 04/11] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-14 19:17 ` SZEDER Gábor
2018-11-14 23:13 ` Elijah Newren
2018-11-14 0:25 ` [PATCH v2 05/11] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-14 0:25 ` [PATCH v2 06/11] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-14 0:25 ` [PATCH v2 07/11] fast-export: ensure we export requested refs Elijah Newren
2018-11-14 0:25 ` [PATCH v2 08/11] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-14 19:27 ` SZEDER Gábor
2018-11-14 23:16 ` Elijah Newren
2018-11-14 0:25 ` [PATCH v2 09/11] fast-import: remove unmaintained duplicate documentation Elijah Newren
2018-11-14 0:25 ` [PATCH v2 10/11] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-14 0:26 ` [PATCH v2 11/11] fast-export: add --always-show-modify-after-rename Elijah Newren
2018-11-14 7:25 ` [PATCH v2 00/11] fast export and import fixes and features Jeff King
2018-11-16 7:59 ` [PATCH v3 " Elijah Newren
2018-11-16 7:59 ` [PATCH v3 01/11] fast-export: convert sha1 to oid Elijah Newren
2018-11-16 7:59 ` [PATCH v3 02/11] git-fast-import.txt: fix documentation for --quiet option Elijah Newren
2018-11-16 7:59 ` [PATCH v3 03/11] git-fast-export.txt: clarify misleading documentation about rev-list args Elijah Newren
2018-11-16 7:59 ` [PATCH v3 04/11] fast-export: use value from correct enum Elijah Newren
2018-11-16 7:59 ` [PATCH v3 05/11] fast-export: avoid dying when filtering by paths and old tags exist Elijah Newren
2018-11-16 7:59 ` [PATCH v3 06/11] fast-export: move commit rewriting logic into a function for reuse Elijah Newren
2018-11-16 7:59 ` [PATCH v3 07/11] fast-export: when using paths, avoid corrupt stream with non-existent mark Elijah Newren
2018-11-16 7:59 ` [PATCH v3 08/11] fast-export: ensure we export requested refs Elijah Newren
2018-11-16 7:59 ` [PATCH v3 09/11] fast-export: add --reference-excluded-parents option Elijah Newren
2018-11-16 7:59 ` [PATCH v3 10/11] fast-import: remove unmaintained duplicate documentation Elijah Newren
2018-11-16 7:59 ` [PATCH v3 11/11] fast-export: add a --show-original-ids option to show original names Elijah Newren
2018-11-16 12:29 ` SZEDER Gábor
2018-11-16 8:50 ` [PATCH v3 00/11] fast export and import fixes and features Jeff King
2018-11-12 9:17 ` Import/Export as a fast way to purge files from Git? Ævar Arnfjörð Bjarmason
2018-11-12 15:34 ` Elijah Newren
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DE9CF60B-7DCB-4B82-9C96-663E145CAD56@gmail.com \
--to=larsxschneider@gmail.com \
--cc=git@vger.kernel.org \
--cc=me@ttaylorr.com \
--cc=peff@peff.net \
--cc=sandals@crustytoothpaste.net \
--cc=sunshine@sunshineco.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).