Re: Git as data archive

[Andreas Kalz <andreas-kalz@gmx.de>]

Hi,

thanks to you both.

@Thomas: are you Thomas Braun who studied at FH Regensburg?

Well, currently the .git repository is 715GB and the maximum file size
is 9.5GB, but I did not get error messages due to that even if the
performance is quite low. The biggest pack* file is 24GB. There are some
files which are modified, but most are not modified.

My question came up as I did not find a documentation about limits of
git, only a lot of entries about github and forum users who are
discussing about old bugs of git. I read about git-lfs and also that it
is not working very stable, due to that I did not use it yet.

How can the delta compression settings and/or the big filethreshold
limits be modified?
Thanks in advance.

All the best,
Andreas


Am 07.12.2019 um 19:04 schrieb Thomas Braun:
> On 07.12.2019 17:54, Philip Oakley wrote:
>> Hi Andreas,
>>
>> On 06/12/2019 18:54, Andreas Kalz wrote:
>>> Hello,
>>> I am using git as archive and versioning also for photos. Apart from
>>> performance issues, I wanted to ask if there are hard limits and
>>> configurable limits (how to configure?) for maximum single file size and
>>> maximum .git archive size (Windows 64 Bit system)?
>>> Thanks in advance for your answer.
>>> All the best,
>>> Andreas
>> On Git the file size is currently limited to size of `long`, rather than
>> `size_t`. Hence on Git-for Windows the size limit is 32bit ~4GiB
>>
>> Any change will be a big change as it ripples through many places in the
>> code base and, for some, will feel 'wrong'. I did some work [1-4] on top
>> of those of many others that was almost there, but...
> Adding to what Philip said. On Windows the size of exported archives
> (git archive) is currently also limited to 4GB. The reason being also
> the long vs size_t issue (which is not present on linux though).
>
> So if you can switch to Linux or even MacOSX these issues are gone.
>
> The number of files in .git, only the number packfiles would be of
> interest here I guess, do not have the long vs size_t issue. So
> packfiles can be larger than 4GB on 64bit Windows (with 64bit git of
> course).
>
> But depending on how large the biggest files are, it might be worth
> tweaking some of the settings, so that the created packfiles are
> readable on all platforms. I once created a repo on linux which could
> not be checked on windows, and that is a bit annoying.
>
> So the questions are how large is each file? And what repository size do
> you expect? Are we talking about 20MB files and 10GB repository? Or a
> factor 100 more? And are you just adding files or are you modifying the
> added files? Depending on the file sizes it might then also be
> beneficial to tweak the delta compression settings and/or the big file
> threshold limits.
>
> Thomas
>
>> The alternative is git-lfs, which I don't personally use (see [4]).
>>
>> Philip
>>
>> [1] https://github.com/git-for-windows/git/pull/2179
>> [2] https://github.com/gitgitgadget/git/pull/115
>> [3] https://github.com/git-for-windows/git/issues/1063
>> [4] https://github.com/git-lfs/git-lfs/issues/2434
>>
>>