From: Son Luong Ngoc <sluongng@gmail.com>
To: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Cc: git <git@vger.kernel.org>
Subject: Re: [Summit topic] Crazy (and not so crazy) ideas
Date: Thu, 21 Oct 2021 14:30:50 +0200 [thread overview]
Message-ID: <CAL3xRKck=cbdo=7gqv0q=MLjn+5J1ES6fbzHP1mEj90pLdUdAA@mail.gmail.com> (raw)
In-Reply-To: <nycvar.QRO.7.76.6.2110211144490.56@tvgsbejvaqbjf.bet>
Hi,
On Thu, Oct 21, 2021 at 1:56 PM Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
>
> This session was led by Elijah Newren. Supporting cast: Johannes "Dscho"
> Schindelin, Jonathan Tan, Jonathan "jrnieder" Nieder, brian m. carlson,
> Jeff "Peff" King, Ævar Arnfjörð Bjarmason, Emily Shaffer, CB Bailey,
> Taylor Blau, and Philip Oakley.
>
> Notes:
>
...
>
> * Biggest idea: there are a lot of people who version control things via
> tarballs or .zip files per version. This prevents history from
> compressing well. Some people check in those compressed files into Git
> for purposes of history.
>
...
>
> * Old suggestion of a “blob-tree” type that allows storing a single
> index entry that corresponds to multiple trees and blobs in the
> background, possibly.
>
> * One long-term dream (inspired by Avery Pennarun’s “bup” tool) is to
> store large binary files in a tree-structured way that can store
> common regions as deltas, improve random access, parallelized
> hashing. Involves a consistent way to split the file into stable
> pieces, like --rsyncable uses (based on a rolling hash being zero).
>
> * Peff: you can do that at the object model layer or at the storage
> layer. The latter is less invasive.
>
> * jrnieder: The benefits of blobtree are greater at the object model
> layer --- e.g. not having to transmit chunks over the wire that you
> already have. I think the main obstacle has been that the benefits
> haven’t been enough to be worth the complexity. If that changes, we
> can imagine bundling it with some other object format changes, e.g.
> putting blob sizes in tree objects, and rolling it out as a new
> object-format.
>
I think this was implemented as 'Blob Ref' in Yandex's vcs named Arc.
I was suggesting this to Gitlab folks earlier (1) as a possible solution to
large file storage.
Very glad to hear that it was brought up during the summit.
Cheers,
Son Luong.
(1): https://gitlab.com/gitlab-org/git/-/issues/93
next prev parent reply other threads:[~2021-10-21 12:31 UTC|newest]
Thread overview: 58+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-21 11:55 Notes from the Git Contributors' Summit 2021, virtual, Oct 19/20 Johannes Schindelin
2021-10-21 11:55 ` [Summit topic] Crazy (and not so crazy) ideas Johannes Schindelin
2021-10-21 12:30 ` Son Luong Ngoc [this message]
2021-10-26 20:14 ` scripting speedups [was: [Summit topic] Crazy (and not so crazy) ideas] Eric Wong
2021-10-30 19:58 ` Ævar Arnfjörð Bjarmason
2021-11-03 9:24 ` test suite speedups via some not-so-crazy ideas (was: scripting speedups[...]) Ævar Arnfjörð Bjarmason
2021-11-03 22:12 ` test suite speedups via some not-so-crazy ideas Junio C Hamano
2021-11-02 13:52 ` scripting speedups [was: [Summit topic] Crazy (and not so crazy) ideas] Johannes Schindelin
2021-10-21 11:55 ` [Summit topic] SHA-256 Updates Johannes Schindelin
2021-10-21 11:56 ` [Summit topic] Server-side merge/rebase: needs and wants? Johannes Schindelin
2021-10-22 3:06 ` Bagas Sanjaya
2021-10-22 10:01 ` Johannes Schindelin
2021-10-23 20:52 ` Ævar Arnfjörð Bjarmason
2021-11-08 18:21 ` Taylor Blau
2021-11-09 2:15 ` Ævar Arnfjörð Bjarmason
2021-11-30 10:06 ` Christian Couder
2021-10-21 11:56 ` [Summit topic] Submodules and how to make them worth using Johannes Schindelin
2021-10-21 11:56 ` [Summit topic] Sparse checkout behavior and plans Johannes Schindelin
2021-10-21 11:56 ` [Summit topic] The state of getting a reftable backend working in git.git Johannes Schindelin
2021-10-25 19:00 ` Han-Wen Nienhuys
2021-10-25 22:09 ` Ævar Arnfjörð Bjarmason
2021-10-26 8:12 ` Han-Wen Nienhuys
2021-10-28 14:17 ` Philip Oakley
2021-10-26 15:51 ` Philip Oakley
2021-10-21 11:56 ` [Summit topic] Documentation (translations, FAQ updates, new user-focused, general improvements, etc.) Johannes Schindelin
2021-10-22 14:20 ` Jean-Noël Avila
2021-10-22 14:31 ` Ævar Arnfjörð Bjarmason
2021-10-27 7:02 ` Jean-Noël Avila
2021-10-27 8:50 ` Jeff King
2021-10-21 11:56 ` [Summit topic] Increasing diversity & inclusion (transition to `main`, etc) Johannes Schindelin
2021-10-21 12:55 ` Son Luong Ngoc
2021-10-22 10:02 ` vale check, was " Johannes Schindelin
2021-10-22 10:03 ` Johannes Schindelin
2021-10-21 11:57 ` [Summit topic] Improving Git UX Johannes Schindelin
2021-10-21 16:45 ` changing the experimental 'git switch' (was: [Summit topic] Improving Git UX) Ævar Arnfjörð Bjarmason
2021-10-21 23:03 ` changing the experimental 'git switch' Junio C Hamano
2021-10-22 3:33 ` changing the experimental 'git switch' (was: [Summit topic] Improving Git UX) Bagas Sanjaya
2021-10-22 14:04 ` martin
2021-10-22 14:24 ` Ævar Arnfjörð Bjarmason
2021-10-22 15:30 ` martin
2021-10-23 8:27 ` changing the experimental 'git switch' Sergey Organov
2021-10-22 21:54 ` Sergey Organov
2021-10-24 6:54 ` changing the experimental 'git switch' (was: [Summit topic] Improving Git UX) Martin
2021-10-24 20:27 ` changing the experimental 'git switch' Junio C Hamano
2021-10-25 12:48 ` Ævar Arnfjörð Bjarmason
2021-10-25 17:06 ` Junio C Hamano
2021-10-25 16:44 ` Sergey Organov
2021-10-25 22:23 ` Ævar Arnfjörð Bjarmason
2021-10-27 18:54 ` Sergey Organov
2021-10-21 11:57 ` [Summit topic] Improving reviewer quality of life (patchwork, subsystem lists?, etc) Johannes Schindelin
2021-10-21 13:41 ` Konstantin Ryabitsev
2021-10-22 22:06 ` Ævar Arnfjörð Bjarmason
2021-10-22 8:02 ` Missing notes, was Re: Notes from the Git Contributors' Summit 2021, virtual, Oct 19/20 Johannes Schindelin
2021-10-22 8:22 ` Johannes Schindelin
2021-10-22 8:30 ` Johannes Schindelin
2021-10-22 9:07 ` Johannes Schindelin
2021-10-22 9:44 ` Let's have public Git chalk talks, " Johannes Schindelin
2021-10-25 12:58 ` Ævar Arnfjörð Bjarmason
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAL3xRKck=cbdo=7gqv0q=MLjn+5J1ES6fbzHP1mEj90pLdUdAA@mail.gmail.com' \
--to=sluongng@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).