mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Scheffenegger, Richard" <>
To: Junio C Hamano <>
Cc: "" <>
Subject: RE: git --archive
Date: Thu, 22 Sep 2022 20:35:08 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <xmqqleqbxj4h.fsf@gitster.g>

Hi Junio,

>> Unless I’m mistaken, the procedure to create a zip archive reads like a recursive collection of all relevant objects, and then writing them out sequentially, in a single thread.
>> Is this assessment correct?
>> I was wondering if a highly concurrent fetching phase could be 
>> optionally added…
> The details matter here, I think.  Enumerating and slurping down the contents to be archived out of the repository/object store to the core can indeed be made parallel, but the end result product being a zip archive or a tarball, which is fairly a serialized output format, there is only so much you can hold in core, and it is not clear what your plan is to do this without filling all the memory.

"core" (presumably referring to the OS kernel memory for IO caching) is not the only cache in play here. 

As mentioned, the use case would be repositories living on storage systems measuring in at around 500 TB on the filesystem - like what is not uncommon on hyperscalers and large commercial scale development environments.

Those external filesystems (the filesystems do NOT live in "core") tend to have some more or less sophisticated form of tiering, destaging cold (infrequently accessed) data out to high latency devices. Amazon Glacier being an extreme example here (on-demand access tape drives, which need to be fetched by a robot and put into a tape drive when data stored there is needed).

To a more realistic scenario, you may have 100 TB in SSD, and a few PB in spinning rust, externally connected to the primary tier. 

An initial phase to simply fetch in as many objects and sub-trees as possible in parallel (ideally with IOs to the objects issued in a non-sequential order, to promote them and their metadata not to get evicted immediately after the first fetch) would heat up those external caches (to some extent the "core" filesystem page cache too, but that is of minor concern). Thus when the tree is then walked in a fully sequential, recursive way just like currently, the hot metadata, and some hot data would dramatically cut the accumulation of latency. 

A 2nd phase, having the tree fetched in parallel, and sent out serial, would work even better.

Also, at least for ZIP (not so much for TAR), objects residing in different subdirectories can be stored in any order - and only need to be referenced properly in the central directory. Thus whenever a subthread has completed the reading of a (sufficiently small) object to be in (git program) memory, it should be sent immediately to the ZIP writer thread. The result would be that small and hot files (which can be read in quickly) end up at the beginning of the zip file, but the parallel threads can already, in parallel, read-in larger and colder object - the absolute wait time within the worker thread reading those objects may be slightly higher, but as many objects are read in in parallel, the absolute time to create the archive would be minimized.

In short - there is no real need with ZIP for the recursive traversal of the objects and trees to deliver them in-sequence at all (beside, the sequence may be determined by the underlying filesystem, which is not necessarily guaranteed to provide a trivially sorted list of any kind - only that the ordering of files within a directory is stable.

Hope this clarifies the background of this ask.

Best regards,

  reply	other threads:[~2022-09-22 20:35 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-22  8:57 git --archive Scheffenegger, Richard
2022-09-22 20:13 ` Junio C Hamano
2022-09-22 20:35   ` Scheffenegger, Richard [this message]
2022-09-23  0:49     ` brian m. carlson
2022-09-23 16:30       ` Junio C Hamano
2022-09-23 16:51         ` Scheffenegger, Richard
2022-09-24  8:58         ` René Scharfe
2022-09-24 11:34           ` Scheffenegger, Richard
2022-09-24 13:19             ` René Scharfe
2022-09-24 18:07               ` René Scharfe
2022-09-25  8:17                 ` René Scharfe
2022-09-24 19:44               ` Scheffenegger, Richard

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).