mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "René Scharfe" <>
To: "Scheffenegger, Richard" <>,
	Junio C Hamano <>,
	"brian m. carlson" <>
Cc: "" <>
Subject: Re: git --archive
Date: Sat, 24 Sep 2022 20:07:54 +0200	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

Am 24.09.22 um 15:19 schrieb René Scharfe:
> It could be done by relying on the randomness of the object IDs and
> partitioning by a sub-string.  Or perhaps using pseudo-random numbers
> is sufficient:
>    git ls-tree -r HEAD |
>    awk '{print $3}' |

No need for awk here, of course; "git ls-tree -r --object-only HEAD"
does the same.  Just saying.

> Here's an idea after all: Using "git ls-tree" without "-r" and handling
> recursing in the prefetch script would allow traversing trees in a
> different order and even in parallel.  Not sure how to limit parallelism
> to a sane degree.

How about something like this?  xargs -P provides a controlled degree of
parallelism.  Sorting by object ID (i.e. hash value) should provide a
fairly random order.  Does this thing work for you?


dir=$(mktemp -d)
echo "$treeish" >"$dir/trees"

# Traverse all sub-trees in randomized order and collect all blob IDs.
while test -s "$dir/trees"
        sort <"$dir/trees" >"$dir/trees.current"
        rm "$dir/trees"
        xargs -P "$parallelism" -L 1 git ls-tree <"$dir/trees.current" |
        awk -v dir="$dir" -v pieces="$parallelism" '
                $2 == "tree" {print $3 > (dir "/trees")}
                $2 == "blob" {print $3 >> (dir "/blobs" int(rand() * pieces))}

# Prefetch all blobs in randomized order.
command="sort $replstr | git cat-file --batch >/dev/null"
ls "$dir/blobs"* | xargs -P "$parallelism" -I "$replstr" -L 1 sh -c "$command"

rm "$dir/trees.current" "$dir/blobs"*
rmdir "$dir"

  reply	other threads:[~2022-09-24 18:08 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-09-22  8:57 git --archive Scheffenegger, Richard
2022-09-22 20:13 ` Junio C Hamano
2022-09-22 20:35   ` Scheffenegger, Richard
2022-09-23  0:49     ` brian m. carlson
2022-09-23 16:30       ` Junio C Hamano
2022-09-23 16:51         ` Scheffenegger, Richard
2022-09-24  8:58         ` René Scharfe
2022-09-24 11:34           ` Scheffenegger, Richard
2022-09-24 13:19             ` René Scharfe
2022-09-24 18:07               ` René Scharfe [this message]
2022-09-25  8:17                 ` René Scharfe
2022-09-24 19:44               ` Scheffenegger, Richard

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).