git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: 程洋 <chengyang@xiaomi.com>
To: Jeff King <peff@peff.net>
Cc: "Taylor Blau" <me@ttaylorr.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"git@vger.kernel.org" <git@vger.kernel.org>
Subject: RE: [External Mail]Re: why git is so slow for a tiny git push?
Date: Thu, 25 Nov 2021 02:53:33 +0000	[thread overview]
Message-ID: <a39b72ed59f045c19fad8fcb259821d2@xiaomi.com> (raw)
In-Reply-To: <YZ6BJpAY4ZAWaqkr@coredump.intra.peff.net>

Well, we do have 300k refs, but only 1000 refs/heads.
However, I think most users only requires refs/heads, a few people only require refs/tags. As for other refs, we hardly see any user case.

So jgit treat it with a smart way,  it will create 2 pack files and 2 bitmaps, pack A contain all refs/heads, and pack B contains other refs. And when user do a fresh clone, it just need to send the pack A without determine if we can reuse or not

-----Original Message-----
From: Jeff King <peff@peff.net>
Sent: Thursday, November 25, 2021 2:15 AM
To: 程洋 <chengyang@xiaomi.com>
Cc: Taylor Blau <me@ttaylorr.com>; Ævar Arnfjörð Bjarmason <avarab@gmail.com>; git@vger.kernel.org
Subject: Re: [External Mail]Re: why git is so slow for a tiny git push?

*This message originated from outside of XIAOMI. Please treat this email with caution*


On Tue, Nov 23, 2021 at 06:42:12AM +0000, 程洋 wrote:

> I got another problem here.
> When I tries to clone from remote server. It took me 25 seconds to enumerating objects. And then 1 second to `couting objects` by bitmap.
> I don't understand, why a fresh clone need `enumerating objects` ? Is `couting objects` enough for the server to determine what to send?

In older versions of Git, the "counting objects" progress meter used to be the actual object graph traversal. That changed in v2.18 (via 5af050437a), but you may still seem some reference to "counting objects is expensive".

These days that is called "enumerating objects", and "counting objects"
is just doing a quick-ish pass over that list to do some light analysis (e.g., if we can reuse an on-disk delta). I'd expect "enumerating" to be expensive in general, and "counting" to be quick in general.

The "enumerating" phase is where we determine what to send whether it's for a clone or a fetch, and may involve opening up a bunch of trees to walk the graph. It's what reachability bitmaps are supposed to make faster. But if you have 300k refs, as you've mentioned, you almost certainly don't have complete coverage of all of the ref tips, so we'll have to fallback to doing at least a partial graph traversal.

Taylor (cc'd) has been looking at some tricks for speeding up cases like this with a lot of refs. But I don't think there's anything to show publicly yet.

-Peff
#/******本邮件及其附件含有小米公司的保密信息,仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本邮件! This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#

  reply	other threads:[~2021-11-25  2:55 UTC|newest]

Thread overview: 16+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <c5a8595658d6416684c2bbd317494c49@xiaomi.com>
     [not found] ` <5a6f3e8f29f74c93bf3af5da636df973@xiaomi.com>
2021-10-09 18:05   ` why git is so slow for a tiny git push? 程洋
2021-10-11 16:53     ` Jeff King
2021-10-12  8:04       ` [External Mail]Re: " 程洋
2021-10-12  8:39         ` Jeff King
2021-10-12  9:08           ` 程洋
2021-10-12 21:39             ` Jeff King
2021-10-14  6:47               ` 程洋
2021-10-26 21:54                 ` Jeff King
2021-10-27  2:48                   ` 程洋
2021-10-12 10:06           ` Ævar Arnfjörð Bjarmason
2021-10-12 21:46             ` Jeff King
2021-11-23  6:42               ` 程洋
2021-11-24 18:15                 ` Jeff King
2021-11-25  2:53                   ` 程洋 [this message]
2021-11-24  8:07               ` 程洋
2021-10-28 13:17     ` Han-Wen Nienhuys

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=a39b72ed59f045c19fad8fcb259821d2@xiaomi.com \
    --to=chengyang@xiaomi.com \
    --cc=avarab@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=me@ttaylorr.com \
    --cc=peff@peff.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).