why git is so slow for a tiny git push?

[程洋 <chengyang@xiaomi.com>]

I have a really big repository which has 9m objects and maybe 300k refs.
I noticed that git push is really slow for a tiny change. An example shows below

3 objects which is only 7 kb takes 36 seconds to pack-objects (it's the time after i enable pack.usesparse)
However if I manually call “pack-objects” with the exactly same objects SHA1. It only take less than 0.005 second
What is really pass to “pack-objects” when I call “git push”?

I read an article says git will enumerate all "uninteresting objects" to determine what to send. but i don't understand, in my case git should only enumerate objects between "1a2d494b1b71469eebbd42aeabe1736bfa4b51fa..ddf3b84dca1aa4fe209a218380df1482af0d6b48". It's insane. I have a master server and a slave server serve this repository to my users. And i have a cron job to push every change from master to slave. And i found my master server CPU is full all the time because of the push jobs

Is there any solution?


Here is my full output
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
work@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$  GIT_TRACE_PERFORMANCE=1 GIT_TRACE_PACKET=1 git push git://10.172.32.31/miui/platform/frameworks/base.git bsp-qcom-s:bsp-qcom-s
.....
.....
.....
.....
01:53:00.595910 pkt-line.c:80           packet:         push< 2d5eb3d37e2aa6659920bda688310584c70a37d0 refs/remotes/origin/v9-l-ido-stable2x
01:53:00.595917 pkt-line.c:80           packet:         push< 120ebf9b593c8f914981506b8390dd317fb8c9f1 refs/remotes/origin/c
01:53:00.595954 pkt-line.c:80           packet:         push< 90403256e6775476bd8dee31f28b31af9f7eac89 refs/remotes/origin/v9-o-d1s-dev-backup
01:53:00.595962 pkt-line.c:80           packet:         push< 9a78ddd5a4d6759585d29549466138291b665d35 refs/remotes/origin/v9-o-dipper-dev
01:53:00.595969 pkt-line.c:80           packet:         push< 343dfeb613e411e0ede4ec6327e2b171e39ff523 refs/remotes/origin/v9-o-nitrogen-dev
01:53:00.595976 pkt-line.c:80           packet:         push< 35e23e916c2dccd8603a6fe485ee56d688003ff5 refs/remotes/origin/v9-o-sagit-dev
01:53:00.595984 pkt-line.c:80           packet:         push< dc62ab5c28f7f6154d41d138301fe7f555db160b refs/remotes/origin/v9-o-sakura-dev
01:53:00.595991 pkt-line.c:80           packet:         push< 00b4763e8219de6e8a76b9ce24ecb3460030783f refs/remotes/origin/v9-o-scorpio-dev
01:53:00.595998 pkt-line.c:80           packet:         push< 2f7b3a268712ba187bb1d399c698cb1836c5d47a refs/remotes/origin/v9-o-sirius-dev
01:53:00.596005 pkt-line.c:80           packet:         push< e14f29817e4128463ceeccc531f313ae1b138780 refs/remotes/origin/v9-o-whyred-dev
01:53:00.596012 pkt-line.c:80           packet:         push< c7620c21cf3aade7666c448471306c1264032c5d refs/remotes/origin/v9-o-ysl-dev
01:53:00.596022 pkt-line.c:80           packet:         push< c1cbe43f1bc50dcd422ef354e694ea45ca8aa797 refs/remotes/origin/wt-p-laurus-native
01:53:00.596029 pkt-line.c:80           packet:         push< 3c736c7b701b9023f08c3641bcc77041e35d0eca refs/remotes/origin/wt-q-laurus-miui
01:53:00.596036 pkt-line.c:80           packet:         push< 3dc0351efe79817f5c8bf23564e3c483ff059833 refs/remotes/origin/wt-q-laurus-native
01:53:00.596044 pkt-line.c:80           packet:         push< 58ac901f9302e7f52325f4ce905cc2cbdfa310ca refs/remotes/origin/wt-r-evergreen
01:53:00.596053 pkt-line.c:80           packet:         push< 0000
01:53:00.601910 pkt-line.c:80           packet:         push> 1a2d494b1b71469eebbd42aeabe1736bfa4b51fa ddf3b84dca1aa4fe209a218380df1482af0d6b48 refs/heads/bsp-qcom-s\0 report-status-v2 side-band-64k object-format=sha1 agent=git/2.32.0
01:53:00.601959 pkt-line.c:80           packet:         push> 0000
Enumerating objects: 3, done.
Counting objects: 100% (3/3), done.
Delta compression using up to 40 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 7.24 KiB | 7.24 MiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
01:53:19.256506 trace.c:487             performance: 18.653337755 s: git command: /usr/libexec/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress
01:53:22.298695 pkt-line.c:80           packet:     sideband< \1000eunpack ok001dok refs/heads/bsp-qcom-s0000
01:53:22.298735 pkt-line.c:80           packet:     sideband< 0000
01:53:22.298745 pkt-line.c:80           packet:         push< unpack ok
01:53:22.298770 pkt-line.c:80           packet:         push< ok refs/heads/bsp-qcom-s
01:53:22.298779 pkt-line.c:80           packet:         push< 0000
To git://10.172.32.31/miui/platform/frameworks/base.git
   1a2d494b1b71..ddf3b84dca1a  bsp-qcom-s -> bsp-qcom-s
01:53:22.316441 trace.c:487             performance: 22.883688573 s: git command: git push git://10.172.32.31/miui/platform/frameworks/base.git bsp-qcom-s:bsp-qcom-s
work@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$ echo 1a2d494b1b71469eebbd42aeabe1736bfa4b51fa..ddf3b84dca1aa4fe209a218380df1482af0d6b48 > 1
work@c5-miui-miuigit-slave15:~/repositories/miui/platform/frameworks/base2/base$ time /usr/libexec/git-core/git pack-objects --all-progress-implied --revs --stdout --thin --delta-base-offset --progress < 1
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 40 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 292 bytes | 292.00 KiB/s, done.

Total 3 (delta 2), reused 0 (delta 0), pack-reused 0

real    0m0.005s
user    0m0.000s
sys 0m0.004s
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
#/******本邮件及其附件含有小米公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！ This e-mail and its attachments contain confidential information from XIAOMI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!******/#