git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Christian Couder <christian.couder@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>, "Jeff King" <peff@peff.net>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>,
	"Karsten Blees" <karsten.blees@gmail.com>,
	"Nguyen Thai Ngoc Duy" <pclouds@gmail.com>,
	"Stefan Beller" <sbeller@google.com>,
	"Eric Sunshine" <sunshine@sunshineco.com>,
	"Ramsay Jones" <ramsay@ramsayjones.plus.com>,
	"Johannes Sixt" <j6t@kdbg.org>, "René Scharfe" <l.s.r@web.de>,
	"Stefan Naewe" <stefan.naewe@atlas-elektronik.com>,
	"Christian Couder" <chriscool@tuxfamily.org>
Subject: [PATCH v13 00/14] libify apply and use lib in am, part 3
Date: Sat, 27 Aug 2016 20:45:33 +0200	[thread overview]
Message-ID: <20160827184547.4365-1-chriscool@tuxfamily.org> (raw)

Goal
~~~~

This is a patch series about libifying `git apply` functionality, and
using this libified functionality in `git am`, so that no 'git apply'
process is spawn anymore. This makes `git am` significantly faster, so
`git rebase`, when it uses the am backend, is also significantly
faster.

Previous discussions and patches series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This has initially been discussed in the following thread:

  http://thread.gmane.org/gmane.comp.version-control.git/287236/

Then the following patch series were sent:

RFC: http://thread.gmane.org/gmane.comp.version-control.git/288489/
v1: http://thread.gmane.org/gmane.comp.version-control.git/292324/
v2: http://thread.gmane.org/gmane.comp.version-control.git/294248/
v3: http://thread.gmane.org/gmane.comp.version-control.git/295429/
v4: http://thread.gmane.org/gmane.comp.version-control.git/296350/
v5: http://thread.gmane.org/gmane.comp.version-control.git/296490/
v6: http://thread.gmane.org/gmane.comp.version-control.git/297024/
v7: http://thread.gmane.org/gmane.comp.version-control.git/297193/
v8: https://public-inbox.org/git/20160627182429.31550-1-chriscool%40tuxfamily.org/
v9: https://public-inbox.org/git/20160730172509.22939-1-chriscool%40tuxfamily.org/
v10: https://public-inbox.org/git/20160808210337.5038-1-chriscool%40tuxfamily.org/
v11: https://public-inbox.org/git/20160811085229.19017-1-chriscool%40tuxfamily.org/
v12: https://public-inbox.org/git/20160811184501.384-1-chriscool@tuxfamily.org/

Highlevel view of the patches in the series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is "part 3" of the full patch series. I am resending only the
last 14 patches of the full series as "part 3", because I don't want
to resend the first 27 patches of v10 nearly as is.

(So "part 2" is made of patch 01/40 from v11 and patches from 02/40 to
27/40 from v10. And "part 1" has been in "master" for some time now.)

  - Patch 01/14 to 04/14 were v1, v2, v6, v7, v8, v9, v10, v11 and v12.

They haven't changed since v12.

They finish libifying the apply functionality that was in
builtin/apply.c and move it into apply.{c,h}, but the libified
functionality is not yet used in `git am`.

  - Patch 05/14 was in v6, v7, v8, v9, v10 and v12 and hasn't changed.

It replaces some calls to error() with calls to error_errno().

  - Patches 06/14 to 10/14 were in v2, v6, v7, v8, v9 and v10.

They haven't changed since v10.

They implement a way to make the libified apply code silent by
changing the bool `apply_verbosely` into a tristate enum called
`apply_verbosity`, that can be one of `verbosity_verbose`,
`verbosity_normal` or `verbosity_silent`.

This is because "git am", since it was a shell script, has been
silencing the apply functionality by redirecting file descriptors to
/dev/null, but this is not acceptable in C.

  - Patch 11/14 was in v9, v10 and v12, and hasn't changed.

It refactors `git apply` option parsing to make it possible for `git
am` to easily pass some command line options to the libified applied
code.

  - Patch 12/14 is new.

It is a refactoring to prepare for some new changes in patch 13/14.

  - Patch 13/14 was in v12.

It replaces patch 33/40 in v10 (environment: add set_index_file())
that was a hack to make it possible for `git am` to use the libified
apply functionality on a different index file.

For this purpose in this patch we add a "const char *index_file" into
"struct apply_state", and when "index_file" is set, we use
hold_lock_file_for_update(), passing it "state->index_file", instead
of using hold_locked_index(), as it is not possible to pass an index
filename in hold_locked_index().

This patch was changed in this version to also use
"read_cache_from(state->index_file)" instead of "read_cache()" when
state->index_file is set.

  - Patch 14/14 was in v1, v2, v6, v7, v8, v9, v10 and v12, and hasn't
    changed since v12.

This patch makes `git am` use the libified functionality.

General comments
~~~~~~~~~~~~~~~~

Now this patch series is shorter than previously. Hopefully also the
early part of this series until 05/14 or 11/14 will be ready soon to
be moved to next and master, and I may only need to resend the last 3
patches if anything.

I will send a diff between this version and v12 as a reply to this
email.

The benefits are not just related to not creating new processes. When
`git am` launched a `git apply` process, this new process had to read
the index from disk. Then after the `git apply`process had terminated,
`git am` dropped its index and read the index from disk to get the
index that had been modified by the `git apply`process. This was
inefficient and also prevented the split-index mechanism to provide
many performance benefits.

By the way, current work is ongoing to make it possible to use
split-index more easily by adding a config variable, see:

https://public-inbox.org/git/20160711172254.13439-1-chriscool%40tuxfamily.org/

Using an earlier version of this series as rebase material, Duy
explained split-index benefits along with this patch series like this:

 > Without the series, the picture is not so surprising. We run git-apply
 > 80+ times, each consists of this sequence
 >
 > read index
 > write index (cache tree updates only)
 > read index again
 > optionally initialize name hash (when new entries are added, I guess)
 > read packed-refs
 > write index
 >
 > With this series, we run a single git-apply which does
 >
 > read index (and sharedindex too if in split-index mode)
 > initialize name hash
 > write index 80+ times

(See: http://thread.gmane.org/gmane.comp.version-control.git/292324/focus=292460)

Links
~~~~~

This patch series is available here:

https://github.com/chriscool/git/commits/libify-apply-use-in-am

The previous versions are available there:

v1: https://github.com/chriscool/git/commits/libify-apply-use-in-am25 
v2: https://github.com/chriscool/git/commits/libify-apply-use-in-am54
v6: https://github.com/chriscool/git/commits/libify-apply-use-in-am65
v7: https://github.com/chriscool/git/commits/libify-apply-use-in-am75
v8: https://github.com/chriscool/git/commits/libify-apply-use-in-am97
v9: https://github.com/chriscool/git/commits/libify-apply-use-in-am106
v10: https://github.com/chriscool/git/commits/libify-apply-use-in-am114
v11: https://github.com/chriscool/git/commits/libify-apply-use-in-am116
v12: https://github.com/chriscool/git/commits/libify-apply-use-in-am118

Performance numbers
~~~~~~~~~~~~~~~~~~~

Numbers are only available for tests that have been performed on Linux
using a very early version of this series, though Johannes Sixt
reported great improvements on Windows. It could be interesting to get
detailed numbers on other platforms like Windows and OSX.

  - Around mid April Ævar did a huge many-hundred commit rebase on the
    kernel with untracked cache.

command: git rebase --onto 1993b17 52bef0c 29dde7c

Vanilla "next" without split index:                1m54.953s
Vanilla "next" with split index:                   1m22.476s
This series on top of "next" without split index:  1m12.034s
This series on top of "next" with split index:     0m15.678s

Ævar used his Debian laptop with SSD.

  - Around mid April I tested rebasing 13 commits in Booking.com's
    monorepo on a Red Hat 6.5 server with split-index and
    GIT_TRACE_PERFORMANCE=1.

With Git v2.8.0, the rebase took 6.375888383 s, with the git am
command launched by the rebase command taking 3.705677431 s.

With this series on top of next, the rebase took 3.044529494 s, with
the git am command launched by the rebase command taking 0.583521168
s.

Christian Couder (14):
  builtin/apply: rename option parsing functions
  apply: rename and move opt constants to apply.h
  Move libified code from builtin/apply.c to apply.{c,h}
  apply: make some parsing functions static again
  apply: use error_errno() where possible
  apply: make it possible to silently apply
  apply: don't print on stdout in verbosity_silent mode
  usage: add set_warn_routine()
  usage: add get_error_routine() and get_warn_routine()
  apply: change error_routine when silent
  apply: refactor `git apply` option parsing
  apply: pass apply state to build_fake_ancestor()
  apply: learn to use a different index file
  builtin/am: use apply API in run_apply()

 apply.c           | 4871 ++++++++++++++++++++++++++++++++++++++++++++++++++++-
 apply.h           |   34 +-
 builtin/am.c      |   65 +-
 builtin/apply.c   | 4814 +---------------------------------------------------
 git-compat-util.h |    3 +
 usage.c           |   15 +
 6 files changed, 4961 insertions(+), 4841 deletions(-)

-- 
2.9.2.770.g14ff7d2


             reply	other threads:[~2016-08-27 18:46 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-08-27 18:45 Christian Couder [this message]
2016-08-27 18:45 ` [PATCH v13 01/14] builtin/apply: rename option parsing functions Christian Couder
2016-08-27 18:45 ` [PATCH v13 02/14] apply: rename and move opt constants to apply.h Christian Couder
2016-08-31 21:46   ` Stefan Beller
2016-09-01  8:03     ` Christian Couder
2016-08-27 18:45 ` [PATCH v13 04/14] apply: make some parsing functions static again Christian Couder
2016-08-31 21:58   ` Stefan Beller
2016-08-27 18:45 ` [PATCH v13 05/14] apply: use error_errno() where possible Christian Couder
2016-08-31 21:59   ` Stefan Beller
2016-08-27 18:45 ` [PATCH v13 06/14] apply: make it possible to silently apply Christian Couder
2016-08-31 22:07   ` Stefan Beller
2016-09-01  8:01     ` Christian Couder
2016-09-01 16:57       ` Stefan Beller
2016-08-27 18:45 ` [PATCH v13 07/14] apply: don't print on stdout in verbosity_silent mode Christian Couder
2016-08-27 18:45 ` [PATCH v13 08/14] usage: add set_warn_routine() Christian Couder
2016-08-27 18:45 ` [PATCH v13 09/14] usage: add get_error_routine() and get_warn_routine() Christian Couder
2016-08-27 18:45 ` [PATCH v13 10/14] apply: change error_routine when silent Christian Couder
2016-08-31 22:20   ` Stefan Beller
2016-09-01  8:19     ` Christian Couder
2016-09-04 10:54       ` Christian Couder
2016-09-04 16:31         ` Ramsay Jones
2016-09-04 16:45           ` Christian Couder
2016-08-27 18:45 ` [PATCH v13 11/14] apply: refactor `git apply` option parsing Christian Couder
2016-08-27 18:45 ` [PATCH v13 12/14] apply: pass apply state to build_fake_ancestor() Christian Couder
2016-08-27 18:45 ` [PATCH v13 13/14] apply: learn to use a different index file Christian Couder
2016-08-27 18:45 ` [PATCH v13 14/14] builtin/am: use apply API in run_apply() Christian Couder
2016-08-31 22:33   ` Stefan Beller
2016-09-01  7:41     ` Christian Couder
2016-08-27 18:49 ` [PATCH v13 00/14] libify apply and use lib in am, part 3 Christian Couder
2016-08-29 19:04 ` Junio C Hamano
2016-08-29 19:52   ` Junio C Hamano
2016-08-30 10:19   ` Christian Couder
2016-08-31 22:15     ` Junio C Hamano
2016-09-01  8:28       ` Christian Couder
2016-09-01 17:48         ` Junio C Hamano
2016-09-04 20:17           ` Christian Couder
     [not found] ` <20160827184547.4365-4-chriscool@tuxfamily.org>
2016-08-31 21:57   ` [PATCH v13 03/14] Move libified code from builtin/apply.c to apply.{c,h} Stefan Beller
2016-09-01  7:46     ` Christian Couder

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160827184547.4365-1-chriscool@tuxfamily.org \
    --to=christian.couder@gmail.com \
    --cc=avarab@gmail.com \
    --cc=chriscool@tuxfamily.org \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=j6t@kdbg.org \
    --cc=karsten.blees@gmail.com \
    --cc=l.s.r@web.de \
    --cc=pclouds@gmail.com \
    --cc=peff@peff.net \
    --cc=ramsay@ramsayjones.plus.com \
    --cc=sbeller@google.com \
    --cc=stefan.naewe@atlas-elektronik.com \
    --cc=sunshine@sunshineco.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).