git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
* [PATCH 00/83] libify apply and use lib in am
@ 2016-04-24 13:33 Christian Couder
  2016-04-24 13:33 ` [PATCH 01/83] builtin/apply: make gitdiff_verify_name() return void Christian Couder
                   ` (79 more replies)
  0 siblings, 80 replies; 156+ messages in thread
From: Christian Couder @ 2016-04-24 13:33 UTC (permalink / raw)
  To: git
  Cc: Junio C Hamano, Jeff King, Ævar Arnfjörð Bjarmason,
	Karsten Blees, Nguyen Thai Ngoc Duy, Johannes Schindelin,
	Stefan Beller, Matthieu Moy, Christian Couder

This is a patch series about libifying `git apply` functionality, and
using this libified functionality in `git am`, so that no 'git apply'
process is spawn anymore. This makes `git am` significantly faster, so
`git rebase`, when it uses the am backend, is also significantly
faster.

This has initially been discussed in the following thread:

http://thread.gmane.org/gmane.comp.version-control.git/287236/

A RFC patch series was sent previously that only got rid of the global
variables and refactored the code around a bit:

http://thread.gmane.org/gmane.comp.version-control.git/288489/

This new patch series is built on top of that previous work, so
patches 1/83 to 47/83 are from the previous RFC patch series and
patches after that are new.

Sorry if this patch series is a bit long. I can split it into two or
more series if it is prefered.

The benefits are not just related to not creating new processes. When
`git am` launched a `git apply` process, this new process had to read
the index from disk. Then after the `git apply`process had terminated,
`git am` dropped its index and read the index from disk to get the
index that had been modified by the `git apply`process. This was
inefficient and also prevented the split-index mechanism to provide
many performance benefits.

Performance numbers:

  - A few days ago Ævar did a huge many-hundred commit rebase on the
    kernel with untracked cache.

command: git rebase --onto 1993b17 52bef0c 29dde7c

Vanilla "next" without split index:                1m54.953s
Vanilla "next" with split index:                   1m22.476s
This series on top of "next" without split index:  1m12.034s
This series on top of "next" with split index:     0m15.678s

Ævar used his Debian laptop with SSD.

  - Some days ago I tested rebasing 13 commits in Booking.com's
    monorepo on a Red Hat 6.5 server with split-index and
    GIT_TRACE_PERFORMANCE=1.

With Git v2.8.0, the rebase took 6.375888383 s, with the git am
command launched by the rebase command taking 3.705677431 s.

With this series on top of next, the rebase took 3.044529494 s, with
the git am command launched by the rebase command taking 0.583521168
s.

No tests on Windows have been performed, but it could be interesting
to test on this platform.

Christian Couder (83):
  builtin/apply: make gitdiff_verify_name() return void
  builtin/apply: avoid parameter shadowing 'p_value' global
  builtin/apply: avoid parameter shadowing 'linenr' global
  builtin/apply: avoid local variable shadowing 'len' parameter
  builtin/apply: extract line_by_line_fuzzy_match() from
    match_fragment()
  builtin/apply: move 'options' variable into cmd_apply()
  builtin/apply: introduce 'struct apply_state' to start libifying
  builtin/apply: move 'unidiff_zero' global into 'struct apply_state'
  builtin/apply: move 'check' global into 'struct apply_state'
  builtin/apply: move 'check_index' global into 'struct apply_state'
  builtin/apply: move 'apply_in_reverse' global into 'struct
    apply_state'
  builtin/apply: move 'apply_with_reject' global into 'struct
    apply_state'
  builtin/apply: move 'apply_verbosely' global into 'struct apply_state'
  builtin/apply: move 'update_index' global into 'struct apply_state'
  builtin/apply: move 'allow_overlap' global into 'struct apply_state'
  builtin/apply: move 'cached' global into 'struct apply_state'
  builtin/apply: move 'diffstat' global into 'struct apply_state'
  builtin/apply: move 'numstat' global into 'struct apply_state'
  builtin/apply: move 'summary' global into 'struct apply_state'
  builtin/apply: move 'threeway' global into 'struct apply_state'
  builtin/apply: move 'no-add' global into 'struct apply_state'
  builtin/apply: move 'unsafe_paths' global into 'struct apply_state'
  builtin/apply: move 'line_termination' global into 'struct
    apply_state'
  builtin/apply: move 'fake_ancestor' global into 'struct apply_state'
  builtin/apply: move 'p_context' global into 'struct apply_state'
  builtin/apply: move 'apply' global into 'struct apply_state'
  builtin/apply: move 'read_stdin' global into cmd_apply()
  builtin/apply: move 'patch_input_file' global into 'struct
    apply_state'
  builtin/apply: move 'limit_by_name' global into 'struct apply_state'
  builtin/apply: move 'has_include' global into 'struct apply_state'
  builtin/apply: move 'p_value' global into 'struct apply_state'
  builtin/apply: move 'p_value_known' global into 'struct apply_state'
  builtin/apply: move 'root' global into 'struct apply_state'
  builtin/apply: move 'whitespace_error' global into 'struct
    apply_state'
  builtin/apply: move 'whitespace_option' into 'struct apply_state'
  builtin/apply: remove whitespace_option arg from
    set_default_whitespace_mode()
  builtin/apply: move 'squelch_whitespace_errors' into 'struct
    apply_state'
  builtin/apply: move 'applied_after_fixing_ws' into 'struct
    apply_state'
  builtin/apply: move 'ws_error_action' into 'struct apply_state'
  builtin/apply: move 'ws_ignore_action' into 'struct apply_state'
  builtin/apply: move 'max_change' and 'max_len' into 'struct
    apply_state'
  builtin/apply: move 'linenr' global into 'struct apply_state'
  builtin/apply: move 'fn_table' global into 'struct apply_state'
  builtin/apply: move 'symlink_changes' global into 'struct apply_state'
  builtin/apply: move 'state' init into init_apply_state()
  builtin/apply: move 'state' check into check_apply_state()
  builtin/apply: move applying patches into apply_all_patches()
  builtin/apply: rename 'prefix_' parameter to 'prefix'
  builtin/apply: move 'lock_file' global into 'struct apply_state'
  builtin/apply: get rid of the 'newfd' global
  builtin/apply: make apply_patch() return -1 instead of die()ing
  builtin/apply: read_patch_file() return -1 instead of die()ing
  builtin/apply: make find_header() return -1 instead of die()ing
  builtin/apply: make parse_chunk() return a negative integer on error
  builtin/apply: make parse_single_patch() return -1 on error
  apply: move 'struct apply_state' to apply.h
  builtin/apply: libify parse_whitespace_option()
  builtin/apply: libify parse_ignorewhitespace_option()
  builtin/apply: move init_apply_state() to apply.c
  apply: libify init_apply_state()
  builtin/apply: libify check_apply_state()
  builtin/apply: move check_apply_state() to apply.c
  builtin/apply: make apply_all_patches() return -1 on error
  builtin/apply: make parse_traditional_patch() return -1 on error
  builtin/apply: make gitdiff_verify_name() return -1 on error
  builtin/apply: change die_on_unsafe_path() to check_unsafe_path()
  builtin/apply: make build_fake_ancestor() return -1 on error
  builtin/apply: make remove_file() return -1 on error
  builtin/apply: make add_conflicted_stages_file() return -1 on error
  builtin/apply: make add_index_file() return -1 on error
  builtin/apply: make create_file() return -1 on error
  builtin/apply: make write_out_one_result() return -1 on error
  builtin/apply: make write_out_results() return -1 on error
  builtin/apply: make try_create_file() return -1 on error
  builtin/apply: make create_one_file() return -1 on error
  builtin/apply: rename option parsing functions
  apply: rename and move opt constants to apply.h
  Move libified code from builtin/apply.c to apply.{c,h}
  apply: make some parsing functions static again
  run-command: make dup_devnull() non static
  apply: roll back index in case of error
  environment: add set_index_file()
  builtin/am: use apply api in run_apply()

 Makefile               |    1 +
 apply.c                | 4796 ++++++++++++++++++++++++++++++++++++++++++++++++
 apply.h                |  149 ++
 builtin/am.c           |  103 +-
 builtin/apply.c        | 4665 +---------------------------------------------
 cache.h                |    1 +
 environment.c          |    5 +
 run-command.c          |    2 +-
 run-command.h          |    6 +
 t/t4012-diff-binary.sh |    4 +-
 t/t4254-am-corrupt.sh  |    2 +-
 11 files changed, 5096 insertions(+), 4638 deletions(-)
 create mode 100644 apply.c
 create mode 100644 apply.h

-- 
2.8.1.300.g5fed0c0

^ permalink raw reply	[flat|nested] 156+ messages in thread

end of thread, other threads:[~2016-05-08 20:11 UTC | newest]

Thread overview: 156+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-24 13:33 [PATCH 00/83] libify apply and use lib in am Christian Couder
2016-04-24 13:33 ` [PATCH 01/83] builtin/apply: make gitdiff_verify_name() return void Christian Couder
2016-04-24 13:33 ` [PATCH 02/83] builtin/apply: avoid parameter shadowing 'p_value' global Christian Couder
2016-04-24 13:33 ` [PATCH 03/83] builtin/apply: avoid parameter shadowing 'linenr' global Christian Couder
2016-04-27 16:27   ` Junio C Hamano
2016-04-27 17:07     ` Christian Couder
2016-04-24 13:33 ` [PATCH 04/83] builtin/apply: avoid local variable shadowing 'len' parameter Christian Couder
2016-04-24 13:33 ` [PATCH 05/83] builtin/apply: extract line_by_line_fuzzy_match() from match_fragment() Christian Couder
2016-04-25 18:50   ` Stefan Beller
2016-04-26 16:15     ` Christian Couder
2016-04-26 20:20       ` Junio C Hamano
2016-04-27 13:31         ` Christian Couder
2016-04-24 13:33 ` [PATCH 06/83] builtin/apply: move 'options' variable into cmd_apply() Christian Couder
2016-04-24 13:33 ` [PATCH 07/83] builtin/apply: introduce 'struct apply_state' to start libifying Christian Couder
2016-04-24 13:33 ` [PATCH 08/83] builtin/apply: move 'unidiff_zero' global into 'struct apply_state' Christian Couder
2016-04-24 13:33 ` [PATCH 09/83] builtin/apply: move 'check' " Christian Couder
2016-04-25 18:57   ` Stefan Beller
2016-04-26 16:26     ` Christian Couder
2016-04-26 16:31       ` Stefan Beller
2016-04-24 13:33 ` [PATCH 10/83] builtin/apply: move 'check_index' " Christian Couder
2016-04-26 20:25   ` Junio C Hamano
2016-04-24 13:33 ` [PATCH 11/83] builtin/apply: move 'apply_in_reverse' " Christian Couder
2016-04-24 13:33 ` [PATCH 12/83] builtin/apply: move 'apply_with_reject' " Christian Couder
2016-04-24 13:33 ` [PATCH 13/83] builtin/apply: move 'apply_verbosely' " Christian Couder
2016-04-24 13:33 ` [PATCH 14/83] builtin/apply: move 'update_index' " Christian Couder
2016-04-24 13:33 ` [PATCH 15/83] builtin/apply: move 'allow_overlap' " Christian Couder
2016-04-24 13:33 ` [PATCH 16/83] builtin/apply: move 'cached' " Christian Couder
2016-04-24 13:33 ` [PATCH 17/83] builtin/apply: move 'diffstat' " Christian Couder
2016-04-24 13:33 ` [PATCH 18/83] builtin/apply: move 'numstat' " Christian Couder
2016-04-25 21:40   ` Stefan Beller
2016-04-26 16:35     ` Christian Couder
2016-04-24 13:33 ` [PATCH 19/83] builtin/apply: move 'summary' " Christian Couder
2016-04-24 13:33 ` [PATCH 20/83] builtin/apply: move 'threeway' " Christian Couder
2016-04-24 13:33 ` [PATCH 21/83] builtin/apply: move 'no-add' " Christian Couder
2016-04-24 13:33 ` [PATCH 22/83] builtin/apply: move 'unsafe_paths' " Christian Couder
2016-04-26 20:27   ` Junio C Hamano
2016-04-27 12:14     ` Christian Couder
2016-04-27 15:26       ` Junio C Hamano
2016-04-24 13:33 ` [PATCH 23/83] builtin/apply: move 'line_termination' " Christian Couder
2016-04-24 13:33 ` [PATCH 24/83] builtin/apply: move 'fake_ancestor' " Christian Couder
2016-04-24 13:33 ` [PATCH 25/83] builtin/apply: move 'p_context' " Christian Couder
2016-04-24 13:33 ` [PATCH 26/83] builtin/apply: move 'apply' " Christian Couder
2016-04-24 13:33 ` [PATCH 27/83] builtin/apply: move 'read_stdin' global into cmd_apply() Christian Couder
2016-04-26 20:28   ` Junio C Hamano
2016-04-27 13:13     ` Christian Couder
2016-04-24 13:33 ` [PATCH 28/83] builtin/apply: move 'patch_input_file' global into 'struct apply_state' Christian Couder
2016-04-24 13:33 ` [PATCH 29/83] builtin/apply: move 'limit_by_name' " Christian Couder
2016-04-24 13:33 ` [PATCH 30/83] builtin/apply: move 'has_include' " Christian Couder
2016-04-24 13:33 ` [PATCH 31/83] builtin/apply: move 'p_value' " Christian Couder
2016-04-24 13:33 ` [PATCH 32/83] builtin/apply: move 'p_value_known' " Christian Couder
2016-04-24 13:33 ` [PATCH 33/83] builtin/apply: move 'root' " Christian Couder
2016-04-25 21:50   ` Stefan Beller
2016-04-25 21:54     ` Stefan Beller
2016-04-26 20:13     ` Christian Couder
2016-04-24 13:33 ` [PATCH 34/83] builtin/apply: move 'whitespace_error' " Christian Couder
2016-04-24 13:33 ` [PATCH 35/83] builtin/apply: move 'whitespace_option' " Christian Couder
2016-04-24 13:33 ` [PATCH 36/83] builtin/apply: remove whitespace_option arg from set_default_whitespace_mode() Christian Couder
2016-04-24 13:33 ` [PATCH 37/83] builtin/apply: move 'squelch_whitespace_errors' into 'struct apply_state' Christian Couder
2016-04-24 13:33 ` [PATCH 38/83] builtin/apply: move 'applied_after_fixing_ws' " Christian Couder
2016-04-24 13:33 ` [PATCH 39/83] builtin/apply: move 'ws_error_action' " Christian Couder
2016-04-26 20:36   ` Junio C Hamano
2016-04-26 20:40     ` Junio C Hamano
2016-04-27 13:23     ` Christian Couder
2016-04-24 13:33 ` [PATCH 40/83] builtin/apply: move 'ws_ignore_action' " Christian Couder
2016-04-24 13:33 ` [PATCH 41/83] builtin/apply: move 'max_change' and 'max_len' " Christian Couder
2016-04-24 13:33 ` [PATCH 42/83] builtin/apply: move 'linenr' global " Christian Couder
2016-04-24 13:33 ` [PATCH 43/83] builtin/apply: move 'fn_table' " Christian Couder
2016-04-24 13:33 ` [PATCH 44/83] builtin/apply: move 'symlink_changes' " Christian Couder
2016-04-24 13:33 ` [PATCH 45/83] builtin/apply: move 'state' init into init_apply_state() Christian Couder
2016-04-25  7:32   ` Eric Sunshine
2016-04-27 12:00     ` Christian Couder
2016-04-24 13:33 ` [PATCH 46/83] builtin/apply: move 'state' check into check_apply_state() Christian Couder
2016-04-24 13:33 ` [PATCH 47/83] builtin/apply: move applying patches into apply_all_patches() Christian Couder
2016-04-25 22:00   ` Stefan Beller
2016-04-27 15:51     ` Christian Couder
2016-04-24 13:33 ` [PATCH 48/83] builtin/apply: rename 'prefix_' parameter to 'prefix' Christian Couder
2016-04-24 13:33 ` [PATCH 49/83] builtin/apply: move 'lock_file' global into 'struct apply_state' Christian Couder
2016-04-25  7:50   ` Eric Sunshine
2016-04-25 17:55     ` Junio C Hamano
2016-04-28 16:30       ` Christian Couder
2016-04-28 20:17         ` Junio C Hamano
2016-04-30 19:39     ` Christian Couder
2016-04-24 13:33 ` [PATCH 50/83] builtin/apply: get rid of the 'newfd' global Christian Couder
2016-04-24 13:33 ` [PATCH 51/83] builtin/apply: make apply_patch() return -1 instead of die()ing Christian Couder
2016-04-26  1:20   ` Eric Sunshine
2016-04-30 19:41     ` Christian Couder
2016-04-24 13:33 ` [PATCH 52/83] builtin/apply: read_patch_file() " Christian Couder
2016-04-24 13:33 ` [PATCH 53/83] builtin/apply: make find_header() " Christian Couder
2016-04-25 13:18   ` Duy Nguyen
2016-04-27 18:10     ` Eric Sunshine
2016-05-01 16:53       ` Christian Couder
2016-04-27 18:08   ` Eric Sunshine
2016-05-01 16:45     ` Christian Couder
2016-04-24 13:33 ` [PATCH 54/83] builtin/apply: make parse_chunk() return a negative integer on error Christian Couder
2016-05-01 19:04   ` Eric Sunshine
2016-05-01 19:51     ` Christian Couder
2016-04-24 13:33 ` [PATCH 55/83] builtin/apply: make parse_single_patch() return -1 " Christian Couder
2016-05-01 19:32   ` Eric Sunshine
2016-04-24 13:33 ` [PATCH 56/83] apply: move 'struct apply_state' to apply.h Christian Couder
2016-04-24 13:33 ` [PATCH 57/83] builtin/apply: libify parse_whitespace_option() Christian Couder
2016-04-24 13:33 ` [PATCH 58/83] builtin/apply: libify parse_ignorewhitespace_option() Christian Couder
2016-04-24 13:33 ` [PATCH 59/83] builtin/apply: move init_apply_state() to apply.c Christian Couder
2016-05-01 19:37   ` Eric Sunshine
2016-05-01 20:19     ` Christian Couder
2016-04-24 13:34 ` [PATCH 60/83] apply: libify init_apply_state() Christian Couder
2016-04-24 13:34 ` [PATCH 61/83] builtin/apply: libify check_apply_state() Christian Couder
2016-04-25 13:26   ` Duy Nguyen
2016-05-01 16:58     ` Christian Couder
2016-04-24 13:34 ` [PATCH 62/83] builtin/apply: move check_apply_state() to apply.c Christian Couder
2016-04-24 13:34 ` [PATCH 63/83] builtin/apply: make apply_all_patches() return -1 on error Christian Couder
2016-04-25 13:30   ` Duy Nguyen
2016-05-01 18:32     ` Christian Couder
2016-05-01 21:03   ` Eric Sunshine
2016-05-02  7:09     ` Johannes Schindelin
2016-05-03 13:01     ` Christian Couder
2016-04-24 13:34 ` [PATCH 64/83] builtin/apply: make parse_traditional_patch() " Christian Couder
2016-04-24 13:34 ` [PATCH 65/83] builtin/apply: make gitdiff_verify_name() " Christian Couder
2016-04-25 13:36   ` Duy Nguyen
2016-05-01 19:31     ` Christian Couder
2016-04-24 13:34 ` [PATCH 66/83] builtin/apply: change die_on_unsafe_path() to check_unsafe_path() Christian Couder
2016-04-24 13:34 ` [PATCH 67/83] builtin/apply: make build_fake_ancestor() return -1 on error Christian Couder
2016-05-02  7:32   ` Eric Sunshine
2016-05-03 13:52     ` Christian Couder
2016-04-24 13:34 ` [PATCH 68/83] builtin/apply: make remove_file() " Christian Couder
2016-04-24 13:34 ` [PATCH 69/83] builtin/apply: make add_conflicted_stages_file() " Christian Couder
2016-05-02  7:36   ` Eric Sunshine
2016-05-03 14:33     ` Christian Couder
2016-04-24 13:34 ` [PATCH 70/83] builtin/apply: make add_index_file() " Christian Couder
2016-04-24 13:34 ` [PATCH 71/83] builtin/apply: make create_file() " Christian Couder
2016-04-24 13:34 ` [PATCH 72/83] builtin/apply: make write_out_one_result() " Christian Couder
2016-04-24 13:34 ` [PATCH 73/83] builtin/apply: make write_out_results() " Christian Couder
2016-05-02 17:42   ` Eric Sunshine
2016-05-03 14:55     ` Christian Couder
2016-04-24 13:34 ` [PATCH 74/83] builtin/apply: make try_create_file() " Christian Couder
2016-05-02 18:01   ` Eric Sunshine
2016-05-03 16:24     ` Christian Couder
2016-04-24 13:34 ` [PATCH 75/83] builtin/apply: make create_one_file() " Christian Couder
2016-04-24 13:34 ` [PATCH 76/83] builtin/apply: rename option parsing functions Christian Couder
2016-04-24 13:34 ` [PATCH 77/83] apply: rename and move opt constants to apply.h Christian Couder
2016-04-24 15:23 ` [PATCH 00/83] libify apply and use lib in am Ramsay Jones
2016-04-24 16:27   ` Christian Couder
2016-04-24 16:56     ` Christian Couder
2016-04-24 17:42       ` Ramsay Jones
2016-04-25  0:14         ` Duy Nguyen
2016-04-25  9:15           ` Christian Couder
2016-04-25 15:09       ` Johannes Schindelin
2016-04-25  9:02 ` Duy Nguyen
2016-04-25  9:57   ` Christian Couder
2016-04-25 10:11     ` Duy Nguyen
     [not found] ` <1461504863-15946-79-git-send-email-chriscool@tuxfamily.org>
2016-04-25 13:46   ` [PATCH 78/83] Move libified code from builtin/apply.c to apply.{c,h} Duy Nguyen
2016-05-04 10:39     ` Christian Couder
2016-05-04 11:32       ` Duy Nguyen
2016-05-05  8:50         ` Christian Couder
2016-05-06 20:18           ` Christian Couder
2016-05-06 21:07             ` Junio C Hamano
2016-05-08 20:11               ` Christian Couder

Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).