mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: "Andrej Shadura via GitGitGadget" <>
Cc: "brian m. carlson" <>,
	Andrej Shadura <>
Subject: [PATCH v2 0/2] Add --no-filters option to git-add
Date: Sat, 20 Feb 2021 17:07:43 +0000	[thread overview]
Message-ID: <> (raw)
In-Reply-To: <>

It is possible for a user to disable attribute-based filtering when
committing by doing one of the following:

 * Create .git/info/attributes unapplying all possible transforming
 * Use git hash-object and git update-index to stage files manually.

Doing the former requires keeping an up-to-date list of all attributes which
can transform files when committing or checking out. Doing the latter is
difficult, error-prone and slow when done from scripts.

Instead, similarly to git hash-object, --no-filter can be added to git add
to enable temporarily disabling filtering in an easy to use way.

The use case here is mostly non-interactive use in scripts creating Git
trees that need to exactly correspond to a working directory regardless of
whether or not they have any .gitattributes files.

For example, for git-buildpackage or dgit, which facilitate Git workflows
with Debian packages, want to ensure the contents of the packages can be
exactly reproduced, which is difficult if the upstream’s tarball has
.gitattributes. It is possible to "defuse" the attributes as demonstrated
above, but this will break if the user modifies the .git/i/a file or if Git
adds more attribute-based conversions. This is what dgit currently does, and
this is what git-buildpackage will soon do too.

Of course, this patch set only addresses staging files. While working on a
patch to git-buildpackage to reproducibly import the contents of tarballs, I
realised that the only realistic way seem to do that is to use hash-object +
update-index manually, which is likely to come with a performance drop
compared to git add (which is what gbp currently uses). A workaround might
be to use Dulwich (which would allow to do hash-object without fork/exec) or
perhaps GitPython (which I haven’t really looked into), or maybe to use git
fast-import, but all of these alternatives are quite complex and don’t
guarantee the same performance.

Adding a new option to git add allows to keep the performance without having
to ensure attributes are set to the right values. The attributes will likely
have to be adjusted anyway for user’s convenience, but at least if they
modify them afterwards, the tools won’t break.

These patches:

 * Add new flag ADD_CACHE_RAW to add_to_index()
 * Add new flag HASH_RAW to index_fd()
 * Make git hash-object use the new HASH_RAW flag for consistency
 * Add tests for the new git-add option.
 * Update the git-add manpage describing the new option and pointing out
   it’s tricky to use.
 * Expand the relevant FAQ entry adding --no-filters as yet one more reason
   for perpetually modified files to appear.

Changes since v1:

 * Removed an extra space left in the option definition. Jessica Clarke on
   GitHub pointed out the inconsistent formatting, but contrary to the
   suggestion I updated the style to the more widespread variation: "(0,"
   occurs at least 689 times across the codebase, while "( 0" only at most
 * Expanded the option description to warn users about potentially confusing
   situations (pointed out by brian m. carlson)
 * Expanded the FAQ entry to mention the new option as one of the cause of
   perpetually modified files.

Andrej Shadura (2):
  add: add option --no-filters to disable attribute-based filtering
  hash-object: use the new HASH_RAW flag instead of setting path to NULL

 Documentation/git-add.txt | 10 ++++++++-
 Documentation/gitfaq.txt  |  7 ++++++
 builtin/add.c             |  3 +++
 builtin/hash-object.c     | 17 ++++++---------
 cache.h                   |  2 ++
 object-file.c             |  2 +-
 read-cache.c              |  3 +++
 t/ | 46 +++++++++++++++++++++++++++++++++++++++
 8 files changed, 78 insertions(+), 12 deletions(-)
 create mode 100755 t/

base-commit: 2283e0e9af55689215afa39c03beb2315ce18e83
Fetch-It-Via: git fetch pr-880/andrewshadura/git-add-no-filters-v2

Range-diff vs v1:

 1:  cf8eb886a1e2 ! 1:  1d84503fff48 add: add option --no-filters to disable attribute-based filtering
     @@ Documentation/git-add.txt: for "git add --no-all <pathspec>...", i.e. ignored re
      +	Add the contents as is, ignoring any input filter that would
      +	have been chosen by the attributes mechanism, including the end-of-line
     -+	conversion.
     ++	conversion. Note that this option is not intended for interactive use,
     ++	since files added this way will always show up as modified if Git were
     ++	to apply transformations to them, making the situation potentially
     ++	very confusing.
       	Override the executable bit of the added files.  The executable
       	bit is only changed in the index, the files on disk are left
     + ## Documentation/gitfaq.txt ##
     +@@ Documentation/gitfaq.txt: following on an otherwise clean working tree:
     + ----
     + $ git add --renormalize .
     + ----
     ++Another situation where perpetually modified may appear on any platform is when
     ++a file has been committed without running any filters (including the end-of-line
     ++conversion), but the `.gitattributes` file states that this file requires a
     ++conversion.  In this case, you can either renormalize the files if this happened
     ++by mistake, or modify `.gitattributes` or `$GIT_DIR/info/attributes` as described
     ++above to exempt the file from the conversion if this was intentional.
     + [[recommended-storage-settings]]
     + What's the recommended way to store files in Git?::
       ## builtin/add.c ##
      @@ builtin/add.c: static const char * const builtin_add_usage[] = {
       static int patch_interactive, add_interactive, edit_interactive;
 2:  0e113e958b35 ! 2:  810d4005fe8f hash-object: use the new HASH_RAW flag instead of setting path to NULL
     @@ builtin/hash-object.c: int cmd_hash_object(int argc, const char **argv, const ch
       		OPT_COUNTUP( 0 , "stdin", &hashstdin, N_("read the object from stdin")),
       		OPT_BOOL( 0 , "stdin-paths", &stdin_paths, N_("read file names from stdin")),
      -		OPT_BOOL( 0 , "no-filters", &no_filters, N_("store file as is without filters")),
     -+		OPT_BIT(0 , "no-filters", &flags, N_("store file as is without filters"),
     ++		OPT_BIT(0, "no-filters", &flags, N_("store file as is without filters"),
      +			HASH_RAW),
       		OPT_BOOL( 0, "literally", &literally, N_("just hash any random garbage to create corrupt objects for debugging Git")),
       		OPT_STRING( 0 , "path", &vpath, N_("file"), N_("process file as it were from this path")),


  parent reply	other threads:[~2021-02-20 17:10 UTC|newest]

Thread overview: 11+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-19 18:12 [PATCH 0/2] Add --no-filters option to git-add Andrej Shadura via GitGitGadget
2021-02-19 18:12 ` [PATCH 1/2] add: add option --no-filters to disable attribute-based filtering Andrej Shadura via GitGitGadget
2021-02-19 18:12 ` [PATCH 2/2] hash-object: use the new HASH_RAW flag instead of setting path to NULL Andrej Shadura via GitGitGadget
2021-02-19 22:36 ` [PATCH 0/2] Add --no-filters option to git-add brian m. carlson
2021-02-20  8:06   ` Andrej Shadura
2021-02-20  9:30     ` Andrej Shadura
2021-02-20 14:10       ` brian m. carlson
2021-02-20 17:07 ` Andrej Shadura via GitGitGadget [this message]
2021-02-20 17:07   ` [PATCH v2 1/2] add: add option --no-filters to disable attribute-based filtering Andrej Shadura via GitGitGadget
2021-02-20 17:07   ` [PATCH v2 2/2] hash-object: use the new HASH_RAW flag instead of setting path to NULL Andrej Shadura via GitGitGadget
2021-02-20 21:34   ` [PATCH v2 0/2] Add --no-filters option to git-add Junio C Hamano

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).