From: lars.schneider@autodesk.com
To: git@vger.kernel.org
Cc: gitster@pobox.com, tboegi@web.de, j6t@kdbg.org,
sunshine@sunshineco.com, peff@peff.net,
ramsay@ramsayjones.plus.com, Johannes.Schindelin@gmx.de,
Lars Schneider <larsxschneider@gmail.com>
Subject: [PATCH v4 0/6] convert: add support for different encodings
Date: Sat, 20 Jan 2018 16:24:12 +0100 [thread overview]
Message-ID: <20180120152418.52859-1-lars.schneider@autodesk.com> (raw)
From: Lars Schneider <larsxschneider@gmail.com>
Hi,
Patches 1-4 and 6 are preparation and helper functions.
Patch 5 is the actual change.
This series depends on Torsten's "convert_to_git(): safe_crlf/checksafe
becomes int conv_flags" patch:
https://public-inbox.org/git/20180113224931.27031-1-tboegi@web.de/
Changes since v3:
* I renamed the attribute from "checkout-encoding" to "working-tree-encoding"
in the hope to convey better what the attribute is about.
* I rebased the series to Git 2.16 and removed Torsten's patch as he
posted the patch on his own.
* Fix documentation wording. (Torsten)
* A macro was used in a commit before it's introduction. Fixed!(Junio)
Thanks,
Lars
RFC: https://public-inbox.org/git/BDB9B884-6D17-4BE3-A83C-F67E2AFA2B46@gmail.com/
v1: https://public-inbox.org/git/20171211155023.1405-1-lars.schneider@autodesk.com/
v2: https://public-inbox.org/git/20171229152222.39680-1-lars.schneider@autodesk.com/
v3: https://public-inbox.org/git/20180106004808.77513-1-lars.schneider@autodesk.com/
Base Ref:
Web-Diff: https://github.com/larsxschneider/git/commit/21f4dac5ab
Checkout: git fetch https://github.com/larsxschneider/git encoding-v4 && git checkout 21f4dac5ab
### Interdiff (v3-rebased-2.16..v4):
diff --git a/Documentation/gitattributes.txt b/Documentation/gitattributes.txt
index 1bc03e69cb..a8dbf4be30 100644
--- a/Documentation/gitattributes.txt
+++ b/Documentation/gitattributes.txt
@@ -272,8 +272,8 @@ few exceptions. Even though...
catch potential problems early, safety triggers.
-`checkout-encoding`
-^^^^^^^^^^^^^^^^^^^
+`working-tree-encoding`
+^^^^^^^^^^^^^^^^^^^^^^^
Git recognizes files encoded with ASCII or one of its supersets (e.g.
UTF-8 or ISO-8859-1) as text files. All other encodings are usually
@@ -281,17 +281,17 @@ interpreted as binary and consequently built-in Git text processing
tools (e.g. 'git diff') as well as most Git web front ends do not
visualize the content.
-In these cases you can teach Git the encoding of a file in the working
-directory with the `checkout-encoding` attribute. If a file with this
+In these cases you can tell Git the encoding of a file in the working
+directory with the `working-tree-encoding` attribute. If a file with this
attributes is added to Git, then Git reencodes the content from the
specified encoding to UTF-8 and stores the result in its internal data
structure (called "the index"). On checkout the content is encoded
back to the specified encoding.
-Please note that using the `checkout-encoding` attribute may have a
+Please note that using the `working-tree-encoding` attribute may have a
number of pitfalls:
-- Git clients that do not support the `checkout-encoding` attribute
+- Git clients that do not support the `working-tree-encoding` attribute
will checkout the respective files UTF-8 encoded and not in the
expected encoding. Consequently, these files will appear different
which typically causes trouble. This is in particular the case for
@@ -304,7 +304,7 @@ number of pitfalls:
- Reencoding content requires resources that might slow down certain
Git operations (e.g 'git checkout' or 'git add').
-Use the `checkout-encoding` attribute only if you cannot store a file in
+Use the `working-tree-encoding` attribute only if you cannot store a file in
UTF-8 encoding and if you want Git to be able to process the content as
text.
@@ -313,7 +313,7 @@ with byte order mark (BOM) and you want Git to perform automatic line
ending conversion based on your platform.
------------------------
-*.txt text checkout-encoding=UTF-16
+*.txt text working-tree-encoding=UTF-16
------------------------
Use the following attributes if your '*.txt' files are UTF-16 little
@@ -321,7 +321,7 @@ endian encoded without BOM and you want Git to use Windows line endings
in the working directory.
------------------------
-*.txt checkout-encoding=UTF-16LE text eol=CRLF
+*.txt working-tree-encoding=UTF-16LE text eol=CRLF
------------------------
You can get a list of all available encodings on your platform with the
diff --git a/convert.c b/convert.c
index 8559651b3f..13fad490ce 100644
--- a/convert.c
+++ b/convert.c
@@ -323,7 +323,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,
const char *advise_msg = _(
"You told Git to treat '%s' as %s. A byte order mark "
"(BOM) is prohibited with this encoding. Either use "
- "%.6s as checkout encoding or remove the BOM from the "
+ "%.6s as working tree encoding or remove the BOM from the "
"file.");
advise(advise_msg, path, enc->name, enc->name, enc->name);
@@ -339,7 +339,7 @@ static int encode_to_git(const char *path, const char *src, size_t src_len,
const char *advise_msg = _(
"You told Git to treat '%s' as %s. A byte order mark "
"(BOM) is required with this encoding. Either use "
- "%sBE/%sLE as checkout encoding or add a BOM to the "
+ "%sBE/%sLE as working tree encoding or add a BOM to the "
"file.");
advise(advise_msg, path, enc->name, enc->name, enc->name);
if (conv_flags & CONV_WRITE_OBJECT)
@@ -1237,7 +1237,7 @@ static void convert_attrs(struct conv_attrs *ca, const char *path)
if (!check) {
check = attr_check_initl("crlf", "ident", "filter",
- "eol", "text", "checkout-encoding",
+ "eol", "text", "working-tree-encoding",
NULL);
user_convert_tail = &user_convert;
encoding_tail = &encoding;
diff --git a/t/t0028-checkout-encoding.sh b/t/t0028-working-tree-encoding.sh
similarity index 89%
rename from t/t0028-checkout-encoding.sh
rename to t/t0028-working-tree-encoding.sh
index 5f1c911c07..0f36d4990a 100755
--- a/t/t0028-checkout-encoding.sh
+++ b/t/t0028-working-tree-encoding.sh
@@ -1,6 +1,6 @@
#!/bin/sh
-test_description='checkout-encoding conversion via gitattributes'
+test_description='working-tree-encoding conversion via gitattributes'
. ./test-lib.sh
@@ -10,7 +10,7 @@ test_expect_success 'setup test repo' '
git config core.eol lf &&
text="hallo there!\ncan you read me?" &&
- echo "*.utf16 text checkout-encoding=utf-16" >.gitattributes &&
+ echo "*.utf16 text working-tree-encoding=utf-16" >.gitattributes &&
printf "$text" >test.utf8.raw &&
printf "$text" | iconv -f UTF-8 -t UTF-16 >test.utf16.raw &&
cp test.utf16.raw test.utf16 &&
@@ -45,10 +45,10 @@ test_expect_success 'check prohibited UTF BOM' '
printf "\0\0\376\777\0\0\0a\0\0\0b\0\0\0c" >bebom.utf32be.raw &&
printf "\777\376\0\0a\0\0\0b\0\0\0c\0\0\0" >lebom.utf32le.raw &&
- echo "*.utf16be text checkout-encoding=utf-16be" >>.gitattributes &&
- echo "*.utf16le text checkout-encoding=utf-16le" >>.gitattributes &&
- echo "*.utf32be text checkout-encoding=utf-32be" >>.gitattributes &&
- echo "*.utf32le text checkout-encoding=utf-32le" >>.gitattributes &&
+ echo "*.utf16be text working-tree-encoding=utf-16be" >>.gitattributes &&
+ echo "*.utf16le text working-tree-encoding=utf-16le" >>.gitattributes &&
+ echo "*.utf32be text working-tree-encoding=utf-32be" >>.gitattributes &&
+ echo "*.utf32le text working-tree-encoding=utf-32le" >>.gitattributes &&
# Here we add a UTF-16 files with BOM (big-endian and little-endian)
# but we tell Git to treat it as UTF-16BE/UTF-16LE. In these cases
@@ -91,7 +91,7 @@ test_expect_success 'check prohibited UTF BOM' '
'
test_expect_success 'check required UTF BOM' '
- echo "*.utf32 text checkout-encoding=utf-32" >>.gitattributes &&
+ echo "*.utf32 text working-tree-encoding=utf-32" >>.gitattributes &&
cp nobom.utf16be.raw nobom.utf16 &&
test_must_fail git add nobom.utf16 2>err.out &&
@@ -143,11 +143,11 @@ test_expect_success 'eol conversion for UTF-16 encoded files on checkout' '
test_expect_success 'check unsupported encodings' '
- echo "*.nothing text checkout-encoding=" >>.gitattributes &&
+ echo "*.nothing text working-tree-encoding=" >>.gitattributes &&
printf "nothing" >t.nothing &&
git add t.nothing &&
- echo "*.garbage text checkout-encoding=garbage" >>.gitattributes &&
+ echo "*.garbage text working-tree-encoding=garbage" >>.gitattributes &&
printf "garbage" >t.garbage &&
test_must_fail git add t.garbage 2>err.out &&
test_i18ngrep "fatal: failed to encode" err.out &&
@@ -161,7 +161,7 @@ test_expect_success 'error if encoding round trip is not the same during refresh
BEFORE_STATE=$(git rev-parse HEAD) &&
# Skip the UTF-16 filter for the added file
- # This simulates a Git version that has no checkoutEncoding support
+ # This simulates a Git version that has no working tree encoding support
echo "hallo" >nonsense.utf16 &&
TEST_HASH=$(git hash-object --no-filters -w nonsense.utf16) &&
git update-index --add --cacheinfo 100644 $TEST_HASH nonsense.utf16 &&
### Patches
Lars Schneider (6):
strbuf: remove unnecessary NUL assignment in xstrdup_tolower()
strbuf: add xstrdup_toupper()
utf8: add function to detect prohibited UTF-16/32 BOM
utf8: add function to detect a missing UTF-16/32 BOM
convert: add 'working-tree-encoding' attribute
convert: add tracing for 'working-tree-encoding' attribute
Documentation/gitattributes.txt | 60 +++++++++++
convert.c | 218 ++++++++++++++++++++++++++++++++++++++-
convert.h | 1 +
sha1_file.c | 2 +-
strbuf.c | 13 ++-
strbuf.h | 1 +
t/t0028-working-tree-encoding.sh | 198 +++++++++++++++++++++++++++++++++++
utf8.c | 37 +++++++
utf8.h | 25 +++++
9 files changed, 552 insertions(+), 3 deletions(-)
create mode 100755 t/t0028-working-tree-encoding.sh
base-commit: 8a2f0888555ce46ac87452b194dec5cb66fb1417
--
2.16.0
next reply other threads:[~2018-01-20 15:24 UTC|newest]
Thread overview: 43+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-20 15:24 lars.schneider [this message]
2018-01-20 15:24 ` [PATCH v4 1/6] strbuf: remove unnecessary NUL assignment in xstrdup_tolower() lars.schneider
2018-01-20 15:24 ` [PATCH v4 2/6] strbuf: add xstrdup_toupper() lars.schneider
2018-01-20 15:24 ` [PATCH v4 3/6] utf8: add function to detect prohibited UTF-16/32 BOM lars.schneider
2018-01-20 15:24 ` [PATCH v4 4/6] utf8: add function to detect a missing " lars.schneider
2018-01-20 15:24 ` [PATCH v4 5/6] convert: add 'working-tree-encoding' attribute lars.schneider
2018-01-21 14:22 ` Simon Ruderich
2018-01-22 12:35 ` Lars Schneider
2018-01-23 0:54 ` Jeff King
2018-01-23 10:25 ` Simon Ruderich
2018-01-23 16:20 ` Jeff King
2018-01-23 21:12 ` Junio C Hamano
2018-01-23 9:27 ` Simon Ruderich
2018-01-22 18:00 ` SQUASH convert: add tracing for " lars.schneider
2018-01-22 19:37 ` Eric Sunshine
2018-01-23 10:17 ` [PATCH v2] " lars.schneider
2018-01-20 15:24 ` [PATCH v4 6/6] " lars.schneider
2018-01-23 20:19 ` [PATCH v4 0/6] convert: add support for different encodings Torsten Bögershausen
2018-01-23 20:53 ` Junio C Hamano
2018-01-29 20:18 ` [PATCH v5 0/7] " tboegi
2018-01-29 20:18 ` [PATCH v5 1/7] strbuf: remove unnecessary NUL assignment in xstrdup_tolower() tboegi
2018-01-29 20:19 ` [PATCH v5 2/7] strbuf: add xstrdup_toupper() tboegi
2018-01-29 20:19 ` [PATCH v5 3/7] utf8: add function to detect prohibited UTF-16/32 BOM tboegi
2018-01-29 20:19 ` [PATCH v5 4/7] utf8: add function to detect a missing " tboegi
2018-01-30 19:15 ` Junio C Hamano
2018-01-30 20:58 ` Lars Schneider
2018-01-30 21:54 ` Junio C Hamano
2018-01-29 20:19 ` [PATCH v5 5/7] convert: add 'working-tree-encoding' attribute tboegi
2018-01-30 20:05 ` Junio C Hamano
2018-01-30 20:31 ` Lars Schneider
2018-01-30 21:56 ` Junio C Hamano
2018-01-31 19:12 ` Lars Schneider
2018-01-31 22:45 ` Junio C Hamano
2018-01-29 20:19 ` [PATCH v5 6/7] convert: add tracing for " tboegi
2018-01-29 20:19 ` [PATCH/RFC v5 7/7] Careful with CRLF when using e.g. UTF-16 for working-tree-encoding tboegi
2018-01-30 11:23 ` Lars Schneider
2018-01-30 14:40 ` Torsten Bögershausen
2018-01-30 15:14 ` Lars Schneider
2018-01-31 17:28 ` Torsten Bögershausen
2018-01-31 19:37 ` Lars Schneider
2018-02-02 19:17 ` Junio C Hamano
2018-02-07 6:31 ` Torsten Bögershausen
2018-02-07 18:12 ` Junio C Hamano
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180120152418.52859-1-lars.schneider@autodesk.com \
--to=lars.schneider@autodesk.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=larsxschneider@gmail.com \
--cc=peff@peff.net \
--cc=ramsay@ramsayjones.plus.com \
--cc=sunshine@sunshineco.com \
--cc=tboegi@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).