From: Elijah Newren <newren@gmail.com>
To: Junio C Hamano <gitster@pobox.com>
Cc: git@vger.kernel.org, "Eric Sunshine" <sunshine@sunshineco.com>,
"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
"Johannes Sixt" <j6t@kdbg.org>,
"Torsten Bögershausen" <tboegi@web.de>,
"Elijah Newren" <newren@gmail.com>
Subject: [PATCH v6 0/5] Fix and extend encoding handling in fast export/import
Date: Mon, 13 May 2019 21:30:57 -0700 [thread overview]
Message-ID: <20190514043102.31979-1-newren@gmail.com> (raw)
In-Reply-To: <20190513231726.16218-1-newren@gmail.com>
While stress testing `git filter-repo`, I noticed an issue with
encoding; further digging led to the fixes and features in this series.
See the individual commit messages for details.
Changes since v5 (full range-diff below):
* s/utf-8/UTF-8/, as pointed out by Torsten (in commit messages and comments)
* small code cleanup pointed out by Eric
* rewrap the first commit message
Elijah Newren (5):
t9350: fix encoding test to actually test reencoding
fast-import: support 'encoding' commit header
fast-export: avoid stripping encoding header if we cannot reencode
fast-export: differentiate between explicitly UTF-8 and implicitly
UTF-8
fast-export: do automatic reencoding of commit messages only if
requested
Documentation/git-fast-export.txt | 7 ++
Documentation/git-fast-import.txt | 7 ++
builtin/fast-export.c | 55 ++++++++++++--
fast-import.c | 11 ++-
t/t9300-fast-import.sh | 20 +++++
t/t9350-fast-export.sh | 78 +++++++++++++++++---
t/t9350/broken-iso-8859-7-commit-message.txt | 1 +
t/t9350/simple-iso-8859-7-commit-message.txt | 1 +
8 files changed, 163 insertions(+), 17 deletions(-)
create mode 100644 t/t9350/broken-iso-8859-7-commit-message.txt
create mode 100644 t/t9350/simple-iso-8859-7-commit-message.txt
Range-diff:
1: 37a68a0ffd ! 1: b5dcdab662 t9350: fix encoding test to actually test reencoding
@@ -2,13 +2,13 @@
t9350: fix encoding test to actually test reencoding
- This test used an author with non-ascii characters in the name, but
- no special commit message. It then grep'ed for those non-ascii
- characters, but those are guaranteed to exist regardless of the
- reencoding process since the reencoding only affects the commit message,
- not the author or committer names. As such, the test would work even if
- the re-encoding process simply stripped the commit message entirely.
- Modify the test to actually check that the reencoding in utf-8 worked.
+ This test used an author with non-ascii characters in the name, but no
+ special commit message. It then grep'ed for those non-ascii characters,
+ but those are guaranteed to exist regardless of the reencoding process
+ since the reencoding only affects the commit message, not the author or
+ committer names. As such, the test would work even if the re-encoding
+ process simply stripped the commit message entirely. Modify the test to
+ actually check that the reencoding into UTF-8 worked.
Signed-off-by: Elijah Newren <newren@gmail.com>
@@ -40,7 +40,7 @@
+ # The commit object, if not re-encoded, would be 240 bytes.
+ # Removing the "encoding iso-8859-7\n" header drops 20 bytes.
+ # Re-encoding the Pi character from \xF0 (\360) in iso-8859-7
-+ # to \xCF\x80 (\317\200) in utf-8 adds a byte. Check for
++ # to \xCF\x80 (\317\200) in UTF-8 adds a byte. Check for
+ # the expected size.
+ test 221 -eq "$(git cat-file -s i18n)" &&
+ # ...and for the expected translation of bytes.
2: 3d84f4613d ! 2: af7d4e18fa fast-import: support 'encoding' commit header
@@ -2,7 +2,7 @@
fast-import: support 'encoding' commit header
- Since git supports commit messages with an encoding other than utf-8,
+ Since git supports commit messages with an encoding other than UTF-8,
allow fast-import to import such commits. This may be useful for folks
who do not want to reencode commit messages from an external system, and
may also be useful to achieve reversible history rewrites (e.g. sha1sum
3: baa8394a3a ! 3: d5b300692a fast-export: avoid stripping encoding header if we cannot reencode
@@ -3,8 +3,8 @@
fast-export: avoid stripping encoding header if we cannot reencode
When fast-export encounters a commit with an 'encoding' header, it tries
- to reencode in utf-8 and then drops the encoding header. However, if it
- fails to reencode in utf-8 because e.g. one of the characters in the
+ to reencode in UTF-8 and then drops the encoding header. However, if it
+ fails to reencode in UTF-8 because e.g. one of the characters in the
commit message was invalid in the old encoding, then we need to retain
the original encoding or otherwise we lose information needed to
understand all the other (valid) characters in the original commit
4: 49960164c6 ! 4: 2cef40c613 fast-export: differentiate between explicitly utf-8 and implicitly utf-8
@@ -1,11 +1,11 @@
Author: Elijah Newren <newren@gmail.com>
- fast-export: differentiate between explicitly utf-8 and implicitly utf-8
+ fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8
The find_encoding() function returned the encoding used by a commit
- message, returning a default of git_commit_encoding (usually utf-8).
+ message, returning a default of git_commit_encoding (usually UTF-8).
Although the current code does not differentiate between a commit which
- explicitly requested utf-8 and one where we just assume utf-8 because no
+ explicitly requested UTF-8 and one where we just assume UTF-8 because no
encoding is set, it will become important when we try to preserve the
encoding header. Since is_encoding_utf8() returns true when passed
NULL, we can just return NULL from find_encoding() instead of returning
5: d8be4ee826 ! 5: d18f03d1bf fast-export: do automatic reencoding of commit messages only if requested
@@ -62,7 +62,7 @@
+ reencode_mode = REENCODE_YES;
+ break;
+ default:
-+ if (arg && !strcasecmp(arg, "abort"))
++ if (!strcasecmp(arg, "abort"))
+ reencode_mode = REENCODE_ABORT;
+ else
+ return error("Unknown reencoding mode: %s", arg);
@@ -156,7 +156,7 @@
+ # The commit object, if not re-encoded, is 240 bytes.
+ # Removing the "encoding iso-8859-7\n" header would drops 20
+ # bytes. Re-encoding the Pi character from \xF0 (\360) in
-+ # iso-8859-7 to \xCF\x80 (\317\200) in utf-8 adds a byte.
++ # iso-8859-7 to \xCF\x80 (\317\200) in UTF-8 adds a byte.
+ # Check for the expected size...
+ test 240 -eq "$(git cat-file -s i18n-no-recoding)" &&
+ # ...as well as the expected byte.
--
2.21.0.782.gd18f03d1bf
next prev parent reply other threads:[~2019-05-14 4:31 UTC|newest]
Thread overview: 45+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-30 18:25 [PATCH v2 0/5] Fix and extend encoding handling in fast export/import Elijah Newren
2019-04-30 18:25 ` [PATCH v2 1/5] t9350: fix encoding test to actually test reencoding Elijah Newren
2019-04-30 18:25 ` [PATCH v2 2/5] fast-import: support 'encoding' commit header Elijah Newren
2019-04-30 18:25 ` [PATCH v2 3/5] fast-export: avoid stripping encoding header if we cannot reencode Elijah Newren
2019-04-30 18:25 ` [PATCH v2 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8 Elijah Newren
2019-04-30 18:25 ` [PATCH v2 5/5] fast-export: do automatic reencoding of commit messages only if requested Elijah Newren
2019-05-10 20:53 ` [PATCH v3 0/5] Fix and extend encoding handling in fast export/import Elijah Newren
2019-05-10 20:53 ` [PATCH v3 1/5] t9350: fix encoding test to actually test reencoding Elijah Newren
2019-05-10 20:53 ` [PATCH v3 2/5] fast-import: support 'encoding' commit header Elijah Newren
2019-05-10 20:53 ` [PATCH v3 3/5] fast-export: avoid stripping encoding header if we cannot reencode Elijah Newren
2019-05-10 20:53 ` [PATCH v3 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8 Elijah Newren
2019-05-10 20:53 ` [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested Elijah Newren
2019-05-11 21:07 ` Torsten Bögershausen
2019-05-11 21:42 ` Elijah Newren
2019-05-13 7:48 ` Junio C Hamano
2019-05-13 13:24 ` Elijah Newren
2019-05-13 10:23 ` Johannes Schindelin
2019-05-13 12:56 ` Torsten Bögershausen
2019-05-13 13:29 ` Elijah Newren
2019-05-13 16:41 ` Elijah Newren
2019-05-13 10:14 ` [PATCH v3 0/5] Fix and extend encoding handling in fast export/import Johannes Schindelin
2019-05-13 16:47 ` [PATCH v4 " Elijah Newren
2019-05-13 16:47 ` [PATCH v4 1/5] t9350: fix encoding test to actually test reencoding Elijah Newren
2019-05-13 16:47 ` [PATCH v4 2/5] fast-import: support 'encoding' commit header Elijah Newren
2019-05-13 16:47 ` [PATCH v4 3/5] fast-export: avoid stripping encoding header if we cannot reencode Elijah Newren
2019-05-13 16:47 ` [PATCH v4 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8 Elijah Newren
2019-05-13 16:47 ` [PATCH v4 5/5] fast-export: do automatic reencoding of commit messages only if requested Elijah Newren
2019-05-13 22:32 ` Junio C Hamano
2019-05-13 23:17 ` [PATCH v5 0/5] Fix and extend encoding handling in fast export/import Elijah Newren
2019-05-13 23:17 ` [PATCH v5 1/5] t9350: fix encoding test to actually test reencoding Elijah Newren
2019-05-14 2:50 ` Torsten Bögershausen
2019-05-13 23:17 ` [PATCH v5 2/5] fast-import: support 'encoding' commit header Elijah Newren
2019-05-13 23:17 ` [PATCH v5 3/5] fast-export: avoid stripping encoding header if we cannot reencode Elijah Newren
2019-05-14 2:56 ` Torsten Bögershausen
2019-05-13 23:17 ` [PATCH v5 4/5] fast-export: differentiate between explicitly utf-8 and implicitly utf-8 Elijah Newren
2019-05-14 3:01 ` Torsten Bögershausen
2019-05-13 23:17 ` [PATCH v5 5/5] fast-export: do automatic reencoding of commit messages only if requested Elijah Newren
2019-05-14 0:19 ` Eric Sunshine
2019-05-14 4:30 ` Elijah Newren [this message]
2019-05-14 4:30 ` [PATCH v6 1/5] t9350: fix encoding test to actually test reencoding Elijah Newren
2019-05-14 4:30 ` [PATCH v6 2/5] fast-import: support 'encoding' commit header Elijah Newren
2019-05-14 4:31 ` [PATCH v6 3/5] fast-export: avoid stripping encoding header if we cannot reencode Elijah Newren
2019-05-14 4:31 ` [PATCH v6 4/5] fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8 Elijah Newren
2019-05-14 4:31 ` [PATCH v6 5/5] fast-export: do automatic reencoding of commit messages only if requested Elijah Newren
2019-05-16 18:15 ` [PATCH v6 0/5] Fix and extend encoding handling in fast export/import Torsten Bögershausen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190514043102.31979-1-newren@gmail.com \
--to=newren@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=j6t@kdbg.org \
--cc=sunshine@sunshineco.com \
--cc=tboegi@web.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).