[PATCH] t: use user-specific utf-8 locale for testing

[PATCH] t: use user-specific utf-8 locale for testing

[Đoàn Trần Công Danh]

In some test-cases, utf-8 locale is required. To find such locale,
we're using the first available UTF-8 locale that returned by
"locale -a".

Despite being required by POSIX, locale(1) is unavailable in some
systems, e.g. Linux with musl libc.  Some of those systems support
utf-8 locale out of the box.

However, without "locale -a", we can't guess provided UTF-8 locale.

Let's give users of those systems an option to have better test
coverage.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
 Makefile                                 |  6 ++++++
 t/lib-git-svn.sh                         | 15 +++++++++------
 t/t9100-git-svn-basic.sh                 | 14 +++-----------
 t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
 t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
 5 files changed, 23 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..4b2c24e5ea 100644
--- a/Makefile
+++ b/Makefile
@@ -398,6 +398,9 @@ all::
 # with a different indexfile format version.  If it isn't set the index
 # file format used is index-v[23].
 #
+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
+# If it isn't set, use the first utf-8 locale returned by "locale -a".
+#
 # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
 #
 # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
@@ -2801,6 +2804,9 @@ ifdef GIT_TEST_CMP
 endif
 ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
 	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
+endif
+ifdef GIT_TEST_UTF8_LOCALE
+	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
 endif
 	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
 ifdef GIT_PERF_REPEAT_COUNT
diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 547eb3c31a..df319593f7 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -121,12 +121,15 @@ start_svnserve () {
 		 --listen-host 127.0.0.1 &
 }
 
-prepare_a_utf8_locale () {
-	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
-	p
-	q
-}')
-	if test -n "$a_utf8_locale"
+prepare_utf8_locale () {
+	if test -z "$GIT_TEST_UTF8_LOCALE"
+	then
+		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
+		p
+		q
+	}')
+	fi
+	if test -n "$GIT_TEST_UTF8_LOCALE"
 	then
 		test_set_prereq UTF8
 	else
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 1d3fdcc997..d5563ec35f 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -4,21 +4,13 @@
 #
 
 test_description='git svn basic tests'
-GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
 
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./lib-git-svn.sh
 
-case "$GIT_SVN_LC_ALL" in
-*.UTF-8)
-	test_set_prereq UTF8
-	;;
-*)
-	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
-	;;
-esac
+prepare_utf8_locale
 
 test_expect_success 'git svn --version works anywhere' '
 	nongit git svn --version
@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
 	test ! -h "$SVN_TREE"/exec-2.sh &&
 	test_cmp help "$SVN_TREE"/exec-2.sh'
 
-name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
-LC_ALL="$GIT_SVN_LC_ALL"
+name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
+LC_ALL="$GIT_TEST_UTF8_LOCALE"
 export LC_ALL
 # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
 test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9b44a44bc1..743fbe1fe4 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 # > ... All of the above characters, except for the backslash, are converted
 # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
 # > "Private use area") when creating or accessing files.
-prepare_a_utf8_locale
+prepare_utf8_locale
 test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
 
 # See the comment on the above test for setting of LC_ALL.
 test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index 2c213ae654..01e1e8a8f7 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -14,12 +14,12 @@ compare_git_head_with () {
 	test_cmp current "$1"
 }
 
-prepare_a_utf8_locale
+prepare_utf8_locale
 
 compare_svn_head_with () {
 	# extract just the log message and strip out committer info.
 	# don't use --limit here since svn 1.1.x doesn't have it,
-	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
+	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
 		use bytes;
 		$/ = ("-"x72) . "\n";
 		my @x = <STDIN>;
-- 
2.32.0.rc2.1.gd85b21334f

Re: [PATCH] t: use user-specific utf-8 locale for testing

[Taylor Blau]

On Wed, Jun 02, 2021 at 06:46:46PM +0700, Đoàn Trần Công Danh wrote:
> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.

Hmmph. I would have imagined that locale was available everywhere, but
unfortunately not.

> diff --git a/Makefile b/Makefile
> index c3565fc0f8..4b2c24e5ea 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -398,6 +398,9 @@ all::
>  # with a different indexfile format version.  If it isn't set the index
>  # file format used is index-v[23].
>  #
> +# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
> +# If it isn't set, use the first utf-8 locale returned by "locale -a".

s/prefered/preferred

> +#
>  # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>  #
>  # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
> @@ -2801,6 +2804,9 @@ ifdef GIT_TEST_CMP
>  endif
>  ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
>  	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
> +endif
> +ifdef GIT_TEST_UTF8_LOCALE
> +	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
>  endif
>  	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>  ifdef GIT_PERF_REPEAT_COUNT
> diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
> index 547eb3c31a..df319593f7 100644
> --- a/t/lib-git-svn.sh
> +++ b/t/lib-git-svn.sh
> @@ -121,12 +121,15 @@ start_svnserve () {
>  		 --listen-host 127.0.0.1 &
>  }
>
> -prepare_a_utf8_locale () {
> -	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> -	p
> -	q
> -}')
> -	if test -n "$a_utf8_locale"
> +prepare_utf8_locale () {
> +	if test -z "$GIT_TEST_UTF8_LOCALE"
> +	then
> +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> +		p
> +		q
> +	}')
> +	fi

OK, so we bind GIT_TEST_UTF8_LOCALE to the value of $a_utf8_locale in
the pre-image, unless the user said otherwise.

> +	if test -n "$GIT_TEST_UTF8_LOCALE"

...Then we go on to handle things like before, except we read from
"$GIT_TEST_UTF8_LOCALE" instead of "$a_utf8_locale". Makes sense to me.

>  	then
>  		test_set_prereq UTF8
>  	else
> diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
> index 1d3fdcc997..d5563ec35f 100755
> --- a/t/t9100-git-svn-basic.sh
> +++ b/t/t9100-git-svn-basic.sh
> @@ -4,21 +4,13 @@
>  #
>
>  test_description='git svn basic tests'
> -GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
>
>  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
>  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>
>  . ./lib-git-svn.sh
>
> -case "$GIT_SVN_LC_ALL" in
> -*.UTF-8)
> -	test_set_prereq UTF8
> -	;;
> -*)
> -	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
> -	;;
> -esac
> +prepare_utf8_locale

This change (and the omitted ones below in later hunks) look like it
isn't changing any behavior (and just running the same code behind the
prepare_utf8_locale function instead of inlining it).

They all look right to me, but it may be helpful to either point it out
in the commit message and/or prepare the separately. I'd probably err on
the side of the former.

That said, this patch looks good to me with minor touch-ups (my only
nits are the above and the spelling mistake in the Makefile).

Thanks,
Taylor

Re: [PATCH] t: use user-specific utf-8 locale for testing

[Jeff King]

On Wed, Jun 02, 2021 at 06:46:46PM +0700, Đoàn Trần Công Danh wrote:

> In some test-cases, utf-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".
> 
> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.
> 
> However, without "locale -a", we can't guess provided UTF-8 locale.
> 
> Let's give users of those systems an option to have better test
> coverage.

That makes sense. At first I thought this was fixing a test breakage,
and wondered if we should simply be skipping the tests. But after
reading more carefully, we already are doing that, and this is just
allowing those tests to run in more places (you even said so in the last
sentence above, but I missed it, so I am calling it out).

I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
that people don't even have to set this extra Makefile knob. But I'm not
sure if we have a good way of testing if that locale works (if we can't
find the "locale" binary). So this is probably a reasonable medium
ground.

> ---
>  Makefile                                 |  6 ++++++
>  t/lib-git-svn.sh                         | 15 +++++++++------
>  t/t9100-git-svn-basic.sh                 | 14 +++-----------
>  t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
>  t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
>  5 files changed, 23 insertions(+), 22 deletions(-)

I think we test musl in our ci builds; should we do something like:

diff --git a/ci/lib.sh b/ci/lib.sh
index d848c036c5..ec67398c18 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -229,6 +229,7 @@ linux-musl)
 	CC=gcc
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=something.UTF-8?"
 	;;
 esac
 

here, too (I don't know what the correct "something" is here; maybe
"C")?

-Peff

Re: [PATCH] t: use user-specific utf-8 locale for testing

[Bagas Sanjaya]

Hi Đoàn,

On 02/06/21 18.46, Đoàn Trần Công Danh wrote:
> In some test-cases, utf-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".
> 

On most systems, the first available such locale is C.UTF-8, so why 
don't we just use it?

> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.
> 
> However, without "locale -a", we can't guess provided UTF-8 locale.
> 
> Let's give users of those systems an option to have better test
> coverage.
> 

So can we assume to UTF-8 locale as default for testing on systems 
without locale(1)?

> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
>   Makefile                                 |  6 ++++++
>   t/lib-git-svn.sh                         | 15 +++++++++------
>   t/t9100-git-svn-basic.sh                 | 14 +++-----------
>   t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
>   t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
>   5 files changed, 23 insertions(+), 22 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index c3565fc0f8..4b2c24e5ea 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -398,6 +398,9 @@ all::
>   # with a different indexfile format version.  If it isn't set the index
>   # file format used is index-v[23].
>   #
> +# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
> +# If it isn't set, use the first utf-8 locale returned by "locale -a".
> +#

Better say "... to prefer utf-8 locale for testing".

-- 
An old man doll... just what I always wanted! - Clara

Re: [PATCH] t: use user-specific utf-8 locale for testing

[Đoàn Trần Công Danh]

On 2021-06-04 10:32:04+0700, Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> Hi Đoàn,
> 
> On 02/06/21 18.46, Đoàn Trần Công Danh wrote:
> > In some test-cases, utf-8 locale is required. To find such locale,
> > we're using the first available UTF-8 locale that returned by
> > "locale -a".
> > 
> 
> On most systems, the first available such locale is C.UTF-8, so why don't we
> just use it?

I don't know. I think in Linux with glibc, C.UTF-8 must be enable
explicitly and run "localedef"?
Worse, systems have a right to not have any UTF-8 locales.
Someone may only enable C and en_US.ISO-8859-1

> > Despite being required by POSIX, locale(1) is unavailable in some
> > systems, e.g. Linux with musl libc.  Some of those systems support
> > utf-8 locale out of the box.
> > 
> > However, without "locale -a", we can't guess provided UTF-8 locale.
> > 
> > Let's give users of those systems an option to have better test
> > coverage.
> > 
> 
> So can we assume to UTF-8 locale as default for testing on systems without
> locale(1)?

I don't know, either. POSIX only requires 2 locales, C and POSIX,
All other locales are optional:

	Conforming implementations shall support one or more coded
	character sets. 

I don't know if such systems exist or not.

C and POSIX treats any bytes higher than 0x80 as arbitrary characters [1].

	The POSIX locale shall contain 256 single-byte characters
	including the characters

> > Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> > ---
> >   Makefile                                 |  6 ++++++
> >   t/lib-git-svn.sh                         | 15 +++++++++------
> >   t/t9100-git-svn-basic.sh                 | 14 +++-----------
> >   t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
> >   t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
> >   5 files changed, 23 insertions(+), 22 deletions(-)
> > 
> > diff --git a/Makefile b/Makefile
> > index c3565fc0f8..4b2c24e5ea 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -398,6 +398,9 @@ all::
> >   # with a different indexfile format version.  If it isn't set the index
> >   # file format used is index-v[23].
> >   #
> > +# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
> > +# If it isn't set, use the first utf-8 locale returned by "locale -a".
> > +#
> 
> Better say "... to prefer utf-8 locale for testing".

I intended to say:

	... to *the* preferred ...

We need utf-8 locale to test anyway ;) So, we don't prefer utf-8 locale,
we pick a utf-8 locale from a handful of utf-8 locales.


1: https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap06.html#tag_06_02

-- 
Danh

[PATCH v2] t: use user-specific utf-8 locale for testing

[Đoàn Trần Công Danh]

In some test-cases, utf-8 locale is required. To find such locale,
we're using the first available UTF-8 locale that returned by
"locale -a".

Despite being required by POSIX, locale(1) is unavailable in some
systems, e.g. Linux with musl libc.  Some of those systems support
utf-8 locale out of the box.

However, without "locale -a", we can't guess provided UTF-8 locale.

Let's give users of those systems an option to have better test
coverage.

This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
prepare_utf8_locale, since we no longer prepare the variable named
"a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
instead.  The fallback will be LC_ALL, LANG environment variable,
or the first utf-8 locale from output of "locale -a", in that order.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---
Range-diff against v1:
1:  d242ce64c4 ! 1:  f299ae2239 t: use user-specific utf-8 locale for testing
    @@ Commit message
         Let's give users of those systems an option to have better test
         coverage.
     
    +    This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
    +    prepare_utf8_locale, since we no longer prepare the variable named
    +    "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
    +    instead.  The fallback will be LC_ALL, LANG environment variable,
    +    or the first utf-8 locale from output of "locale -a", in that order.
    +
         Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
     
      ## Makefile ##
    @@ Makefile: all::
      # with a different indexfile format version.  If it isn't set the index
      # file format used is index-v[23].
      #
    -+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
    -+# If it isn't set, use the first utf-8 locale returned by "locale -a".
    ++# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
    ++# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
    ++# locale returned by "locale -a".
     +#
      # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
      #
    @@ Makefile: ifdef GIT_TEST_CMP
      	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
      ifdef GIT_PERF_REPEAT_COUNT
     
    + ## ci/lib.sh ##
    +@@ ci/lib.sh: linux-musl)
    + 	CC=gcc
    + 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
    + 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
    ++	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
    + 	;;
    + esac
    + 
    +
      ## t/lib-git-svn.sh ##
     @@ t/lib-git-svn.sh: start_svnserve () {
      		 --listen-host 127.0.0.1 &
    @@ t/lib-git-svn.sh: start_svnserve () {
     -}')
     -	if test -n "$a_utf8_locale"
     +prepare_utf8_locale () {
    -+	if test -z "$GIT_TEST_UTF8_LOCALE"
    ++	if test -n "$GIT_TEST_UTF8_LOCALE"
    ++	then
    ++		: test_set_prereq UTF8
    ++	elif test -n "${LC_ALL:-$LANG}"
     +	then
    ++		case "${LC_ALL:-$LANG}" in
    ++		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
    ++			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
    ++			;;
    ++		esac
    ++	else
     +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
     +		p
     +		q

 Makefile                                 |  7 +++++++
 ci/lib.sh                                |  1 +
 t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
 t/t9100-git-svn-basic.sh                 | 14 +++-----------
 t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
 t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
 6 files changed, 34 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..502e0c9a81 100644
--- a/Makefile
+++ b/Makefile
@@ -398,6 +398,10 @@ all::
 # with a different indexfile format version.  If it isn't set the index
 # file format used is index-v[23].
 #
+# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
+# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
+# locale returned by "locale -a".
+#
 # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
 #
 # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
@@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
 endif
 ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
 	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
+endif
+ifdef GIT_TEST_UTF8_LOCALE
+	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
 endif
 	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
 ifdef GIT_PERF_REPEAT_COUNT
diff --git a/ci/lib.sh b/ci/lib.sh
index d848c036c5..476c3f369f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -229,6 +229,7 @@ linux-musl)
 	CC=gcc
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
 	;;
 esac
 
diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 547eb3c31a..83efc17661 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -121,12 +121,24 @@ start_svnserve () {
 		 --listen-host 127.0.0.1 &
 }
 
-prepare_a_utf8_locale () {
-	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
-	p
-	q
-}')
-	if test -n "$a_utf8_locale"
+prepare_utf8_locale () {
+	if test -n "$GIT_TEST_UTF8_LOCALE"
+	then
+		: test_set_prereq UTF8
+	elif test -n "${LC_ALL:-$LANG}"
+	then
+		case "${LC_ALL:-$LANG}" in
+		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
+			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
+			;;
+		esac
+	else
+		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
+		p
+		q
+	}')
+	fi
+	if test -n "$GIT_TEST_UTF8_LOCALE"
 	then
 		test_set_prereq UTF8
 	else
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 1d3fdcc997..d5563ec35f 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -4,21 +4,13 @@
 #
 
 test_description='git svn basic tests'
-GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
 
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./lib-git-svn.sh
 
-case "$GIT_SVN_LC_ALL" in
-*.UTF-8)
-	test_set_prereq UTF8
-	;;
-*)
-	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
-	;;
-esac
+prepare_utf8_locale
 
 test_expect_success 'git svn --version works anywhere' '
 	nongit git svn --version
@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
 	test ! -h "$SVN_TREE"/exec-2.sh &&
 	test_cmp help "$SVN_TREE"/exec-2.sh'
 
-name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
-LC_ALL="$GIT_SVN_LC_ALL"
+name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
+LC_ALL="$GIT_TEST_UTF8_LOCALE"
 export LC_ALL
 # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
 test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9b44a44bc1..743fbe1fe4 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 # > ... All of the above characters, except for the backslash, are converted
 # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
 # > "Private use area") when creating or accessing files.
-prepare_a_utf8_locale
+prepare_utf8_locale
 test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
 
 # See the comment on the above test for setting of LC_ALL.
 test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index 2c213ae654..01e1e8a8f7 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -14,12 +14,12 @@ compare_git_head_with () {
 	test_cmp current "$1"
 }
 
-prepare_a_utf8_locale
+prepare_utf8_locale
 
 compare_svn_head_with () {
 	# extract just the log message and strip out committer info.
 	# don't use --limit here since svn 1.1.x doesn't have it,
-	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
+	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
 		use bytes;
 		$/ = ("-"x72) . "\n";
 		my @x = <STDIN>;
-- 
2.32.0.rc3.5.gf3d78db977

Re: [PATCH v2] t: use user-specific utf-8 locale for testing

[Torsten Bögershausen]

This all looks good.
Some suggestions about the commit message are inline.

On Sun, Jun 06, 2021 at 11:33:16PM +0700, Đoàn Trần Công Danh wrote:
> In some test-cases, utf-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".

Good explanation.
I think that in generaral "utf-8" as a specification/specifier is better
written as "UTF-8", with uppercase.
"utf-8" or utf8 may be used inside the code, depending on the language.

>
> Despite being required by POSIX, locale(1) is unavailable in some
> systems, e.g. Linux with musl libc.  Some of those systems support
> utf-8 locale out of the box.
This reads a little bit harsh (the first sentence) and it is not
fully clear which systems do what (the second sentence).
Or are 2 things mentioned - the locale(1) utility and the support
of one UTF-8 locale "out of the box" ?
Does Linux with musl libs support an UTF-8 locale, but not
the locale(1) untility ?
Git itself supports many systems, that are not POSIX compliant,
strictly speaking. But if avaliable, the functions defined in POSIX
are used, whenever available.

Could we write:
However, the locale(1) utility is unavailable on some systems,
e.g. Linux with musl libc.

>
> However, without "locale -a", we can't guess provided UTF-8 locale.
>
> Let's give users of those systems an option to have better test
> coverage.

Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for linux-musl

>
> This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
> prepare_utf8_locale, since we no longer prepare the variable named
> "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
> instead.  The fallback will be LC_ALL, LANG environment variable,
> or the first utf-8 locale from output of "locale -a", in that order.

rename -> renames, may be drop "This change", like this ?
Rename t/lib-git-svn.sh:prepare_a_utf8_locale into prepare_utf8_locale,
since we no longer prepare the variable named "a_utf8_locale",
but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
The fallback will be LC_ALL, LANG environment variable,
or the first UTF-8 locale from output of "locale -a", in that order.


>
> Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
> ---
> Range-diff against v1:
> 1:  d242ce64c4 ! 1:  f299ae2239 t: use user-specific utf-8 locale for testing
>     @@ Commit message
>          Let's give users of those systems an option to have better test
>          coverage.
>
>     +    This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
>     +    prepare_utf8_locale, since we no longer prepare the variable named
>     +    "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
>     +    instead.  The fallback will be LC_ALL, LANG environment variable,
>     +    or the first utf-8 locale from output of "locale -a", in that order.
>     +
>          Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
>
>       ## Makefile ##
>     @@ Makefile: all::
>       # with a different indexfile format version.  If it isn't set the index
>       # file format used is index-v[23].
>       #
>     -+# Define GIT_TEST_UTF8_LOCALE to prefered utf-8 locale for testing.
>     -+# If it isn't set, use the first utf-8 locale returned by "locale -a".
>     ++# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
>     ++# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
>     ++# locale returned by "locale -a".
>      +#
>       # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>       #
>     @@ Makefile: ifdef GIT_TEST_CMP
>       	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>       ifdef GIT_PERF_REPEAT_COUNT
>
>     + ## ci/lib.sh ##
>     +@@ ci/lib.sh: linux-musl)
>     + 	CC=gcc
>     + 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>     + 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
>     ++	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>     + 	;;
>     + esac
>     +
>     +
>       ## t/lib-git-svn.sh ##
>      @@ t/lib-git-svn.sh: start_svnserve () {
>       		 --listen-host 127.0.0.1 &
>     @@ t/lib-git-svn.sh: start_svnserve () {
>      -}')
>      -	if test -n "$a_utf8_locale"
>      +prepare_utf8_locale () {
>     -+	if test -z "$GIT_TEST_UTF8_LOCALE"
>     ++	if test -n "$GIT_TEST_UTF8_LOCALE"
>     ++	then
>     ++		: test_set_prereq UTF8
>     ++	elif test -n "${LC_ALL:-$LANG}"
>      +	then
>     ++		case "${LC_ALL:-$LANG}" in
>     ++		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
>     ++			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
>     ++			;;
>     ++		esac
>     ++	else
>      +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
>      +		p
>      +		q
>
>  Makefile                                 |  7 +++++++
>  ci/lib.sh                                |  1 +
>  t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
>  t/t9100-git-svn-basic.sh                 | 14 +++-----------
>  t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
>  t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
>  6 files changed, 34 insertions(+), 22 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index c3565fc0f8..502e0c9a81 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -398,6 +398,10 @@ all::
>  # with a different indexfile format version.  If it isn't set the index
>  # file format used is index-v[23].
>  #
> +# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
> +# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
> +# locale returned by "locale -a".
> +#
>  # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
>  #
>  # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
> @@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
>  endif
>  ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
>  	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
> +endif
> +ifdef GIT_TEST_UTF8_LOCALE
> +	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
>  endif
>  	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
>  ifdef GIT_PERF_REPEAT_COUNT
> diff --git a/ci/lib.sh b/ci/lib.sh
> index d848c036c5..476c3f369f 100755
> --- a/ci/lib.sh
> +++ b/ci/lib.sh
> @@ -229,6 +229,7 @@ linux-musl)
>  	CC=gcc
>  	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
>  	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
> +	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
>  	;;
>  esac
>
> diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
> index 547eb3c31a..83efc17661 100644
> --- a/t/lib-git-svn.sh
> +++ b/t/lib-git-svn.sh
> @@ -121,12 +121,24 @@ start_svnserve () {
>  		 --listen-host 127.0.0.1 &
>  }
>
> -prepare_a_utf8_locale () {
> -	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> -	p
> -	q
> -}')
> -	if test -n "$a_utf8_locale"
> +prepare_utf8_locale () {
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
> +	then
> +		: test_set_prereq UTF8
> +	elif test -n "${LC_ALL:-$LANG}"
> +	then
> +		case "${LC_ALL:-$LANG}" in
> +		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
> +			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
> +			;;
> +		esac
> +	else
> +		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
> +		p
> +		q
> +	}')
> +	fi
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
>  	then
>  		test_set_prereq UTF8
>  	else
> diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
> index 1d3fdcc997..d5563ec35f 100755
> --- a/t/t9100-git-svn-basic.sh
> +++ b/t/t9100-git-svn-basic.sh
> @@ -4,21 +4,13 @@
>  #
>
>  test_description='git svn basic tests'
> -GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
>
>  GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
>  export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
>
>  . ./lib-git-svn.sh
>
> -case "$GIT_SVN_LC_ALL" in
> -*.UTF-8)
> -	test_set_prereq UTF8
> -	;;
> -*)
> -	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
> -	;;
> -esac
> +prepare_utf8_locale
>
>  test_expect_success 'git svn --version works anywhere' '
>  	nongit git svn --version
> @@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
>  	test ! -h "$SVN_TREE"/exec-2.sh &&
>  	test_cmp help "$SVN_TREE"/exec-2.sh'
>
> -name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
> -LC_ALL="$GIT_SVN_LC_ALL"
> +name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
> +LC_ALL="$GIT_TEST_UTF8_LOCALE"
>  export LC_ALL
>  # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
>  test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
> diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
> index 9b44a44bc1..743fbe1fe4 100755
> --- a/t/t9115-git-svn-dcommit-funky-renames.sh
> +++ b/t/t9115-git-svn-dcommit-funky-renames.sh
> @@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
>  # > ... All of the above characters, except for the backslash, are converted
>  # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
>  # > "Private use area") when creating or accessing files.
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>  test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	neq=$(printf "\201\202") &&
>  	git config svn.pathnameencoding cp932 &&
> @@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
>
>  # See the comment on the above test for setting of LC_ALL.
>  test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
> -	LC_ALL=$a_utf8_locale &&
> +	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
>  	export LC_ALL &&
>  	inf=$(printf "\201\207") &&
>  	git config svn.pathnameencoding cp932 &&
> diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
> index 2c213ae654..01e1e8a8f7 100755
> --- a/t/t9129-git-svn-i18n-commitencoding.sh
> +++ b/t/t9129-git-svn-i18n-commitencoding.sh
> @@ -14,12 +14,12 @@ compare_git_head_with () {
>  	test_cmp current "$1"
>  }
>
> -prepare_a_utf8_locale
> +prepare_utf8_locale
>
>  compare_svn_head_with () {
>  	# extract just the log message and strip out committer info.
>  	# don't use --limit here since svn 1.1.x doesn't have it,
> -	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
> +	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
>  		use bytes;
>  		$/ = ("-"x72) . "\n";
>  		my @x = <STDIN>;
> --
> 2.32.0.rc3.5.gf3d78db977
>

Re: [PATCH v2] t: use user-specific utf-8 locale for testing

[Junio C Hamano]

Torsten Bögershausen <tboegi@web.de> writes:

> This all looks good.
> Some suggestions about the commit message are inline.

All good comments.

I have another one on the title.

> Subject: Re: [PATCH v2] t: use user-specific utf-8 locale for testing

If we auto-detect to customize which locale to use per user, that is
already user-specific; letting the user to explicitly tell us which
locale to use, that is "use user-specified UTF-8 locale".

This affects only SVN tests and the change is in lib-git-svn, so
when we test other parts of the system for UTF-8, this change will
not directly help, right?  Perhaps we want to mention SVN somewhere
on the title?  Perhaps "for testing git-svn"?

Thanks.

[PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Đoàn Trần Công Danh]

In some test-cases, UTF-8 locale is required. To find such locale,
we're using the first available UTF-8 locale that returned by
"locale -a".

However, the locale(1) utility is unavailable on some systems,
e.g. Linux with musl libc.

However, without "locale -a", we can't guess provided UTF-8 locale.

Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for
linux-musl in our CI system.

Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale,
since we no longer prepare the variable named "a_utf8_locale",
but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
The fallback will be LC_ALL, LANG environment variable,
or the first UTF-8 locale from output of "locale -a", in that order.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---

Jeff King wrote:

> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
> that people don't even have to set this extra Makefile knob. But I'm not
> sure if we have a good way of testing if that locale works (if we can't
> find the "locale" binary).

I also think we should fallback to "C.UTF-8" instead of not testing those
tests.  However, I don't know if there're any systems that not have "C.UTF-8"
locale.


Range-diff against v2:
1:  f299ae2239 ! 1:  064cfc0fe3 t: use user-specific utf-8 locale for testing
    @@ Metadata
     Author: Đoàn Trần Công Danh <congdanhqx@gmail.com>
     
      ## Commit message ##
    -    t: use user-specific utf-8 locale for testing
    +    t: use pre-defined utf-8 locale for testing svn
     
    -    In some test-cases, utf-8 locale is required. To find such locale,
    +    In some test-cases, UTF-8 locale is required. To find such locale,
         we're using the first available UTF-8 locale that returned by
         "locale -a".
     
    -    Despite being required by POSIX, locale(1) is unavailable in some
    -    systems, e.g. Linux with musl libc.  Some of those systems support
    -    utf-8 locale out of the box.
    +    However, the locale(1) utility is unavailable on some systems,
    +    e.g. Linux with musl libc.
     
         However, without "locale -a", we can't guess provided UTF-8 locale.
     
    -    Let's give users of those systems an option to have better test
    -    coverage.
    +    Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for
    +    linux-musl in our CI system.
     
    -    This change also rename t/lib-git-svn.sh:prepare_a_utf8_locale to
    -    prepare_utf8_locale, since we no longer prepare the variable named
    -    "a_utf8_locale" but set up a fallback value for GIT_TEST_UTF8_LOCALE
    -    instead.  The fallback will be LC_ALL, LANG environment variable,
    -    or the first utf-8 locale from output of "locale -a", in that order.
    +    Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale,
    +    since we no longer prepare the variable named "a_utf8_locale",
    +    but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
    +    The fallback will be LC_ALL, LANG environment variable,
    +    or the first UTF-8 locale from output of "locale -a", in that order.
     
         Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
     

 Makefile                                 |  7 +++++++
 ci/lib.sh                                |  1 +
 t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
 t/t9100-git-svn-basic.sh                 | 14 +++-----------
 t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
 t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
 6 files changed, 34 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..502e0c9a81 100644
--- a/Makefile
+++ b/Makefile
@@ -398,6 +398,10 @@ all::
 # with a different indexfile format version.  If it isn't set the index
 # file format used is index-v[23].
 #
+# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
+# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
+# locale returned by "locale -a".
+#
 # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
 #
 # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
@@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
 endif
 ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
 	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
+endif
+ifdef GIT_TEST_UTF8_LOCALE
+	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
 endif
 	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
 ifdef GIT_PERF_REPEAT_COUNT
diff --git a/ci/lib.sh b/ci/lib.sh
index d848c036c5..476c3f369f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -229,6 +229,7 @@ linux-musl)
 	CC=gcc
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
 	;;
 esac
 
diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 547eb3c31a..83efc17661 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -121,12 +121,24 @@ start_svnserve () {
 		 --listen-host 127.0.0.1 &
 }
 
-prepare_a_utf8_locale () {
-	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
-	p
-	q
-}')
-	if test -n "$a_utf8_locale"
+prepare_utf8_locale () {
+	if test -n "$GIT_TEST_UTF8_LOCALE"
+	then
+		: test_set_prereq UTF8
+	elif test -n "${LC_ALL:-$LANG}"
+	then
+		case "${LC_ALL:-$LANG}" in
+		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
+			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
+			;;
+		esac
+	else
+		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
+		p
+		q
+	}')
+	fi
+	if test -n "$GIT_TEST_UTF8_LOCALE"
 	then
 		test_set_prereq UTF8
 	else
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 1d3fdcc997..d5563ec35f 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -4,21 +4,13 @@
 #
 
 test_description='git svn basic tests'
-GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
 
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./lib-git-svn.sh
 
-case "$GIT_SVN_LC_ALL" in
-*.UTF-8)
-	test_set_prereq UTF8
-	;;
-*)
-	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
-	;;
-esac
+prepare_utf8_locale
 
 test_expect_success 'git svn --version works anywhere' '
 	nongit git svn --version
@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
 	test ! -h "$SVN_TREE"/exec-2.sh &&
 	test_cmp help "$SVN_TREE"/exec-2.sh'
 
-name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
-LC_ALL="$GIT_SVN_LC_ALL"
+name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
+LC_ALL="$GIT_TEST_UTF8_LOCALE"
 export LC_ALL
 # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
 test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9b44a44bc1..743fbe1fe4 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 # > ... All of the above characters, except for the backslash, are converted
 # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
 # > "Private use area") when creating or accessing files.
-prepare_a_utf8_locale
+prepare_utf8_locale
 test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
 
 # See the comment on the above test for setting of LC_ALL.
 test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index 2c213ae654..01e1e8a8f7 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -14,12 +14,12 @@ compare_git_head_with () {
 	test_cmp current "$1"
 }
 
-prepare_a_utf8_locale
+prepare_utf8_locale
 
 compare_svn_head_with () {
 	# extract just the log message and strip out committer info.
 	# don't use --limit here since svn 1.1.x doesn't have it,
-	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
+	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
 		use bytes;
 		$/ = ("-"x72) . "\n";
 		my @x = <STDIN>;
-- 
2.32.0.rc3.5.gf3d78db977

Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Junio C Hamano]

Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:

> Subject: Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

That certainly is better than "user-specific", but a lot worse than
"user-specified".  "pre-defined" leaves it open to question "who
defines it?", and it is a reasonable interpretation that the locale
may be hardcoded in the makefile, but that is not what this patch
does.  Saying "user-specified" would not have such a problem.

>> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
>> that people don't even have to set this extra Makefile knob. But I'm not
>> sure if we have a good way of testing if that locale works (if we can't
>> find the "locale" binary).
>
> I also think we should fallback to "C.UTF-8" instead of not testing those
> tests.  However, I don't know if there're any systems that not have "C.UTF-8"
> locale.

I do share the feeling, but have a hunch that systems lacking
"locale -a" may be either superset of, or has at least large overlap
with, those lacking "C.UTF-8", and the new mechanism introduced here
will cover both of them, so I think it is OK to stop here, at least
for now.

Thanks.

[PATCH v4] t: use user-specified utf-8 locale for testing svn

[Đoàn Trần Công Danh]

In some test-cases, UTF-8 locale is required. To find such locale,
we're using the first available UTF-8 locale that returned by
"locale -a".

However, the locale(1) utility is unavailable on some systems,
e.g. Linux with musl libc.

However, without "locale -a", we can't guess provided UTF-8 locale.

Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for
linux-musl in our CI system.

Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale,
since we no longer prepare the variable named "a_utf8_locale",
but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
The fallback will be LC_ALL, LANG environment variable,
or the first UTF-8 locale from output of "locale -a", in that order.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---

Junio C. Hamano wrote:

> That certainly is better than "user-specific", but a lot worse than
> "user-specified".  "pre-defined" leaves it open to question "who
> defines it?", and it is a reasonable interpretation that the locale
> may be hardcoded in the makefile, but that is not what this patch
> does.  Saying "user-specified" would not have such a problem.

Ahh, I scanned your comment too fast. Sorry for the noise.

Thanks for the quick turn-around.

Range-diff against v3:
1:  064cfc0fe3 ! 1:  6036f670d9 t: use pre-defined utf-8 locale for testing svn
    @@ Metadata
     Author: Đoàn Trần Công Danh <congdanhqx@gmail.com>
     
      ## Commit message ##
    -    t: use pre-defined utf-8 locale for testing svn
    +    t: use user-specified utf-8 locale for testing svn
     
         In some test-cases, UTF-8 locale is required. To find such locale,
         we're using the first available UTF-8 locale that returned by

 Makefile                                 |  7 +++++++
 ci/lib.sh                                |  1 +
 t/lib-git-svn.sh                         | 24 ++++++++++++++++++------
 t/t9100-git-svn-basic.sh                 | 14 +++-----------
 t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
 t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
 6 files changed, 34 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..502e0c9a81 100644
--- a/Makefile
+++ b/Makefile
@@ -398,6 +398,10 @@ all::
 # with a different indexfile format version.  If it isn't set the index
 # file format used is index-v[23].
 #
+# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
+# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
+# locale returned by "locale -a".
+#
 # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
 #
 # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
@@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
 endif
 ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
 	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
+endif
+ifdef GIT_TEST_UTF8_LOCALE
+	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
 endif
 	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
 ifdef GIT_PERF_REPEAT_COUNT
diff --git a/ci/lib.sh b/ci/lib.sh
index d848c036c5..476c3f369f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -229,6 +229,7 @@ linux-musl)
 	CC=gcc
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
 	;;
 esac
 
diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 547eb3c31a..83efc17661 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -121,12 +121,24 @@ start_svnserve () {
 		 --listen-host 127.0.0.1 &
 }
 
-prepare_a_utf8_locale () {
-	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
-	p
-	q
-}')
-	if test -n "$a_utf8_locale"
+prepare_utf8_locale () {
+	if test -n "$GIT_TEST_UTF8_LOCALE"
+	then
+		: test_set_prereq UTF8
+	elif test -n "${LC_ALL:-$LANG}"
+	then
+		case "${LC_ALL:-$LANG}" in
+		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
+			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
+			;;
+		esac
+	else
+		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
+		p
+		q
+	}')
+	fi
+	if test -n "$GIT_TEST_UTF8_LOCALE"
 	then
 		test_set_prereq UTF8
 	else
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 1d3fdcc997..d5563ec35f 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -4,21 +4,13 @@
 #
 
 test_description='git svn basic tests'
-GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
 
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./lib-git-svn.sh
 
-case "$GIT_SVN_LC_ALL" in
-*.UTF-8)
-	test_set_prereq UTF8
-	;;
-*)
-	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
-	;;
-esac
+prepare_utf8_locale
 
 test_expect_success 'git svn --version works anywhere' '
 	nongit git svn --version
@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
 	test ! -h "$SVN_TREE"/exec-2.sh &&
 	test_cmp help "$SVN_TREE"/exec-2.sh'
 
-name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
-LC_ALL="$GIT_SVN_LC_ALL"
+name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
+LC_ALL="$GIT_TEST_UTF8_LOCALE"
 export LC_ALL
 # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
 test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9b44a44bc1..743fbe1fe4 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 # > ... All of the above characters, except for the backslash, are converted
 # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
 # > "Private use area") when creating or accessing files.
-prepare_a_utf8_locale
+prepare_utf8_locale
 test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
 
 # See the comment on the above test for setting of LC_ALL.
 test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index 2c213ae654..01e1e8a8f7 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -14,12 +14,12 @@ compare_git_head_with () {
 	test_cmp current "$1"
 }
 
-prepare_a_utf8_locale
+prepare_utf8_locale
 
 compare_svn_head_with () {
 	# extract just the log message and strip out committer info.
 	# don't use --limit here since svn 1.1.x doesn't have it,
-	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
+	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
 		use bytes;
 		$/ = ("-"x72) . "\n";
 		my @x = <STDIN>;
-- 
2.32.0.rc3.5.gf3d78db977

Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Torsten Bögershausen]

On Mon, Jun 07, 2021 at 10:01:12AM +0900, Junio C Hamano wrote:
> Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
>
> > Subject: Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn
>
> That certainly is better than "user-specific", but a lot worse than
> "user-specified".  "pre-defined" leaves it open to question "who
> defines it?", and it is a reasonable interpretation that the locale
> may be hardcoded in the makefile, but that is not what this patch
> does.  Saying "user-specified" would not have such a problem.
>
> >> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
> >> that people don't even have to set this extra Makefile knob. But I'm not
> >> sure if we have a good way of testing if that locale works (if we can't
> >> find the "locale" binary).
> >
> > I also think we should fallback to "C.UTF-8" instead of not testing those
> > tests.  However, I don't know if there're any systems that not have "C.UTF-8"
> > locale.
>
> I do share the feeling, but have a hunch that systems lacking
> "locale -a" may be either superset of, or has at least large overlap
> with, those lacking "C.UTF-8", and the new mechanism introduced here
> will cover both of them, so I think it is OK to stop here, at least
> for now.
>
> Thanks.

I did a little bit of digging, the first (?) usage of locale was probably
introduced in the i18n patch to use gettext and msgfmt and stuff:

commit 5e9637c629702e3d41ad01d95956d1835d7338e0
Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date:   Fri Nov 18 00:14:42 2011 +0100

    i18n: add infrastructure for translating Git with gettext

--------------------------

I can't get rid of the feeling, that most systems have an UTF-8
locale enabled by default these days.

What does the following give us under Linux musl ?
set | grep UTF


MacOs (my box here) has this:

LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
LC_CTYPE=UTF-8
XTERM_LOCALE=en_US.UTF-8

If, and only if, Linux musl has a similar setup,
then we may be able to skip the Makefile knob for the moment.
Just skip locale -a, if locale is not available.
This is just a loose idea, I need to install the git-svn bindings and
test if the git-svn tests pass.

Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Đoàn Trần Công Danh]

On 2021-06-07 16:38:16+0200, Torsten Bögershausen <tboegi@web.de> wrote:
> On Mon, Jun 07, 2021 at 10:01:12AM +0900, Junio C Hamano wrote:
> > Đoàn Trần Công Danh  <congdanhqx@gmail.com> writes:
> >
> > > Subject: Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn
> >
> > That certainly is better than "user-specific", but a lot worse than
> > "user-specified".  "pre-defined" leaves it open to question "who
> > defines it?", and it is a reasonable interpretation that the locale
> > may be hardcoded in the makefile, but that is not what this patch
> > does.  Saying "user-specified" would not have such a problem.
> >
> > >> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
> > >> that people don't even have to set this extra Makefile knob. But I'm not
> > >> sure if we have a good way of testing if that locale works (if we can't
> > >> find the "locale" binary).
> > >
> > > I also think we should fallback to "C.UTF-8" instead of not testing those
> > > tests.  However, I don't know if there're any systems that not have "C.UTF-8"
> > > locale.
> >
> > I do share the feeling, but have a hunch that systems lacking
> > "locale -a" may be either superset of, or has at least large overlap
> > with, those lacking "C.UTF-8", and the new mechanism introduced here
> > will cover both of them, so I think it is OK to stop here, at least
> > for now.
> >
> > Thanks.
> 
> I did a little bit of digging, the first (?) usage of locale was probably
> introduced in the i18n patch to use gettext and msgfmt and stuff:
> 
> commit 5e9637c629702e3d41ad01d95956d1835d7338e0
> Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> Date:   Fri Nov 18 00:14:42 2011 +0100
> 
>     i18n: add infrastructure for translating Git with gettext
> 
> --------------------------
> 
> I can't get rid of the feeling, that most systems have an UTF-8
> locale enabled by default these days.
> 
> What does the following give us under Linux musl ?
> set | grep UTF

It's empty.

OK, I lied.  Distributions include a file in /etc/profile.d to enable
LC_ALL/LANG.  Without that file, it's empty.  However, such use-case
is covered by our fallback to LC_ALL and LANG.

Technically, Linux with musl always knows those locales C, POSIX,
C.UTF-8

I worried more about other systems instead.

> MacOs (my box here) has this:
> 
> LANG=en_US.UTF-8
> LC_ALL=en_US.UTF-8
> LC_CTYPE=UTF-8
> XTERM_LOCALE=en_US.UTF-8
> 
> If, and only if, Linux musl has a similar setup,
> then we may be able to skip the Makefile knob for the moment.

Yes, this patch allows to skip the Makefile knob.  Because LC_ALL and
LANG will be the first and second fallback, respectively.  And we can
skip a (potentially expensive) call to "locale -a".

> Just skip locale -a, if locale is not available.
> This is just a loose idea, I need to install the git-svn bindings and
> test if the git-svn tests pass.

-- 
Danh

Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Jeff King]

On Mon, Jun 07, 2021 at 10:01:12AM +0900, Junio C Hamano wrote:

> >> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
> >> that people don't even have to set this extra Makefile knob. But I'm not
> >> sure if we have a good way of testing if that locale works (if we can't
> >> find the "locale" binary).
> >
> > I also think we should fallback to "C.UTF-8" instead of not testing those
> > tests.  However, I don't know if there're any systems that not have "C.UTF-8"
> > locale.
> 
> I do share the feeling, but have a hunch that systems lacking
> "locale -a" may be either superset of, or has at least large overlap
> with, those lacking "C.UTF-8", and the new mechanism introduced here
> will cover both of them, so I think it is OK to stop here, at least
> for now.

Yeah, I think it is OK to stop here, too. I'd worry that we'd make life
unnecessarily complicated for people testing on platforms that lack
"locale" and C.UTF-8, for little gain.

The v4 patch looks good to me.

-Peff

Re: [PATCH v4] t: use user-specified utf-8 locale for testing svn

[Jeff King]

On Mon, Jun 07, 2021 at 08:08:44AM +0700, Đoàn Trần Công Danh wrote:

> In some test-cases, UTF-8 locale is required. To find such locale,
> we're using the first available UTF-8 locale that returned by
> "locale -a".
> 
> However, the locale(1) utility is unavailable on some systems,
> e.g. Linux with musl libc.
> 
> However, without "locale -a", we can't guess provided UTF-8 locale.
> 
> Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for
> linux-musl in our CI system.
> 
> Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale,
> since we no longer prepare the variable named "a_utf8_locale",
> but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
> The fallback will be LC_ALL, LANG environment variable,
> or the first UTF-8 locale from output of "locale -a", in that order.

This patch looks good to me. I have one tiny nit below, but I'm OK if
it's left as-is:

> +prepare_utf8_locale () {
> +	if test -n "$GIT_TEST_UTF8_LOCALE"
> +	then
> +		: test_set_prereq UTF8

This ":" is really just a comment, but since it contains code we would
like to run (but which actually gets run later), it confused me.
Something like:

  : already set by the caller, nothing to do

would have made it easier to understand.

And then the rest of the function is pretty self-explanatory.

-Peff

Re: [PATCH v3] t: use pre-defined utf-8 locale for testing svn

[Đoàn Trần Công Danh]

On 2021-06-08 02:35:30-0400, Jeff King <peff@peff.net> wrote:
> On Mon, Jun 07, 2021 at 10:01:12AM +0900, Junio C Hamano wrote:
> 
> > >> I kind of wonder if trying "C.UTF-8" would be a reasonable fallback so
> > >> that people don't even have to set this extra Makefile knob. But I'm not
> > >> sure if we have a good way of testing if that locale works (if we can't
> > >> find the "locale" binary).
> > >
> > > I also think we should fallback to "C.UTF-8" instead of not testing those
> > > tests.  However, I don't know if there're any systems that not have "C.UTF-8"
> > > locale.
> > 
> > I do share the feeling, but have a hunch that systems lacking
> > "locale -a" may be either superset of, or has at least large overlap
> > with, those lacking "C.UTF-8", and the new mechanism introduced here
> > will cover both of them, so I think it is OK to stop here, at least
> > for now.
> 
> Yeah, I think it is OK to stop here, too. I'd worry that we'd make life
> unnecessarily complicated for people testing on platforms that lack
> "locale" and C.UTF-8, for little gain.

Argh, reading this and my patch again, I figured out a regression with
my patch when either LC_ALL and LANG is set but to something not
ending with utf-8 will ignore "locale -a" completely. I'll send
a reroll later. (With this reroll, we'll ignore the dumb ":", too.

> 
> The v4 patch looks good to me.
> 
> -Peff

-- 
Danh

[PATCH v5] t: use user-specified utf-8 locale for testing svn

[Đoàn Trần Công Danh]

In some test-cases, UTF-8 locale is required. To find such locale,
we're using the first available UTF-8 locale that returned by
"locale -a".

However, the locale(1) utility is unavailable on some systems,
e.g. Linux with musl libc.

However, without "locale -a", we can't guess provided UTF-8 locale.

Add a Makefile knob GIT_TEST_UTF8_LOCALE and activate it for
linux-musl in our CI system.

Rename t/lib-git-svn.sh:prepare_a_utf8_locale to prepare_utf8_locale,
since we no longer prepare the variable named "a_utf8_locale",
but set up a fallback value for GIT_TEST_UTF8_LOCALE instead.
The fallback will be LC_ALL, LANG environment variable,
or the first UTF-8 locale from output of "locale -a", in that order.

Signed-off-by: Đoàn Trần Công Danh <congdanhqx@gmail.com>
---

 New in v5: fix a regression when LC_ALL or LANG set to something that not
 ends with utf-8, e.g. "C", "locale -a" will be ignored completely.

 Thanks to Peff's comment, I didn't notice this regression earlier.

Range-diff against v4:
1:  6036f670d9 ! 1:  6675de6866 t: use user-specified utf-8 locale for testing svn
    @@ t/lib-git-svn.sh: start_svnserve () {
     -}')
     -	if test -n "$a_utf8_locale"
     +prepare_utf8_locale () {
    -+	if test -n "$GIT_TEST_UTF8_LOCALE"
    -+	then
    -+		: test_set_prereq UTF8
    -+	elif test -n "${LC_ALL:-$LANG}"
    ++	if test -z "$GIT_TEST_UTF8_LOCALE"
     +	then
     +		case "${LC_ALL:-$LANG}" in
     +		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
     +			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
     +			;;
    ++		*)
    ++			GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
    ++				p
    ++				q
    ++			}')
    ++			;;
     +		esac
    -+	else
    -+		GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
    -+		p
    -+		q
    -+	}')
     +	fi
     +	if test -n "$GIT_TEST_UTF8_LOCALE"
      	then

 Makefile                                 |  7 +++++++
 ci/lib.sh                                |  1 +
 t/lib-git-svn.sh                         | 22 ++++++++++++++++------
 t/t9100-git-svn-basic.sh                 | 14 +++-----------
 t/t9115-git-svn-dcommit-funky-renames.sh |  6 +++---
 t/t9129-git-svn-i18n-commitencoding.sh   |  4 ++--
 6 files changed, 32 insertions(+), 22 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..502e0c9a81 100644
--- a/Makefile
+++ b/Makefile
@@ -398,6 +398,10 @@ all::
 # with a different indexfile format version.  If it isn't set the index
 # file format used is index-v[23].
 #
+# Define GIT_TEST_UTF8_LOCALE to preferred utf-8 locale for testing.
+# If it isn't set, fallback to $LC_ALL, $LANG or use the first utf-8
+# locale returned by "locale -a".
+#
 # Define HAVE_CLOCK_GETTIME if your platform has clock_gettime.
 #
 # Define HAVE_CLOCK_MONOTONIC if your platform has CLOCK_MONOTONIC.
@@ -2801,6 +2805,9 @@ ifdef GIT_TEST_CMP
 endif
 ifdef GIT_TEST_CMP_USE_COPIED_CONTEXT
 	@echo GIT_TEST_CMP_USE_COPIED_CONTEXT=YesPlease >>$@+
+endif
+ifdef GIT_TEST_UTF8_LOCALE
+	@echo GIT_TEST_UTF8_LOCALE=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_UTF8_LOCALE)))'\' >>$@+
 endif
 	@echo NO_GETTEXT=\''$(subst ','\'',$(subst ','\'',$(NO_GETTEXT)))'\' >>$@+
 ifdef GIT_PERF_REPEAT_COUNT
diff --git a/ci/lib.sh b/ci/lib.sh
index d848c036c5..476c3f369f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -229,6 +229,7 @@ linux-musl)
 	CC=gcc
 	MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python3 USE_LIBPCRE2=Yes"
 	MAKEFLAGS="$MAKEFLAGS NO_REGEX=Yes ICONV_OMITS_BOM=Yes"
+	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
 	;;
 esac
 
diff --git a/t/lib-git-svn.sh b/t/lib-git-svn.sh
index 547eb3c31a..2fde2353fd 100644
--- a/t/lib-git-svn.sh
+++ b/t/lib-git-svn.sh
@@ -121,12 +121,22 @@ start_svnserve () {
 		 --listen-host 127.0.0.1 &
 }
 
-prepare_a_utf8_locale () {
-	a_utf8_locale=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
-	p
-	q
-}')
-	if test -n "$a_utf8_locale"
+prepare_utf8_locale () {
+	if test -z "$GIT_TEST_UTF8_LOCALE"
+	then
+		case "${LC_ALL:-$LANG}" in
+		*.[Uu][Tt][Ff]8 | *.[Uu][Tt][Ff]-8)
+			GIT_TEST_UTF8_LOCALE="${LC_ALL:-$LANG}"
+			;;
+		*)
+			GIT_TEST_UTF8_LOCALE=$(locale -a | sed -n '/\.[uU][tT][fF]-*8$/{
+				p
+				q
+			}')
+			;;
+		esac
+	fi
+	if test -n "$GIT_TEST_UTF8_LOCALE"
 	then
 		test_set_prereq UTF8
 	else
diff --git a/t/t9100-git-svn-basic.sh b/t/t9100-git-svn-basic.sh
index 1d3fdcc997..d5563ec35f 100755
--- a/t/t9100-git-svn-basic.sh
+++ b/t/t9100-git-svn-basic.sh
@@ -4,21 +4,13 @@
 #
 
 test_description='git svn basic tests'
-GIT_SVN_LC_ALL=${LC_ALL:-$LANG}
 
 GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME
 
 . ./lib-git-svn.sh
 
-case "$GIT_SVN_LC_ALL" in
-*.UTF-8)
-	test_set_prereq UTF8
-	;;
-*)
-	say "# UTF-8 locale not set, some tests skipped ($GIT_SVN_LC_ALL)"
-	;;
-esac
+prepare_utf8_locale
 
 test_expect_success 'git svn --version works anywhere' '
 	nongit git svn --version
@@ -187,8 +179,8 @@ test_expect_success POSIXPERM,SYMLINKS "$name" '
 	test ! -h "$SVN_TREE"/exec-2.sh &&
 	test_cmp help "$SVN_TREE"/exec-2.sh'
 
-name="commit with UTF-8 message: locale: $GIT_SVN_LC_ALL"
-LC_ALL="$GIT_SVN_LC_ALL"
+name="commit with UTF-8 message: locale: $GIT_TEST_UTF8_LOCALE"
+LC_ALL="$GIT_TEST_UTF8_LOCALE"
 export LC_ALL
 # This test relies on the previous test, hence requires POSIXPERM,SYMLINKS
 test_expect_success UTF8,POSIXPERM,SYMLINKS "$name" "
diff --git a/t/t9115-git-svn-dcommit-funky-renames.sh b/t/t9115-git-svn-dcommit-funky-renames.sh
index 9b44a44bc1..743fbe1fe4 100755
--- a/t/t9115-git-svn-dcommit-funky-renames.sh
+++ b/t/t9115-git-svn-dcommit-funky-renames.sh
@@ -93,9 +93,9 @@ test_expect_success 'git svn rebase works inside a fresh-cloned repository' '
 # > ... All of the above characters, except for the backslash, are converted
 # > to special UNICODE characters in the range 0xf000 to 0xf0ff (the
 # > "Private use area") when creating or accessing files.
-prepare_a_utf8_locale
+prepare_utf8_locale
 test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new file on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	neq=$(printf "\201\202") &&
 	git config svn.pathnameencoding cp932 &&
@@ -107,7 +107,7 @@ test_expect_success UTF8,!MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 new
 
 # See the comment on the above test for setting of LC_ALL.
 test_expect_success !MINGW,!UTF8_NFD_TO_NFC 'svn.pathnameencoding=cp932 rename on dcommit' '
-	LC_ALL=$a_utf8_locale &&
+	LC_ALL=$GIT_TEST_UTF8_LOCALE &&
 	export LC_ALL &&
 	inf=$(printf "\201\207") &&
 	git config svn.pathnameencoding cp932 &&
diff --git a/t/t9129-git-svn-i18n-commitencoding.sh b/t/t9129-git-svn-i18n-commitencoding.sh
index 2c213ae654..01e1e8a8f7 100755
--- a/t/t9129-git-svn-i18n-commitencoding.sh
+++ b/t/t9129-git-svn-i18n-commitencoding.sh
@@ -14,12 +14,12 @@ compare_git_head_with () {
 	test_cmp current "$1"
 }
 
-prepare_a_utf8_locale
+prepare_utf8_locale
 
 compare_svn_head_with () {
 	# extract just the log message and strip out committer info.
 	# don't use --limit here since svn 1.1.x doesn't have it,
-	LC_ALL="$a_utf8_locale" svn log $(git svn info --url) | perl -w -e '
+	LC_ALL="$GIT_TEST_UTF8_LOCALE" svn log $(git svn info --url) | perl -w -e '
 		use bytes;
 		$/ = ("-"x72) . "\n";
 		my @x = <STDIN>;
-- 
2.32.0.278.gd42b80f139

Re: [PATCH v5] t: use user-specified utf-8 locale for testing svn

[Jeff King]

On Tue, Jun 08, 2021 at 01:56:28PM +0700, Đoàn Trần Công Danh wrote:

>  New in v5: fix a regression when LC_ALL or LANG set to something that not
>  ends with utf-8, e.g. "C", "locale -a" will be ignored completely.
> 
>  Thanks to Peff's comment, I didn't notice this regression earlier.

Doh, I totally missed that, too. I'm glad my comment was helpful in that
respect, at least. :)

Your fix here looks good.

-Peff

Re: [PATCH] t: use user-specific utf-8 locale for testing

[Ævar Arnfjörð Bjarmason]

On Wed, Jun 02 2021, Taylor Blau wrote:

> On Wed, Jun 02, 2021 at 06:46:46PM +0700, Đoàn Trần Công Danh wrote:
>> Despite being required by POSIX, locale(1) is unavailable in some
>> systems, e.g. Linux with musl libc.  Some of those systems support
>> utf-8 locale out of the box.
>
> Hmmph. I would have imagined that locale was available everywhere, but
> unfortunately not.

Small and unsolicited history lesson from a person with funny characters
in their name & language :)

Today it seems like *nix systems have always had UTF-8, but this was a
relatively late development.  

It's Plan9 that had UTF-8 from the start, on *nix systems it was
US-ASCII, and anything else was tacked on top later on.

When I started using *nix systems I belive it was quite common to have
default configurations with only ISO-8859-1 locales installed, and
certainly that's what a lot of or most users who had the need for
locales in European languages not covered by US-ASCII used by default.

This is from hazy memory, but I think it was even actively recommended
against having or using UTF-8 locales on the system. If you
e.g. connected to an IRC channel, or copy/pasted from your text editor
into an E-Mail you could easily send the other end misencodedgibberish.

Later on things like IRC channels in these languages had a "switch day",
it was a complete mess. Nowadays mostly nobody really notices or
remembers anymore these encoding issues since we've mostly got UTF-8
everywhere as a result.

I mean, at least in the case of European languages, I understand
e.g. Japanese and Chinese still have their own persistent encoding
issues related to competing standards.

Even today you can't rely on UTF-8 even on Linux systems, and I think
this has become even more true of late with minimal CI systems or other
chroot-like test environments.