RFC: Native clean/smudge filter for UTF-16 files

[Lars Schneider <larsxschneider@gmail.com>]

Hi,

I am working with a team that owns a repository with lots of UTF-16 files.
Converting these files to UTF-8 is no good option as downstream applications
require the UTF-16 encoding. Keeping the files in UTF-16 is no good option
either as Git and Git related tools (e.g. GitHub) consider the files binary
and consequently do not render diffs.

The obvious solution is to setup a clean/smudge filter like this [1]:
    [filter "winutf16"]
        clean = iconv -f utf-16 -t utf-8
        smudge = iconv -f utf-8 -t utf-16

In general this works well but the "per-file" clean/smudge process invocation 
can be slow for many files. I could apply the same trick that we used for Git
LFS and write a iconv that processes all files with a single invocation (see
"edcc85814c convert: add filter.<driver>.process option" [2]). 

Alternatively, I could add a native attribute to Git that translates UTF-16 
to UTF-8 and back. A conversion function is already available in "mingw.h" [3]
on Windows. Limiting this feature to Windows wouldn't be a problem from my
point of view as UTF-16 is only relevant on Windows anyways. The attribute 
could look like this:

    *.txt        text encoding=utf-16

There was a previous discussion on the topic and Jonathan already suggested
a "native" clean/smudge filter in 2010 [4]. Also the "encoding" attribute
is already present but, as far as I can tell, is only used by the git gui
for viewing [5].

Do you think a patch that converts UTF-16 files to UTF-8 via an attribute
"encoding=utf-16" on Windows would have a chance to get accepted?

Thanks,
Lars

[1] https://github.com/msysgit/msysgit/issues/113#issue-13142846
[2] https://github.com/git/git/commit/edcc85814c87ebd7f3b1b7d3979fac3dfb84d308
[3] https://github.com/git/git/blob/14c63a9dc093d6738454f6369a4f5663ca732cf7/compat/mingw.h#L501-L533
[4] https://public-inbox.org/git/20101022195331.GA12014@burratino/
[5] https://github.com/git/git/commit/1ffca60f0b0395e1e593e64d66e7ed3c47d8517e