git-am fails for emails with UTF8 characters in email header

git-am fails for emails with UTF8 characters in email header

[Siddharth Chandrasekaran]

Hi Junio,

My email server (zoho) puts leading UTF-8 non-breaking spaces ("\xC2\xA0")
when folding long mail headers. Due to this, git-am is failing as it expects
only '\t' or ' ' characters.

RFC2822 [1] on page 7 states:

> The general rule is that wherever this standard allows for folding white
> space (not simply WSP characters), a CRLF may be inserted before any WSP.

It appears the RFC isn't too strict about ASCII only headers (correct me if I'm
wrong here); ergo, if the mail file is in UTF-8, isn't logical to expect UTF-8
whitespace characters also to be allowed in the fold/unfold of headers?

I was able to get git-am to work with modifications to read_one_header_line()
in mailinfo.c need your view on whether a patch is necessary for this. If so, I can
send a patch for this.

Thanks
Sid.

Footnotes:
[1]: https://www.ietf.org/rfc/rfc2822.txt

Re: git-am fails for emails with UTF8 characters in email header

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

On 2019-09-13 at 20:07:24, Siddharth Chandrasekaran wrote:
> Hi Junio,
> 
> My email server (zoho) puts leading UTF-8 non-breaking spaces ("\xC2\xA0")
> when folding long mail headers. Due to this, git-am is failing as it expects
> only '\t' or ' ' characters.
> 
> RFC2822 [1] on page 7 states:
> 
> > The general rule is that wherever this standard allows for folding white
> > space (not simply WSP characters), a CRLF may be inserted before any WSP.
> 
> It appears the RFC isn't too strict about ASCII only headers (correct me if I'm
> wrong here); ergo, if the mail file is in UTF-8, isn't logical to expect UTF-8
> whitespace characters also to be allowed in the fold/unfold of headers?

RFC 5322 (the email specification) doesn't allow non-ASCII characters
anywhere in the header of an email.  RFC 6532, which allows UTF-8 in
header fields if the message is transported via SMTP using the SMTPUTF8
extension, doesn't allow UTF-8 characters to appear in the WSP
production, and therefore doesn't allow them as folding whitespace.
Only space and tab are permitted in WSP.

It appears that your mail server is broken, and it would probably be
best to get it fixed instead of trying to get git-am to work with it.
There are servers which reject messages which are malformed, and even if
your mail gets delivered, there are a bunch of mail clients which will
mishandle it.

> I was able to get git-am to work with modifications to read_one_header_line()
> in mailinfo.c need your view on whether a patch is necessary for this. If so, I can
> send a patch for this.

I don't think such a patch would be a good idea.  I definitely think we
should support SMTPUTF8 and internationalized email, but this particular
issue isn't such a case.
-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 868 bytes --]