Re: [PATCH] disallow NUL characters in Message-ID and List-Id

["Eric W. Biederman" <ebiederm@xmission.com>]

Eric Wong <e@80x24.org> writes:

> While MTAs seem to stop '\0' from appearing in headers, users
> fetching archives via git remain susceptible to having '\0' land
> in archives.  So we'll filter them out of Xapian and SQLite DBs
> to avoid interopability problems with CLI tools since there's no
> known messages in lore or any of my archives which feature them.
>
> Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
> can be specified from the command-line (although some characters
> will still require $(printf) contortions).
>
> As with Message-ID, List-Id fields with /\n\t\r/ characters will
> also be stripped for indexing.  I will assume whatever went wrong
> with the References: header in
> <https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
> could also happen to the List-Id header.
>
> This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
> (Import: Don't copy nulls from emails into git, 2018-07-07)

That seems reasonable to me.

Eric


> ---
>  lib/PublicInbox/MID.pm       | 2 +-
>  lib/PublicInbox/SearchIdx.pm | 1 +
>  2 files changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
> index 97cf3a54..36c05855 100644
> --- a/lib/PublicInbox/MID.pm
> +++ b/lib/PublicInbox/MID.pm
> @@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
>  	my @ret;
>  	$seen ||= {};
>  	foreach my $mid (@$mids) {
> -		$mid =~ tr/\n\t\r//d;
> +		$mid =~ tr/\n\t\r\0//d;
>  		if (length($mid) > MAX_MID_SIZE) {
>  			warn "Message-ID: <$mid> too long, truncating\n";
>  			$mid = substr($mid, 0, MAX_MID_SIZE);
> diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
> index 32598b7c..f569428c 100644
> --- a/lib/PublicInbox/SearchIdx.pm
> +++ b/lib/PublicInbox/SearchIdx.pm
> @@ -414,6 +414,7 @@ sub index_list_id ($$$) {
>  	for my $l ($hdr->header_raw('List-Id')) {
>  		$l =~ /<([^>]+)>/ or next;
>  		my $lid = lc $1;
> +		$lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
>  		$doc->add_boolean_term('G' . $lid);
>  		index_phrase($self, $lid, 1, 'XL'); # probabilistic
>  	}