user/dev discussion of public-inbox itself
 help / color / Atom feed
* [PATCH 00/24] fix IDN linkification, add paranoia
@ 2019-06-04 11:27 Eric Wong
  2019-06-04 11:27 ` [PATCH 01/24] linkify: support Internationalized Domain Names in URLs Eric Wong
                   ` (24 more replies)
  0 siblings, 25 replies; 26+ messages in thread
From: Eric Wong @ 2019-06-04 11:27 UTC (permalink / raw)
  To: meta

Looks like I wasn't up-to-date with the Perl 5.6 Unicode changes
from the turn-of-the century :x   So, regexp character classes
like \w, \s, \d, etc are all locale-aware and could match
characters I didn't expect.

AFAIK, the worst thing from this screwup was a message with an
IDN (internationalizd domain name) in a URL causing a message to
fail rendering as HTML during linkification.

Now, creating attachment and git blob download links with
non-ASCII characters could cause some confusion; and lead to
difficulty removing said files for people who can't type or
tab-complete their way to non-ASCII characters, or lack the
proper fonts to display the character.

The handy "/a" regexp modifier is off-limits for now, since
that's only in Perl 5.14+ and we are just barely taking the bold
step of requiring Perl 5.10.1+ a decade after its release.

Most of the other changes are probably paranoia at best, and/or
optimizations intended to stop bad inputs earlier rather than
later.  Deeper parts of the stack which actually interpret
strings of digits as integers will treat non-ASCII digits as
zero.

Anyways, internationalized domain names and email addresses are
a real thing whether or not it's a security and maintenance
nightmare.  And it looks like some of the code can already
support them (or can be easily tweaked to support them, like
Linkify.pm for IDN).

There's probably more things along these lines which can be
done.  Right now, I'm treating git-fast-import and
"git-log --raw" output be trusted in that they won't surprise
us with Unicode digits and such...

But yeah, probably more auditing and eyes for stuff like this
would be helpful...

Eric Wong (24):
  linkify: support Internationalized Domain Names in URLs
  nntp: be explicit about ASCII digit matches
  nntp: ensure we only handle ASCII whitespace
  mid: id_compress requires ASCII-clean words
  feed: only accept ASCII digits for ref~$N
  http: require SERVER_PORT to be ASCII digit
  wwwlisting: require ASCII digit for port number
  wwwattach: only pass the charset through if ASCII
  www: only emit ASCII chars in attachment filenames
  www: require ASCII filenames in git blob downloads
  config: do not accept non-ASCII digits in cgitrc params
  newswww: only accept ASCII digits as article numbers
  view: require YYYYmmDD(HHMMSS) timestamps to be ASCII
  githttpbackend: require Range:, Status: to be ASCII digits
  searchview: do not allow non-ASCII offsets and limits
  msgtime: require ASCII digits for parsing dates
  filter/rubylang: require ASCII digit for mailcount
  inbox: require ASCII digits for feedmax var
  solver|viewdiff: restrict digit matches to ASCII
  www: require ASCII digit for git epoch
  require ASCII digits for local FS items
  githttpbackend: require ASCII in path
  www: require ASCII range for mbox downloads
  www: require ASCII word characters for CSS filenames

 lib/PublicInbox/Config.pm          |  2 +-
 lib/PublicInbox/Feed.pm            |  2 +-
 lib/PublicInbox/Filter/RubyLang.pm |  2 +-
 lib/PublicInbox/GitHTTPBackend.pm  |  8 ++++----
 lib/PublicInbox/HTTP.pm            |  2 +-
 lib/PublicInbox/Hval.pm            |  3 +++
 lib/PublicInbox/Inbox.pm           |  6 +++---
 lib/PublicInbox/Linkify.pm         |  5 +++--
 lib/PublicInbox/MID.pm             |  4 ++--
 lib/PublicInbox/MsgTime.pm         |  7 ++++---
 lib/PublicInbox/NNTP.pm            | 16 ++++++++--------
 lib/PublicInbox/NewsWWW.pm         |  2 +-
 lib/PublicInbox/Search.pm          |  2 +-
 lib/PublicInbox/SearchView.pm      |  4 ++--
 lib/PublicInbox/SolverGit.pm       |  2 +-
 lib/PublicInbox/V2Writable.pm      |  4 ++--
 lib/PublicInbox/View.pm            |  6 +++---
 lib/PublicInbox/ViewDiff.pm        |  4 ++--
 lib/PublicInbox/WWW.pm             | 20 +++++++++++++-------
 lib/PublicInbox/WwwAttach.pm       |  2 +-
 lib/PublicInbox/WwwListing.pm      |  4 ++--
 lib/PublicInbox/Xapcmd.pm          |  6 +++---
 script/public-inbox-purge          |  2 +-
 t/linkify.t                        | 12 ++++++++++++
 24 files changed, 75 insertions(+), 52 deletions(-)

-- 
EW


^ permalink raw reply	[flat|nested] 26+ messages in thread

end of thread, back to index

Thread overview: 26+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-06-04 11:27 [PATCH 00/24] fix IDN linkification, add paranoia Eric Wong
2019-06-04 11:27 ` [PATCH 01/24] linkify: support Internationalized Domain Names in URLs Eric Wong
2019-06-04 11:27 ` [PATCH 02/24] nntp: be explicit about ASCII digit matches Eric Wong
2019-06-04 11:27 ` [PATCH 03/24] nntp: ensure we only handle ASCII whitespace Eric Wong
2019-06-04 11:27 ` [PATCH 04/24] mid: id_compress requires ASCII-clean words Eric Wong
2019-06-04 11:27 ` [PATCH 05/24] feed: only accept ASCII digits for ref~$N Eric Wong
2019-06-04 11:27 ` [PATCH 06/24] http: require SERVER_PORT to be ASCII digit Eric Wong
2019-06-04 11:27 ` [PATCH 07/24] wwwlisting: require ASCII digit for port number Eric Wong
2019-06-04 11:27 ` [PATCH 08/24] wwwattach: only pass the charset through if ASCII Eric Wong
2019-06-04 11:27 ` [PATCH 09/24] www: only emit ASCII chars in attachment filenames Eric Wong
2019-06-04 11:27 ` [PATCH 10/24] www: require ASCII filenames in git blob downloads Eric Wong
2019-06-04 11:27 ` [PATCH 11/24] config: do not accept non-ASCII digits in cgitrc params Eric Wong
2019-06-04 11:27 ` [PATCH 12/24] newswww: only accept ASCII digits as article numbers Eric Wong
2019-06-04 11:27 ` [PATCH 13/24] view: require YYYYmmDD(HHMMSS) timestamps to be ASCII Eric Wong
2019-06-04 11:27 ` [PATCH 14/24] githttpbackend: require Range:, Status: to be ASCII digits Eric Wong
2019-06-04 11:27 ` [PATCH 15/24] searchview: do not allow non-ASCII offsets and limits Eric Wong
2019-06-04 11:27 ` [PATCH 16/24] msgtime: require ASCII digits for parsing dates Eric Wong
2019-06-04 11:27 ` [PATCH 17/24] filter/rubylang: require ASCII digit for mailcount Eric Wong
2019-06-04 11:27 ` [PATCH 18/24] inbox: require ASCII digits for feedmax var Eric Wong
2019-06-04 11:27 ` [PATCH 19/24] solver|viewdiff: restrict digit matches to ASCII Eric Wong
2019-06-04 11:27 ` [PATCH 20/24] www: require ASCII digit for git epoch Eric Wong
2019-06-04 11:27 ` [PATCH 21/24] require ASCII digits for local FS items Eric Wong
2019-06-04 11:27 ` [PATCH 22/24] githttpbackend: require ASCII in path Eric Wong
2019-06-04 11:27 ` [PATCH 23/24] www: require ASCII range for mbox downloads Eric Wong
2019-06-04 11:27 ` [PATCH 24/24] www: require ASCII word characters for CSS filenames Eric Wong
2019-06-05  2:18 ` [PATCH 25/24] tighten up digit matches to ASCII for git output Eric Wong

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Example config snippet for mirrors

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox