* [PATCH] nntp: fixup "Wide character" warnings
2016-05-13 11:37 7% [PATCH] Revert "nntp: proper UTF-8 support (hopefully?)" Eric Wong
@ 2016-05-13 12:17 5% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2016-05-13 12:17 UTC (permalink / raw)
To: meta
We need Perl to believe everything we send is UTF-8,
make it so, even if it may not be.
Fixes: 265e79ff82ce 'Revert "nntp: proper UTF-8 support (hopefully?)"'
---
lib/PublicInbox/NNTP.pm | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index d177bdc..a632148 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -16,6 +16,8 @@ use Data::Dumper qw(Dumper);
use POSIX qw(strftime);
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
use URI::Escape qw(uri_escape_utf8);
+use Encode qw(find_encoding);
+my $enc_utf8 = find_encoding('UTF-8');
use constant {
r501 => '501 command syntax error',
r221 => '221 Header follows',
@@ -900,6 +902,7 @@ sub cmd_xpath ($$) {
sub res ($$) {
my ($self, $line) = @_;
+ $line = $enc_utf8->encode($line);
do_write($self, $line . "\r\n");
}
@@ -934,6 +937,7 @@ use constant MSG_MORE => ($^O eq 'linux') ? 0x8000 : 0;
sub do_more ($$) {
my ($self, $data) = @_;
+ $data = $enc_utf8->encode($data);
if (MSG_MORE && !$self->{write_buf_size}) {
my $n = send($self->{sock}, $data, MSG_MORE);
if (defined $n) {
^ permalink raw reply related [relevance 5%]
* [PATCH] Revert "nntp: proper UTF-8 support (hopefully?)"
@ 2016-05-13 11:37 7% Eric Wong
2016-05-13 12:17 5% ` [PATCH] nntp: fixup "Wide character" warnings Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2016-05-13 11:37 UTC (permalink / raw)
To: meta
This reverts commit f81ad477cb013d05b9b11fa051a9ebc5983a5be6.
The raw, undecoded body is probably what should be sent over the
wire anyways for clients to deal with. We'll need this to avoid
deprecation warnings with Perl 5.24+ since we use
send()/recv()/sysread().
---
lib/PublicInbox/NNTP.pm | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 3e0faaf..d177bdc 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -11,7 +11,7 @@ use PublicInbox::Search;
use PublicInbox::Msgmap;
use PublicInbox::Git;
use PublicInbox::MID qw(mid2path);
-use Email::MIME;
+use Email::Simple;
use Data::Dumper qw(Dumper);
use POSIX qw(strftime);
use Time::HiRes qw(clock_gettime CLOCK_MONOTONIC);
@@ -114,7 +114,6 @@ sub expire_old () {
sub new ($$$) {
my ($class, $sock, $nntpd) = @_;
my $self = fields::new($class);
- binmode $sock, ':utf8'; # RFC 3977
$self->SUPER::new($sock);
$self->{nntpd} = $nntpd;
res($self, '201 server ready - post via email');
@@ -488,7 +487,7 @@ find_mid:
found:
my $o = 'HEAD:' . mid2path($mid);
my $bytes;
- my $s = eval { Email::MIME->new($ng->gcf->cat_file($o, \$bytes)) };
+ my $s = eval { Email::Simple->new($ng->gcf->cat_file($o, \$bytes)) };
return $err unless $s;
my $lines;
if ($set_headers) {
^ permalink raw reply related [relevance 7%]
* [PATCH 6/6] nntp: proper UTF-8 support (hopefully?)
2015-09-21 11:11 5% [PATCH 0/6] more NNTP server updates Eric Wong
@ 2015-09-21 11:11 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2015-09-21 11:11 UTC (permalink / raw)
To: meta
RFC 3977 stipulates the use of UTF-8 as the default charset,
so we shall try using that and hopefully not mangle things.
---
lib/PublicInbox/NNTP.pm | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/NNTP.pm b/lib/PublicInbox/NNTP.pm
index 2b580d1..91b10f2 100644
--- a/lib/PublicInbox/NNTP.pm
+++ b/lib/PublicInbox/NNTP.pm
@@ -8,7 +8,7 @@ use fields qw(nntpd article rbuf ng long_res);
use PublicInbox::Msgmap;
use PublicInbox::GitCatFile;
use PublicInbox::MID qw(mid2path);
-use Email::Simple;
+use Email::MIME;
use Data::Dumper qw(Dumper);
use POSIX qw(strftime);
use Time::HiRes qw(clock_gettime ualarm CLOCK_MONOTONIC);
@@ -29,6 +29,7 @@ my %DISABLED; # = map { $_ => 1 } qw(xover list_overview_fmt newnews xhdr);
sub new ($$$) {
my ($class, $sock, $nntpd) = @_;
my $self = fields::new($class);
+ binmode $sock, ':utf8'; # RFC 3977
$self->SUPER::new($sock);
$self->{nntpd} = $nntpd;
res($self, '201 server ready - post via email');
@@ -367,7 +368,7 @@ find_mid:
found:
my $o = 'HEAD:' . mid2path($mid);
my $bytes;
- my $s = eval { Email::Simple->new($ng->gcf->cat_file($o, \$bytes)) };
+ my $s = eval { Email::MIME->new($ng->gcf->cat_file($o, \$bytes)) };
return $err unless $s;
if ($set_headers) {
$s->header_set('Newsgroups', $ng->{name});
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/6] more NNTP server updates
@ 2015-09-21 11:11 5% Eric Wong
2015-09-21 11:11 7% ` [PATCH 6/6] nntp: proper UTF-8 support (hopefully?) Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2015-09-21 11:11 UTC (permalink / raw)
To: meta
Always finding new bugs. The XHDR Xref performance improvement
is pretty huge, and the XOVER speedup is noticeable as well.
We will begin to implement RFC 3977, AFAIK the latest NNTP RFC.
Eric Wong (6):
msgmap: minor cleanup to move constant declaration
nntp: speed up XHDR Xref for rtin
nntp: speed up xover slightly
t/nntpd.t: improve test runnability
remove bytes:: for length checks
nntp: proper UTF-8 support (hopefully?)
lib/PublicInbox/Filter.pm | 2 +-
lib/PublicInbox/GitCatFile.pm | 5 +++--
lib/PublicInbox/Msgmap.pm | 2 +-
lib/PublicInbox/NNTP.pm | 47 ++++++++++++++++++++++++++++++++++++-------
public-inbox-nntpd | 4 ++--
t/nntpd.t | 3 ++-
6 files changed, 49 insertions(+), 14 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-09-21 11:11 5% [PATCH 0/6] more NNTP server updates Eric Wong
2015-09-21 11:11 7% ` [PATCH 6/6] nntp: proper UTF-8 support (hopefully?) Eric Wong
2016-05-13 11:37 7% [PATCH] Revert "nntp: proper UTF-8 support (hopefully?)" Eric Wong
2016-05-13 12:17 5% ` [PATCH] nntp: fixup "Wide character" warnings Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).