* Re: [PATCH] overidx: fix compatibility with current versions
2020-07-26 19:43 6% [PATCH] overidx: fix compatibility with current versions Eric Wong
@ 2020-07-26 19:51 0% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-07-26 19:51 UTC (permalink / raw)
To: meta
Eric Wong <e@yhbt.net> wrote:
> We still need to use SQL_BLOB to ensure existing versions
> of public-inbox can read over.sqlite3.
... because they're still using {sqlite_unicode}
Will squash that in before pushing.
> This partially
> reverts commit e9fc1290ead44e06d20ff58e0a6acb5306d4fbe2.
>
> Fixes: e9fc1290ead44e06 ("over: unset sqlite_unicode attribute")
^ permalink raw reply [relevance 0%]
* [PATCH] overidx: fix compatibility with current versions
@ 2020-07-26 19:43 6% Eric Wong
2020-07-26 19:51 0% ` Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-07-26 19:43 UTC (permalink / raw)
To: meta
We still need to use SQL_BLOB to ensure existing versions
of public-inbox can read over.sqlite3. This partially
reverts commit e9fc1290ead44e06d20ff58e0a6acb5306d4fbe2.
Fixes: e9fc1290ead44e06 ("over: unset sqlite_unicode attribute")
---
lib/PublicInbox/OverIdx.pm | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index fcb450794..5821c562b 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -12,7 +12,7 @@ use strict;
use warnings;
use base qw(PublicInbox::Over);
use IO::Handle;
-use DBI;
+use DBI qw(:sql_types); # SQL_BLOB
use PublicInbox::MID qw/id_compress mids_for_index references/;
use PublicInbox::Smsg qw(subject_normalized);
use Compress::Zlib qw(compress);
@@ -337,7 +337,7 @@ VALUES (?,?,?,?,?,?)
my $n = 0;
my @v = ($num, $tid, $sid, $ts, $ds);
foreach (@v) { $sth->bind_param(++$n, $_) }
- $sth->bind_param(++$n, $ddd);
+ $sth->bind_param(++$n, $ddd, SQL_BLOB);
$sth->execute;
$sth = $dbh->prepare_cached(<<'');
INSERT INTO id2num (id, num) VALUES (?,?)
^ permalink raw reply related [relevance 6%]
* [PATCH 1/3] over: unset sqlite_unicode attribute
2020-07-14 2:14 6% [PATCH 0/3] avoid msgmap reopens in long-lived processes Eric Wong
@ 2020-07-14 2:14 7% ` Eric Wong
0 siblings, 0 replies; 4+ results
From: Eric Wong @ 2020-07-14 2:14 UTC (permalink / raw)
To: meta
None of the human-readable strings stored in over.sqlite3
require UTF-8. Message-IDs do not, nor do the compressed
Subject IDs (sid) we use for Subject-based threading. And the
`ddd' (doc-data-deflated) column is of course binary data.
This frees us of having to use SQL_BLOB for the `ddd', column,
and will open the door for us to use dbh_new for Msgmap, too.
---
lib/PublicInbox/Over.pm | 1 -
lib/PublicInbox/OverIdx.pm | 4 ++--
2 files changed, 2 insertions(+), 3 deletions(-)
diff --git a/lib/PublicInbox/Over.pm b/lib/PublicInbox/Over.pm
index e5a980d5..5d285057 100644
--- a/lib/PublicInbox/Over.pm
+++ b/lib/PublicInbox/Over.pm
@@ -36,7 +36,6 @@ sub dbh_new {
$st = pack('dd', $st[0], $st[1]);
} while ($st ne $self->{st} && $tries++ < 3);
warn "W: $f: .st_dev, .st_ino unstable\n" if $st ne $self->{st};
- $dbh->{sqlite_unicode} = 1;
$dbh;
}
diff --git a/lib/PublicInbox/OverIdx.pm b/lib/PublicInbox/OverIdx.pm
index 008a5d1a..13aa2d74 100644
--- a/lib/PublicInbox/OverIdx.pm
+++ b/lib/PublicInbox/OverIdx.pm
@@ -12,7 +12,7 @@ use strict;
use warnings;
use base qw(PublicInbox::Over);
use IO::Handle;
-use DBI qw(:sql_types); # SQL_BLOB
+use DBI;
use PublicInbox::MID qw/id_compress mids_for_index references/;
use PublicInbox::Smsg qw(subject_normalized);
use Compress::Zlib qw(compress);
@@ -309,7 +309,7 @@ VALUES (?,?,?,?,?,?)
my $n = 0;
my @v = ($num, $tid, $sid, $ts, $ds);
foreach (@v) { $sth->bind_param(++$n, $_) }
- $sth->bind_param(++$n, $ddd, SQL_BLOB);
+ $sth->bind_param(++$n, $ddd);
$sth->execute;
$sth = $dbh->prepare_cached(<<'');
INSERT INTO id2num (id, num) VALUES (?,?)
^ permalink raw reply related [relevance 7%]
* [PATCH 0/3] avoid msgmap reopens in long-lived processes
@ 2020-07-14 2:14 6% Eric Wong
2020-07-14 2:14 7% ` [PATCH 1/3] over: unset sqlite_unicode attribute Eric Wong
0 siblings, 1 reply; 4+ results
From: Eric Wong @ 2020-07-14 2:14 UTC (permalink / raw)
To: meta
As with commit 2a717d13f10fcdc69921d80cf94c47a694a175d4
("nntpd+imapd: detect replaced over.sqlite3"), this is
another step towards eliminating needless wakeups on
systems with inotify or kqueue.
To save memory, we'll also stop storing {filename} in Perl once
the SQLite DB is open, since we expect to have thousands of
inboxes soon.
Eric Wong (3):
over: unset sqlite_unicode attribute
nntpd+imapd: detect unlinked msgmap
over+msgmap: do not store filename after DBI->connect
lib/PublicInbox/Inbox.pm | 11 +++----
lib/PublicInbox/Msgmap.pm | 67 ++++++++++++++++++++------------------
lib/PublicInbox/Over.pm | 31 +++++++++++++-----
lib/PublicInbox/OverIdx.pm | 6 ++--
t/nntpd.t | 8 +++++
5 files changed, 74 insertions(+), 49 deletions(-)
^ permalink raw reply [relevance 6%]
Results 1-4 of 4 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-07-14 2:14 6% [PATCH 0/3] avoid msgmap reopens in long-lived processes Eric Wong
2020-07-14 2:14 7% ` [PATCH 1/3] over: unset sqlite_unicode attribute Eric Wong
2020-07-26 19:43 6% [PATCH] overidx: fix compatibility with current versions Eric Wong
2020-07-26 19:51 0% ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).