* [PATCH 12/21] sharedkv: lock and explicitly disconnect {dbh}
2021-02-01 8:28 6% [PATCH 00/21] lei2mail worker segfault finally fixed Eric Wong
@ 2021-02-01 8:28 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-02-01 8:28 UTC (permalink / raw)
To: meta
It may be possible for updates or changes to be uncommitted
until disconnect, so we'll use flock() as we do elsewhere
to avoid the polling retry behavior of SQLite.
We also need to clear CachedKids before disconnecting to
to avoid warnings like:
->disconnect invalidates 1 active statement handle
(either destroy statement handles or call finish on
them before disconnecting)
---
lib/PublicInbox/LeiDedupe.pm | 1 +
lib/PublicInbox/SharedKV.pm | 10 +++++++++-
2 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/lib/PublicInbox/LeiDedupe.pm b/lib/PublicInbox/LeiDedupe.pm
index 55488376..5c83fd80 100644
--- a/lib/PublicInbox/LeiDedupe.pm
+++ b/lib/PublicInbox/LeiDedupe.pm
@@ -126,6 +126,7 @@ sub prepare_dedupe {
sub pause_dedupe {
my ($self) = @_;
my $skv = $self->[0];
+ $skv->dbh_release;
delete($skv->{dbh}) if $skv;
}
diff --git a/lib/PublicInbox/SharedKV.pm b/lib/PublicInbox/SharedKV.pm
index f5d09cc1..77df0fb4 100644
--- a/lib/PublicInbox/SharedKV.pm
+++ b/lib/PublicInbox/SharedKV.pm
@@ -143,9 +143,17 @@ SELECT COUNT(k) FROM kv
$sth->fetchrow_array;
}
+sub dbh_release {
+ my ($self, $lock) = @_;
+ my $dbh = delete $self->{dbh} or return;
+ $lock //= $self->lock_for_scope; # may be needed for WAL
+ %{$dbh->{CachedKids}} = (); # cleanup prepare_cached
+ $dbh->disconnect;
+}
+
sub DESTROY {
my ($self) = @_;
- delete $self->{dbh};
+ dbh_release($self);
my $dir = delete $self->{"tmp$$.$self"} or return;
my $tries = 0;
do {
^ permalink raw reply related [relevance 7%]
* [PATCH 00/21] lei2mail worker segfault finally fixed
@ 2021-02-01 8:28 6% Eric Wong
2021-02-01 8:28 7% ` [PATCH 12/21] sharedkv: lock and explicitly disconnect {dbh} Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-02-01 8:28 UTC (permalink / raw)
To: meta
This lei2mail segfault turned out to be an old Perl 5 quirk
which plagued many before me. It was not consistently
reproducible, and random changes seemed to make it happen more
or less frequently. There were several times when I thought I
fixed it (and maybe this is still one of them!) only to have it
pop up again.
Still, I found many other little bugs and improvements worth
doing along the way. Hope things go more smoothly in the
future...
Anyways, [PATCH 18/21] is the fix (and I'll followup with more
on how I found the fix). 19/21 is purely defensive
future-proofing.
Eric Wong (21):
lei: more consistent dedupe and ovv_buf init
ipc: switch wq to use the event loop
lei: remove per-child SIG{__WARN__}
lei: remove SIGPIPE handler
ipc: more helpful ETOOMANYREFS error messages
lei: remove syslog dependency
sharedkv: release {dbh} before rmtree
lei: keep $lei around until workers are reaped
lei_dedupe: use Digest::SHA
lei_xsearch: load PublicInbox::Smsg
lei: deep clone {ovv} for l2m workers
sharedkv: lock and explicitly disconnect {dbh}
lei: increase initial timeout
sharedkv: use lock_for_scope_fast
lei_to_mail: reduce spew on Maildir removal
sharedkv: do not set cache_size by default
import: reap git-config(1) synchronously
ds: guard against stack-not-refcounted quirk of Perl 5
ds: next_tick: avoid $_ in top-level loop iterator
lei: avoid ETOOMANYREFS, cleanup imports
doc: note optional BSD::Resource use
Documentation/public-inbox-config.pod | 2 +-
INSTALL | 6 ++
MANIFEST | 2 +
lib/PublicInbox/DS.pm | 12 ++--
lib/PublicInbox/IPC.pm | 43 +++++++-----
lib/PublicInbox/Import.pm | 1 +
lib/PublicInbox/LEI.pm | 95 +++++++++++++++------------
lib/PublicInbox/LeiDedupe.pm | 6 +-
lib/PublicInbox/LeiExternal.pm | 3 +-
lib/PublicInbox/LeiOverview.pm | 51 +++++++-------
lib/PublicInbox/LeiToMail.pm | 84 +++++++++++------------
lib/PublicInbox/LeiXSearch.pm | 36 +++++-----
lib/PublicInbox/Lock.pm | 17 +++++
lib/PublicInbox/SharedKV.pm | 33 +++++++---
lib/PublicInbox/WQWorker.pm | 34 ++++++++++
script/lei | 28 +++++---
t/lei_to_mail.t | 31 +++++----
xt/stress-sharedkv.t | 50 ++++++++++++++
18 files changed, 342 insertions(+), 192 deletions(-)
create mode 100644 lib/PublicInbox/WQWorker.pm
create mode 100644 xt/stress-sharedkv.t
^ permalink raw reply [relevance 6%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2021-02-01 8:28 6% [PATCH 00/21] lei2mail worker segfault finally fixed Eric Wong
2021-02-01 8:28 7% ` [PATCH 12/21] sharedkv: lock and explicitly disconnect {dbh} Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).