From: Eric Wong <firstname.lastname@example.org> To: email@example.com Subject: anybody hit SQLite "database is locked" errors? Date: Tue, 25 Aug 2020 00:12:04 +0000 Message-ID: <20200825001204.GA840@dcvr> (raw) Hey all, I've been reindexing frequently ahead of THREADED changes while another process is doing grok-pull and triggering SQLite reads in the post_update_hook. This problem causes an exception in the --reindex process because a read-only SQLite process is holding a SHARED lock. In particular, the expensive (for LKML) `SELECT COUNT(*) FROM msgmap' statement in our current examples/grok-pull.post_update_hook.sh seems to be a culprit. That's even opened with the sqlite3 `-readonly' flag, it still needs to acquire a SHARED lock in SQLite. Other statements run by -httpd/-nntpd/-imapd may still cause this, I think; but I haven't seen it... Maybe others have? WAL (write-ahead-log in SQLite) would be nice; but we can't use it since we expect writers (-mda/-watch) to be a different Unix user than -httpd/-nntpd/-imapd. The Perl binding we use, DBD::SQLite sets it to 30000ms by default, but the busy handler doesn't get fired in these cases. See comments for sqlite3PagerSetBusyHandler in src/pager.c: https://repo.or.cz/sqlite.git/blob_plain/33c269d09c0ec55e61cfe145b75e8cda59dc1d2d:/src/pager.c (SQLite itself doesn't enable by busy handler by default, DBD::SQLite does) I'm thinking of some possible solutions around it, but they're fairly ugly and would make reads more expensive... SQLite itself seems saddled with retaining Windows 95 compatibility; something we obviously won't have to care about.
next reply other threads:[~2020-08-25 0:12 UTC|newest] Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top 2020-08-25 0:12 Eric Wong [this message] 2020-08-25 10:23 ` [PATCH] grok-pull.post_update_hook: flock(2) before SQLite check Eric Wong 2020-08-25 16:29 ` anybody hit SQLite "database is locked" errors? Eric Wong
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style List information: http://public-inbox.org/README * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20200825001204.GA840@dcvr \ --firstname.lastname@example.org \ --email@example.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: link
user/dev discussion of public-inbox itself This inbox may be cloned and mirrored by anyone: git clone --mirror http://public-inbox.org/meta git clone --mirror http://czquwvybam4bgbro.onion/meta git clone --mirror http://hjrcffqmbrq6wope.onion/meta git clone --mirror http://ou63pmih66umazou.onion/meta # If you have public-inbox 1.1+ installed, you may # initialize and index your mirror using the following commands: public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \ firstname.lastname@example.org public-inbox-index meta Example config snippet for mirrors. Newsgroups are available over NNTP: nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta nntp://news.gmane.io/gmane.mail.public-inbox.general note: .onion URLs require Tor: https://www.torproject.org/ code repositories for the project(s) associated with this inbox: https://80x24.org/public-inbox.git AGPL code for this site: git clone https://public-inbox.org/public-inbox.git