From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=ALL_TRUSTED,AWL,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id 9837B1F66E; Tue, 25 Aug 2020 00:12:04 +0000 (UTC) Date: Tue, 25 Aug 2020 00:12:04 +0000 From: Eric Wong To: meta@public-inbox.org Subject: anybody hit SQLite "database is locked" errors? Message-ID: <20200825001204.GA840@dcvr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline List-Id: Hey all, I've been reindexing frequently ahead of THREADED changes while another process is doing grok-pull and triggering SQLite reads in the post_update_hook. This problem causes an exception in the --reindex process because a read-only SQLite process is holding a SHARED lock. In particular, the expensive (for LKML) `SELECT COUNT(*) FROM msgmap' statement in our current examples/grok-pull.post_update_hook.sh seems to be a culprit. That's even opened with the sqlite3 `-readonly' flag, it still needs to acquire a SHARED lock in SQLite. Other statements run by -httpd/-nntpd/-imapd may still cause this, I think; but I haven't seen it... Maybe others have? WAL (write-ahead-log in SQLite) would be nice; but we can't use it since we expect writers (-mda/-watch) to be a different Unix user than -httpd/-nntpd/-imapd. The Perl binding we use, DBD::SQLite sets it to 30000ms by default, but the busy handler doesn't get fired in these cases. See comments for sqlite3PagerSetBusyHandler in src/pager.c: https://repo.or.cz/sqlite.git/blob_plain/33c269d09c0ec55e61cfe145b75e8cda59dc1d2d:/src/pager.c (SQLite itself doesn't enable by busy handler by default, DBD::SQLite does) I'm thinking of some possible solutions around it, but they're fairly ugly and would make reads more expensive... SQLite itself seems saddled with retaining Windows 95 compatibility; something we obviously won't have to care about.