Re: Bug#808610: libxapian22: xapian database corruption causes recollindex loop

[Eric Wong <e@80x24.org>]

Olly Betts <olly@survex.com> wrote:
> On Mon, Dec 28, 2015 at 02:45:43AM +0000, Olly Betts wrote:
> > There haven't been many backend changes since 1.2.19 - my guess is it's
> > this one:
> > 
> > http://trac.xapian.org/changeset/826d1a19cc356e7bf66c1681626e70af32967447/git
> > 
> > Can you test 1.2.19 plus that patch to see if this solves your problem?
> > 
> > (If that is the fix, then testing and unstable aren't affected).
> 
> Still awaiting your response on this.

I encountered a problem with Jessie, too, and this might be it.
I'm testing the above change on the public-inbox.git tree
(instructions below).  Will update in 24-48 hours...

Meanwhile, in case anybody isn't familiar with patching + building
Debian packages.... (and most of my Debian knowledge is 15 years old,
so maybe I'm not up-to-date with latest practices).

Download the unified diff from the above Trac URL, it should
save to a file like:

changeset_826d1a19cc356e7bf66c1681626e70af32967447.diff

Make sure deb-src lines for jessie are available in
/etc/apt/sources.list (or a file in /etc/apt/sources.list.d/)

	deb-src http://http.us.debian.org/debian/ jessie-updates main
	deb-src http://http.us.debian.org/debian/ jessie main
	deb-src http://http.us.debian.org/debian/ jessie-backports main

# install build prereqs (need root):
sudo apt-get update # make sure deb-src sources are known
sudo apt-get install build-essential fakeroot
sudo apt-get build-dep xapian-core

# grab the source (as a normal user)
apt-get source xapian-core=1.2.19-1

# enter the unpatched source directory
cd xapian-core-1.2.19

# apply the patch you downloaded above
# you'll see some warnings about fuzz which seem harmless for this:
patch -p1 </path/to/changeset_826d1a19cc356e7bf66c1681626e70af32967447.diff

# Still in the xapian-core-1.2.19 dir, run the build and wait a while:
debian/rules build

# build the Debian binary packages (.deb files)
fakeroot debian/rules binary

# At the end, you should see messages about .deb files being
# built in the parent directory, install the libxapian22 deb:
sudo dpkg -i ../libxapian22_1.2.19-1_amd64.deb


I'm testing this against t/search.t in the public-inbox.git sources

# fetch prereqs and sources:
sudo apt-get install libsearch-xapian-perl libemail-mime-perl
git clone git://80x24.org/public-inbox

# enter the source tree
cd public-inbox

# reset to the HEAD as of this writing
git reset --hard 9bae4c858d983d0f0cf7314b50f613818a58d5c5

# Audit t/search.t and any files it uses/requires (in lib)
# to ensure it won't eat your hard drive or send me all your
# personal info :)

# test in a loop:
while perl -I lib t/search.t; do :; done
# let the loop run for a few hours/days...

Even without the patch, it can take thousands of tries before a
failure manifests along the lines of:

  Exception: find_entry failed to find any entry at all!

Or something with "basic_string" in it...

I never had any problems under Debian wheezy.