Re: Warnings from git fsck after lkml import

[ebiederm@xmission.com (Eric W. Biederman)]

Konstantin Ryabitsev <konstantin@linuxfoundation.org> writes:

> On Thu, Jul 05, 2018 at 11:13:46PM +0000, Eric Wong wrote:
>>"Eric W. Biederman" <ebiederm@xmission.com> wrote:
>>> It looks like public-inbox has some challenges when importing some
>>> questionable emails.  The import of lkml has resulted in several commits
>>> with bad dates that git fsck complains about.  I have previously
>>> reported this to Konstantin Ryabitsev who maintains kernel.org but since
>>> I have not seen any discussion of this I thought I should report it
>>> directly here as well.
>>
>>Thanks for bringing this up publically.
>>
>>Yes, I early during v2 development I noticed old mails had some
>>-1400 timezone values (but the furthest is -1200).  I opted to
>>attempt to preserve the wonky timezones since fast-import
>>happily accepts -1400 and I didn't anticipate problems...
>
> So, I can fix those in the archives, but this obviously requires
> rebasing the whole repo, and I'm not sure what kind of impact that
> would have. I'm assuming it's not sufficient to just fix the git repo,
> as all commit IDs after the modified commit are going to be different
> -- so additional changes to sqlite and xapian dbs would be required?

Unless I am mistaken the cheap/clever version is to
- Rebuild the 3 .git trees.
- Notice that the object id's (aka sha1 hashes) of the emails remains
  the same.
- Use sqlite3 to update the meta table of msgmap.sqlite3

My currently msgmap.sqlite3 contains:
CREATE TABLE meta (key VARCHAR(32) PRIMARY KEY, val VARCHAR(255) NOT NULL);
/* No STAT tables available */
sqlite> select * from meta;
created_at|1530525399
last_xap15-6|c8f95c6728579303c200adbfb5469215da7e7836
last_xap15-5|31ed379430c456f90bdd172b223020c0e6d7cb8d
last_xap15-4|88294f6d487193f5984791ee81213a25130d0559
last_xap15-3|93d9eace2721494d8457c7f5f6de803c0d648172
last_xap15-2|d48078ceeec1f51313253a56ed3ba0eae7fde909
last_xap15-1|6b67b9f5e0cd82d3c734e6cdc44c1f722ab6fb6a
last_xap15-0|b67bf7f62c8125d67461cc6e7d1736ddc8844a18

Which matches the HEAD commits the lkml git trees.
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/0.git show --pretty=oneline | head  -1
b67bf7f62c8125d67461cc6e7d1736ddc8844a18 [-mm patch] drivers/firewire/: cleanups
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/1.git show --pretty=oneline | head  -1
6b67b9f5e0cd82d3c734e6cdc44c1f722ab6fb6a Re: [git patches] libata updates for 2.6.34
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/2.git show --pretty=oneline | head  -1
d48078ceeec1f51313253a56ed3ba0eae7fde909 Re: linux-next: Tree for Jan 10 (staging/sb105x)
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/3.git show --pretty=oneline | head  -1
93d9eace2721494d8457c7f5f6de803c0d648172 Re: randconfig bug: ARM/KVM link error in hyp_idmap section
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/4.git show --pretty=oneline | head  -1
88294f6d487193f5984791ee81213a25130d0559 Re: [PATCH 2/2] sdhci-of-arasan: Set controller to test mode when fails-without-test-cd is present
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/5.git show --pretty=oneline | head  -1
31ed379430c456f90bdd172b223020c0e6d7cb8d Re: [PATCH 0/2] of: change overlay apply input data from EDT to FDT
eric@x220:~/public-inbox/vger.kernel.org/linux-kernel-good$ git --git-dir=git/6.git show --pretty=oneline | head  -1
c8f95c6728579303c200adbfb5469215da7e7836 [PATCH] slimbus: stream: Fix htmldocs warnings

However all you have to do is ensure you preserve msgmap.sqlite3 and
public-inbox-index is capable of rebuilding everything else.


Eric