user/dev discussion of public-inbox itself
 help / color / mirror / code / Atom feed
From: Eric Wong <>
Subject: xref3 + NNTP problems...
Date: Mon, 30 Nov 2020 19:42:01 +0000	[thread overview]
Message-ID: <20201130194201.GA6687@dcvr> (raw)
In-Reply-To: <>

Eric Wong <> wrote:
> Subject: Re: [PATCH 12/12] nntp: xref: use ->ALL extindex if available
> Getting Xref for cross-posted messages is an O(n) operation
> where `n' is the number of newsgroups on the server.  This works
> acceptably when there are dozens of groups, but would be
> unnacceptable when there's tens of thousands of newsgroups.
> With ~140 newsgroups, a mirror already handles
> "XHDR Xref $MESSAGE_ID" requests around 30% faster after
> creating the xref3.idx_nntp index.

xref3-based on ContentHash is great for internal use to ensure
we don't archive redundant messages; but don't get fooled by
reused Message-IDs, either (because users can and do reuse
Message-IDs for completely different messages)

But there's also more cases where content-matching for messages
we don't match because of mailing list-added trailers
(e.g. unsubscribe and other list info info)

So every message posted to
fails to match cross posts to Xref:, which is bad, too...

I have an idea to make it less bad, maybe it'll work...

  reply	other threads:[~2020-11-30 19:42 UTC|newest]

Thread overview: 15+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-27  9:52 [PATCH 00/12] some NNTP-related fixes + speedups Eric Wong
2020-11-27  9:52 ` [PATCH 01/12] nntp: use Inbox->uidvalidity instead of ->mm->created_at Eric Wong
2020-11-27  9:52 ` [PATCH 02/12] nntpd: share {groups} hash with {-by_newsgroup} in Config Eric Wong
2020-11-27  9:52 ` [PATCH 03/12] mm: min/max: return 0 instead of undef Eric Wong
2020-11-27  9:52 ` [PATCH 04/12] nntp: use grep operation for wildmat matching Eric Wong
2020-11-27  9:52 ` [PATCH 05/12] nntp: NEWNEWS: speed up filtering Eric Wong
2020-11-27  9:52 ` [PATCH 06/12] miscsearch: implement ->newsgroup_matches Eric Wong
2020-11-27  9:52 ` [PATCH 07/12] nntp: LIST ACTIVE.TIMES use angle brackets around address Eric Wong
2020-11-27  9:52 ` [PATCH 08/12] nntp: move LIST iterators to long_response Eric Wong
2020-11-27  9:52 ` [PATCH 09/12] t/extsearch: show a more realistic case Eric Wong
2020-11-27  9:52 ` [PATCH 10/12] nntp: some minor golfing Eric Wong
2020-11-27  9:52 ` [PATCH 11/12] nntp: xref: simplify sub signature Eric Wong
2020-11-27  9:52 ` [PATCH 12/12] nntp: xref: use ->ALL extindex if available Eric Wong
2020-11-30 19:42   ` Eric Wong [this message]
2020-11-30 23:37     ` [PATCH] nntp: make ->ALL Xref generation more fuzzy Eric Wong

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

  List information:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201130194201.GA6687@dcvr \ \ \
    --subject='Re: xref3 + NNTP problems...' \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ \
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
 note: .onion URLs require Tor:

code repositories for project(s) associated with this inbox:

AGPL code for this site: git clone