user/dev discussion of public-inbox itself
 help / color / mirror / Atom feed
* thoughts improving duplicate message handling...
@ 2020-12-28 21:41 Eric Wong
  2020-12-29 15:50 ` Konstantin Ryabitsev
  0 siblings, 1 reply; 2+ messages in thread
From: Eric Wong @ 2020-12-28 21:41 UTC (permalink / raw)
  To: meta

Spinning off of https://public-inbox.org/meta/20201228213139.GA17600@dcvr/

> (*) The downside to this approach is IMAP UIDs (NNTP article
> numbers) get changed, but I think I can workaround that.  The
> workaround I'm thinking of involves capturing exact blob OIDs
> during the unindex phase to create an OID => UID mapping.
> reindex would reuse the OID => UID mapping to keep the same
> IMAP UID.  It could be loosened to use ContentHash, or
> whatever combination of Message-ID/From/Date/etc, too.

I've noticed quite a few messages with identical Message-IDs,
especially with /all/ (*)

a) There are occasionally resent revisions of patches with the
   same Message-ID

b) More often, a cross-posted message has different trailers
   depending on which list it was posted to.  (And the
   PublicInbox::Filter::* API doesn't apply at indexing,
   only at git injection time)

I'm thinking of two new features:

1) the WWW and lei UIs could have a "diff mode"
   that clearly shows differences between rendered texts
   This could be expensive with malicious inputs, but the
   existing "limiter" infrastructure used for serving packs
   should help, here.

2) support PublicInbox::Filter at index-time
   (requires reindexing to take effect, will cause new IMAP
   UIDs+NNTP article numbers to show up, possibly en-mass)

(*) https://public-inbox.org/meta/20201126194543.GA30337@dcvr/

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: thoughts improving duplicate message handling...
  2020-12-28 21:41 thoughts improving duplicate message handling Eric Wong
@ 2020-12-29 15:50 ` Konstantin Ryabitsev
  0 siblings, 0 replies; 2+ messages in thread
From: Konstantin Ryabitsev @ 2020-12-29 15:50 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

On Mon, Dec 28, 2020 at 09:41:46PM +0000, Eric Wong wrote:
> a) There are occasionally resent revisions of patches with the
>    same Message-ID

This is broken and is unwanted. :)

> b) More often, a cross-posted message has different trailers
>    depending on which list it was posted to.  (And the
>    PublicInbox::Filter::* API doesn't apply at indexing,
>    only at git injection time)

In this case, my preferred solution is "first in wins", with ability to affect
which one shows up first, using a weighting metric such as "comes first in the
config file" or "has a higher priority setting."

-K

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2020-12-29 15:50 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-28 21:41 thoughts improving duplicate message handling Eric Wong
2020-12-29 15:50 ` Konstantin Ryabitsev

user/dev discussion of public-inbox itself

This inbox may be cloned and mirrored by anyone:

	git clone --mirror http://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 meta meta/ http://public-inbox.org/meta \
		meta@public-inbox.org
	public-inbox-index meta

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.io/gmane.mail.public-inbox.general
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/public-inbox.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git