* Re: [PATCH] v2writable: exact discontiguous history handling
2021-01-05 1:29 14% ` [PATCH] v2writable: exact discontiguous history handling Eric Wong
@ 2021-01-09 22:21 8% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2021-01-09 22:21 UTC (permalink / raw)
To: meta
Pushed as 392533147f50061d93cb9ed82abf98067dde5472
^ permalink raw reply [relevance 8%]
* [PATCH] v2writable: exact discontiguous history handling
@ 2021-01-05 1:29 14% ` Eric Wong
2021-01-09 22:21 8% ` Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2021-01-05 1:29 UTC (permalink / raw)
To: meta
Eric Wong <e@80x24.org> wrote:
> That would allow the new version of the edited message to be
> piped and seen by NNTP/IMAP readers.
>
> You *do* want to pipe the new version of the message you've
> edited, right?
---------8<--------
Subject: [PATCH] v2writable: exact discontiguous history handling
We've always temporarily unindexeded messages before reindexing
them again if there's discontiguous history.
This change improves the mechanism we use to prevent NNTP and
IMAP clients from seeing duplicate messages.
Previously, we relied on mapping Message-IDs to NNTP article
numbers to ensure clients would not see the same message twice.
This worked for most messages, but not for for messages with
reused or duplicate Message-IDs.
Instead of relying on Message-IDs as a key, we now rely on the
git blob object ID for exact content matching. This allows
truly different messages to show up for NNTP|IMAP clients, while
still those clients from seeing the message again.
---
lib/PublicInbox/V2Writable.pm | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/lib/PublicInbox/V2Writable.pm b/lib/PublicInbox/V2Writable.pm
index 459c7e86..54004fd7 100644
--- a/lib/PublicInbox/V2Writable.pm
+++ b/lib/PublicInbox/V2Writable.pm
@@ -888,12 +888,16 @@ sub index_oid { # cat_async callback
}
# {unindexed} is unlikely
- if ((my $unindexed = $arg->{unindexed}) && scalar(@$mids) == 1) {
- $num = delete($unindexed->{$mids->[0]});
+ if (my $unindexed = $arg->{unindexed}) {
+ my $oidbin = pack('H*', $oid);
+ my $u = $unindexed->{$oidbin};
+ ($num, $mid0) = splice(@$u, 0, 2) if $u;
if (defined $num) {
- $mid0 = $mids->[0];
$self->{mm}->mid_set($num, $mid0);
- delete($arg->{unindexed}) if !keys(%$unindexed);
+ if (scalar(@$u) == 0) { # done with current OID
+ delete $unindexed->{$oidbin};
+ delete($arg->{unindexed}) if !keys(%$unindexed);
+ }
}
}
if (!defined($num)) { # reuse if reindexing (or duplicates)
@@ -1160,10 +1164,13 @@ sub unindex_oid ($$;$) { # git->cat_async callback
warn "BUG: multiple articles linked to $oid\n",
join(',',sort keys %gone), "\n";
}
- foreach my $num (keys %gone) {
+ # reuse (num => mid) mapping in ascending numeric order
+ for my $num (sort { $a <=> $b } keys %gone) {
+ $num += 0;
if ($unindexed) {
my $mid0 = $mm->mid_for($num);
- $unindexed->{$mid0} = $num;
+ my $oidbin = pack('H*', $oid);
+ push @{$unindexed->{$oidbin}}, $num, $mid0;
}
$mm->num_delete($num);
}
@@ -1179,7 +1186,7 @@ sub git { $_[0]->{ibx}->git }
sub unindex_todo ($$$) {
my ($self, $sync, $unit) = @_;
my $unindex_range = delete($unit->{unindex_range}) // return;
- my $unindexed = $sync->{unindexed} //= {}; # $mid0 => $num
+ my $unindexed = $sync->{unindexed} //= {}; # $oidbin => [$num, $mid0]
my $before = scalar keys %$unindexed;
# order does not matter, here:
my $fh = $unit->{git}->popen(qw(log --raw -r --no-notes --no-color
^ permalink raw reply related [relevance 14%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2020-12-21 21:20 public-inbox + mlmmj best practices? Konstantin Ryabitsev
2020-12-21 21:39 ` Eric Wong
2020-12-22 6:28 ` Eric Wong
2020-12-28 16:22 ` Konstantin Ryabitsev
2020-12-28 21:31 ` Eric Wong
2021-01-04 20:12 ` Konstantin Ryabitsev
2021-01-05 1:06 ` Eric Wong
2021-01-05 1:29 14% ` [PATCH] v2writable: exact discontiguous history handling Eric Wong
2021-01-09 22:21 8% ` Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).