Re: [PATCH 3/3] v2writable: reindex handles 3-headered monsters

[Eric Wong <e@80x24.org>]

Eric Wong <e@80x24.org> wrote:
> +++ b/lib/PublicInbox/V2Writable.pm
> +sub reindex_oid ($$$$) {

<snip>

> +	} else { # multiple MIDs are a weird case:
> +		my $del = 0;
> +		for (@$mids) {
> +			$del += delete($sync->{D}->{"$_\0$cid"}) // 0;
> +		}
> +		if ($del) {
> +			unindex_oid_remote($self, $oid, $_) for @$mids;
> +			# do not delete from {mm_tmp}, since another
> +			# single-MID message may use it.
> +		} else { # handle them at the end:
> +			push @{$sync->{multi_mid} //= []}, $oid;

Part of me worres @$multi_mid can be abused here by people
trying to OOM the indexing process...

<snip>

> @@ -1184,10 +1265,22 @@ sub index_sync {
>  
>  	# unindex is required for leftovers if "deletes" affect messages
>  	# in a previous fetch+index window:
> +	my $git;
>  	if (my @leftovers = values %{delete $sync->{D}}) {
> -		my $git = $self->{-inbox}->git;
> -		unindex_oid($self, $git, $_) for @leftovers;
> -		$git->cleanup;
> +		$git = $self->{-inbox}->git;
> +		for my $oid (@leftovers) {
> +			$self->{current_info} = "leftover $oid";
> +			unindex_oid($self, $git, $oid);
> +		}
> +	}
> +	if (my $multi_mid = delete $sync->{multi_mid}) {
> +		$git //= $self->{-inbox}->git;
> +
> +		while (defined(my $oid = pop(@$multi_mid))) {
> +			$self->{current_info} = "multi_mid $oid";
> +			reindex_oid_m($self, $sync, $git, $oid);
> +		}
> +		$git->cleanup if $git;
>  	}
>  	$self->done;

I suppose we could easily write to a temporary file and use
fixed offsets, too; but fixed offsets would make git + SHA-256
more difficult.  So maybe Tie::File (stdlib) or a SQLite-based
stack could work, too...