user/dev discussion of public-inbox itself
 help / Atom feed
* [PATCH] Import.pm: When purging replace a purged file with a zero length file
@ 2018-08-10  0:08 ebiederm
  2018-08-10 17:47 ` Eric Wong
  0 siblings, 1 reply; 3+ messages in thread
From: ebiederm @ 2018-08-10  0:08 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta


This ensures that the number of added files remains the same and thus
the article numbers derived from a repository will remain the same.

I think this is the last place in public-inbox that has to be tweaked to
guarantee the generated article number will remain the same in an public
inbox archive.

Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
---
 lib/PublicInbox/Import.pm | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
index bfa7a8053297..3df7d98f298b 100644
--- a/lib/PublicInbox/Import.pm
+++ b/lib/PublicInbox/Import.pm
@@ -519,11 +519,12 @@ sub purge_oids {
 			push @buf, $buf;
 		} elsif (/^M 100644 ([a-f0-9]+) (\w+)/) {
 			my ($oid, $path) = ($1, $2);
+			$tree->{$path} = 1;
 			if ($purge->{$oid}) {
 				push @oids, $oid;
-				delete $tree->{$path};
+				my $cmd = "M 100644 inline $path\ndata 0\n\n";
+				push @buf, $cmd;
 			} else {
-				$tree->{$path} = 1;
 				push @buf, $_;
 			}
 		} elsif (/^D (\w+)/) {
-- 
2.17.1


^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] Import.pm: When purging replace a purged file with a zero length file
  2018-08-10  0:08 [PATCH] Import.pm: When purging replace a purged file with a zero length file ebiederm
@ 2018-08-10 17:47 ` Eric Wong
  2018-08-11  1:12   ` ebiederm
  0 siblings, 1 reply; 3+ messages in thread
From: Eric Wong @ 2018-08-10 17:47 UTC (permalink / raw)
  To: Eric W. Biederman; +Cc: meta

"Eric W. Biederman" <ebiederm@xmission.com> wrote:
> 
> This ensures that the number of added files remains the same and thus
> the article numbers derived from a repository will remain the same.
> 
> I think this is the last place in public-inbox that has to be tweaked to
> guarantee the generated article number will remain the same in an public
> inbox archive.

OK, definitely desirable.

> Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
> ---
>  lib/PublicInbox/Import.pm | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
> index bfa7a8053297..3df7d98f298b 100644
> --- a/lib/PublicInbox/Import.pm
> +++ b/lib/PublicInbox/Import.pm
> @@ -519,11 +519,12 @@ sub purge_oids {
>  			push @buf, $buf;
>  		} elsif (/^M 100644 ([a-f0-9]+) (\w+)/) {
>  			my ($oid, $path) = ($1, $2);
> +			$tree->{$path} = 1;
>  			if ($purge->{$oid}) {
>  				push @oids, $oid;
> -				delete $tree->{$path};
> +				my $cmd = "M 100644 inline $path\ndata 0\n\n";
> +				push @buf, $cmd;
>  			} else {
> -				$tree->{$path} = 1;
>  				push @buf, $_;
>  			}
>  		} elsif (/^D (\w+)/) {
> -- 

OK.  I haven't checked, but is the indexing/re-indexing code
able to deal with zero-byte messages?  Thanks.

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH] Import.pm: When purging replace a purged file with a zero length file
  2018-08-10 17:47 ` Eric Wong
@ 2018-08-11  1:12   ` ebiederm
  0 siblings, 0 replies; 3+ messages in thread
From: ebiederm @ 2018-08-11  1:12 UTC (permalink / raw)
  To: Eric Wong; +Cc: meta

Eric Wong <e@80x24.org> writes:

> "Eric W. Biederman" <ebiederm@xmission.com> wrote:
>> 
>> This ensures that the number of added files remains the same and thus
>> the article numbers derived from a repository will remain the same.
>> 
>> I think this is the last place in public-inbox that has to be tweaked to
>> guarantee the generated article number will remain the same in an public
>> inbox archive.
>
> OK, definitely desirable.
>
>> Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
>> ---
>>  lib/PublicInbox/Import.pm | 5 +++--
>>  1 file changed, 3 insertions(+), 2 deletions(-)
>> 
>> diff --git a/lib/PublicInbox/Import.pm b/lib/PublicInbox/Import.pm
>> index bfa7a8053297..3df7d98f298b 100644
>> --- a/lib/PublicInbox/Import.pm
>> +++ b/lib/PublicInbox/Import.pm
>> @@ -519,11 +519,12 @@ sub purge_oids {
>>  			push @buf, $buf;
>>  		} elsif (/^M 100644 ([a-f0-9]+) (\w+)/) {
>>  			my ($oid, $path) = ($1, $2);
>> +			$tree->{$path} = 1;
>>  			if ($purge->{$oid}) {
>>  				push @oids, $oid;
>> -				delete $tree->{$path};
>> +				my $cmd = "M 100644 inline $path\ndata 0\n\n";
>> +				push @buf, $cmd;
>>  			} else {
>> -				$tree->{$path} = 1;
>>  				push @buf, $_;
>>  			}
>>  		} elsif (/^D (\w+)/) {
>> -- 
>
> OK.  I haven't checked, but is the indexing/re-indexing code
> able to deal with zero-byte messages?  Thanks.

The v2mirror test covers this case and it doesn't seem to have any
problems.  The v2mirror performs an inex_sync after the purge and looks
for warnings and doesn't get any.  So I think we are ok.  Skimming
through the code I don't see any obvious issues either.

Eric


^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, back to index

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-08-10  0:08 [PATCH] Import.pm: When purging replace a purged file with a zero length file ebiederm
2018-08-10 17:47 ` Eric Wong
2018-08-11  1:12   ` ebiederm

user/dev discussion of public-inbox itself

Archives are clonable:
	git clone --mirror https://public-inbox.org/meta
	git clone --mirror http://czquwvybam4bgbro.onion/meta
	git clone --mirror http://hjrcffqmbrq6wope.onion/meta
	git clone --mirror http://ou63pmih66umazou.onion/meta

Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.mail.public-inbox.meta
	nntp://ou63pmih66umazou.onion/inbox.comp.mail.public-inbox.meta
	nntp://czquwvybam4bgbro.onion/inbox.comp.mail.public-inbox.meta
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.mail.public-inbox.meta
	nntp://news.gmane.org/gmane.mail.public-inbox.general

 note: .onion URLs require Tor: https://www.torproject.org/
       or Tor2web: https://www.tor2web.org/

AGPL code for this site: git clone https://public-inbox.org/ public-inbox