* [PATCH 4/4] import_slrnspool: fork a process for each message
2015-01-12 2:30 5% [PATCH 0/4] misc import_slrnspool improvements Eric Wong
@ 2015-01-12 2:30 7% ` Eric Wong
0 siblings, 0 replies; 2+ results
From: Eric Wong @ 2015-01-12 2:30 UTC (permalink / raw)
To: meta
This prevents process growth when importing large messages.
Memory growth could be due to the sliding sbrk window in glibc malloc
or a circular reference in the Email::* Perl code somewhere.
---
scripts/import_slrnspool | 46 ++++++++++++++++++++++++++++++----------------
1 file changed, 30 insertions(+), 16 deletions(-)
diff --git a/scripts/import_slrnspool b/scripts/import_slrnspool
index e55c96c..dd612b1 100755
--- a/scripts/import_slrnspool
+++ b/scripts/import_slrnspool
@@ -66,27 +66,41 @@ for (; $exit == 0 && $n < $max; $n++) {
print STDERR $fn, "\n";
open(my $fh, '<', $fn) or next;
$max = $n + $max_gap;
- my $f = Email::Filter->new(data => eval { local $/; <$fh> });
- my $s = $f->simple;
- # gmane rewrites Received headers, which increases spamminess
- # Some older archives set Original-To
- foreach my $x (qw(Received To)) {
- my @h = $s->header("Original-$x");
- if (@h) {
- $s->header_set($x, @h);
- $s->header_set("Original-$x");
+ # prevent process growth by forking a new process for each message
+ my $pid = fork;
+ die "failed to fork: $!\n" unless defined $pid;
+
+ if ($pid == 0) {
+ my $f = Email::Filter->new(data => eval { local $/; <$fh> });
+ close $fh;
+ $fh = undef;
+ my $s = $f->simple;
+
+ # gmane rewrites Received headers, which increases spamminess
+ # Some older archives set Original-To
+ foreach my $x (qw(Received To)) {
+ my @h = $s->header("Original-$x");
+ if (@h) {
+ $s->header_set($x, @h);
+ $s->header_set("Original-$x");
+ }
}
- }
- # triggers for the SA HEADER_SPAM rule
- foreach my $drop (qw(Approved)) { $s->header_set($drop) }
+ # triggers for the SA HEADER_SPAM rule
+ foreach my $drop (qw(Approved)) { $s->header_set($drop) }
- # appears to be an old gmane bug:
- $s->header_set('connect()');
+ # appears to be an old gmane bug:
+ $s->header_set('connect()');
- $f->exit(0);
- $f->pipe(@mda);
+ $f->exit(0);
+ $f->pipe(@mda);
+ exit 0;
+ } else {
+ close $fh;
+ waitpid($pid, 0);
+ die "error: $?\n" if $?;
+ }
$ok = $n + 1;
set_min($cfg, $ok);
}
--
EW
^ permalink raw reply related [relevance 7%]
* [PATCH 0/4] misc import_slrnspool improvements
@ 2015-01-12 2:30 5% Eric Wong
2015-01-12 2:30 7% ` [PATCH 4/4] import_slrnspool: fork a process for each message Eric Wong
0 siblings, 1 reply; 2+ results
From: Eric Wong @ 2015-01-12 2:30 UTC (permalink / raw)
To: meta
Eric Wong (4):
import_slrnspool: make filtering optional
import_slrnspool: graceful exit for interruptibility
import_slrnspool: load private config key
import_slrnspool: fork a process for each message
scripts/import_slrnspool | 79 ++++++++++++++++++++++++++++++++++--------------
1 file changed, 57 insertions(+), 22 deletions(-)
^ permalink raw reply [relevance 5%]
Results 1-2 of 2 | reverse | options above
-- pct% links below jump to the message on this page, permalinks otherwise --
2015-01-12 2:30 5% [PATCH 0/4] misc import_slrnspool improvements Eric Wong
2015-01-12 2:30 7% ` [PATCH 4/4] import_slrnspool: fork a process for each message Eric Wong
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).