From: Eric Wong <e@yhbt.net>
To: meta@public-inbox.org
Subject: [PATCH 1/3] release large (non ref) scalars using `undef $sv'
Date: Sun, 5 Apr 2020 07:53:47 +0000 [thread overview]
Message-ID: <20200405075349.2173-2-e@yhbt.net> (raw)
In-Reply-To: <20200405075349.2173-1-e@yhbt.net>
Using `undef EXPR' like a function call actually frees the heap
memory associated with the scalar, whereas `$sv = undef' or
`$sv = ""' will hold the buffer around until $sv goes out
of scope.
The `sv_set_undef' documentation in the perlapi(1) manpage
explicitly states this:
The perl equivalent is "$sv = undef;". Note that it doesn't
free any string buffer, unlike "undef $sv".
And I've confirmed by reading Dump() output from Devel::Peek.
We'll also inline the old index_body sub in SearchIdx.pm to make
the scope of the scalar more obvious.
This change saves several hundred kB RSS on both -index and
-httpd when hitting large emails with thousands of lines.
---
lib/PublicInbox/SearchIdx.pm | 33 ++++++++++++++++-----------------
lib/PublicInbox/View.pm | 4 ++--
lib/PublicInbox/ViewDiff.pm | 2 +-
3 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 89d8bc2b..9a5484e3 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -275,22 +275,8 @@ sub index_diff ($$$) {
index_text($self, join("\n", @xnq), 1, 'XNQ');
}
-sub index_body ($$$) {
- my ($self, $txt, $doc) = @_;
- if ($doc) {
- # does it look like a diff?
- if ($txt =~ /^(?:diff|---|\+\+\+) /ms) {
- index_diff($self, $txt, $doc);
- } else {
- index_text($self, $txt, 1, 'XNQ');
- }
- } else {
- index_text($self, $txt, 0, 'XQUOT');
- }
-}
-
sub index_xapian { # msg_iter callback
- my ($part, $depth, @idx) = @{$_[0]};
+ my $part = $_[0]->[0]; # ignore $depth and @idx
my ($self, $doc) = @{$_[1]};
my $ct = $part->content_type || 'text/plain';
my $fn = $part->filename;
@@ -300,11 +286,24 @@ sub index_xapian { # msg_iter callback
my ($s, undef) = msg_part_text($part, $ct);
defined $s or return;
+ $_[0]->[0] = $part = undef; # free memory
# split off quoted and unquoted blocks:
my @sections = PublicInbox::MsgIter::split_quotes($s);
- $part = $s = undef;
- index_body($self, $_, /\A>/ ? 0 : $doc) for @sections;
+ undef $s; # free memory
+ for my $txt (@sections) {
+ if ($txt =~ /\A>/) {
+ index_text($self, $txt, 0, 'XQUOT');
+ } else {
+ # does it look like a diff?
+ if ($txt =~ /^(?:diff|---|\+\+\+) /ms) {
+ index_diff($self, $txt, $doc);
+ } else {
+ index_text($self, $txt, 1, 'XNQ');
+ }
+ }
+ undef $txt; # free memory
+ }
}
sub add_xapian ($$$$) {
diff --git a/lib/PublicInbox/View.pm b/lib/PublicInbox/View.pm
index 1e53d8dc..ddd94e48 100644
--- a/lib/PublicInbox/View.pm
+++ b/lib/PublicInbox/View.pm
@@ -566,7 +566,7 @@ sub add_text_body { # callback for msg_iter
# split off quoted and unquoted blocks:
my @sections = PublicInbox::MsgIter::split_quotes($s);
- $s = '';
+ undef $s; # free memory
my $rv = $ctx->{obuf};
if (defined($fn) || $depth > 0 || $err) {
# badly-encoded message with $err? tell the world about it!
@@ -587,7 +587,7 @@ sub add_text_body { # callback for msg_iter
# regular lines, OK
$$rv .= $l->to_html($cur);
}
- $cur = undef;
+ undef $cur; # free memory
}
obfuscate_addrs($ibx, $$rv) if $ibx->{obfuscate};
diff --git a/lib/PublicInbox/ViewDiff.pm b/lib/PublicInbox/ViewDiff.pm
index f7422712..3d6058a9 100644
--- a/lib/PublicInbox/ViewDiff.pm
+++ b/lib/PublicInbox/ViewDiff.pm
@@ -192,7 +192,7 @@ sub flush_diff ($$) {
my ($ctx, $cur) = @_;
my @top = split($EXTRACT_DIFFS, $$cur);
- $$cur = undef;
+ undef $$cur; # free memory
my $linkify = $ctx->{-linkify};
my $dst = $ctx->{obuf};
next prev parent reply other threads:[~2020-04-05 7:53 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-04-05 7:53 [PATCH 0/3] some efficiency improvements Eric Wong
2020-04-05 7:53 ` Eric Wong [this message]
2020-04-05 7:53 ` [PATCH 2/3] mbox: halve ->getline "context switches" Eric Wong
2020-04-05 7:53 ` [PATCH 3/3] git: reduce stat buffer storage overhead Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200405075349.2173-2-e@yhbt.net \
--to=e@yhbt.net \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/public-inbox.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).