From: Jonathan Nieder <jrnieder@gmail.com>
To: Ramkumar Ramachandra <artagnon@gmail.com>
Cc: Git Mailing List <git@vger.kernel.org>,
David Michael Barr <david.barr@cordelta.com>,
Sverre Rabbelier <srabbelier@gmail.com>,
Junio C Hamano <gitster@pobox.com>
Subject: [PATCH 05/10] Add string-specific memory pool
Date: Mon, 9 Aug 2010 17:34:42 -0500 [thread overview]
Message-ID: <20100809223442.GA4429@burratino> (raw)
In-Reply-To: <20100809215719.GA4203@burratino>
From: David Barr <david.barr@cordelta.com>
Intern strings so they can be compared by address and stored without
wasting space.
This library uses the macros in the obj_pool.h and trp.h to create a
memory pool for strings and expose an API for handling them.
[rr: added API docs]
[jn: with some API simplifications, new documentation and tests]
Signed-off-by: David Barr <david.barr@cordelta.com>
Signed-off-by: Ramkumar Ramachandra <artagnon@gmail.com>
Signed-off-by: Jonathan Nieder <jrnieder@gmail.com>
---
New test. The return value from pool_tok_seq is not checked by
the vcs-svn lib but trying to use it in tests revealed it was not
so intuitive. pool_tok_seq() was behaving strangely when passed
an array of size 0; I think there is nothing sane to do in that
case --- maybe it should abort(). The API was passing around
char * that cannot be modified; changed to const char *.
Another set of eyes on this would be welcome.
.gitignore | 1 +
Makefile | 9 +++-
t/t0080-vcs-svn.sh | 16 +++++++
test-string-pool.c | 31 ++++++++++++++
vcs-svn/string_pool.c | 102 +++++++++++++++++++++++++++++++++++++++++++++++
vcs-svn/string_pool.h | 11 +++++
vcs-svn/string_pool.txt | 43 ++++++++++++++++++++
7 files changed, 210 insertions(+), 3 deletions(-)
create mode 100644 test-string-pool.c
create mode 100644 vcs-svn/string_pool.c
create mode 100644 vcs-svn/string_pool.h
create mode 100644 vcs-svn/string_pool.txt
diff --git a/.gitignore b/.gitignore
index af47653..9f109db 100644
--- a/.gitignore
+++ b/.gitignore
@@ -173,6 +173,7 @@
/test-run-command
/test-sha1
/test-sigchain
+/test-string-pool
/test-treap
/common-cmds.h
*.tar.gz
diff --git a/Makefile b/Makefile
index e7c33ec..24103c9 100644
--- a/Makefile
+++ b/Makefile
@@ -415,6 +415,7 @@ TEST_PROGRAMS_NEED_X += test-path-utils
TEST_PROGRAMS_NEED_X += test-run-command
TEST_PROGRAMS_NEED_X += test-sha1
TEST_PROGRAMS_NEED_X += test-sigchain
+TEST_PROGRAMS_NEED_X += test-string-pool
TEST_PROGRAMS_NEED_X += test-treap
TEST_PROGRAMS_NEED_X += test-index-version
@@ -1742,7 +1743,7 @@ ifndef NO_CURL
endif
XDIFF_OBJS = xdiff/xdiffi.o xdiff/xprepare.o xdiff/xutils.o xdiff/xemit.o \
xdiff/xmerge.o xdiff/xpatience.o
-VCSSVN_OBJS =
+VCSSVN_OBJS = vcs-svn/string_pool.o
OBJECTS := $(GIT_OBJS) $(XDIFF_OBJS) $(VCSSVN_OBJS)
dep_files := $(foreach f,$(OBJECTS),$(dir $f).depend/$(notdir $f).d)
@@ -1866,7 +1867,7 @@ xdiff-interface.o $(XDIFF_OBJS): \
xdiff/xutils.h xdiff/xprepare.h xdiff/xdiffi.h xdiff/xemit.h
$(VCSSVN_OBJS): \
- vcs-svn/obj_pool.h vcs-svn/trp.h
+ vcs-svn/obj_pool.h vcs-svn/trp.h vcs-svn/string_pool.h
endif
exec_cmd.s exec_cmd.o: EXTRA_CPPFLAGS = \
@@ -2017,10 +2018,12 @@ test-delta$X: diff-delta.o patch-delta.o
test-parse-options$X: parse-options.o
+test-string-pool$X: vcs-svn/lib.a
+
.PRECIOUS: $(TEST_OBJS)
test-%$X: test-%.o $(GITLIBS)
- $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(LIBS)
+ $(QUIET_LINK)$(CC) $(ALL_CFLAGS) -o $@ $(ALL_LDFLAGS) $(filter %.o,$^) $(filter %.a,$^) $(LIBS)
check-sha1:: test-sha1$X
./test-sha1.sh
diff --git a/t/t0080-vcs-svn.sh b/t/t0080-vcs-svn.sh
index ce02c58..99a314b 100755
--- a/t/t0080-vcs-svn.sh
+++ b/t/t0080-vcs-svn.sh
@@ -76,6 +76,22 @@ test_expect_success 'obj pool: high-water mark' '
test_cmp expected actual
'
+test_expect_success 'string pool' '
+ echo a does not equal b >expected.differ &&
+ echo a equals a >expected.match &&
+ echo equals equals equals >expected.matchmore &&
+
+ test-string-pool "a,--b" >actual.differ &&
+ test-string-pool "a,a" >actual.match &&
+ test-string-pool "equals-equals" >actual.matchmore &&
+ test_must_fail test-string-pool a,a,a &&
+ test_must_fail test-string-pool a &&
+
+ test_cmp expected.differ actual.differ &&
+ test_cmp expected.match actual.match &&
+ test_cmp expected.matchmore actual.matchmore
+'
+
test_expect_success 'treap sort' '
cat <<-\EOF >unsorted &&
68
diff --git a/test-string-pool.c b/test-string-pool.c
new file mode 100644
index 0000000..2adf84b
--- /dev/null
+++ b/test-string-pool.c
@@ -0,0 +1,31 @@
+/*
+ * test-string-pool.c: code to exercise the svn importer's string pool
+ */
+
+#include "git-compat-util.h"
+#include "vcs-svn/string_pool.h"
+
+int main(int argc, char *argv[])
+{
+ const uint32_t unequal = pool_intern("does not equal");
+ const uint32_t equal = pool_intern("equals");
+ uint32_t buf[3];
+ uint32_t n;
+
+ if (argc != 2)
+ usage("test-string-pool <string>,<string>");
+
+ n = pool_tok_seq(3, buf, ",-", argv[1]);
+ if (n >= 3)
+ die("too many strings");
+ if (n <= 1)
+ die("too few strings");
+
+ buf[2] = buf[1];
+ buf[1] = (buf[0] == buf[2]) ? equal : unequal;
+ pool_print_seq(3, buf, ' ', stdout);
+ fputc('\n', stdout);
+
+ pool_reset();
+ return 0;
+}
diff --git a/vcs-svn/string_pool.c b/vcs-svn/string_pool.c
new file mode 100644
index 0000000..550f0e5
--- /dev/null
+++ b/vcs-svn/string_pool.c
@@ -0,0 +1,102 @@
+/*
+ * Licensed under a two-clause BSD-style license.
+ * See LICENSE for details.
+ */
+
+#include "git-compat-util.h"
+#include "trp.h"
+#include "obj_pool.h"
+#include "string_pool.h"
+
+static struct trp_root tree = { ~0 };
+
+struct node {
+ uint32_t offset;
+ struct trp_node children;
+};
+
+/* Two memory pools: one for struct node, and another for strings */
+obj_pool_gen(node, struct node, 4096)
+obj_pool_gen(string, char, 4096)
+
+static char *node_value(struct node *node)
+{
+ return node ? string_pointer(node->offset) : NULL;
+}
+
+static int node_cmp(struct node *a, struct node *b)
+{
+ return strcmp(node_value(a), node_value(b));
+}
+
+/* Build a Treap from the node structure (a trp_node w/ offset) */
+trp_gen(static, tree_, struct node, children, node, node_cmp);
+
+const char *pool_fetch(uint32_t entry)
+{
+ return node_value(node_pointer(entry));
+}
+
+uint32_t pool_intern(const char *key)
+{
+ /* Canonicalize key */
+ struct node *match = NULL;
+ uint32_t key_len;
+ if (key == NULL)
+ return ~0;
+ key_len = strlen(key) + 1;
+ struct node *node = node_pointer(node_alloc(1));
+ node->offset = string_alloc(key_len);
+ strcpy(node_value(node), key);
+ match = tree_search(&tree, node);
+ if (!match) {
+ tree_insert(&tree, node);
+ } else {
+ node_free(1);
+ string_free(key_len);
+ node = match;
+ }
+ return node_offset(node);
+}
+
+uint32_t pool_tok_r(char *str, const char *delim, char **saveptr)
+{
+ char *token = strtok_r(str, delim, saveptr);
+ return token ? pool_intern(token) : ~0;
+}
+
+void pool_print_seq(uint32_t len, uint32_t *seq, char delim, FILE *stream)
+{
+ uint32_t i;
+ for (i = 0; i < len && ~seq[i]; i++) {
+ fputs(pool_fetch(seq[i]), stream);
+ if (i < len - 1 && ~seq[i + 1])
+ fputc(delim, stream);
+ }
+}
+
+uint32_t pool_tok_seq(uint32_t sz, uint32_t *seq, const char *delim, char *str)
+{
+ char *context = NULL;
+ uint32_t token = ~0;
+ uint32_t length;
+
+ if (sz == 0)
+ return ~0;
+ if (str)
+ token = pool_tok_r(str, delim, &context);
+ for (length = 0; length < sz; length++) {
+ seq[length] = token;
+ if (token == ~0)
+ return length;
+ token = pool_tok_r(NULL, delim, &context);
+ }
+ seq[sz - 1] = ~0;
+ return sz;
+}
+
+void pool_reset(void)
+{
+ node_reset();
+ string_reset();
+}
diff --git a/vcs-svn/string_pool.h b/vcs-svn/string_pool.h
new file mode 100644
index 0000000..222fb66
--- /dev/null
+++ b/vcs-svn/string_pool.h
@@ -0,0 +1,11 @@
+#ifndef STRING_POOL_H_
+#define STRING_POOL_H_
+
+uint32_t pool_intern(const char *key);
+const char *pool_fetch(uint32_t entry);
+uint32_t pool_tok_r(char *str, const char *delim, char **saveptr);
+void pool_print_seq(uint32_t len, uint32_t *seq, char delim, FILE *stream);
+uint32_t pool_tok_seq(uint32_t sz, uint32_t *seq, const char *delim, char *str);
+void pool_reset(void);
+
+#endif
diff --git a/vcs-svn/string_pool.txt b/vcs-svn/string_pool.txt
new file mode 100644
index 0000000..1b41f15
--- /dev/null
+++ b/vcs-svn/string_pool.txt
@@ -0,0 +1,43 @@
+string_pool API
+===============
+
+The string_pool API provides facilities for replacing strings
+with integer keys that can be more easily compared and stored.
+The facilities are designed so that one could teach Git without
+too much trouble to store the information needed for these keys to
+remain valid over multiple executions.
+
+Functions
+---------
+
+pool_intern::
+ Include a string in the string pool and get its key.
+ If that string is already in the pool, retrieves its
+ existing key.
+
+pool_fetch::
+ Retrieve the string associated to a given key.
+
+pool_tok_r::
+ Extract the key of the next token from a string.
+ Interface mimics strtok_r.
+
+pool_print_seq::
+ Print a sequence of strings named by key to a file, using the
+ specified delimiter to separate them.
+
+ If NULL (key ~0) appears in the sequence, the sequence ends
+ early.
+
+pool_tok_seq::
+ Split a string into tokens, storing the keys of segments
+ into a caller-provided array.
+
+ Unless sz is 0, the array will always be ~0-terminated.
+ If there is not enough room for all the tokens, the
+ array holds as many tokens as fit in the entries before
+ the terminating ~0. Return value is the index after the
+ last token, or sz if the tokens did not fit.
+
+pool_reset::
+ Deallocate storage for the string pool.
--
1.7.2.1.544.ga752d.dirty
next prev parent reply other threads:[~2010-08-09 22:36 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-07-15 16:22 [PATCH 0/8] Resurrect rr/svn-export Ramkumar Ramachandra
2010-07-15 16:22 ` [PATCH 1/8] Export parse_date_basic() to convert a date string to timestamp Ramkumar Ramachandra
2010-07-15 17:25 ` Jonathan Nieder
2010-07-15 22:54 ` Junio C Hamano
2010-07-15 16:22 ` [PATCH 2/8] Introduce vcs-svn lib Ramkumar Ramachandra
2010-07-15 17:46 ` Jonathan Nieder
2010-07-15 19:15 ` Ramkumar Ramachandra
2010-07-15 16:22 ` [PATCH 3/8] Add memory pool library Ramkumar Ramachandra
2010-07-15 18:57 ` Jonathan Nieder
2010-07-15 19:12 ` Ramkumar Ramachandra
2010-07-15 16:23 ` [PATCH 4/8] Add treap implementation Ramkumar Ramachandra
2010-07-15 19:09 ` Jonathan Nieder
2010-07-15 19:18 ` Ramkumar Ramachandra
2010-07-15 16:23 ` [PATCH 5/8] Add string-specific memory pool Ramkumar Ramachandra
2010-07-15 16:23 ` [PATCH 6/8] Add stream helper library Ramkumar Ramachandra
2010-07-15 19:19 ` Jonathan Nieder
2010-07-15 16:23 ` [PATCH 7/8] Add infrastructure to write revisions in fast-export format Ramkumar Ramachandra
2010-07-15 19:28 ` Jonathan Nieder
2010-07-15 16:23 ` [PATCH 8/8] Add SVN dump parser Ramkumar Ramachandra
2010-07-15 19:52 ` Jonathan Nieder
2010-07-15 20:04 ` Jonathan Nieder
2010-07-16 10:13 ` [PATCH 0/8] Resurrect rr/svn-export Jonathan Nieder
2010-07-16 10:16 ` [PATCH 3/9] Add memory pool library Jonathan Nieder
2010-07-16 10:23 ` [PATCH 4/9] Add treap implementation Jonathan Nieder
2010-07-16 18:26 ` Jonathan Nieder
2010-08-09 21:57 ` [PATCH 0/10] rr/svn-export reroll Jonathan Nieder
2010-08-09 22:01 ` [PATCH 01/10] Export parse_date_basic() to convert a date string to timestamp Jonathan Nieder
2010-08-09 22:04 ` [PATCH 02/10] Introduce vcs-svn lib Jonathan Nieder
2010-08-09 22:11 ` [PATCH 03/10] Add memory pool library Jonathan Nieder
2010-08-09 22:17 ` [PATCH 04/10] Add treap implementation Jonathan Nieder
2010-08-12 17:22 ` Junio C Hamano
2010-08-12 22:02 ` Jonathan Nieder
2010-08-12 22:11 ` Jonathan Nieder
2010-08-12 22:44 ` Junio C Hamano
2010-08-09 22:34 ` Jonathan Nieder [this message]
2010-08-12 17:22 ` [PATCH 05/10] Add string-specific memory pool Junio C Hamano
2010-08-12 21:30 ` Jonathan Nieder
2010-08-09 22:39 ` [PATCH 06/10] Add stream helper library Jonathan Nieder
2010-08-09 22:48 ` [PATCH 07/10] Infrastructure to write revisions in fast-export format Jonathan Nieder
2010-08-09 22:55 ` [PATCH 08/10] SVN dump parser Jonathan Nieder
2010-08-12 17:22 ` Junio C Hamano
2010-08-09 22:55 ` PATCH 09/10] Update svn-fe manual Jonathan Nieder
2010-08-09 22:58 ` [PATCH 10/10] svn-fe manual: Clarify warning about deltas in dump files Jonathan Nieder
2010-08-10 12:53 ` [PATCH 0/10] rr/svn-export reroll Ramkumar Ramachandra
2010-08-11 1:53 ` Jonathan Nieder
2010-10-11 2:34 ` [PATCH/WIP 00/16] svn delta applier Jonathan Nieder
2010-10-11 2:37 ` [PATCH 01/16] vcs-svn: Eliminate global byte_buffer[] array Jonathan Nieder
2010-10-11 2:39 ` [PATCH 03/16] vcs-svn: Collect line_buffer data in a struct Jonathan Nieder
2010-10-11 2:41 ` [PATCH 04/16] vcs-svn: Teach line_buffer to handle multiple input files Jonathan Nieder
2010-10-11 2:44 ` [PATCH 05/16] vcs-svn: Make buffer_skip_bytes() report partial reads Jonathan Nieder
2010-10-11 2:46 ` [PATCH 06/16] vcs-svn: Improve support for reading large files Jonathan Nieder
2010-10-11 2:47 ` [PATCH 07/16] vcs-svn: Add binary-safe read() function Jonathan Nieder
2010-10-11 2:47 ` [PATCH 08/16] vcs-svn: Let callers peek ahead to find stream end Jonathan Nieder
2010-10-11 2:51 ` [PATCH 09/16] vcs-svn: Allow input errors to be detected early Jonathan Nieder
2010-10-11 2:52 ` [PATCH 10/16] vcs-svn: Allow character-oriented input Jonathan Nieder
2010-10-11 2:53 ` [PATCH 11/16] vcs-svn: Add code to maintain a sliding view of a file Jonathan Nieder
2010-10-11 2:55 ` [PATCH 12/16] vcs-svn: Learn to parse variable-length integers Jonathan Nieder
2010-10-11 2:58 ` [PATCH 13/16] vcs-svn: Learn to check for SVN\0 magic Jonathan Nieder
2010-10-11 2:59 ` [PATCH 14/16] compat: helper for detecting unsigned overflow Jonathan Nieder
2010-10-11 3:00 ` [PATCH 15/16] t9010 (svn-fe): Eliminate dependency on svn perl bindings Jonathan Nieder
2010-10-11 3:11 ` [PATCH 02/16] vcs-svn: Replace buffer_read_string() memory pool with a strbuf Jonathan Nieder
2010-10-11 4:01 ` [PATCH/RFC 16'/16] vcs-svn: Add svn delta parser Jonathan Nieder
2010-10-13 9:17 ` [PATCH/RFC 0/11] Building up the " Jonathan Nieder
2010-10-13 9:19 ` [PATCH 01/11] fixup! vcs-svn: Learn to parse variable-length integers Jonathan Nieder
2010-10-13 9:21 ` [PATCH 02/11] vcs-svn: Skeleton of an svn delta parser Jonathan Nieder
2010-10-13 9:30 ` [PATCH 03/11] vcs-svn: Read the preimage while applying deltas Jonathan Nieder
2010-10-14 21:45 ` Sam Vilain
2010-10-14 23:40 ` Jonathan Nieder
2010-10-13 9:35 ` [PATCH 04/11] vcs-svn: Read inline data from deltas Jonathan Nieder
2010-10-13 9:38 ` [PATCH 05/11] vcs-svn: Read instructions " Jonathan Nieder
2010-10-13 9:39 ` [PATCH 06/11] vcs-svn: Implement copyfrom_data delta instruction Jonathan Nieder
2010-10-13 9:41 ` [PATCH 07/11] vcs-svn: Check declared number of output bytes Jonathan Nieder
2010-10-13 9:48 ` [PATCH 08/11] vcs-svn: Reject deltas that do not consume all inline data Jonathan Nieder
2010-10-13 9:50 ` [PATCH 09/11] vcs-svn: Let deltas use data from postimage Jonathan Nieder
2010-10-13 9:53 ` [PATCH 10/11] vcs-svn: Reject deltas that read past end of preimage Jonathan Nieder
2010-10-13 9:58 ` [PATCH 11/11] vcs-svn: Allow deltas to copy from preimage Jonathan Nieder
2010-10-13 10:00 ` Jonathan Nieder
2010-10-18 17:00 ` [PATCH/RFC 0/11] Building up the delta parser Ramkumar Ramachandra
2010-10-18 17:03 ` Jonathan Nieder
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100809223442.GA4429@burratino \
--to=jrnieder@gmail.com \
--cc=artagnon@gmail.com \
--cc=david.barr@cordelta.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=srabbelier@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).