[PATCH 00/10] Reftable coverity fixes

[PATCH 00/10] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'next'.

This addresses some complaints from Coverity that Peff reported.

CI on GGG shows a segfault on windows
(https://github.com/git/git/runs/4446635428?check_suite_focus=true), which I
can't make sense of. Any hints would be greatly appreciated.

Han-Wen Nienhuys (10):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: fix NULL derefs in error paths
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: make reftable_record a tagged union

 reftable/block.c          |  29 ++++--
 reftable/block_test.c     |  22 ++---
 reftable/blocksource.c    |   6 +-
 reftable/generic.c        |  35 ++++----
 reftable/iter.c           |   4 +-
 reftable/merged.c         |  37 ++++----
 reftable/pq.c             |   3 +-
 reftable/pq_test.c        |  31 +++----
 reftable/reader.c         | 108 ++++++++++++-----------
 reftable/readwrite_test.c |   1 -
 reftable/record.c         | 180 +++++++++++++++-----------------------
 reftable/record.h         |  45 ++++------
 reftable/record_test.c    | 162 +++++++++++++++++-----------------
 reftable/stack.c          |  15 ++--
 reftable/stack_test.c     |   3 +-
 reftable/writer.c         |  46 +++++-----
 t/helper/test-reftable.c  |   9 +-
 17 files changed, 360 insertions(+), 376 deletions(-)


base-commit: a8338297339b1ab064e799c15c4fc56c122ef967
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v1
Pull-Request: https://github.com/git/git/pull/1152
-- 
gitgitgadget

[PATCH 01/10] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH 02/10] reftable: fix resource leak in error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered by corrupt files, so it doesn't have test coverage. This
was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c | 25 ++++++++++++++++++-------
 1 file changed, 18 insertions(+), 7 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..d7347bb3152 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -213,15 +216,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
 			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +258,11 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	if (uncompressed) {
+		reftable_free(uncompressed);
+	}
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
-- 
gitgitgadget

[PATCH 03/10] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an opened
file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH 04/10] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index eb0b7228b0c..d628420e63a 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -814,6 +814,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH 05/10] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index d628420e63a..4b7292945c3 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -89,7 +89,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH 06/10] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and the next
available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index df5021ebf08..10dfd370e8e 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -877,7 +877,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -911,7 +911,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -932,7 +932,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 
 		sublock_file_fd = open(subtab_lock.buf,
 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
-		if (sublock_file_fd > 0) {
+		if (sublock_file_fd >= 0) {
 			close(sublock_file_fd);
 		} else if (sublock_file_fd < 0) {
 			if (errno == EEXIST) {
@@ -1013,7 +1013,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1050,9 +1050,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH 07/10] reftable: fix NULL derefs in error paths

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 3 ++-
 reftable/stack.c  | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..8d308d858f8 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -796,6 +796,7 @@ int reftable_reader_print_file(const char *tablename)
 	reftable_table_from_reader(&tab, r);
 	err = reftable_table_print(&tab);
 done:
-	reftable_reader_free(r);
+	if (r)
+		reftable_reader_free(r);
 	return err;
 }
diff --git a/reftable/stack.c b/reftable/stack.c
index 10dfd370e8e..eb03b6c74f6 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -707,7 +707,8 @@ done:
 	strbuf_release(&temp_tab_file_name);
 	strbuf_release(&tab_file_name);
 	strbuf_release(&next_name);
-	reftable_writer_free(wr);
+	if (wr)
+		reftable_writer_free(wr);
 	return err;
 }
 
-- 
gitgitgadget

[PATCH 08/10] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more pratical ordering when working on refactorings of the reftable
code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..f08d66df949 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	// test from simple to complex.
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH 09/10] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 5f6bcc2f775..7c40b9b77f3 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -597,7 +597,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH 10/10] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void pointer or
vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it cannot be
zero-initialized. To address this, introduce reftable_record_for() to create a
fresh instance, given a record type.

Thanks to Peff for the suggestion.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   4 +-
 reftable/block_test.c  |  22 +++---
 reftable/generic.c     |  35 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  37 ++++-----
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  31 ++++----
 reftable/reader.c      | 105 ++++++++++++------------
 reftable/record.c      | 176 ++++++++++++++++-------------------------
 reftable/record.h      |  45 +++++------
 reftable/record_test.c | 162 +++++++++++++++++++------------------
 reftable/writer.c      |  46 ++++++-----
 12 files changed, 319 insertions(+), 351 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index d7347bb3152..ad713caa57b 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -382,7 +382,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 		.key = *want,
 		.r = br,
 	};
-	struct reftable_record rec = reftable_new_record(block_reader_type(br));
+	struct reftable_record rec = reftable_record_for(block_reader_type(br));
 	struct strbuf key = STRBUF_INIT;
 	int err = 0;
 	struct block_iter next = {
@@ -424,7 +424,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..670329f2224 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.ref.refname = name;
+		rec.ref.value_type = REFTABLE_REF_VAL1;
+		rec.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.ref.refname = NULL;
+		rec.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..7f00aa9cdbe 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,19 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
+	int err = iterator_next(it, &rec);
+	*ref = rec.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
+	int err = iterator_next(it, &rec);
+	*log = rec.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..fedf5d8ab0b 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..e69955d7c2d 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -22,7 +22,7 @@ static int merged_iter_init(struct merged_iter *mi)
 {
 	int i = 0;
 	for (i = 0; i < mi->stack_len; i++) {
-		struct reftable_record rec = reftable_new_record(mi->typ);
+		struct reftable_record rec = reftable_record_for(mi->typ);
 		int err = iterator_next(&mi->stack[i], &rec);
 		if (err < 0) {
 			return err;
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_record_for(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -246,7 +245,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,
 		sizeof(struct reftable_iterator) * mt->stack_len);
 	struct merged_iter merged = {
 		.stack = iters,
-		.typ = reftable_record_type(rec),
+		.typ = rec->type,
 		.hash_id = mt->hash_id,
 		.suppress_deletions = mt->suppress_deletions,
 	};
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,13 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..89d069d674d 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,14 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = {
+			.rec = {
+				.type = BLOCK_TYPE_REF,
+				.ref = {
+					.refname = names[i],
+				}
+			}
+		};
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +57,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(rec->type == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->ref.refname;
+		rec->ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 8d308d858f8..0159f131122 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -238,9 +238,8 @@ static int table_iter_next_in_block(struct table_iter *ti,
 				    struct reftable_record *rec)
 {
 	int res = block_iter_next(&ti->bi, rec);
-	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+	if (res == 0 && rec->type == BLOCK_TYPE_REF) {
+		rec->ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -345,7 +344,7 @@ static int table_iter_next_block(struct table_iter *dest,
 
 static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)
 {
-	if (reftable_record_type(rec) != ti->typ)
+	if (rec->type != ti->typ)
 		return REFTABLE_API_ERROR;
 
 	while (1) {
@@ -437,8 +436,7 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,
 static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 			      struct reftable_record *want)
 {
-	struct reftable_record rec =
-		reftable_new_record(reftable_record_type(want));
+	struct reftable_record rec = reftable_record_for(want->type);
 	struct strbuf want_key = STRBUF_INIT;
 	struct strbuf got_key = STRBUF_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
@@ -475,7 +473,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -485,38 +483,40 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX,
+		.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
-	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
+	reftable_record_key(rec, &want_index.idx.last_key);
+	err = reader_start(r, &index_iter, rec->type, 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.idx.last_key);
 		if (err < 0)
 			goto done;
 
-		if (next.typ == reftable_record_type(rec)) {
+		if (next.typ == rec->type) {
 			err = 0;
 			break;
 		}
@@ -540,8 +540,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -549,15 +549,14 @@ static int reader_seek_internal(struct reftable_reader *r,
 				struct reftable_iterator *it,
 				struct reftable_record *rec)
 {
-	struct reftable_reader_offsets *offs =
-		reader_offsets_for(r, reftable_record_type(rec));
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, rec->type);
 	uint64_t idx = offs->index_offset;
 	struct table_iter ti = TABLE_ITER_INIT;
 	int err = 0;
 	if (idx > 0)
 		return reader_seek_indexed(r, it, rec);
 
-	err = reader_start(r, &ti, reftable_record_type(rec), 0);
+	err = reader_start(r, &ti, rec->type, 0);
 	if (err < 0)
 		return err;
 	err = reader_seek_linear(r, &ti, rec);
@@ -576,7 +575,7 @@ static int reader_seek_internal(struct reftable_reader *r,
 static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 		       struct reftable_record *rec)
 {
-	uint8_t typ = reftable_record_type(rec);
+	uint8_t typ = rec->type;
 
 	struct reftable_reader_offsets *offs = reader_offsets_for(r, typ);
 	if (!offs->is_present) {
@@ -590,11 +589,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -602,12 +602,13 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return reader_seek(r, it, &rec);
 }
 
@@ -649,31 +650,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.obj.hash_prefix, got.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -681,15 +684,15 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.obj.offsets, got.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..b665feb709f 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -926,58 +930,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.is_deletion = &reftable_log_record_is_deletion_void,
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1055,91 +1007,47 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
-}
-
-uint8_t reftable_record_type(struct reftable_record *rec)
-{
-	return rec->ops->type;
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1210,3 +1118,53 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_record_for(uint8_t typ)
+{
+	struct reftable_record clean_idx = {
+		.type = BLOCK_TYPE_INDEX,
+		.idx = {
+			.last_key = STRBUF_INIT,
+		},
+	};
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	if (typ == BLOCK_TYPE_INDEX) {
+		return clean_idx;
+	}
+
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..a702c67e857 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -60,18 +60,9 @@ struct reftable_record_vtable {
 	int (*is_deletion)(const void *rec);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
-struct reftable_record reftable_new_record(uint8_t typ);
-
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
  * number of bytes written. */
 int reftable_encode_key(int *is_restart, struct string_view dest,
@@ -97,8 +88,26 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_record_for() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	};
+};
+
 /* see struct record_vtable */
 
+/* return an initialized record for the given type */
+struct reftable_record reftable_record_for(uint8_t typ);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
@@ -111,25 +120,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..23abfb037b3 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,24 +16,22 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = reftable_record_for(rec->type);
+
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
+	switch (copy.type) {
 	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
+		EXPECT(reftable_ref_record_equal(&copy.ref, &rec->ref,
 						 GIT_SHA1_RAWSZ));
 		break;
 	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
+		EXPECT(reftable_log_record_equal(&copy.log, &rec->log,
 						 GIT_SHA1_RAWSZ));
 		break;
 	}
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -106,61 +104,60 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_REF
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.ref.value.val2.value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.ref.value.val2.value, 1);
+			in.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.ref, &out.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -213,7 +210,9 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = {
+			.type = BLOCK_TYPE_LOG
+		};
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -221,8 +220,10 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.log = {
+				.refname = xstrdup("old name"),
 			.value_type = REFTABLE_LOG_UPDATE,
 			.value = {
 				.update = {
@@ -233,11 +234,11 @@ static void test_reftable_log_record_roundtrip(void)
 					.message = xstrdup("old message"),
 				},
 			},
+			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.log = in[i];
 
 		test_copy(&rec);
 
@@ -245,16 +246,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -322,47 +323,51 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_OBJ
+		};
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
+		EXPECT(in.obj.hash_prefix_len == out.obj.hash_prefix_len);
+		EXPECT(in.obj.offset_len == out.obj.offset_len);
 
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(!memcmp(in.obj.hash_prefix, out.obj.hash_prefix,
+			       in.obj.hash_prefix_len));
+		EXPECT(0 == memcmp(in.obj.offsets, out.obj.offsets,
+				   sizeof(uint64_t) * in.obj.offset_len));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -370,31 +375,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(in.idx.offset == out.idx.offset);
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 3ca721e9f64..df256f3a8b5 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -221,10 +221,10 @@ static int writer_add_record(struct reftable_writer *w,
 	strbuf_reset(&w->last_key);
 	strbuf_addbuf(&w->last_key, &key);
 	if (w->block_writer == NULL) {
-		writer_reinit_block_writer(w, reftable_record_type(rec));
+		writer_reinit_block_writer(w, rec->type);
 	}
 
-	assert(block_writer_type(w->block_writer) == reftable_record_type(rec));
+	assert(block_writer_type(w->block_writer) == rec->type);
 
 	if (block_writer_add(w->block_writer, rec) == 0) {
 		err = 0;
@@ -236,7 +236,7 @@ static int writer_add_record(struct reftable_writer *w,
 		goto done;
 	}
 
-	writer_reinit_block_writer(w, reftable_record_type(rec));
+	writer_reinit_block_writer(w, rec->type);
 	err = block_writer_add(w->block_writer, rec);
 	if (err < 0) {
 		goto done;
@@ -251,8 +251,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -261,8 +263,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -301,7 +302,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -311,8 +315,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -393,8 +395,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -462,17 +466,18 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_OBJ,
+		.obj = {
+			.hash_prefix = (uint8_t *)entry->hash.buf,
+			.hash_prefix_len = arg->w->stats.object_id_len,
+			.offsets = entry->offsets,
+			.offset_len = entry->offset_len,
+		}
 	};
-	struct reftable_record rec = { NULL };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -485,7 +490,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> +struct reftable_record {
> +	uint8_t type;
> +	union {
> +		struct reftable_ref_record ref;
> +		struct reftable_log_record log;
> +		struct reftable_obj_record obj;
> +		struct reftable_index_record idx;
> +	};
> +};

error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Jeff King]

On Tue, Dec 07, 2021 at 01:56:07PM -0800, Junio C Hamano wrote:

> "Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
> > +struct reftable_record {
> > +	uint8_t type;
> > +	union {
> > +		struct reftable_ref_record ref;
> > +		struct reftable_log_record log;
> > +		struct reftable_obj_record obj;
> > +		struct reftable_index_record idx;
> > +	};
> > +};
> 
> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]

Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
It's because we don't specify -std there, and newer gcc defaults to
gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
helpful to teach config.mak.dev to pass -std=c99.

Anyway, the usual fix is to stick a "u.foo" everywhere. Patch is below
for convenience. Two things to note:

  - I used ".u.foo = {...}" as a designated initializer. I think this is
    allowed by the standard, but I don't think it's something we've used
    before. If it's a problem, it can be solved by using an extra level
    of braces (which is OK, just uglier).

  - Han-Wen, if you don't just squash in the patch below, note that it
    also has a fixup for some misleading indentation in an initializer
    in test_reftable_log_record_roundtrip(). You might want to grab that
    separately.

---
 reftable/block_test.c  | 16 ++++----
 reftable/generic.c     |  8 ++--
 reftable/iter.c        |  4 +-
 reftable/merged.c      |  4 +-
 reftable/pq_test.c     |  8 ++--
 reftable/reader.c      | 26 ++++++------
 reftable/record.c      | 10 ++---
 reftable/record.h      |  2 +-
 reftable/record_test.c | 74 +++++++++++++++++------------------
 reftable/writer.c      | 12 +++---
 10 files changed, 82 insertions(+), 82 deletions(-)

diff --git a/reftable/block_test.c b/reftable/block_test.c
index 670329f222..fa2ee092ec 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		rec.ref.refname = name;
-		rec.ref.value_type = REFTABLE_REF_VAL1;
-		rec.ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		rec.ref.refname = NULL;
-		rec.ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], rec.ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,15 +92,15 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], rec.ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
 		EXPECT(n == 0);
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], rec.ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7f00aa9cdb..4446b8ed36 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -17,7 +17,7 @@ int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
 	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
-				       .ref = {
+				       .u.ref = {
 					       .refname = (char *)name,
 				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
@@ -27,7 +27,7 @@ int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
 	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
-				       .log = {
+				       .u.log = {
 					       .refname = (char *)name,
 					       .update_index = ~((uint64_t)0),
 				       } };
@@ -130,7 +130,7 @@ int reftable_iterator_next_ref(struct reftable_iterator *it,
 {
 	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
 	int err = iterator_next(it, &rec);
-	*ref = rec.ref;
+	*ref = rec.u.ref;
 	return err;
 }
 
@@ -139,7 +139,7 @@ int reftable_iterator_next_log(struct reftable_iterator *it,
 {
 	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 	int err = iterator_next(it, &rec);
-	*log = rec.log;
+	*log = rec.u.log;
 	return err;
 }
 
diff --git a/reftable/iter.c b/reftable/iter.c
index fedf5d8ab0..a8d174c040 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = &rec->ref;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = &rec->ref;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e69955d7c2..e9b0edec97 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -291,7 +291,7 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.ref = {
+		.u.ref = {
 			.refname = (char *)name,
 		},
 	};
@@ -304,7 +304,7 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.log = {
+		.u.log = {
 			.refname = (char *)name,
 			.update_index = update_index,
 		}
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index 89d069d674..9f04d84fed 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -45,7 +45,7 @@ static void test_pq(void)
 		struct pq_entry e = {
 			.rec = {
 				.type = BLOCK_TYPE_REF,
-				.ref = {
+				.u.ref = {
 					.refname = names[i],
 				}
 			}
@@ -62,11 +62,11 @@ static void test_pq(void)
 
 		EXPECT(rec->type == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, rec->ref.refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
 		// this is names[i], so don't dealloc.
-		last = rec->ref.refname;
-		rec->ref.refname = NULL;
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
 		reftable_record_release(rec);
 	}
 	for (i = 0; i < N; i++) {
diff --git a/reftable/reader.c b/reftable/reader.c
index 0159f13112..ef322791fd 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,7 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && rec->type == BLOCK_TYPE_REF) {
-		rec->ref.update_index += ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -485,17 +485,17 @@ static int reader_seek_indexed(struct reftable_reader *r,
 {
 	struct reftable_record want_index = {
 		.type = BLOCK_TYPE_INDEX,
-		.idx = { .last_key = STRBUF_INIT }
+		.u.idx = { .last_key = STRBUF_INIT }
 	};
 	struct reftable_record index_result = {
 		.type = BLOCK_TYPE_INDEX,
-		.idx = { .last_key = STRBUF_INIT },
+		.u.idx = { .last_key = STRBUF_INIT },
 	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.idx.last_key);
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 	err = reader_start(r, &index_iter, rec->type, 1);
 	if (err < 0)
 		goto done;
@@ -507,12 +507,12 @@ static int reader_seek_indexed(struct reftable_reader *r,
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.idx.offset,
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
 					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.idx.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -591,7 +591,7 @@ int reftable_reader_seek_ref(struct reftable_reader *r,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.ref = {
+		.u.ref = {
 			.refname = (char *)name,
 		},
 	};
@@ -604,7 +604,7 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.log = {
+		.u.log = {
 			.refname = (char *)name,
 			.update_index = update_index,
 		}
@@ -652,15 +652,15 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 {
 	struct reftable_record want = {
 		.type = BLOCK_TYPE_OBJ,
-		.obj = {
+		.u.obj = {
 			.hash_prefix = oid,
 			.hash_prefix_len = r->object_id_len,
 		},
 	};
 	struct reftable_iterator oit = { NULL };
 	struct reftable_record got = {
 		.type = BLOCK_TYPE_OBJ,
-		.obj = { 0 },
+		.u.obj = { 0 },
 	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
@@ -675,7 +675,7 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	if (err < 0)
 		goto done;
 
-	if (err > 0 || memcmp(want.obj.hash_prefix, got.obj.hash_prefix,
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
 			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
@@ -684,10 +684,10 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.obj.offsets, got.obj.offset_len);
+					 got.u.obj.offsets, got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.obj.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
diff --git a/reftable/record.c b/reftable/record.c
index b665feb709..68fdde9d7c 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1123,13 +1123,13 @@ static void *reftable_record_data(struct reftable_record *rec)
 {
 	switch (rec->type) {
 	case BLOCK_TYPE_REF:
-		return &rec->ref;
+		return &rec->u.ref;
 	case BLOCK_TYPE_LOG:
-		return &rec->log;
+		return &rec->u.log;
 	case BLOCK_TYPE_INDEX:
-		return &rec->idx;
+		return &rec->u.idx;
 	case BLOCK_TYPE_OBJ:
-		return &rec->obj;
+		return &rec->u.obj;
 	}
 	abort();
 }
@@ -1154,7 +1154,7 @@ struct reftable_record reftable_record_for(uint8_t typ)
 {
 	struct reftable_record clean_idx = {
 		.type = BLOCK_TYPE_INDEX,
-		.idx = {
+		.u.idx = {
 			.last_key = STRBUF_INIT,
 		},
 	};
diff --git a/reftable/record.h b/reftable/record.h
index a702c67e85..1fe0c14a19 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -101,7 +101,7 @@ struct reftable_record {
 		struct reftable_log_record log;
 		struct reftable_obj_record obj;
 		struct reftable_index_record idx;
-	};
+	} u;
 };
 
 /* see struct record_vtable */
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 23abfb037b..f234a0382a 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -23,11 +23,11 @@ static void test_copy(struct reftable_record *rec)
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	switch (copy.type) {
 	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(&copy.ref, &rec->ref,
+		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
 						 GIT_SHA1_RAWSZ));
 		break;
 	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(&copy.log, &rec->log,
+		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
 						 GIT_SHA1_RAWSZ));
 		break;
 	}
@@ -118,27 +118,27 @@ static void test_reftable_ref_record_roundtrip(void)
 		};
 		int n, m;
 
-		in.ref.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.ref.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.ref.value.val2.value =
+			in.u.ref.value.val2.value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.ref.value.val2.value, 1);
-			in.ref.value.val2.target_value =
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.ref.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.ref.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.ref.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
 		test_copy(&in);
 
@@ -152,7 +152,7 @@ static void test_reftable_ref_record_roundtrip(void)
 		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in.ref, &out.ref,
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
 						 GIT_SHA1_RAWSZ));
 		reftable_record_release(&in);
 
@@ -222,23 +222,23 @@ static void test_reftable_log_record_roundtrip(void)
 		/* populate out, to check for leaks. */
 		struct reftable_record out = {
 			.type = BLOCK_TYPE_LOG,
-			.log = {
+			.u.log = {
 				.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
-			},
 		};
 		int n, m, valtype;
 
-		rec.log = in[i];
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -251,7 +251,7 @@ static void test_reftable_log_record_roundtrip(void)
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out.log,
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
 						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
@@ -330,7 +330,7 @@ static void test_reftable_obj_record_roundtrip(void)
 		};
 		struct reftable_record in = {
 			.type = BLOCK_TYPE_OBJ,
-			.obj = recs[i],
+			.u.obj = recs[i],
 		};
 		struct strbuf key = STRBUF_INIT;
 		struct reftable_record out = {
@@ -348,13 +348,13 @@ static void test_reftable_obj_record_roundtrip(void)
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.obj.hash_prefix_len == out.obj.hash_prefix_len);
-		EXPECT(in.obj.offset_len == out.obj.offset_len);
+		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
+		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
 
-		EXPECT(!memcmp(in.obj.hash_prefix, out.obj.hash_prefix,
-			       in.obj.hash_prefix_len));
-		EXPECT(0 == memcmp(in.obj.offsets, out.obj.offsets,
-				   sizeof(uint64_t) * in.obj.offset_len));
+		EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
+			       in.u.obj.hash_prefix_len));
+		EXPECT(0 == memcmp(in.u.obj.offsets, out.u.obj.offsets,
+				   sizeof(uint64_t) * in.u.obj.offset_len));
 		strbuf_release(&key);
 		reftable_record_release(&out);
 	}
@@ -364,7 +364,7 @@ static void test_reftable_index_record_roundtrip(void)
 {
 	struct reftable_record in = {
 		.type = BLOCK_TYPE_INDEX,
-		.idx = {
+		.u.idx = {
 			.offset = 42,
 			.last_key = STRBUF_INIT,
 		},
@@ -377,28 +377,28 @@ static void test_reftable_index_record_roundtrip(void)
 	struct strbuf key = STRBUF_INIT;
 	struct reftable_record out = {
 		.type = BLOCK_TYPE_INDEX,
-		.idx = { .last_key = STRBUF_INIT },
+		.u.idx = { .last_key = STRBUF_INIT },
 	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.idx.last_key, "refs/heads/master");
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
 	reftable_record_key(&in, &key);
 	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.idx.last_key));
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
 	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
 	extra = reftable_record_val_type(&in);
 	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.idx.offset == out.idx.offset);
+	EXPECT(in.u.idx.offset == out.u.idx.offset);
 
 	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.idx.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index df256f3a8b..f9544c5f97 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -253,7 +253,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.ref = *ref,
+		.u.ref = *ref,
 	};
 	int err = 0;
 
@@ -263,7 +263,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	rec.ref.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -304,7 +304,7 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.log = *log,
+		.u.log = *log,
 	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
@@ -397,7 +397,7 @@ static int writer_finish_section(struct reftable_writer *w)
 		for (i = 0; i < idx_len; i++) {
 			struct reftable_record rec = {
 				.type = BLOCK_TYPE_INDEX,
-				.idx = idx[i],
+				.u.idx = idx[i],
 			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
@@ -468,7 +468,7 @@ static void write_object_record(void *void_arg, void *key)
 	struct obj_index_tree_node *entry = key;
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_OBJ,
-		.obj = {
+		.u.obj = {
 			.hash_prefix = (uint8_t *)entry->hash.buf,
 			.hash_prefix_len = arg->w->stats.object_id_len,
 			.offsets = entry->offsets,
@@ -491,7 +491,7 @@ static void write_object_record(void *void_arg, void *key)
 	if (arg->err == 0)
 		goto done;
 
-	rec.obj.offset_len = 0;
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Junio C Hamano]

Jeff King <peff@peff.net> writes:

>> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]
>
> Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
> It's because we don't specify -std there, and newer gcc defaults to
> gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
> helpful to teach config.mak.dev to pass -std=c99.

FWIW, I use -std=gnu99 as our Makefile suggests.

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Wed, Dec 8, 2021 at 5:13 AM Junio C Hamano <gitster@pobox.com> wrote:
> >> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]
> >
> > Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
> > It's because we don't specify -std there, and newer gcc defaults to
> > gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
> > helpful to teach config.mak.dev to pass -std=c99.
>
> FWIW, I use -std=gnu99 as our Makefile suggests.

I understand that the default build should be lenient rather than
strict for portability reasons. However, it would be good if the CI
was strict with this.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH 02/10] reftable: fix resource leak in error path

[Derrick Stolee]

On 12/7/2021 12:45 PM, Han-Wen Nienhuys via GitGitGadget wrote:
> From: Han-Wen Nienhuys <hanwen@google.com>
> 
> This would be triggered by corrupt files, so it doesn't have test coverage. This
> was discovered by a Coverity scan.
> 
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/block.c | 25 ++++++++++++++++++-------
>  1 file changed, 18 insertions(+), 7 deletions(-)
> 
> diff --git a/reftable/block.c b/reftable/block.c
> index 855e3f5c947..d7347bb3152 100644
> --- a/reftable/block.c
> +++ b/reftable/block.c
> @@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
>  	uint32_t full_block_size = table_block_size;
>  	uint8_t typ = block->data[header_off];
>  	uint32_t sz = get_be24(block->data + header_off + 1);
> -
> +	int err = 0;
>  	uint16_t restart_count = 0;
>  	uint32_t restart_start = 0;
>  	uint8_t *restart_bytes = NULL;
> +	uint8_t *uncompressed = NULL;

You define this here...

>  
> -	if (!reftable_is_block_type(typ))
> -		return REFTABLE_FORMAT_ERROR;
> +	if (!reftable_is_block_type(typ)) {
> +		err =  REFTABLE_FORMAT_ERROR;
> +		goto done;
> +	}
>  
>  	if (typ == BLOCK_TYPE_LOG) {
>  		int block_header_skip = 4 + header_off;
> @@ -213,15 +216,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
>  		    uncompress2(uncompressed + block_header_skip, &dst_len,
>  				block->data + block_header_skip, &src_len)) {
>  			reftable_free(uncompressed);

But it is already used here, because it is defined within the if()
block. You need to remove that definition, too, or your change here
does nothing.

...

>  		/* We're done with the input data. */
>  		reftable_block_done(block);
>  		block->data = uncompressed;
> +		uncompressed = NULL;

For example, this nullifies the local version of uncompressed.

(I had to double-check that this even compiled, which was
surprising to me.)

> +done:
> +	if (uncompressed) {
> +		reftable_free(uncompressed);
> +	}
> +	return err;

Remove that local declaration and this will be a good patch.

Thanks,
-Stolee

Re: [PATCH 08/10] reftable: order unittests by complexity

[Derrick Stolee]

On 12/7/2021 12:45 PM, Han-Wen Nienhuys via GitGitGadget wrote:
> From: Han-Wen Nienhuys <hanwen@google.com>
...
>  int cmd__reftable(int argc, const char **argv)
>  {
> +	// test from simple to complex.

Use /* ... */ style comments, even for a single line.

Thanks,
-Stolee

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Derrick Stolee]

On 12/7/2021 12:45 PM, Han-Wen Nienhuys via GitGitGadget wrote:
> From: Han-Wen Nienhuys <hanwen@google.com>
> 
> This reduces the amount of glue code, because we don't need a void pointer or
> vtable within the structure.
> 
> The only snag is that reftable_index_record contain a strbuf, so it cannot be
> zero-initialized. To address this, introduce reftable_record_for() to create a
> fresh instance, given a record type.
> 
> Thanks to Peff for the suggestion.
> 
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/block.c       |   4 +-
>  reftable/block_test.c  |  22 +++---
>  reftable/generic.c     |  35 ++++----
>  reftable/iter.c        |   4 +-
>  reftable/merged.c      |  37 ++++-----
>  reftable/pq.c          |   3 +-
>  reftable/pq_test.c     |  31 ++++----
>  reftable/reader.c      | 105 ++++++++++++------------
>  reftable/record.c      | 176 ++++++++++++++++-------------------------
>  reftable/record.h      |  45 +++++------
>  reftable/record_test.c | 162 +++++++++++++++++++------------------
>  reftable/writer.c      |  46 ++++++-----

This is a HUGE diff, especially compared to the previous changes
in this series. I recommend splitting this out into its own series
and finding a way to break it down into smaller changes.

Thanks,
-Stolee

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Wed, Dec 8, 2021 at 3:35 PM Derrick Stolee <stolee@gmail.com> wrote:
> This is a HUGE diff, especially compared to the previous changes
> in this series. I recommend splitting this out into its own series
> and finding a way to break it down into smaller changes.

Would you have a suggestion how? The reftable_record type is used
across the reftable library, so if we change its definition, that
impacts most callsites.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@google.com> writes:

> On Wed, Dec 8, 2021 at 5:13 AM Junio C Hamano <gitster@pobox.com> wrote:
>> >> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]
>> >
>> > Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
>> > It's because we don't specify -std there, and newer gcc defaults to
>> > gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
>> > helpful to teach config.mak.dev to pass -std=c99.
>>
>> FWIW, I use -std=gnu99 as our Makefile suggests.
>
> I understand that the default build should be lenient rather than
> strict for portability reasons. However, it would be good if the CI
> was strict with this.

Yeah, I agree with the above, and was a bit surprised that I found
the issue via my local build after it came from GGG.

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Junio C Hamano]

Derrick Stolee <stolee@gmail.com> writes:

> On 12/7/2021 12:45 PM, Han-Wen Nienhuys via GitGitGadget wrote:
>> From: Han-Wen Nienhuys <hanwen@google.com>
>> 
>> This reduces the amount of glue code, because we don't need a void pointer or
>> vtable within the structure.
>> 
>> The only snag is that reftable_index_record contain a strbuf, so it cannot be
>> zero-initialized. To address this, introduce reftable_record_for() to create a
>> fresh instance, given a record type.
>> 
>> Thanks to Peff for the suggestion.
>> 
>> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
>> ---
>>  reftable/block.c       |   4 +-
>>  reftable/block_test.c  |  22 +++---
>>  reftable/generic.c     |  35 ++++----
>>  reftable/iter.c        |   4 +-
>>  reftable/merged.c      |  37 ++++-----
>>  reftable/pq.c          |   3 +-
>>  reftable/pq_test.c     |  31 ++++----
>>  reftable/reader.c      | 105 ++++++++++++------------
>>  reftable/record.c      | 176 ++++++++++++++++-------------------------
>>  reftable/record.h      |  45 +++++------
>>  reftable/record_test.c | 162 +++++++++++++++++++------------------
>>  reftable/writer.c      |  46 ++++++-----
>
> This is a HUGE diff, especially compared to the previous changes
> in this series. I recommend splitting this out into its own series
> and finding a way to break it down into smaller changes.

As the reftable_record structure is used everywhere (and that is why
this step has to touch everywhere), I suspect that a reviewable fix
in small chunks would be achievable only if we redo the topic that
introduces this hierarchy and fix the type at the source, as if the
reftable_record structure was a struct with union in it from the
beginning, I am afraid.

Perhaps reftable_record_for() can be implemented without changing
the shape of the underlying reftable_record structure in an earlier
step, then all the users of reftable_record instances can be
migrated to call it, and then finally the shape of the structure and
the implementation of reftable_record_for() can be updated?  

If that is doable, then the "migrate each users" part can be split
purely by size.  But (1) I do not know if the first step is even
doable, and (2) I am not sure if it is worth going a somewhat
roundabout route to get to the same destination in this case.

So...

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Wed, Dec 8, 2021 at 5:47 PM Junio C Hamano <gitster@pobox.com> wrote:
> Perhaps reftable_record_for() can be implemented without changing
> the shape of the underlying reftable_record structure in an earlier
> step, then all the users of reftable_record instances can be
> migrated to call it, and then finally the shape of the structure and
> the implementation of reftable_record_for() can be updated?

We are going from

 reftable_ref_record ref = { 0 };
 reftable_record rec =  { 0 };
 record_from_ref_record(&rec, &ref);

to

 reftable_record rec =  { .type = BLOCK_TYPE_REF };

which is why this change is so nice.  reftable_record_for() is used in
just a few places, so it wouldn't make the change much smaller.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Derrick Stolee]

On 12/8/2021 9:48 AM, Han-Wen Nienhuys wrote:
> On Wed, Dec 8, 2021 at 3:35 PM Derrick Stolee <stolee@gmail.com> wrote:
>> This is a HUGE diff, especially compared to the previous changes
>> in this series. I recommend splitting this out into its own series
>> and finding a way to break it down into smaller changes.
> 
> Would you have a suggestion how? The reftable_record type is used
> across the reftable library, so if we change its definition, that
> impacts most callsites.
 
Looking at the diff, I'm seeing a lot of things that don't seem
necessary to changing the struct, like renaming "reftable_new_record"
to "reftable_record_for". Yes, the implementation changes, but that
implementation is related to the struct changing and can be done in
a more isolated way.

Some things can be split out easier, such as the addition of
"uint8_t type" into the struct and replacing all reftable_record_type()
calls with <var>.type.

The big one is definitely the addition of a union, but it could be
done in a way that only changes the implementation of methods with
names such as reftable_record_from_obj() then we can replace those with
"<var>.obj" later.

Some of these changes are definitely going to have a lot of lines,
but when it's the same kind of change it is easy to verify. I'm
getting lost in this diff because it's doing too many things at once.

Thanks,
-Stolee

[PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang

[Jeff King]

On Tue, Dec 07, 2021 at 08:13:43PM -0800, Junio C Hamano wrote:

> Jeff King <peff@peff.net> writes:
> 
> >> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]
> >
> > Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
> > It's because we don't specify -std there, and newer gcc defaults to
> > gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
> > helpful to teach config.mak.dev to pass -std=c99.
> 
> FWIW, I use -std=gnu99 as our Makefile suggests.

I think the patch below would have detected this both locally for
Han-Wen, but also in C.

-- >8 --
Subject: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang

The point of DEVELOPER=1 is to turn up the warnings so we can catch
portability or correctness mistakes at the compiler level. But since
modern compilers tend to default to modern standards like gnu17, we
might miss warnings about older standards, even though we expect Git to
build with compilers that use them.

So it's helpful for developer builds to set the -std argument to our
lowest-common denominator. Traditionally this was c89, but since we're
moving to assuming c99 in 7bc341e21b (git-compat-util: add a test
balloon for C99 support, 2021-12-01) that seems like a good spot to
land. And as explained in that commit, we want "gnu99" because we still
want to take advantage of some extensions when they're available.

The new argument kicks in only for clang and gcc (which we know to
support "-std=" and "gnu" standards). And only for compiler versions
which default to a newer standard. That will avoid accidentally
silencing any build problems that non-developers would run into on older
compilers that default to c89.

My digging found that the default switched to gnu11 in gcc 5.1.0.
Clang's documentation is less clear, but has done so since at least
clang-7. So that's what I put in the conditional here. It's OK to err on
the side of not-enabling this for older compilers. Most developers (as
well as CI) are using much more recent versions, so any warnings will
eventually surface.

A concrete example is anonymous unions, which became legal in c11.
Without this patch, "gcc -pedantic" will not complain about them, but
will if we add in "-std=gnu99".

Signed-off-by: Jeff King <peff@peff.net>
---
 config.mak.dev | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/config.mak.dev b/config.mak.dev
index 7673fed114..d4afac6b51 100644
--- a/config.mak.dev
+++ b/config.mak.dev
@@ -19,6 +19,11 @@ endif
 endif
 endif
 endif
+
+ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
+DEVELOPER_CFLAGS += -std=gnu99
+endif
+
 DEVELOPER_CFLAGS += -Wdeclaration-after-statement
 DEVELOPER_CFLAGS += -Wformat-security
 DEVELOPER_CFLAGS += -Wold-style-definition
-- 
2.34.1.503.gbf394de69d

[PATCH v2 00/11] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'next'.

This addresses some complaints from Coverity that Peff reported.

CI on GGG shows a segfault on windows
(https://github.com/git/git/runs/4446635428?check_suite_focus=true), which I
can't make sense of. Any hints would be greatly appreciated.

Han-Wen Nienhuys (11):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: fix NULL derefs in error paths
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable_record a tagged union

 reftable/block.c          |  31 +++++--
 reftable/block_test.c     |  22 ++---
 reftable/blocksource.c    |   6 +-
 reftable/generic.c        |  35 +++----
 reftable/iter.c           |   4 +-
 reftable/merged.c         |  37 ++++----
 reftable/pq.c             |   3 +-
 reftable/pq_test.c        |  31 ++++---
 reftable/reader.c         | 108 +++++++++++-----------
 reftable/readwrite_test.c |   1 -
 reftable/record.c         | 188 +++++++++++++++-----------------------
 reftable/record.h         |  45 ++++-----
 reftable/record_test.c    | 181 ++++++++++++++++++------------------
 reftable/stack.c          |  15 +--
 reftable/stack_test.c     |   3 +-
 reftable/writer.c         |  46 ++++++----
 t/helper/test-reftable.c  |   9 +-
 17 files changed, 376 insertions(+), 389 deletions(-)


base-commit: a8338297339b1ab064e799c15c4fc56c122ef967
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v2
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v1:

  1:  7c033815183 =  1:  7c033815183 reftable: fix OOB stack write in print functions
  2:  603bd1d4f6e !  2:  1ddcfe61ebc reftable: fix resource leak in error path
     @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
       
       	if (typ == BLOCK_TYPE_LOG) {
       		int block_header_skip = 4 + header_off;
     +@@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,
     + 		uLongf src_len = block->len - block_header_skip;
     + 		/* Log blocks specify the *uncompressed* size in their header.
     + 		 */
     +-		uint8_t *uncompressed = reftable_malloc(sz);
     ++		uncompressed = reftable_malloc(sz);
     + 
     + 		/* Copy over the block header verbatim. It's not compressed. */
     + 		memcpy(uncompressed, block->data, block_header_skip);
      @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,
       		    uncompress2(uncompressed + block_header_skip, &dst_len,
       				block->data + block_header_skip, &src_len)) {
  3:  97279040dde =  3:  e052b2a61d6 reftable: fix resource leak blocksource.c
  4:  9288289ebcb =  4:  9063137457b reftable: check reftable_stack_auto_compact() return value
  5:  9269977a1c3 =  5:  5020be156ae reftable: ignore remove() return value in stack_test.c
  6:  eab71ab008e =  6:  64c18d01cad reftable: fix resource warning
  7:  47fef87eb4a =  7:  700387ac5d3 reftable: fix NULL derefs in error paths
  8:  458adf0fa9b =  8:  713f1d09f68 reftable: order unittests by complexity
  9:  cd5d8d3607d =  9:  cb601b51a47 reftable: drop stray printf in readwrite_test
  -:  ----------- > 10:  a0f83eff19f reftable: handle null refnames in reftable_ref_record_equal
 10:  8deccc3a1df ! 11:  a2743033cfd reftable: make reftable_record a tagged union
     @@ Commit message
      
          Thanks to Peff for the suggestion.
      
     +    Helped-by: Jeff King <peff@peff.net>
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
       ## reftable/block.c ##
     @@ reftable/block_test.c: static void test_block_read_write(void)
      -		ref.refname = name;
      -		ref.value_type = REFTABLE_REF_VAL1;
      -		ref.value.val1 = hash;
     -+		rec.ref.refname = name;
     -+		rec.ref.value_type = REFTABLE_REF_VAL1;
     -+		rec.ref.value.val1 = hash;
     ++		rec.u.ref.refname = name;
     ++		rec.u.ref.value_type = REFTABLE_REF_VAL1;
     ++		rec.u.ref.value.val1 = hash;
       
       		names[i] = xstrdup(name);
       		n = block_writer_add(&bw, &rec);
      -		ref.refname = NULL;
      -		ref.value_type = REFTABLE_REF_DELETION;
     -+		rec.ref.refname = NULL;
     -+		rec.ref.value_type = REFTABLE_REF_DELETION;
     ++		rec.u.ref.refname = NULL;
     ++		rec.u.ref.value_type = REFTABLE_REF_DELETION;
       		EXPECT(n == 0);
       	}
       
     @@ reftable/block_test.c: static void test_block_read_write(void)
       			break;
       		}
      -		EXPECT_STREQ(names[j], ref.refname);
     -+		EXPECT_STREQ(names[j], rec.ref.refname);
     ++		EXPECT_STREQ(names[j], rec.u.ref.refname);
       		j++;
       	}
       
     @@ reftable/block_test.c: static void test_block_read_write(void)
       		EXPECT(n == 0);
       
      -		EXPECT_STREQ(names[i], ref.refname);
     -+		EXPECT_STREQ(names[i], rec.ref.refname);
     ++		EXPECT_STREQ(names[i], rec.u.ref.refname);
       
       		want.len--;
       		n = block_reader_seek(&br, &it, &want);
     @@ reftable/block_test.c: static void test_block_read_write(void)
       		n = block_iter_next(&it, &rec);
       		EXPECT(n == 0);
      -		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
     -+		EXPECT_STREQ(names[10 * (i / 10)], rec.ref.refname);
     ++		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
       
       		block_iter_close(&it);
       	}
     @@ reftable/generic.c: https://developers.google.com/open-source/licenses/bsd
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_ref(&rec, &ref);
      +	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
     -+				       .ref = {
     ++				       .u.ref = {
      +					       .refname = (char *)name,
      +				       } };
       	return tab->ops->seek_record(tab->table_arg, it, &rec);
     @@ reftable/generic.c: https://developers.google.com/open-source/licenses/bsd
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
      +	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     -+				       .log = {
     ++				       .u.log = {
      +					       .refname = (char *)name,
      +					       .update_index = ~((uint64_t)0),
      +				       } };
     @@ reftable/generic.c: void reftable_iterator_destroy(struct reftable_iterator *it)
      -	return iterator_next(it, &rec);
      +	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
      +	int err = iterator_next(it, &rec);
     -+	*ref = rec.ref;
     ++	*ref = rec.u.ref;
      +	return err;
       }
       
     @@ reftable/generic.c: void reftable_iterator_destroy(struct reftable_iterator *it)
      -	return iterator_next(it, &rec);
      +	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
      +	int err = iterator_next(it, &rec);
     -+	*log = rec.log;
     ++	*log = rec.u.log;
      +	return err;
       }
       
     @@ reftable/iter.c: static int filtering_ref_iterator_next(void *iter_arg,
       {
       	struct filtering_ref_iterator *fri = iter_arg;
      -	struct reftable_ref_record *ref = rec->data;
     -+	struct reftable_ref_record *ref = &rec->ref;
     ++	struct reftable_ref_record *ref = &rec->u.ref;
       	int err = 0;
       	while (1) {
       		err = reftable_iterator_next_ref(&fri->it, ref);
     @@ reftable/iter.c: static int indexed_table_ref_iter_next_block(struct indexed_tab
       {
       	struct indexed_table_ref_iter *it = p;
      -	struct reftable_ref_record *ref = rec->data;
     -+	struct reftable_ref_record *ref = &rec->ref;
     ++	struct reftable_ref_record *ref = &rec->u.ref;
       
       	while (1) {
       		int err = block_iter_next(&it->cur, rec);
     @@ reftable/merged.c: int reftable_merged_table_seek_ref(struct reftable_merged_tab
      -		.refname = (char *)name,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_REF,
     -+		.ref = {
     ++		.u.ref = {
      +			.refname = (char *)name,
      +		},
       	};
     @@ reftable/merged.c: int reftable_merged_table_seek_log_at(struct reftable_merged_
      -		.update_index = update_index,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_LOG,
     -+		.log = {
     ++		.u.log = {
      +			.refname = (char *)name,
      +			.update_index = update_index,
      +		}
     @@ reftable/pq_test.c: static void test_pq(void)
      +		struct pq_entry e = {
      +			.rec = {
      +				.type = BLOCK_TYPE_REF,
     -+				.ref = {
     ++				.u.ref = {
      +					.refname = names[i],
      +				}
      +			}
     @@ reftable/pq_test.c: static void test_pq(void)
      +		EXPECT(rec->type == BLOCK_TYPE_REF);
       		if (last) {
      -			EXPECT(strcmp(last, ref->refname) < 0);
     -+			EXPECT(strcmp(last, rec->ref.refname) < 0);
     ++			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
       		}
      -		last = ref->refname;
      -		ref->refname = NULL;
      -		reftable_free(ref);
      +		// this is names[i], so don't dealloc.
     -+		last = rec->ref.refname;
     -+		rec->ref.refname = NULL;
     ++		last = rec->u.ref.refname;
     ++		rec->u.ref.refname = NULL;
      +		reftable_record_release(rec);
       	}
      -
     @@ reftable/reader.c: static int table_iter_next_in_block(struct table_iter *ti,
      -		((struct reftable_ref_record *)rec->data)->update_index +=
      -			ti->r->min_update_index;
      +	if (res == 0 && rec->type == BLOCK_TYPE_REF) {
     -+		rec->ref.update_index += ti->r->min_update_index;
     ++		rec->u.ref.update_index += ti->r->min_update_index;
       	}
       
       	return res;
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
      -	struct reftable_record index_result_rec = { NULL };
      +	struct reftable_record want_index = {
      +		.type = BLOCK_TYPE_INDEX,
     -+		.idx = { .last_key = STRBUF_INIT }
     ++		.u.idx = { .last_key = STRBUF_INIT }
      +	};
      +	struct reftable_record index_result = {
      +		.type = BLOCK_TYPE_INDEX,
     -+		.idx = { .last_key = STRBUF_INIT },
     ++		.u.idx = { .last_key = STRBUF_INIT },
      +	};
       	struct table_iter index_iter = TABLE_ITER_INIT;
       	struct table_iter next = TABLE_ITER_INIT;
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
      -	reftable_record_from_index(&index_result_rec, &index_result);
      -
      -	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
     -+	reftable_record_key(rec, &want_index.idx.last_key);
     ++	reftable_record_key(rec, &want_index.u.idx.last_key);
      +	err = reader_start(r, &index_iter, rec->type, 1);
       	if (err < 0)
       		goto done;
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
       			goto done;
       
      -		err = reader_table_iter_at(r, &next, index_result.offset, 0);
     -+		err = reader_table_iter_at(r, &next, index_result.idx.offset,
     ++		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
      +					   0);
       		if (err != 0)
       			goto done;
       
      -		err = block_iter_seek(&next.bi, &want_index.last_key);
     -+		err = block_iter_seek(&next.bi, &want_index.idx.last_key);
     ++		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
       		if (err < 0)
       			goto done;
       
     @@ reftable/reader.c: static int reader_seek(struct reftable_reader *r, struct reft
      -		.refname = (char *)name,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_REF,
     -+		.ref = {
     ++		.u.ref = {
      +			.refname = (char *)name,
      +		},
       	};
     @@ reftable/reader.c: int reftable_reader_seek_log_at(struct reftable_reader *r,
      -		.update_index = update_index,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_LOG,
     -+		.log = {
     ++		.u.log = {
      +			.refname = (char *)name,
      +			.update_index = update_index,
      +		}
     @@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_r
      -		.hash_prefix_len = r->object_id_len,
      +	struct reftable_record want = {
      +		.type = BLOCK_TYPE_OBJ,
     -+		.obj = {
     ++		.u.obj = {
      +			.hash_prefix = oid,
      +			.hash_prefix_len = r->object_id_len,
      +		},
     @@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_r
      -	struct reftable_record got_rec = { NULL };
      +	struct reftable_record got = {
      +		.type = BLOCK_TYPE_OBJ,
     -+		.obj = { 0 },
     ++		.u.obj = { 0 },
      +	};
       	int err = 0;
       	struct indexed_table_ref_iter *itr = NULL;
     @@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_r
       
      -	if (err > 0 ||
      -	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
     -+	if (err > 0 || memcmp(want.obj.hash_prefix, got.obj.hash_prefix,
     ++	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
      +			      r->object_id_len)) {
       		/* didn't find it; return empty iterator */
       		iterator_set_empty(it);
     @@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_r
       
       	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
      -					 got.offsets, got.offset_len);
     -+					 got.obj.offsets, got.obj.offset_len);
     ++					 got.u.obj.offsets, got.u.obj.offset_len);
       	if (err < 0)
       		goto done;
      -	got.offsets = NULL;
     -+	got.obj.offsets = NULL;
     ++	got.u.obj.offsets = NULL;
       	iterator_from_indexed_table_ref_iter(it, itr);
       
       done:
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
      +{
      +	switch (rec->type) {
      +	case BLOCK_TYPE_REF:
     -+		return &rec->ref;
     ++		return &rec->u.ref;
      +	case BLOCK_TYPE_LOG:
     -+		return &rec->log;
     ++		return &rec->u.log;
      +	case BLOCK_TYPE_INDEX:
     -+		return &rec->idx;
     ++		return &rec->u.idx;
      +	case BLOCK_TYPE_OBJ:
     -+		return &rec->obj;
     ++		return &rec->u.obj;
      +	}
      +	abort();
      +}
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
      +{
      +	struct reftable_record clean_idx = {
      +		.type = BLOCK_TYPE_INDEX,
     -+		.idx = {
     ++		.u.idx = {
      +			.last_key = STRBUF_INIT,
      +		},
      +	};
     @@ reftable/record.h: struct reftable_obj_record {
      +		struct reftable_log_record log;
      +		struct reftable_obj_record obj;
      +		struct reftable_index_record idx;
     -+	};
     ++	} u;
      +};
      +
       /* see struct record_vtable */
     @@ reftable/record_test.c
       	case BLOCK_TYPE_REF:
      -		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
      -						 reftable_record_as_ref(rec),
     -+		EXPECT(reftable_ref_record_equal(&copy.ref, &rec->ref,
     ++		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
       						 GIT_SHA1_RAWSZ));
       		break;
       	case BLOCK_TYPE_LOG:
      -		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
      -						 reftable_record_as_log(rec),
     -+		EXPECT(reftable_log_record_equal(&copy.log, &rec->log,
     ++		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
       						 GIT_SHA1_RAWSZ));
       		break;
       	}
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
       		int n, m;
       
      -		in.value_type = i;
     -+		in.ref.value_type = i;
     ++		in.u.ref.value_type = i;
       		switch (i) {
       		case REFTABLE_REF_DELETION:
       			break;
       		case REFTABLE_REF_VAL1:
      -			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
      -			set_hash(in.value.val1, 1);
     -+			in.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
     -+			set_hash(in.ref.value.val1, 1);
     ++			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
     ++			set_hash(in.u.ref.value.val1, 1);
       			break;
       		case REFTABLE_REF_VAL2:
      -			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
      -			set_hash(in.value.val2.value, 1);
      -			in.value.val2.target_value =
     -+			in.ref.value.val2.value =
     ++			in.u.ref.value.val2.value =
      +				reftable_malloc(GIT_SHA1_RAWSZ);
     -+			set_hash(in.ref.value.val2.value, 1);
     -+			in.ref.value.val2.target_value =
     ++			set_hash(in.u.ref.value.val2.value, 1);
     ++			in.u.ref.value.val2.target_value =
       				reftable_malloc(GIT_SHA1_RAWSZ);
      -			set_hash(in.value.val2.target_value, 2);
     -+			set_hash(in.ref.value.val2.target_value, 2);
     ++			set_hash(in.u.ref.value.val2.target_value, 2);
       			break;
       		case REFTABLE_REF_SYMREF:
      -			in.value.symref = xstrdup("target");
     -+			in.ref.value.symref = xstrdup("target");
     ++			in.u.ref.value.symref = xstrdup("target");
       			break;
       		}
      -		in.refname = xstrdup("refs/heads/master");
     -+		in.ref.refname = xstrdup("refs/heads/master");
     ++		in.u.ref.refname = xstrdup("refs/heads/master");
       
      -		reftable_record_from_ref(&rec, &in);
      -		test_copy(&rec);
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
       
      -		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
      -		reftable_record_release(&rec_out);
     -+		EXPECT(reftable_ref_record_equal(&in.ref, &out.ref,
     ++		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
      +						 GIT_SHA1_RAWSZ));
      +		reftable_record_release(&in);
       
     @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
       		/* populate out, to check for leaks. */
      -		struct reftable_log_record out = {
      -			.refname = xstrdup("old name"),
     +-			.value_type = REFTABLE_LOG_UPDATE,
     +-			.value = {
     +-				.update = {
     +-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
     +-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
     +-					.name = xstrdup("old name"),
     +-					.email = xstrdup("old@email"),
     +-					.message = xstrdup("old message"),
      +		struct reftable_record out = {
      +			.type = BLOCK_TYPE_LOG,
     -+			.log = {
     ++			.u.log = {
      +				.refname = xstrdup("old name"),
     - 			.value_type = REFTABLE_LOG_UPDATE,
     - 			.value = {
     - 				.update = {
     -@@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
     - 					.message = xstrdup("old message"),
     ++				.value_type = REFTABLE_LOG_UPDATE,
     ++				.value = {
     ++					.update = {
     ++						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
     ++						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
     ++						.name = xstrdup("old name"),
     ++						.email = xstrdup("old@email"),
     ++						.message = xstrdup("old message"),
     ++					},
       				},
       			},
     -+			},
       		};
      -		struct reftable_record rec_out = { NULL };
       		int n, m, valtype;
       
      -		reftable_record_from_log(&rec, &in[i]);
     -+		rec.log = in[i];
     ++		rec.u.log = in[i];
       
       		test_copy(&rec);
       
     @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
       		EXPECT(n == m);
       
      -		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
     -+		EXPECT(reftable_log_record_equal(&in[i], &out.log,
     ++		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
      +						 GIT_SHA1_RAWSZ));
       		reftable_log_record_release(&in[i]);
       		strbuf_release(&key);
     @@ reftable/record_test.c: static void test_reftable_obj_record_roundtrip(void)
      -		struct reftable_record rec = { NULL };
      +		struct reftable_record in = {
      +			.type = BLOCK_TYPE_OBJ,
     -+			.obj = recs[i],
     ++			.u.obj = recs[i],
      +		};
       		struct strbuf key = STRBUF_INIT;
      -		struct reftable_obj_record out = { NULL };
     @@ reftable/record_test.c: static void test_reftable_obj_record_roundtrip(void)
       
      -		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
      -		EXPECT(in.offset_len == out.offset_len);
     -+		EXPECT(in.obj.hash_prefix_len == out.obj.hash_prefix_len);
     -+		EXPECT(in.obj.offset_len == out.obj.offset_len);
     - 
     +-
      -		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
      -			       in.hash_prefix_len));
      -		EXPECT(0 == memcmp(in.offsets, out.offsets,
      -				   sizeof(uint64_t) * in.offset_len));
     -+		EXPECT(!memcmp(in.obj.hash_prefix, out.obj.hash_prefix,
     -+			       in.obj.hash_prefix_len));
     -+		EXPECT(0 == memcmp(in.obj.offsets, out.obj.offsets,
     -+				   sizeof(uint64_t) * in.obj.offset_len));
     ++		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
     ++		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
     ++		if (in.u.obj.hash_prefix_len)
     ++			EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
     ++				       in.u.obj.hash_prefix_len));
     ++		if (in.u.obj.offset_len)
     ++			EXPECT(!memcmp(in.u.obj.offsets, out.u.obj.offsets,
     ++					   sizeof(uint64_t) * in.u.obj.offset_len));
       		strbuf_release(&key);
      -		reftable_record_release(&rec_out);
      +		reftable_record_release(&out);
     @@ reftable/record_test.c: static void test_reftable_obj_record_roundtrip(void)
      -		.last_key = STRBUF_INIT,
      +	struct reftable_record in = {
      +		.type = BLOCK_TYPE_INDEX,
     -+		.idx = {
     ++		.u.idx = {
      +			.offset = 42,
      +			.last_key = STRBUF_INIT,
      +		},
     @@ reftable/record_test.c: static void test_reftable_index_record_roundtrip(void)
      -	struct reftable_record out_rec = { NULL };
      +	struct reftable_record out = {
      +		.type = BLOCK_TYPE_INDEX,
     -+		.idx = { .last_key = STRBUF_INIT },
     ++		.u.idx = { .last_key = STRBUF_INIT },
      +	};
       	int n, m;
       	uint8_t extra;
     @@ reftable/record_test.c: static void test_reftable_index_record_roundtrip(void)
      -	reftable_record_from_index(&rec, &in);
      -	reftable_record_key(&rec, &key);
      -	test_copy(&rec);
     -+	strbuf_addstr(&in.idx.last_key, "refs/heads/master");
     ++	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
      +	reftable_record_key(&in, &key);
      +	test_copy(&in);
       
      -	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
      -	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
     -+	EXPECT(0 == strbuf_cmp(&key, &in.idx.last_key));
     ++	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
      +	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
       	EXPECT(n > 0);
       
     @@ reftable/record_test.c: static void test_reftable_index_record_roundtrip(void)
       	EXPECT(m == n);
       
      -	EXPECT(in.offset == out.offset);
     -+	EXPECT(in.idx.offset == out.idx.offset);
     ++	EXPECT(in.u.idx.offset == out.u.idx.offset);
       
      -	reftable_record_release(&out_rec);
      +	reftable_record_release(&out);
       	strbuf_release(&key);
      -	strbuf_release(&in.last_key);
     -+	strbuf_release(&in.idx.last_key);
     ++	strbuf_release(&in.u.idx.last_key);
       }
       
       int record_test_main(int argc, const char *argv[])
     @@ reftable/writer.c: done:
      -	struct reftable_ref_record copy = *ref;
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_REF,
     -+		.ref = *ref,
     ++		.u.ref = *ref,
      +	};
       	int err = 0;
       
     @@ reftable/writer.c: int reftable_writer_add_ref(struct reftable_writer *w,
       
      -	reftable_record_from_ref(&rec, &copy);
      -	copy.update_index -= w->min_update_index;
     -+	rec.ref.update_index -= w->min_update_index;
     ++	rec.u.ref.update_index -= w->min_update_index;
       
       	err = writer_add_record(w, &rec);
       	if (err < 0)
     @@ reftable/writer.c: int reftable_writer_add_refs(struct reftable_writer *w,
      -	struct reftable_record rec = { NULL };
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_LOG,
     -+		.log = *log,
     ++		.u.log = *log,
      +	};
       	if (w->block_writer &&
       	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
     @@ reftable/writer.c: static int writer_finish_section(struct reftable_writer *w)
      -			reftable_record_from_index(&rec, idx + i);
      +			struct reftable_record rec = {
      +				.type = BLOCK_TYPE_INDEX,
     -+				.idx = idx[i],
     ++				.u.idx = idx[i],
      +			};
       			if (block_writer_add(w->block_writer, &rec) == 0) {
       				continue;
     @@ reftable/writer.c: static void write_object_record(void *void_arg, void *key)
      -		.offset_len = entry->offset_len,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_OBJ,
     -+		.obj = {
     ++		.u.obj = {
      +			.hash_prefix = (uint8_t *)entry->hash.buf,
      +			.hash_prefix_len = arg->w->stats.object_id_len,
      +			.offsets = entry->offsets,
     @@ reftable/writer.c: static void write_object_record(void *void_arg, void *key)
       		goto done;
      -	obj_rec.offset_len = 0;
      +
     -+	rec.obj.offset_len = 0;
     ++	rec.u.obj.offset_len = 0;
       	arg->err = block_writer_add(arg->w->block_writer, &rec);
       
       	/* Should be able to write into a fresh block. */

-- 
gitgitgadget

[PATCH v2 01/11] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v2 02/11] reftable: fix resource leak in error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered by corrupt files, so it doesn't have test coverage. This
was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c | 27 +++++++++++++++++++--------
 1 file changed, 19 insertions(+), 8 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..79b6f3aac72 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +206,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -213,15 +216,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
 			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +258,11 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	if (uncompressed) {
+		reftable_free(uncompressed);
+	}
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
-- 
gitgitgadget

[PATCH v2 03/11] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an opened
file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v2 04/11] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index eb0b7228b0c..d628420e63a 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -814,6 +814,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v2 05/11] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index d628420e63a..4b7292945c3 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -89,7 +89,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v2 06/11] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and the next
available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index df5021ebf08..10dfd370e8e 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -877,7 +877,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -911,7 +911,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -932,7 +932,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 
 		sublock_file_fd = open(subtab_lock.buf,
 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
-		if (sublock_file_fd > 0) {
+		if (sublock_file_fd >= 0) {
 			close(sublock_file_fd);
 		} else if (sublock_file_fd < 0) {
 			if (errno == EEXIST) {
@@ -1013,7 +1013,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1050,9 +1050,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v2 07/11] reftable: fix NULL derefs in error paths

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 3 ++-
 reftable/stack.c  | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..8d308d858f8 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -796,6 +796,7 @@ int reftable_reader_print_file(const char *tablename)
 	reftable_table_from_reader(&tab, r);
 	err = reftable_table_print(&tab);
 done:
-	reftable_reader_free(r);
+	if (r)
+		reftable_reader_free(r);
 	return err;
 }
diff --git a/reftable/stack.c b/reftable/stack.c
index 10dfd370e8e..eb03b6c74f6 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -707,7 +707,8 @@ done:
 	strbuf_release(&temp_tab_file_name);
 	strbuf_release(&tab_file_name);
 	strbuf_release(&next_name);
-	reftable_writer_free(wr);
+	if (wr)
+		reftable_writer_free(wr);
 	return err;
 }
 
-- 
gitgitgadget

[PATCH v2 08/11] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more pratical ordering when working on refactorings of the reftable
code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..f08d66df949 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	// test from simple to complex.
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v2 09/11] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 5f6bcc2f775..7c40b9b77f3 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -597,7 +597,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v2 10/11] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..8bbcbff1e69 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v2 11/11] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void pointer or
vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it cannot be
zero-initialized. To address this, introduce reftable_record_for() to create a
fresh instance, given a record type.

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   4 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  35 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  37 +++++----
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  31 +++----
 reftable/reader.c      | 105 ++++++++++++------------
 reftable/record.c      | 176 +++++++++++++++------------------------
 reftable/record.h      |  45 +++++-----
 reftable/record_test.c | 181 +++++++++++++++++++++--------------------
 reftable/writer.c      |  46 ++++++-----
 12 files changed, 329 insertions(+), 360 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 79b6f3aac72..0bc4a5d84b3 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -382,7 +382,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 		.key = *want,
 		.r = br,
 	};
-	struct reftable_record rec = reftable_new_record(block_reader_type(br));
+	struct reftable_record rec = reftable_record_for(block_reader_type(br));
 	struct strbuf key = STRBUF_INIT;
 	int err = 0;
 	struct block_iter next = {
@@ -424,7 +424,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..4446b8ed360 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .u.ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,19 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..e9b0edec97d 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -22,7 +22,7 @@ static int merged_iter_init(struct merged_iter *mi)
 {
 	int i = 0;
 	for (i = 0; i < mi->stack_len; i++) {
-		struct reftable_record rec = reftable_new_record(mi->typ);
+		struct reftable_record rec = reftable_record_for(mi->typ);
 		int err = iterator_next(&mi->stack[i], &rec);
 		if (err < 0) {
 			return err;
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_record_for(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -246,7 +245,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,
 		sizeof(struct reftable_iterator) * mt->stack_len);
 	struct merged_iter merged = {
 		.stack = iters,
-		.typ = reftable_record_type(rec),
+		.typ = rec->type,
 		.hash_id = mt->hash_id,
 		.suppress_deletions = mt->suppress_deletions,
 	};
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,13 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..9f04d84fed6 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,14 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = {
+			.rec = {
+				.type = BLOCK_TYPE_REF,
+				.u.ref = {
+					.refname = names[i],
+				}
+			}
+		};
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +57,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(rec->type == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 8d308d858f8..ef322791fda 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -238,9 +238,8 @@ static int table_iter_next_in_block(struct table_iter *ti,
 				    struct reftable_record *rec)
 {
 	int res = block_iter_next(&ti->bi, rec);
-	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+	if (res == 0 && rec->type == BLOCK_TYPE_REF) {
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -345,7 +344,7 @@ static int table_iter_next_block(struct table_iter *dest,
 
 static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)
 {
-	if (reftable_record_type(rec) != ti->typ)
+	if (rec->type != ti->typ)
 		return REFTABLE_API_ERROR;
 
 	while (1) {
@@ -437,8 +436,7 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,
 static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 			      struct reftable_record *want)
 {
-	struct reftable_record rec =
-		reftable_new_record(reftable_record_type(want));
+	struct reftable_record rec = reftable_record_for(want->type);
 	struct strbuf want_key = STRBUF_INIT;
 	struct strbuf got_key = STRBUF_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
@@ -475,7 +473,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -485,38 +483,40 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
-	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
+	reftable_record_key(rec, &want_index.u.idx.last_key);
+	err = reader_start(r, &index_iter, rec->type, 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
-		if (next.typ == reftable_record_type(rec)) {
+		if (next.typ == rec->type) {
 			err = 0;
 			break;
 		}
@@ -540,8 +540,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -549,15 +549,14 @@ static int reader_seek_internal(struct reftable_reader *r,
 				struct reftable_iterator *it,
 				struct reftable_record *rec)
 {
-	struct reftable_reader_offsets *offs =
-		reader_offsets_for(r, reftable_record_type(rec));
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, rec->type);
 	uint64_t idx = offs->index_offset;
 	struct table_iter ti = TABLE_ITER_INIT;
 	int err = 0;
 	if (idx > 0)
 		return reader_seek_indexed(r, it, rec);
 
-	err = reader_start(r, &ti, reftable_record_type(rec), 0);
+	err = reader_start(r, &ti, rec->type, 0);
 	if (err < 0)
 		return err;
 	err = reader_seek_linear(r, &ti, rec);
@@ -576,7 +575,7 @@ static int reader_seek_internal(struct reftable_reader *r,
 static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 		       struct reftable_record *rec)
 {
-	uint8_t typ = reftable_record_type(rec);
+	uint8_t typ = rec->type;
 
 	struct reftable_reader_offsets *offs = reader_offsets_for(r, typ);
 	if (!offs->is_present) {
@@ -590,11 +589,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -602,12 +602,13 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return reader_seek(r, it, &rec);
 }
 
@@ -649,31 +650,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -681,15 +684,15 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets, got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 8bbcbff1e69..8cfd67b21e6 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -926,58 +930,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.is_deletion = &reftable_log_record_is_deletion_void,
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1055,91 +1007,47 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
-}
-
-uint8_t reftable_record_type(struct reftable_record *rec)
-{
-	return rec->ops->type;
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1212,3 +1120,53 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_record_for(uint8_t typ)
+{
+	struct reftable_record clean_idx = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.last_key = STRBUF_INIT,
+		},
+	};
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	if (typ == BLOCK_TYPE_INDEX) {
+		return clean_idx;
+	}
+
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..1fe0c14a198 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -60,18 +60,9 @@ struct reftable_record_vtable {
 	int (*is_deletion)(const void *rec);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
-struct reftable_record reftable_new_record(uint8_t typ);
-
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
  * number of bytes written. */
 int reftable_encode_key(int *is_restart, struct string_view dest,
@@ -97,8 +88,26 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_record_for() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 
+/* return an initialized record for the given type */
+struct reftable_record reftable_record_for(uint8_t typ);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
@@ -111,25 +120,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..12161c2895a 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,24 +16,22 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = reftable_record_for(rec->type);
+
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
+	switch (copy.type) {
 	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
+		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
 						 GIT_SHA1_RAWSZ));
 		break;
 	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
+		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
 						 GIT_SHA1_RAWSZ));
 		break;
 	}
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -106,61 +104,60 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_REF
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -213,7 +210,9 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = {
+			.type = BLOCK_TYPE_LOG
+		};
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -221,23 +220,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -245,16 +246,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -322,47 +323,52 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_OBJ
+		};
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
+		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
+		if (in.u.obj.hash_prefix_len)
+			EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
+				       in.u.obj.hash_prefix_len));
+		if (in.u.obj.offset_len)
+			EXPECT(!memcmp(in.u.obj.offsets, out.u.obj.offsets,
+					   sizeof(uint64_t) * in.u.obj.offset_len));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -370,31 +376,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(in.u.idx.offset == out.u.idx.offset);
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 3ca721e9f64..f9544c5f97b 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -221,10 +221,10 @@ static int writer_add_record(struct reftable_writer *w,
 	strbuf_reset(&w->last_key);
 	strbuf_addbuf(&w->last_key, &key);
 	if (w->block_writer == NULL) {
-		writer_reinit_block_writer(w, reftable_record_type(rec));
+		writer_reinit_block_writer(w, rec->type);
 	}
 
-	assert(block_writer_type(w->block_writer) == reftable_record_type(rec));
+	assert(block_writer_type(w->block_writer) == rec->type);
 
 	if (block_writer_add(w->block_writer, rec) == 0) {
 		err = 0;
@@ -236,7 +236,7 @@ static int writer_add_record(struct reftable_writer *w,
 		goto done;
 	}
 
-	writer_reinit_block_writer(w, reftable_record_type(rec));
+	writer_reinit_block_writer(w, rec->type);
 	err = block_writer_add(w->block_writer, rec);
 	if (err < 0) {
 		goto done;
@@ -251,8 +251,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -261,8 +263,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -301,7 +302,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -311,8 +315,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -393,8 +395,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -462,17 +466,18 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = (uint8_t *)entry->hash.buf,
+			.hash_prefix_len = arg->w->stats.object_id_len,
+			.offsets = entry->offsets,
+			.offset_len = entry->offset_len,
+		}
 	};
-	struct reftable_record rec = { NULL };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -485,7 +490,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

Re: [PATCH v2 00/11] Reftable coverity fixes

[Jeff King]

On Wed, Dec 08, 2021 at 09:49:35PM +0000, Han-Wen Nienhuys via GitGitGadget wrote:

>   2:  603bd1d4f6e !  2:  1ddcfe61ebc reftable: fix resource leak in error path
>      @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
>        
>        	if (typ == BLOCK_TYPE_LOG) {
>        		int block_header_skip = 4 + header_off;
>      +@@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,
>      + 		uLongf src_len = block->len - block_header_skip;
>      + 		/* Log blocks specify the *uncompressed* size in their header.
>      + 		 */
>      +-		uint8_t *uncompressed = reftable_malloc(sz);
>      ++		uncompressed = reftable_malloc(sz);
>      + 
>      + 		/* Copy over the block header verbatim. It's not compressed. */
>      + 		memcpy(uncompressed, block->data, block_header_skip);

This drops the shadowed variable, but it needs a little more. In one of
the error blocks we still free the variable, and then jump to the
"done" label, which will try to free it again. So you need:

diff --git a/reftable/block.c b/reftable/block.c
index 0bc4a5d84b..6d55817e0d 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -215,7 +215,6 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
 			err = REFTABLE_ZLIB_ERROR;
 			goto done;
 		}

to avoid the double-free. I think it's otherwise OK. The other error
case does not try to free, and in the success case we hand off ownership
of "uncompressed" and set it back to NULL.

I doubt this is the cause of the segfault on Windows, though (I saw it
via Coverity). That one still puzzles me, and I suspect will need more
output to figure out. It's possible to get an interactive session on the
Windows VM running the CI:

  https://github.com/marketplace/actions/debugging-with-tmate

but I haven't tried it myself. I'm also not sure what tools are
available there (gdb sure would be nice, but even a backtrace of the
segfault would be quite a lot; I wonder if we could build with ASan on
Windows).

-Peff

Re: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 08 2021, Jeff King wrote:

> On Tue, Dec 07, 2021 at 08:13:43PM -0800, Junio C Hamano wrote:
>
>> Jeff King <peff@peff.net> writes:
>> 
>> >> error: ISO C99 doesn't support unnamed structs/unions [-Werror=pedantic]
>> >
>> > Hmm. It's interesting that the regular DEVELOPER=1 doesn't catch this.
>> > It's because we don't specify -std there, and newer gcc defaults to
>> > gnu17 (unnamed unions appeared in c11, I think). I wonder if it would be
>> > helpful to teach config.mak.dev to pass -std=c99.
>> 
>> FWIW, I use -std=gnu99 as our Makefile suggests.
>
> I think the patch below would have detected this both locally for
> Han-Wen, but also in C.
>
> -- >8 --
> Subject: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang
>
> The point of DEVELOPER=1 is to turn up the warnings so we can catch
> portability or correctness mistakes at the compiler level. But since
> modern compilers tend to default to modern standards like gnu17, we
> might miss warnings about older standards, even though we expect Git to
> build with compilers that use them.
>
> So it's helpful for developer builds to set the -std argument to our
> lowest-common denominator. Traditionally this was c89, but since we're
> moving to assuming c99 in 7bc341e21b (git-compat-util: add a test
> balloon for C99 support, 2021-12-01) that seems like a good spot to
> land. And as explained in that commit, we want "gnu99" because we still
> want to take advantage of some extensions when they're available.
>
> The new argument kicks in only for clang and gcc (which we know to
> support "-std=" and "gnu" standards). And only for compiler versions
> which default to a newer standard. That will avoid accidentally
> silencing any build problems that non-developers would run into on older
> compilers that default to c89.
>
> My digging found that the default switched to gnu11 in gcc 5.1.0.
> Clang's documentation is less clear, but has done so since at least
> clang-7. So that's what I put in the conditional here. It's OK to err on
> the side of not-enabling this for older compilers. Most developers (as
> well as CI) are using much more recent versions, so any warnings will
> eventually surface.
>
> A concrete example is anonymous unions, which became legal in c11.
> Without this patch, "gcc -pedantic" will not complain about them, but
> will if we add in "-std=gnu99".
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
>  config.mak.dev | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git a/config.mak.dev b/config.mak.dev
> index 7673fed114..d4afac6b51 100644
> --- a/config.mak.dev
> +++ b/config.mak.dev
> @@ -19,6 +19,11 @@ endif
>  endif
>  endif
>  endif
> +
> +ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
> +DEVELOPER_CFLAGS += -std=gnu99
> +endif
> +
>  DEVELOPER_CFLAGS += -Wdeclaration-after-statement
>  DEVELOPER_CFLAGS += -Wformat-security
>  DEVELOPER_CFLAGS += -Wold-style-definition

This approach looks good & the rationale make sense.

I mentioned in [1] that this might be a bad idea because:

    And as you note it's not only that older or non-gcc non-clang compilers
    may not understand this at all, but are we getting worse behavior on
    modern versions of those two because they're forced into some 20 year
    old C99 standard mode, instead of the current preferred default?

But from some short testing of GCC it will generate the exact same
<file>.s regardless of -std=* option, so I think this indeed only
impacts the warnings we'll emit. So pinning this seems to categorically
be a good thing.

A bad thing about this is that we'll explicitly avoid happy accidents
where we start relying on some newer C standard, and discover N releases
later that it was no big deal, and can thus just use that feature.

On the other hand having this means less back & forth churn of adding
such a dependency only to find it breaks on one of our platforms
etc. Overall I think this makes sense, just say'n.

I don't think this needs to change, but FWIW this would benefit from the
same sort of "let's just compile it" step as [2]. I.e. I think you'll
find that we could just check the exit code of:

    $(CC) -E -std=gnu99 - </dev/null

This works on GCC/Clang, and will die on xlc/suncc, and I assume any
other compiler that doesn't grok this. But I think it's better to avoid
a $(shell) here just for that, and any such change can wait until we
have some proper "compile this once and cache it" probing for
config.mak.dev.

1. https://lore.kernel.org/git/211116.86pmr0p82k.gmgdl@evledraar.gmail.com/
2. https://lore.kernel.org/git/211118.86lf1m5h1y.gmgdl@evledraar.gmail.com/

Re: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang

[Jeff King]

On Thu, Dec 09, 2021 at 01:05:44PM +0100, Ævar Arnfjörð Bjarmason wrote:

> > +
> > +ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
> > +DEVELOPER_CFLAGS += -std=gnu99
> > +endif
> [...]
> 
> This approach looks good & the rationale make sense.
> 
> I mentioned in [1] that this might be a bad idea because:
> 
>     And as you note it's not only that older or non-gcc non-clang compilers
>     may not understand this at all, but are we getting worse behavior on
>     modern versions of those two because they're forced into some 20 year
>     old C99 standard mode, instead of the current preferred default?
> 
> But from some short testing of GCC it will generate the exact same
> <file>.s regardless of -std=* option, so I think this indeed only
> impacts the warnings we'll emit. So pinning this seems to categorically
> be a good thing.

Thanks for looking into that. The thought crossed my mind, too, but I
didn't have any actual data. I think some of this is due to the standard
committee keeping backwards compatibility. I.e., it's unlikely for there
to be a case where the N standard says "you must do X", and N+1 says
"that's dumb, you can do Y instead". Usually it is about new features
that were syntactically invalid or created undefined behavior in version
N.

> A bad thing about this is that we'll explicitly avoid happy accidents
> where we start relying on some newer C standard, and discover N releases
> later that it was no big deal, and can thus just use that feature.
> 
> On the other hand having this means less back & forth churn of adding
> such a dependency only to find it breaks on one of our platforms
> etc. Overall I think this makes sense, just say'n.

Right. I think it forces us to be more deliberate, but that's probably
OK.

> I don't think this needs to change, but FWIW this would benefit from the
> same sort of "let's just compile it" step as [2]. I.e. I think you'll
> find that we could just check the exit code of:
> 
>     $(CC) -E -std=gnu99 - </dev/null
> 
> This works on GCC/Clang, and will die on xlc/suncc, and I assume any
> other compiler that doesn't grok this. But I think it's better to avoid
> a $(shell) here just for that, and any such change can wait until we
> have some proper "compile this once and cache it" probing for
> config.mak.dev.

You'd lose one property that the version-check has, which is that we
don't ever _upgrade_ to gnu99 from gnu89. I'm not sure how important
that is, though. It would have been bad if we took something like
brian's c99 patch, and developers and CI both failed to realize that it
was breaking the out-of-the-box build for older compilers (because
behind the scenes they were getting -std=gnu99 that non-developers do
not). But now that we've decided on that direction anyway, I'm not sure
there's anything left to be learned.

-Peff

[PATCH v3 00/11] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'next'.

This addresses some complaints from Coverity that Peff reported.

v3:

 * revert some changes suggested by stolee
 * add unittest for zlib corruption.

Han-Wen Nienhuys (11):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: fix NULL derefs in error paths
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable_record a tagged union

 reftable/block.c          |  30 ++++---
 reftable/block_test.c     |  22 ++---
 reftable/blocksource.c    |   6 +-
 reftable/generic.c        |  35 ++++----
 reftable/iter.c           |   4 +-
 reftable/merged.c         |  33 +++----
 reftable/pq.c             |   3 +-
 reftable/pq_test.c        |  31 +++----
 reftable/reader.c         |  96 ++++++++++----------
 reftable/readwrite_test.c |  68 +++++++++++++-
 reftable/record.c         | 185 ++++++++++++++++----------------------
 reftable/record.h         |  42 ++++-----
 reftable/record_test.c    | 181 +++++++++++++++++++------------------
 reftable/stack.c          |  15 ++--
 reftable/stack_test.c     |   3 +-
 reftable/writer.c         |  40 +++++----
 t/helper/test-reftable.c  |   9 +-
 17 files changed, 431 insertions(+), 372 deletions(-)


base-commit: fae76fe5da3df25d752f2251b7ccda3f62813aa9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v3
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v2:

  1:  7c033815183 =  1:  1c1a3ff92bd reftable: fix OOB stack write in print functions
  2:  1ddcfe61ebc !  2:  975a570d388 reftable: fix resource leak in error path
     @@ Metadata
       ## Commit message ##
          reftable: fix resource leak in error path
      
     -    This would be triggered by corrupt files, so it doesn't have test coverage. This
     -    was discovered by a Coverity scan.
     +    Add test coverage for corrupt zlib data.
     +
     +    This problem was discovered by a Coverity scan.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
     @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
       		/* Copy over the block header verbatim. It's not compressed. */
       		memcpy(uncompressed, block->data, block_header_skip);
      @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,
     + 		if (Z_OK !=
       		    uncompress2(uncompressed + block_header_skip, &dst_len,
       				block->data + block_header_skip, &src_len)) {
     - 			reftable_free(uncompressed);
     +-			reftable_free(uncompressed);
      -			return REFTABLE_ZLIB_ERROR;
      +			err = REFTABLE_ZLIB_ERROR;
      +			goto done;
     @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
       }
       
       static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
     +
     + ## reftable/readwrite_test.c ##
     +@@ reftable/readwrite_test.c: static void test_log_write_read(void)
     + 	reader_close(&rd);
     + }
     + 
     ++static void test_log_zlib_corruption(void)
     ++{
     ++	struct reftable_write_options opts = {
     ++		.block_size = 256,
     ++	};
     ++	struct reftable_iterator it = { NULL };
     ++	struct reftable_reader rd = { NULL };
     ++	struct reftable_block_source source = { NULL };
     ++	struct strbuf buf = STRBUF_INIT;
     ++	struct reftable_writer *w =
     ++		reftable_new_writer(&strbuf_add_void, &buf, &opts);
     ++	const struct reftable_stats *stats = NULL;
     ++	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
     ++	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
     ++	char message[100] = { 0 };
     ++	int err, i, n;
     ++
     ++	struct reftable_log_record log = {
     ++		.refname = "refname",
     ++		.value_type = REFTABLE_LOG_UPDATE,
     ++		.value = {
     ++			.update = {
     ++				.new_hash = hash1,
     ++				.old_hash = hash2,
     ++				.name = "My Name",
     ++				.email = "myname@invalid",
     ++				.message = message,
     ++			},
     ++		},
     ++	};
     ++
     ++	for (i = 0; i < sizeof(message)-1; i++) {
     ++		message[i] = (uint8_t)(rand() % 64 + ' ');
     ++	}
     ++
     ++	reftable_writer_set_limits(w, 1, 1);
     ++
     ++	err = reftable_writer_add_log(w, &log);
     ++	EXPECT_ERR(err);
     ++
     ++	n = reftable_writer_close(w);
     ++	EXPECT(n == 0);
     ++
     ++	stats = writer_stats(w);
     ++	EXPECT(stats->log_stats.blocks > 0);
     ++	reftable_writer_free(w);
     ++	w = NULL;
     ++
     ++	/* corrupt the data. */
     ++	buf.buf[50] ^= 0x99;
     ++
     ++	block_source_from_strbuf(&source, &buf);
     ++
     ++	err = init_reader(&rd, &source, "file.log");
     ++	EXPECT_ERR(err);
     ++
     ++	err = reftable_reader_seek_log(&rd, &it, "refname");
     ++	EXPECT(err == REFTABLE_ZLIB_ERROR);
     ++
     ++	reftable_iterator_destroy(&it);
     ++
     ++	/* cleanup. */
     ++	strbuf_release(&buf);
     ++	reader_close(&rd);
     ++}
     ++
     + static void test_table_read_write_sequential(void)
     + {
     + 	char **names;
     +@@ reftable/readwrite_test.c: static void test_corrupt_table(void)
     + 
     + int readwrite_test_main(int argc, const char *argv[])
     + {
     ++	RUN_TEST(test_log_zlib_corruption);
     + 	RUN_TEST(test_corrupt_table);
     + 	RUN_TEST(test_corrupt_table_empty);
     + 	RUN_TEST(test_log_write_read);
  3:  e052b2a61d6 =  3:  0b9c7176d71 reftable: fix resource leak blocksource.c
  4:  9063137457b =  4:  1dda4ee717f reftable: check reftable_stack_auto_compact() return value
  5:  5020be156ae =  5:  36858e2070b reftable: ignore remove() return value in stack_test.c
  6:  64c18d01cad =  6:  80b1988b885 reftable: fix resource warning
  7:  700387ac5d3 =  7:  2939286924c reftable: fix NULL derefs in error paths
  8:  713f1d09f68 =  8:  9dce18d7349 reftable: order unittests by complexity
  9:  cb601b51a47 =  9:  6b0af68f0b9 reftable: drop stray printf in readwrite_test
 10:  a0f83eff19f = 10:  bff85cb0809 reftable: handle null refnames in reftable_ref_record_equal
 11:  a2743033cfd ! 11:  b3e592b9c27 reftable: make reftable_record a tagged union
     @@ Commit message
          vtable within the structure.
      
          The only snag is that reftable_index_record contain a strbuf, so it cannot be
     -    zero-initialized. To address this, introduce reftable_record_for() to create a
     -    fresh instance, given a record type.
     +    zero-initialized. To address this, use reftable_new_record() to return fresh
     +    instance, given a record type. Since reftable_new_record() doesn't cause heap
     +    allocation anymore, it should be balanced with reftable_record_release() rather
     +    than reftable_record_destroy().
      
          Thanks to Peff for the suggestion.
      
     @@ Commit message
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
       ## reftable/block.c ##
     -@@ reftable/block.c: int block_reader_seek(struct block_reader *br, struct block_iter *it,
     - 		.key = *want,
     - 		.r = br,
     - 	};
     --	struct reftable_record rec = reftable_new_record(block_reader_type(br));
     -+	struct reftable_record rec = reftable_record_for(block_reader_type(br));
     - 	struct strbuf key = STRBUF_INIT;
     - 	int err = 0;
     - 	struct block_iter next = {
      @@ reftable/block.c: int block_reader_seek(struct block_reader *br, struct block_iter *it,
       done:
       	strbuf_release(&key);
     @@ reftable/iter.c: static int indexed_table_ref_iter_next_block(struct indexed_tab
      
       ## reftable/merged.c ##
      @@ reftable/merged.c: static int merged_iter_init(struct merged_iter *mi)
     - {
     - 	int i = 0;
     - 	for (i = 0; i < mi->stack_len; i++) {
     --		struct reftable_record rec = reftable_new_record(mi->typ);
     -+		struct reftable_record rec = reftable_record_for(mi->typ);
     - 		int err = iterator_next(&mi->stack[i], &rec);
     - 		if (err < 0) {
     - 			return err;
     -@@ reftable/merged.c: static int merged_iter_init(struct merged_iter *mi)
       
       		if (err > 0) {
       			reftable_iterator_destroy(&mi->stack[i]);
     @@ reftable/merged.c: static void merged_iter_close(void *p)
      -	struct reftable_record rec = reftable_new_record(mi->typ);
       	struct pq_entry e = {
      -		.rec = rec,
     -+		.rec = reftable_record_for(mi->typ),
     ++		.rec = reftable_new_record(mi->typ),
       		.index = idx,
       	};
      -	int err = iterator_next(&mi->stack[idx], &rec);
     @@ reftable/merged.c: static int merged_iter_next_entry(struct merged_iter *mi,
       	strbuf_release(&entry_key);
       	return 0;
       }
     -@@ reftable/merged.c: static int merged_table_seek_record(struct reftable_merged_table *mt,
     - 		sizeof(struct reftable_iterator) * mt->stack_len);
     - 	struct merged_iter merged = {
     - 		.stack = iters,
     --		.typ = reftable_record_type(rec),
     -+		.typ = rec->type,
     - 		.hash_id = mt->hash_id,
     - 		.suppress_deletions = mt->suppress_deletions,
     - 	};
      @@ reftable/merged.c: int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
       				   struct reftable_iterator *it,
       				   const char *name)
     @@ reftable/pq_test.c: static void test_pq(void)
      +		struct reftable_record *rec = &e.rec;
       		merged_iter_pqueue_check(pq);
       
     -+		EXPECT(rec->type == BLOCK_TYPE_REF);
     ++		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
       		if (last) {
      -			EXPECT(strcmp(last, ref->refname) < 0);
      +			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
     @@ reftable/pq_test.c: static void test_pq(void)
      
       ## reftable/reader.c ##
      @@ reftable/reader.c: static int table_iter_next_in_block(struct table_iter *ti,
     - 				    struct reftable_record *rec)
       {
       	int res = block_iter_next(&ti->bi, rec);
     --	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
     + 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
      -		((struct reftable_ref_record *)rec->data)->update_index +=
      -			ti->r->min_update_index;
     -+	if (res == 0 && rec->type == BLOCK_TYPE_REF) {
      +		rec->u.ref.update_index += ti->r->min_update_index;
       	}
       
       	return res;
     -@@ reftable/reader.c: static int table_iter_next_block(struct table_iter *dest,
     - 
     - static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)
     - {
     --	if (reftable_record_type(rec) != ti->typ)
     -+	if (rec->type != ti->typ)
     - 		return REFTABLE_API_ERROR;
     - 
     - 	while (1) {
      @@ reftable/reader.c: static int reader_start(struct reftable_reader *r, struct table_iter *ti,
       static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
       			      struct reftable_record *want)
       {
      -	struct reftable_record rec =
      -		reftable_new_record(reftable_record_type(want));
     -+	struct reftable_record rec = reftable_record_for(want->type);
     ++	struct reftable_record rec = reftable_new_record(reftable_record_type(want));
       	struct strbuf want_key = STRBUF_INIT;
       	struct strbuf got_key = STRBUF_INIT;
       	struct table_iter next = TABLE_ITER_INIT;
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
      -	reftable_record_from_index(&want_index_rec, &want_index);
      -	reftable_record_from_index(&index_result_rec, &index_result);
      -
     --	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
      +	reftable_record_key(rec, &want_index.u.idx.last_key);
     -+	err = reader_start(r, &index_iter, rec->type, 1);
     + 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
       	if (err < 0)
       		goto done;
       
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
       		if (err < 0)
       			goto done;
       
     --		if (next.typ == reftable_record_type(rec)) {
     -+		if (next.typ == rec->type) {
     - 			err = 0;
     - 			break;
     - 		}
      @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
       done:
       	block_iter_close(&next.bi);
     @@ reftable/reader.c: static int reader_seek_internal(struct reftable_reader *r,
       {
      -	struct reftable_reader_offsets *offs =
      -		reader_offsets_for(r, reftable_record_type(rec));
     -+	struct reftable_reader_offsets *offs = reader_offsets_for(r, rec->type);
     ++	struct reftable_reader_offsets *offs = reader_offsets_for(r, reftable_record_type(rec));
       	uint64_t idx = offs->index_offset;
       	struct table_iter ti = TABLE_ITER_INIT;
       	int err = 0;
     - 	if (idx > 0)
     - 		return reader_seek_indexed(r, it, rec);
     - 
     --	err = reader_start(r, &ti, reftable_record_type(rec), 0);
     -+	err = reader_start(r, &ti, rec->type, 0);
     - 	if (err < 0)
     - 		return err;
     - 	err = reader_seek_linear(r, &ti, rec);
     -@@ reftable/reader.c: static int reader_seek_internal(struct reftable_reader *r,
     - static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
     - 		       struct reftable_record *rec)
     - {
     --	uint8_t typ = reftable_record_type(rec);
     -+	uint8_t typ = rec->type;
     - 
     - 	struct reftable_reader_offsets *offs = reader_offsets_for(r, typ);
     - 	if (!offs->is_present) {
      @@ reftable/reader.c: static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
       int reftable_reader_seek_ref(struct reftable_reader *r,
       			     struct reftable_iterator *it, const char *name)
     @@ reftable/record.c: static struct reftable_record_vtable reftable_index_record_vt
       void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
       {
      -	rec->ops->key(rec->data, dest);
     --}
     --
     --uint8_t reftable_record_type(struct reftable_record *rec)
     --{
     --	return rec->ops->type;
      +	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
       }
       
     + uint8_t reftable_record_type(struct reftable_record *rec)
     + {
     +-	return rec->ops->type;
     ++	return rec->type;
     + }
     + 
       int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
       			   int hash_size)
       {
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
      +	abort();
      +}
      +
     -+struct reftable_record reftable_record_for(uint8_t typ)
     ++struct reftable_record reftable_new_record(uint8_t typ)
      +{
      +	struct reftable_record clean_idx = {
      +		.type = BLOCK_TYPE_INDEX,
     @@ reftable/record.h: struct reftable_record_vtable {
       int reftable_is_block_type(uint8_t typ);
       
      -/* creates a malloced record of the given type. Dispose with record_destroy */
     --struct reftable_record reftable_new_record(uint8_t typ);
     --
     ++/* return an initialized record for the given type */
     + struct reftable_record reftable_new_record(uint8_t typ);
     + 
       /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
     -  * number of bytes written. */
     - int reftable_encode_key(int *is_restart, struct string_view dest,
      @@ reftable/record.h: struct reftable_obj_record {
       	int offset_len;
       };
     @@ reftable/record.h: struct reftable_obj_record {
      +/* record is a generic wrapper for different types of records. It is normally
      + * created on the stack, or embedded within another struct. If the type is
      + * known, a fresh instance can be initialized explicitly. Otherwise, use
     -+ * reftable_record_for() to initialize generically (as the index_record is not
     ++ * reftable_new_record() to initialize generically (as the index_record is not
      + * valid as 0-initialized structure)
      + */
      +struct reftable_record {
     @@ reftable/record.h: struct reftable_obj_record {
      +
       /* see struct record_vtable */
       
     -+/* return an initialized record for the given type */
     -+struct reftable_record reftable_record_for(uint8_t typ);
       void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
     - uint8_t reftable_record_type(struct reftable_record *rec);
     - void reftable_record_copy_from(struct reftable_record *rec,
      @@ reftable/record.h: int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
       			   int hash_size);
       int reftable_record_is_deletion(struct reftable_record *rec);
     @@ reftable/record_test.c
       {
      -	struct reftable_record copy =
      -		reftable_new_record(reftable_record_type(rec));
     -+	struct reftable_record copy = reftable_record_for(rec->type);
     ++	struct reftable_record copy = reftable_new_record(reftable_record_type(rec));
      +
       	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
       	/* do it twice to catch memory leaks */
     @@ reftable/record_test.c: static void test_reftable_index_record_roundtrip(void)
       int record_test_main(int argc, const char *argv[])
      
       ## reftable/writer.c ##
     -@@ reftable/writer.c: static int writer_add_record(struct reftable_writer *w,
     - 	strbuf_reset(&w->last_key);
     - 	strbuf_addbuf(&w->last_key, &key);
     - 	if (w->block_writer == NULL) {
     --		writer_reinit_block_writer(w, reftable_record_type(rec));
     -+		writer_reinit_block_writer(w, rec->type);
     - 	}
     - 
     --	assert(block_writer_type(w->block_writer) == reftable_record_type(rec));
     -+	assert(block_writer_type(w->block_writer) == rec->type);
     - 
     - 	if (block_writer_add(w->block_writer, rec) == 0) {
     - 		err = 0;
     -@@ reftable/writer.c: static int writer_add_record(struct reftable_writer *w,
     - 		goto done;
     - 	}
     - 
     --	writer_reinit_block_writer(w, reftable_record_type(rec));
     -+	writer_reinit_block_writer(w, rec->type);
     - 	err = block_writer_add(w->block_writer, rec);
     - 	if (err < 0) {
     - 		goto done;
      @@ reftable/writer.c: done:
       int reftable_writer_add_ref(struct reftable_writer *w,
       			    struct reftable_ref_record *ref)

-- 
gitgitgadget

[PATCH v3 01/11] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v3 02/11] reftable: fix resource leak in error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Add test coverage for corrupt zlib data.

This problem was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c          | 28 ++++++++++------
 reftable/readwrite_test.c | 67 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 86 insertions(+), 9 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..37a85b9576d 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +206,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -212,16 +215,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +257,11 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	if (uncompressed) {
+		reftable_free(uncompressed);
+	}
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 5f6bcc2f775..42caf0bde4c 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -254,6 +254,72 @@ static void test_log_write_read(void)
 	reader_close(&rd);
 }
 
+static void test_log_zlib_corruption(void)
+{
+	struct reftable_write_options opts = {
+		.block_size = 256,
+	};
+	struct reftable_iterator it = { NULL };
+	struct reftable_reader rd = { NULL };
+	struct reftable_block_source source = { NULL };
+	struct strbuf buf = STRBUF_INIT;
+	struct reftable_writer *w =
+		reftable_new_writer(&strbuf_add_void, &buf, &opts);
+	const struct reftable_stats *stats = NULL;
+	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
+	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
+	char message[100] = { 0 };
+	int err, i, n;
+
+	struct reftable_log_record log = {
+		.refname = "refname",
+		.value_type = REFTABLE_LOG_UPDATE,
+		.value = {
+			.update = {
+				.new_hash = hash1,
+				.old_hash = hash2,
+				.name = "My Name",
+				.email = "myname@invalid",
+				.message = message,
+			},
+		},
+	};
+
+	for (i = 0; i < sizeof(message)-1; i++) {
+		message[i] = (uint8_t)(rand() % 64 + ' ');
+	}
+
+	reftable_writer_set_limits(w, 1, 1);
+
+	err = reftable_writer_add_log(w, &log);
+	EXPECT_ERR(err);
+
+	n = reftable_writer_close(w);
+	EXPECT(n == 0);
+
+	stats = writer_stats(w);
+	EXPECT(stats->log_stats.blocks > 0);
+	reftable_writer_free(w);
+	w = NULL;
+
+	/* corrupt the data. */
+	buf.buf[50] ^= 0x99;
+
+	block_source_from_strbuf(&source, &buf);
+
+	err = init_reader(&rd, &source, "file.log");
+	EXPECT_ERR(err);
+
+	err = reftable_reader_seek_log(&rd, &it, "refname");
+	EXPECT(err == REFTABLE_ZLIB_ERROR);
+
+	reftable_iterator_destroy(&it);
+
+	/* cleanup. */
+	strbuf_release(&buf);
+	reader_close(&rd);
+}
+
 static void test_table_read_write_sequential(void)
 {
 	char **names;
@@ -633,6 +699,7 @@ static void test_corrupt_table(void)
 
 int readwrite_test_main(int argc, const char *argv[])
 {
+	RUN_TEST(test_log_zlib_corruption);
 	RUN_TEST(test_corrupt_table);
 	RUN_TEST(test_corrupt_table_empty);
 	RUN_TEST(test_log_write_read);
-- 
gitgitgadget

[PATCH v3 03/11] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an opened
file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v3 04/11] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index eb0b7228b0c..d628420e63a 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -814,6 +814,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v3 05/11] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index d628420e63a..4b7292945c3 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -89,7 +89,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v3 06/11] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and the next
available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index df5021ebf08..10dfd370e8e 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -877,7 +877,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -911,7 +911,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -932,7 +932,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 
 		sublock_file_fd = open(subtab_lock.buf,
 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
-		if (sublock_file_fd > 0) {
+		if (sublock_file_fd >= 0) {
 			close(sublock_file_fd);
 		} else if (sublock_file_fd < 0) {
 			if (errno == EEXIST) {
@@ -1013,7 +1013,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1050,9 +1050,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v3 07/11] reftable: fix NULL derefs in error paths

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 3 ++-
 reftable/stack.c  | 3 ++-
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..8d308d858f8 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -796,6 +796,7 @@ int reftable_reader_print_file(const char *tablename)
 	reftable_table_from_reader(&tab, r);
 	err = reftable_table_print(&tab);
 done:
-	reftable_reader_free(r);
+	if (r)
+		reftable_reader_free(r);
 	return err;
 }
diff --git a/reftable/stack.c b/reftable/stack.c
index 10dfd370e8e..eb03b6c74f6 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -707,7 +707,8 @@ done:
 	strbuf_release(&temp_tab_file_name);
 	strbuf_release(&tab_file_name);
 	strbuf_release(&next_name);
-	reftable_writer_free(wr);
+	if (wr)
+		reftable_writer_free(wr);
 	return err;
 }
 
-- 
gitgitgadget

[PATCH v3 08/11] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more pratical ordering when working on refactorings of the reftable
code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..f08d66df949 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	// test from simple to complex.
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v3 09/11] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 42caf0bde4c..f6546d7b8d1 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -663,7 +663,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v3 10/11] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..8bbcbff1e69 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v3 11/11] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void pointer or
vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it cannot be
zero-initialized. To address this, use reftable_new_record() to return fresh
instance, given a record type. Since reftable_new_record() doesn't cause heap
allocation anymore, it should be balanced with reftable_record_release() rather
than reftable_record_destroy().

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   2 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  35 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  33 ++++----
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  31 +++----
 reftable/reader.c      |  93 +++++++++++----------
 reftable/record.c      | 173 ++++++++++++++++-----------------------
 reftable/record.h      |  42 ++++------
 reftable/record_test.c | 181 +++++++++++++++++++++--------------------
 reftable/writer.c      |  40 +++++----
 12 files changed, 317 insertions(+), 342 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 37a85b9576d..872bedaa2f0 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -423,7 +423,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..4446b8ed360 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .u.ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,19 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..a7c0df91ff0 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_new_record(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,13 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..e0f5c584ada 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,14 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = {
+			.rec = {
+				.type = BLOCK_TYPE_REF,
+				.u.ref = {
+					.refname = names[i],
+				}
+			}
+		};
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +57,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 8d308d858f8..4a25c1d44f4 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,8 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -437,8 +436,7 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,
 static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 			      struct reftable_record *want)
 {
-	struct reftable_record rec =
-		reftable_new_record(reftable_record_type(want));
+	struct reftable_record rec = reftable_new_record(reftable_record_type(want));
 	struct strbuf want_key = STRBUF_INIT;
 	struct strbuf got_key = STRBUF_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
@@ -475,7 +473,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -485,34 +483,36 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -540,8 +540,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -549,8 +549,7 @@ static int reader_seek_internal(struct reftable_reader *r,
 				struct reftable_iterator *it,
 				struct reftable_record *rec)
 {
-	struct reftable_reader_offsets *offs =
-		reader_offsets_for(r, reftable_record_type(rec));
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, reftable_record_type(rec));
 	uint64_t idx = offs->index_offset;
 	struct table_iter ti = TABLE_ITER_INIT;
 	int err = 0;
@@ -590,11 +589,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -602,12 +602,13 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return reader_seek(r, it, &rec);
 }
 
@@ -649,31 +650,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -681,15 +684,15 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets, got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 8bbcbff1e69..f766ee20ace 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -926,58 +930,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.is_deletion = &reftable_log_record_is_deletion_void,
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1055,91 +1007,52 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 uint8_t reftable_record_type(struct reftable_record *rec)
 {
-	return rec->ops->type;
+	return rec->type;
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1212,3 +1125,53 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_new_record(uint8_t typ)
+{
+	struct reftable_record clean_idx = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.last_key = STRBUF_INIT,
+		},
+	};
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	if (typ == BLOCK_TYPE_INDEX) {
+		return clean_idx;
+	}
+
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..407cdaba018 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -60,16 +60,10 @@ struct reftable_record_vtable {
 	int (*is_deletion)(const void *rec);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
+/* return an initialized record for the given type */
 struct reftable_record reftable_new_record(uint8_t typ);
 
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
@@ -97,6 +91,22 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_new_record() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
@@ -111,25 +121,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..96f65035c1b 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,24 +16,22 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = reftable_new_record(reftable_record_type(rec));
+
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
+	switch (copy.type) {
 	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
+		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
 						 GIT_SHA1_RAWSZ));
 		break;
 	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
+		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
 						 GIT_SHA1_RAWSZ));
 		break;
 	}
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -106,61 +104,60 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_REF
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -213,7 +210,9 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = {
+			.type = BLOCK_TYPE_LOG
+		};
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -221,23 +220,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -245,16 +246,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -322,47 +323,52 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_OBJ
+		};
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
+		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
+		if (in.u.obj.hash_prefix_len)
+			EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
+				       in.u.obj.hash_prefix_len));
+		if (in.u.obj.offset_len)
+			EXPECT(!memcmp(in.u.obj.offsets, out.u.obj.offsets,
+					   sizeof(uint64_t) * in.u.obj.offset_len));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -370,31 +376,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(in.u.idx.offset == out.u.idx.offset);
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 3ca721e9f64..990f78ff7ed 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -251,8 +251,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -261,8 +263,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -301,7 +302,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -311,8 +315,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -393,8 +395,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -462,17 +466,18 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = (uint8_t *)entry->hash.buf,
+			.hash_prefix_len = arg->w->stats.object_id_len,
+			.offsets = entry->offsets,
+			.offset_len = entry->offset_len,
+		}
 	};
-	struct reftable_record rec = { NULL };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -485,7 +490,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

Re: [PATCH v3 02/11] reftable: fix resource leak in error path

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 13 2021, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
> [...]
> -	return 0;
> +done:
> +	if (uncompressed) {
> +		reftable_free(uncompressed);
> +	}
> +	return err;
>  }

Other things in the codebase don't check for NULL before feeding things
to reftable_free(), and our own free() has a coccicheck rule to catch
this sort of code, we should probably add reftable_free to that list...

>  
>  static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
> diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
> index 5f6bcc2f775..42caf0bde4c 100644
> --- a/reftable/readwrite_test.c
> +++ b/reftable/readwrite_test.c
> @@ -254,6 +254,72 @@ static void test_log_write_read(void)
>  	reader_close(&rd);
>  }
>  
> +static void test_log_zlib_corruption(void)
> +{
> +	struct reftable_write_options opts = {
> +		.block_size = 256,
> +	};
> +	struct reftable_iterator it = { NULL };
> +	struct reftable_reader rd = { NULL };
> +	struct reftable_block_source source = { NULL };

Nit: It doesn't matter for semantics, but usually we use "{ 0 }", and
your 1/11 does too. Would be better to do that here for consistency.

> +	for (i = 0; i < sizeof(message)-1; i++) {
> +		message[i] = (uint8_t)(rand() % 64 + ' ');
> +	}

style: braces not needede.

Re: [PATCH v3 07/11] reftable: fix NULL derefs in error paths

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 13 2021, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Spotted by Coverity.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/reader.c | 3 ++-
>  reftable/stack.c  | 3 ++-
>  2 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/reftable/reader.c b/reftable/reader.c
> index 006709a645a..8d308d858f8 100644
> --- a/reftable/reader.c
> +++ b/reftable/reader.c
> @@ -796,6 +796,7 @@ int reftable_reader_print_file(const char *tablename)
>  	reftable_table_from_reader(&tab, r);
>  	err = reftable_table_print(&tab);
>  done:
> -	reftable_reader_free(r);
> +	if (r)
> +		reftable_reader_free(r);
>  	return err;
>  }
> diff --git a/reftable/stack.c b/reftable/stack.c
> index 10dfd370e8e..eb03b6c74f6 100644
> --- a/reftable/stack.c
> +++ b/reftable/stack.c
> @@ -707,7 +707,8 @@ done:
>  	strbuf_release(&temp_tab_file_name);
>  	strbuf_release(&tab_file_name);
>  	strbuf_release(&next_name);
> -	reftable_writer_free(wr);
> +	if (wr)
> +		reftable_writer_free(wr);
>  	return err;
>  }

I'd think this would be better:

diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..3eec915830d 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -641,7 +641,8 @@ int reftable_new_reader(struct reftable_reader **p,
 
 void reftable_reader_free(struct reftable_reader *r)
 {
-       reader_close(r);
+       if (r)
+               reader_close(r);
        reftable_free(r);
 }

No?

Re: [PATCH v3 08/11] reftable: order unittests by complexity

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 13 2021, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> This is a more pratical ordering when working on refactorings of the reftable
> code.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  t/helper/test-reftable.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
> index 26b03d7b789..f08d66df949 100644
> --- a/t/helper/test-reftable.c
> +++ b/t/helper/test-reftable.c
> @@ -3,15 +3,16 @@
>  
>  int cmd__reftable(int argc, const char **argv)
>  {
> +	// test from simple to complex.

A C99 comment, which I don't mind, but is explicitly forbidden by the
style guide (and when I mentioned recently that we didn't have any
portability reasons not to use these, there wasn't much/any interest in
changing that...).

Re: [PATCH v3 09/11] reftable: drop stray printf in readwrite_test

[Ævar Arnfjörð Bjarmason]

On Mon, Dec 13 2021, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/readwrite_test.c | 1 -
>  1 file changed, 1 deletion(-)
>
> diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
> index 42caf0bde4c..f6546d7b8d1 100644
> --- a/reftable/readwrite_test.c
> +++ b/reftable/readwrite_test.c
> @@ -663,7 +663,6 @@ static void test_write_key_order(void)
>  	err = reftable_writer_add_ref(w, &refs[0]);
>  	EXPECT_ERR(err);
>  	err = reftable_writer_add_ref(w, &refs[1]);
> -	printf("%d\n", err);
>  	EXPECT(err == REFTABLE_API_ERROR);
>  	reftable_writer_close(w);
>  	reftable_writer_free(w);

Is this something coverity raised and I'm missing why it's bad (per the
CL), or just a stray unrelated cleanup?

Re: [PATCH v3 02/11] reftable: fix resource leak in error path

[Han-Wen Nienhuys]

On Mon, Dec 13, 2021 at 5:21 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Other things in the codebase don't check for NULL before feeding things
> to reftable_free(), and our own free() has a coccicheck rule to catch
> this sort of code, we should probably add reftable_free to that list...

Thanks, changed.

> > +     struct reftable_block_source source = { NULL };
>
> Nit: It doesn't matter for semantics, but usually we use "{ 0 }", and
> your 1/11 does too. Would be better to do that here for consistency.

I got a past review where someone complained about this. I don't mind
either way, but would rather not flipflop.

>
> > +     for (i = 0; i < sizeof(message)-1; i++) {
> > +             message[i] = (uint8_t)(rand() % 64 + ' ');
> > +     }
>
> style: braces not needede.

Done.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v3 08/11] reftable: order unittests by complexity

[Han-Wen Nienhuys]

On Mon, Dec 13, 2021 at 5:26 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> A C99 comment, which I don't mind, but is explicitly forbidden by the
> style guide (and when I mentioned recently that we didn't have any
> portability reasons not to use these, there wasn't much/any interest in
> changing that...).

sorry, I missed this comment in a previous round. Fixed now.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v3 09/11] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys]

On Mon, Dec 13, 2021 at 5:26 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> Is this something coverity raised and I'm missing why it's bad (per the
> CL), or just a stray unrelated cleanup?

I spotted it while working on this series. It prints something
unexpected into the output of the unittest, so better remove it.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v3 02/11] reftable: fix resource leak in error path

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@google.com> writes:

> On Mon, Dec 13, 2021 at 5:21 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> Other things in the codebase don't check for NULL before feeding things
>> to reftable_free(), and our own free() has a coccicheck rule to catch
>> this sort of code, we should probably add reftable_free to that list...
>
> Thanks, changed.
>
>> > +     struct reftable_block_source source = { NULL };
>>
>> Nit: It doesn't matter for semantics, but usually we use "{ 0 }", and
>> your 1/11 does too. Would be better to do that here for consistency.
>
> I got a past review where someone complained about this. I don't mind
> either way, but would rather not flipflop.

The last part is important.  

For initializers that show the value for the first member in {} to
mean "everything is zero-initialized", only because the language
does not allow us to write

	struct foo var = {};

we historically used { NULL } for pointers and { 0 } for integral
types (primarily because auto checkers like sparse did not like us
to write a NULL pointer as 0), but we started preferring { 0 } as
more recent versions of checkers understand it as an idiom, and we
can freely reorder the struct members if we consistently spell the
"everything is zero-initialized" that way.

So, let's use { 0 } here, too.

Thanks.

Re: [PATCH v3 08/11] reftable: order unittests by complexity

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>>  int cmd__reftable(int argc, const char **argv)
>>  {
>> +	// test from simple to complex.
>
> A C99 comment, which I don't mind, but is explicitly forbidden by the
> style guide (and when I mentioned recently that we didn't have any
> portability reasons not to use these, there wasn't much/any interest in
> changing that...).

Unlike the decl-after-statement thing, I personally do not mind
something that does not upset compilers and does not hurt human
readability greatly, like the use of "//".  But I do not care about
it deeply enough to find it worth my time to raise it as a separate
topic and update the guideline.

[PATCH v4 00/11] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'next'.

This addresses some complaints from Coverity that Peff reported.

v3:

 * revert some changes suggested by stolee
 * add unittest for zlib corruption.

Han-Wen Nienhuys (11):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in block.c error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: all xxx_free() functions accept NULL arguments
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable_record a tagged union

 reftable/block.c          |  28 +++---
 reftable/block_test.c     |  22 ++---
 reftable/blocksource.c    |   6 +-
 reftable/generic.c        |  35 ++++----
 reftable/iter.c           |   4 +-
 reftable/merged.c         |  33 +++----
 reftable/pq.c             |   3 +-
 reftable/pq_test.c        |  31 +++----
 reftable/reader.c         |  95 ++++++++++----------
 reftable/readwrite_test.c |  67 +++++++++++++-
 reftable/record.c         | 185 ++++++++++++++++----------------------
 reftable/record.h         |  42 ++++-----
 reftable/record_test.c    | 181 +++++++++++++++++++------------------
 reftable/stack.c          |  12 +--
 reftable/stack_test.c     |   3 +-
 reftable/writer.c         |  42 +++++----
 t/helper/test-reftable.c  |   9 +-
 17 files changed, 428 insertions(+), 370 deletions(-)


base-commit: fae76fe5da3df25d752f2251b7ccda3f62813aa9
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v4
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v4
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v3:

  1:  1c1a3ff92bd =  1:  1c1a3ff92bd reftable: fix OOB stack write in print functions
  2:  975a570d388 !  2:  923293b79b4 reftable: fix resource leak in error path
     @@ Metadata
      Author: Han-Wen Nienhuys <hanwen@google.com>
      
       ## Commit message ##
     -    reftable: fix resource leak in error path
     +    reftable: fix resource leak in block.c error path
      
          Add test coverage for corrupt zlib data.
      
     @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
       
      -	return 0;
      +done:
     -+	if (uncompressed) {
     -+		reftable_free(uncompressed);
     -+	}
     ++	reftable_free(uncompressed);
      +	return err;
       }
       
     @@ reftable/readwrite_test.c: static void test_log_write_read(void)
      +	struct reftable_write_options opts = {
      +		.block_size = 256,
      +	};
     -+	struct reftable_iterator it = { NULL };
     -+	struct reftable_reader rd = { NULL };
     -+	struct reftable_block_source source = { NULL };
     ++	struct reftable_iterator it = { 0 };
     ++	struct reftable_reader rd = { 0 };
     ++	struct reftable_block_source source = { 0 };
      +	struct strbuf buf = STRBUF_INIT;
      +	struct reftable_writer *w =
      +		reftable_new_writer(&strbuf_add_void, &buf, &opts);
     @@ reftable/readwrite_test.c: static void test_log_write_read(void)
      +		},
      +	};
      +
     -+	for (i = 0; i < sizeof(message)-1; i++) {
     ++	for (i = 0; i < sizeof(message)-1; i++)
      +		message[i] = (uint8_t)(rand() % 64 + ' ');
     -+	}
      +
      +	reftable_writer_set_limits(w, 1, 1);
      +
  3:  0b9c7176d71 =  3:  7a914f77756 reftable: fix resource leak blocksource.c
  4:  1dda4ee717f =  4:  51b4a17a2e1 reftable: check reftable_stack_auto_compact() return value
  5:  36858e2070b =  5:  43989afcb5a reftable: ignore remove() return value in stack_test.c
  6:  80b1988b885 =  6:  1c7f15d811c reftable: fix resource warning
  7:  2939286924c <  -:  ----------- reftable: fix NULL derefs in error paths
  -:  ----------- >  7:  47ba5ddceb8 reftable: all xxx_free() functions accept NULL arguments
  8:  9dce18d7349 !  8:  aba8b8113ad reftable: order unittests by complexity
     @@ Metadata
       ## Commit message ##
          reftable: order unittests by complexity
      
     -    This is a more pratical ordering when working on refactorings of the reftable
     +    This is a more practical ordering when working on refactorings of the reftable
          code.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
     @@ t/helper/test-reftable.c
       
       int cmd__reftable(int argc, const char **argv)
       {
     -+	// test from simple to complex.
     ++	/* test from simple to complex. */
       	basics_test_main(argc, argv);
      +	record_test_main(argc, argv);
       	block_test_main(argc, argv);
  9:  6b0af68f0b9 =  9:  f2af404d7f1 reftable: drop stray printf in readwrite_test
 10:  bff85cb0809 = 10:  d2ce1bfef9f reftable: handle null refnames in reftable_ref_record_equal
 11:  b3e592b9c27 = 11:  0ffc1cf0e42 reftable: make reftable_record a tagged union

-- 
gitgitgadget

[PATCH v4 01/11] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v4 02/11] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Add test coverage for corrupt zlib data.

This problem was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c          | 26 +++++++++------
 reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+), 9 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..6c8e8705205 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +206,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -212,16 +215,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +257,9 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	reftable_free(uncompressed);
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 5f6bcc2f775..6e88182a83a 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -254,6 +254,71 @@ static void test_log_write_read(void)
 	reader_close(&rd);
 }
 
+static void test_log_zlib_corruption(void)
+{
+	struct reftable_write_options opts = {
+		.block_size = 256,
+	};
+	struct reftable_iterator it = { 0 };
+	struct reftable_reader rd = { 0 };
+	struct reftable_block_source source = { 0 };
+	struct strbuf buf = STRBUF_INIT;
+	struct reftable_writer *w =
+		reftable_new_writer(&strbuf_add_void, &buf, &opts);
+	const struct reftable_stats *stats = NULL;
+	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
+	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
+	char message[100] = { 0 };
+	int err, i, n;
+
+	struct reftable_log_record log = {
+		.refname = "refname",
+		.value_type = REFTABLE_LOG_UPDATE,
+		.value = {
+			.update = {
+				.new_hash = hash1,
+				.old_hash = hash2,
+				.name = "My Name",
+				.email = "myname@invalid",
+				.message = message,
+			},
+		},
+	};
+
+	for (i = 0; i < sizeof(message)-1; i++)
+		message[i] = (uint8_t)(rand() % 64 + ' ');
+
+	reftable_writer_set_limits(w, 1, 1);
+
+	err = reftable_writer_add_log(w, &log);
+	EXPECT_ERR(err);
+
+	n = reftable_writer_close(w);
+	EXPECT(n == 0);
+
+	stats = writer_stats(w);
+	EXPECT(stats->log_stats.blocks > 0);
+	reftable_writer_free(w);
+	w = NULL;
+
+	/* corrupt the data. */
+	buf.buf[50] ^= 0x99;
+
+	block_source_from_strbuf(&source, &buf);
+
+	err = init_reader(&rd, &source, "file.log");
+	EXPECT_ERR(err);
+
+	err = reftable_reader_seek_log(&rd, &it, "refname");
+	EXPECT(err == REFTABLE_ZLIB_ERROR);
+
+	reftable_iterator_destroy(&it);
+
+	/* cleanup. */
+	strbuf_release(&buf);
+	reader_close(&rd);
+}
+
 static void test_table_read_write_sequential(void)
 {
 	char **names;
@@ -633,6 +698,7 @@ static void test_corrupt_table(void)
 
 int readwrite_test_main(int argc, const char *argv[])
 {
+	RUN_TEST(test_log_zlib_corruption);
 	RUN_TEST(test_corrupt_table);
 	RUN_TEST(test_corrupt_table_empty);
 	RUN_TEST(test_log_write_read);
-- 
gitgitgadget

[PATCH v4 03/11] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an opened
file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v4 04/11] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index eb0b7228b0c..d628420e63a 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -814,6 +814,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v4 05/11] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index d628420e63a..4b7292945c3 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -89,7 +89,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v4 06/11] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and the next
available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index df5021ebf08..10dfd370e8e 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -877,7 +877,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -911,7 +911,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -932,7 +932,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 
 		sublock_file_fd = open(subtab_lock.buf,
 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
-		if (sublock_file_fd > 0) {
+		if (sublock_file_fd >= 0) {
 			close(sublock_file_fd);
 		} else if (sublock_file_fd < 0) {
 			if (errno == EEXIST) {
@@ -1013,7 +1013,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1050,9 +1050,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v4 07/11] reftable: all xxx_free() functions accept NULL arguments

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This fixes NULL derefs in error paths. Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 2 ++
 reftable/writer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..733509606a9 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -641,6 +641,8 @@ int reftable_new_reader(struct reftable_reader **p,
 
 void reftable_reader_free(struct reftable_reader *r)
 {
+	if (!r)
+		return;
 	reader_close(r);
 	reftable_free(r);
 }
diff --git a/reftable/writer.c b/reftable/writer.c
index 3ca721e9f64..9fd24fa93ce 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -150,6 +150,8 @@ void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,
 
 void reftable_writer_free(struct reftable_writer *w)
 {
+	if (!w)
+		return;
 	reftable_free(w->block);
 	reftable_free(w);
 }
-- 
gitgitgadget

[PATCH v4 08/11] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more practical ordering when working on refactorings of the reftable
code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..1f0a28cbb64 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	/* test from simple to complex. */
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v4 09/11] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 6e88182a83a..9b89a5da103 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -662,7 +662,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v4 10/11] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..8bbcbff1e69 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v4 11/11] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void pointer or
vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it cannot be
zero-initialized. To address this, use reftable_new_record() to return fresh
instance, given a record type. Since reftable_new_record() doesn't cause heap
allocation anymore, it should be balanced with reftable_record_release() rather
than reftable_record_destroy().

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   2 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  35 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  33 ++++----
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  31 +++----
 reftable/reader.c      |  93 +++++++++++----------
 reftable/record.c      | 173 ++++++++++++++++-----------------------
 reftable/record.h      |  42 ++++------
 reftable/record_test.c | 181 +++++++++++++++++++++--------------------
 reftable/writer.c      |  40 +++++----
 12 files changed, 317 insertions(+), 342 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 6c8e8705205..2170748c5e9 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -421,7 +421,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..4446b8ed360 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .u.ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,19 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..a7c0df91ff0 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_new_record(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,13 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..e0f5c584ada 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,14 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = {
+			.rec = {
+				.type = BLOCK_TYPE_REF,
+				.u.ref = {
+					.refname = names[i],
+				}
+			}
+		};
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +57,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 733509606a9..0790f654366 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,8 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -437,8 +436,7 @@ static int reader_start(struct reftable_reader *r, struct table_iter *ti,
 static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 			      struct reftable_record *want)
 {
-	struct reftable_record rec =
-		reftable_new_record(reftable_record_type(want));
+	struct reftable_record rec = reftable_new_record(reftable_record_type(want));
 	struct strbuf want_key = STRBUF_INIT;
 	struct strbuf got_key = STRBUF_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
@@ -475,7 +473,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -485,34 +483,36 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -540,8 +540,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -549,8 +549,7 @@ static int reader_seek_internal(struct reftable_reader *r,
 				struct reftable_iterator *it,
 				struct reftable_record *rec)
 {
-	struct reftable_reader_offsets *offs =
-		reader_offsets_for(r, reftable_record_type(rec));
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, reftable_record_type(rec));
 	uint64_t idx = offs->index_offset;
 	struct table_iter ti = TABLE_ITER_INIT;
 	int err = 0;
@@ -590,11 +589,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -602,12 +602,13 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return reader_seek(r, it, &rec);
 }
 
@@ -651,31 +652,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -683,15 +686,15 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets, got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 8bbcbff1e69..f766ee20ace 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -926,58 +930,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.is_deletion = &reftable_log_record_is_deletion_void,
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1055,91 +1007,52 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 uint8_t reftable_record_type(struct reftable_record *rec)
 {
-	return rec->ops->type;
+	return rec->type;
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1212,3 +1125,53 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_new_record(uint8_t typ)
+{
+	struct reftable_record clean_idx = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.last_key = STRBUF_INIT,
+		},
+	};
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	if (typ == BLOCK_TYPE_INDEX) {
+		return clean_idx;
+	}
+
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..407cdaba018 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -60,16 +60,10 @@ struct reftable_record_vtable {
 	int (*is_deletion)(const void *rec);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
+/* return an initialized record for the given type */
 struct reftable_record reftable_new_record(uint8_t typ);
 
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
@@ -97,6 +91,22 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_new_record() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
@@ -111,25 +121,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..96f65035c1b 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,24 +16,22 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = reftable_new_record(reftable_record_type(rec));
+
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
+	switch (copy.type) {
 	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
+		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
 						 GIT_SHA1_RAWSZ));
 		break;
 	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
+		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
 						 GIT_SHA1_RAWSZ));
 		break;
 	}
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -106,61 +104,60 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_REF
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -213,7 +210,9 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = {
+			.type = BLOCK_TYPE_LOG
+		};
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -221,23 +220,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -245,16 +246,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -322,47 +323,52 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_OBJ
+		};
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
+		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
+		if (in.u.obj.hash_prefix_len)
+			EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
+				       in.u.obj.hash_prefix_len));
+		if (in.u.obj.offset_len)
+			EXPECT(!memcmp(in.u.obj.offsets, out.u.obj.offsets,
+					   sizeof(uint64_t) * in.u.obj.offset_len));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -370,31 +376,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(in.u.idx.offset == out.u.idx.offset);
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 9fd24fa93ce..51066c36a61 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -253,8 +253,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -263,8 +265,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -303,7 +304,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -313,8 +317,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -395,8 +397,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -464,17 +468,18 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = (uint8_t *)entry->hash.buf,
+			.hash_prefix_len = arg->w->stats.object_id_len,
+			.offsets = entry->offsets,
+			.offset_len = entry->offset_len,
+		}
 	};
-	struct reftable_record rec = { NULL };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -487,7 +492,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

[PATCH v5 00/16] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'next'.

This addresses some complaints from Coverity that Peff reported.

v5:

 * add generic record equality
 * add generic record printing
 * const correctness for record-reftable.h
 * fix windows crash.
 * 0-length memcpy paranoia
 * drop unused file.

Han-Wen Nienhuys (16):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in block.c error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: all xxx_free() functions accept NULL arguments
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable-record.h function signatures const correct
  reftable: implement record equality generically
  reftable: remove outdated file reftable.c
  reftable: make reftable_record a tagged union
  reftable: add print functions to the record types
  reftable: be more paranoid about 0-length memcpy calls

 reftable/block.c           |  28 ++-
 reftable/block_test.c      |  22 +--
 reftable/blocksource.c     |   6 +-
 reftable/generic.c         |  41 ++--
 reftable/iter.c            |   4 +-
 reftable/merged.c          |  33 ++--
 reftable/pq.c              |   3 +-
 reftable/pq_test.c         |  27 ++-
 reftable/reader.c          | 113 ++++++-----
 reftable/readwrite_test.c  |  67 ++++++-
 reftable/record.c          | 376 ++++++++++++++++++++++++-------------
 reftable/record.h          |  49 ++---
 reftable/record_test.c     | 197 +++++++++----------
 reftable/reftable-record.h |  14 +-
 reftable/reftable.c        | 115 ------------
 reftable/stack.c           |  12 +-
 reftable/stack_test.c      |   3 +-
 reftable/writer.c          |  43 +++--
 t/helper/test-reftable.c   |   9 +-
 19 files changed, 629 insertions(+), 533 deletions(-)
 delete mode 100644 reftable/reftable.c


base-commit: 69a9c10c95e28df457e33b3c7400b16caf2e2962
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v5
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v4:

  1:  1c1a3ff92bd =  1:  e7f1be7bbec reftable: fix OOB stack write in print functions
  2:  923293b79b4 !  2:  9ab631a3b29 reftable: fix resource leak in block.c error path
     @@ Metadata
       ## Commit message ##
          reftable: fix resource leak in block.c error path
      
     -    Add test coverage for corrupt zlib data.
     +    Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
     +    unittest.
      
          This problem was discovered by a Coverity scan.
      
     @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable
       
       static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
      
     + ## reftable/reader.c ##
     +@@ reftable/reader.c: int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
     + 
     + 	err = reader_get_block(r, &block, next_off, guess_block_size);
     + 	if (err < 0)
     +-		return err;
     ++		goto done;
     + 
     + 	block_size = extract_block_size(block.data, &block_typ, next_off,
     + 					r->version);
     +-	if (block_size < 0)
     +-		return block_size;
     +-
     ++	if (block_size < 0) {
     ++		err = block_size;
     ++		goto done;
     ++	}
     + 	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
     +-		reftable_block_done(&block);
     +-		return 1;
     ++		err = 1;
     ++		goto done;
     + 	}
     + 
     + 	if (block_size > guess_block_size) {
     + 		reftable_block_done(&block);
     + 		err = reader_get_block(r, &block, next_off, block_size);
     + 		if (err < 0) {
     +-			return err;
     ++			goto done;
     + 		}
     + 	}
     + 
     +-	return block_reader_init(br, &block, header_off, r->block_size,
     +-				 hash_size(r->hash_id));
     ++	err = block_reader_init(br, &block, header_off, r->block_size,
     ++				hash_size(r->hash_id));
     ++done:
     ++	if (err)
     ++		reftable_block_done(&block);
     ++
     ++	return err;
     + }
     + 
     + static int table_iter_next_block(struct table_iter *dest,
     +
       ## reftable/readwrite_test.c ##
      @@ reftable/readwrite_test.c: static void test_log_write_read(void)
       	reader_close(&rd);
  3:  7a914f77756 =  3:  8301000e83b reftable: fix resource leak blocksource.c
  4:  51b4a17a2e1 =  4:  e6e53aabc39 reftable: check reftable_stack_auto_compact() return value
  5:  43989afcb5a =  5:  7551bcdd917 reftable: ignore remove() return value in stack_test.c
  6:  1c7f15d811c =  6:  700a4e247e7 reftable: fix resource warning
  7:  47ba5ddceb8 =  7:  30016fee4c8 reftable: all xxx_free() functions accept NULL arguments
  8:  aba8b8113ad =  8:  f90846cc103 reftable: order unittests by complexity
  9:  f2af404d7f1 =  9:  425e12d3667 reftable: drop stray printf in readwrite_test
 10:  d2ce1bfef9f = 10:  67d858ec59e reftable: handle null refnames in reftable_ref_record_equal
  -:  ----------- > 11:  1e50924894f reftable: make reftable-record.h function signatures const correct
  -:  ----------- > 12:  c81b17ad57e reftable: implement record equality generically
  -:  ----------- > 13:  4175089ec43 reftable: remove outdated file reftable.c
 11:  0ffc1cf0e42 ! 14:  ede2e792ab6 reftable: make reftable_record a tagged union
     @@ reftable/generic.c: void reftable_iterator_destroy(struct reftable_iterator *it)
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_ref(&rec, ref);
      -	return iterator_next(it, &rec);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_REF };
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_REF,
     ++		.u.ref = *ref,
     ++	};
      +	int err = iterator_next(it, &rec);
      +	*ref = rec.u.ref;
      +	return err;
     @@ reftable/generic.c: void reftable_iterator_destroy(struct reftable_iterator *it)
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, log);
      -	return iterator_next(it, &rec);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_LOG,
     ++		.u.log = *log,
     ++	};
      +	int err = iterator_next(it, &rec);
      +	*log = rec.u.log;
      +	return err;
     @@ reftable/merged.c: int reftable_merged_table_seek_log_at(struct reftable_merged_
      -	struct reftable_log_record log = {
      -		.refname = (char *)name,
      -		.update_index = update_index,
     -+	struct reftable_record rec = {
     -+		.type = BLOCK_TYPE_LOG,
     -+		.u.log = {
     -+			.refname = (char *)name,
     -+			.update_index = update_index,
     -+		}
     - 	};
     +-	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
     ++	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     ++				       .u.log = {
     ++					       .refname = (char *)name,
     ++					       .update_index = update_index,
     ++				       } };
       	return merged_table_seek_record(mt, it, &rec);
       }
       
     @@ reftable/pq_test.c: static void test_pq(void)
      -
      -		reftable_record_as_ref(&rec)->refname = names[i];
      -		e.rec = rec;
     -+		struct pq_entry e = {
     -+			.rec = {
     -+				.type = BLOCK_TYPE_REF,
     -+				.u.ref = {
     -+					.refname = names[i],
     -+				}
     -+			}
     -+		};
     ++		struct pq_entry e = { .rec = { .type = BLOCK_TYPE_REF,
     ++					       .u.ref = {
     ++						       .refname = names[i],
     ++					       } } };
       		merged_iter_pqueue_add(&pq, e);
       		merged_iter_pqueue_check(pq);
       		i = (i * 7) % N;
     @@ reftable/reader.c: static int table_iter_next_in_block(struct table_iter *ti,
       	}
       
       	return res;
     -@@ reftable/reader.c: static int reader_start(struct reftable_reader *r, struct table_iter *ti,
     - static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
     - 			      struct reftable_record *want)
     - {
     --	struct reftable_record rec =
     --		reftable_new_record(reftable_record_type(want));
     -+	struct reftable_record rec = reftable_new_record(reftable_record_type(want));
     - 	struct strbuf want_key = STRBUF_INIT;
     - 	struct strbuf got_key = STRBUF_INIT;
     - 	struct table_iter next = TABLE_ITER_INIT;
      @@ reftable/reader.c: static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
       
       done:
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
      -	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
      -	struct reftable_record index_result_rec = { NULL };
      +	struct reftable_record want_index = {
     -+		.type = BLOCK_TYPE_INDEX,
     -+		.u.idx = { .last_key = STRBUF_INIT }
     ++		.type = BLOCK_TYPE_INDEX, .u.idx = { .last_key = STRBUF_INIT }
      +	};
      +	struct reftable_record index_result = {
      +		.type = BLOCK_TYPE_INDEX,
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
       	return err;
       }
       
     -@@ reftable/reader.c: static int reader_seek_internal(struct reftable_reader *r,
     - 				struct reftable_iterator *it,
     - 				struct reftable_record *rec)
     - {
     --	struct reftable_reader_offsets *offs =
     --		reader_offsets_for(r, reftable_record_type(rec));
     -+	struct reftable_reader_offsets *offs = reader_offsets_for(r, reftable_record_type(rec));
     - 	uint64_t idx = offs->index_offset;
     - 	struct table_iter ti = TABLE_ITER_INIT;
     - 	int err = 0;
      @@ reftable/reader.c: static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
       int reftable_reader_seek_ref(struct reftable_reader *r,
       			     struct reftable_iterator *it, const char *name)
     @@ reftable/reader.c: int reftable_reader_seek_log_at(struct reftable_reader *r,
      -	struct reftable_log_record log = {
      -		.refname = (char *)name,
      -		.update_index = update_index,
     -+	struct reftable_record rec = {
     -+		.type = BLOCK_TYPE_LOG,
     -+		.u.log = {
     -+			.refname = (char *)name,
     -+			.update_index = update_index,
     -+		}
     - 	};
     +-	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
     ++	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     ++				       .u.log = {
     ++					       .refname = (char *)name,
     ++					       .update_index = update_index,
     ++				       } };
       	return reader_seek(r, it, &rec);
       }
       
     @@ reftable/reader.c: static int reftable_reader_refs_for_indexed(struct reftable_r
       
       	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
      -					 got.offsets, got.offset_len);
     -+					 got.u.obj.offsets, got.u.obj.offset_len);
     ++					 got.u.obj.offsets,
     ++					 got.u.obj.offset_len);
       	if (err < 0)
       		goto done;
      -	got.offsets = NULL;
     @@ reftable/record.c: https://developers.google.com/open-source/licenses/bsd
       int get_var_int(uint64_t *dest, struct string_view *in)
       {
       	int ptr = 0;
     +@@ reftable/record.c: static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
     + 		(const struct reftable_obj_record *)src_rec;
     + 
     + 	reftable_obj_record_release(obj);
     +-	*obj = *src;
     +-	obj->hash_prefix = reftable_malloc(obj->hash_prefix_len);
     +-	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
     ++	obj->hash_prefix = reftable_malloc(src->hash_prefix_len);
     ++	obj->hash_prefix_len = src->hash_prefix_len;
     ++	if (src->hash_prefix_len)
     ++		memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
     + 
     +-	obj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));
     +-	COPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);
     ++	obj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));
     ++	obj->offset_len = src->offset_len;
     ++	COPY_ARRAY(obj->offsets, src->offsets, src->offset_len);
     + }
     + 
     + static uint8_t reftable_obj_record_val_type(const void *rec)
      @@ reftable/record.c: static struct reftable_record_vtable reftable_log_record_vtable = {
     - 	.is_deletion = &reftable_log_record_is_deletion_void,
     + 	.equal = &reftable_log_record_equal_void
       };
       
      -struct reftable_record reftable_new_record(uint8_t typ)
     @@ reftable/record.c: static struct reftable_record_vtable reftable_index_record_vt
       int reftable_record_is_deletion(struct reftable_record *rec)
       {
      -	return rec->ops->is_deletion(rec->data);
     ++	return reftable_record_vtable(rec)->is_deletion(
     ++		reftable_record_data(rec));
     + }
     + 
     + int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
     + {
     +-	if (a->ops != b->ops)
     ++	if (a->type != b->type)
     + 		return 0;
     +-	return a->ops->equal(a->data, b->data, hash_size);
      -}
      -
      -void reftable_record_from_ref(struct reftable_record *rec,
     @@ reftable/record.c: static struct reftable_record_vtable reftable_index_record_vt
      -{
      -	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
      -	return rec->data;
     -+	return reftable_record_vtable(rec)->is_deletion(
     -+		reftable_record_data(rec));
     ++	return reftable_record_vtable(a)->equal(
     ++		reftable_record_data(a), reftable_record_data(b), hash_size);
       }
       
       static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
      +
      +struct reftable_record reftable_new_record(uint8_t typ)
      +{
     -+	struct reftable_record clean_idx = {
     -+		.type = BLOCK_TYPE_INDEX,
     -+		.u.idx = {
     -+			.last_key = STRBUF_INIT,
     -+		},
     -+	};
      +	struct reftable_record clean = {
      +		.type = typ,
      +	};
      +
     -+	if (typ == BLOCK_TYPE_INDEX) {
     -+		return clean_idx;
     ++	/* the following is involved, but the naive solution (just return
     ++	 * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage
     ++	 * clean.u.obj.offsets pointer on Windows VS CI.  Go figure.
     ++	 */
     ++	switch (typ) {
     ++	case BLOCK_TYPE_OBJ:
     ++	{
     ++		struct reftable_obj_record obj = { 0 };
     ++		clean.u.obj = obj;
     ++		break;
     ++	}
     ++	case BLOCK_TYPE_INDEX:
     ++	{
     ++		struct reftable_index_record idx = {
     ++			.last_key = STRBUF_INIT,
     ++		};
     ++		clean.u.idx = idx;
     ++		break;
     ++	}
     ++	case BLOCK_TYPE_REF:
     ++	{
     ++		struct reftable_ref_record ref = { 0 };
     ++		clean.u.ref = ref;
     ++		break;
     ++	}
     ++	case BLOCK_TYPE_LOG:
     ++	{
     ++		struct reftable_log_record log = { 0 };
     ++		clean.u.log = log;
     ++		break;
     ++	}
      +	}
     -+
      +	return clean;
      +}
      
       ## reftable/record.h ##
      @@ reftable/record.h: struct reftable_record_vtable {
     - 	int (*is_deletion)(const void *rec);
     + 	int (*equal)(const void *a, const void *b, int hash_size);
       };
       
      -/* record is a generic wrapper for different types of records. */
     @@ reftable/record.h: struct reftable_obj_record {
      +};
      +
       /* see struct record_vtable */
     - 
     + int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
       void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
      @@ reftable/record.h: int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
       			   int hash_size);
     @@ reftable/record_test.c
       {
      -	struct reftable_record copy =
      -		reftable_new_record(reftable_record_type(rec));
     -+	struct reftable_record copy = reftable_new_record(reftable_record_type(rec));
     ++	struct reftable_record copy = { 0 };
     ++	uint8_t typ;
      +
     ++	typ = reftable_record_type(rec);
     ++	copy = reftable_new_record(typ);
       	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
       	/* do it twice to catch memory leaks */
       	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
     --	switch (reftable_record_type(&copy)) {
     -+	switch (copy.type) {
     - 	case BLOCK_TYPE_REF:
     --		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
     --						 reftable_record_as_ref(rec),
     -+		EXPECT(reftable_ref_record_equal(&copy.u.ref, &rec->u.ref,
     - 						 GIT_SHA1_RAWSZ));
     - 		break;
     - 	case BLOCK_TYPE_LOG:
     --		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
     --						 reftable_record_as_log(rec),
     -+		EXPECT(reftable_log_record_equal(&copy.u.log, &rec->u.log,
     - 						 GIT_SHA1_RAWSZ));
     - 		break;
     - 	}
     + 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
      -	reftable_record_destroy(&copy);
      +	reftable_record_release(&copy);
       }
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
      +		struct reftable_record in = {
      +			.type = BLOCK_TYPE_REF,
      +		};
     -+		struct reftable_record out = {
     -+			.type = BLOCK_TYPE_REF
     -+		};
     ++		struct reftable_record out = { .type = BLOCK_TYPE_REF };
       		struct strbuf key = STRBUF_INIT;
      -		struct reftable_record rec = { NULL };
       		uint8_t buffer[1024] = { 0 };
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
      -			set_hash(in.value.val2.value, 1);
      -			in.value.val2.target_value =
      +			in.u.ref.value.val2.value =
     -+				reftable_malloc(GIT_SHA1_RAWSZ);
     -+			set_hash(in.u.ref.value.val2.value, 1);
     -+			in.u.ref.value.val2.target_value =
       				reftable_malloc(GIT_SHA1_RAWSZ);
      -			set_hash(in.value.val2.target_value, 2);
     ++			set_hash(in.u.ref.value.val2.value, 1);
     ++			in.u.ref.value.val2.target_value =
     ++				reftable_malloc(GIT_SHA1_RAWSZ);
      +			set_hash(in.u.ref.value.val2.target_value, 2);
       			break;
       		case REFTABLE_REF_SYMREF:
     @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
       	set_test_hash(in[0].value.update.old_hash, 2);
       	for (i = 0; i < ARRAY_SIZE(in); i++) {
      -		struct reftable_record rec = { NULL };
     -+		struct reftable_record rec = {
     -+			.type = BLOCK_TYPE_LOG
     -+		};
     ++		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
       		struct strbuf key = STRBUF_INIT;
       		uint8_t buffer[1024] = { 0 };
       		struct string_view dest = {
     @@ reftable/record_test.c: static void test_reftable_obj_record_roundtrip(void)
       		struct strbuf key = STRBUF_INIT;
      -		struct reftable_obj_record out = { NULL };
      -		struct reftable_record rec_out = { NULL };
     -+		struct reftable_record out = {
     -+			.type = BLOCK_TYPE_OBJ
     -+		};
     ++		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
       		int n, m;
       		uint8_t extra;
       
     @@ reftable/record_test.c: static void test_reftable_obj_record_roundtrip(void)
       					   GIT_SHA1_RAWSZ);
       		EXPECT(n == m);
       
     --		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
     --		EXPECT(in.offset_len == out.offset_len);
     --
     --		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
     --			       in.hash_prefix_len));
     --		EXPECT(0 == memcmp(in.offsets, out.offsets,
     --				   sizeof(uint64_t) * in.offset_len));
     -+		EXPECT(in.u.obj.hash_prefix_len == out.u.obj.hash_prefix_len);
     -+		EXPECT(in.u.obj.offset_len == out.u.obj.offset_len);
     -+		if (in.u.obj.hash_prefix_len)
     -+			EXPECT(!memcmp(in.u.obj.hash_prefix, out.u.obj.hash_prefix,
     -+				       in.u.obj.hash_prefix_len));
     -+		if (in.u.obj.offset_len)
     -+			EXPECT(!memcmp(in.u.obj.offsets, out.u.obj.offsets,
     -+					   sizeof(uint64_t) * in.u.obj.offset_len));
     +-		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
     ++		EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
       		strbuf_release(&key);
      -		reftable_record_release(&rec_out);
      +		reftable_record_release(&out);
     @@ reftable/record_test.c: static void test_reftable_index_record_roundtrip(void)
      +	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
       	EXPECT(m == n);
       
     --	EXPECT(in.offset == out.offset);
     -+	EXPECT(in.u.idx.offset == out.u.idx.offset);
     +-	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
     ++	EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
       
      -	reftable_record_release(&out_rec);
      +	reftable_record_release(&out);
     @@ reftable/writer.c: static void write_object_record(void *void_arg, void *key)
      -		.hash_prefix_len = arg->w->stats.object_id_len,
      -		.offsets = entry->offsets,
      -		.offset_len = entry->offset_len,
     -+	struct reftable_record rec = {
     -+		.type = BLOCK_TYPE_OBJ,
     -+		.u.obj = {
     -+			.hash_prefix = (uint8_t *)entry->hash.buf,
     -+			.hash_prefix_len = arg->w->stats.object_id_len,
     -+			.offsets = entry->offsets,
     -+			.offset_len = entry->offset_len,
     -+		}
     - 	};
     +-	};
      -	struct reftable_record rec = { NULL };
     ++	struct reftable_record
     ++		rec = { .type = BLOCK_TYPE_OBJ,
     ++			.u.obj = {
     ++				.hash_prefix = (uint8_t *)entry->hash.buf,
     ++				.hash_prefix_len = arg->w->stats.object_id_len,
     ++				.offsets = entry->offsets,
     ++				.offset_len = entry->offset_len,
     ++			} };
       	if (arg->err < 0)
       		goto done;
       
  -:  ----------- > 15:  6c06bd91662 reftable: add print functions to the record types
  -:  ----------- > 16:  e16bf0c5212 reftable: be more paranoid about 0-length memcpy calls

-- 
gitgitgadget

[PATCH v5 01/16] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
unittest.

This problem was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c          | 26 +++++++++------
 reftable/reader.c         | 24 ++++++++------
 reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 98 insertions(+), 18 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..6c8e8705205 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +206,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -212,16 +215,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +257,9 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	reftable_free(uncompressed);
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..0d16b098f5e 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -290,28 +290,34 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
 
 	err = reader_get_block(r, &block, next_off, guess_block_size);
 	if (err < 0)
-		return err;
+		goto done;
 
 	block_size = extract_block_size(block.data, &block_typ, next_off,
 					r->version);
-	if (block_size < 0)
-		return block_size;
-
+	if (block_size < 0) {
+		err = block_size;
+		goto done;
+	}
 	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
-		reftable_block_done(&block);
-		return 1;
+		err = 1;
+		goto done;
 	}
 
 	if (block_size > guess_block_size) {
 		reftable_block_done(&block);
 		err = reader_get_block(r, &block, next_off, block_size);
 		if (err < 0) {
-			return err;
+			goto done;
 		}
 	}
 
-	return block_reader_init(br, &block, header_off, r->block_size,
-				 hash_size(r->hash_id));
+	err = block_reader_init(br, &block, header_off, r->block_size,
+				hash_size(r->hash_id));
+done:
+	if (err)
+		reftable_block_done(&block);
+
+	return err;
 }
 
 static int table_iter_next_block(struct table_iter *dest,
diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 5f6bcc2f775..6e88182a83a 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -254,6 +254,71 @@ static void test_log_write_read(void)
 	reader_close(&rd);
 }
 
+static void test_log_zlib_corruption(void)
+{
+	struct reftable_write_options opts = {
+		.block_size = 256,
+	};
+	struct reftable_iterator it = { 0 };
+	struct reftable_reader rd = { 0 };
+	struct reftable_block_source source = { 0 };
+	struct strbuf buf = STRBUF_INIT;
+	struct reftable_writer *w =
+		reftable_new_writer(&strbuf_add_void, &buf, &opts);
+	const struct reftable_stats *stats = NULL;
+	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
+	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
+	char message[100] = { 0 };
+	int err, i, n;
+
+	struct reftable_log_record log = {
+		.refname = "refname",
+		.value_type = REFTABLE_LOG_UPDATE,
+		.value = {
+			.update = {
+				.new_hash = hash1,
+				.old_hash = hash2,
+				.name = "My Name",
+				.email = "myname@invalid",
+				.message = message,
+			},
+		},
+	};
+
+	for (i = 0; i < sizeof(message)-1; i++)
+		message[i] = (uint8_t)(rand() % 64 + ' ');
+
+	reftable_writer_set_limits(w, 1, 1);
+
+	err = reftable_writer_add_log(w, &log);
+	EXPECT_ERR(err);
+
+	n = reftable_writer_close(w);
+	EXPECT(n == 0);
+
+	stats = writer_stats(w);
+	EXPECT(stats->log_stats.blocks > 0);
+	reftable_writer_free(w);
+	w = NULL;
+
+	/* corrupt the data. */
+	buf.buf[50] ^= 0x99;
+
+	block_source_from_strbuf(&source, &buf);
+
+	err = init_reader(&rd, &source, "file.log");
+	EXPECT_ERR(err);
+
+	err = reftable_reader_seek_log(&rd, &it, "refname");
+	EXPECT(err == REFTABLE_ZLIB_ERROR);
+
+	reftable_iterator_destroy(&it);
+
+	/* cleanup. */
+	strbuf_release(&buf);
+	reader_close(&rd);
+}
+
 static void test_table_read_write_sequential(void)
 {
 	char **names;
@@ -633,6 +698,7 @@ static void test_corrupt_table(void)
 
 int readwrite_test_main(int argc, const char *argv[])
 {
+	RUN_TEST(test_log_zlib_corruption);
 	RUN_TEST(test_corrupt_table);
 	RUN_TEST(test_corrupt_table_empty);
 	RUN_TEST(test_log_write_read);
-- 
gitgitgadget

[PATCH v5 03/16] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an opened
file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v5 04/16] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index eb0b7228b0c..d628420e63a 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -814,6 +814,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v5 05/16] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index d628420e63a..4b7292945c3 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -89,7 +89,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v5 06/16] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and the next
available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index df5021ebf08..10dfd370e8e 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -877,7 +877,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -911,7 +911,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -932,7 +932,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 
 		sublock_file_fd = open(subtab_lock.buf,
 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
-		if (sublock_file_fd > 0) {
+		if (sublock_file_fd >= 0) {
 			close(sublock_file_fd);
 		} else if (sublock_file_fd < 0) {
 			if (errno == EEXIST) {
@@ -1013,7 +1013,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1050,9 +1050,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v5 07/16] reftable: all xxx_free() functions accept NULL arguments

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This fixes NULL derefs in error paths. Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 2 ++
 reftable/writer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/reftable/reader.c b/reftable/reader.c
index 0d16b098f5e..c7a32331c76 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -647,6 +647,8 @@ int reftable_new_reader(struct reftable_reader **p,
 
 void reftable_reader_free(struct reftable_reader *r)
 {
+	if (!r)
+		return;
 	reader_close(r);
 	reftable_free(r);
 }
diff --git a/reftable/writer.c b/reftable/writer.c
index 3ca721e9f64..9fd24fa93ce 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -150,6 +150,8 @@ void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,
 
 void reftable_writer_free(struct reftable_writer *w)
 {
+	if (!w)
+		return;
 	reftable_free(w->block);
 	reftable_free(w);
 }
-- 
gitgitgadget

[PATCH v5 08/16] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more practical ordering when working on refactorings of the reftable
code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..1f0a28cbb64 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	/* test from simple to complex. */
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v5 09/16] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 6e88182a83a..9b89a5da103 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -662,7 +662,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v5 10/16] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..8bbcbff1e69 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v5 11/16] reftable: make reftable-record.h function signatures const correct

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c          | 14 +++++++-------
 reftable/reftable-record.h | 14 +++++++-------
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8bbcbff1e69..f7c77c51539 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -72,7 +72,7 @@ int reftable_is_block_type(uint8_t typ)
 	return 0;
 }
 
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL1:
@@ -84,7 +84,7 @@ uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
 	}
 }
 
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL2:
@@ -251,7 +251,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
@@ -881,8 +881,8 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size)
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size)
 {
 	if (!(null_streq(a->refname, b->refname) &&
 	      a->update_index == b->update_index &&
@@ -1150,8 +1150,8 @@ static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
 	return a == b;
 }
 
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size)
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
 	if (!null_streq(a->refname, b->refname))
diff --git a/reftable/reftable-record.h b/reftable/reftable-record.h
index 5370d2288c7..67104f8fbfe 100644
--- a/reftable/reftable-record.h
+++ b/reftable/reftable-record.h
@@ -49,25 +49,25 @@ struct reftable_ref_record {
 
 /* Returns the first hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec);
 
 /* Returns the second hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec);
 
 /* returns whether 'ref' represents a deletion */
 int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);
 
 /* prints a reftable_ref_record onto stdout. Useful for debugging. */
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id);
 
 /* frees and nulls all pointer values inside `ref`. */
 void reftable_ref_record_release(struct reftable_ref_record *ref);
 
 /* returns whether two reftable_ref_records are the same. Useful for testing. */
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size);
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size);
 
 /* reftable_log_record holds a reflog entry */
 struct reftable_log_record {
@@ -104,8 +104,8 @@ int reftable_log_record_is_deletion(const struct reftable_log_record *log);
 void reftable_log_record_release(struct reftable_log_record *log);
 
 /* returns whether two records are equal. Useful for testing. */
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size);
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size);
 
 /* dumps a reftable_log_record on stdout, for debugging/testing. */
 void reftable_log_record_print(struct reftable_log_record *log,
-- 
gitgitgadget

[PATCH v5 12/16] reftable: implement record equality generically

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This simplifies unittests a little, and provides further coverage for
reftable_record_copy().

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 57 +++++++++++++++++++++++++++++++++++++++++-
 reftable/record.h      |  5 +++-
 reftable/record_test.c | 23 +++--------------
 3 files changed, 63 insertions(+), 22 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index f7c77c51539..2a9e41a992e 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -430,6 +430,15 @@ static int reftable_ref_record_is_deletion_void(const void *p)
 		(const struct reftable_ref_record *)p);
 }
 
+
+static int reftable_ref_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	struct reftable_ref_record *ra = (struct reftable_ref_record *) a;
+	struct reftable_ref_record *rb = (struct reftable_ref_record *) b;
+	return reftable_ref_record_equal(ra, rb, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -439,6 +448,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.decode = &reftable_ref_record_decode,
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
+	.equal = &reftable_ref_record_equal_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -572,6 +582,25 @@ static int not_a_deletion(const void *p)
 	return 0;
 }
 
+static int reftable_obj_record_equal_void(const void *a, const void *b, int hash_size)
+{
+	struct reftable_obj_record *ra = (struct reftable_obj_record *) a;
+	struct reftable_obj_record *rb = (struct reftable_obj_record *) b;
+
+	if (ra->hash_prefix_len != rb->hash_prefix_len
+	    || ra->offset_len != rb->offset_len)
+		return 0;
+
+	if (ra->hash_prefix_len &&
+	    memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
+		return 0;
+	if (ra->offset_len &&
+	    memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
+		return 0;
+
+	return 1;
+}
+
 static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.key = &reftable_obj_record_key,
 	.type = BLOCK_TYPE_OBJ,
@@ -580,7 +609,8 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.encode = &reftable_obj_record_encode,
 	.decode = &reftable_obj_record_decode,
 	.release = &reftable_obj_record_release,
-	.is_deletion = not_a_deletion,
+	.is_deletion = &not_a_deletion,
+	.equal = &reftable_obj_record_equal_void,
 };
 
 void reftable_log_record_print(struct reftable_log_record *log,
@@ -881,6 +911,14 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
+static int reftable_log_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	return reftable_log_record_equal((struct reftable_log_record *) a,
+					 (struct reftable_log_record *) b,
+					 hash_size);
+}
+
 int reftable_log_record_equal(const struct reftable_log_record *a,
 			      const struct reftable_log_record *b, int hash_size)
 {
@@ -924,6 +962,7 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
+	.equal = &reftable_log_record_equal_void
 };
 
 struct reftable_record reftable_new_record(uint8_t typ)
@@ -1042,6 +1081,14 @@ static int reftable_index_record_decode(void *rec, struct strbuf key,
 	return start.len - in.len;
 }
 
+static int reftable_index_record_equal(const void *a, const void *b, int hash_size)
+{
+	struct reftable_index_record *ia = (struct reftable_index_record *) a;
+	struct reftable_index_record *ib = (struct reftable_index_record *) b;
+
+	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1051,6 +1098,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.decode = &reftable_index_record_decode,
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
+	.equal = &reftable_index_record_equal,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1098,6 +1146,13 @@ int reftable_record_is_deletion(struct reftable_record *rec)
 	return rec->ops->is_deletion(rec->data);
 }
 
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
+{
+	if (a->ops != b->ops)
+		return 0;
+	return a->ops->equal(a->data, b->data, hash_size);
+}
+
 void reftable_record_from_ref(struct reftable_record *rec,
 			      struct reftable_ref_record *ref_rec)
 {
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..da75d7d1f11 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -58,6 +58,9 @@ struct reftable_record_vtable {
 
 	/* is this a tombstone? */
 	int (*is_deletion)(const void *rec);
+
+	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
+	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
 /* record is a generic wrapper for different types of records. */
@@ -98,7 +101,7 @@ struct reftable_obj_record {
 };
 
 /* see struct record_vtable */
-
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..92680848156 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -21,18 +21,7 @@ static void test_copy(struct reftable_record *rec)
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
-	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	}
+	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
 	reftable_record_destroy(&copy);
 }
 
@@ -346,13 +335,7 @@ static void test_reftable_obj_record_roundtrip(void)
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
 		reftable_record_release(&rec_out);
 	}
@@ -390,7 +373,7 @@ static void test_reftable_index_record_roundtrip(void)
 	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
 
 	reftable_record_release(&out_rec);
 	strbuf_release(&key);
-- 
gitgitgadget

[PATCH v5 13/16] reftable: remove outdated file reftable.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This was renamed to generic.c, but the origin was never removed

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reftable.c | 115 --------------------------------------------
 1 file changed, 115 deletions(-)
 delete mode 100644 reftable/reftable.c

diff --git a/reftable/reftable.c b/reftable/reftable.c
deleted file mode 100644
index 0e4607a7cd6..00000000000
--- a/reftable/reftable.c
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
-Copyright 2020 Google LLC
-
-Use of this source code is governed by a BSD-style
-license that can be found in the LICENSE file or at
-https://developers.google.com/open-source/licenses/bsd
-*/
-
-#include "basics.h"
-#include "record.h"
-#include "generic.h"
-#include "reftable-iterator.h"
-#include "reftable-generic.h"
-
-int reftable_table_seek_ref(struct reftable_table *tab,
-			    struct reftable_iterator *it, const char *name)
-{
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
-	return tab->ops->seek_record(tab->table_arg, it, &rec);
-}
-
-int reftable_table_read_ref(struct reftable_table *tab, const char *name,
-			    struct reftable_ref_record *ref)
-{
-	struct reftable_iterator it = { NULL };
-	int err = reftable_table_seek_ref(tab, &it, name);
-	if (err)
-		goto done;
-
-	err = reftable_iterator_next_ref(&it, ref);
-	if (err)
-		goto done;
-
-	if (strcmp(ref->refname, name) ||
-	    reftable_ref_record_is_deletion(ref)) {
-		reftable_ref_record_release(ref);
-		err = 1;
-		goto done;
-	}
-
-done:
-	reftable_iterator_destroy(&it);
-	return err;
-}
-
-uint64_t reftable_table_max_update_index(struct reftable_table *tab)
-{
-	return tab->ops->max_update_index(tab->table_arg);
-}
-
-uint64_t reftable_table_min_update_index(struct reftable_table *tab)
-{
-	return tab->ops->min_update_index(tab->table_arg);
-}
-
-uint32_t reftable_table_hash_id(struct reftable_table *tab)
-{
-	return tab->ops->hash_id(tab->table_arg);
-}
-
-void reftable_iterator_destroy(struct reftable_iterator *it)
-{
-	if (!it->ops) {
-		return;
-	}
-	it->ops->close(it->iter_arg);
-	it->ops = NULL;
-	FREE_AND_NULL(it->iter_arg);
-}
-
-int reftable_iterator_next_ref(struct reftable_iterator *it,
-			       struct reftable_ref_record *ref)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
-}
-
-int reftable_iterator_next_log(struct reftable_iterator *it,
-			       struct reftable_log_record *log)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
-}
-
-int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
-{
-	return it->ops->next(it->iter_arg, rec);
-}
-
-static int empty_iterator_next(void *arg, struct reftable_record *rec)
-{
-	return 1;
-}
-
-static void empty_iterator_close(void *arg)
-{
-}
-
-static struct reftable_iterator_vtable empty_vtable = {
-	.next = &empty_iterator_next,
-	.close = &empty_iterator_close,
-};
-
-void iterator_set_empty(struct reftable_iterator *it)
-{
-	assert(!it->ops);
-	it->iter_arg = NULL;
-	it->ops = &empty_vtable;
-}
-- 
gitgitgadget

[PATCH v5 14/16] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void pointer or
vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it cannot be
zero-initialized. To address this, use reftable_new_record() to return fresh
instance, given a record type. Since reftable_new_record() doesn't cause heap
allocation anymore, it should be balanced with reftable_record_release() rather
than reftable_record_destroy().

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   2 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  41 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  33 ++++---
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  27 +++---
 reftable/reader.c      |  87 +++++++++--------
 reftable/record.c      | 212 +++++++++++++++++++----------------------
 reftable/record.h      |  42 ++++----
 reftable/record_test.c | 157 +++++++++++++++---------------
 reftable/writer.c      |  41 ++++----
 12 files changed, 334 insertions(+), 337 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 6c8e8705205..2170748c5e9 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -421,7 +421,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..b27d152e89a 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .u.ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,25 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..2a6efa110d5 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_new_record(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,11 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = update_index,
+				       } };
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..7de5e886f35 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,10 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = { .rec = { .type = BLOCK_TYPE_REF,
+					       .u.ref = {
+						       .refname = names[i],
+					       } } };
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +53,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index c7a32331c76..a537af5f9a8 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,8 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -481,7 +480,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -491,34 +490,35 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX, .u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -546,8 +546,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -596,11 +596,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -608,12 +609,11 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = update_index,
+				       } };
 	return reader_seek(r, it, &rec);
 }
 
@@ -657,31 +657,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -689,15 +691,16 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets,
+					 got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 2a9e41a992e..a8cee628942 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -475,12 +479,14 @@ static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 		(const struct reftable_obj_record *)src_rec;
 
 	reftable_obj_record_release(obj);
-	*obj = *src;
-	obj->hash_prefix = reftable_malloc(obj->hash_prefix_len);
-	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
+	obj->hash_prefix = reftable_malloc(src->hash_prefix_len);
+	obj->hash_prefix_len = src->hash_prefix_len;
+	if (src->hash_prefix_len)
+		memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
 
-	obj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));
-	COPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);
+	obj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));
+	obj->offset_len = src->offset_len;
+	COPY_ARRAY(obj->offsets, src->offsets, src->offset_len);
 }
 
 static uint8_t reftable_obj_record_val_type(const void *rec)
@@ -965,58 +971,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.equal = &reftable_log_record_equal_void
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1103,98 +1057,60 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 uint8_t reftable_record_type(struct reftable_record *rec)
 {
-	return rec->ops->type;
+	return rec->type;
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
 {
-	if (a->ops != b->ops)
+	if (a->type != b->type)
 		return 0;
-	return a->ops->equal(a->data, b->data, hash_size);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(a)->equal(
+		reftable_record_data(a), reftable_record_data(b), hash_size);
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1267,3 +1183,75 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_new_record(uint8_t typ)
+{
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	/* the following is involved, but the naive solution (just return
+	 * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage
+	 * clean.u.obj.offsets pointer on Windows VS CI.  Go figure.
+	 */
+	switch (typ) {
+	case BLOCK_TYPE_OBJ:
+	{
+		struct reftable_obj_record obj = { 0 };
+		clean.u.obj = obj;
+		break;
+	}
+	case BLOCK_TYPE_INDEX:
+	{
+		struct reftable_index_record idx = {
+			.last_key = STRBUF_INIT,
+		};
+		clean.u.idx = idx;
+		break;
+	}
+	case BLOCK_TYPE_REF:
+	{
+		struct reftable_ref_record ref = { 0 };
+		clean.u.ref = ref;
+		break;
+	}
+	case BLOCK_TYPE_LOG:
+	{
+		struct reftable_log_record log = { 0 };
+		clean.u.log = log;
+		break;
+	}
+	}
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index da75d7d1f11..010a322e901 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -63,16 +63,10 @@ struct reftable_record_vtable {
 	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
+/* return an initialized record for the given type */
 struct reftable_record reftable_new_record(uint8_t typ);
 
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
@@ -100,6 +94,22 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_new_record() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
@@ -114,25 +124,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 92680848156..c6fdd1925a9 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,13 +16,16 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = { 0 };
+	uint8_t typ;
+
+	typ = reftable_record_type(rec);
+	copy = reftable_new_record(typ);
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -95,61 +98,58 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = { .type = BLOCK_TYPE_REF };
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -202,7 +202,7 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -210,23 +210,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -234,16 +236,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -311,41 +313,43 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -353,31 +357,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
+	EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 9fd24fa93ce..02b74d25c0f 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -253,8 +253,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -263,8 +265,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -303,7 +304,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -313,8 +317,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -395,8 +397,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -464,17 +468,17 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
-	};
-	struct reftable_record rec = { NULL };
+	struct reftable_record
+		rec = { .type = BLOCK_TYPE_OBJ,
+			.u.obj = {
+				.hash_prefix = (uint8_t *)entry->hash.buf,
+				.hash_prefix_len = arg->w->stats.object_id_len,
+				.offsets = entry->offsets,
+				.offset_len = entry->offset_len,
+			} };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -487,7 +491,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

[PATCH v5 15/16] reftable: add print functions to the record types

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This isn't used per se, but it is useful for debugging, especially
Windows CI failures.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 85 +++++++++++++++++++++++++++++++++++-------
 reftable/record.h      |  4 ++
 reftable/record_test.c | 21 ++++++++++-
 3 files changed, 95 insertions(+), 15 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index a8cee628942..fbaa1fbef56 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -255,8 +255,8 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(const struct reftable_ref_record *ref,
-			       uint32_t hash_id)
+static void reftable_ref_record_print_sz(const struct reftable_ref_record *ref,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
@@ -265,14 +265,14 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 		printf("=> %s", ref->value.symref);
 		break;
 	case REFTABLE_REF_VAL2:
-		hex_format(hex, ref->value.val2.value, hash_size(hash_id));
+		hex_format(hex, ref->value.val2.value, hash_size);
 		printf("val 2 %s", hex);
 		hex_format(hex, ref->value.val2.target_value,
-			   hash_size(hash_id));
+			   hash_size);
 		printf("(T %s)", hex);
 		break;
 	case REFTABLE_REF_VAL1:
-		hex_format(hex, ref->value.val1, hash_size(hash_id));
+		hex_format(hex, ref->value.val1, hash_size);
 		printf("val 1 %s", hex);
 		break;
 	case REFTABLE_REF_DELETION:
@@ -282,6 +282,11 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 	printf("}\n");
 }
 
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
+			       uint32_t hash_id) {
+	reftable_ref_record_print_sz(ref, hash_size(hash_id));
+}
+
 static void reftable_ref_record_release_void(void *rec)
 {
 	reftable_ref_record_release(rec);
@@ -443,6 +448,12 @@ static int reftable_ref_record_equal_void(const void *a,
 	return reftable_ref_record_equal(ra, rb, hash_size);
 }
 
+static void reftable_ref_record_print_void(const void *rec,
+					   int hash_size)
+{
+	reftable_ref_record_print_sz((struct reftable_ref_record *) rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -453,6 +464,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
 	.equal = &reftable_ref_record_equal_void,
+	.print = &reftable_ref_record_print_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -471,6 +483,21 @@ static void reftable_obj_record_release(void *rec)
 	memset(obj, 0, sizeof(struct reftable_obj_record));
 }
 
+static void reftable_obj_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_obj_record *obj = rec;
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
+	struct strbuf offset_str = STRBUF_INIT;
+	int i;
+
+	for (i = 0; i < obj->offset_len; i++)
+		strbuf_addf(&offset_str, "%" PRIu64 " ", obj->offsets[i]);
+	hex_format(hex, obj->hash_prefix, obj->hash_prefix_len);
+	printf("prefix %s (len %d), offsets [%s]\n",
+	       hex, obj->hash_prefix_len, offset_str.buf);
+	strbuf_release(&offset_str);
+}
+
 static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 					  int hash_size)
 {
@@ -617,31 +644,41 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.release = &reftable_obj_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_obj_record_equal_void,
+	.print = &reftable_obj_record_print,
 };
 
-void reftable_log_record_print(struct reftable_log_record *log,
-			       uint32_t hash_id)
+static void reftable_log_record_print_sz(struct reftable_log_record *log,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-		printf("log{%s(%" PRIu64 ") delete", log->refname,
+		printf("log{%s(%" PRIu64 ") delete\n", log->refname,
 		       log->update_index);
 		break;
 	case REFTABLE_LOG_UPDATE:
 		printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
-		       log->refname, log->update_index, log->value.update.name,
-		       log->value.update.email, log->value.update.time,
+		       log->refname, log->update_index,
+		       log->value.update.name ? log->value.update.name : "",
+		       log->value.update.email ? log->value.update.email : "",
+		       log->value.update.time,
 		       log->value.update.tz_offset);
-		hex_format(hex, log->value.update.old_hash, hash_size(hash_id));
+		hex_format(hex, log->value.update.old_hash, hash_size);
 		printf("%s => ", hex);
-		hex_format(hex, log->value.update.new_hash, hash_size(hash_id));
-		printf("%s\n\n%s\n}\n", hex, log->value.update.message);
+		hex_format(hex, log->value.update.new_hash, hash_size);
+		printf("%s\n\n%s\n}\n", hex,
+		       log->value.update.message ? log->value.update.message : "");
 		break;
 	}
 }
 
+void reftable_log_record_print(struct reftable_log_record *log,
+				      uint32_t hash_id)
+{
+	reftable_log_record_print_sz(log, hash_size(hash_id));
+}
+
 static void reftable_log_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_log_record *rec =
@@ -959,6 +996,11 @@ static int reftable_log_record_is_deletion_void(const void *p)
 		(const struct reftable_log_record *)p);
 }
 
+static void reftable_log_record_print_void(const void *rec, int hash_size)
+{
+	reftable_log_record_print_sz((struct reftable_log_record*)rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_log_record_vtable = {
 	.key = &reftable_log_record_key,
 	.type = BLOCK_TYPE_LOG,
@@ -968,7 +1010,8 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
-	.equal = &reftable_log_record_equal_void
+	.equal = &reftable_log_record_equal_void,
+	.print = &reftable_log_record_print_void,
 };
 
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
@@ -1043,6 +1086,13 @@ static int reftable_index_record_equal(const void *a, const void *b, int hash_si
 	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
 }
 
+static void reftable_index_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_index_record *idx = rec;
+	/* TODO: escape null chars? */
+	printf("\"%s\" %" PRIu64 "\n", idx->last_key.buf, idx->offset);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1053,6 +1103,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_index_record_equal,
+	.print = &reftable_index_record_print,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1255,3 +1306,9 @@ struct reftable_record reftable_new_record(uint8_t typ)
 	}
 	return clean;
 }
+
+void reftable_record_print(struct reftable_record *rec, int hash_size)
+{
+	printf("'%c': ", rec->type);
+	reftable_record_vtable(rec)->print(reftable_record_data(rec), hash_size);
+}
diff --git a/reftable/record.h b/reftable/record.h
index 010a322e901..fd80cd451d5 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -61,6 +61,9 @@ struct reftable_record_vtable {
 
 	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
 	int (*equal)(const void *a, const void *b, int hash_size);
+
+	/* Print on stdout, for debugging. */
+	void (*print)(const void *rec, int hash_size);
 };
 
 /* returns true for recognized block types. Block start with the block type. */
@@ -112,6 +115,7 @@ struct reftable_record {
 
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
+void reftable_record_print(struct reftable_record *rec, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index c6fdd1925a9..f91ea5e8830 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -25,6 +25,10 @@ static void test_copy(struct reftable_record *rec)
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
+
+	puts("testing print coverage:\n");
+	reftable_record_print(&copy, GIT_SHA1_RAWSZ);
+
 	reftable_record_release(&copy);
 }
 
@@ -176,7 +180,8 @@ static void test_reftable_log_record_equal(void)
 static void test_reftable_log_record_roundtrip(void)
 {
 	int i;
-	struct reftable_log_record in[2] = {
+
+	struct reftable_log_record in[] = {
 		{
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 42,
@@ -197,10 +202,24 @@ static void test_reftable_log_record_roundtrip(void)
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 22,
 			.value_type = REFTABLE_LOG_DELETION,
+		},
+		{
+			.refname = xstrdup("branch"),
+			.update_index = 33,
+			.value_type = REFTABLE_LOG_UPDATE,
+			.value = {
+				.update = {
+					.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+					.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+					/* rest of fields left empty. */
+				},
+			},
 		}
 	};
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
+	set_test_hash(in[2].value.update.new_hash, 3);
+	set_test_hash(in[2].value.update.old_hash, 4);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
 		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
-- 
gitgitgadget

[PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index fbaa1fbef56..423e687b220 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -126,7 +126,8 @@ static int encode_string(char *str, struct string_view s)
 	string_view_consume(&s, n);
 	if (s.len < l)
 		return -1;
-	memcpy(s.buf, str, l);
+	if (l)
+		memcpy(s.buf, str, l);
 	string_view_consume(&s, l);
 
 	return start.len - s.len;
@@ -153,7 +154,9 @@ int reftable_encode_key(int *restart, struct string_view dest,
 
 	if (dest.len < suffix_len)
 		return -1;
-	memcpy(dest.buf, key.buf + prefix_len, suffix_len);
+
+	if (suffix_len)
+		memcpy(dest.buf, key.buf + prefix_len, suffix_len);
 	string_view_consume(&dest, suffix_len);
 
 	return start.len - dest.len;
@@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
 	uint64_t last;
 	int j;
 	r->hash_prefix = reftable_malloc(key.len);
-	memcpy(r->hash_prefix, key.buf, key.len);
+	if (key.len)
+		memcpy(r->hash_prefix, key.buf, key.len);
 	r->hash_prefix_len = key.len;
 
 	if (val_type == 0) {
-- 
gitgitgadget

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index fbaa1fbef56..423e687b220 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -126,7 +126,8 @@ static int encode_string(char *str, struct string_view s)
>  	string_view_consume(&s, n);
>  	if (s.len < l)
>  		return -1;
> -	memcpy(s.buf, str, l);
> +	if (l)
> +		memcpy(s.buf, str, l);
>  	string_view_consume(&s, l);
>  
>  	return start.len - s.len;
> @@ -153,7 +154,9 @@ int reftable_encode_key(int *restart, struct string_view dest,
>  
>  	if (dest.len < suffix_len)
>  		return -1;
> -	memcpy(dest.buf, key.buf + prefix_len, suffix_len);
> +
> +	if (suffix_len)
> +		memcpy(dest.buf, key.buf + prefix_len, suffix_len);
>  	string_view_consume(&dest, suffix_len);
>  
>  	return start.len - dest.len;
> @@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
>  	uint64_t last;
>  	int j;
>  	r->hash_prefix = reftable_malloc(key.len);
> -	memcpy(r->hash_prefix, key.buf, key.len);
> +	if (key.len)
> +		memcpy(r->hash_prefix, key.buf, key.len);
>  	r->hash_prefix_len = key.len;
>  
>  	if (val_type == 0) {

I am not sure why any of these are needed.

For a code path that involves a <ptr, len> pair, where ptr is lazily
allocated only when we have contents to store, i.e. ptr can remain
NULL until len becomes non-zero, memcpy(dst, ptr, len) can become a
problem as the standard requires ptr to be a valid even when len is
0 (ISO/IEC 9899:1999, 7.21.1 String function conventions, paragraph
2), but none of these three calls to memcpy() do any such thing.

Puzzled.

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> diff --git a/reftable/reader.c b/reftable/reader.c
> index 006709a645a..0d16b098f5e 100644
> --- a/reftable/reader.c
> +++ b/reftable/reader.c
> @@ -290,28 +290,34 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
>  
>  	err = reader_get_block(r, &block, next_off, guess_block_size);
>  	if (err < 0)
> -		return err;
> +		goto done;
>  
>  	block_size = extract_block_size(block.data, &block_typ, next_off,
>  					r->version);
> -	if (block_size < 0)
> -		return block_size;
> -
> +	if (block_size < 0) {
> +		err = block_size;
> +		goto done;
> +	}
>  	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
> -		reftable_block_done(&block);
> -		return 1;
> +		err = 1;
> +		goto done;
>  	}
>  
>  	if (block_size > guess_block_size) {
>  		reftable_block_done(&block);
>  		err = reader_get_block(r, &block, next_off, block_size);
>  		if (err < 0) {
> -			return err;
> +			goto done;
>  		}
>  	}
>  
> -	return block_reader_init(br, &block, header_off, r->block_size,
> -				 hash_size(r->hash_id));
> +	err = block_reader_init(br, &block, header_off, r->block_size,
> +				hash_size(r->hash_id));
> +done:
> +	if (err)

Is the convention for reader_init() different from all other
functions?  It makes reader wonder why this is not

	if (err < 0)

even though it is not wrong per-se (as long as "zero means success"
is a part of the return value convention).

> +		reftable_block_done(&block);
> +
> +	return err;
>  }

This one is new in this round.  All look good, other than that one
check for error return.

> diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
> index 5f6bcc2f775..6e88182a83a 100644
> --- a/reftable/readwrite_test.c
> +++ b/reftable/readwrite_test.c
> @@ -254,6 +254,71 @@ static void test_log_write_read(void)
>  	reader_close(&rd);
>  }
>  
> +static void test_log_zlib_corruption(void)
> +{
> +	struct reftable_write_options opts = {
> +		.block_size = 256,
> +	};
> +	struct reftable_iterator it = { 0 };
> +	struct reftable_reader rd = { 0 };
> +	struct reftable_block_source source = { 0 };
> +	struct strbuf buf = STRBUF_INIT;
> +	struct reftable_writer *w =
> +		reftable_new_writer(&strbuf_add_void, &buf, &opts);
> +	const struct reftable_stats *stats = NULL;
> +	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
> +	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };

Will this code be exercised when compiling with SHA256 support?  If
not, this is perfectly fine, but otherwise, this needs to be MAX,
not SHA1, no?

> +	char message[100] = { 0 };

You're filling this to the sizeof(message)-1, so we can afford to
leave it uninitialized.

> +	int err, i, n;
> +
> +	struct reftable_log_record log = {
> +		.refname = "refname",
> +		.value_type = REFTABLE_LOG_UPDATE,
> +		.value = {
> +			.update = {
> +				.new_hash = hash1,
> +				.old_hash = hash2,
> +				.name = "My Name",
> +				.email = "myname@invalid",
> +				.message = message,
> +			},
> +		},
> +	};
> +
> +	for (i = 0; i < sizeof(message)-1; i++)

Style: SP around "-" on both sides.

> +		message[i] = (uint8_t)(rand() % 64 + ' ');
> +
> +	reftable_writer_set_limits(w, 1, 1);

Re: [PATCH v5 13/16] reftable: remove outdated file reftable.c

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> This was renamed to generic.c, but the origin was never removed
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/reftable.c | 115 --------------------------------------------
>  1 file changed, 115 deletions(-)
>  delete mode 100644 reftable/reftable.c

That's embarrassing for all reviewers of past reftable patches.

Re: [PATCH v5 10/16] reftable: handle null refnames in reftable_ref_record_equal

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Spotted by Coverity.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c | 8 +++++---
>  1 file changed, 5 insertions(+), 3 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index 8536bd03aa9..8bbcbff1e69 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
>  			      struct reftable_ref_record *b, int hash_size)
>  {
>  	assert(hash_size > 0);
> -	if (!(0 == strcmp(a->refname, b->refname) &&
> -	      a->update_index == b->update_index &&
> -	      a->value_type == b->value_type))
> +	if (!null_streq(a->refname, b->refname))
> +		return 0;
> +
> +	if (a->update_index != b->update_index ||
> +	    a->value_type != b->value_type)
>  		return 0;

The original assumed that the .refname member of these two records
are filled, so strcmp() would have segfaulted if they were set to
NULL, which equates to an empty string.  I assume that this patch
is to fix that and change nothing else?

The original said: we'll make an early return with 0 unless
all three conditions hold true; (1) names are the same, (2)
update_index members are the same, and (3) value_type members
are the same.

We now say, "we return early with 0 if names are different", and "we
also return early with 0 if update-index or value-type members are
different".

I just found the splitting into two separate statements looked
different from the original, but they do mean the same, so it is OK.

In fact, I think the !(A && B && C) is much less readable than
(!A || !B || !C) in this case, so I am OK with the new version.

Looking good.  Thanks.

Re: [PATCH v5 01/16] reftable: fix OOB stack write in print functions

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index 6a5dac32dc6..8536bd03aa9 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
>  void reftable_ref_record_print(struct reftable_ref_record *ref,
>  			       uint32_t hash_id)
>  {
> -	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
> +	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
>  	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
>  	switch (ref->value_type) {
>  	case REFTABLE_REF_SYMREF:
> @@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
>  void reftable_log_record_print(struct reftable_log_record *log,
>  			       uint32_t hash_id)
>  {
> -	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
> +	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
>  
>  	switch (log->value_type) {
>  	case REFTABLE_LOG_DELETION:

Wow.  The first one is future-proofing in case we learn to use even
larger hash, but the latter is an outright bug that makes reviewers
feel embarrassed, isn't it?

Thanks for noticing and fixing it.

Re: [PATCH v5 00/16] Reftable coverity fixes

[Junio C Hamano]

"Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:

> This series was targeted to 'next'.
>
> This addresses some complaints from Coverity that Peff reported.
>
> v5:
>
>  * add generic record equality
>  * add generic record printing
>  * const correctness for record-reftable.h
>  * fix windows crash.
>  * 0-length memcpy paranoia
>  * drop unused file.

Thanks.  Will replace.

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[René Scharfe]

Am 22.12.21 um 23:50 schrieb Junio C Hamano:
> "Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> From: Han-Wen Nienhuys <hanwen@google.com>
>>
>> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
>> ---
>>  reftable/record.c | 10 +++++++---
>>  1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/reftable/record.c b/reftable/record.c
>> index fbaa1fbef56..423e687b220 100644
>> --- a/reftable/record.c
>> +++ b/reftable/record.c
>> @@ -126,7 +126,8 @@ static int encode_string(char *str, struct string_view s)
>>  	string_view_consume(&s, n);
>>  	if (s.len < l)
>>  		return -1;
>> -	memcpy(s.buf, str, l);
>> +	if (l)
>> +		memcpy(s.buf, str, l);
>>  	string_view_consume(&s, l);
>>
>>  	return start.len - s.len;
>> @@ -153,7 +154,9 @@ int reftable_encode_key(int *restart, struct string_view dest,
>>
>>  	if (dest.len < suffix_len)
>>  		return -1;
>> -	memcpy(dest.buf, key.buf + prefix_len, suffix_len);
>> +
>> +	if (suffix_len)
>> +		memcpy(dest.buf, key.buf + prefix_len, suffix_len);
>>  	string_view_consume(&dest, suffix_len);
>>
>>  	return start.len - dest.len;
>> @@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
>>  	uint64_t last;
>>  	int j;
>>  	r->hash_prefix = reftable_malloc(key.len);
>> -	memcpy(r->hash_prefix, key.buf, key.len);
>> +	if (key.len)
>> +		memcpy(r->hash_prefix, key.buf, key.len);
>>  	r->hash_prefix_len = key.len;
>>
>>  	if (val_type == 0) {
>
> I am not sure why any of these are needed.
>
> For a code path that involves a <ptr, len> pair, where ptr is lazily
> allocated only when we have contents to store, i.e. ptr can remain
> NULL until len becomes non-zero, memcpy(dst, ptr, len) can become a
> problem as the standard requires ptr to be a valid even when len is
> 0 (ISO/IEC 9899:1999, 7.21.1 String function conventions, paragraph
> 2), but none of these three calls to memcpy() do any such thing.
>
> Puzzled.

I don't know about the first two, but in the third case dst (i.e.
r->hash_prefix) might be NULL if key.len == 0, reftable_malloc is malloc
(which it is, because reftable_set_alloc is never called) and malloc(0)
returns NULL (which it might do according to
https://www.man7.org/linux/man-pages/man3/malloc.3.html).

malloc can return NULL on failure, too, of course, and none of the
reftable_malloc callers check for that.  That seems a bit too
optimistic.

René

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Han-Wen Nienhuys]

On Wed, Dec 22, 2021 at 11:50 PM Junio C Hamano <gitster@pobox.com> wrote:

> > -     memcpy(r->hash_prefix, key.buf, key.len);
> > +     if (key.len)
> > +             memcpy(r->hash_prefix, key.buf, key.len);
> >       r->hash_prefix_len = key.len;
> >
> >       if (val_type == 0) {
>
> I am not sure why any of these are needed.

I'm not sure they are needed, but IMO it's not worth spending brain
cycles on deciding either way. Checking the length is always a safe
alternative.

I would support having a safe_memcpy() that does this check centrally
(note how our array_copy() array function also does this check, even
if it's not always needed.)

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 01/16] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys]

On Wed, Dec 22, 2021 at 11:51 PM Junio C Hamano <gitster@pobox.com> wrote:>
> Wow.  The first one is future-proofing in case we learn to use even
> larger hash, but the latter is an outright bug that makes reviewers
> feel embarrassed, isn't it?
>
> Thanks for noticing and fixing it.

You should thank Coverity :-)

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys]

On Wed, Dec 22, 2021 at 11:51 PM Junio C Hamano <gitster@pobox.com> wrote:
> > +     if (err)
>
> Is the convention for reader_init() different from all other
> functions?  It makes reader wonder why this is not
>
>         if (err < 0)
> > +             reftable_block_done(&block);
>
> even though it is not wrong per-se (as long as "zero means success"
> is a part of the return value convention).

err > 0 is returned when we reach the end of the iteration, and this
function can generate err==1.

Normally, block_reader_init() transfers the block to the block_reader.
If err > 0, we skip that, so we'd be leaking the block.

At the same time, this means the "if (err)" is superfluous. In the
success case, the block was transferred to the block_reader, so the
reftable_block_done() call is a nop.

>
> > +
> > +     return err;
> >  }
>
> This one is new in this round.  All look good, other than that one
> check for error return.
>
> > diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
> > index 5f6bcc2f775..6e88182a83a 100644
> > --- a/reftable/readwrite_test.c
> > +++ b/reftable/readwrite_test.c
> > @@ -254,6 +254,71 @@ static void test_log_write_read(void)
> >       reader_close(&rd);
> >  }
> >
> > +static void test_log_zlib_corruption(void)
> > +{
> > +     struct reftable_write_options opts = {
> > +             .block_size = 256,
> > +     };
> > +     struct reftable_iterator it = { 0 };
> > +     struct reftable_reader rd = { 0 };
> > +     struct reftable_block_source source = { 0 };
> > +     struct strbuf buf = STRBUF_INIT;
> > +     struct reftable_writer *w =
> > +             reftable_new_writer(&strbuf_add_void, &buf, &opts);
> > +     const struct reftable_stats *stats = NULL;
> > +     uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
> > +     uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
>
> Will this code be exercised when compiling with SHA256 support?  If
> not, this is perfectly fine, but otherwise, this needs to be MAX,
> not SHA1, no?

The code is parameterized in terms of hash_size, so we don't have to
test both flavors exhaustively. There is a
test_table_read_write_seek_linear_sha256() that ensures that the basic
functionality works for SHA256.

> > +     char message[100] = { 0 };
>
> You're filling this to the sizeof(message)-1, so we can afford to
> leave it uninitialized.

At the same time, we can afford to initialize it :-)

I'd rather not think about this, and always initialize everything.

> > +     for (i = 0; i < sizeof(message)-1; i++)
>
> Style: SP around "-" on both sides.

done.

(I assume I don't have to resend the whole series for these small
tweaks? I'll wait if anyone else has comments, and send a reroll early
January)
-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH 10/10] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Wed, Dec 8, 2021 at 7:17 PM Derrick Stolee <stolee@gmail.com> wrote:
> Some of these changes are definitely going to have a lot of lines,
> but when it's the same kind of change it is easy to verify. I'm
> getting lost in this diff because it's doing too many things at once.

I took some of your advice in the last version of the patch. PTAL.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

>>> @@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
>>>  	uint64_t last;
>>>  	int j;
>>>  	r->hash_prefix = reftable_malloc(key.len);
>>> -	memcpy(r->hash_prefix, key.buf, key.len);
>>> +	if (key.len)
>>> +		memcpy(r->hash_prefix, key.buf, key.len);
>>>  	r->hash_prefix_len = key.len;
>>>
>>>  	if (val_type == 0) {
>>
>> I am not sure why any of these are needed.
>> ...
> I don't know about the first two, but in the third case dst (i.e.
> r->hash_prefix) might be NULL if key.len == 0, reftable_malloc is malloc
> (which it is, because reftable_set_alloc is never called) and malloc(0)
> returns NULL (which it might do according to
> https://www.man7.org/linux/man-pages/man3/malloc.3.html).
>
> malloc can return NULL on failure, too, of course, and none of the
> reftable_malloc callers check for that.  That seems a bit too
> optimistic.

Yeah, I agree that the real bug in this code is that the returned
value of malloc() is not checked.  But checking if key.len is zero
before using memcpy() would not help fix it, or hide it.  So I am
not sure if that is a counter-argument against "this check is
pointless".

Thanks.

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@google.com> writes:

> Normally, block_reader_init() transfers the block to the block_reader.
> If err > 0, we skip that, so we'd be leaking the block.

Thanks.  I was about to say that 

    <reftable/block.h> wants to say more than "initializes a block
    reader."; perhaps "returns the number of blocks" or something
    needs to be added.

but I do not think the lack of the documentation was the source of
my confusion.

> At the same time, this means the "if (err)" is superfluous. In the
> success case, the block was transferred to the block_reader, so the
> reftable_block_done() call is a nop.

I agree that having the "if (err)" there is highly misleading.  "if
(err < 0 || 0 < err)" would be the more faithful expression of what
you explained, and if it were written that way, I wouldn't have been
as confused as I was.

In any case, if _done() becomes a safe and quick no-op when 0 block
was transferred, losing the conditional would be the way to make the
resulting code the most readable, I think.

>> > +     uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
>> > +     uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
>>
>> Will this code be exercised when compiling with SHA256 support?  If
>> not, this is perfectly fine, but otherwise, this needs to be MAX,
>> not SHA1, no?
>
> The code is parameterized in terms of hash_size, so we don't have to
> test both flavors exhaustively.

If it is safe for this function to assume that it will be called
only with hash_size of SHA1, then what you wrote is perfectly fine,
and I agree that there is no particular reason why this test must be
repeated for all available hashes.

I just don't know if we have a mechanism to prevent clueless code
churners from saying "let's test both flavors exhaustively".  For
example, I see nothing in this function, other than the size of
these two hashes, that says that this test function will be compiled
with reftable library that expects that the hash function is SHA1.
How does this function protect itself from being run in a CI job
that uses GIT_TEST_DEFAULT_HASH=sha256, for example?  It was why I
asked the question.  It wasn't that I necessarily thought we need to
test all combinations (but there is no need not to, if it takes more
engineering time to exclude it when testing Git as a whole in sha256
mode).

>> > +     char message[100] = { 0 };
>>
>> You're filling this to the sizeof(message)-1, so we can afford to
>> leave it uninitialized.
>
> At the same time, we can afford to initialize it :-)
>
> I'd rather not think about this, and always initialize everything.

I do not care too deeply in this test-only function, but as a
general principle, primarily to help less experienced developers who
may be reading from the sidelines to avoid copying such an attitude,
I have to make a note here.

If you know you will have to fill an array with, or assign to a
variable, meaningful value(s), leaving the array or variable
uninitialized at the declaration time is a much better thing to do
than initializing them with less meaningful value(s).  It will help
compilers and tools to detect a lack of proper assignment and use of
uninitialized variable (iow, you may know you will have to fill, but
in the code to do so, your implementation may be botched).  Once you
initialize at the declaration with "less meaningful" value (like
zero initialization), the tools won't be able to tell when the code
uses that variable "uninitialized" (because the assignment was
skipped by a bug), since it appears to always be initialied to them.

Helping the tools help us is what those of us, who would rather not
think about it, should be doing.

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@google.com> writes:

> On Wed, Dec 22, 2021 at 11:50 PM Junio C Hamano <gitster@pobox.com> wrote:
>
>> > -     memcpy(r->hash_prefix, key.buf, key.len);
>> > +     if (key.len)
>> > +             memcpy(r->hash_prefix, key.buf, key.len);
>> >       r->hash_prefix_len = key.len;
>> >
>> >       if (val_type == 0) {
>>
>> I am not sure why any of these are needed.
>
> I'm not sure they are needed, but IMO it's not worth spending brain
> cycles on deciding either way. Checking the length is always a safe
> alternative.
>
> I would support having a safe_memcpy() that does this check centrally
> (note how our array_copy() array function also does this check, even
> if it's not always needed.)

But your safe_memcpy() should not be

    safe_memcpy(dst, src, n)
    {
	if (n)
		memcpy(dst, src, n);
	return dst;
    }

Using memcpy() with size==0 is not a crime wrt the language.
Passing an invalid pointer while doing so is.

It should rather be

    safe_memcpy(dst, src, n)
    {
	if (dst)
		memcpy(dst, src, n);
	else if (n)
               	BUG(...);
	return dst;
    }

to support a common pattern of <ptr, len> that lazily allocates the
ptr only when len goes more than zero from triggering an error from
picky runtime, compiler and tools.  We want to allow "dst==NULL &&
n==0", while still catching "dst==NULL && n!=0" as an error.

And these places, I do not think use of safe_memcpy() is relevant.

Re: [PATCH v5 13/16] reftable: remove outdated file reftable.c

[Ævar Arnfjörð Bjarmason]

On Wed, Dec 22 2021, Junio C Hamano wrote:

> "Han-Wen Nienhuys via GitGitGadget" <gitgitgadget@gmail.com> writes:
>
>> From: Han-Wen Nienhuys <hanwen@google.com>
>>
>> This was renamed to generic.c, but the origin was never removed
>>
>> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
>> ---
>>  reftable/reftable.c | 115 --------------------------------------------
>>  1 file changed, 115 deletions(-)
>>  delete mode 100644 reftable/reftable.c
>
> That's embarrassing for all reviewers of past reftable patches.

As one of those reviewers: I think it suggests something different:

Various people looked it over, but this amount of code is just too much
for someone to review in one sitting, so it's not unexpected that
various things like this slipped through.

The current landing of this topic on "master" is based on a plan I
suggested, i.e. to have the library and its own tests in "master" first,
and not to do any of the integration yet.

Exactly because we expected that we would not be getting any of this
right the first time around.

So if anything I think these current topics and recent reftable activity
suggest that we should have been more willing to get it into "master"
earlier, and not wait until all bugs in it were solved beforehand.

I.e. there's bugs in the code, but they're well-contained, since nothing
in-tree uses it except its own tests.

And by having merged it down we're benefiting from more eyes, both in
terms of human review, and in the review of various tooling (like
coverity) that's being run on "master" but not on some random branch
that's languishing in "seen" for months.

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[René Scharfe]

Am 23.12.21 um 19:59 schrieb Junio C Hamano:
> René Scharfe <l.s.r@web.de> writes:
>
>>>> @@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
>>>>  	uint64_t last;
>>>>  	int j;
>>>>  	r->hash_prefix = reftable_malloc(key.len);
>>>> -	memcpy(r->hash_prefix, key.buf, key.len);
>>>> +	if (key.len)
>>>> +		memcpy(r->hash_prefix, key.buf, key.len);
>>>>  	r->hash_prefix_len = key.len;
>>>>
>>>>  	if (val_type == 0) {
>>>
>>> I am not sure why any of these are needed.
>>> ...
>> I don't know about the first two, but in the third case dst (i.e.
>> r->hash_prefix) might be NULL if key.len == 0, reftable_malloc is malloc
>> (which it is, because reftable_set_alloc is never called) and malloc(0)
>> returns NULL (which it might do according to
>> https://www.man7.org/linux/man-pages/man3/malloc.3.html).
>>
>> malloc can return NULL on failure, too, of course, and none of the
>> reftable_malloc callers check for that.  That seems a bit too
>> optimistic.
>
> Yeah, I agree that the real bug in this code is that the returned
> value of malloc() is not checked.  But checking if key.len is zero
> before using memcpy() would not help fix it, or hide it.  So I am
> not sure if that is a counter-argument against "this check is
> pointless".

It's not -- I was just trying to find a scenario which would cause a
Coverity warning that could be silenced by such a zero check.

We can use xmalloc and xrealloc to handle allocation failures and to get
valid pointers for zero-sized allocations, like in the patch below.

I don't understand why reftable_set_alloc exists, though -- is there
really a use case that requires changing the allocator at runtime?


diff --git a/reftable/publicbasics.c b/reftable/publicbasics.c
index 0ad7d5c0ff..438a1b0a7d 100644
--- a/reftable/publicbasics.c
+++ b/reftable/publicbasics.c
@@ -19,14 +19,14 @@ void *reftable_malloc(size_t sz)
 {
 	if (reftable_malloc_ptr)
 		return (*reftable_malloc_ptr)(sz);
-	return malloc(sz);
+	return xmalloc(sz);
 }

 void *reftable_realloc(void *p, size_t sz)
 {
 	if (reftable_realloc_ptr)
 		return (*reftable_realloc_ptr)(p, sz);
-	return realloc(p, sz);
+	return xrealloc(p, sz);
 }

 void reftable_free(void *p)

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Ævar Arnfjörð Bjarmason]

On Sun, Dec 26 2021, René Scharfe wrote:

> Am 23.12.21 um 19:59 schrieb Junio C Hamano:
>> René Scharfe <l.s.r@web.de> writes:
>>
>>>>> @@ -569,7 +572,8 @@ static int reftable_obj_record_decode(void *rec, struct strbuf key,
>>>>>  	uint64_t last;
>>>>>  	int j;
>>>>>  	r->hash_prefix = reftable_malloc(key.len);
>>>>> -	memcpy(r->hash_prefix, key.buf, key.len);
>>>>> +	if (key.len)
>>>>> +		memcpy(r->hash_prefix, key.buf, key.len);
>>>>>  	r->hash_prefix_len = key.len;
>>>>>
>>>>>  	if (val_type == 0) {
>>>>
>>>> I am not sure why any of these are needed.
>>>> ...
>>> I don't know about the first two, but in the third case dst (i.e.
>>> r->hash_prefix) might be NULL if key.len == 0, reftable_malloc is malloc
>>> (which it is, because reftable_set_alloc is never called) and malloc(0)
>>> returns NULL (which it might do according to
>>> https://www.man7.org/linux/man-pages/man3/malloc.3.html).
>>>
>>> malloc can return NULL on failure, too, of course, and none of the
>>> reftable_malloc callers check for that.  That seems a bit too
>>> optimistic.
>>
>> Yeah, I agree that the real bug in this code is that the returned
>> value of malloc() is not checked.  But checking if key.len is zero
>> before using memcpy() would not help fix it, or hide it.  So I am
>> not sure if that is a counter-argument against "this check is
>> pointless".
>
> It's not -- I was just trying to find a scenario which would cause a
> Coverity warning that could be silenced by such a zero check.
>
> We can use xmalloc and xrealloc to handle allocation failures and to get
> valid pointers for zero-sized allocations, like in the patch below.
>
> I don't understand why reftable_set_alloc exists, though -- is there
> really a use case that requires changing the allocator at runtime?

I don't know why it's there, but I suppose it's for the same reason as
PCREv2's integration, whose rabbit hole starts at 513f2b0bbd4 (grep:
make PCRE2 aware of custom allocator, 2019-10-16).

I.e. for building it as part of git.git we can just replace
malloc()/free(), but for other uses as a general linkable library you'd
want to replace it at runtime.

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Han-Wen Nienhuys]

On Fri, Dec 24, 2021 at 5:16 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> But your safe_memcpy() should not be
>
>     safe_memcpy(dst, src, n)
>     {
>         if (n)
>                 memcpy(dst, src, n);
>         return dst;
>     }
>
> Using memcpy() with size==0 is not a crime wrt the language.
> Passing an invalid pointer while doing so is.

It's not a crime, but what is the benefit of calling memcpy with n == 0 ?

>     safe_memcpy(dst, src, n)
>     {
>         if (dst)
>                 memcpy(dst, src, n);
>         else if (n)
>                 BUG(...);

I think this is suboptimal. Sure, a segfault is uglier than "out of
memory" error, but both effectively crash the program, so the
difference isn't that big.

The nice way is if the reftable library grows an error-code
REFTABLE_OOM, which is propagated once a malloc or realloc returns
NULL.

We could test this exhaustively in the unittest by swapping in a
malloc that starts failing after N allocations, and then running a
transaction in a loop, increasing N.
I'll have to look more closely if this is possible throughout, so for
this series, I'll just take a closer look at the current call-sites to
see if NULL can really occur or not.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys]

On Fri, Dec 24, 2021 at 5:16 AM Junio C Hamano <gitster@pobox.com> wrote:
> Once you
> initialize at the declaration with "less meaningful" value (like
> zero initialization), the tools won't be able to tell when the code
> uses that variable "uninitialized" (because the assignment was
> skipped by a bug), since it appears to always be initialied to them.

Which tools are these? When I add

static void test_memcpy(void)
{
 uint32_t dest;
 char not_init[200];
 int i;
 memcpy(&dest, not_init, sizeof(dest));

 for (i = 0 ; i < 10; i++)
  not_init[i] = rand() % 255 + 1;
 printf("%d", (int) strlen(not_init));
}

to the C code, it compiles cleanly if I do "make DEVELOPER=1".


-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 16/16] reftable: be more paranoid about 0-length memcpy calls

[Han-Wen Nienhuys]

On Wed, Jan 12, 2022 at 12:39 PM Han-Wen Nienhuys <hanwen@google.com> wrote:
> this series, I'll just take a closer look at the current call-sites to
> see if NULL can really occur or not.

I've dropped this patch from the series. The only suspect callsite is
in obj_record_decode(),
and that is better addressed by other changes to make sure we never
try to do a 0 length copy.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[René Scharfe]

Am 12.01.22 um 12:58 schrieb Han-Wen Nienhuys:
> On Fri, Dec 24, 2021 at 5:16 AM Junio C Hamano <gitster@pobox.com> wrote:
>> Once you
>> initialize at the declaration with "less meaningful" value (like
>> zero initialization), the tools won't be able to tell when the code
>> uses that variable "uninitialized" (because the assignment was
>> skipped by a bug), since it appears to always be initialied to them.
>
> Which tools are these? When I add
>
> static void test_memcpy(void)
> {
>  uint32_t dest;
>  char not_init[200];
>  int i;
>  memcpy(&dest, not_init, sizeof(dest));
>
>  for (i = 0 ; i < 10; i++)
>   not_init[i] = rand() % 255 + 1;
>  printf("%d", (int) strlen(not_init));
> }
>
> to the C code, it compiles cleanly if I do "make DEVELOPER=1".

https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html says about
-Wuninitialized:

   "Note that there may be no warning about a variable that is used only
    to compute a value that itself is never used, because such
    computations may be deleted by data flow analysis before the
    warnings are printed."

And indeed, dest is not used above.  But even if we change the funtion
to use dest by returning it, GCC versions 9.1 and higher don't warn
about the use of the uninitialized buffer.  GCC 4.7.1 to 8.5 do warn,
though: https://godbolt.org/z/zYz9KvPdK

René

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Ævar Arnfjörð Bjarmason]

On Wed, Jan 12 2022, Han-Wen Nienhuys wrote:

> On Fri, Dec 24, 2021 at 5:16 AM Junio C Hamano <gitster@pobox.com> wrote:
>> Once you
>> initialize at the declaration with "less meaningful" value (like
>> zero initialization), the tools won't be able to tell when the code
>> uses that variable "uninitialized" (because the assignment was
>> skipped by a bug), since it appears to always be initialied to them.
>
> Which tools are these? When I add
>
> static void test_memcpy(void)
> {
>  uint32_t dest;
>  char not_init[200];
>  int i;
>  memcpy(&dest, not_init, sizeof(dest));
>
>  for (i = 0 ; i < 10; i++)
>   not_init[i] = rand() % 255 + 1;
>  printf("%d", (int) strlen(not_init));
> }
>
> to the C code, it compiles cleanly if I do "make DEVELOPER=1".

Aside from what René said in his follow-up, I think what Junio's
pointing out here has to do with the use of this pattern in general, not
the specific code being discussed here.

I.e. if you get into the habit of needless initialization it may not
matter right now, but once the function grows an if/else branch, or
someone copies the pattern to such a function it may hide a logic error.

So it's not about analyzing the control specific flow here, but that
your upthread patch is separating a variable and its actual
internalization by ~20 lines.

So in the general case, by initializing it when it's declared it's more
likely that we'll introduce a logic error where it won't be initialized
at all.

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys]

On Thu, Jan 13, 2022 at 11:02 AM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> Aside from what René said in his follow-up, I think what Junio's
> pointing out here has to do with the use of this pattern in general, not
> the specific code being discussed here.
>
> I.e. if you get into the habit of needless initialization it may not
> matter right now, but once the function grows an if/else branch, or
> someone copies the pattern to such a function it may hide a logic error.
>
> So it's not about analyzing the control specific flow here, but that
> your upthread patch is separating a variable and its actual
> internalization by ~20 lines.

I know this is the Git project's preferred style, so I'm OK with
adapting to that, but I'm also sad about it.

Sure, you can introduce logic errors by refactoring things, but with
initialized data, you get a reproducible logic error, rather than
something whose failure mode depends on platform, compiler and
surrounding function calls. This makes debugging problems across
platforms easier. In particular, I don't have a functioning Windows
environment, so anything that helps minimize differences across
platforms is welcome to me.

--
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v5 02/16] reftable: fix resource leak in block.c error path

[Junio C Hamano]

René Scharfe <l.s.r@web.de> writes:

> Am 12.01.22 um 12:58 schrieb Han-Wen Nienhuys:
>> On Fri, Dec 24, 2021 at 5:16 AM Junio C Hamano <gitster@pobox.com> wrote:
>>> Once you
>>> initialize at the declaration with "less meaningful" value (like
>>> zero initialization), the tools won't be able to tell when the code
>>> uses that variable "uninitialized" (because the assignment was
>>> skipped by a bug), since it appears to always be initialied to them.
>>
>> Which tools are these? When I add
>>
>> static void test_memcpy(void)
>> {
>>  uint32_t dest;
>>  char not_init[200];
>>  int i;
>>  memcpy(&dest, not_init, sizeof(dest));
>>
>>  for (i = 0 ; i < 10; i++)
>>   not_init[i] = rand() % 255 + 1;
>>  printf("%d", (int) strlen(not_init));
>> }
>>
>> to the C code, it compiles cleanly if I do "make DEVELOPER=1".
>
> https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html says about
> -Wuninitialized:
>
>    "Note that there may be no warning about a variable that is used only
>     to compute a value that itself is never used, because such
>     computations may be deleted by data flow analysis before the
>     warnings are printed."
>
> And indeed, dest is not used above.  But even if we change the funtion
> to use dest by returning it, GCC versions 9.1 and higher don't warn
> about the use of the uninitialized buffer.  GCC 4.7.1 to 8.5 do warn,
> though: https://godbolt.org/z/zYz9KvPdK

What I meant in the original discussion was a runtime checker that
notices a read from an uninitialized location (I've also written one
myself way before my Git time, which was how I noticed strcpy() on
old SunOS, in an attempt to optimize by loading a word at a time,
sometimes read one word too much beyond the end of a page).

But if a static analysis may catch itas a potential error, that is
another reason to worry about it, too.

The original code wanted to do

	char message[100] = { 0 };

	for (i = 0; i < sizeof(message) - 1; i++)
        	message[i] = something();

	use_NUL_terminated_string(message);

and it allowed to omit an assigmnent

	message[sizeof(message) - 1] = '\0';

after the loop.

As I already said, I do not care too deeply in a test-only helper
like this.

Thanks.

Re: v2.35.0 DEVELOPER=1 regression (was: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang)

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1565 bytes --]

On 2022-01-13 at 10:45:49, Ævar Arnfjörð Bjarmason wrote:
> [I'm aware that Jeff probably won't read this in his sabbatical, just
> replying to the relevant thread]
> 
> Something neither of us considered at the time, but which is pretty
> obvious in retrospect, is that this new -std=gnu99 dosen't only apply to
> our own code, but any system and library includes that we need.
> 
> Thus e.g. FreeBSD 13.0 which previously was happy to compile under
> DEVELOPER=1 will now die because its core libraries use C11-specific
> code:
>     
>     archive.c:337:35: error: '_Generic' is a C11 extension [-Werror,-Wc11-extensions]
>                     strbuf_addstr(&path_in_archive, basename(path));
>                                                     ^
>     /usr/include/libgen.h:61:21: note: expanded from macro 'basename'
>     #define basename(x)     __generic(x, const char *, __old_basename, basename)(x)
>                             ^
>     /usr/include/sys/cdefs.h:325:2: note: expanded from macro '__generic'
>             _Generic(expr, t: yes, default: no)
>             ^

I think we had this discussion about FreeBSD before and that's why I
specifically dropped that option from the main makefile.  We can either
drop that patch, or we can set it to -std=gnu11 and tell folks setting
DEVELOPER to use a system released in the last five years.  I think we
can be a little stricter with what we require in the case of DEVELOPER
than we might be otherwise.
-- 
brian m. carlson (he/him or they/them)
Toronto, Ontario, CA

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: v2.35.0 DEVELOPER=1 regression (was: [PATCH] config.mak.dev: specify -std=gnu99 for gcc/clang)

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 14 2022, brian m. carlson wrote:

> [[PGP Signed Part:Undecided]]
> On 2022-01-13 at 10:45:49, Ævar Arnfjörð Bjarmason wrote:
>> [I'm aware that Jeff probably won't read this in his sabbatical, just
>> replying to the relevant thread]
>> 
>> Something neither of us considered at the time, but which is pretty
>> obvious in retrospect, is that this new -std=gnu99 dosen't only apply to
>> our own code, but any system and library includes that we need.
>> 
>> Thus e.g. FreeBSD 13.0 which previously was happy to compile under
>> DEVELOPER=1 will now die because its core libraries use C11-specific
>> code:
>>     
>>     archive.c:337:35: error: '_Generic' is a C11 extension [-Werror,-Wc11-extensions]
>>                     strbuf_addstr(&path_in_archive, basename(path));
>>                                                     ^
>>     /usr/include/libgen.h:61:21: note: expanded from macro 'basename'
>>     #define basename(x)     __generic(x, const char *, __old_basename, basename)(x)
>>                             ^
>>     /usr/include/sys/cdefs.h:325:2: note: expanded from macro '__generic'
>>             _Generic(expr, t: yes, default: no)
>>             ^
>
> I think we had this discussion about FreeBSD before and that's why I
> specifically dropped that option from the main makefile.  We can either
> drop that patch, or we can set it to -std=gnu11 and tell folks setting
> DEVELOPER to use a system released in the last five years.  I think we
> can be a little stricter with what we require in the case of DEVELOPER
> than we might be otherwise.

I guess, yeah.

As a practical matter the changes in this cycle have made DEVELOPER=1
much more fragile as a cross-platform facility. I test on various
platforms/OS's (across the GCC farm), and before this cycle only AIX and
Solaris were something I had to pay special attention to.

Now we've got things like this breaking (by default) non-obscure things
like FreeBSD.

Of course I can manually set -Wall etc. which is what I previously got
out of this, but before e.g. pedantic was opt-in, now an effective
C-version-a-pedantic is the default, and doesn't have a tweaking knob at
all.

Anyway, we'll see how much of a hassle it is, and it's not too painful
for me right now, so I don't think anything needs to be done in the RC
period.

But I wonder if the X-Y problem we're trying to solve is making sure we
don't move past C99 unintentionally whether this wouldn't be better
solved by dropping this -std=gnu99 approach in the Makefile, and instead
just do that in one of the CI jobs (whose OS includes would be known to
be C99-OK).

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

>> DEVELOPER=1 will now die because its core libraries use C11-specific
>> code:
>>     
>>     archive.c:337:35: error: '_Generic' is a C11 extension [-Werror,-Wc11-extensions]
>>                     strbuf_addstr(&path_in_archive, basename(path));
>>                                                     ^
>>     /usr/include/libgen.h:61:21: note: expanded from macro 'basename'
>>     #define basename(x)     __generic(x, const char *, __old_basename, basename)(x)
>>                             ^
>>     /usr/include/sys/cdefs.h:325:2: note: expanded from macro '__generic'
>>             _Generic(expr, t: yes, default: no)
>>             ^

Wow, that sounds horribly broken.

> I think we had this discussion about FreeBSD before and that's why I
> specifically dropped that option from the main makefile.  We can either
> drop that patch, or we can set it to -std=gnu11 and tell folks setting
> DEVELOPER to use a system released in the last five years.  I think we
> can be a little stricter with what we require in the case of DEVELOPER
> than we might be otherwise.

But that is not being stricter, but looser, no?  I thought that the
point of -std=gnu99 was to allow us to use C99 features while catching
use of language features newer than that, and use of -std=gnu11 will
defeat half the point, wouldn't it?

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 14 2022, Junio C Hamano wrote:

> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
>
>>> DEVELOPER=1 will now die because its core libraries use C11-specific
>>> code:
>>>     
>>>     archive.c:337:35: error: '_Generic' is a C11 extension [-Werror,-Wc11-extensions]
>>>                     strbuf_addstr(&path_in_archive, basename(path));
>>>                                                     ^
>>>     /usr/include/libgen.h:61:21: note: expanded from macro 'basename'
>>>     #define basename(x)     __generic(x, const char *, __old_basename, basename)(x)
>>>                             ^
>>>     /usr/include/sys/cdefs.h:325:2: note: expanded from macro '__generic'
>>>             _Generic(expr, t: yes, default: no)
>>>             ^
>
> Wow, that sounds horribly broken.

Yes, but it's also working as designed :) We're erroring because the C
library headers on the OS aren't C99-compliant. That it would apply to
only git.git's sources was only ever wishful thinking.

>> I think we had this discussion about FreeBSD before and that's why I
>> specifically dropped that option from the main makefile.  We can either
>> drop that patch, or we can set it to -std=gnu11 and tell folks setting
>> DEVELOPER to use a system released in the last five years.  I think we
>> can be a little stricter with what we require in the case of DEVELOPER
>> than we might be otherwise.
>
> But that is not being stricter, but looser, no?  I thought that the
> point of -std=gnu99 was to allow us to use C99 features while catching
> use of language features newer than that, and use of -std=gnu11 will
> defeat half the point, wouldn't it?

Indeed. But also as noted in my side-thread reply in
<220114.86a6fytte9.gmgdl@evledraar.gmail.com> I think that being able to
use it like this was always wishful thinking.

I.e. to brian's "tell folks setting DEVELOPER to use a system released
in the last five years". This is the exact opposite of that.

We are implicitly expecting that the OS will forever be using the
20+-year old C99 standard, and not trying out fetures in C11
standardized 10+ years ago. The FreeBSD 13.0 release is less than a year
old[1].

When I went looking I found that the change to use __generic in its
libgen.h just so happens to have been made in FreeBSD sources around 5
years ago: [2].

1. https://www.freebsd.org/releases/13.0R/announce/
2. https://github.com/freebsd/freebsd-src/commit/34168b28e99b0bb328c7bb49cd91cb942181056f

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

>> Wow, that sounds horribly broken.
>
> Yes, but it's also working as designed :) We're erroring because the C
> library headers on the OS aren't C99-compliant. That it would apply to
> only git.git's sources was only ever wishful thinking.

No, C library supporting only C11 is perfectly fine.  On such a
system, the compiler shouldn't even support -std=gnu99.  That is
what I consider broken.

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

>> I think we had this discussion about FreeBSD before and that's why I
>> specifically dropped that option from the main makefile.  We can either
>> drop that patch, or we can set it to -std=gnu11 and tell folks setting
>> DEVELOPER to use a system released in the last five years.  I think we
>> can be a little stricter with what we require in the case of DEVELOPER
>> than we might be otherwise.
>
> But that is not being stricter, but looser, no?  I thought that the
> point of -std=gnu99 was to allow us to use C99 features while catching
> use of language features newer than that, and use of -std=gnu11 will
> defeat half the point, wouldn't it?

If FreeBSD (or any other platform) cannot do "reject features beyond
C99", I am perfectly OK to drop -std=gnu99 on such a place.

DEVELOPER=YesPlease ought to be a collection of settings that helps
the developers the most.  So on platforms that *can* do "reject
features beyond C99", I prefer to have it enabled when
DEVELOPER=YesPlease is given.

It seems that -std=gnu99 is only added conditionally even in today's
config.mak.dev, so it is fine if we dropped -std=gnu99 from there.
Which means that developers on FreeBSD cannot participate in vetting
use of features beyond C99, but there are developers on other
platforms who will, so it's not too bad, I would say.

As config.mak.dev is included after config.mak.uname, something like
this may be sufficient, perhaps?

 config.mak.dev | 5 +++++
 1 file changed, 5 insertions(+)

diff --git i/config.mak.dev w/config.mak.dev
index d4afac6b51..3deb076d5e 100644
--- i/config.mak.dev
+++ w/config.mak.dev
@@ -20,9 +20,14 @@ endif
 endif
 endif
 
+ifneq ($(uname_S),FreeBSD)
 ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
 DEVELOPER_CFLAGS += -std=gnu99
 endif
+else
+# FreeBSD cannot limit to C99 because its system headers unconditionally
+# rely on C11 features.
+endif
 
 DEVELOPER_CFLAGS += -Wdeclaration-after-statement
 DEVELOPER_CFLAGS += -Wformat-security

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 14 2022, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
>
>>> I think we had this discussion about FreeBSD before and that's why I
>>> specifically dropped that option from the main makefile.  We can either
>>> drop that patch, or we can set it to -std=gnu11 and tell folks setting
>>> DEVELOPER to use a system released in the last five years.  I think we
>>> can be a little stricter with what we require in the case of DEVELOPER
>>> than we might be otherwise.
>>
>> But that is not being stricter, but looser, no?  I thought that the
>> point of -std=gnu99 was to allow us to use C99 features while catching
>> use of language features newer than that, and use of -std=gnu11 will
>> defeat half the point, wouldn't it?
>
> If FreeBSD (or any other platform) cannot do "reject features beyond
> C99", I am perfectly OK to drop -std=gnu99 on such a place.
>
> DEVELOPER=YesPlease ought to be a collection of settings that helps
> the developers the most.  So on platforms that *can* do "reject
> features beyond C99", I prefer to have it enabled when
> DEVELOPER=YesPlease is given.
>
> It seems that -std=gnu99 is only added conditionally even in today's
> config.mak.dev, so it is fine if we dropped -std=gnu99 from there.
> Which means that developers on FreeBSD cannot participate in vetting
> use of features beyond C99, but there are developers on other
> platforms who will, so it's not too bad, I would say.
>
> As config.mak.dev is included after config.mak.uname, something like
> this may be sufficient, perhaps?

Wasn't the initial goal here to check whether we'd accidentally include
C99? Just checking on GCC/Clang somewhere seems sufficient enough,
I.e. something like:
    
    diff --git a/ci/lib.sh b/ci/lib.sh
    index 9d28ab50fb4..94d0d4127b9 100755
    --- a/ci/lib.sh
    +++ b/ci/lib.sh
    @@ -209,6 +209,9 @@ linux-leaks)
            export SANITIZE=leak
            export GIT_TEST_PASSING_SANITIZE_LEAK=true
            ;;
    +linux-gcc|linux-clang)
    +       MAKEFLAGS="$MAKEFLAGS CFLAGS=-std=gnu99"
    +       ;;
     esac
     
     MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    diff --git a/config.mak.dev b/config.mak.dev
    index d4afac6b51f..216f92493fc 100644
    --- a/config.mak.dev
    +++ b/config.mak.dev
    @@ -20,10 +20,6 @@ endif
     endif
     endif
     
    -ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
    -DEVELOPER_CFLAGS += -std=gnu99
    -endif
    -
     DEVELOPER_CFLAGS += -Wdeclaration-after-statement
     DEVELOPER_CFLAGS += -Wformat-security
     DEVELOPER_CFLAGS += -Wold-style-definition
    
We're just starting to get into the whack-a-mole of hardcoding OS's and
compiler versions, it's not all versions of FreeBSD, probably not just
that OS etc.

I've also wondered why we can't just ship our own super-light version of
autoconf to run some real probes, and as a result eventually get rid of
configure.ac, ./detect-compiler, the version hardcoding in
config.mak.dev etc.

Just something as simple as extending this:
    
    diff --git a/Makefile b/Makefile
    index 5580859afdb..f197a07c100 100644
    --- a/Makefile
    +++ b/Makefile
    @@ -2557,6 +2557,15 @@ else
     $(OBJECTS): $(LIB_H) $(GENERATED_H)
     endif
     
    +ifdef DEVELOPER
    +probe/std.mak:
    +       @mkdir -p $(@D)
    +       $(QUIET_GEN)if $(CC) -E -std=gnu99 - </dev/null >/dev/null; then \
    +               echo 'DEVELOPER_CFLAGS += -std=gnu99'; \
    +       fi >$@
    +include probe/std.mak
    +endif
    +
     ifeq ($(GENERATE_COMPILATION_DATABASE),yes)
     all:: compile_commands.json
     compile_commands.json:

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Junio C Hamano <gitster@pobox.com> writes:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>>> Wow, that sounds horribly broken.
>>
>> Yes, but it's also working as designed :) We're erroring because the C
>> library headers on the OS aren't C99-compliant. That it would apply to
>> only git.git's sources was only ever wishful thinking.
>
> No, C library supporting only C11 is perfectly fine.  On such a
> system, the compiler shouldn't even support -std=gnu99.  That is
> what I consider broken.

Or, the system headers should be arranged in such a way that
depending on __STDC_VERSION__, it should refrain from using features
of the language that is not supported.  So supporting -std=gnu99 in
their compilers may not be a bug---but in that case, their system
headers are buggy.

Anyway.  I think <xmqqzgny7xo7.fsf@gitster.g> shows a viable way
forward.

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> Wasn't the initial goal here to check whether we'd accidentally include
> C99? Just checking on GCC/Clang somewhere seems sufficient enough,

What do you mean by "accidentally include"?

The goal here is that developer build should give developer a set of
options that help the most---"not going beyond C99" is something we
want them to be checking, if able (and we have established that
users on FreeBSD are not capable).

Doesn't your "something like" limits the check to CI?  Developers
with compilers that _can_ help ensuring that we do not go beyond C99
should be able to do so by their local developer build.

> I.e. something like:
>     
>     diff --git a/ci/lib.sh b/ci/lib.sh
>     index 9d28ab50fb4..94d0d4127b9 100755
>     --- a/ci/lib.sh
>     +++ b/ci/lib.sh
>     @@ -209,6 +209,9 @@ linux-leaks)
>             export SANITIZE=leak
>             export GIT_TEST_PASSING_SANITIZE_LEAK=true
>             ;;
>     +linux-gcc|linux-clang)
>     +       MAKEFLAGS="$MAKEFLAGS CFLAGS=-std=gnu99"
>     +       ;;
>      esac
>      
>      MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
>     diff --git a/config.mak.dev b/config.mak.dev
>     index d4afac6b51f..216f92493fc 100644
>     --- a/config.mak.dev
>     +++ b/config.mak.dev
>     @@ -20,10 +20,6 @@ endif
>      endif
>      endif
>      
>     -ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
>     -DEVELOPER_CFLAGS += -std=gnu99
>     -endif
>     -
>      DEVELOPER_CFLAGS += -Wdeclaration-after-statement
>      DEVELOPER_CFLAGS += -Wformat-security
>      DEVELOPER_CFLAGS += -Wold-style-definition

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Fri, Jan 14 2022, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> Wasn't the initial goal here to check whether we'd accidentally include
>> C99? Just checking on GCC/Clang somewhere seems sufficient enough,
>
> What do you mean by "accidentally include"?
>
> The goal here is that developer build should give developer a set of
> options that help the most---"not going beyond C99" is something we
> want them to be checking, if able (and we have established that
> users on FreeBSD are not capable).

The reason we ended up with -std=gnu99 is because of this thread where
Han-Wen accidentally used an unnamed structs/unions:
https://lore.kernel.org/git/xmqqlf0w5bbc.fsf@gitster.g/

He then pointed out that it would be nice if CI caught this, which would
have been sufficient:
https://lore.kernel.org/git/CAFQ2z_NuOy+-pfSoNAYjJhS9jZCYOfoFue10=k=iyPVsPYrB3g@mail.gmail.com/

> Doesn't your "something like" limits the check to CI?  Developers
> with compilers that _can_ help ensuring that we do not go beyond C99
> should be able to do so by their local developer build.

Yes, I'm suggesting that what we're trying to catch here is rare enough
that that may be sufficient, and which would sidestep the issue of
wasting time on trying to make this portable everywhere.

The DEVELOPER=1 feature has recently gone from something I used on a
wide varity on of platforms and the worst I got was notices about
unknown -W flags in a couple of them, to something where recently due to
-fno-common and now more widely with -std=gnu99 I've ended up having to
manually blacklist it on a few boxes where it Just Worked.

So maybe you think we absolutely need to spend the effort to run less
portable checks like that locally if at all possible, with all the
config.mak.uname etc. work that entails, if so OK.

I'm just suggesting that perhaps a more practical solution is to
side-step those portability issues by checking this in CI, which I think
would be Good Enough on this case.

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> I'm just suggesting that perhaps a more practical solution is to
> side-step those portability issues by checking this in CI, which I think
> would be Good Enough on this case.

Use of GitHub PR + GGG is required to take advantage of that, no?  I
do not think I want to see a change that makes it _more_ likely that
a patch I pick up from the list will have problems that my local
integration work will not catch (because DEVELOPER=YesPlease build
is castrated at the original contributor side, and also on my box)
and will only be caught after I push out 'seen'.

Re: v2.35.0 DEVELOPER=1 regression

[Johannes Schindelin]

Hi Junio,

On Fri, 14 Jan 2022, Junio C Hamano wrote:

> Junio C Hamano <gitster@pobox.com> writes:
>
> >> I think we had this discussion about FreeBSD before and that's why I
> >> specifically dropped that option from the main makefile.  We can either
> >> drop that patch, or we can set it to -std=gnu11 and tell folks setting
> >> DEVELOPER to use a system released in the last five years.  I think we
> >> can be a little stricter with what we require in the case of DEVELOPER
> >> than we might be otherwise.
> >
> > But that is not being stricter, but looser, no?  I thought that the
> > point of -std=gnu99 was to allow us to use C99 features while catching
> > use of language features newer than that, and use of -std=gnu11 will
> > defeat half the point, wouldn't it?
>
> If FreeBSD (or any other platform) cannot do "reject features beyond
> C99", I am perfectly OK to drop -std=gnu99 on such a place.
>
> DEVELOPER=YesPlease ought to be a collection of settings that helps
> the developers the most.  So on platforms that *can* do "reject
> features beyond C99", I prefer to have it enabled when
> DEVELOPER=YesPlease is given.
>
> It seems that -std=gnu99 is only added conditionally even in today's
> config.mak.dev, so it is fine if we dropped -std=gnu99 from there.
> Which means that developers on FreeBSD cannot participate in vetting
> use of features beyond C99, but there are developers on other
> platforms who will, so it's not too bad, I would say.

Plus, we have CI runs that do help us, thanks to setting `DEVELOPER=1`
(see https://github.com/git/git/blob/v2.35.0-rc1/ci/lib.sh#L154).

> As config.mak.dev is included after config.mak.uname, something like
> this may be sufficient, perhaps?
>
>  config.mak.dev | 5 +++++
>  1 file changed, 5 insertions(+)
>
> diff --git i/config.mak.dev w/config.mak.dev
> index d4afac6b51..3deb076d5e 100644
> --- i/config.mak.dev
> +++ w/config.mak.dev
> @@ -20,9 +20,14 @@ endif
>  endif
>  endif
>
> +ifneq ($(uname_S),FreeBSD)
>  ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
>  DEVELOPER_CFLAGS += -std=gnu99
>  endif
> +else
> +# FreeBSD cannot limit to C99 because its system headers unconditionally
> +# rely on C11 features.
> +endif
>
>  DEVELOPER_CFLAGS += -Wdeclaration-after-statement
>  DEVELOPER_CFLAGS += -Wformat-security
>

I applied this patch on top of the current tip of `seen`, opened a PR at
https://github.com/gitgitgadget/git/pull/1116. The corresponding FreeBSD
run is here: https://cirrus-ci.com/task/5867558132776960, and it
succeeded.

In addition, I concur that this is the best we can do, and I really would
like to have that patch as soon as possible because it confuses new
contributors when their PR builds fail out of no mistake of their own.

Please count this as a vote for including this patch in -rc2.

Thank you,
Dscho

[PATCH] config.mak.dev: fix DEVELOPER=1 on FreeBSD with -std=gnu99

[Ævar Arnfjörð Bjarmason]

Since the addition of -std=gnu99 to CFLAGS under DEVELOPER=1 in
5f46385309b (config.mak.dev: specify -std=gnu99 for gcc/clang,
2021-12-08) the compilation has been broken on newer FreeBSD
systems. E.g. on FreeBSD 13 we'll now get this when compiling
archive.o:

    archive.c:337:35: error: '_Generic' is a C11 extension [-Werror,-Wc11-extensions]
                    strbuf_addstr(&path_in_archive, basename(path));
                                                    ^
    /usr/include/libgen.h:61:21: note: expanded from macro 'basename'
    #define basename(x)     __generic(x, const char *, __old_basename, basename)(x)
                            ^
    /usr/include/sys/cdefs.h:325:2: note: expanded from macro '__generic'
            _Generic(expr, t: yes, default: no)
            ^

This issue is a bug in FreeBSD introduced in
freebsd/freebsd-src@62b7f85d4749 (Leave the C11 keywords alone when we
have a recent version of GCC., 2014-09-03)[1].

That commit started conditionally using the C11 _Generic() keyword as
a function of either a __STDC_VERSION__ check, or
__has_extension(c_generic_selections). The current FreeBSD code in
sys/cdefs.h being (slightly modified since that commit):

    #if (defined(__STDC_VERSION__) && __STDC_VERSION__ >= 201112L) || \
        __has_extension(c_generic_selections)
    [...]

The inherent problem is that __has_extension() is a way to ask
Clang (and GCC) whether the compiler supports that feature, but the
-Wc11-extensions warning will be issued on the basis of the selected
__STDC_VERSION__. With -std=gnu99 the __has_extension() built-in will
return true, but the warning will still fire.

Let's narrowly work around this by checking whether:

 A. We are under FreeBSD
 B. We are using a C standard older than C11

In that case we'll include sys/cdefs.h, and undefine FreeBSD's
__generic() wrapper if it's been defined. We'll then load libgen.h,
and restore whatever __generic definition we temporarily undefined
earlier.

An alternate solution would be to simply define NO_LIBGEN_H=Y in
config.mak.uname for FreeBSD, but this way we'll use its OS-provided
basename() and dirname(), and in the general case ensure that nothing
changes outside of DEVELOPER=1 builds.

1. https://github.com/freebsd/freebsd-src/commit/62b7f85d4749

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 git-compat-util.h | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/git-compat-util.h b/git-compat-util.h
index 1229c8296b9..69d9b5f202f 100644
--- a/git-compat-util.h
+++ b/git-compat-util.h
@@ -318,7 +318,25 @@ static inline int setitimer(int which, const struct itimerval *value, struct iti
 #endif
 
 #ifndef NO_LIBGEN_H
+/*
+ * FreeBSD's libgen.h inadvertently requires C11 features, due to its
+ * sys/cdefs.h using _Generic() if
+ * __has_extension(c_generic_selections) is true, regardless of
+ * __STDC_VERSION__....
+ */
+#if defined(__FreeBSD__) && __STDC_VERSION__ - 0 < 201112L
+#include <sys/cdefs.h>
+#ifdef __generic
+#define __fbsd_generic __generic
+#endif
+#undef __generic
+#endif
 #include <libgen.h>
+/* ...continue FreeBSD-specific hack above */
+#ifdef __fbsd_generic
+#define __generic __fbsd_generic
+#undef __fbsd_generic
+#endif
 #else
 #define basename gitbasename
 char *gitbasename(char *);
-- 
2.35.0.rc1.862.gadd77378860

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Tue, Jan 18 2022, Johannes Schindelin wrote:

> Hi Junio,
>
> On Fri, 14 Jan 2022, Junio C Hamano wrote:
>
>> Junio C Hamano <gitster@pobox.com> writes:
>>
>> >> I think we had this discussion about FreeBSD before and that's why I
>> >> specifically dropped that option from the main makefile.  We can either
>> >> drop that patch, or we can set it to -std=gnu11 and tell folks setting
>> >> DEVELOPER to use a system released in the last five years.  I think we
>> >> can be a little stricter with what we require in the case of DEVELOPER
>> >> than we might be otherwise.
>> >
>> > But that is not being stricter, but looser, no?  I thought that the
>> > point of -std=gnu99 was to allow us to use C99 features while catching
>> > use of language features newer than that, and use of -std=gnu11 will
>> > defeat half the point, wouldn't it?
>>
>> If FreeBSD (or any other platform) cannot do "reject features beyond
>> C99", I am perfectly OK to drop -std=gnu99 on such a place.
>>
>> DEVELOPER=YesPlease ought to be a collection of settings that helps
>> the developers the most.  So on platforms that *can* do "reject
>> features beyond C99", I prefer to have it enabled when
>> DEVELOPER=YesPlease is given.
>>
>> It seems that -std=gnu99 is only added conditionally even in today's
>> config.mak.dev, so it is fine if we dropped -std=gnu99 from there.
>> Which means that developers on FreeBSD cannot participate in vetting
>> use of features beyond C99, but there are developers on other
>> platforms who will, so it's not too bad, I would say.
>
> Plus, we have CI runs that do help us, thanks to setting `DEVELOPER=1`
> (see https://github.com/git/git/blob/v2.35.0-rc1/ci/lib.sh#L154).
>
>> As config.mak.dev is included after config.mak.uname, something like
>> this may be sufficient, perhaps?
>>
>>  config.mak.dev | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git i/config.mak.dev w/config.mak.dev
>> index d4afac6b51..3deb076d5e 100644
>> --- i/config.mak.dev
>> +++ w/config.mak.dev
>> @@ -20,9 +20,14 @@ endif
>>  endif
>>  endif
>>
>> +ifneq ($(uname_S),FreeBSD)
>>  ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
>>  DEVELOPER_CFLAGS += -std=gnu99
>>  endif
>> +else
>> +# FreeBSD cannot limit to C99 because its system headers unconditionally
>> +# rely on C11 features.
>> +endif
>>
>>  DEVELOPER_CFLAGS += -Wdeclaration-after-statement
>>  DEVELOPER_CFLAGS += -Wformat-security
>>
>
> I applied this patch on top of the current tip of `seen`, opened a PR at
> https://github.com/gitgitgadget/git/pull/1116. The corresponding FreeBSD
> run is here: https://cirrus-ci.com/task/5867558132776960, and it
> succeeded.
>
> In addition, I concur that this is the best we can do, and I really would
> like to have that patch as soon as possible because it confuses new
> contributors when their PR builds fail out of no mistake of their own.

I just submitted a more narrow fix in the side-thread[1]. I've tested it
on the FreeBSD 13.0 box I have access to, but perhaps you'd like to test
it too?

> Please count this as a vote for including this patch in -rc2.

I'd like to have it too, but for context needing to add NO_UNCOMPRESS2=Y
(which Junio's punted on for the final[2]) is a much more widespread
issue of needing new post-install build tweaking than this issue that
only affects developer builds on FreeBSD.

What makes that FreeBSD run a part of the GGG CI? Another and more
narrow workaround for that in particular is to have whatever's driving
it add -Wno-c11-extensions to the CFLAGS.

1. https://lore.kernel.org/git/patch-1.1-06cc12be94d-20220118T151234Z-avarab@gmail.com/
2. https://lore.kernel.org/git/patch-1.1-9cea01a1395-20220117T170457Z-avarab@gmail.com/

Re: [PATCH] config.mak.dev: fix DEVELOPER=1 on FreeBSD with -std=gnu99

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> The inherent problem is that __has_extension() is a way to ask
> Clang (and GCC) whether the compiler supports that feature, but the
> -Wc11-extensions warning will be issued on the basis of the selected
> __STDC_VERSION__. With -std=gnu99 the __has_extension() built-in will
> return true, but the warning will still fire.
>
> Let's narrowly work around this by checking whether:

The end result, if and when a change along this line proves OK for
all versions of FreeBSD we care about, may be narrow, but as a band
aid to discuss just before the final release, I do not know if
anybody can sensibly assess the ramifications of such a change,
except for somebody whose primary development environment has been
FreeBSD for the past 6 months or more.

This looks way more than we can take comfortably at this point in
the release cycle, compared to "fixing" the developer build by
kicking developers on FreeBSD out of the "we are C99 or below at
this point" enforcement mechanism.

>  A. We are under FreeBSD
>  B. We are using a C standard older than C11
>
> In that case we'll include sys/cdefs.h, and undefine FreeBSD's
> __generic() wrapper if it's been defined. We'll then load libgen.h,
> and restore whatever __generic definition we temporarily undefined
> earlier.
>
> An alternate solution would be to simply define NO_LIBGEN_H=Y in
> config.mak.uname for FreeBSD, but this way we'll use its OS-provided
> basename() and dirname(), and in the general case ensure that nothing
> changes outside of DEVELOPER=1 builds.
>
> 1. https://github.com/freebsd/freebsd-src/commit/62b7f85d4749
>
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  git-compat-util.h | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
>
> diff --git a/git-compat-util.h b/git-compat-util.h
> index 1229c8296b9..69d9b5f202f 100644
> --- a/git-compat-util.h
> +++ b/git-compat-util.h
> @@ -318,7 +318,25 @@ static inline int setitimer(int which, const struct itimerval *value, struct iti
>  #endif
>  
>  #ifndef NO_LIBGEN_H
> +/*
> + * FreeBSD's libgen.h inadvertently requires C11 features, due to its
> + * sys/cdefs.h using _Generic() if
> + * __has_extension(c_generic_selections) is true, regardless of
> + * __STDC_VERSION__....
> + */
> +#if defined(__FreeBSD__) && __STDC_VERSION__ - 0 < 201112L
> +#include <sys/cdefs.h>
> +#ifdef __generic
> +#define __fbsd_generic __generic
> +#endif
> +#undef __generic
> +#endif
>  #include <libgen.h>
> +/* ...continue FreeBSD-specific hack above */
> +#ifdef __fbsd_generic
> +#define __generic __fbsd_generic
> +#undef __fbsd_generic
> +#endif
>  #else
>  #define basename gitbasename
>  char *gitbasename(char *);

[PATCH] Makefile: FreeBSD cannot do C99-or-below build

[Junio C Hamano]

In "make DEVELOPER=YesPlease" builds, we try to help developers to
catch as many potential issues as they can by using -Wall and
turning compilation warnings into errors.  In the same spirit, we
recently started adding -std=gnu99 to their CFLAGS, so that they can
notice when they accidentally used language features beyond C99.

It however turns out that FreeBSD 13.0 mistakenly uses C11 extension
in its system header files regardless of what __STDC_VERSION__ says,
which means that the platform (unless we tweak their system headers)
cannot be used for this purpose.

It seems that -std=gnu99 is only added conditionally even in today's
config.mak.dev, so it is fine if we dropped -std=gnu99 from there.
Which means that developers on FreeBSD cannot participate in vetting
use of features beyond C99, but there are developers on other
platforms who will, so it's not too bad.

We might want a more "fundamental" fix to make the platform capable
of taking -std=gnu99, like working around the use of unconditional
C11 extension in its system header files by supplying a set of
"replacement" definitions in our header files.  We chose not to
pursue such an approach for two reasons at this point:

 (1) The fix belongs to the FreeBSD project, not this project, and
     such an upstream fix may happen hopefully in a not-too-distant
     future.

 (2) Fixing such a bug in system header files and working it around
     can lead to unexpected breakages (other parts of their system
     header files may not be expecting to see and do not work well
     with our "replacement" definitions).  This close to the final
     release of this cycle, we have no time for that.

Signed-off-by: Junio C Hamano <gitster@pobox.com>
---
 config.mak.dev | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/config.mak.dev b/config.mak.dev
index d4afac6b51..3deb076d5e 100644
--- a/config.mak.dev
+++ b/config.mak.dev
@@ -20,9 +20,14 @@ endif
 endif
 endif
 
+ifneq ($(uname_S),FreeBSD)
 ifneq ($(or $(filter gcc6,$(COMPILER_FEATURES)),$(filter clang7,$(COMPILER_FEATURES))),)
 DEVELOPER_CFLAGS += -std=gnu99
 endif
+else
+# FreeBSD cannot limit to C99 because its system headers unconditionally
+# rely on C11 features.
+endif
 
 DEVELOPER_CFLAGS += -Wdeclaration-after-statement
 DEVELOPER_CFLAGS += -Wformat-security
-- 
2.35.0-rc1-135-g45b839adb0

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> I'd like to have it too, but for context needing to add NO_UNCOMPRESS2=Y
> (which Junio's punted on for the final[2]) is a much more widespread
> issue of needing new post-install build tweaking than this issue that
> only affects developer builds on FreeBSD.

I hate it when people misrepresent what I said, even in an attempt
to spite me.  For the "check ZLIB_VERSION_NUM" topic, I gave you
specific things that needs to be different from the version posted
with reasons, fully expecting that a better version of the patch to
materialize soon (knowing how proliferate you can be when you want
to) to give us enough time to assess the potential damage.

I wouldn't call that "punted".

For this one, config.mak.dev patch WOULD only affect developer
builds, which is a much better solution than overriding what their
system headers want to do and force compiling with C99 mode.  With
the status quo with today's code, with or without the patch Dscho
wants in this thread, means ANYBODY will be stopped when they
attempt to build with -std=gnu99 on FreeBSD, which is a GOOD thing.

The reason why it is a much better solution to PUNT on using C99
mode on FreeBSD is because this episode makes it very clear that
nobody tested building anything that use basename(), dirname(),
etc. with C99 mode on the platform.  I do not trust such a build,
even if we could work around the system header breakage.

This time we got lucky and wereq stopped by a compilation error, but
I have a strong suspicion that C99-only mode of compiler on this
platform is not as well battle tested as their standard mode of
compilation that allows C11.  I simply do not think we want to waste
developer's time with C99-only mode on this platform which may end
up debugging the platform and not our program.  Next bug that will
hit us may not be so friendly to clearly break compilation.
Instead, the symptom may be a subtle runtime breakage that wastes
people's time.

After developers who work on FreeBSD (not Git developers who uses
FreeBSD) ships an updated system headers so that more programs, not
just us, are built and testsd with C99-only mode, perhaps it becomes
usable as a platform to catch use of language features beyond C99,
like everybody else.  But with such a clearly broken system headers,
I do not think today's FreeBSD is ready for that.  I do trust their
default settings that allows C11, a lot more than C99-only
compilation with the "their libgen is broken, so here is a user side
workaround" patch.

Re: [PATCH] Makefile: FreeBSD cannot do C99-or-below build

[Neeraj Singh]

The approach of building with C11 on FreeBSD is a good one
compared to trying to hack around the headers.

It appears more like a compiler bug that's being worked around
here. The FreeBSD header supposedly uses a GCC extension if the
C standard version is less than C11.  See: 
https://github.com/freebsd/freebsd-src/blob/1e7b5f950b2d54ddb257d008592563c4d753aa54/sys/sys/cdefs.h#L317

Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>

Re: [PATCH] Makefile: FreeBSD cannot do C99-or-below build

[Ævar Arnfjörð Bjarmason]

On Tue, Jan 18 2022, Neeraj Singh wrote:

> The approach of building with C11 on FreeBSD is a good one
> compared to trying to hack around the headers.

The "hack around the headers" suggests that you've seen my
alternate[1]....

> It appears more like a compiler bug that's being worked around
> here. The FreeBSD header supposedly uses a GCC extension if the
> C standard version is less than C11.  See: 
> https://github.com/freebsd/freebsd-src/blob/1e7b5f950b2d54ddb257d008592563c4d753aa54/sys/sys/cdefs.h#L317

...which discusses how the line you're linking to here and
__has_extension() interact, it's not a compiler bug, unless I'm missing
something.

I.e. it's just a FreeBSD include asking the compiler the wrong question
and/or not suppressing the warning (e.g. via "#pragma clang
system_header").

> Signed-off-by: Neeraj Singh <neerajsi@microsoft.com>

Accidentally addition?

1. https://lore.kernel.org/git/patch-1.1-06cc12be94d-20220118T151234Z-avarab@gmail.com/

Re: [PATCH] Makefile: FreeBSD cannot do C99-or-below build

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Tue, Jan 18 2022, Neeraj Singh wrote:
>
>> The approach of building with C11 on FreeBSD is a good one
>> compared to trying to hack around the headers.
>
> The "hack around the headers" suggests that you've seen my
> alternate[1]....
>
>> It appears more like a compiler bug that's being worked around
>> here. The FreeBSD header supposedly uses a GCC extension if the
>> C standard version is less than C11.  See: 
>> https://github.com/freebsd/freebsd-src/blob/1e7b5f950b2d54ddb257d008592563c4d753aa54/sys/sys/cdefs.h#L317
>
> ...which discusses how the line you're linking to here and
> __has_extension() interact, it's not a compiler bug, unless I'm missing
> something.
>
> I.e. it's just a FreeBSD include asking the compiler the wrong question
> and/or not suppressing the warning (e.g. via "#pragma clang
> system_header").

Yup, I do agree with you that this looks more like a bug in the
header files.

Re: [PATCH] config.mak.dev: fix DEVELOPER=1 on FreeBSD with -std=gnu99

[Ævar Arnfjörð Bjarmason]

On Tue, Jan 18 2022, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
>
>> The inherent problem is that __has_extension() is a way to ask
>> Clang (and GCC) whether the compiler supports that feature, but the
>> -Wc11-extensions warning will be issued on the basis of the selected
>> __STDC_VERSION__. With -std=gnu99 the __has_extension() built-in will
>> return true, but the warning will still fire.
>>
>> Let's narrowly work around this by checking whether:
>
> The end result, if and when a change along this line proves OK for
> all versions of FreeBSD we care about, may be narrow, but as a band
> aid to discuss just before the final release, I do not know if
> anybody can sensibly assess the ramifications of such a change,
> except for somebody whose primary development environment has been
> FreeBSD for the past 6 months or more.

libgen.h is a trivially sized header (~20 lines of non-comment
non-whitespace), and uses __generic for exactly one thing.

If you'd like to go with your patch instead that's fair enough, but I
don't see what using FreeBSD as a primary development environment for an
extended time would buy anyone in understanding this narrow issue
causing the C11 warning.

> This looks way more than we can take comfortably at this point in
> the release cycle, compared to "fixing" the developer build by
> kicking developers on FreeBSD out of the "we are C99 or below at
> this point" enforcement mechanism.
>
>>  A. We are under FreeBSD
>>  B. We are using a C standard older than C11
>>
>> In that case we'll include sys/cdefs.h, and undefine FreeBSD's
>> __generic() wrapper if it's been defined. We'll then load libgen.h,
>> and restore whatever __generic definition we temporarily undefined
>> earlier.
>>
>> An alternate solution would be to simply define NO_LIBGEN_H=Y in
>> config.mak.uname for FreeBSD, but this way we'll use its OS-provided
>> basename() and dirname(), and in the general case ensure that nothing
>> changes outside of DEVELOPER=1 builds.
>>
>> 1. https://github.com/freebsd/freebsd-src/commit/62b7f85d4749
>>
>> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
>> ---
>>  git-compat-util.h | 18 ++++++++++++++++++
>>  1 file changed, 18 insertions(+)
>>
>> diff --git a/git-compat-util.h b/git-compat-util.h
>> index 1229c8296b9..69d9b5f202f 100644
>> --- a/git-compat-util.h
>> +++ b/git-compat-util.h
>> @@ -318,7 +318,25 @@ static inline int setitimer(int which, const struct itimerval *value, struct iti
>>  #endif
>>  
>>  #ifndef NO_LIBGEN_H
>> +/*
>> + * FreeBSD's libgen.h inadvertently requires C11 features, due to its
>> + * sys/cdefs.h using _Generic() if
>> + * __has_extension(c_generic_selections) is true, regardless of
>> + * __STDC_VERSION__....
>> + */
>> +#if defined(__FreeBSD__) && __STDC_VERSION__ - 0 < 201112L
>> +#include <sys/cdefs.h>
>> +#ifdef __generic
>> +#define __fbsd_generic __generic
>> +#endif
>> +#undef __generic
>> +#endif
>>  #include <libgen.h>
>> +/* ...continue FreeBSD-specific hack above */
>> +#ifdef __fbsd_generic
>> +#define __generic __fbsd_generic
>> +#undef __fbsd_generic
>> +#endif
>>  #else
>>  #define basename gitbasename
>>  char *gitbasename(char *);

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Tue, Jan 18 2022, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> I'd like to have it too, but for context needing to add NO_UNCOMPRESS2=Y
>> (which Junio's punted on for the final[2]) is a much more widespread
>> issue of needing new post-install build tweaking than this issue that
>> only affects developer builds on FreeBSD.
>
> I hate it when people misrepresent what I said, even in an attempt
> to spite me.

Let's assume some mutual good faith here. I'm just trying to help debug
last-minute issues in this release to help get it out the door.

If I've misrepresented your views that's clearly true if you're the one
claiming it, but it's not intentional.

I was trying to read the tea leaves here in terms of helping patch
triage along. My assessment of that NO_UNCOMPRESS2=Y issue from the
boxes I've tested on is that it's a much more widespread problem new in
this release (this is purely from testing on the GCC farm, however
representative that is). It'll impact multiple OS's, linux distros &
versions.

Whereas the C11 warning is "just" recent FreeBSD && DEVELOPER=1.

So I assumed if you weren't interested in the former before the final
you probably wouldn't be in the latter, but wanted to provide a more
narrow fix in case you were.

> For the "check ZLIB_VERSION_NUM" topic, I gave you
> specific things that needs to be different from the version posted
> with reasons, fully expecting that a better version of the patch to
> materialize soon (knowing how proliferate you can be when you want
> to) to give us enough time to assess the potential damage.

I can re-roll it sooner than later if you'd like, but figured per your
"I like the basic idea of the patch, but I am afraid that it is way too
late in the cycle"[1] that the resulting on-list distraction wouldn't be
worth it at this point, if it wasn't being considered for being applied
in the release window.

Aside from "I can re-roll it". I also think that the point of making
that change mostly (but not entirely) disappears if it isn't included in
v2.35.0.

I.e. the point of doing it is to avoid the one-time pain of anyone
building new releases of git on $OLD_OS/$OLD_DISTRO not having to run
into the compilation error that's fixed with NO_UNCOMPRESS2=Y.

So if we put out a release without it anyone who's compiling git
releases will need to adjust their build system for that anyway (or
they're using autoconf, where it'll Just Work).

If we then get this into v2.36.0 there'll be someone somewhere that
benefits, but I'd think the ship has sailed for most of those who'd
avoid the needless flag twiddling (git-packagers@ et al).

So I don't know if it's even worth it to pursue that patch in that
case...

> I wouldn't call that "punted".

I don't think I've used or heard that word outside of this ML. FWIW I'm
using it in the sense of "kicking the can down the road" or "deferred
it".

Maybe that's incorrect, and I certainly stand corrected if I implied
something that wasn't true about our views by using that word.

> For this one, config.mak.dev patch WOULD only affect developer
> builds, which is a much better solution than overriding what their
> system headers want to do and force compiling with C99 mode.  With
> the status quo with today's code, with or without the patch Dscho
> wants in this thread, means ANYBODY will be stopped when they
> attempt to build with -std=gnu99 on FreeBSD, which is a GOOD thing.
>
> The reason why it is a much better solution to PUNT on using C99
> mode on FreeBSD is because this episode makes it very clear that
> nobody tested building anything that use basename(), dirname(),
> etc. with C99 mode on the platform.  I do not trust such a build,
> even if we could work around the system header breakage.

Aside from anything else I think you're assuming a lot about warning
hygiene in a typical C project. Git's use of even optional -Werror is
fairly unusual.

If you Google search that error you'll find numerous past reports of it,
it just hasn't been solved.

I also think we discussed around the -std=gnu99 change that the "C99
mode" in compilers we tested isn't impacting generated code (although of
course it will if it's conditional on __STDC_VERSION___). Otherwise the
generated objects are the same, it's just what warnings/errors you get
that changes.

In this case the fallback case already existed without my __generic
hack, so forcing that codepath for libgen.h isn't new.

> This time we got lucky and wereq stopped by a compilation error, but
> I have a strong suspicion that C99-only mode of compiler on this
> platform is not as well battle tested as their standard mode of
> compilation that allows C11.  I simply do not think we want to waste
> developer's time with C99-only mode on this platform which may end
> up debugging the platform and not our program.  Next bug that will
> hit us may not be so friendly to clearly break compilation.
> Instead, the symptom may be a subtle runtime breakage that wastes
> people's time.

I'd share that opinion if this was aCC on HP/UX or xlc on AIX or
whatever. But in this case it's just vanilla clang, not some scary
platform-specific compiler. Its "C99-only mode" isn't anything different
on FreeBSD than Linux.

We're just talking about one specific and avoidable warning in
/usr/include/libgen.h on that OS, once we're past that its "C99-mode"
works just fine.

> After developers who work on FreeBSD (not Git developers who uses
> FreeBSD) ships an updated system headers so that more programs, not
> just us, are built and testsd with C99-only mode, perhaps it becomes
> usable as a platform to catch use of language features beyond C99,
> like everybody else.  But with such a clearly broken system headers,
> I do not think today's FreeBSD is ready for that.  I do trust their
> default settings that allows C11, a lot more than C99-only
> compilation with the "their libgen is broken, so here is a user side
> workaround" patch.

Fair enough. I'm happy for us to get past this one way or the other,
whether it's with your patch or mine.

I took your previous reservations about my suggestions to just do this
C99 checking in CI-only to mean that you'd like to keep -std=gnu99 on
DEVELOPER=1 if at all possible, which my approach does.

1. https://lore.kernel.org/git/xmqq7dayw732.fsf@gitster.g/

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> Whereas the C11 warning is "just" recent FreeBSD && DEVELOPER=1.
>
> So I assumed if you weren't interested in the former before the final
> you probably wouldn't be in the latter, but wanted to provide a more
> narrow fix in case you were.

If we muck with the inclusion of libgen.h, it then becomes a problem
for everybody who builds on FreeBSD, not just the developer builds.
IOW, it is not even narrower to begin with.  Giving the same
potential breakage to everybody will make it easier to diagnose it,
but because I do not trust -std=gnu99 on today's FreeBSD, I think it
is a problem we do not even have to solve.

> I.e. the point of doing it is to avoid the one-time pain of anyone
> building new releases of git on $OLD_OS/$OLD_DISTRO not having to run
> into the compilation error that's fixed with NO_UNCOMPRESS2=Y.
> ...
> If we then get this into v2.36.0 there'll be someone somewhere that
> benefits, but I'd think the ship has sailed for most of those who'd
> avoid the needless flag twiddling (git-packagers@ et al).

I actually think it is a good thing.  It is what they brought onto
themselves.  They can follow David's example next time.

Re: v2.35.0 DEVELOPER=1 regression

[Ævar Arnfjörð Bjarmason]

On Tue, Jan 18 2022, Junio C Hamano wrote:

> Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:
>
>> Whereas the C11 warning is "just" recent FreeBSD && DEVELOPER=1.
>>
>> So I assumed if you weren't interested in the former before the final
>> you probably wouldn't be in the latter, but wanted to provide a more
>> narrow fix in case you were.
>
> If we muck with the inclusion of libgen.h, it then becomes a problem
> for everybody who builds on FreeBSD, not just the developer builds.
> IOW, it is not even narrower to begin with.  Giving the same
> potential breakage to everybody will make it easier to diagnose it,
> but because I do not trust -std=gnu99 on today's FreeBSD, I think it
> is a problem we do not even have to solve.

The patch I submitted could trivially have an additional "#ifdef
DEVELOPER" or equivalent, which would narrow it down even further.

The reason it doesn't is because we provide -std=gnu99 in
config.mak.dev, but that doesn't mean that we don't have to deal with
e.g. a user-supplied -std=gnu99.

Except of course if we declare that we're not going to generally support
such a thing, and you should only provide such flags via
DEVELOPER/DEVOPTS, not manually in CFLAGS, which is fair enough.

>> I.e. the point of doing it is to avoid the one-time pain of anyone
>> building new releases of git on $OLD_OS/$OLD_DISTRO not having to run
>> into the compilation error that's fixed with NO_UNCOMPRESS2=Y.
>> ...
>> If we then get this into v2.36.0 there'll be someone somewhere that
>> benefits, but I'd think the ship has sailed for most of those who'd
>> avoid the needless flag twiddling (git-packagers@ et al).
>
> I actually think it is a good thing.  It is what they brought onto
> themselves.  They can follow David's example next time.

Do you mean David Aguilar's addition of "NO_UNCOMPRESS2 = YesPlease" in
[1]? Isn't that mutually exclusive with pursuing my change to
auto-detect it[2] instead?

As noted I'm a bit "meh" on my own patch if it's not in the release. I'm
just trying to see where you stand, since you seemed to want a re-roll
of it post-release....

1. https://lore.kernel.org/git/20220116020520.26895-1-davvid@gmail.com/
2. https://lore.kernel.org/git/patch-1.1-9cea01a1395-20220117T170457Z-avarab@gmail.com/

Re: v2.35.0 DEVELOPER=1 regression

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> As noted I'm a bit "meh" on my own patch if it's not in the release. I'm
> just trying to see where you stand, since you seemed to want a re-roll
> of it post-release....

I was hoping to see a quick reroll that would be in time for the
release, but it seems it was overly optimistic back when I gave my
initial review.  I do not see it happening given that I'd be tagging
the -rc2 tomorrow.

I do think in the longer term, we should aim to retire the current
mechanism and depend more on ZLIB_VERSION_NUM.  I am not optimistic
that we can have a reasonable version before the final.

[PATCH v6 00/15] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'master'.

This addresses some complaints from Coverity that Peff reported.

v6:

 * drop 0-length memcpy paranoia patch.

Han-Wen Nienhuys (15):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in block.c error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: all xxx_free() functions accept NULL arguments
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable-record.h function signatures const correct
  reftable: implement record equality generically
  reftable: remove outdated file reftable.c
  reftable: make reftable_record a tagged union
  reftable: add print functions to the record types

 reftable/block.c           |  28 ++-
 reftable/block_test.c      |  22 +--
 reftable/blocksource.c     |   6 +-
 reftable/generic.c         |  41 +++--
 reftable/iter.c            |   4 +-
 reftable/merged.c          |  33 ++--
 reftable/pq.c              |   3 +-
 reftable/pq_test.c         |  27 ++-
 reftable/reader.c          | 112 ++++++------
 reftable/readwrite_test.c  |  67 ++++++-
 reftable/record.c          | 366 ++++++++++++++++++++++++-------------
 reftable/record.h          |  49 ++---
 reftable/record_test.c     | 197 ++++++++++----------
 reftable/reftable-record.h |  14 +-
 reftable/reftable.c        | 115 ------------
 reftable/stack.c           |  10 +-
 reftable/stack_test.c      |   3 +-
 reftable/writer.c          |  43 +++--
 t/helper/test-reftable.c   |   9 +-
 19 files changed, 620 insertions(+), 529 deletions(-)
 delete mode 100644 reftable/reftable.c


base-commit: 50b2d72e110cad39ecaf2322bfdf1b60cd13dd96
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v6
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v6
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v5:

  1:  e7f1be7bbec =  1:  9218bd59b25 reftable: fix OOB stack write in print functions
  2:  9ab631a3b29 !  2:  315ce62e710 reftable: fix resource leak in block.c error path
     @@ reftable/reader.c: int reader_init_block_reader(struct reftable_reader *r, struc
      +	err = block_reader_init(br, &block, header_off, r->block_size,
      +				hash_size(r->hash_id));
      +done:
     -+	if (err)
     -+		reftable_block_done(&block);
     ++	reftable_block_done(&block);
      +
      +	return err;
       }
     @@ reftable/readwrite_test.c: static void test_log_write_read(void)
      +		},
      +	};
      +
     -+	for (i = 0; i < sizeof(message)-1; i++)
     ++	for (i = 0; i < sizeof(message) - 1; i++)
      +		message[i] = (uint8_t)(rand() % 64 + ' ');
      +
      +	reftable_writer_set_limits(w, 1, 1);
  3:  8301000e83b !  3:  7cf65e141cf reftable: fix resource leak blocksource.c
     @@ Metadata
       ## Commit message ##
          reftable: fix resource leak blocksource.c
      
     -    This would be triggered in the unlikely event of fstat() failing on an opened
     -    file.
     +    This would be triggered in the unlikely event of fstat() failing on an
     +    opened file.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
  4:  e6e53aabc39 =  4:  8740ba92981 reftable: check reftable_stack_auto_compact() return value
  5:  7551bcdd917 =  5:  08be6d90a48 reftable: ignore remove() return value in stack_test.c
  6:  700a4e247e7 !  6:  630f67bdb51 reftable: fix resource warning
     @@ Metadata
       ## Commit message ##
          reftable: fix resource warning
      
     -    This would trigger in the unlikely event that we are compacting, and the next
     -    available file handle is 0.
     +    This would trigger in the unlikely event that we are compacting, and
     +    the next available file handle is 0.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
     @@ reftable/stack.c: static int stack_compact_range(struct reftable_stack *st, int
       	have_lock = 1;
       	err = stack_uptodate(st);
      @@ reftable/stack.c: static int stack_compact_range(struct reftable_stack *st, int first, int last,
     - 
     - 		sublock_file_fd = open(subtab_lock.buf,
     - 				       O_EXCL | O_CREAT | O_WRONLY, 0644);
     --		if (sublock_file_fd > 0) {
     -+		if (sublock_file_fd >= 0) {
     - 			close(sublock_file_fd);
     - 		} else if (sublock_file_fd < 0) {
     - 			if (errno == EEXIST) {
     -@@ reftable/stack.c: static int stack_compact_range(struct reftable_stack *st, int first, int last,
       		goto done;
       	}
       	err = close(lock_file_fd);
  7:  30016fee4c8 =  7:  afee67fb60d reftable: all xxx_free() functions accept NULL arguments
  8:  f90846cc103 !  8:  6f4f24c17ad reftable: order unittests by complexity
     @@ Metadata
       ## Commit message ##
          reftable: order unittests by complexity
      
     -    This is a more practical ordering when working on refactorings of the reftable
     -    code.
     +    This is a more practical ordering when working on refactorings of the
     +    reftable code.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
  9:  425e12d3667 =  9:  0abaab1d7c4 reftable: drop stray printf in readwrite_test
 10:  67d858ec59e = 10:  004b7a736bf reftable: handle null refnames in reftable_ref_record_equal
 11:  1e50924894f = 11:  951efcebcd2 reftable: make reftable-record.h function signatures const correct
 12:  c81b17ad57e = 12:  6385e449ba7 reftable: implement record equality generically
 13:  4175089ec43 = 13:  69c4a554882 reftable: remove outdated file reftable.c
 14:  ede2e792ab6 ! 14:  54bcac325bd reftable: make reftable_record a tagged union
     @@ Metadata
       ## Commit message ##
          reftable: make reftable_record a tagged union
      
     -    This reduces the amount of glue code, because we don't need a void pointer or
     -    vtable within the structure.
     +    This reduces the amount of glue code, because we don't need a void
     +    pointer or vtable within the structure.
      
     -    The only snag is that reftable_index_record contain a strbuf, so it cannot be
     -    zero-initialized. To address this, use reftable_new_record() to return fresh
     -    instance, given a record type. Since reftable_new_record() doesn't cause heap
     -    allocation anymore, it should be balanced with reftable_record_release() rather
     -    than reftable_record_destroy().
     +    The only snag is that reftable_index_record contain a strbuf, so it
     +    cannot be zero-initialized. To address this, use reftable_new_record()
     +    to return fresh instance, given a record type. Since
     +    reftable_new_record() doesn't cause heap allocation anymore, it should
     +    be balanced with reftable_record_release() rather than
     +    reftable_record_destroy().
      
          Thanks to Peff for the suggestion.
      
 15:  6c06bd91662 = 15:  82f140cab5c reftable: add print functions to the record types
 16:  e16bf0c5212 <  -:  ----------- reftable: be more paranoid about 0-length memcpy calls

-- 
gitgitgadget

[PATCH v6 01/15] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..8536bd03aa9 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v6 02/15] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
unittest.

This problem was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c          | 26 +++++++++------
 reftable/reader.c         | 23 ++++++++------
 reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+), 18 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..6c8e8705205 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint32_t full_block_size = table_block_size;
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
-
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +206,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -212,16 +215,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +257,9 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	reftable_free(uncompressed);
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..35781593a29 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -290,28 +290,33 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
 
 	err = reader_get_block(r, &block, next_off, guess_block_size);
 	if (err < 0)
-		return err;
+		goto done;
 
 	block_size = extract_block_size(block.data, &block_typ, next_off,
 					r->version);
-	if (block_size < 0)
-		return block_size;
-
+	if (block_size < 0) {
+		err = block_size;
+		goto done;
+	}
 	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
-		reftable_block_done(&block);
-		return 1;
+		err = 1;
+		goto done;
 	}
 
 	if (block_size > guess_block_size) {
 		reftable_block_done(&block);
 		err = reader_get_block(r, &block, next_off, block_size);
 		if (err < 0) {
-			return err;
+			goto done;
 		}
 	}
 
-	return block_reader_init(br, &block, header_off, r->block_size,
-				 hash_size(r->hash_id));
+	err = block_reader_init(br, &block, header_off, r->block_size,
+				hash_size(r->hash_id));
+done:
+	reftable_block_done(&block);
+
+	return err;
 }
 
 static int table_iter_next_block(struct table_iter *dest,
diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 70c7aedba2c..edb9cfc1930 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -288,6 +288,71 @@ static void test_log_write_read(void)
 	reader_close(&rd);
 }
 
+static void test_log_zlib_corruption(void)
+{
+	struct reftable_write_options opts = {
+		.block_size = 256,
+	};
+	struct reftable_iterator it = { 0 };
+	struct reftable_reader rd = { 0 };
+	struct reftable_block_source source = { 0 };
+	struct strbuf buf = STRBUF_INIT;
+	struct reftable_writer *w =
+		reftable_new_writer(&strbuf_add_void, &buf, &opts);
+	const struct reftable_stats *stats = NULL;
+	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
+	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
+	char message[100] = { 0 };
+	int err, i, n;
+
+	struct reftable_log_record log = {
+		.refname = "refname",
+		.value_type = REFTABLE_LOG_UPDATE,
+		.value = {
+			.update = {
+				.new_hash = hash1,
+				.old_hash = hash2,
+				.name = "My Name",
+				.email = "myname@invalid",
+				.message = message,
+			},
+		},
+	};
+
+	for (i = 0; i < sizeof(message) - 1; i++)
+		message[i] = (uint8_t)(rand() % 64 + ' ');
+
+	reftable_writer_set_limits(w, 1, 1);
+
+	err = reftable_writer_add_log(w, &log);
+	EXPECT_ERR(err);
+
+	n = reftable_writer_close(w);
+	EXPECT(n == 0);
+
+	stats = writer_stats(w);
+	EXPECT(stats->log_stats.blocks > 0);
+	reftable_writer_free(w);
+	w = NULL;
+
+	/* corrupt the data. */
+	buf.buf[50] ^= 0x99;
+
+	block_source_from_strbuf(&source, &buf);
+
+	err = init_reader(&rd, &source, "file.log");
+	EXPECT_ERR(err);
+
+	err = reftable_reader_seek_log(&rd, &it, "refname");
+	EXPECT(err == REFTABLE_ZLIB_ERROR);
+
+	reftable_iterator_destroy(&it);
+
+	/* cleanup. */
+	strbuf_release(&buf);
+	reader_close(&rd);
+}
+
 static void test_table_read_write_sequential(void)
 {
 	char **names;
@@ -667,6 +732,7 @@ static void test_corrupt_table(void)
 
 int readwrite_test_main(int argc, const char *argv[])
 {
+	RUN_TEST(test_log_zlib_corruption);
 	RUN_TEST(test_corrupt_table);
 	RUN_TEST(test_corrupt_table_empty);
 	RUN_TEST(test_log_write_read);
-- 
gitgitgadget

[PATCH v6 03/15] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an
opened file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v6 04/15] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Fixes a problem detected by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index f4c743db80c..e84f50d27ff 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -839,6 +839,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v6 05/15] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index e84f50d27ff..19fe4e20085 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -90,7 +90,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	(void) remove(fn);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v6 06/15] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and
the next available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index 56bf5f2d84a..ddbdf1b9c8b 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -889,7 +889,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -923,7 +923,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -1031,7 +1031,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1068,9 +1068,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v6 07/15] reftable: all xxx_free() functions accept NULL arguments

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This fixes NULL derefs in error paths. Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 2 ++
 reftable/writer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/reftable/reader.c b/reftable/reader.c
index 35781593a29..272378ed1d5 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -646,6 +646,8 @@ int reftable_new_reader(struct reftable_reader **p,
 
 void reftable_reader_free(struct reftable_reader *r)
 {
+	if (!r)
+		return;
 	reader_close(r);
 	reftable_free(r);
 }
diff --git a/reftable/writer.c b/reftable/writer.c
index 35c8649c9b7..83a23daf60b 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -150,6 +150,8 @@ void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,
 
 void reftable_writer_free(struct reftable_writer *w)
 {
+	if (!w)
+		return;
 	reftable_free(w->block);
 	reftable_free(w);
 }
-- 
gitgitgadget

[PATCH v6 08/15] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more practical ordering when working on refactorings of the
reftable code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..1f0a28cbb64 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	/* test from simple to complex. */
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v6 09/15] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index edb9cfc1930..605ba0f9fd4 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -696,7 +696,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v6 10/15] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8536bd03aa9..8bbcbff1e69 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v6 11/15] reftable: make reftable-record.h function signatures const correct

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c          | 14 +++++++-------
 reftable/reftable-record.h | 14 +++++++-------
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 8bbcbff1e69..f7c77c51539 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -72,7 +72,7 @@ int reftable_is_block_type(uint8_t typ)
 	return 0;
 }
 
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL1:
@@ -84,7 +84,7 @@ uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
 	}
 }
 
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL2:
@@ -251,7 +251,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
@@ -881,8 +881,8 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size)
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size)
 {
 	if (!(null_streq(a->refname, b->refname) &&
 	      a->update_index == b->update_index &&
@@ -1150,8 +1150,8 @@ static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
 	return a == b;
 }
 
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size)
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
 	if (!null_streq(a->refname, b->refname))
diff --git a/reftable/reftable-record.h b/reftable/reftable-record.h
index 5370d2288c7..67104f8fbfe 100644
--- a/reftable/reftable-record.h
+++ b/reftable/reftable-record.h
@@ -49,25 +49,25 @@ struct reftable_ref_record {
 
 /* Returns the first hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec);
 
 /* Returns the second hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec);
 
 /* returns whether 'ref' represents a deletion */
 int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);
 
 /* prints a reftable_ref_record onto stdout. Useful for debugging. */
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id);
 
 /* frees and nulls all pointer values inside `ref`. */
 void reftable_ref_record_release(struct reftable_ref_record *ref);
 
 /* returns whether two reftable_ref_records are the same. Useful for testing. */
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size);
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size);
 
 /* reftable_log_record holds a reflog entry */
 struct reftable_log_record {
@@ -104,8 +104,8 @@ int reftable_log_record_is_deletion(const struct reftable_log_record *log);
 void reftable_log_record_release(struct reftable_log_record *log);
 
 /* returns whether two records are equal. Useful for testing. */
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size);
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size);
 
 /* dumps a reftable_log_record on stdout, for debugging/testing. */
 void reftable_log_record_print(struct reftable_log_record *log,
-- 
gitgitgadget

[PATCH v6 12/15] reftable: implement record equality generically

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This simplifies unittests a little, and provides further coverage for
reftable_record_copy().

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 57 +++++++++++++++++++++++++++++++++++++++++-
 reftable/record.h      |  5 +++-
 reftable/record_test.c | 23 +++--------------
 3 files changed, 63 insertions(+), 22 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index f7c77c51539..2a9e41a992e 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -430,6 +430,15 @@ static int reftable_ref_record_is_deletion_void(const void *p)
 		(const struct reftable_ref_record *)p);
 }
 
+
+static int reftable_ref_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	struct reftable_ref_record *ra = (struct reftable_ref_record *) a;
+	struct reftable_ref_record *rb = (struct reftable_ref_record *) b;
+	return reftable_ref_record_equal(ra, rb, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -439,6 +448,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.decode = &reftable_ref_record_decode,
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
+	.equal = &reftable_ref_record_equal_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -572,6 +582,25 @@ static int not_a_deletion(const void *p)
 	return 0;
 }
 
+static int reftable_obj_record_equal_void(const void *a, const void *b, int hash_size)
+{
+	struct reftable_obj_record *ra = (struct reftable_obj_record *) a;
+	struct reftable_obj_record *rb = (struct reftable_obj_record *) b;
+
+	if (ra->hash_prefix_len != rb->hash_prefix_len
+	    || ra->offset_len != rb->offset_len)
+		return 0;
+
+	if (ra->hash_prefix_len &&
+	    memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
+		return 0;
+	if (ra->offset_len &&
+	    memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
+		return 0;
+
+	return 1;
+}
+
 static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.key = &reftable_obj_record_key,
 	.type = BLOCK_TYPE_OBJ,
@@ -580,7 +609,8 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.encode = &reftable_obj_record_encode,
 	.decode = &reftable_obj_record_decode,
 	.release = &reftable_obj_record_release,
-	.is_deletion = not_a_deletion,
+	.is_deletion = &not_a_deletion,
+	.equal = &reftable_obj_record_equal_void,
 };
 
 void reftable_log_record_print(struct reftable_log_record *log,
@@ -881,6 +911,14 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
+static int reftable_log_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	return reftable_log_record_equal((struct reftable_log_record *) a,
+					 (struct reftable_log_record *) b,
+					 hash_size);
+}
+
 int reftable_log_record_equal(const struct reftable_log_record *a,
 			      const struct reftable_log_record *b, int hash_size)
 {
@@ -924,6 +962,7 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
+	.equal = &reftable_log_record_equal_void
 };
 
 struct reftable_record reftable_new_record(uint8_t typ)
@@ -1042,6 +1081,14 @@ static int reftable_index_record_decode(void *rec, struct strbuf key,
 	return start.len - in.len;
 }
 
+static int reftable_index_record_equal(const void *a, const void *b, int hash_size)
+{
+	struct reftable_index_record *ia = (struct reftable_index_record *) a;
+	struct reftable_index_record *ib = (struct reftable_index_record *) b;
+
+	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1051,6 +1098,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.decode = &reftable_index_record_decode,
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
+	.equal = &reftable_index_record_equal,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1098,6 +1146,13 @@ int reftable_record_is_deletion(struct reftable_record *rec)
 	return rec->ops->is_deletion(rec->data);
 }
 
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
+{
+	if (a->ops != b->ops)
+		return 0;
+	return a->ops->equal(a->data, b->data, hash_size);
+}
+
 void reftable_record_from_ref(struct reftable_record *rec,
 			      struct reftable_ref_record *ref_rec)
 {
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..da75d7d1f11 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -58,6 +58,9 @@ struct reftable_record_vtable {
 
 	/* is this a tombstone? */
 	int (*is_deletion)(const void *rec);
+
+	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
+	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
 /* record is a generic wrapper for different types of records. */
@@ -98,7 +101,7 @@ struct reftable_obj_record {
 };
 
 /* see struct record_vtable */
-
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..92680848156 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -21,18 +21,7 @@ static void test_copy(struct reftable_record *rec)
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
-	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	}
+	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
 	reftable_record_destroy(&copy);
 }
 
@@ -346,13 +335,7 @@ static void test_reftable_obj_record_roundtrip(void)
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
 		reftable_record_release(&rec_out);
 	}
@@ -390,7 +373,7 @@ static void test_reftable_index_record_roundtrip(void)
 	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
 
 	reftable_record_release(&out_rec);
 	strbuf_release(&key);
-- 
gitgitgadget

[PATCH v6 13/15] reftable: remove outdated file reftable.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This was renamed to generic.c, but the origin was never removed

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reftable.c | 115 --------------------------------------------
 1 file changed, 115 deletions(-)
 delete mode 100644 reftable/reftable.c

diff --git a/reftable/reftable.c b/reftable/reftable.c
deleted file mode 100644
index 0e4607a7cd6..00000000000
--- a/reftable/reftable.c
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
-Copyright 2020 Google LLC
-
-Use of this source code is governed by a BSD-style
-license that can be found in the LICENSE file or at
-https://developers.google.com/open-source/licenses/bsd
-*/
-
-#include "basics.h"
-#include "record.h"
-#include "generic.h"
-#include "reftable-iterator.h"
-#include "reftable-generic.h"
-
-int reftable_table_seek_ref(struct reftable_table *tab,
-			    struct reftable_iterator *it, const char *name)
-{
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
-	return tab->ops->seek_record(tab->table_arg, it, &rec);
-}
-
-int reftable_table_read_ref(struct reftable_table *tab, const char *name,
-			    struct reftable_ref_record *ref)
-{
-	struct reftable_iterator it = { NULL };
-	int err = reftable_table_seek_ref(tab, &it, name);
-	if (err)
-		goto done;
-
-	err = reftable_iterator_next_ref(&it, ref);
-	if (err)
-		goto done;
-
-	if (strcmp(ref->refname, name) ||
-	    reftable_ref_record_is_deletion(ref)) {
-		reftable_ref_record_release(ref);
-		err = 1;
-		goto done;
-	}
-
-done:
-	reftable_iterator_destroy(&it);
-	return err;
-}
-
-uint64_t reftable_table_max_update_index(struct reftable_table *tab)
-{
-	return tab->ops->max_update_index(tab->table_arg);
-}
-
-uint64_t reftable_table_min_update_index(struct reftable_table *tab)
-{
-	return tab->ops->min_update_index(tab->table_arg);
-}
-
-uint32_t reftable_table_hash_id(struct reftable_table *tab)
-{
-	return tab->ops->hash_id(tab->table_arg);
-}
-
-void reftable_iterator_destroy(struct reftable_iterator *it)
-{
-	if (!it->ops) {
-		return;
-	}
-	it->ops->close(it->iter_arg);
-	it->ops = NULL;
-	FREE_AND_NULL(it->iter_arg);
-}
-
-int reftable_iterator_next_ref(struct reftable_iterator *it,
-			       struct reftable_ref_record *ref)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
-}
-
-int reftable_iterator_next_log(struct reftable_iterator *it,
-			       struct reftable_log_record *log)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
-}
-
-int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
-{
-	return it->ops->next(it->iter_arg, rec);
-}
-
-static int empty_iterator_next(void *arg, struct reftable_record *rec)
-{
-	return 1;
-}
-
-static void empty_iterator_close(void *arg)
-{
-}
-
-static struct reftable_iterator_vtable empty_vtable = {
-	.next = &empty_iterator_next,
-	.close = &empty_iterator_close,
-};
-
-void iterator_set_empty(struct reftable_iterator *it)
-{
-	assert(!it->ops);
-	it->iter_arg = NULL;
-	it->ops = &empty_vtable;
-}
-- 
gitgitgadget

[PATCH v6 14/15] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void
pointer or vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it
cannot be zero-initialized. To address this, use reftable_new_record()
to return fresh instance, given a record type. Since
reftable_new_record() doesn't cause heap allocation anymore, it should
be balanced with reftable_record_release() rather than
reftable_record_destroy().

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   2 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  41 ++++----
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  33 ++++---
 reftable/pq.c          |   3 +-
 reftable/pq_test.c     |  27 +++---
 reftable/reader.c      |  87 +++++++++--------
 reftable/record.c      | 212 +++++++++++++++++++----------------------
 reftable/record.h      |  42 ++++----
 reftable/record_test.c | 157 +++++++++++++++---------------
 reftable/writer.c      |  41 ++++----
 12 files changed, 334 insertions(+), 337 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 6c8e8705205..2170748c5e9 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -421,7 +421,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..b27d152e89a 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,21 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
+				       .u.ref = {
+					       .refname = (char *)name,
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = ~((uint64_t)0),
+				       } };
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +128,25 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..2a6efa110d5 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_new_record(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -290,11 +289,12 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +302,11 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = update_index,
+				       } };
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..96ca6dd37b3 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
 void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
 {
 	int i = 0;
+
 	if (pq->len == pq->cap) {
 		pq->cap = 2 * pq->cap + 1;
 		pq->heap = reftable_realloc(pq->heap,
@@ -98,7 +99,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..7de5e886f35 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,10 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = { .rec = { .type = BLOCK_TYPE_REF,
+					       .u.ref = {
+						       .refname = names[i],
+					       } } };
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +53,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		// this is names[i], so don't dealloc.
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 272378ed1d5..00906e7a2de 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,8 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -480,7 +479,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -490,34 +489,35 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX, .u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
-
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -545,8 +545,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -595,11 +595,12 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = {
+			.refname = (char *)name,
+		},
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -607,12 +608,11 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
+				       .u.log = {
+					       .refname = (char *)name,
+					       .update_index = update_index,
+				       } };
 	return reader_seek(r, it, &rec);
 }
 
@@ -656,31 +656,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -688,15 +690,16 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets,
+					 got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index 2a9e41a992e..a8cee628942 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+static void *reftable_record_data(struct reftable_record *rec);
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -475,12 +479,14 @@ static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 		(const struct reftable_obj_record *)src_rec;
 
 	reftable_obj_record_release(obj);
-	*obj = *src;
-	obj->hash_prefix = reftable_malloc(obj->hash_prefix_len);
-	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
+	obj->hash_prefix = reftable_malloc(src->hash_prefix_len);
+	obj->hash_prefix_len = src->hash_prefix_len;
+	if (src->hash_prefix_len)
+		memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
 
-	obj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));
-	COPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);
+	obj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));
+	obj->offset_len = src->offset_len;
+	COPY_ARRAY(obj->offsets, src->offsets, src->offset_len);
 }
 
 static uint8_t reftable_obj_record_val_type(const void *rec)
@@ -965,58 +971,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.equal = &reftable_log_record_equal_void
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1103,98 +1057,60 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 uint8_t reftable_record_type(struct reftable_record *rec)
 {
-	return rec->ops->type;
+	return rec->type;
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
 {
-	if (a->ops != b->ops)
+	if (a->type != b->type)
 		return 0;
-	return a->ops->equal(a->data, b->data, hash_size);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(a)->equal(
+		reftable_record_data(a), reftable_record_data(b), hash_size);
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1267,3 +1183,75 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_new_record(uint8_t typ)
+{
+	struct reftable_record clean = {
+		.type = typ,
+	};
+
+	/* the following is involved, but the naive solution (just return
+	 * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage
+	 * clean.u.obj.offsets pointer on Windows VS CI.  Go figure.
+	 */
+	switch (typ) {
+	case BLOCK_TYPE_OBJ:
+	{
+		struct reftable_obj_record obj = { 0 };
+		clean.u.obj = obj;
+		break;
+	}
+	case BLOCK_TYPE_INDEX:
+	{
+		struct reftable_index_record idx = {
+			.last_key = STRBUF_INIT,
+		};
+		clean.u.idx = idx;
+		break;
+	}
+	case BLOCK_TYPE_REF:
+	{
+		struct reftable_ref_record ref = { 0 };
+		clean.u.ref = ref;
+		break;
+	}
+	case BLOCK_TYPE_LOG:
+	{
+		struct reftable_log_record log = { 0 };
+		clean.u.log = log;
+		break;
+	}
+	}
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index da75d7d1f11..010a322e901 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -63,16 +63,10 @@ struct reftable_record_vtable {
 	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
+/* return an initialized record for the given type */
 struct reftable_record reftable_new_record(uint8_t typ);
 
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
@@ -100,6 +94,22 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_new_record() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
@@ -114,25 +124,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 92680848156..c6fdd1925a9 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,13 +16,16 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = { 0 };
+	uint8_t typ;
+
+	typ = reftable_record_type(rec);
+	copy = reftable_new_record(typ);
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -95,61 +98,58 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = { .type = BLOCK_TYPE_REF };
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -202,7 +202,7 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -210,23 +210,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -234,16 +236,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -311,41 +313,43 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -353,31 +357,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
+	EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 83a23daf60b..944c2329ab5 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -256,8 +256,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -266,8 +268,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -306,7 +307,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -316,8 +320,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
-
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -398,8 +400,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -467,17 +471,17 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
-	};
-	struct reftable_record rec = { NULL };
+	struct reftable_record
+		rec = { .type = BLOCK_TYPE_OBJ,
+			.u.obj = {
+				.hash_prefix = (uint8_t *)entry->hash.buf,
+				.hash_prefix_len = arg->w->stats.object_id_len,
+				.offsets = entry->offsets,
+				.offset_len = entry->offset_len,
+			} };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -490,7 +494,8 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

[PATCH v6 15/15] reftable: add print functions to the record types

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This isn't used per se, but it is useful for debugging, especially
Windows CI failures.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 85 +++++++++++++++++++++++++++++++++++-------
 reftable/record.h      |  4 ++
 reftable/record_test.c | 21 ++++++++++-
 3 files changed, 95 insertions(+), 15 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index a8cee628942..fbaa1fbef56 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -255,8 +255,8 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(const struct reftable_ref_record *ref,
-			       uint32_t hash_id)
+static void reftable_ref_record_print_sz(const struct reftable_ref_record *ref,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
@@ -265,14 +265,14 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 		printf("=> %s", ref->value.symref);
 		break;
 	case REFTABLE_REF_VAL2:
-		hex_format(hex, ref->value.val2.value, hash_size(hash_id));
+		hex_format(hex, ref->value.val2.value, hash_size);
 		printf("val 2 %s", hex);
 		hex_format(hex, ref->value.val2.target_value,
-			   hash_size(hash_id));
+			   hash_size);
 		printf("(T %s)", hex);
 		break;
 	case REFTABLE_REF_VAL1:
-		hex_format(hex, ref->value.val1, hash_size(hash_id));
+		hex_format(hex, ref->value.val1, hash_size);
 		printf("val 1 %s", hex);
 		break;
 	case REFTABLE_REF_DELETION:
@@ -282,6 +282,11 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 	printf("}\n");
 }
 
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
+			       uint32_t hash_id) {
+	reftable_ref_record_print_sz(ref, hash_size(hash_id));
+}
+
 static void reftable_ref_record_release_void(void *rec)
 {
 	reftable_ref_record_release(rec);
@@ -443,6 +448,12 @@ static int reftable_ref_record_equal_void(const void *a,
 	return reftable_ref_record_equal(ra, rb, hash_size);
 }
 
+static void reftable_ref_record_print_void(const void *rec,
+					   int hash_size)
+{
+	reftable_ref_record_print_sz((struct reftable_ref_record *) rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -453,6 +464,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
 	.equal = &reftable_ref_record_equal_void,
+	.print = &reftable_ref_record_print_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -471,6 +483,21 @@ static void reftable_obj_record_release(void *rec)
 	memset(obj, 0, sizeof(struct reftable_obj_record));
 }
 
+static void reftable_obj_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_obj_record *obj = rec;
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
+	struct strbuf offset_str = STRBUF_INIT;
+	int i;
+
+	for (i = 0; i < obj->offset_len; i++)
+		strbuf_addf(&offset_str, "%" PRIu64 " ", obj->offsets[i]);
+	hex_format(hex, obj->hash_prefix, obj->hash_prefix_len);
+	printf("prefix %s (len %d), offsets [%s]\n",
+	       hex, obj->hash_prefix_len, offset_str.buf);
+	strbuf_release(&offset_str);
+}
+
 static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 					  int hash_size)
 {
@@ -617,31 +644,41 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.release = &reftable_obj_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_obj_record_equal_void,
+	.print = &reftable_obj_record_print,
 };
 
-void reftable_log_record_print(struct reftable_log_record *log,
-			       uint32_t hash_id)
+static void reftable_log_record_print_sz(struct reftable_log_record *log,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-		printf("log{%s(%" PRIu64 ") delete", log->refname,
+		printf("log{%s(%" PRIu64 ") delete\n", log->refname,
 		       log->update_index);
 		break;
 	case REFTABLE_LOG_UPDATE:
 		printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
-		       log->refname, log->update_index, log->value.update.name,
-		       log->value.update.email, log->value.update.time,
+		       log->refname, log->update_index,
+		       log->value.update.name ? log->value.update.name : "",
+		       log->value.update.email ? log->value.update.email : "",
+		       log->value.update.time,
 		       log->value.update.tz_offset);
-		hex_format(hex, log->value.update.old_hash, hash_size(hash_id));
+		hex_format(hex, log->value.update.old_hash, hash_size);
 		printf("%s => ", hex);
-		hex_format(hex, log->value.update.new_hash, hash_size(hash_id));
-		printf("%s\n\n%s\n}\n", hex, log->value.update.message);
+		hex_format(hex, log->value.update.new_hash, hash_size);
+		printf("%s\n\n%s\n}\n", hex,
+		       log->value.update.message ? log->value.update.message : "");
 		break;
 	}
 }
 
+void reftable_log_record_print(struct reftable_log_record *log,
+				      uint32_t hash_id)
+{
+	reftable_log_record_print_sz(log, hash_size(hash_id));
+}
+
 static void reftable_log_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_log_record *rec =
@@ -959,6 +996,11 @@ static int reftable_log_record_is_deletion_void(const void *p)
 		(const struct reftable_log_record *)p);
 }
 
+static void reftable_log_record_print_void(const void *rec, int hash_size)
+{
+	reftable_log_record_print_sz((struct reftable_log_record*)rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_log_record_vtable = {
 	.key = &reftable_log_record_key,
 	.type = BLOCK_TYPE_LOG,
@@ -968,7 +1010,8 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
-	.equal = &reftable_log_record_equal_void
+	.equal = &reftable_log_record_equal_void,
+	.print = &reftable_log_record_print_void,
 };
 
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
@@ -1043,6 +1086,13 @@ static int reftable_index_record_equal(const void *a, const void *b, int hash_si
 	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
 }
 
+static void reftable_index_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_index_record *idx = rec;
+	/* TODO: escape null chars? */
+	printf("\"%s\" %" PRIu64 "\n", idx->last_key.buf, idx->offset);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1053,6 +1103,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_index_record_equal,
+	.print = &reftable_index_record_print,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1255,3 +1306,9 @@ struct reftable_record reftable_new_record(uint8_t typ)
 	}
 	return clean;
 }
+
+void reftable_record_print(struct reftable_record *rec, int hash_size)
+{
+	printf("'%c': ", rec->type);
+	reftable_record_vtable(rec)->print(reftable_record_data(rec), hash_size);
+}
diff --git a/reftable/record.h b/reftable/record.h
index 010a322e901..fd80cd451d5 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -61,6 +61,9 @@ struct reftable_record_vtable {
 
 	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
 	int (*equal)(const void *a, const void *b, int hash_size);
+
+	/* Print on stdout, for debugging. */
+	void (*print)(const void *rec, int hash_size);
 };
 
 /* returns true for recognized block types. Block start with the block type. */
@@ -112,6 +115,7 @@ struct reftable_record {
 
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
+void reftable_record_print(struct reftable_record *rec, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index c6fdd1925a9..f91ea5e8830 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -25,6 +25,10 @@ static void test_copy(struct reftable_record *rec)
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
+
+	puts("testing print coverage:\n");
+	reftable_record_print(&copy, GIT_SHA1_RAWSZ);
+
 	reftable_record_release(&copy);
 }
 
@@ -176,7 +180,8 @@ static void test_reftable_log_record_equal(void)
 static void test_reftable_log_record_roundtrip(void)
 {
 	int i;
-	struct reftable_log_record in[2] = {
+
+	struct reftable_log_record in[] = {
 		{
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 42,
@@ -197,10 +202,24 @@ static void test_reftable_log_record_roundtrip(void)
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 22,
 			.value_type = REFTABLE_LOG_DELETION,
+		},
+		{
+			.refname = xstrdup("branch"),
+			.update_index = 33,
+			.value_type = REFTABLE_LOG_UPDATE,
+			.value = {
+				.update = {
+					.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+					.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+					/* rest of fields left empty. */
+				},
+			},
 		}
 	};
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
+	set_test_hash(in[2].value.update.new_hash, 3);
+	set_test_hash(in[2].value.update.old_hash, 4);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
 		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
-- 
gitgitgadget

Re: [PATCH v6 01/15] reftable: fix OOB stack write in print functions

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index 6a5dac32dc6..8536bd03aa9 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
>  void reftable_ref_record_print(struct reftable_ref_record *ref,
>  			       uint32_t hash_id)
>  {
> -	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
> +	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */

Is whatever "BUG" this is still current after this change?

Re: [PATCH v6 02/15] reftable: fix resource leak in block.c error path

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
> unittest.
>
> This problem was discovered by a Coverity scan.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/block.c          | 26 +++++++++------
>  reftable/reader.c         | 23 ++++++++------
>  reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
>  3 files changed, 97 insertions(+), 18 deletions(-)
>
> diff --git a/reftable/block.c b/reftable/block.c
> index 855e3f5c947..6c8e8705205 100644
> --- a/reftable/block.c
> +++ b/reftable/block.c
> @@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
>  	uint32_t full_block_size = table_block_size;
>  	uint8_t typ = block->data[header_off];
>  	uint32_t sz = get_be24(block->data + header_off + 1);
> -

stray extra whitespace being added in the middle of variable
declarations.

Re: [PATCH v6 04/15] reftable: check reftable_stack_auto_compact() return value

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Fixes a problem detected by Coverity.

Detected by coverity, but I think it's better to describe the
issue. Here it seems to be that we didn't use the "err" at all, so
before this change the test wasn't doing anything to check that we
didn't have an error?

Or would the subsequent EXPECT() catch it in practice?

> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/stack_test.c | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/reftable/stack_test.c b/reftable/stack_test.c
> index f4c743db80c..e84f50d27ff 100644
> --- a/reftable/stack_test.c
> +++ b/reftable/stack_test.c
> @@ -839,6 +839,7 @@ static void test_reftable_stack_auto_compaction(void)
>  		EXPECT_ERR(err);
>  
>  		err = reftable_stack_auto_compact(st);
> +		EXPECT_ERR(err);
>  		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
>  	}

Re: [PATCH v6 05/15] reftable: ignore remove() return value in stack_test.c

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> If the cleanup fails, there is nothing we can do.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/stack_test.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/reftable/stack_test.c b/reftable/stack_test.c
> index e84f50d27ff..19fe4e20085 100644
> --- a/reftable/stack_test.c
> +++ b/reftable/stack_test.c
> @@ -90,7 +90,7 @@ static void test_read_file(void)
>  		EXPECT(0 == strcmp(want[i], names[i]));
>  	}
>  	free_names(names);
> -	remove(fn);
> +	(void) remove(fn);
>  }
>  
>  static void test_parse_names(void)

Well, if we fail here due to a permission error or other I/O weirdness
surely it's better to:

	if (remove(fn) < 0)
		die_errno("unable to remove '%s'", fn);

Otherwise we're just silently sweeping that under the rug, and likely
having the "rm -rf" we'll shortly do in test-lib.sh catch it at a
distance.

Also why are we using remove() here at all? Shouldn't this just be
unlink()? Or per the feedback above unlink_or_warn() or
remove_or_warn()? I.e. looking at the context we just open()'d this
"fn", so we're not unsure if it's a directory or a file, are we?

Re: [PATCH v6 12/15] reftable: implement record equality generically

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> This simplifies unittests a little, and provides further coverage for
> reftable_record_copy().
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c      | 57 +++++++++++++++++++++++++++++++++++++++++-
>  reftable/record.h      |  5 +++-
>  reftable/record_test.c | 23 +++--------------
>  3 files changed, 63 insertions(+), 22 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index f7c77c51539..2a9e41a992e 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -430,6 +430,15 @@ static int reftable_ref_record_is_deletion_void(const void *p)
>  		(const struct reftable_ref_record *)p);
>  }
>  
> +

stray extra newline being added here.

> +static int reftable_ref_record_equal_void(const void *a,
> +					  const void *b, int hash_size)
> +{
> +	struct reftable_ref_record *ra = (struct reftable_ref_record *) a;
> +	struct reftable_ref_record *rb = (struct reftable_ref_record *) b;
> +	return reftable_ref_record_equal(ra, rb, hash_size);
> +}
> +
>  static struct reftable_record_vtable reftable_ref_record_vtable = {
>  	.key = &reftable_ref_record_key,
>  	.type = BLOCK_TYPE_REF,
> @@ -439,6 +448,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
>  	.decode = &reftable_ref_record_decode,
>  	.release = &reftable_ref_record_release_void,
>  	.is_deletion = &reftable_ref_record_is_deletion_void,
> +	.equal = &reftable_ref_record_equal_void,
>  };
>  
>  static void reftable_obj_record_key(const void *r, struct strbuf *dest)
> @@ -572,6 +582,25 @@ static int not_a_deletion(const void *p)
>  	return 0;
>  }
>  
> +static int reftable_obj_record_equal_void(const void *a, const void *b, int hash_size)
> +{
> +	struct reftable_obj_record *ra = (struct reftable_obj_record *) a;
> +	struct reftable_obj_record *rb = (struct reftable_obj_record *) b;
> +
> +	if (ra->hash_prefix_len != rb->hash_prefix_len
> +	    || ra->offset_len != rb->offset_len)
> +		return 0;
> +
> +	if (ra->hash_prefix_len &&
> +	    memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
> +		return 0;

Similar to the memcpy() paranoia isn't this memcmp() paranoia?
I.e. memcmp() returns 0 on a n=0, so we can lose the
"ra->hash_prefix_len &&" here, no?

> +	if (ra->offset_len &&
> +	    memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
> +		return 0;

...and here, since 0 * sizeof() will just get us zero.

Re: [PATCH v6 13/15] reftable: remove outdated file reftable.c

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> This was renamed to generic.c, but the origin was never removed

I thought "hrm, didn't I have a way to assert this?" before I remembered
that yes, we could use $(wildcard) in the Makefile to avoid this class
of problem in the future:
https://lore.kernel.org/git/220121.86o845jnvv.gmgdl@evledraar.gmail.com/

Just food for thought :) The change itself LGTM.

Re: [PATCH v6 14/15] reftable: make reftable_record a tagged union

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> This reduces the amount of glue code, because we don't need a void
> pointer or vtable within the structure.
> [...]
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_ref(&rec, &ref);
> +	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
> +				       .u.ref = {
> +					       .refname = (char *)name,
> +				       } };
>  	return tab->ops->seek_record(tab->table_arg, it, &rec);
>  }

Both for this & the below don't we prefer to have such assignments on
their own line? I.e.:
	
	struct reftable_record rec = {
		.type = BLOCK_TYPE_REF,
	        .u.ref = { .refname = (char *)name },
	};

In this case the snuggling is buying is nothing in terms of line count,
and for the .u.ref assignment we were split across 3 lines instead of
1. So this seems easier to read to me....
	
>  int reftable_table_seek_log(struct reftable_table *tab,
>  			    struct reftable_iterator *it, const char *name)
>  {
> -	struct reftable_log_record log = {
> -		.refname = (char *)name,
> -		.update_index = ~((uint64_t)0),
> -	};
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_log(&rec, &log);
> +	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
> +				       .u.log = {
> +					       .refname = (char *)name,
> +					       .update_index = ~((uint64_t)0),
> +				       } };

...ditto..

>  	return tab->ops->seek_record(tab->table_arg, it, &rec);
>  }
>  
> @@ -129,17 +128,25 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
>  int reftable_iterator_next_ref(struct reftable_iterator *it,
>  			       struct reftable_ref_record *ref)
>  {
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_ref(&rec, ref);
> -	return iterator_next(it, &rec);
> +	struct reftable_record rec = {
> +		.type = BLOCK_TYPE_REF,
> +		.u.ref = *ref,
> +	};

...and here we have the sort of assignment I'm suggesting, not

	struct reftable_record rec = { .type = [...]

...

>  int reftable_iterator_next_log(struct reftable_iterator *it,
>  			       struct reftable_log_record *log)
>  {
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_log(&rec, log);
> -	return iterator_next(it, &rec);
> +	struct reftable_record rec = {
> +		.type = BLOCK_TYPE_LOG,
> +		.u.log = *log,
> +	};

...ditto (good)....

> -	struct reftable_ref_record ref = {
> -		.refname = (char *)name,
> +	struct reftable_record rec = {
> +		.type = BLOCK_TYPE_REF,
> +		.u.ref = {
> +			.refname = (char *)name,
> +		},

... yet another style that's a mix between the two. I suppose always
putting the braces on their own line is consistent, but for these
one-member assignments just doing:

	struct reftable_record rec = {
		.type = BLOCK_TYPE_REF,
		.u.ref = { .refname = (char *)name },
  	};

seems easier to read....

> +	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
> +				       .u.log = {
> +					       .refname = (char *)name,
> +					       .update_index = update_index,
> +				       } };

...ditto (exhaustively wrapped)

>  	return merged_table_seek_record(mt, it, &rec);
>  }
>  
> diff --git a/reftable/pq.c b/reftable/pq.c
> index efc474017a2..96ca6dd37b3 100644
> --- a/reftable/pq.c
> +++ b/reftable/pq.c
> @@ -74,6 +74,7 @@ struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
>  void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
>  {
>  	int i = 0;
> +
>  	if (pq->len == pq->cap) {

A stray whitespace change. A good one, but maybe (depending on others in
the topic) bundle those into their own patch (or maybe it's not worth
it, haven't looked at the whole thing again).

In this case we don't otherwise touch this function, so maybe just
leaving it for some future re-formatting change is better?

> -		last = ref->refname;
> -		ref->refname = NULL;
> -		reftable_free(ref);
> +		// this is names[i], so don't dealloc.
> +		last = rec->u.ref.refname;
> +		rec->u.ref.refname = NULL;
> +		reftable_record_release(rec);

Per https://lore.kernel.org/git/87wnmwpwyf.fsf@evledraar.gmail.com/ I
think Junio at al prefer not to have C++-style comments in git, even
though they're also in C99.

>  	}
> -

...more stray whitespace...

> [...]
> -	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
> -	struct reftable_record want_index_rec = { NULL };
> -	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
> -	struct reftable_record index_result_rec = { NULL };
> +	struct reftable_record want_index = {
> +		.type = BLOCK_TYPE_INDEX, .u.idx = { .last_key = STRBUF_INIT }
> +	};
> +	struct reftable_record index_result = {
> +		.type = BLOCK_TYPE_INDEX,
> +		.u.idx = { .last_key = STRBUF_INIT },
> +	};

Here you are formatting/indenting these struct assignments exactly in
the style I suggested above (but within just this patch I think it's the
3rd or 4th style you're using picking).

>  	struct table_iter index_iter = TABLE_ITER_INIT;
>  	struct table_iter next = TABLE_ITER_INIT;
>  	int err = 0;
>  
> -	reftable_record_key(rec, &want_index.last_key);
> -	reftable_record_from_index(&want_index_rec, &want_index);
> -	reftable_record_from_index(&index_result_rec, &index_result);
> -
> +	reftable_record_key(rec, &want_index.u.idx.last_key);

Since you're adjusting whitespace that splits blocks for readability,
doesn't it mke sense to retain the reftable_record_key() as a "block"
here before reader_start(), even if there's only one of them left?

I.e. to split up data setup from "real work" in the function.

>  {
> -	struct reftable_ref_record ref = {
> -		.refname = (char *)name,
> +	struct reftable_record rec = {
> +		.type = BLOCK_TYPE_REF,
> +		.u.ref = {
> +			.refname = (char *)name,
> +		},
>  	};
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_ref(&rec, &ref);
>  	return reader_seek(r, it, &rec);
>  }

...more struct formatting...

> @@ -607,12 +608,11 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
>  				struct reftable_iterator *it, const char *name,
>  				uint64_t update_index)
>  {
> -	struct reftable_log_record log = {
> -		.refname = (char *)name,
> -		.update_index = update_index,
> -	};
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_log(&rec, &log);
> +	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
> +				       .u.log = {
> +					       .refname = (char *)name,
> +					       .update_index = update_index,
> +				       } };

..ditto...

>  	return reader_seek(r, it, &rec);
>  }
>  
> @@ -656,31 +656,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
>  					    struct reftable_iterator *it,
>  					    uint8_t *oid)
>  {
> -	struct reftable_obj_record want = {
> -		.hash_prefix = oid,
> -		.hash_prefix_len = r->object_id_len,
> +	struct reftable_record want = {
> +		.type = BLOCK_TYPE_OBJ,
> +		.u.obj = {
> +			.hash_prefix = oid,
> +			.hash_prefix_len = r->object_id_len,
> +		},
>  	};

..ditto...

> -	struct reftable_record want_rec = { NULL };
>  	struct reftable_iterator oit = { NULL };
> -	struct reftable_obj_record got = { NULL };
> -	struct reftable_record got_rec = { NULL };
> +	struct reftable_record got = {
> +		.type = BLOCK_TYPE_OBJ,
> +		.u.obj = { 0 },
> +	};
>  	int err = 0;
>  	struct indexed_table_ref_iter *itr = NULL;
>  
>  	/* Look through the reverse index. */
> -	reftable_record_from_obj(&want_rec, &want);
> -	err = reader_seek(r, &oit, &want_rec);
> +	err = reader_seek(r, &oit, &want);
>  	if (err != 0)
>  		goto done;
>  
>  	/* read out the reftable_obj_record */
> -	reftable_record_from_obj(&got_rec, &got);
> -	err = iterator_next(&oit, &got_rec);
> +	err = iterator_next(&oit, &got);
>  	if (err < 0)
>  		goto done;
>  
> -	if (err > 0 ||
> -	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
> +	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
> +			      r->object_id_len)) {
>  		/* didn't find it; return empty iterator */
>  		iterator_set_empty(it);
>  		err = 0;
> @@ -688,15 +690,16 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
>  	}
>  
>  	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
> -					 got.offsets, got.offset_len);
> +					 got.u.obj.offsets,
> +					 got.u.obj.offset_len);
>  	if (err < 0)
>  		goto done;
> -	got.offsets = NULL;
> +	got.u.obj.offsets = NULL;
>  	iterator_from_indexed_table_ref_iter(it, itr);
>  
>  done:
>  	reftable_iterator_destroy(&oit);
> -	reftable_record_release(&got_rec);
> +	reftable_record_release(&got);
>  	return err;
>  }
>  
> diff --git a/reftable/record.c b/reftable/record.c
> index 2a9e41a992e..a8cee628942 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -15,6 +15,10 @@ https://developers.google.com/open-source/licenses/bsd
>  #include "reftable-error.h"
>  #include "basics.h"
>  
> +static struct reftable_record_vtable *
> +reftable_record_vtable(struct reftable_record *rec);
> +static void *reftable_record_data(struct reftable_record *rec);

The reftable_record_data() function can just be moved above its first
user, and we can avoid this forward deceleration for a new static
function.

> +	if (src->hash_prefix_len)
> +		memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);

Earlier I commented on other mem*() usage, but this goes against the
memcpy() paranoia being dropped referenced in the CL. I.e. we can do
away with the guard clause here too.

> +static void *reftable_record_data(struct reftable_record *rec)
> +{
> +	switch (rec->type) {
> +	case BLOCK_TYPE_REF:
> +		return &rec->u.ref;
> +	case BLOCK_TYPE_LOG:
> +		return &rec->u.log;
> +	case BLOCK_TYPE_INDEX:
> +		return &rec->u.idx;
> +	case BLOCK_TYPE_OBJ:
> +		return &rec->u.obj;
> +	}
> +	abort();
> +}

The static function we forward-declared above, and could just move up.

> +struct reftable_record reftable_new_record(uint8_t typ)
> +{

nit: There's no "type" variable in the below function, so just call this "type"?...

> +	struct reftable_record clean = {
> +		.type = typ,

...only this "type" member, which won't confuse a compiler.

> +	switch (typ) {

...but actually...


> +	case BLOCK_TYPE_OBJ:
> +	{
> +		struct reftable_obj_record obj = { 0 };
> +		clean.u.obj = obj;
> +		break;
> +	}
> +	case BLOCK_TYPE_INDEX:
> +	{
> +		struct reftable_index_record idx = {
> +			.last_key = STRBUF_INIT,
> +		};
> +		clean.u.idx = idx;
> +		break;
> +	}
> +	case BLOCK_TYPE_REF:
> +	{
> +		struct reftable_ref_record ref = { 0 };
> +		clean.u.ref = ref;
> +		break;
> +	}
> +	case BLOCK_TYPE_LOG:
> +	{
> +		struct reftable_log_record log = { 0 };
> +		clean.u.log = log;
> +		break;
> +	}
> +	}
> +	return clean;
> +}

...reading this whole function shouldn't this be a more normal *_init()
pattern function where the caller just populates the ".type = ", and we
init the rest here? That would also make the ownership more obvious, and
if any future API user needs to pass in variable on the heap instead of
us returning it on the stack here...

> -/* zeroes out the embedded record */
> +/* frees and zeroes out the embedded record */
>  void reftable_record_release(struct reftable_record *rec);

I didn't follow all the vtable entries, but for these:
    
    4 matches for ".release =" in buffer: record.c     
        440:        .release = &reftable_ref_record_release_void,
        582:        .release = &reftable_obj_record_release,
        925:        .release = &reftable_log_record_release_void,
       1052:        .release = &reftable_index_record_release,

Some zero'd the data out already, but for
"reftable_index_record_release" isn't promising this a bug, as we don't
want to memset() to 0 a strbuf_init()?

>  static void test_copy(struct reftable_record *rec)
>  {
> -	struct reftable_record copy =
> -		reftable_new_record(reftable_record_type(rec));
> +	struct reftable_record copy = { 0 };
> +	uint8_t typ;
> +
> +	typ = reftable_record_type(rec);

More odd s/typ/type/

Also for the whitespace here we can just add this assignment above where
we declare it.

> [...]
>  		};
> -
>  		int n, m;

Good whitespace change, but another thing to split up in unrelated
whitespace fixes...

> [...]
>  	struct strbuf key = STRBUF_INIT;
> -	struct reftable_record rec = { NULL };
> -	struct reftable_index_record out = { .last_key = STRBUF_INIT };
> -	struct reftable_record out_rec = { NULL };
> +	struct reftable_record out = {
> +		.type = BLOCK_TYPE_INDEX,
> +		.u.idx = { .last_key = STRBUF_INIT },
> +	};

Another case of different struct formatting (but the one I suggested using uniformly).

>  		int err = writer_finish_public_section(w);
> @@ -316,8 +320,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
>  
>  	w->next -= w->pending_padding;
>  	w->pending_padding = 0;
> -

stray whitespace change...


>  	if (arg->err == 0)
>  		goto done;
> -	obj_rec.offset_len = 0;
> +

..stray whitespace change...

> +	rec.u.obj.offset_len = 0;
>  	arg->err = block_writer_add(arg->w->block_writer, &rec);
>  
>  	/* Should be able to write into a fresh block. */

Re: [PATCH v6 15/15] reftable: add print functions to the record types

[Ævar Arnfjörð Bjarmason]

On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
> [...]
> +void reftable_ref_record_print(const struct reftable_ref_record *ref,
> +			       uint32_t hash_id) {

...here...

> +	reftable_ref_record_print_sz(ref, hash_size(hash_id));
> +}
> +
>  static void reftable_ref_record_release_void(void *rec)
>  {
>  	reftable_ref_record_release(rec);
> @@ -443,6 +448,12 @@ static int reftable_ref_record_equal_void(const void *a,
>  	return reftable_ref_record_equal(ra, rb, hash_size);
>  }
>  
> +static void reftable_ref_record_print_void(const void *rec,
> +					   int hash_size)

...and here you avoid a line longer than 79 characters....

> +{
> +	reftable_ref_record_print_sz((struct reftable_ref_record *) rec, hash_size);

...but not here...

> +		strbuf_addf(&offset_str, "%" PRIu64 " ", obj->offsets[i]);

"%"PRIu64 not "%" PRIu64, per the usual coding style....

> -		printf("log{%s(%" PRIu64 ") delete", log->refname,
> +		printf("log{%s(%" PRIu64 ") delete\n", log->refname,

...although I see we have some existing code here in reftable/ using the other style..

> +static void reftable_index_record_print(const void *rec, int hash_size)
> +{
> +	const struct reftable_index_record *idx = rec;
> +	/* TODO: escape null chars? */

That's quite scary as a TODO comment, can we think about whether we're
doing the wrong thing here? I.e. the printf() will stop on a \0, or
maybe I'm misunderstanding this...

>  	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
>  	int (*equal)(const void *a, const void *b, int hash_size);
> +
> +	/* Print on stdout, for debugging. */

Inaccurate comment, this isn't for ad-hoc debugging, we're using this in
the reftable tests....

> +	void (*print)(const void *rec, int hash_size);
>  };
>  
>  /* returns true for recognized block types. Block start with the block type. */
> @@ -112,6 +115,7 @@ struct reftable_record {
>  
>  /* see struct record_vtable */
>  int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
> +void reftable_record_print(struct reftable_record *rec, int hash_size);
>  void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
>  uint8_t reftable_record_type(struct reftable_record *rec);
>  void reftable_record_copy_from(struct reftable_record *rec,
> diff --git a/reftable/record_test.c b/reftable/record_test.c
> index c6fdd1925a9..f91ea5e8830 100644
> --- a/reftable/record_test.c
> +++ b/reftable/record_test.c
> @@ -25,6 +25,10 @@ static void test_copy(struct reftable_record *rec)
>  	/* do it twice to catch memory leaks */
>  	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
>  	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
> +
> +	puts("testing print coverage:\n");
> +	reftable_record_print(&copy, GIT_SHA1_RAWSZ);

...i.e. here.

> +
>  	reftable_record_release(&copy);
>  }
>  
> @@ -176,7 +180,8 @@ static void test_reftable_log_record_equal(void)
>  static void test_reftable_log_record_roundtrip(void)
>  {
>  	int i;
> -	struct reftable_log_record in[2] = {
> +

...more stray whitespace...

> +	struct reftable_log_record in[] = {
>  		{
>  			.refname = xstrdup("refs/heads/master"),
>  			.update_index = 42,
> @@ -197,10 +202,24 @@ static void test_reftable_log_record_roundtrip(void)
>  			.refname = xstrdup("refs/heads/master"),
>  			.update_index = 22,
>  			.value_type = REFTABLE_LOG_DELETION,
> +		},
> +		{
> +			.refname = xstrdup("branch"),
> +			.update_index = 33,
> +			.value_type = REFTABLE_LOG_UPDATE,
> +			.value = {
> +				.update = {
> +					.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
> +					.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
> +					/* rest of fields left empty. */

But in test_reftable_log_record_roundtrip() we have another existing
user who leaves out fields, but doesn't have such a comment.

I think it's better to just leave it out here, clearly it's intentional,
but if it's going to be added to anything surely the other user defining
5/7 fields is better than this user of 2/7 fields (i.e. the 5/7 is more
likely to be an unintentional omission), this one is clearly
intentional.

Re: [PATCH v6 02/15] reftable: fix resource leak in block.c error path

[Junio C Hamano]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> On Thu, Jan 20 2022, Han-Wen Nienhuys via GitGitGadget wrote:
>
>> From: Han-Wen Nienhuys <hanwen@google.com>
>>
>> Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
>> unittest.
>>
>> This problem was discovered by a Coverity scan.
>>
>> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
>> ---
>>  reftable/block.c          | 26 +++++++++------
>>  reftable/reader.c         | 23 ++++++++------
>>  reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 97 insertions(+), 18 deletions(-)
>>
>> diff --git a/reftable/block.c b/reftable/block.c
>> index 855e3f5c947..6c8e8705205 100644
>> --- a/reftable/block.c
>> +++ b/reftable/block.c
>> @@ -188,13 +188,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
>>  	uint32_t full_block_size = table_block_size;
>>  	uint8_t typ = block->data[header_off];
>>  	uint32_t sz = get_be24(block->data + header_off + 1);
>> -
>
> stray extra whitespace being added in the middle of variable
> declarations.

Hmph.  Isn't it removing a blank line?

Re: [PATCH v6 01/15] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys]

On Fri, Jan 21, 2022 at 12:42 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> > -     char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
> > +     char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
>
> Is whatever "BUG" this is still current after this change?

I can't remember what this was about. Dropping.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v6 14/15] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Fri, Jan 21, 2022 at 1:32 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> > From: Han-Wen Nienhuys <hanwen@google.com>
> >
> > This reduces the amount of glue code, because we don't need a void
> > pointer or vtable within the structure.
> > [...]
> > -     struct reftable_record rec = { NULL };
> > -     reftable_record_from_ref(&rec, &ref);
> > +     struct reftable_record rec = { .type = BLOCK_TYPE_REF,
> > +                                    .u.ref = {
> > +                                            .refname = (char *)name,
> > +                                    } };
> >       return tab->ops->seek_record(tab->table_arg, it, &rec);
> >  }
>
> Both for this & the below don't we prefer to have such assignments on
> their own line? I.e.:

I generally let clang-format do formatting for me, but it looks like
it doesn't want to make a decision about where to put line breaks
here.

I've made the style consistent.
> > +     struct reftable_record clean = {
> > +             .type = typ,
>
> ...only this "type" member, which won't confuse a compiler.
>
> > +     switch (typ) {
>
> ...but actually...

typ vs type is a remnant of the Go origins (type is a reserved word in Go).

I fixed up a couple of places where I add it newly in the commit, but
it makes more sense to change this throughout in a separate commit.

> > +     return clean;
> > +}
>
> ...reading this whole function shouldn't this be a more normal *_init()
> pattern function where the caller just populates the ".type = ", and we
> init the rest here? That would also make the ownership more obvious, and
> if any future API user needs to pass in variable on the heap instead of
> us returning it on the stack here...

this function exists so you can initialize as part of a list of
declarations , eg.

 struct reftable_record rec = reftable_new_record(block_reader_type(br));
 struct strbuf key = STRBUF_INIT;
 int err = 0;
 struct block_iter next = {
  .last_key = STRBUF_INIT,
 };
 int i = binsearch(br->restart_count, &restart_key_less, &args);

with init functions, you have reorganize all of these blocks.


> > -/* zeroes out the embedded record */
> > +/* frees and zeroes out the embedded record */
> >  void reftable_record_release(struct reftable_record *rec);
>
> I didn't follow all the vtable entries, but for these:
>
>     4 matches for ".release =" in buffer: record.c
>         440:        .release = &reftable_ref_record_release_void,
>         582:        .release = &reftable_obj_record_release,
>         925:        .release = &reftable_log_record_release_void,
>        1052:        .release = &reftable_index_record_release,
>
> Some zero'd the data out already, but for
> "reftable_index_record_release" isn't promising this a bug, as we don't
> want to memset() to 0 a strbuf_init()?

Isn't strbuf_release() -which is used- the correct way to clear out a strbuf?

>>       }
>> -
>
>...more stray whitespace...

If you trim this much context, can you provide a line number?

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--

Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich

Registergericht und -nummer: Hamburg, HRB 86891

Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v6 15/15] reftable: add print functions to the record types

[Han-Wen Nienhuys]

On Fri, Jan 21, 2022 at 1:40 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
> > +static void reftable_index_record_print(const void *rec, int hash_size)
> > +{
> > +     const struct reftable_index_record *idx = rec;
> > +     /* TODO: escape null chars? */
>
> That's quite scary as a TODO comment, can we think about whether we're
> doing the wrong thing here? I.e. the printf() will stop on a \0, or
> maybe I'm misunderstanding this...

It's a print function, so we'd print a truncated last_key. This can
happen if we were printing an index record that points to a log
record. I'd address this if and when I have to debug into indexing
going wrong (which hasn't happened so far.)


> >       /* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
> >       int (*equal)(const void *a, const void *b, int hash_size);
> > +
> > +     /* Print on stdout, for debugging. */
>
> Inaccurate comment, this isn't for ad-hoc debugging, we're using this in
> the reftable tests....

The tests say

> > +     puts("testing print coverage:\n");
> > +     reftable_record_print(&copy, GIT_SHA1_RAWSZ);

ie. I'm calling the print functions to make sure they don't crash or
leak memory. They're not used as part of functionality of either the
reftable code or its tests.

> > +                                     .old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
> > +                                     .new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
> > +                                     /* rest of fields left empty. */
>
> But in test_reftable_log_record_roundtrip() we have another existing
> user who leaves out fields, but doesn't have such a comment.

rephrased comment.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

[PATCH v7 00/16] Reftable coverity fixes

[Han-Wen Nienhuys via GitGitGadget]

This series was targeted to 'master'.

This addresses some complaints from Coverity that Peff reported.

v7:

 * address avarab's comments.

Han-Wen Nienhuys (16):
  reftable: fix OOB stack write in print functions
  reftable: fix resource leak in block.c error path
  reftable: fix resource leak blocksource.c
  reftable: check reftable_stack_auto_compact() return value
  reftable: ignore remove() return value in stack_test.c
  reftable: fix resource warning
  reftable: all xxx_free() functions accept NULL arguments
  reftable: order unittests by complexity
  reftable: drop stray printf in readwrite_test
  reftable: handle null refnames in reftable_ref_record_equal
  reftable: make reftable-record.h function signatures const correct
  reftable: implement record equality generically
  reftable: remove outdated file reftable.c
  reftable: make reftable_record a tagged union
  reftable: add print functions to the record types
  reftable: rename typ to type

 reftable/block.c           |  35 ++--
 reftable/block_test.c      |  22 +--
 reftable/blocksource.c     |   6 +-
 reftable/generic.c         |  39 ++--
 reftable/iter.c            |   4 +-
 reftable/merged.c          |  35 ++--
 reftable/merged.h          |   2 +-
 reftable/pq.c              |   2 +-
 reftable/pq_test.c         |  29 ++-
 reftable/reader.c          | 167 +++++++++--------
 reftable/readwrite_test.c  |  67 ++++++-
 reftable/record.c          | 367 +++++++++++++++++++++++--------------
 reftable/record.h          |  49 ++---
 reftable/record_test.c     | 193 +++++++++----------
 reftable/reftable-record.h |  14 +-
 reftable/reftable.c        | 115 ------------
 reftable/stack.c           |  10 +-
 reftable/stack_test.c      |   3 +-
 reftable/writer.c          |  72 ++++----
 t/helper/test-reftable.c   |   9 +-
 20 files changed, 666 insertions(+), 574 deletions(-)
 delete mode 100644 reftable/reftable.c


base-commit: 50b2d72e110cad39ecaf2322bfdf1b60cd13dd96
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1152%2Fhanwen%2Freftable-coverity-v7
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1152/hanwen/reftable-coverity-v7
Pull-Request: https://github.com/git/git/pull/1152

Range-diff vs v6:

  1:  9218bd59b25 !  1:  cc47ec7a50b reftable: fix OOB stack write in print functions
     @@ Metadata
       ## Commit message ##
          reftable: fix OOB stack write in print functions
      
     +    Remove /* BUG */ comment, whose meaning has escaped my memory.
     +
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
       ## reftable/record.c ##
     @@ reftable/record.c: static void hex_format(char *dest, uint8_t *src, int hash_siz
       			       uint32_t hash_id)
       {
      -	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
     -+	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
     ++	char hex[GIT_MAX_HEXSZ + 1];
       	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
       	switch (ref->value_type) {
       	case REFTABLE_REF_SYMREF:
  2:  315ce62e710 !  2:  1c5441d2f7e reftable: fix resource leak in block.c error path
     @@ Commit message
      
       ## reftable/block.c ##
      @@ reftable/block.c: int block_reader_init(struct block_reader *br, struct reftable_block *block,
     - 	uint32_t full_block_size = table_block_size;
       	uint8_t typ = block->data[header_off];
       	uint32_t sz = get_be24(block->data + header_off + 1);
     --
     + 
      +	int err = 0;
       	uint16_t restart_count = 0;
       	uint32_t restart_start = 0;
  3:  7cf65e141cf =  3:  f11a9964383 reftable: fix resource leak blocksource.c
  4:  8740ba92981 !  4:  b0bffb6e29d reftable: check reftable_stack_auto_compact() return value
     @@ Metadata
       ## Commit message ##
          reftable: check reftable_stack_auto_compact() return value
      
     -    Fixes a problem detected by Coverity.
     +    If compaction consistently errors out, we would accumulate reftables,
     +    and the follow-on EXPECT would start failing, but catching errors
     +    early helps debugging.
     +
     +    Spotted by Coverity.
      
          Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
      
  5:  08be6d90a48 !  5:  e3d575ae2b4 reftable: ignore remove() return value in stack_test.c
     @@ reftable/stack_test.c: static void test_read_file(void)
       	}
       	free_names(names);
      -	remove(fn);
     -+	(void) remove(fn);
     ++	EXPECT(unlink(fn) == 0);
       }
       
       static void test_parse_names(void)
  6:  630f67bdb51 =  6:  a21d934f37d reftable: fix resource warning
  7:  afee67fb60d =  7:  b40d91c50a2 reftable: all xxx_free() functions accept NULL arguments
  8:  6f4f24c17ad =  8:  7a1f32a2ff0 reftable: order unittests by complexity
  9:  0abaab1d7c4 =  9:  a118cd40975 reftable: drop stray printf in readwrite_test
 10:  004b7a736bf = 10:  1d5cfbbc47a reftable: handle null refnames in reftable_ref_record_equal
 11:  951efcebcd2 ! 11:  90c72a2df0c reftable: make reftable-record.h function signatures const correct
     @@ reftable/record.c: static void hex_format(char *dest, uint8_t *src, int hash_siz
      +void reftable_ref_record_print(const struct reftable_ref_record *ref,
       			       uint32_t hash_id)
       {
     - 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
     + 	char hex[GIT_MAX_HEXSZ + 1];
      @@ reftable/record.c: static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
       	return !memcmp(a, b, sz);
       }
 12:  6385e449ba7 ! 12:  a83a9cf59b7 reftable: implement record equality generically
     @@ reftable/record.c: static int reftable_ref_record_is_deletion_void(const void *p
       		(const struct reftable_ref_record *)p);
       }
       
     -+
      +static int reftable_ref_record_equal_void(const void *a,
      +					  const void *b, int hash_size)
      +{
     @@ reftable/record.c: static int not_a_deletion(const void *p)
      +	    || ra->offset_len != rb->offset_len)
      +		return 0;
      +
     -+	if (ra->hash_prefix_len &&
     -+	    memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
     ++	if (memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
      +		return 0;
     -+	if (ra->offset_len &&
     -+	    memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
     ++	if (memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
      +		return 0;
      +
      +	return 1;
 13:  69c4a554882 = 13:  7e23d14d1af reftable: remove outdated file reftable.c
 14:  54bcac325bd ! 14:  047bbb75fcb reftable: make reftable_record a tagged union
     @@ reftable/generic.c: https://developers.google.com/open-source/licenses/bsd
       {
      -	struct reftable_ref_record ref = {
      -		.refname = (char *)name,
     --	};
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_REF,
     ++		.u.ref = { .refname = (char *)name, }
     + 	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_ref(&rec, &ref);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_REF,
     -+				       .u.ref = {
     -+					       .refname = (char *)name,
     -+				       } };
       	return tab->ops->seek_record(tab->table_arg, it, &rec);
       }
       
     @@ reftable/generic.c: https://developers.google.com/open-source/licenses/bsd
      -	struct reftable_log_record log = {
      -		.refname = (char *)name,
      -		.update_index = ~((uint64_t)0),
     --	};
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_LOG,
     ++		.u.log = {
     ++			.refname = (char *)name,
     ++			.update_index = ~((uint64_t)0),
     ++		}
     + 	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     -+				       .u.log = {
     -+					       .refname = (char *)name,
     -+					       .update_index = ~((uint64_t)0),
     -+				       } };
       	return tab->ops->seek_record(tab->table_arg, it, &rec);
       }
       
     @@ reftable/merged.c: int reftable_merged_table_seek_ref(struct reftable_merged_tab
      -		.refname = (char *)name,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_REF,
     -+		.u.ref = {
     -+			.refname = (char *)name,
     -+		},
     ++		.u.ref = { .refname = (char *)name, },
       	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_ref(&rec, &ref);
     @@ reftable/merged.c: int reftable_merged_table_seek_log_at(struct reftable_merged_
      -	struct reftable_log_record log = {
      -		.refname = (char *)name,
      -		.update_index = update_index,
     --	};
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_LOG,
     ++		.u.log = {
     ++			.refname = (char *)name,
     ++			.update_index = update_index,
     ++		}
     + 	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     -+				       .u.log = {
     -+					       .refname = (char *)name,
     -+					       .update_index = update_index,
     -+				       } };
       	return merged_table_seek_record(mt, it, &rec);
       }
       
      
       ## reftable/pq.c ##
     -@@ reftable/pq.c: struct pq_entry merged_iter_pqueue_remove(struct merged_iter_pqueue *pq)
     - void merged_iter_pqueue_add(struct merged_iter_pqueue *pq, struct pq_entry e)
     - {
     - 	int i = 0;
     -+
     - 	if (pq->len == pq->cap) {
     - 		pq->cap = 2 * pq->cap + 1;
     - 		pq->heap = reftable_realloc(pq->heap,
      @@ reftable/pq.c: void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
       {
       	int i = 0;
     @@ reftable/pq_test.c: static void test_pq(void)
      -
      -		reftable_record_as_ref(&rec)->refname = names[i];
      -		e.rec = rec;
     -+		struct pq_entry e = { .rec = { .type = BLOCK_TYPE_REF,
     -+					       .u.ref = {
     -+						       .refname = names[i],
     -+					       } } };
     ++		struct pq_entry e = {
     ++			.rec = {
     ++				.type = BLOCK_TYPE_REF,
     ++				.u.ref = { .refname = names[i], }
     ++			}
     ++		};
       		merged_iter_pqueue_add(&pq, e);
       		merged_iter_pqueue_check(pq);
       		i = (i * 7) % N;
     @@ reftable/pq_test.c: static void test_pq(void)
      -		last = ref->refname;
      -		ref->refname = NULL;
      -		reftable_free(ref);
     -+		// this is names[i], so don't dealloc.
     ++		/* this is names[i], so don't dealloc. */
      +		last = rec->u.ref.refname;
      +		rec->u.ref.refname = NULL;
      +		reftable_record_release(rec);
     @@ reftable/reader.c: static int reader_seek_indexed(struct reftable_reader *r,
      -	reftable_record_key(rec, &want_index.last_key);
      -	reftable_record_from_index(&want_index_rec, &want_index);
      -	reftable_record_from_index(&index_result_rec, &index_result);
     --
      +	reftable_record_key(rec, &want_index.u.idx.last_key);
     + 
       	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
       	if (err < 0)
       		goto done;
     @@ reftable/reader.c: static int reader_seek(struct reftable_reader *r, struct reft
      -		.refname = (char *)name,
      +	struct reftable_record rec = {
      +		.type = BLOCK_TYPE_REF,
     -+		.u.ref = {
     -+			.refname = (char *)name,
     -+		},
     ++		.u.ref = { .refname = (char *)name, },
       	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_ref(&rec, &ref);
     @@ reftable/reader.c: int reftable_reader_seek_log_at(struct reftable_reader *r,
      -	struct reftable_log_record log = {
      -		.refname = (char *)name,
      -		.update_index = update_index,
     --	};
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_LOG,
     ++		.u.log = {
     ++			.refname = (char *)name,
     ++			.update_index = update_index,
     ++		}
     + 	};
      -	struct reftable_record rec = { NULL };
      -	reftable_record_from_log(&rec, &log);
     -+	struct reftable_record rec = { .type = BLOCK_TYPE_LOG,
     -+				       .u.log = {
     -+					       .refname = (char *)name,
     -+					       .update_index = update_index,
     -+				       } };
       	return reader_seek(r, it, &rec);
       }
       
     @@ reftable/record.c: https://developers.google.com/open-source/licenses/bsd
       
      +static struct reftable_record_vtable *
      +reftable_record_vtable(struct reftable_record *rec);
     -+static void *reftable_record_data(struct reftable_record *rec);
     ++
     ++static void *reftable_record_data(struct reftable_record *rec)
     ++{
     ++	switch (rec->type) {
     ++	case BLOCK_TYPE_REF:
     ++		return &rec->u.ref;
     ++	case BLOCK_TYPE_LOG:
     ++		return &rec->u.log;
     ++	case BLOCK_TYPE_INDEX:
     ++		return &rec->u.idx;
     ++	case BLOCK_TYPE_OBJ:
     ++		return &rec->u.obj;
     ++	}
     ++	abort();
     ++}
      +
       int get_var_int(uint64_t *dest, struct string_view *in)
       {
     @@ reftable/record.c: static void reftable_obj_record_copy_from(void *rec, const vo
       	reftable_obj_record_release(obj);
      -	*obj = *src;
      -	obj->hash_prefix = reftable_malloc(obj->hash_prefix_len);
     --	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
      +	obj->hash_prefix = reftable_malloc(src->hash_prefix_len);
      +	obj->hash_prefix_len = src->hash_prefix_len;
     -+	if (src->hash_prefix_len)
     -+		memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
     + 	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
       
      -	obj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));
      -	COPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);
     @@ reftable/record.c: static struct reftable_record_vtable reftable_log_record_vtab
       static void reftable_index_record_key(const void *r, struct strbuf *dest)
       {
       	const struct reftable_index_record *rec = r;
     +@@ reftable/record.c: static void reftable_index_record_release(void *rec)
     + {
     + 	struct reftable_index_record *idx = rec;
     + 	strbuf_release(&idx->last_key);
     ++	idx->offset = 0;
     + }
     + 
     + static uint8_t reftable_index_record_val_type(const void *rec)
      @@ reftable/record.c: static struct reftable_record_vtable reftable_index_record_vtable = {
       
       void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
       	s->len -= n;
       }
      +
     -+static void *reftable_record_data(struct reftable_record *rec)
     -+{
     -+	switch (rec->type) {
     -+	case BLOCK_TYPE_REF:
     -+		return &rec->u.ref;
     -+	case BLOCK_TYPE_LOG:
     -+		return &rec->u.log;
     -+	case BLOCK_TYPE_INDEX:
     -+		return &rec->u.idx;
     -+	case BLOCK_TYPE_OBJ:
     -+		return &rec->u.obj;
     -+	}
     -+	abort();
     -+}
     -+
      +static struct reftable_record_vtable *
      +reftable_record_vtable(struct reftable_record *rec)
      +{
     @@ reftable/record.c: void string_view_consume(struct string_view *s, int n)
      +	abort();
      +}
      +
     -+struct reftable_record reftable_new_record(uint8_t typ)
     ++struct reftable_record reftable_new_record(uint8_t type)
      +{
      +	struct reftable_record clean = {
     -+		.type = typ,
     ++		.type = type,
      +	};
      +
      +	/* the following is involved, but the naive solution (just return
      +	 * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage
      +	 * clean.u.obj.offsets pointer on Windows VS CI.  Go figure.
      +	 */
     -+	switch (typ) {
     ++	switch (type) {
      +	case BLOCK_TYPE_OBJ:
      +	{
      +		struct reftable_obj_record obj = { 0 };
     @@ reftable/record_test.c
      -	struct reftable_record copy =
      -		reftable_new_record(reftable_record_type(rec));
      +	struct reftable_record copy = { 0 };
     -+	uint8_t typ;
     -+
     -+	typ = reftable_record_type(rec);
     -+	copy = reftable_new_record(typ);
     ++	uint8_t type = reftable_record_type(rec);
     ++	copy = reftable_new_record(type);
       	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
       	/* do it twice to catch memory leaks */
       	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
       		uint8_t buffer[1024] = { 0 };
       		struct string_view dest = {
       			.buf = buffer,
     - 			.len = sizeof(buffer),
     - 		};
     --
     +@@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
     + 
       		int n, m;
       
      -		in.value_type = i;
     @@ reftable/record_test.c: static void test_reftable_ref_record_roundtrip(void)
      -			set_hash(in.value.val2.value, 1);
      -			in.value.val2.target_value =
      +			in.u.ref.value.val2.value =
     - 				reftable_malloc(GIT_SHA1_RAWSZ);
     --			set_hash(in.value.val2.target_value, 2);
     ++				reftable_malloc(GIT_SHA1_RAWSZ);
      +			set_hash(in.u.ref.value.val2.value, 1);
      +			in.u.ref.value.val2.target_value =
     -+				reftable_malloc(GIT_SHA1_RAWSZ);
     + 				reftable_malloc(GIT_SHA1_RAWSZ);
     +-			set_hash(in.value.val2.target_value, 2);
      +			set_hash(in.u.ref.value.val2.target_value, 2);
       			break;
       		case REFTABLE_REF_SYMREF:
     @@ reftable/writer.c: int reftable_writer_add_refs(struct reftable_writer *w,
       	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
       		int err = writer_finish_public_section(w);
      @@ reftable/writer.c: static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
     - 
       	w->next -= w->pending_padding;
       	w->pending_padding = 0;
     --
     + 
      -	reftable_record_from_log(&rec, log);
       	return writer_add_record(w, &rec);
       }
     @@ reftable/writer.c: static void write_object_record(void *void_arg, void *key)
      -		.hash_prefix_len = arg->w->stats.object_id_len,
      -		.offsets = entry->offsets,
      -		.offset_len = entry->offset_len,
     --	};
     ++	struct reftable_record rec = {
     ++		.type = BLOCK_TYPE_OBJ,
     ++		.u.obj = {
     ++			.hash_prefix = (uint8_t *)entry->hash.buf,
     ++			.hash_prefix_len = arg->w->stats.object_id_len,
     ++			.offsets = entry->offsets,
     ++			.offset_len = entry->offset_len,
     ++		}
     + 	};
      -	struct reftable_record rec = { NULL };
     -+	struct reftable_record
     -+		rec = { .type = BLOCK_TYPE_OBJ,
     -+			.u.obj = {
     -+				.hash_prefix = (uint8_t *)entry->hash.buf,
     -+				.hash_prefix_len = arg->w->stats.object_id_len,
     -+				.offsets = entry->offsets,
     -+				.offset_len = entry->offset_len,
     -+			} };
       	if (arg->err < 0)
       		goto done;
       
     @@ reftable/writer.c: static void write_object_record(void *void_arg, void *key)
       	if (arg->err == 0)
       		goto done;
      -	obj_rec.offset_len = 0;
     -+
      +	rec.u.obj.offset_len = 0;
       	arg->err = block_writer_add(arg->w->block_writer, &rec);
       
 15:  82f140cab5c ! 15:  1e8785bbc26 reftable: add print functions to the record types
     @@ reftable/record.c: static void hex_format(char *dest, uint8_t *src, int hash_siz
      +static void reftable_ref_record_print_sz(const struct reftable_ref_record *ref,
      +					 int hash_size)
       {
     - 	char hex[GIT_MAX_HEXSZ + 1] = { 0 }; /* BUG */
     + 	char hex[GIT_MAX_HEXSZ + 1];
       	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
      @@ reftable/record.c: void reftable_ref_record_print(const struct reftable_ref_record *ref,
       		printf("=> %s", ref->value.symref);
     @@ reftable/record.c: static int reftable_ref_record_equal_void(const void *a,
      +static void reftable_ref_record_print_void(const void *rec,
      +					   int hash_size)
      +{
     -+	reftable_ref_record_print_sz((struct reftable_ref_record *) rec, hash_size);
     ++	reftable_ref_record_print_sz(
     ++		(struct reftable_ref_record *) rec, hash_size);
      +}
      +
       static struct reftable_record_vtable reftable_ref_record_vtable = {
     @@ reftable/record.c: static void reftable_obj_record_release(void *rec)
      +	int i;
      +
      +	for (i = 0; i < obj->offset_len; i++)
     -+		strbuf_addf(&offset_str, "%" PRIu64 " ", obj->offsets[i]);
     ++		strbuf_addf(&offset_str, "%"PRIu64 " ", obj->offsets[i]);
      +	hex_format(hex, obj->hash_prefix, obj->hash_prefix_len);
      +	printf("prefix %s (len %d), offsets [%s]\n",
      +	       hex, obj->hash_prefix_len, offset_str.buf);
     @@ reftable/record.c: static struct reftable_record_vtable reftable_obj_record_vtab
       	switch (log->value_type) {
       	case REFTABLE_LOG_DELETION:
      -		printf("log{%s(%" PRIu64 ") delete", log->refname,
     -+		printf("log{%s(%" PRIu64 ") delete\n", log->refname,
     ++		printf("log{%s(%"PRIu64 ") delete\n", log->refname,
       		       log->update_index);
       		break;
       	case REFTABLE_LOG_UPDATE:
     - 		printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
     +-		printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
      -		       log->refname, log->update_index, log->value.update.name,
      -		       log->value.update.email, log->value.update.time,
     ++		printf("log{%s(%"PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
      +		       log->refname, log->update_index,
      +		       log->value.update.name ? log->value.update.name : "",
      +		       log->value.update.email ? log->value.update.email : "",
     @@ reftable/record.c: static struct reftable_record_vtable reftable_index_record_vt
       };
       
       void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
     -@@ reftable/record.c: struct reftable_record reftable_new_record(uint8_t typ)
     +@@ reftable/record.c: struct reftable_record reftable_new_record(uint8_t type)
       	}
       	return clean;
       }
     @@ reftable/record_test.c: static void test_reftable_log_record_equal(void)
       {
       	int i;
      -	struct reftable_log_record in[2] = {
     -+
      +	struct reftable_log_record in[] = {
       		{
       			.refname = xstrdup("refs/heads/master"),
     @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
       			.value_type = REFTABLE_LOG_DELETION,
      +		},
      +		{
     ++			/* make sure we handle optional fields correctly. */
      +			.refname = xstrdup("branch"),
      +			.update_index = 33,
      +			.value_type = REFTABLE_LOG_UPDATE,
     @@ reftable/record_test.c: static void test_reftable_log_record_roundtrip(void)
      +				.update = {
      +					.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
      +					.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
     -+					/* rest of fields left empty. */
      +				},
      +			},
       		}
  -:  ----------- > 16:  556fc11080a reftable: rename typ to type

-- 
gitgitgadget

[PATCH v7 01/16] reftable: fix OOB stack write in print functions

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Remove /* BUG */ comment, whose meaning has escaped my memory.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 6a5dac32dc6..1f236b47d1c 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 void reftable_ref_record_print(struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
-	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
+	char hex[GIT_MAX_HEXSZ + 1];
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
 	switch (ref->value_type) {
 	case REFTABLE_REF_SYMREF:
@@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 void reftable_log_record_print(struct reftable_log_record *log,
 			       uint32_t hash_id)
 {
-	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-- 
gitgitgadget

[PATCH v7 02/16] reftable: fix resource leak in block.c error path

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Add test coverage for corrupt zlib data. Fix memory leaks demonstrated by
unittest.

This problem was discovered by a Coverity scan.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c          | 25 ++++++++++-----
 reftable/reader.c         | 23 ++++++++------
 reftable/readwrite_test.c | 66 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 97 insertions(+), 17 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 855e3f5c947..5832f85501d 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -189,12 +189,16 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint8_t typ = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
 
+	int err = 0;
 	uint16_t restart_count = 0;
 	uint32_t restart_start = 0;
 	uint8_t *restart_bytes = NULL;
+	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ))
-		return REFTABLE_FORMAT_ERROR;
+	if (!reftable_is_block_type(typ)) {
+		err =  REFTABLE_FORMAT_ERROR;
+		goto done;
+	}
 
 	if (typ == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
@@ -203,7 +207,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		uLongf src_len = block->len - block_header_skip;
 		/* Log blocks specify the *uncompressed* size in their header.
 		 */
-		uint8_t *uncompressed = reftable_malloc(sz);
+		uncompressed = reftable_malloc(sz);
 
 		/* Copy over the block header verbatim. It's not compressed. */
 		memcpy(uncompressed, block->data, block_header_skip);
@@ -212,16 +216,19 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		if (Z_OK !=
 		    uncompress2(uncompressed + block_header_skip, &dst_len,
 				block->data + block_header_skip, &src_len)) {
-			reftable_free(uncompressed);
-			return REFTABLE_ZLIB_ERROR;
+			err = REFTABLE_ZLIB_ERROR;
+			goto done;
 		}
 
-		if (dst_len + block_header_skip != sz)
-			return REFTABLE_FORMAT_ERROR;
+		if (dst_len + block_header_skip != sz) {
+			err = REFTABLE_FORMAT_ERROR;
+			goto done;
+		}
 
 		/* We're done with the input data. */
 		reftable_block_done(block);
 		block->data = uncompressed;
+		uncompressed = NULL;
 		block->len = sz;
 		block->source = malloc_block_source();
 		full_block_size = src_len + block_header_skip;
@@ -251,7 +258,9 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	br->restart_count = restart_count;
 	br->restart_bytes = restart_bytes;
 
-	return 0;
+done:
+	reftable_free(uncompressed);
+	return err;
 }
 
 static uint32_t block_reader_restart_offset(struct block_reader *br, int i)
diff --git a/reftable/reader.c b/reftable/reader.c
index 006709a645a..35781593a29 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -290,28 +290,33 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
 
 	err = reader_get_block(r, &block, next_off, guess_block_size);
 	if (err < 0)
-		return err;
+		goto done;
 
 	block_size = extract_block_size(block.data, &block_typ, next_off,
 					r->version);
-	if (block_size < 0)
-		return block_size;
-
+	if (block_size < 0) {
+		err = block_size;
+		goto done;
+	}
 	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
-		reftable_block_done(&block);
-		return 1;
+		err = 1;
+		goto done;
 	}
 
 	if (block_size > guess_block_size) {
 		reftable_block_done(&block);
 		err = reader_get_block(r, &block, next_off, block_size);
 		if (err < 0) {
-			return err;
+			goto done;
 		}
 	}
 
-	return block_reader_init(br, &block, header_off, r->block_size,
-				 hash_size(r->hash_id));
+	err = block_reader_init(br, &block, header_off, r->block_size,
+				hash_size(r->hash_id));
+done:
+	reftable_block_done(&block);
+
+	return err;
 }
 
 static int table_iter_next_block(struct table_iter *dest,
diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index 70c7aedba2c..edb9cfc1930 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -288,6 +288,71 @@ static void test_log_write_read(void)
 	reader_close(&rd);
 }
 
+static void test_log_zlib_corruption(void)
+{
+	struct reftable_write_options opts = {
+		.block_size = 256,
+	};
+	struct reftable_iterator it = { 0 };
+	struct reftable_reader rd = { 0 };
+	struct reftable_block_source source = { 0 };
+	struct strbuf buf = STRBUF_INIT;
+	struct reftable_writer *w =
+		reftable_new_writer(&strbuf_add_void, &buf, &opts);
+	const struct reftable_stats *stats = NULL;
+	uint8_t hash1[GIT_SHA1_RAWSZ] = { 1 };
+	uint8_t hash2[GIT_SHA1_RAWSZ] = { 2 };
+	char message[100] = { 0 };
+	int err, i, n;
+
+	struct reftable_log_record log = {
+		.refname = "refname",
+		.value_type = REFTABLE_LOG_UPDATE,
+		.value = {
+			.update = {
+				.new_hash = hash1,
+				.old_hash = hash2,
+				.name = "My Name",
+				.email = "myname@invalid",
+				.message = message,
+			},
+		},
+	};
+
+	for (i = 0; i < sizeof(message) - 1; i++)
+		message[i] = (uint8_t)(rand() % 64 + ' ');
+
+	reftable_writer_set_limits(w, 1, 1);
+
+	err = reftable_writer_add_log(w, &log);
+	EXPECT_ERR(err);
+
+	n = reftable_writer_close(w);
+	EXPECT(n == 0);
+
+	stats = writer_stats(w);
+	EXPECT(stats->log_stats.blocks > 0);
+	reftable_writer_free(w);
+	w = NULL;
+
+	/* corrupt the data. */
+	buf.buf[50] ^= 0x99;
+
+	block_source_from_strbuf(&source, &buf);
+
+	err = init_reader(&rd, &source, "file.log");
+	EXPECT_ERR(err);
+
+	err = reftable_reader_seek_log(&rd, &it, "refname");
+	EXPECT(err == REFTABLE_ZLIB_ERROR);
+
+	reftable_iterator_destroy(&it);
+
+	/* cleanup. */
+	strbuf_release(&buf);
+	reader_close(&rd);
+}
+
 static void test_table_read_write_sequential(void)
 {
 	char **names;
@@ -667,6 +732,7 @@ static void test_corrupt_table(void)
 
 int readwrite_test_main(int argc, const char *argv[])
 {
+	RUN_TEST(test_log_zlib_corruption);
 	RUN_TEST(test_corrupt_table);
 	RUN_TEST(test_corrupt_table_empty);
 	RUN_TEST(test_log_write_read);
-- 
gitgitgadget

[PATCH v7 03/16] reftable: fix resource leak blocksource.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would be triggered in the unlikely event of fstat() failing on an
opened file.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/blocksource.c | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/reftable/blocksource.c b/reftable/blocksource.c
index 0044eecd9aa..2605371c28d 100644
--- a/reftable/blocksource.c
+++ b/reftable/blocksource.c
@@ -134,8 +134,10 @@ int reftable_block_source_from_file(struct reftable_block_source *bs,
 	}
 
 	err = fstat(fd, &st);
-	if (err < 0)
-		return -1;
+	if (err < 0) {
+		close(fd);
+		return REFTABLE_IO_ERROR;
+	}
 
 	p = reftable_calloc(sizeof(struct file_block_source));
 	p->size = st.st_size;
-- 
gitgitgadget

[PATCH v7 04/16] reftable: check reftable_stack_auto_compact() return value

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If compaction consistently errors out, we would accumulate reftables,
and the follow-on EXPECT would start failing, but catching errors
early helps debugging.

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index f4c743db80c..e84f50d27ff 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -839,6 +839,7 @@ static void test_reftable_stack_auto_compaction(void)
 		EXPECT_ERR(err);
 
 		err = reftable_stack_auto_compact(st);
+		EXPECT_ERR(err);
 		EXPECT(i < 3 || st->merged->stack_len < 2 * fastlog2(i));
 	}
 
-- 
gitgitgadget

[PATCH v7 05/16] reftable: ignore remove() return value in stack_test.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

If the cleanup fails, there is nothing we can do.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack_test.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/reftable/stack_test.c b/reftable/stack_test.c
index e84f50d27ff..f0be0be9fee 100644
--- a/reftable/stack_test.c
+++ b/reftable/stack_test.c
@@ -90,7 +90,7 @@ static void test_read_file(void)
 		EXPECT(0 == strcmp(want[i], names[i]));
 	}
 	free_names(names);
-	remove(fn);
+	EXPECT(unlink(fn) == 0);
 }
 
 static void test_parse_names(void)
-- 
gitgitgadget

[PATCH v7 06/16] reftable: fix resource warning

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This would trigger in the unlikely event that we are compacting, and
the next available file handle is 0.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/stack.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/reftable/stack.c b/reftable/stack.c
index 56bf5f2d84a..ddbdf1b9c8b 100644
--- a/reftable/stack.c
+++ b/reftable/stack.c
@@ -889,7 +889,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	struct strbuf new_table_path = STRBUF_INIT;
 	int err = 0;
 	int have_lock = 0;
-	int lock_file_fd = 0;
+	int lock_file_fd = -1;
 	int compact_count = last - first + 1;
 	char **listp = NULL;
 	char **delete_on_success =
@@ -923,7 +923,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 	}
 	/* Don't want to write to the lock for now.  */
 	close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 
 	have_lock = 1;
 	err = stack_uptodate(st);
@@ -1031,7 +1031,7 @@ static int stack_compact_range(struct reftable_stack *st, int first, int last,
 		goto done;
 	}
 	err = close(lock_file_fd);
-	lock_file_fd = 0;
+	lock_file_fd = -1;
 	if (err < 0) {
 		err = REFTABLE_IO_ERROR;
 		unlink(new_table_path.buf);
@@ -1068,9 +1068,9 @@ done:
 		listp++;
 	}
 	free_names(subtable_locks);
-	if (lock_file_fd > 0) {
+	if (lock_file_fd >= 0) {
 		close(lock_file_fd);
-		lock_file_fd = 0;
+		lock_file_fd = -1;
 	}
 	if (have_lock) {
 		unlink(lock_file_name.buf);
-- 
gitgitgadget

[PATCH v7 07/16] reftable: all xxx_free() functions accept NULL arguments

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This fixes NULL derefs in error paths. Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reader.c | 2 ++
 reftable/writer.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/reftable/reader.c b/reftable/reader.c
index 35781593a29..272378ed1d5 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -646,6 +646,8 @@ int reftable_new_reader(struct reftable_reader **p,
 
 void reftable_reader_free(struct reftable_reader *r)
 {
+	if (!r)
+		return;
 	reader_close(r);
 	reftable_free(r);
 }
diff --git a/reftable/writer.c b/reftable/writer.c
index 35c8649c9b7..83a23daf60b 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -150,6 +150,8 @@ void reftable_writer_set_limits(struct reftable_writer *w, uint64_t min,
 
 void reftable_writer_free(struct reftable_writer *w)
 {
+	if (!w)
+		return;
 	reftable_free(w->block);
 	reftable_free(w);
 }
-- 
gitgitgadget

[PATCH v7 08/16] reftable: order unittests by complexity

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This is a more practical ordering when working on refactorings of the
reftable code.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 t/helper/test-reftable.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c
index 26b03d7b789..1f0a28cbb64 100644
--- a/t/helper/test-reftable.c
+++ b/t/helper/test-reftable.c
@@ -3,15 +3,16 @@
 
 int cmd__reftable(int argc, const char **argv)
 {
+	/* test from simple to complex. */
 	basics_test_main(argc, argv);
+	record_test_main(argc, argv);
 	block_test_main(argc, argv);
-	merged_test_main(argc, argv);
+	tree_test_main(argc, argv);
 	pq_test_main(argc, argv);
-	record_test_main(argc, argv);
-	refname_test_main(argc, argv);
 	readwrite_test_main(argc, argv);
+	merged_test_main(argc, argv);
 	stack_test_main(argc, argv);
-	tree_test_main(argc, argv);
+	refname_test_main(argc, argv);
 	return 0;
 }
 
-- 
gitgitgadget

[PATCH v7 09/16] reftable: drop stray printf in readwrite_test

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/readwrite_test.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c
index edb9cfc1930..605ba0f9fd4 100644
--- a/reftable/readwrite_test.c
+++ b/reftable/readwrite_test.c
@@ -696,7 +696,6 @@ static void test_write_key_order(void)
 	err = reftable_writer_add_ref(w, &refs[0]);
 	EXPECT_ERR(err);
 	err = reftable_writer_add_ref(w, &refs[1]);
-	printf("%d\n", err);
 	EXPECT(err == REFTABLE_API_ERROR);
 	reftable_writer_close(w);
 	reftable_writer_free(w);
-- 
gitgitgadget

[PATCH v7 10/16] reftable: handle null refnames in reftable_ref_record_equal

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Spotted by Coverity.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 1f236b47d1c..53d83e4a41a 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -1154,9 +1154,11 @@ int reftable_ref_record_equal(struct reftable_ref_record *a,
 			      struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
-	if (!(0 == strcmp(a->refname, b->refname) &&
-	      a->update_index == b->update_index &&
-	      a->value_type == b->value_type))
+	if (!null_streq(a->refname, b->refname))
+		return 0;
+
+	if (a->update_index != b->update_index ||
+	    a->value_type != b->value_type)
 		return 0;
 
 	switch (a->value_type) {
-- 
gitgitgadget

[PATCH v7 11/16] reftable: make reftable-record.h function signatures const correct

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c          | 14 +++++++-------
 reftable/reftable-record.h | 14 +++++++-------
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 53d83e4a41a..1ae344c745b 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -72,7 +72,7 @@ int reftable_is_block_type(uint8_t typ)
 	return 0;
 }
 
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL1:
@@ -84,7 +84,7 @@ uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec)
 	}
 }
 
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec)
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec)
 {
 	switch (rec->value_type) {
 	case REFTABLE_REF_VAL2:
@@ -251,7 +251,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id)
 {
 	char hex[GIT_MAX_HEXSZ + 1];
@@ -881,8 +881,8 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size)
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size)
 {
 	if (!(null_streq(a->refname, b->refname) &&
 	      a->update_index == b->update_index &&
@@ -1150,8 +1150,8 @@ static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
 	return a == b;
 }
 
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size)
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size)
 {
 	assert(hash_size > 0);
 	if (!null_streq(a->refname, b->refname))
diff --git a/reftable/reftable-record.h b/reftable/reftable-record.h
index 5370d2288c7..67104f8fbfe 100644
--- a/reftable/reftable-record.h
+++ b/reftable/reftable-record.h
@@ -49,25 +49,25 @@ struct reftable_ref_record {
 
 /* Returns the first hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL1 or REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val1(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val1(const struct reftable_ref_record *rec);
 
 /* Returns the second hash, or NULL if `rec` is not of type
  * REFTABLE_REF_VAL2. */
-uint8_t *reftable_ref_record_val2(struct reftable_ref_record *rec);
+uint8_t *reftable_ref_record_val2(const struct reftable_ref_record *rec);
 
 /* returns whether 'ref' represents a deletion */
 int reftable_ref_record_is_deletion(const struct reftable_ref_record *ref);
 
 /* prints a reftable_ref_record onto stdout. Useful for debugging. */
-void reftable_ref_record_print(struct reftable_ref_record *ref,
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
 			       uint32_t hash_id);
 
 /* frees and nulls all pointer values inside `ref`. */
 void reftable_ref_record_release(struct reftable_ref_record *ref);
 
 /* returns whether two reftable_ref_records are the same. Useful for testing. */
-int reftable_ref_record_equal(struct reftable_ref_record *a,
-			      struct reftable_ref_record *b, int hash_size);
+int reftable_ref_record_equal(const struct reftable_ref_record *a,
+			      const struct reftable_ref_record *b, int hash_size);
 
 /* reftable_log_record holds a reflog entry */
 struct reftable_log_record {
@@ -104,8 +104,8 @@ int reftable_log_record_is_deletion(const struct reftable_log_record *log);
 void reftable_log_record_release(struct reftable_log_record *log);
 
 /* returns whether two records are equal. Useful for testing. */
-int reftable_log_record_equal(struct reftable_log_record *a,
-			      struct reftable_log_record *b, int hash_size);
+int reftable_log_record_equal(const struct reftable_log_record *a,
+			      const struct reftable_log_record *b, int hash_size);
 
 /* dumps a reftable_log_record on stdout, for debugging/testing. */
 void reftable_log_record_print(struct reftable_log_record *log,
-- 
gitgitgadget

[PATCH v7 12/16] reftable: implement record equality generically

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This simplifies unittests a little, and provides further coverage for
reftable_record_copy().

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 54 +++++++++++++++++++++++++++++++++++++++++-
 reftable/record.h      |  5 +++-
 reftable/record_test.c | 23 +++---------------
 3 files changed, 60 insertions(+), 22 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 1ae344c745b..b543585ad39 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -430,6 +430,14 @@ static int reftable_ref_record_is_deletion_void(const void *p)
 		(const struct reftable_ref_record *)p);
 }
 
+static int reftable_ref_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	struct reftable_ref_record *ra = (struct reftable_ref_record *) a;
+	struct reftable_ref_record *rb = (struct reftable_ref_record *) b;
+	return reftable_ref_record_equal(ra, rb, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -439,6 +447,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.decode = &reftable_ref_record_decode,
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
+	.equal = &reftable_ref_record_equal_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -572,6 +581,23 @@ static int not_a_deletion(const void *p)
 	return 0;
 }
 
+static int reftable_obj_record_equal_void(const void *a, const void *b, int hash_size)
+{
+	struct reftable_obj_record *ra = (struct reftable_obj_record *) a;
+	struct reftable_obj_record *rb = (struct reftable_obj_record *) b;
+
+	if (ra->hash_prefix_len != rb->hash_prefix_len
+	    || ra->offset_len != rb->offset_len)
+		return 0;
+
+	if (memcmp(ra->hash_prefix, rb->hash_prefix, ra->hash_prefix_len))
+		return 0;
+	if (memcmp(ra->offsets, rb->offsets, ra->offset_len * sizeof(uint64_t)))
+		return 0;
+
+	return 1;
+}
+
 static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.key = &reftable_obj_record_key,
 	.type = BLOCK_TYPE_OBJ,
@@ -580,7 +606,8 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.encode = &reftable_obj_record_encode,
 	.decode = &reftable_obj_record_decode,
 	.release = &reftable_obj_record_release,
-	.is_deletion = not_a_deletion,
+	.is_deletion = &not_a_deletion,
+	.equal = &reftable_obj_record_equal_void,
 };
 
 void reftable_log_record_print(struct reftable_log_record *log,
@@ -881,6 +908,14 @@ static int zero_hash_eq(uint8_t *a, uint8_t *b, int sz)
 	return !memcmp(a, b, sz);
 }
 
+static int reftable_log_record_equal_void(const void *a,
+					  const void *b, int hash_size)
+{
+	return reftable_log_record_equal((struct reftable_log_record *) a,
+					 (struct reftable_log_record *) b,
+					 hash_size);
+}
+
 int reftable_log_record_equal(const struct reftable_log_record *a,
 			      const struct reftable_log_record *b, int hash_size)
 {
@@ -924,6 +959,7 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
+	.equal = &reftable_log_record_equal_void
 };
 
 struct reftable_record reftable_new_record(uint8_t typ)
@@ -1042,6 +1078,14 @@ static int reftable_index_record_decode(void *rec, struct strbuf key,
 	return start.len - in.len;
 }
 
+static int reftable_index_record_equal(const void *a, const void *b, int hash_size)
+{
+	struct reftable_index_record *ia = (struct reftable_index_record *) a;
+	struct reftable_index_record *ib = (struct reftable_index_record *) b;
+
+	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1051,6 +1095,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.decode = &reftable_index_record_decode,
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
+	.equal = &reftable_index_record_equal,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1098,6 +1143,13 @@ int reftable_record_is_deletion(struct reftable_record *rec)
 	return rec->ops->is_deletion(rec->data);
 }
 
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
+{
+	if (a->ops != b->ops)
+		return 0;
+	return a->ops->equal(a->data, b->data, hash_size);
+}
+
 void reftable_record_from_ref(struct reftable_record *rec,
 			      struct reftable_ref_record *ref_rec)
 {
diff --git a/reftable/record.h b/reftable/record.h
index 498e8c50bf4..da75d7d1f11 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -58,6 +58,9 @@ struct reftable_record_vtable {
 
 	/* is this a tombstone? */
 	int (*is_deletion)(const void *rec);
+
+	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
+	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
 /* record is a generic wrapper for different types of records. */
@@ -98,7 +101,7 @@ struct reftable_obj_record {
 };
 
 /* see struct record_vtable */
-
+int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f4ad7cace41..92680848156 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -21,18 +21,7 @@ static void test_copy(struct reftable_record *rec)
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
-	switch (reftable_record_type(&copy)) {
-	case BLOCK_TYPE_REF:
-		EXPECT(reftable_ref_record_equal(reftable_record_as_ref(&copy),
-						 reftable_record_as_ref(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	case BLOCK_TYPE_LOG:
-		EXPECT(reftable_log_record_equal(reftable_record_as_log(&copy),
-						 reftable_record_as_log(rec),
-						 GIT_SHA1_RAWSZ));
-		break;
-	}
+	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
 	reftable_record_destroy(&copy);
 }
 
@@ -346,13 +335,7 @@ static void test_reftable_obj_record_roundtrip(void)
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(in.hash_prefix_len == out.hash_prefix_len);
-		EXPECT(in.offset_len == out.offset_len);
-
-		EXPECT(!memcmp(in.hash_prefix, out.hash_prefix,
-			       in.hash_prefix_len));
-		EXPECT(0 == memcmp(in.offsets, out.offsets,
-				   sizeof(uint64_t) * in.offset_len));
+		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
 		reftable_record_release(&rec_out);
 	}
@@ -390,7 +373,7 @@ static void test_reftable_index_record_roundtrip(void)
 	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(in.offset == out.offset);
+	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
 
 	reftable_record_release(&out_rec);
 	strbuf_release(&key);
-- 
gitgitgadget

[PATCH v7 13/16] reftable: remove outdated file reftable.c

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This was renamed to generic.c, but the origin was never removed

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/reftable.c | 115 --------------------------------------------
 1 file changed, 115 deletions(-)
 delete mode 100644 reftable/reftable.c

diff --git a/reftable/reftable.c b/reftable/reftable.c
deleted file mode 100644
index 0e4607a7cd6..00000000000
--- a/reftable/reftable.c
+++ /dev/null
@@ -1,115 +0,0 @@
-/*
-Copyright 2020 Google LLC
-
-Use of this source code is governed by a BSD-style
-license that can be found in the LICENSE file or at
-https://developers.google.com/open-source/licenses/bsd
-*/
-
-#include "basics.h"
-#include "record.h"
-#include "generic.h"
-#include "reftable-iterator.h"
-#include "reftable-generic.h"
-
-int reftable_table_seek_ref(struct reftable_table *tab,
-			    struct reftable_iterator *it, const char *name)
-{
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
-	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
-	return tab->ops->seek_record(tab->table_arg, it, &rec);
-}
-
-int reftable_table_read_ref(struct reftable_table *tab, const char *name,
-			    struct reftable_ref_record *ref)
-{
-	struct reftable_iterator it = { NULL };
-	int err = reftable_table_seek_ref(tab, &it, name);
-	if (err)
-		goto done;
-
-	err = reftable_iterator_next_ref(&it, ref);
-	if (err)
-		goto done;
-
-	if (strcmp(ref->refname, name) ||
-	    reftable_ref_record_is_deletion(ref)) {
-		reftable_ref_record_release(ref);
-		err = 1;
-		goto done;
-	}
-
-done:
-	reftable_iterator_destroy(&it);
-	return err;
-}
-
-uint64_t reftable_table_max_update_index(struct reftable_table *tab)
-{
-	return tab->ops->max_update_index(tab->table_arg);
-}
-
-uint64_t reftable_table_min_update_index(struct reftable_table *tab)
-{
-	return tab->ops->min_update_index(tab->table_arg);
-}
-
-uint32_t reftable_table_hash_id(struct reftable_table *tab)
-{
-	return tab->ops->hash_id(tab->table_arg);
-}
-
-void reftable_iterator_destroy(struct reftable_iterator *it)
-{
-	if (!it->ops) {
-		return;
-	}
-	it->ops->close(it->iter_arg);
-	it->ops = NULL;
-	FREE_AND_NULL(it->iter_arg);
-}
-
-int reftable_iterator_next_ref(struct reftable_iterator *it,
-			       struct reftable_ref_record *ref)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
-}
-
-int reftable_iterator_next_log(struct reftable_iterator *it,
-			       struct reftable_log_record *log)
-{
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
-}
-
-int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
-{
-	return it->ops->next(it->iter_arg, rec);
-}
-
-static int empty_iterator_next(void *arg, struct reftable_record *rec)
-{
-	return 1;
-}
-
-static void empty_iterator_close(void *arg)
-{
-}
-
-static struct reftable_iterator_vtable empty_vtable = {
-	.next = &empty_iterator_next,
-	.close = &empty_iterator_close,
-};
-
-void iterator_set_empty(struct reftable_iterator *it)
-{
-	assert(!it->ops);
-	it->iter_arg = NULL;
-	it->ops = &empty_vtable;
-}
-- 
gitgitgadget

[PATCH v7 14/16] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This reduces the amount of glue code, because we don't need a void
pointer or vtable within the structure.

The only snag is that reftable_index_record contain a strbuf, so it
cannot be zero-initialized. To address this, use reftable_new_record()
to return fresh instance, given a record type. Since
reftable_new_record() doesn't cause heap allocation anymore, it should
be balanced with reftable_record_release() rather than
reftable_record_destroy().

Thanks to Peff for the suggestion.

Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c       |   2 +-
 reftable/block_test.c  |  22 ++---
 reftable/generic.c     |  39 +++++---
 reftable/iter.c        |   4 +-
 reftable/merged.c      |  31 +++---
 reftable/pq.c          |   2 +-
 reftable/pq_test.c     |  29 +++---
 reftable/reader.c      |  84 +++++++++--------
 reftable/record.c      | 209 +++++++++++++++++++----------------------
 reftable/record.h      |  42 ++++-----
 reftable/record_test.c | 154 +++++++++++++++---------------
 reftable/writer.c      |  38 ++++----
 12 files changed, 328 insertions(+), 328 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 5832f85501d..80ad10bb794 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -422,7 +422,7 @@ int block_reader_seek(struct block_reader *br, struct block_iter *it,
 done:
 	strbuf_release(&key);
 	strbuf_release(&next.last_key);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 
 	return err;
 }
diff --git a/reftable/block_test.c b/reftable/block_test.c
index 4b3ea262dcb..fa2ee092ec0 100644
--- a/reftable/block_test.c
+++ b/reftable/block_test.c
@@ -26,8 +26,9 @@ static void test_block_read_write(void)
 	struct block_writer bw = {
 		.last_key = STRBUF_INIT,
 	};
-	struct reftable_ref_record ref = { NULL };
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+	};
 	int i = 0;
 	int n;
 	struct block_reader br = { 0 };
@@ -40,7 +41,6 @@ static void test_block_read_write(void)
 	block.source = malloc_block_source();
 	block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size,
 			  header_off, hash_size(GIT_SHA1_FORMAT_ID));
-	reftable_record_from_ref(&rec, &ref);
 
 	for (i = 0; i < N; i++) {
 		char name[100];
@@ -48,14 +48,14 @@ static void test_block_read_write(void)
 		snprintf(name, sizeof(name), "branch%02d", i);
 		memset(hash, i, sizeof(hash));
 
-		ref.refname = name;
-		ref.value_type = REFTABLE_REF_VAL1;
-		ref.value.val1 = hash;
+		rec.u.ref.refname = name;
+		rec.u.ref.value_type = REFTABLE_REF_VAL1;
+		rec.u.ref.value.val1 = hash;
 
 		names[i] = xstrdup(name);
 		n = block_writer_add(&bw, &rec);
-		ref.refname = NULL;
-		ref.value_type = REFTABLE_REF_DELETION;
+		rec.u.ref.refname = NULL;
+		rec.u.ref.value_type = REFTABLE_REF_DELETION;
 		EXPECT(n == 0);
 	}
 
@@ -74,7 +74,7 @@ static void test_block_read_write(void)
 		if (r > 0) {
 			break;
 		}
-		EXPECT_STREQ(names[j], ref.refname);
+		EXPECT_STREQ(names[j], rec.u.ref.refname);
 		j++;
 	}
 
@@ -92,7 +92,7 @@ static void test_block_read_write(void)
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
 
-		EXPECT_STREQ(names[i], ref.refname);
+		EXPECT_STREQ(names[i], rec.u.ref.refname);
 
 		want.len--;
 		n = block_reader_seek(&br, &it, &want);
@@ -100,7 +100,7 @@ static void test_block_read_write(void)
 
 		n = block_iter_next(&it, &rec);
 		EXPECT(n == 0);
-		EXPECT_STREQ(names[10 * (i / 10)], ref.refname);
+		EXPECT_STREQ(names[10 * (i / 10)], rec.u.ref.refname);
 
 		block_iter_close(&it);
 	}
diff --git a/reftable/generic.c b/reftable/generic.c
index 7a8a738d860..b176c1cde0a 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -7,6 +7,7 @@ https://developers.google.com/open-source/licenses/bsd
 */
 
 #include "basics.h"
+#include "constants.h"
 #include "record.h"
 #include "generic.h"
 #include "reftable-iterator.h"
@@ -15,23 +16,23 @@ https://developers.google.com/open-source/licenses/bsd
 int reftable_table_seek_ref(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = { .refname = (char *)name, }
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
 int reftable_table_seek_log(struct reftable_table *tab,
 			    struct reftable_iterator *it, const char *name)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = ~((uint64_t)0),
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = ~((uint64_t)0),
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return tab->ops->seek_record(tab->table_arg, it, &rec);
 }
 
@@ -129,17 +130,25 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
 int reftable_iterator_next_ref(struct reftable_iterator *it,
 			       struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, ref);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
+	int err = iterator_next(it, &rec);
+	*ref = rec.u.ref;
+	return err;
 }
 
 int reftable_iterator_next_log(struct reftable_iterator *it,
 			       struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, log);
-	return iterator_next(it, &rec);
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
+	int err = iterator_next(it, &rec);
+	*log = rec.u.log;
+	return err;
 }
 
 int iterator_next(struct reftable_iterator *it, struct reftable_record *rec)
diff --git a/reftable/iter.c b/reftable/iter.c
index 93d04f735b8..a8d174c0406 100644
--- a/reftable/iter.c
+++ b/reftable/iter.c
@@ -32,7 +32,7 @@ static int filtering_ref_iterator_next(void *iter_arg,
 				       struct reftable_record *rec)
 {
 	struct filtering_ref_iterator *fri = iter_arg;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 	int err = 0;
 	while (1) {
 		err = reftable_iterator_next_ref(&fri->it, ref);
@@ -127,7 +127,7 @@ static int indexed_table_ref_iter_next_block(struct indexed_table_ref_iter *it)
 static int indexed_table_ref_iter_next(void *p, struct reftable_record *rec)
 {
 	struct indexed_table_ref_iter *it = p;
-	struct reftable_ref_record *ref = rec->data;
+	struct reftable_ref_record *ref = &rec->u.ref;
 
 	while (1) {
 		int err = block_iter_next(&it->cur, rec);
diff --git a/reftable/merged.c b/reftable/merged.c
index e5b53da6db3..82511762d2a 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -30,7 +30,7 @@ static int merged_iter_init(struct merged_iter *mi)
 
 		if (err > 0) {
 			reftable_iterator_destroy(&mi->stack[i]);
-			reftable_record_destroy(&rec);
+			reftable_record_release(&rec);
 		} else {
 			struct pq_entry e = {
 				.rec = rec,
@@ -57,18 +57,17 @@ static void merged_iter_close(void *p)
 static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
-	struct reftable_record rec = reftable_new_record(mi->typ);
 	struct pq_entry e = {
-		.rec = rec,
+		.rec = reftable_new_record(mi->typ),
 		.index = idx,
 	};
-	int err = iterator_next(&mi->stack[idx], &rec);
+	int err = iterator_next(&mi->stack[idx], &e.rec);
 	if (err < 0)
 		return err;
 
 	if (err > 0) {
 		reftable_iterator_destroy(&mi->stack[idx]);
-		reftable_record_destroy(&rec);
+		reftable_record_release(&e.rec);
 		return 0;
 	}
 
@@ -126,11 +125,11 @@ static int merged_iter_next_entry(struct merged_iter *mi,
 		if (err < 0) {
 			return err;
 		}
-		reftable_record_destroy(&top.rec);
+		reftable_record_release(&top.rec);
 	}
 
 	reftable_record_copy_from(rec, &entry.rec, hash_size(mi->hash_id));
-	reftable_record_destroy(&entry.rec);
+	reftable_record_release(&entry.rec);
 	strbuf_release(&entry_key);
 	return 0;
 }
@@ -290,11 +289,10 @@ int reftable_merged_table_seek_ref(struct reftable_merged_table *mt,
 				   struct reftable_iterator *it,
 				   const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = { .refname = (char *)name, },
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
@@ -302,12 +300,13 @@ int reftable_merged_table_seek_log_at(struct reftable_merged_table *mt,
 				      struct reftable_iterator *it,
 				      const char *name, uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return merged_table_seek_record(mt, it, &rec);
 }
 
diff --git a/reftable/pq.c b/reftable/pq.c
index efc474017a2..5ffc8380c4a 100644
--- a/reftable/pq.c
+++ b/reftable/pq.c
@@ -98,7 +98,7 @@ void merged_iter_pqueue_release(struct merged_iter_pqueue *pq)
 {
 	int i = 0;
 	for (i = 0; i < pq->len; i++) {
-		reftable_record_destroy(&pq->heap[i].rec);
+		reftable_record_release(&pq->heap[i].rec);
 	}
 	FREE_AND_NULL(pq->heap);
 	pq->len = pq->cap = 0;
diff --git a/reftable/pq_test.c b/reftable/pq_test.c
index c9bb05e37b7..5d58a4c05b6 100644
--- a/reftable/pq_test.c
+++ b/reftable/pq_test.c
@@ -31,7 +31,7 @@ static void test_pq(void)
 	int N = ARRAY_SIZE(names) - 1;
 
 	struct merged_iter_pqueue pq = { NULL };
-	const char *last = NULL;
+	char *last = NULL;
 
 	int i = 0;
 	for (i = 0; i < N; i++) {
@@ -42,12 +42,12 @@ static void test_pq(void)
 
 	i = 1;
 	do {
-		struct reftable_record rec =
-			reftable_new_record(BLOCK_TYPE_REF);
-		struct pq_entry e = { 0 };
-
-		reftable_record_as_ref(&rec)->refname = names[i];
-		e.rec = rec;
+		struct pq_entry e = {
+			.rec = {
+				.type = BLOCK_TYPE_REF,
+				.u.ref = { .refname = names[i], }
+			}
+		};
 		merged_iter_pqueue_add(&pq, e);
 		merged_iter_pqueue_check(pq);
 		i = (i * 7) % N;
@@ -55,19 +55,18 @@ static void test_pq(void)
 
 	while (!merged_iter_pqueue_is_empty(pq)) {
 		struct pq_entry e = merged_iter_pqueue_remove(&pq);
-		struct reftable_ref_record *ref =
-			reftable_record_as_ref(&e.rec);
-
+		struct reftable_record *rec = &e.rec;
 		merged_iter_pqueue_check(pq);
 
+		EXPECT(reftable_record_type(rec) == BLOCK_TYPE_REF);
 		if (last) {
-			EXPECT(strcmp(last, ref->refname) < 0);
+			EXPECT(strcmp(last, rec->u.ref.refname) < 0);
 		}
-		last = ref->refname;
-		ref->refname = NULL;
-		reftable_free(ref);
+		/* this is names[i], so don't dealloc. */
+		last = rec->u.ref.refname;
+		rec->u.ref.refname = NULL;
+		reftable_record_release(rec);
 	}
-
 	for (i = 0; i < N; i++) {
 		reftable_free(names[i]);
 	}
diff --git a/reftable/reader.c b/reftable/reader.c
index 272378ed1d5..565cda8d5a2 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -239,8 +239,7 @@ static int table_iter_next_in_block(struct table_iter *ti,
 {
 	int res = block_iter_next(&ti->bi, rec);
 	if (res == 0 && reftable_record_type(rec) == BLOCK_TYPE_REF) {
-		((struct reftable_ref_record *)rec->data)->update_index +=
-			ti->r->min_update_index;
+		rec->u.ref.update_index += ti->r->min_update_index;
 	}
 
 	return res;
@@ -480,7 +479,7 @@ static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
 
 done:
 	block_iter_close(&next.bi);
-	reftable_record_destroy(&rec);
+	reftable_record_release(&rec);
 	strbuf_release(&want_key);
 	strbuf_release(&got_key);
 	return err;
@@ -490,34 +489,36 @@ static int reader_seek_indexed(struct reftable_reader *r,
 			       struct reftable_iterator *it,
 			       struct reftable_record *rec)
 {
-	struct reftable_index_record want_index = { .last_key = STRBUF_INIT };
-	struct reftable_record want_index_rec = { NULL };
-	struct reftable_index_record index_result = { .last_key = STRBUF_INIT };
-	struct reftable_record index_result_rec = { NULL };
+	struct reftable_record want_index = {
+		.type = BLOCK_TYPE_INDEX, .u.idx = { .last_key = STRBUF_INIT }
+	};
+	struct reftable_record index_result = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	struct table_iter index_iter = TABLE_ITER_INIT;
 	struct table_iter next = TABLE_ITER_INIT;
 	int err = 0;
 
-	reftable_record_key(rec, &want_index.last_key);
-	reftable_record_from_index(&want_index_rec, &want_index);
-	reftable_record_from_index(&index_result_rec, &index_result);
+	reftable_record_key(rec, &want_index.u.idx.last_key);
 
 	err = reader_start(r, &index_iter, reftable_record_type(rec), 1);
 	if (err < 0)
 		goto done;
 
-	err = reader_seek_linear(r, &index_iter, &want_index_rec);
+	err = reader_seek_linear(r, &index_iter, &want_index);
 	while (1) {
-		err = table_iter_next(&index_iter, &index_result_rec);
+		err = table_iter_next(&index_iter, &index_result);
 		table_iter_block_done(&index_iter);
 		if (err != 0)
 			goto done;
 
-		err = reader_table_iter_at(r, &next, index_result.offset, 0);
+		err = reader_table_iter_at(r, &next, index_result.u.idx.offset,
+					   0);
 		if (err != 0)
 			goto done;
 
-		err = block_iter_seek(&next.bi, &want_index.last_key);
+		err = block_iter_seek(&next.bi, &want_index.u.idx.last_key);
 		if (err < 0)
 			goto done;
 
@@ -545,8 +546,8 @@ static int reader_seek_indexed(struct reftable_reader *r,
 done:
 	block_iter_close(&next.bi);
 	table_iter_close(&index_iter);
-	reftable_record_release(&want_index_rec);
-	reftable_record_release(&index_result_rec);
+	reftable_record_release(&want_index);
+	reftable_record_release(&index_result);
 	return err;
 }
 
@@ -595,11 +596,10 @@ static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 int reftable_reader_seek_ref(struct reftable_reader *r,
 			     struct reftable_iterator *it, const char *name)
 {
-	struct reftable_ref_record ref = {
-		.refname = (char *)name,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = { .refname = (char *)name, },
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_ref(&rec, &ref);
 	return reader_seek(r, it, &rec);
 }
 
@@ -607,12 +607,13 @@ int reftable_reader_seek_log_at(struct reftable_reader *r,
 				struct reftable_iterator *it, const char *name,
 				uint64_t update_index)
 {
-	struct reftable_log_record log = {
-		.refname = (char *)name,
-		.update_index = update_index,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = {
+			.refname = (char *)name,
+			.update_index = update_index,
+		}
 	};
-	struct reftable_record rec = { NULL };
-	reftable_record_from_log(&rec, &log);
 	return reader_seek(r, it, &rec);
 }
 
@@ -656,31 +657,33 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 					    struct reftable_iterator *it,
 					    uint8_t *oid)
 {
-	struct reftable_obj_record want = {
-		.hash_prefix = oid,
-		.hash_prefix_len = r->object_id_len,
+	struct reftable_record want = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = oid,
+			.hash_prefix_len = r->object_id_len,
+		},
 	};
-	struct reftable_record want_rec = { NULL };
 	struct reftable_iterator oit = { NULL };
-	struct reftable_obj_record got = { NULL };
-	struct reftable_record got_rec = { NULL };
+	struct reftable_record got = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = { 0 },
+	};
 	int err = 0;
 	struct indexed_table_ref_iter *itr = NULL;
 
 	/* Look through the reverse index. */
-	reftable_record_from_obj(&want_rec, &want);
-	err = reader_seek(r, &oit, &want_rec);
+	err = reader_seek(r, &oit, &want);
 	if (err != 0)
 		goto done;
 
 	/* read out the reftable_obj_record */
-	reftable_record_from_obj(&got_rec, &got);
-	err = iterator_next(&oit, &got_rec);
+	err = iterator_next(&oit, &got);
 	if (err < 0)
 		goto done;
 
-	if (err > 0 ||
-	    memcmp(want.hash_prefix, got.hash_prefix, r->object_id_len)) {
+	if (err > 0 || memcmp(want.u.obj.hash_prefix, got.u.obj.hash_prefix,
+			      r->object_id_len)) {
 		/* didn't find it; return empty iterator */
 		iterator_set_empty(it);
 		err = 0;
@@ -688,15 +691,16 @@ static int reftable_reader_refs_for_indexed(struct reftable_reader *r,
 	}
 
 	err = new_indexed_table_ref_iter(&itr, r, oid, hash_size(r->hash_id),
-					 got.offsets, got.offset_len);
+					 got.u.obj.offsets,
+					 got.u.obj.offset_len);
 	if (err < 0)
 		goto done;
-	got.offsets = NULL;
+	got.u.obj.offsets = NULL;
 	iterator_from_indexed_table_ref_iter(it, itr);
 
 done:
 	reftable_iterator_destroy(&oit);
-	reftable_record_release(&got_rec);
+	reftable_record_release(&got);
 	return err;
 }
 
diff --git a/reftable/record.c b/reftable/record.c
index b543585ad39..238ff882aab 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -15,6 +15,24 @@ https://developers.google.com/open-source/licenses/bsd
 #include "reftable-error.h"
 #include "basics.h"
 
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec);
+
+static void *reftable_record_data(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &rec->u.ref;
+	case BLOCK_TYPE_LOG:
+		return &rec->u.log;
+	case BLOCK_TYPE_INDEX:
+		return &rec->u.idx;
+	case BLOCK_TYPE_OBJ:
+		return &rec->u.obj;
+	}
+	abort();
+}
+
 int get_var_int(uint64_t *dest, struct string_view *in)
 {
 	int ptr = 0;
@@ -474,12 +492,13 @@ static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 		(const struct reftable_obj_record *)src_rec;
 
 	reftable_obj_record_release(obj);
-	*obj = *src;
-	obj->hash_prefix = reftable_malloc(obj->hash_prefix_len);
+	obj->hash_prefix = reftable_malloc(src->hash_prefix_len);
+	obj->hash_prefix_len = src->hash_prefix_len;
 	memcpy(obj->hash_prefix, src->hash_prefix, obj->hash_prefix_len);
 
-	obj->offsets = reftable_malloc(obj->offset_len * sizeof(uint64_t));
-	COPY_ARRAY(obj->offsets, src->offsets, obj->offset_len);
+	obj->offsets = reftable_malloc(src->offset_len * sizeof(uint64_t));
+	obj->offset_len = src->offset_len;
+	COPY_ARRAY(obj->offsets, src->offsets, src->offset_len);
 }
 
 static uint8_t reftable_obj_record_val_type(const void *rec)
@@ -962,58 +981,6 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.equal = &reftable_log_record_equal_void
 };
 
-struct reftable_record reftable_new_record(uint8_t typ)
-{
-	struct reftable_record rec = { NULL };
-	switch (typ) {
-	case BLOCK_TYPE_REF: {
-		struct reftable_ref_record *r =
-			reftable_calloc(sizeof(struct reftable_ref_record));
-		reftable_record_from_ref(&rec, r);
-		return rec;
-	}
-
-	case BLOCK_TYPE_OBJ: {
-		struct reftable_obj_record *r =
-			reftable_calloc(sizeof(struct reftable_obj_record));
-		reftable_record_from_obj(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_LOG: {
-		struct reftable_log_record *r =
-			reftable_calloc(sizeof(struct reftable_log_record));
-		reftable_record_from_log(&rec, r);
-		return rec;
-	}
-	case BLOCK_TYPE_INDEX: {
-		struct reftable_index_record empty = { .last_key =
-							       STRBUF_INIT };
-		struct reftable_index_record *r =
-			reftable_calloc(sizeof(struct reftable_index_record));
-		*r = empty;
-		reftable_record_from_index(&rec, r);
-		return rec;
-	}
-	}
-	abort();
-	return rec;
-}
-
-/* clear out the record, yielding the reftable_record data that was
- * encapsulated. */
-static void *reftable_record_yield(struct reftable_record *rec)
-{
-	void *p = rec->data;
-	rec->data = NULL;
-	return p;
-}
-
-void reftable_record_destroy(struct reftable_record *rec)
-{
-	reftable_record_release(rec);
-	reftable_free(reftable_record_yield(rec));
-}
-
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_index_record *rec = r;
@@ -1036,6 +1003,7 @@ static void reftable_index_record_release(void *rec)
 {
 	struct reftable_index_record *idx = rec;
 	strbuf_release(&idx->last_key);
+	idx->offset = 0;
 }
 
 static uint8_t reftable_index_record_val_type(const void *rec)
@@ -1100,98 +1068,60 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
 {
-	rec->ops->key(rec->data, dest);
+	reftable_record_vtable(rec)->key(reftable_record_data(rec), dest);
 }
 
 uint8_t reftable_record_type(struct reftable_record *rec)
 {
-	return rec->ops->type;
+	return rec->type;
 }
 
 int reftable_record_encode(struct reftable_record *rec, struct string_view dest,
 			   int hash_size)
 {
-	return rec->ops->encode(rec->data, dest, hash_size);
+	return reftable_record_vtable(rec)->encode(reftable_record_data(rec),
+						   dest, hash_size);
 }
 
 void reftable_record_copy_from(struct reftable_record *rec,
 			       struct reftable_record *src, int hash_size)
 {
-	assert(src->ops->type == rec->ops->type);
+	assert(src->type == rec->type);
 
-	rec->ops->copy_from(rec->data, src->data, hash_size);
+	reftable_record_vtable(rec)->copy_from(reftable_record_data(rec),
+					       reftable_record_data(src),
+					       hash_size);
 }
 
 uint8_t reftable_record_val_type(struct reftable_record *rec)
 {
-	return rec->ops->val_type(rec->data);
+	return reftable_record_vtable(rec)->val_type(reftable_record_data(rec));
 }
 
 int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   uint8_t extra, struct string_view src, int hash_size)
 {
-	return rec->ops->decode(rec->data, key, extra, src, hash_size);
+	return reftable_record_vtable(rec)->decode(reftable_record_data(rec),
+						   key, extra, src, hash_size);
 }
 
 void reftable_record_release(struct reftable_record *rec)
 {
-	rec->ops->release(rec->data);
+	reftable_record_vtable(rec)->release(reftable_record_data(rec));
 }
 
 int reftable_record_is_deletion(struct reftable_record *rec)
 {
-	return rec->ops->is_deletion(rec->data);
+	return reftable_record_vtable(rec)->is_deletion(
+		reftable_record_data(rec));
 }
 
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size)
 {
-	if (a->ops != b->ops)
+	if (a->type != b->type)
 		return 0;
-	return a->ops->equal(a->data, b->data, hash_size);
-}
-
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *ref_rec)
-{
-	assert(!rec->ops);
-	rec->data = ref_rec;
-	rec->ops = &reftable_ref_record_vtable;
-}
-
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *obj_rec)
-{
-	assert(!rec->ops);
-	rec->data = obj_rec;
-	rec->ops = &reftable_obj_record_vtable;
-}
-
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *index_rec)
-{
-	assert(!rec->ops);
-	rec->data = index_rec;
-	rec->ops = &reftable_index_record_vtable;
-}
-
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *log_rec)
-{
-	assert(!rec->ops);
-	rec->data = log_rec;
-	rec->ops = &reftable_log_record_vtable;
-}
-
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_REF);
-	return rec->data;
-}
-
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *rec)
-{
-	assert(reftable_record_type(rec) == BLOCK_TYPE_LOG);
-	return rec->data;
+	return reftable_record_vtable(a)->equal(
+		reftable_record_data(a), reftable_record_data(b), hash_size);
 }
 
 static int hash_equal(uint8_t *a, uint8_t *b, int hash_size)
@@ -1264,3 +1194,60 @@ void string_view_consume(struct string_view *s, int n)
 	s->buf += n;
 	s->len -= n;
 }
+
+static struct reftable_record_vtable *
+reftable_record_vtable(struct reftable_record *rec)
+{
+	switch (rec->type) {
+	case BLOCK_TYPE_REF:
+		return &reftable_ref_record_vtable;
+	case BLOCK_TYPE_LOG:
+		return &reftable_log_record_vtable;
+	case BLOCK_TYPE_INDEX:
+		return &reftable_index_record_vtable;
+	case BLOCK_TYPE_OBJ:
+		return &reftable_obj_record_vtable;
+	}
+	abort();
+}
+
+struct reftable_record reftable_new_record(uint8_t type)
+{
+	struct reftable_record clean = {
+		.type = type,
+	};
+
+	/* the following is involved, but the naive solution (just return
+	 * `clean` as is, except for BLOCK_TYPE_INDEX), returns a garbage
+	 * clean.u.obj.offsets pointer on Windows VS CI.  Go figure.
+	 */
+	switch (type) {
+	case BLOCK_TYPE_OBJ:
+	{
+		struct reftable_obj_record obj = { 0 };
+		clean.u.obj = obj;
+		break;
+	}
+	case BLOCK_TYPE_INDEX:
+	{
+		struct reftable_index_record idx = {
+			.last_key = STRBUF_INIT,
+		};
+		clean.u.idx = idx;
+		break;
+	}
+	case BLOCK_TYPE_REF:
+	{
+		struct reftable_ref_record ref = { 0 };
+		clean.u.ref = ref;
+		break;
+	}
+	case BLOCK_TYPE_LOG:
+	{
+		struct reftable_log_record log = { 0 };
+		clean.u.log = log;
+		break;
+	}
+	}
+	return clean;
+}
diff --git a/reftable/record.h b/reftable/record.h
index da75d7d1f11..010a322e901 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -63,16 +63,10 @@ struct reftable_record_vtable {
 	int (*equal)(const void *a, const void *b, int hash_size);
 };
 
-/* record is a generic wrapper for different types of records. */
-struct reftable_record {
-	void *data;
-	struct reftable_record_vtable *ops;
-};
-
 /* returns true for recognized block types. Block start with the block type. */
 int reftable_is_block_type(uint8_t typ);
 
-/* creates a malloced record of the given type. Dispose with record_destroy */
+/* return an initialized record for the given type */
 struct reftable_record reftable_new_record(uint8_t typ);
 
 /* Encode `key` into `dest`. Sets `is_restart` to indicate a restart. Returns
@@ -100,6 +94,22 @@ struct reftable_obj_record {
 	int offset_len;
 };
 
+/* record is a generic wrapper for different types of records. It is normally
+ * created on the stack, or embedded within another struct. If the type is
+ * known, a fresh instance can be initialized explicitly. Otherwise, use
+ * reftable_new_record() to initialize generically (as the index_record is not
+ * valid as 0-initialized structure)
+ */
+struct reftable_record {
+	uint8_t type;
+	union {
+		struct reftable_ref_record ref;
+		struct reftable_log_record log;
+		struct reftable_obj_record obj;
+		struct reftable_index_record idx;
+	} u;
+};
+
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
@@ -114,25 +124,9 @@ int reftable_record_decode(struct reftable_record *rec, struct strbuf key,
 			   int hash_size);
 int reftable_record_is_deletion(struct reftable_record *rec);
 
-/* zeroes out the embedded record */
+/* frees and zeroes out the embedded record */
 void reftable_record_release(struct reftable_record *rec);
 
-/* clear and deallocate embedded record, and zero `rec`. */
-void reftable_record_destroy(struct reftable_record *rec);
-
-/* initialize generic records from concrete records. The generic record should
- * be zeroed out. */
-void reftable_record_from_obj(struct reftable_record *rec,
-			      struct reftable_obj_record *objrec);
-void reftable_record_from_index(struct reftable_record *rec,
-				struct reftable_index_record *idxrec);
-void reftable_record_from_ref(struct reftable_record *rec,
-			      struct reftable_ref_record *refrec);
-void reftable_record_from_log(struct reftable_record *rec,
-			      struct reftable_log_record *logrec);
-struct reftable_ref_record *reftable_record_as_ref(struct reftable_record *ref);
-struct reftable_log_record *reftable_record_as_log(struct reftable_record *ref);
-
 /* for qsort. */
 int reftable_ref_record_compare_name(const void *a, const void *b);
 
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 92680848156..4a025c48104 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -16,13 +16,14 @@
 
 static void test_copy(struct reftable_record *rec)
 {
-	struct reftable_record copy =
-		reftable_new_record(reftable_record_type(rec));
+	struct reftable_record copy = { 0 };
+	uint8_t type = reftable_record_type(rec);
+	copy = reftable_new_record(type);
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
-	reftable_record_destroy(&copy);
+	reftable_record_release(&copy);
 }
 
 static void test_varint_roundtrip(void)
@@ -95,11 +96,11 @@ static void test_reftable_ref_record_roundtrip(void)
 	int i = 0;
 
 	for (i = REFTABLE_REF_DELETION; i < REFTABLE_NR_REF_VALUETYPES; i++) {
-		struct reftable_ref_record in = { NULL };
-		struct reftable_ref_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_REF,
+		};
+		struct reftable_record out = { .type = BLOCK_TYPE_REF };
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_record rec = { NULL };
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
@@ -108,48 +109,46 @@ static void test_reftable_ref_record_roundtrip(void)
 
 		int n, m;
 
-		in.value_type = i;
+		in.u.ref.value_type = i;
 		switch (i) {
 		case REFTABLE_REF_DELETION:
 			break;
 		case REFTABLE_REF_VAL1:
-			in.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val1, 1);
+			in.u.ref.value.val1 = reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val1, 1);
 			break;
 		case REFTABLE_REF_VAL2:
-			in.value.val2.value = reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.value, 1);
-			in.value.val2.target_value =
+			in.u.ref.value.val2.value =
+				reftable_malloc(GIT_SHA1_RAWSZ);
+			set_hash(in.u.ref.value.val2.value, 1);
+			in.u.ref.value.val2.target_value =
 				reftable_malloc(GIT_SHA1_RAWSZ);
-			set_hash(in.value.val2.target_value, 2);
+			set_hash(in.u.ref.value.val2.target_value, 2);
 			break;
 		case REFTABLE_REF_SYMREF:
-			in.value.symref = xstrdup("target");
+			in.u.ref.value.symref = xstrdup("target");
 			break;
 		}
-		in.refname = xstrdup("refs/heads/master");
+		in.u.ref.refname = xstrdup("refs/heads/master");
 
-		reftable_record_from_ref(&rec, &in);
-		test_copy(&rec);
+		test_copy(&in);
 
-		EXPECT(reftable_record_val_type(&rec) == i);
+		EXPECT(reftable_record_val_type(&in) == i);
 
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
 
 		/* decode into a non-zero reftable_record to test for leaks. */
-
-		reftable_record_from_ref(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, i, dest,
-					   GIT_SHA1_RAWSZ);
+		m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_ref_record_equal(&in, &out, GIT_SHA1_RAWSZ));
-		reftable_record_release(&rec_out);
+		EXPECT(reftable_ref_record_equal(&in.u.ref, &out.u.ref,
+						 GIT_SHA1_RAWSZ));
+		reftable_record_release(&in);
 
 		strbuf_release(&key);
-		reftable_ref_record_release(&in);
+		reftable_record_release(&out);
 	}
 }
 
@@ -202,7 +201,7 @@ static void test_reftable_log_record_roundtrip(void)
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
-		struct reftable_record rec = { NULL };
+		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
@@ -210,23 +209,25 @@ static void test_reftable_log_record_roundtrip(void)
 			.len = sizeof(buffer),
 		};
 		/* populate out, to check for leaks. */
-		struct reftable_log_record out = {
-			.refname = xstrdup("old name"),
-			.value_type = REFTABLE_LOG_UPDATE,
-			.value = {
-				.update = {
-					.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
-					.name = xstrdup("old name"),
-					.email = xstrdup("old@email"),
-					.message = xstrdup("old message"),
+		struct reftable_record out = {
+			.type = BLOCK_TYPE_LOG,
+			.u.log = {
+				.refname = xstrdup("old name"),
+				.value_type = REFTABLE_LOG_UPDATE,
+				.value = {
+					.update = {
+						.new_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.old_hash = reftable_calloc(GIT_SHA1_RAWSZ),
+						.name = xstrdup("old name"),
+						.email = xstrdup("old@email"),
+						.message = xstrdup("old message"),
+					},
 				},
 			},
 		};
-		struct reftable_record rec_out = { NULL };
 		int n, m, valtype;
 
-		reftable_record_from_log(&rec, &in[i]);
+		rec.u.log = in[i];
 
 		test_copy(&rec);
 
@@ -234,16 +235,16 @@ static void test_reftable_log_record_roundtrip(void)
 
 		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n >= 0);
-		reftable_record_from_log(&rec_out, &out);
 		valtype = reftable_record_val_type(&rec);
-		m = reftable_record_decode(&rec_out, key, valtype, dest,
+		m = reftable_record_decode(&out, key, valtype, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_log_record_equal(&in[i], &out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_log_record_equal(&in[i], &out.u.log,
+						 GIT_SHA1_RAWSZ));
 		reftable_log_record_release(&in[i]);
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
@@ -311,41 +312,43 @@ static void test_reftable_obj_record_roundtrip(void)
 					       } };
 	int i = 0;
 	for (i = 0; i < ARRAY_SIZE(recs); i++) {
-		struct reftable_obj_record in = recs[i];
 		uint8_t buffer[1024] = { 0 };
 		struct string_view dest = {
 			.buf = buffer,
 			.len = sizeof(buffer),
 		};
-		struct reftable_record rec = { NULL };
+		struct reftable_record in = {
+			.type = BLOCK_TYPE_OBJ,
+			.u.obj = recs[i],
+		};
 		struct strbuf key = STRBUF_INIT;
-		struct reftable_obj_record out = { NULL };
-		struct reftable_record rec_out = { NULL };
+		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
 		int n, m;
 		uint8_t extra;
 
-		reftable_record_from_obj(&rec, &in);
-		test_copy(&rec);
-		reftable_record_key(&rec, &key);
-		n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+		test_copy(&in);
+		reftable_record_key(&in, &key);
+		n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 		EXPECT(n > 0);
-		extra = reftable_record_val_type(&rec);
-		reftable_record_from_obj(&rec_out, &out);
-		m = reftable_record_decode(&rec_out, key, extra, dest,
+		extra = reftable_record_val_type(&in);
+		m = reftable_record_decode(&out, key, extra, dest,
 					   GIT_SHA1_RAWSZ);
 		EXPECT(n == m);
 
-		EXPECT(reftable_record_equal(&rec, &rec_out, GIT_SHA1_RAWSZ));
+		EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 		strbuf_release(&key);
-		reftable_record_release(&rec_out);
+		reftable_record_release(&out);
 	}
 }
 
 static void test_reftable_index_record_roundtrip(void)
 {
-	struct reftable_index_record in = {
-		.offset = 42,
-		.last_key = STRBUF_INIT,
+	struct reftable_record in = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = {
+			.offset = 42,
+			.last_key = STRBUF_INIT,
+		},
 	};
 	uint8_t buffer[1024] = { 0 };
 	struct string_view dest = {
@@ -353,31 +356,30 @@ static void test_reftable_index_record_roundtrip(void)
 		.len = sizeof(buffer),
 	};
 	struct strbuf key = STRBUF_INIT;
-	struct reftable_record rec = { NULL };
-	struct reftable_index_record out = { .last_key = STRBUF_INIT };
-	struct reftable_record out_rec = { NULL };
+	struct reftable_record out = {
+		.type = BLOCK_TYPE_INDEX,
+		.u.idx = { .last_key = STRBUF_INIT },
+	};
 	int n, m;
 	uint8_t extra;
 
-	strbuf_addstr(&in.last_key, "refs/heads/master");
-	reftable_record_from_index(&rec, &in);
-	reftable_record_key(&rec, &key);
-	test_copy(&rec);
+	strbuf_addstr(&in.u.idx.last_key, "refs/heads/master");
+	reftable_record_key(&in, &key);
+	test_copy(&in);
 
-	EXPECT(0 == strbuf_cmp(&key, &in.last_key));
-	n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ);
+	EXPECT(0 == strbuf_cmp(&key, &in.u.idx.last_key));
+	n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ);
 	EXPECT(n > 0);
 
-	extra = reftable_record_val_type(&rec);
-	reftable_record_from_index(&out_rec, &out);
-	m = reftable_record_decode(&out_rec, key, extra, dest, GIT_SHA1_RAWSZ);
+	extra = reftable_record_val_type(&in);
+	m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ);
 	EXPECT(m == n);
 
-	EXPECT(reftable_record_equal(&rec, &out_rec, GIT_SHA1_RAWSZ));
+	EXPECT(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ));
 
-	reftable_record_release(&out_rec);
+	reftable_record_release(&out);
 	strbuf_release(&key);
-	strbuf_release(&in.last_key);
+	strbuf_release(&in.u.idx.last_key);
 }
 
 int record_test_main(int argc, const char *argv[])
diff --git a/reftable/writer.c b/reftable/writer.c
index 83a23daf60b..96a5155ea77 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -256,8 +256,10 @@ done:
 int reftable_writer_add_ref(struct reftable_writer *w,
 			    struct reftable_ref_record *ref)
 {
-	struct reftable_record rec = { NULL };
-	struct reftable_ref_record copy = *ref;
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_REF,
+		.u.ref = *ref,
+	};
 	int err = 0;
 
 	if (ref->refname == NULL)
@@ -266,8 +268,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 	    ref->update_index > w->max_update_index)
 		return REFTABLE_API_ERROR;
 
-	reftable_record_from_ref(&rec, &copy);
-	copy.update_index -= w->min_update_index;
+	rec.u.ref.update_index -= w->min_update_index;
 
 	err = writer_add_record(w, &rec);
 	if (err < 0)
@@ -306,7 +307,10 @@ int reftable_writer_add_refs(struct reftable_writer *w,
 static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 					    struct reftable_log_record *log)
 {
-	struct reftable_record rec = { NULL };
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_LOG,
+		.u.log = *log,
+	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
 		int err = writer_finish_public_section(w);
@@ -317,7 +321,6 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 	w->next -= w->pending_padding;
 	w->pending_padding = 0;
 
-	reftable_record_from_log(&rec, log);
 	return writer_add_record(w, &rec);
 }
 
@@ -398,8 +401,10 @@ static int writer_finish_section(struct reftable_writer *w)
 		w->index_len = 0;
 		w->index_cap = 0;
 		for (i = 0; i < idx_len; i++) {
-			struct reftable_record rec = { NULL };
-			reftable_record_from_index(&rec, idx + i);
+			struct reftable_record rec = {
+				.type = BLOCK_TYPE_INDEX,
+				.u.idx = idx[i],
+			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
 			}
@@ -467,17 +472,18 @@ static void write_object_record(void *void_arg, void *key)
 {
 	struct write_record_arg *arg = void_arg;
 	struct obj_index_tree_node *entry = key;
-	struct reftable_obj_record obj_rec = {
-		.hash_prefix = (uint8_t *)entry->hash.buf,
-		.hash_prefix_len = arg->w->stats.object_id_len,
-		.offsets = entry->offsets,
-		.offset_len = entry->offset_len,
+	struct reftable_record rec = {
+		.type = BLOCK_TYPE_OBJ,
+		.u.obj = {
+			.hash_prefix = (uint8_t *)entry->hash.buf,
+			.hash_prefix_len = arg->w->stats.object_id_len,
+			.offsets = entry->offsets,
+			.offset_len = entry->offset_len,
+		}
 	};
-	struct reftable_record rec = { NULL };
 	if (arg->err < 0)
 		goto done;
 
-	reftable_record_from_obj(&rec, &obj_rec);
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
@@ -490,7 +496,7 @@ static void write_object_record(void *void_arg, void *key)
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 	if (arg->err == 0)
 		goto done;
-	obj_rec.offset_len = 0;
+	rec.u.obj.offset_len = 0;
 	arg->err = block_writer_add(arg->w->block_writer, &rec);
 
 	/* Should be able to write into a fresh block. */
-- 
gitgitgadget

[PATCH v7 15/16] reftable: add print functions to the record types

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

This isn't used per se, but it is useful for debugging, especially
Windows CI failures.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/record.c      | 88 +++++++++++++++++++++++++++++++++++-------
 reftable/record.h      |  4 ++
 reftable/record_test.c | 20 +++++++++-
 3 files changed, 96 insertions(+), 16 deletions(-)

diff --git a/reftable/record.c b/reftable/record.c
index 238ff882aab..c57bac0e96a 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -269,8 +269,8 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
 	}
 }
 
-void reftable_ref_record_print(const struct reftable_ref_record *ref,
-			       uint32_t hash_id)
+static void reftable_ref_record_print_sz(const struct reftable_ref_record *ref,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1];
 	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
@@ -279,14 +279,14 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 		printf("=> %s", ref->value.symref);
 		break;
 	case REFTABLE_REF_VAL2:
-		hex_format(hex, ref->value.val2.value, hash_size(hash_id));
+		hex_format(hex, ref->value.val2.value, hash_size);
 		printf("val 2 %s", hex);
 		hex_format(hex, ref->value.val2.target_value,
-			   hash_size(hash_id));
+			   hash_size);
 		printf("(T %s)", hex);
 		break;
 	case REFTABLE_REF_VAL1:
-		hex_format(hex, ref->value.val1, hash_size(hash_id));
+		hex_format(hex, ref->value.val1, hash_size);
 		printf("val 1 %s", hex);
 		break;
 	case REFTABLE_REF_DELETION:
@@ -296,6 +296,11 @@ void reftable_ref_record_print(const struct reftable_ref_record *ref,
 	printf("}\n");
 }
 
+void reftable_ref_record_print(const struct reftable_ref_record *ref,
+			       uint32_t hash_id) {
+	reftable_ref_record_print_sz(ref, hash_size(hash_id));
+}
+
 static void reftable_ref_record_release_void(void *rec)
 {
 	reftable_ref_record_release(rec);
@@ -456,6 +461,13 @@ static int reftable_ref_record_equal_void(const void *a,
 	return reftable_ref_record_equal(ra, rb, hash_size);
 }
 
+static void reftable_ref_record_print_void(const void *rec,
+					   int hash_size)
+{
+	reftable_ref_record_print_sz(
+		(struct reftable_ref_record *) rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.key = &reftable_ref_record_key,
 	.type = BLOCK_TYPE_REF,
@@ -466,6 +478,7 @@ static struct reftable_record_vtable reftable_ref_record_vtable = {
 	.release = &reftable_ref_record_release_void,
 	.is_deletion = &reftable_ref_record_is_deletion_void,
 	.equal = &reftable_ref_record_equal_void,
+	.print = &reftable_ref_record_print_void,
 };
 
 static void reftable_obj_record_key(const void *r, struct strbuf *dest)
@@ -484,6 +497,21 @@ static void reftable_obj_record_release(void *rec)
 	memset(obj, 0, sizeof(struct reftable_obj_record));
 }
 
+static void reftable_obj_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_obj_record *obj = rec;
+	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
+	struct strbuf offset_str = STRBUF_INIT;
+	int i;
+
+	for (i = 0; i < obj->offset_len; i++)
+		strbuf_addf(&offset_str, "%"PRIu64 " ", obj->offsets[i]);
+	hex_format(hex, obj->hash_prefix, obj->hash_prefix_len);
+	printf("prefix %s (len %d), offsets [%s]\n",
+	       hex, obj->hash_prefix_len, offset_str.buf);
+	strbuf_release(&offset_str);
+}
+
 static void reftable_obj_record_copy_from(void *rec, const void *src_rec,
 					  int hash_size)
 {
@@ -627,31 +655,41 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
 	.release = &reftable_obj_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_obj_record_equal_void,
+	.print = &reftable_obj_record_print,
 };
 
-void reftable_log_record_print(struct reftable_log_record *log,
-			       uint32_t hash_id)
+static void reftable_log_record_print_sz(struct reftable_log_record *log,
+					 int hash_size)
 {
 	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
 
 	switch (log->value_type) {
 	case REFTABLE_LOG_DELETION:
-		printf("log{%s(%" PRIu64 ") delete", log->refname,
+		printf("log{%s(%"PRIu64 ") delete\n", log->refname,
 		       log->update_index);
 		break;
 	case REFTABLE_LOG_UPDATE:
-		printf("log{%s(%" PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
-		       log->refname, log->update_index, log->value.update.name,
-		       log->value.update.email, log->value.update.time,
+		printf("log{%s(%"PRIu64 ") %s <%s> %" PRIu64 " %04d\n",
+		       log->refname, log->update_index,
+		       log->value.update.name ? log->value.update.name : "",
+		       log->value.update.email ? log->value.update.email : "",
+		       log->value.update.time,
 		       log->value.update.tz_offset);
-		hex_format(hex, log->value.update.old_hash, hash_size(hash_id));
+		hex_format(hex, log->value.update.old_hash, hash_size);
 		printf("%s => ", hex);
-		hex_format(hex, log->value.update.new_hash, hash_size(hash_id));
-		printf("%s\n\n%s\n}\n", hex, log->value.update.message);
+		hex_format(hex, log->value.update.new_hash, hash_size);
+		printf("%s\n\n%s\n}\n", hex,
+		       log->value.update.message ? log->value.update.message : "");
 		break;
 	}
 }
 
+void reftable_log_record_print(struct reftable_log_record *log,
+				      uint32_t hash_id)
+{
+	reftable_log_record_print_sz(log, hash_size(hash_id));
+}
+
 static void reftable_log_record_key(const void *r, struct strbuf *dest)
 {
 	const struct reftable_log_record *rec =
@@ -969,6 +1007,11 @@ static int reftable_log_record_is_deletion_void(const void *p)
 		(const struct reftable_log_record *)p);
 }
 
+static void reftable_log_record_print_void(const void *rec, int hash_size)
+{
+	reftable_log_record_print_sz((struct reftable_log_record*)rec, hash_size);
+}
+
 static struct reftable_record_vtable reftable_log_record_vtable = {
 	.key = &reftable_log_record_key,
 	.type = BLOCK_TYPE_LOG,
@@ -978,7 +1021,8 @@ static struct reftable_record_vtable reftable_log_record_vtable = {
 	.decode = &reftable_log_record_decode,
 	.release = &reftable_log_record_release_void,
 	.is_deletion = &reftable_log_record_is_deletion_void,
-	.equal = &reftable_log_record_equal_void
+	.equal = &reftable_log_record_equal_void,
+	.print = &reftable_log_record_print_void,
 };
 
 static void reftable_index_record_key(const void *r, struct strbuf *dest)
@@ -1054,6 +1098,13 @@ static int reftable_index_record_equal(const void *a, const void *b, int hash_si
 	return ia->offset == ib->offset && !strbuf_cmp(&ia->last_key, &ib->last_key);
 }
 
+static void reftable_index_record_print(const void *rec, int hash_size)
+{
+	const struct reftable_index_record *idx = rec;
+	/* TODO: escape null chars? */
+	printf("\"%s\" %" PRIu64 "\n", idx->last_key.buf, idx->offset);
+}
+
 static struct reftable_record_vtable reftable_index_record_vtable = {
 	.key = &reftable_index_record_key,
 	.type = BLOCK_TYPE_INDEX,
@@ -1064,6 +1115,7 @@ static struct reftable_record_vtable reftable_index_record_vtable = {
 	.release = &reftable_index_record_release,
 	.is_deletion = &not_a_deletion,
 	.equal = &reftable_index_record_equal,
+	.print = &reftable_index_record_print,
 };
 
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest)
@@ -1251,3 +1303,9 @@ struct reftable_record reftable_new_record(uint8_t type)
 	}
 	return clean;
 }
+
+void reftable_record_print(struct reftable_record *rec, int hash_size)
+{
+	printf("'%c': ", rec->type);
+	reftable_record_vtable(rec)->print(reftable_record_data(rec), hash_size);
+}
diff --git a/reftable/record.h b/reftable/record.h
index 010a322e901..fd80cd451d5 100644
--- a/reftable/record.h
+++ b/reftable/record.h
@@ -61,6 +61,9 @@ struct reftable_record_vtable {
 
 	/* Are two records equal? This assumes they have the same type. Returns 0 for non-equal. */
 	int (*equal)(const void *a, const void *b, int hash_size);
+
+	/* Print on stdout, for debugging. */
+	void (*print)(const void *rec, int hash_size);
 };
 
 /* returns true for recognized block types. Block start with the block type. */
@@ -112,6 +115,7 @@ struct reftable_record {
 
 /* see struct record_vtable */
 int reftable_record_equal(struct reftable_record *a, struct reftable_record *b, int hash_size);
+void reftable_record_print(struct reftable_record *rec, int hash_size);
 void reftable_record_key(struct reftable_record *rec, struct strbuf *dest);
 uint8_t reftable_record_type(struct reftable_record *rec);
 void reftable_record_copy_from(struct reftable_record *rec,
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 4a025c48104..847abfa2b04 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -23,6 +23,10 @@ static void test_copy(struct reftable_record *rec)
 	/* do it twice to catch memory leaks */
 	reftable_record_copy_from(&copy, rec, GIT_SHA1_RAWSZ);
 	EXPECT(reftable_record_equal(rec, &copy, GIT_SHA1_RAWSZ));
+
+	puts("testing print coverage:\n");
+	reftable_record_print(&copy, GIT_SHA1_RAWSZ);
+
 	reftable_record_release(&copy);
 }
 
@@ -175,7 +179,7 @@ static void test_reftable_log_record_equal(void)
 static void test_reftable_log_record_roundtrip(void)
 {
 	int i;
-	struct reftable_log_record in[2] = {
+	struct reftable_log_record in[] = {
 		{
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 42,
@@ -196,10 +200,24 @@ static void test_reftable_log_record_roundtrip(void)
 			.refname = xstrdup("refs/heads/master"),
 			.update_index = 22,
 			.value_type = REFTABLE_LOG_DELETION,
+		},
+		{
+			/* make sure we handle optional fields correctly. */
+			.refname = xstrdup("branch"),
+			.update_index = 33,
+			.value_type = REFTABLE_LOG_UPDATE,
+			.value = {
+				.update = {
+					.old_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+					.new_hash = reftable_malloc(GIT_SHA1_RAWSZ),
+				},
+			},
 		}
 	};
 	set_test_hash(in[0].value.update.new_hash, 1);
 	set_test_hash(in[0].value.update.old_hash, 2);
+	set_test_hash(in[2].value.update.new_hash, 3);
+	set_test_hash(in[2].value.update.old_hash, 4);
 	for (i = 0; i < ARRAY_SIZE(in); i++) {
 		struct reftable_record rec = { .type = BLOCK_TYPE_LOG };
 		struct strbuf key = STRBUF_INIT;
-- 
gitgitgadget

[PATCH v7 16/16] reftable: rename typ to type

[Han-Wen Nienhuys via GitGitGadget]

From: Han-Wen Nienhuys <hanwen@google.com>

The 'typ' variable name betrays the Go origins of this code (in Go,
'type' is a reserved word). Since we now have parted ways with the Go
code, simply use 'type'.

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
---
 reftable/block.c  | 10 ++++----
 reftable/merged.c |  6 ++---
 reftable/merged.h |  2 +-
 reftable/reader.c | 58 +++++++++++++++++++++++------------------------
 reftable/record.c |  4 ++--
 reftable/writer.c | 32 +++++++++++++-------------
 6 files changed, 56 insertions(+), 56 deletions(-)

diff --git a/reftable/block.c b/reftable/block.c
index 80ad10bb794..6c1f70ba563 100644
--- a/reftable/block.c
+++ b/reftable/block.c
@@ -68,14 +68,14 @@ static int block_writer_register_restart(struct block_writer *w, int n,
 	return 0;
 }
 
-void block_writer_init(struct block_writer *bw, uint8_t typ, uint8_t *buf,
+void block_writer_init(struct block_writer *bw, uint8_t type, uint8_t *buf,
 		       uint32_t block_size, uint32_t header_off, int hash_size)
 {
 	bw->buf = buf;
 	bw->hash_size = hash_size;
 	bw->block_size = block_size;
 	bw->header_off = header_off;
-	bw->buf[header_off] = typ;
+	bw->buf[header_off] = type;
 	bw->next = header_off + 4;
 	bw->restart_interval = 16;
 	bw->entries = 0;
@@ -186,7 +186,7 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 		      int hash_size)
 {
 	uint32_t full_block_size = table_block_size;
-	uint8_t typ = block->data[header_off];
+	uint8_t type = block->data[header_off];
 	uint32_t sz = get_be24(block->data + header_off + 1);
 
 	int err = 0;
@@ -195,12 +195,12 @@ int block_reader_init(struct block_reader *br, struct reftable_block *block,
 	uint8_t *restart_bytes = NULL;
 	uint8_t *uncompressed = NULL;
 
-	if (!reftable_is_block_type(typ)) {
+	if (!reftable_is_block_type(type)) {
 		err =  REFTABLE_FORMAT_ERROR;
 		goto done;
 	}
 
-	if (typ == BLOCK_TYPE_LOG) {
+	if (type == BLOCK_TYPE_LOG) {
 		int block_header_skip = 4 + header_off;
 		uLongf dst_len = sz - block_header_skip; /* total size of dest
 							    buffer. */
diff --git a/reftable/merged.c b/reftable/merged.c
index 82511762d2a..b5710d5f7a9 100644
--- a/reftable/merged.c
+++ b/reftable/merged.c
@@ -22,7 +22,7 @@ static int merged_iter_init(struct merged_iter *mi)
 {
 	int i = 0;
 	for (i = 0; i < mi->stack_len; i++) {
-		struct reftable_record rec = reftable_new_record(mi->typ);
+		struct reftable_record rec = reftable_new_record(mi->type);
 		int err = iterator_next(&mi->stack[i], &rec);
 		if (err < 0) {
 			return err;
@@ -58,7 +58,7 @@ static int merged_iter_advance_nonnull_subiter(struct merged_iter *mi,
 					       size_t idx)
 {
 	struct pq_entry e = {
-		.rec = reftable_new_record(mi->typ),
+		.rec = reftable_new_record(mi->type),
 		.index = idx,
 	};
 	int err = iterator_next(&mi->stack[idx], &e.rec);
@@ -245,7 +245,7 @@ static int merged_table_seek_record(struct reftable_merged_table *mt,
 		sizeof(struct reftable_iterator) * mt->stack_len);
 	struct merged_iter merged = {
 		.stack = iters,
-		.typ = reftable_record_type(rec),
+		.type = reftable_record_type(rec),
 		.hash_id = mt->hash_id,
 		.suppress_deletions = mt->suppress_deletions,
 	};
diff --git a/reftable/merged.h b/reftable/merged.h
index 7d9f95d27ed..63f288ff8d6 100644
--- a/reftable/merged.h
+++ b/reftable/merged.h
@@ -28,7 +28,7 @@ struct merged_iter {
 	struct reftable_iterator *stack;
 	uint32_t hash_id;
 	size_t stack_len;
-	uint8_t typ;
+	uint8_t type;
 	int suppress_deletions;
 	struct merged_iter_pqueue pq;
 };
diff --git a/reftable/reader.c b/reftable/reader.c
index 565cda8d5a2..1ecd23bba36 100644
--- a/reftable/reader.c
+++ b/reftable/reader.c
@@ -43,9 +43,9 @@ void block_source_close(struct reftable_block_source *source)
 }
 
 static struct reftable_reader_offsets *
-reader_offsets_for(struct reftable_reader *r, uint8_t typ)
+reader_offsets_for(struct reftable_reader *r, uint8_t type)
 {
-	switch (typ) {
+	switch (type) {
 	case BLOCK_TYPE_REF:
 		return &r->ref_offsets;
 	case BLOCK_TYPE_LOG:
@@ -84,7 +84,7 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,
 			uint8_t *header)
 {
 	uint8_t *f = footer;
-	uint8_t first_block_typ;
+	uint8_t first_block_type;
 	int err = 0;
 	uint32_t computed_crc;
 	uint32_t file_crc;
@@ -149,10 +149,10 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer,
 		goto done;
 	}
 
-	first_block_typ = header[header_size(r->version)];
-	r->ref_offsets.is_present = (first_block_typ == BLOCK_TYPE_REF);
+	first_block_type = header[header_size(r->version)];
+	r->ref_offsets.is_present = (first_block_type == BLOCK_TYPE_REF);
 	r->ref_offsets.offset = 0;
-	r->log_offsets.is_present = (first_block_typ == BLOCK_TYPE_LOG ||
+	r->log_offsets.is_present = (first_block_type == BLOCK_TYPE_LOG ||
 				     r->log_offsets.offset > 0);
 	r->obj_offsets.is_present = r->obj_offsets.offset > 0;
 	err = 0;
@@ -214,7 +214,7 @@ done:
 
 struct table_iter {
 	struct reftable_reader *r;
-	uint8_t typ;
+	uint8_t type;
 	uint64_t block_off;
 	struct block_iter bi;
 	int is_finished;
@@ -228,7 +228,7 @@ static void table_iter_copy_from(struct table_iter *dest,
 				 struct table_iter *src)
 {
 	dest->r = src->r;
-	dest->typ = src->typ;
+	dest->type = src->type;
 	dest->block_off = src->block_off;
 	dest->is_finished = src->is_finished;
 	block_iter_copy_from(&dest->bi, &src->bi);
@@ -257,7 +257,7 @@ static void table_iter_block_done(struct table_iter *ti)
 	ti->bi.next_off = 0;
 }
 
-static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,
+static int32_t extract_block_size(uint8_t *data, uint8_t *type, uint64_t off,
 				  int version)
 {
 	int32_t result = 0;
@@ -266,20 +266,20 @@ static int32_t extract_block_size(uint8_t *data, uint8_t *typ, uint64_t off,
 		data += header_size(version);
 	}
 
-	*typ = data[0];
-	if (reftable_is_block_type(*typ)) {
+	*type = data[0];
+	if (reftable_is_block_type(*type)) {
 		result = get_be24(data + 1);
 	}
 	return result;
 }
 
 int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
-			     uint64_t next_off, uint8_t want_typ)
+			     uint64_t next_off, uint8_t want_type)
 {
 	int32_t guess_block_size = r->block_size ? r->block_size :
 							 DEFAULT_BLOCK_SIZE;
 	struct reftable_block block = { NULL };
-	uint8_t block_typ = 0;
+	uint8_t block_type = 0;
 	int err = 0;
 	uint32_t header_off = next_off ? 0 : header_size(r->version);
 	int32_t block_size = 0;
@@ -291,13 +291,13 @@ int reader_init_block_reader(struct reftable_reader *r, struct block_reader *br,
 	if (err < 0)
 		goto done;
 
-	block_size = extract_block_size(block.data, &block_typ, next_off,
+	block_size = extract_block_size(block.data, &block_type, next_off,
 					r->version);
 	if (block_size < 0) {
 		err = block_size;
 		goto done;
 	}
-	if (want_typ != BLOCK_TYPE_ANY && block_typ != want_typ) {
+	if (want_type != BLOCK_TYPE_ANY && block_type != want_type) {
 		err = 1;
 		goto done;
 	}
@@ -326,10 +326,10 @@ static int table_iter_next_block(struct table_iter *dest,
 	int err = 0;
 
 	dest->r = src->r;
-	dest->typ = src->typ;
+	dest->type = src->type;
 	dest->block_off = next_block_off;
 
-	err = reader_init_block_reader(src->r, &br, next_block_off, src->typ);
+	err = reader_init_block_reader(src->r, &br, next_block_off, src->type);
 	if (err > 0) {
 		dest->is_finished = 1;
 		return 1;
@@ -349,7 +349,7 @@ static int table_iter_next_block(struct table_iter *dest,
 
 static int table_iter_next(struct table_iter *ti, struct reftable_record *rec)
 {
-	if (reftable_record_type(rec) != ti->typ)
+	if (reftable_record_type(rec) != ti->type)
 		return REFTABLE_API_ERROR;
 
 	while (1) {
@@ -404,38 +404,38 @@ static void iterator_from_table_iter(struct reftable_iterator *it,
 
 static int reader_table_iter_at(struct reftable_reader *r,
 				struct table_iter *ti, uint64_t off,
-				uint8_t typ)
+				uint8_t type)
 {
 	struct block_reader br = { 0 };
 	struct block_reader *brp = NULL;
 
-	int err = reader_init_block_reader(r, &br, off, typ);
+	int err = reader_init_block_reader(r, &br, off, type);
 	if (err != 0)
 		return err;
 
 	brp = reftable_malloc(sizeof(struct block_reader));
 	*brp = br;
 	ti->r = r;
-	ti->typ = block_reader_type(brp);
+	ti->type = block_reader_type(brp);
 	ti->block_off = off;
 	block_reader_start(brp, &ti->bi);
 	return 0;
 }
 
 static int reader_start(struct reftable_reader *r, struct table_iter *ti,
-			uint8_t typ, int index)
+			uint8_t type, int index)
 {
-	struct reftable_reader_offsets *offs = reader_offsets_for(r, typ);
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, type);
 	uint64_t off = offs->offset;
 	if (index) {
 		off = offs->index_offset;
 		if (off == 0) {
 			return 1;
 		}
-		typ = BLOCK_TYPE_INDEX;
+		type = BLOCK_TYPE_INDEX;
 	}
 
-	return reader_table_iter_at(r, ti, off, typ);
+	return reader_table_iter_at(r, ti, off, type);
 }
 
 static int reader_seek_linear(struct reftable_reader *r, struct table_iter *ti,
@@ -522,12 +522,12 @@ static int reader_seek_indexed(struct reftable_reader *r,
 		if (err < 0)
 			goto done;
 
-		if (next.typ == reftable_record_type(rec)) {
+		if (next.type == reftable_record_type(rec)) {
 			err = 0;
 			break;
 		}
 
-		if (next.typ != BLOCK_TYPE_INDEX) {
+		if (next.type != BLOCK_TYPE_INDEX) {
 			err = REFTABLE_FORMAT_ERROR;
 			break;
 		}
@@ -582,9 +582,9 @@ static int reader_seek_internal(struct reftable_reader *r,
 static int reader_seek(struct reftable_reader *r, struct reftable_iterator *it,
 		       struct reftable_record *rec)
 {
-	uint8_t typ = reftable_record_type(rec);
+	uint8_t type = reftable_record_type(rec);
 
-	struct reftable_reader_offsets *offs = reader_offsets_for(r, typ);
+	struct reftable_reader_offsets *offs = reader_offsets_for(r, type);
 	if (!offs->is_present) {
 		iterator_set_empty(it);
 		return 0;
diff --git a/reftable/record.c b/reftable/record.c
index c57bac0e96a..8330b7a2589 100644
--- a/reftable/record.c
+++ b/reftable/record.c
@@ -78,9 +78,9 @@ int put_var_int(struct string_view *dest, uint64_t val)
 	return n;
 }
 
-int reftable_is_block_type(uint8_t typ)
+int reftable_is_block_type(uint8_t type)
 {
-	switch (typ) {
+	switch (type) {
 	case BLOCK_TYPE_REF:
 	case BLOCK_TYPE_LOG:
 	case BLOCK_TYPE_OBJ:
diff --git a/reftable/writer.c b/reftable/writer.c
index 96a5155ea77..5b768438625 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -26,9 +26,9 @@ static void writer_clear_index(struct reftable_writer *w);
 static int writer_finish_public_section(struct reftable_writer *w);
 
 static struct reftable_block_stats *
-writer_reftable_block_stats(struct reftable_writer *w, uint8_t typ)
+writer_reftable_block_stats(struct reftable_writer *w, uint8_t type)
 {
-	switch (typ) {
+	switch (type) {
 	case 'r':
 		return &w->stats.ref_stats;
 	case 'o':
@@ -102,7 +102,7 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest)
 	return header_size(writer_version(w));
 }
 
-static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)
+static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t type)
 {
 	int block_start = 0;
 	if (w->next == 0) {
@@ -110,7 +110,7 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ)
 	}
 
 	strbuf_release(&w->last_key);
-	block_writer_init(&w->block_writer_data, typ, w->block,
+	block_writer_init(&w->block_writer_data, type, w->block,
 			  w->opts.block_size, block_start,
 			  hash_size(w->opts.hash_id));
 	w->block_writer = &w->block_writer_data;
@@ -375,7 +375,7 @@ int reftable_writer_add_logs(struct reftable_writer *w,
 
 static int writer_finish_section(struct reftable_writer *w)
 {
-	uint8_t typ = block_writer_type(w->block_writer);
+	uint8_t type = block_writer_type(w->block_writer);
 	uint64_t index_start = 0;
 	int max_level = 0;
 	int threshold = w->opts.unpadded ? 1 : 3;
@@ -434,7 +434,7 @@ static int writer_finish_section(struct reftable_writer *w)
 	if (err < 0)
 		return err;
 
-	bstats = writer_reftable_block_stats(w, typ);
+	bstats = writer_reftable_block_stats(w, type);
 	bstats->index_blocks = w->stats.idx_stats.blocks - before_blocks;
 	bstats->index_offset = index_start;
 	bstats->max_index_level = max_level;
@@ -536,17 +536,17 @@ static int writer_dump_object_index(struct reftable_writer *w)
 
 static int writer_finish_public_section(struct reftable_writer *w)
 {
-	uint8_t typ = 0;
+	uint8_t type = 0;
 	int err = 0;
 
 	if (w->block_writer == NULL)
 		return 0;
 
-	typ = block_writer_type(w->block_writer);
+	type = block_writer_type(w->block_writer);
 	err = writer_finish_section(w);
 	if (err < 0)
 		return err;
-	if (typ == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&
+	if (type == BLOCK_TYPE_REF && !w->opts.skip_index_objects &&
 	    w->stats.ref_stats.index_blocks > 0) {
 		err = writer_dump_object_index(w);
 		if (err < 0)
@@ -630,10 +630,10 @@ static const int debug = 0;
 
 static int writer_flush_nonempty_block(struct reftable_writer *w)
 {
-	uint8_t typ = block_writer_type(w->block_writer);
+	uint8_t type = block_writer_type(w->block_writer);
 	struct reftable_block_stats *bstats =
-		writer_reftable_block_stats(w, typ);
-	uint64_t block_typ_off = (bstats->blocks == 0) ? w->next : 0;
+		writer_reftable_block_stats(w, type);
+	uint64_t block_type_off = (bstats->blocks == 0) ? w->next : 0;
 	int raw_bytes = block_writer_finish(w->block_writer);
 	int padding = 0;
 	int err = 0;
@@ -641,12 +641,12 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)
 	if (raw_bytes < 0)
 		return raw_bytes;
 
-	if (!w->opts.unpadded && typ != BLOCK_TYPE_LOG) {
+	if (!w->opts.unpadded && type != BLOCK_TYPE_LOG) {
 		padding = w->opts.block_size - raw_bytes;
 	}
 
-	if (block_typ_off > 0) {
-		bstats->offset = block_typ_off;
+	if (block_type_off > 0) {
+		bstats->offset = block_type_off;
 	}
 
 	bstats->entries += w->block_writer->entries;
@@ -655,7 +655,7 @@ static int writer_flush_nonempty_block(struct reftable_writer *w)
 	w->stats.blocks++;
 
 	if (debug) {
-		fprintf(stderr, "block %c off %" PRIu64 " sz %d (%d)\n", typ,
+		fprintf(stderr, "block %c off %" PRIu64 " sz %d (%d)\n", type,
 			w->next, raw_bytes,
 			get_be24(w->block + w->block_writer->header_off + 1));
 	}
-- 
gitgitgadget

Re: [PATCH v7 01/16] reftable: fix OOB stack write in print functions

[Ævar Arnfjörð Bjarmason]

On Mon, Jan 24 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
>
> Remove /* BUG */ comment, whose meaning has escaped my memory.
>
> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com>
> ---
>  reftable/record.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/reftable/record.c b/reftable/record.c
> index 6a5dac32dc6..1f236b47d1c 100644
> --- a/reftable/record.c
> +++ b/reftable/record.c
> @@ -254,7 +254,7 @@ static void hex_format(char *dest, uint8_t *src, int hash_size)
>  void reftable_ref_record_print(struct reftable_ref_record *ref,
>  			       uint32_t hash_id)
>  {
> -	char hex[2 * GIT_SHA256_RAWSZ + 1] = { 0 }; /* BUG */
> +	char hex[GIT_MAX_HEXSZ + 1];
>  	printf("ref{%s(%" PRIu64 ") ", ref->refname, ref->update_index);
>  	switch (ref->value_type) {
>  	case REFTABLE_REF_SYMREF:
> @@ -586,7 +586,7 @@ static struct reftable_record_vtable reftable_obj_record_vtable = {
>  void reftable_log_record_print(struct reftable_log_record *log,
>  			       uint32_t hash_id)
>  {
> -	char hex[GIT_SHA256_RAWSZ + 1] = { 0 };
> +	char hex[GIT_MAX_HEXSZ + 1] = { 0 };
>  
>  	switch (log->value_type) {
>  	case REFTABLE_LOG_DELETION:

FWIW it seems to originate from
https://github.com/hanwen/reftable/commit/62bbd28 and first appeared
on-list in
https://lore.kernel.org/git/pull.847.v4.git.git.1607522429.gitgitgadget@gmail.com/
(i.e. any /SHA.*BUG/ in this code). Probably just in reference to having
to hardcode the "2 * " and/or SHA256 which later/now is using a more
generic macro.

Re: [PATCH v7 14/16] reftable: make reftable_record a tagged union

[Ævar Arnfjörð Bjarmason]

On Mon, Jan 24 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> From: Han-Wen Nienhuys <hanwen@google.com>
> [...]
> @@ -129,17 +130,25 @@ void reftable_iterator_destroy(struct reftable_iterator *it)
>  int reftable_iterator_next_ref(struct reftable_iterator *it,
>  			       struct reftable_ref_record *ref)
>  {
> -	struct reftable_record rec = { NULL };
> -	reftable_record_from_ref(&rec, ref);
> -	return iterator_next(it, &rec);
> +	struct reftable_record rec = {
> +		.type = BLOCK_TYPE_REF,
> +		.u.ref = *ref,
> +	};
> +	int err = iterator_next(it, &rec);
> +	*ref = rec.u.ref;
> +	return err;
>  }

I didn't test the v6 on xlc on AIX 7.1, but found that it refuses to
compile this code (but the one on AIX 7.2 is OK with it):
    
    "reftable/generic.c", line 135.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
    "reftable/generic.c", line 147.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
    "reftable/writer.c", line 261.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
    "reftable/writer.c", line 312.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
    "reftable/writer.c", line 406.45: 1506-196 (S) Initialization between types "unsigned long long" and "struct reftable_index_record" is not allowed.

I.e. you're dereferencing a struct type here to get at its first member,
I'm surprised that gcc and clang don't complain about it.

I think that the below fix-up for this series as a whole (didn't check
if it was just this patch) should fix it:

diff --git a/reftable/generic.c b/reftable/generic.c
index b176c1cde0a..fe8f060a689 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -132,7 +132,7 @@ int reftable_iterator_next_ref(struct reftable_iterator *it,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.u.ref = *ref,
+		.u.ref = { ref->refname },
 	};
 	int err = iterator_next(it, &rec);
 	*ref = rec.u.ref;
@@ -144,7 +144,7 @@ int reftable_iterator_next_log(struct reftable_iterator *it,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.u.log = *log,
+		.u.log = { log->refname },
 	};
 	int err = iterator_next(it, &rec);
 	*log = rec.u.log;
diff --git a/reftable/record_test.c b/reftable/record_test.c
index 847abfa2b04..be99de5194b 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -337,7 +337,7 @@ static void test_reftable_obj_record_roundtrip(void)
 		};
 		struct reftable_record in = {
 			.type = BLOCK_TYPE_OBJ,
-			.u.obj = recs[i],
+			.u.obj = { recs[i].hash_prefix },
 		};
 		struct strbuf key = STRBUF_INIT;
 		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
diff --git a/reftable/writer.c b/reftable/writer.c
index 5b768438625..6cc2ef9acaf 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -258,7 +258,7 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.u.ref = *ref,
+		.u.ref = { ref->refname },
 	};
 	int err = 0;
 
@@ -309,7 +309,7 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.u.log = *log,
+		.u.log = { log->refname },
 	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
@@ -403,7 +403,7 @@ static int writer_finish_section(struct reftable_writer *w)
 		for (i = 0; i < idx_len; i++) {
 			struct reftable_record rec = {
 				.type = BLOCK_TYPE_INDEX,
-				.u.idx = idx[i],
+				.u.idx = { idx[i].offset },
 			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;

Re: [PATCH v7 00/16] Reftable coverity fixes

[Ævar Arnfjörð Bjarmason]

On Mon, Jan 24 2022, Han-Wen Nienhuys via GitGitGadget wrote:

> This series was targeted to 'master'.
>
> This addresses some complaints from Coverity that Peff reported.
>
> v7:
>
>  * address avarab's comments.

Thanks. I just had time to skim it now but this overall LGTM. I noted a
compilation error on xlc in another reply that I'm sorry I didn't test
before, but provided a hopefully easy fix for.

>      @@ reftable/record.c: static int reftable_ref_record_equal_void(const void *a,
>       +static void reftable_ref_record_print_void(const void *rec,
>       +					   int hash_size)
>       +{
>      -+	reftable_ref_record_print_sz((struct reftable_ref_record *) rec, hash_size);
>      ++	reftable_ref_record_print_sz(
>      ++		(struct reftable_ref_record *) rec, hash_size);
>       +}
>       +

Just a nit, and only noting this since you seemed to be adjusting your
clang-format settings (or so I recall), but I think the more usual
wrapping in the codebase is:
    
    diff --git a/reftable/record.c b/reftable/record.c
    index 8330b7a2589..ee42897460d 100644
    --- a/reftable/record.c
    +++ b/reftable/record.c
    @@ -464,8 +464,8 @@ static int reftable_ref_record_equal_void(const void *a,
     static void reftable_ref_record_print_void(const void *rec,
                                               int hash_size)
     {
    -       reftable_ref_record_print_sz(
    -               (struct reftable_ref_record *) rec, hash_size);
    +       reftable_ref_record_print_sz((struct reftable_ref_record *)rec,
    +                                    hash_size);
     }
     
     static struct reftable_record_vtable reftable_ref_record_vtable = {
    

I.e. to "keep going" with the initial line until we hit 79 characters,
and then wrap it. Then for casts "(x *)y" rather than "(x *) y".

Re: [PATCH v7 14/16] reftable: make reftable_record a tagged union

[Han-Wen Nienhuys]

On Mon, Jan 24, 2022 at 8:44 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:

> I didn't test the v6 on xlc on AIX 7.1, but found that it refuses to
> compile this code (but the one on AIX 7.2 is OK with it):
>
>     "reftable/generic.c", line 135.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>     "reftable/generic.c", line 147.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>     "reftable/writer.c", line 261.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>     "reftable/writer.c", line 312.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>     "reftable/writer.c", line 406.45: 1506-196 (S) Initialization between types "unsigned long long" and "struct reftable_index_record" is not allowed.
>

> I.e. you're dereferencing a struct type here to get at its first member,

No, that's not what I'm doing.

> > +   struct reftable_record rec = {
> > +             .type = BLOCK_TYPE_REF,
> > +             .u.ref = *ref,
> > +     };

reftable_record is tagged union, and u.ref is a reftable_ref_record. I
intend to shallow-copying the parameter (a reftable_record called
'ref') into 'rec.u.ref' , precisely as it is written.

Does this patch pass the unittests?

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.
--
Google Germany GmbH, Erika-Mann-Strasse 33, 80636 Munich
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

Re: [PATCH v7 14/16] reftable: make reftable_record a tagged union

[Ævar Arnfjörð Bjarmason]

On Mon, Jan 24 2022, Han-Wen Nienhuys wrote:

> On Mon, Jan 24, 2022 at 8:44 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>
>> I didn't test the v6 on xlc on AIX 7.1, but found that it refuses to
>> compile this code (but the one on AIX 7.2 is OK with it):
>>
>>     "reftable/generic.c", line 135.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>     "reftable/generic.c", line 147.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>     "reftable/writer.c", line 261.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>     "reftable/writer.c", line 312.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>     "reftable/writer.c", line 406.45: 1506-196 (S) Initialization between types "unsigned long long" and "struct reftable_index_record" is not allowed.
>>
>
>> I.e. you're dereferencing a struct type here to get at its first member,
>
> No, that's not what I'm doing.
>
>> > +   struct reftable_record rec = {
>> > +             .type = BLOCK_TYPE_REF,
>> > +             .u.ref = *ref,
>> > +     };
>
> reftable_record is tagged union, and u.ref is a reftable_ref_record. I
> intend to shallow-copying the parameter (a reftable_record called
> 'ref') into 'rec.u.ref' , precisely as it is written.
>
> Does this patch pass the unittests?

No, sorry about that. It fails. I didn't have time to run them earlier
(and it takes a while on that platform).

I don't know how to work around it then...

master doesn't compile on xlc 21.01 anymore (old AIX compiler) (was: [PATCH v7 14/16] reftable: make reftable_record a tagged union)

[Ævar Arnfjörð Bjarmason]

On Mon, Jan 24 2022, Ævar Arnfjörð Bjarmason wrote:

> On Mon, Jan 24 2022, Han-Wen Nienhuys wrote:
>
>> On Mon, Jan 24, 2022 at 8:44 PM Ævar Arnfjörð Bjarmason
>> <avarab@gmail.com> wrote:
>>
>>> I didn't test the v6 on xlc on AIX 7.1, but found that it refuses to
>>> compile this code (but the one on AIX 7.2 is OK with it):
>>>
>>>     "reftable/generic.c", line 135.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>>     "reftable/generic.c", line 147.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>>     "reftable/writer.c", line 261.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>>     "reftable/writer.c", line 312.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>>     "reftable/writer.c", line 406.45: 1506-196 (S) Initialization between types "unsigned long long" and "struct reftable_index_record" is not allowed.
>>>
>>
>>> I.e. you're dereferencing a struct type here to get at its first member,
>>
>> No, that's not what I'm doing.
>>
>>> > +   struct reftable_record rec = {
>>> > +             .type = BLOCK_TYPE_REF,
>>> > +             .u.ref = *ref,
>>> > +     };
>>
>> reftable_record is tagged union, and u.ref is a reftable_ref_record. I
>> intend to shallow-copying the parameter (a reftable_record called
>> 'ref') into 'rec.u.ref' , precisely as it is written.
>>
>> Does this patch pass the unittests?
>
> No, sorry about that. It fails. I didn't have time to run them earlier
> (and it takes a while on that platform).
>
> I don't know how to work around it then...

Just a reminder that now with this landed on master we've hard broken
compilation on that version of xlc:
    
    avar@gcc111:[/home/avar]xlc -qversion
    IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
    Version: 12.01.0000.0000

The error is:
    
    "reftable/generic.c", line 133.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
    "reftable/generic.c", line 145.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.

Is there really no workaround we can think of for this?

Re: master doesn't compile on xlc 21.01 anymore (old AIX compiler) (was: [PATCH v7 14/16] reftable: make reftable_record a tagged union)

[René Scharfe]

Am 19.02.22 um 13:54 schrieb Ævar Arnfjörð Bjarmason:
>
> On Mon, Jan 24 2022, Ævar Arnfjörð Bjarmason wrote:
>
>> On Mon, Jan 24 2022, Han-Wen Nienhuys wrote:
>>
>>> On Mon, Jan 24, 2022 at 8:44 PM Ævar Arnfjörð Bjarmason
>>> <avarab@gmail.com> wrote:
>>>
>>>> I didn't test the v6 on xlc on AIX 7.1, but found that it refuses to
>>>> compile this code (but the one on AIX 7.2 is OK with it):
>>>>
>>>>     "reftable/generic.c", line 135.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>>>     "reftable/generic.c", line 147.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>>>     "reftable/writer.c", line 261.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>>>>     "reftable/writer.c", line 312.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>>>>     "reftable/writer.c", line 406.45: 1506-196 (S) Initialization between types "unsigned long long" and "struct reftable_index_record" is not allowed.
>>>>
>>>
>>>> I.e. you're dereferencing a struct type here to get at its first member,
>>>
>>> No, that's not what I'm doing.
>>>
>>>>> +   struct reftable_record rec = {
>>>>> +             .type = BLOCK_TYPE_REF,
>>>>> +             .u.ref = *ref,
>>>>> +     };
>>>
>>> reftable_record is tagged union, and u.ref is a reftable_ref_record. I
>>> intend to shallow-copying the parameter (a reftable_record called
>>> 'ref') into 'rec.u.ref' , precisely as it is written.
>>>
>>> Does this patch pass the unittests?
>>
>> No, sorry about that. It fails. I didn't have time to run them earlier
>> (and it takes a while on that platform).
>>
>> I don't know how to work around it then...
>
> Just a reminder that now with this landed on master we've hard broken
> compilation on that version of xlc:
>
>     avar@gcc111:[/home/avar]xlc -qversion
>     IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
>     Version: 12.01.0000.0000
>
> The error is:
>
>     "reftable/generic.c", line 133.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_ref_record" is not allowed.
>     "reftable/generic.c", line 145.26: 1506-196 (S) Initialization between types "char*" and "struct reftable_log_record" is not allowed.
>
> Is there really no workaround we can think of for this?

The code is hard to read for me in general due to its many types and
function dispatch tables, but this compiler seems to have an even harder
time.  Where does it even get that char pointer type from?

Can it compile this?

	struct reftable_record rec = {
		.type = BLOCK_TYPE_REF,
		.u = {
			.ref = *ref
		},
	};

If it can't, what about this?

	struct reftable_record rec = {
		.type = BLOCK_TYPE_REF,
	};
	rec.u.ref = *ref;

René

[PATCH] reftable: make assignments portable to AIX xlc v12.01

[Ævar Arnfjörð Bjarmason]

Change the assignment syntax introduced in 66c0dabab5e (reftable: make
reftable_record a tagged union, 2022-01-20) to be portable to AIX xlc
v12.1:

    avar@gcc111:[/home/avar]xlc -qversion
    IBM XL C/C++ for AIX, V12.1 (5765-J02, 5725-C72)
    Version: 12.01.0000.0000

The error emitted before this was e.g.:

    "reftable/generic.c", line 133.26: 1506-196 (S) Initialization
    between types "char*" and "struct reftable_ref_record" is not
    allowed.

The syntax in the pre-image is supported by e.g. xlc 13.01 on a newer
AIX version:

    avar@gcc119:[/home/avar]xlc -qversion
    IBM XL C/C++ for AIX, V13.1.3 (5725-C72, 5765-J07)
    Version: 13.01.0003.0006

But as we've otherwise supported this compiler let's not break it
entirely if it's easy to work around it.

Suggested-by: René Scharfe <l.s.r@web.de>
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---

On Sat, Feb 19 2022, René Scharfe wrote:

> Am 19.02.22 um 13:54 schrieb Ævar Arnfjörð Bjarmason:
> [...]
> The code is hard to read for me in general due to its many types and
> function dispatch tables, but this compiler seems to have an even harder
> time.  Where does it even get that char pointer type from?

Sorry about the very late reply, yes thath first suggestion worked,
d'oh in retrospect. This patch fixes it.


 reftable/generic.c     |  8 ++++++--
 reftable/record_test.c |  4 +++-
 reftable/writer.c      | 12 +++++++++---
 3 files changed, 18 insertions(+), 6 deletions(-)

diff --git a/reftable/generic.c b/reftable/generic.c
index b27d152e89a..57f8032db94 100644
--- a/reftable/generic.c
+++ b/reftable/generic.c
@@ -130,7 +130,9 @@ int reftable_iterator_next_ref(struct reftable_iterator *it,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.u.ref = *ref,
+		.u = {
+			.ref = *ref
+		},
 	};
 	int err = iterator_next(it, &rec);
 	*ref = rec.u.ref;
@@ -142,7 +144,9 @@ int reftable_iterator_next_log(struct reftable_iterator *it,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.u.log = *log,
+		.u = {
+			.log = *log,
+		},
 	};
 	int err = iterator_next(it, &rec);
 	*log = rec.u.log;
diff --git a/reftable/record_test.c b/reftable/record_test.c
index f91ea5e8830..70ae78feca4 100644
--- a/reftable/record_test.c
+++ b/reftable/record_test.c
@@ -339,7 +339,9 @@ static void test_reftable_obj_record_roundtrip(void)
 		};
 		struct reftable_record in = {
 			.type = BLOCK_TYPE_OBJ,
-			.u.obj = recs[i],
+			.u = {
+				.obj = recs[i],
+			},
 		};
 		struct strbuf key = STRBUF_INIT;
 		struct reftable_record out = { .type = BLOCK_TYPE_OBJ };
diff --git a/reftable/writer.c b/reftable/writer.c
index 6d979e245ff..427f1317c6b 100644
--- a/reftable/writer.c
+++ b/reftable/writer.c
@@ -257,7 +257,9 @@ int reftable_writer_add_ref(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_REF,
-		.u.ref = *ref,
+		.u = {
+			.ref = *ref
+		},
 	};
 	int err = 0;
 
@@ -308,7 +310,9 @@ static int reftable_writer_add_log_verbatim(struct reftable_writer *w,
 {
 	struct reftable_record rec = {
 		.type = BLOCK_TYPE_LOG,
-		.u.log = *log,
+		.u = {
+			.log = *log,
+		},
 	};
 	if (w->block_writer &&
 	    block_writer_type(w->block_writer) == BLOCK_TYPE_REF) {
@@ -401,7 +405,9 @@ static int writer_finish_section(struct reftable_writer *w)
 		for (i = 0; i < idx_len; i++) {
 			struct reftable_record rec = {
 				.type = BLOCK_TYPE_INDEX,
-				.u.idx = idx[i],
+				.u = {
+					.idx = idx[i],
+				},
 			};
 			if (block_writer_add(w->block_writer, &rec) == 0) {
 				continue;
-- 
2.35.1.1548.gaad2ca01774

Re: [PATCH] reftable: make assignments portable to AIX xlc v12.01

[Junio C Hamano]

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:

> diff --git a/reftable/generic.c b/reftable/generic.c
> index b27d152e89a..57f8032db94 100644
> --- a/reftable/generic.c
> +++ b/reftable/generic.c
> @@ -130,7 +130,9 @@ int reftable_iterator_next_ref(struct reftable_iterator *it,
>  {
>  	struct reftable_record rec = {
>  		.type = BLOCK_TYPE_REF,
> -		.u.ref = *ref,
> +		.u = {
> +			.ref = *ref
> +		},
>  	};

Having to do this does *not* look so bad.

Re: [PATCH] reftable: make assignments portable to AIX xlc v12.01

[Han-Wen Nienhuys]

On Mon, Mar 28, 2022 at 9:10 PM Ævar Arnfjörð Bjarmason
<avarab@gmail.com> wrote:
>
> Change the assignment syntax introduced in 66c0dabab5e (reftable: make
> reftable_record a tagged union, 2022-01-20) to be portable to AIX xlc
> v12.1:

LGTM.

-- 
Han-Wen Nienhuys - Google Munich
I work 80%. Don't expect answers from me on Fridays.

Re: [PATCH] reftable: make assignments portable to AIX xlc v12.01

[Junio C Hamano]

Han-Wen Nienhuys <hanwen@google.com> writes:

> On Mon, Mar 28, 2022 at 9:10 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>>
>> Change the assignment syntax introduced in 66c0dabab5e (reftable: make
>> reftable_record a tagged union, 2022-01-20) to be portable to AIX xlc
>> v12.1:
>
> LGTM.

Thanks; queued.