* [RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer
@ 2018-10-04 23:01 Josh Steadmon
2018-10-04 23:01 ` [RFC PATCH 1/2] fuzz: Add basic fuzz testing target Josh Steadmon
` (2 more replies)
0 siblings, 3 replies; 10+ messages in thread
From: Josh Steadmon @ 2018-10-04 23:01 UTC (permalink / raw)
To: git; +Cc: Josh Steadmon
libFuzzer[1] is a fuzzing engine included in recent versions of LLVM. It
is used by OSS-Fuzz[2] for continuous fuzzing of OSS projects.
This series adds two basic fuzzing targets covering packfile header and
index code. It is not particularly portable, and requires the use of
LLVM v4.0 (the latest version available on my workstation). I would
particularly appreciate advice on how to make the Makefile more
portable.
[1]: https://llvm.org/docs/LibFuzzer.html
[2]: https://github.com/google/oss-fuzz
Josh Steadmon (2):
fuzz: Add basic fuzz testing target.
fuzz: Add fuzz testing for packfile indices.
.gitignore | 3 +++
Makefile | 33 ++++++++++++++++++++++++++++++++-
fuzz-pack-headers.c | 14 ++++++++++++++
fuzz-pack-idx.c | 13 +++++++++++++
packfile.c | 44 +++++++++++++++++++++++++-------------------
packfile.h | 13 +++++++++++++
6 files changed, 100 insertions(+), 20 deletions(-)
create mode 100644 fuzz-pack-headers.c
create mode 100644 fuzz-pack-idx.c
--
2.19.0.605.g01d371f741-goog
^ permalink raw reply [flat|nested] 10+ messages in thread* [RFC PATCH 1/2] fuzz: Add basic fuzz testing target. 2018-10-04 23:01 [RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer Josh Steadmon @ 2018-10-04 23:01 ` Josh Steadmon 2018-10-10 2:14 ` Junio C Hamano 2018-10-04 23:01 ` [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices Josh Steadmon 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon 2 siblings, 1 reply; 10+ messages in thread From: Josh Steadmon @ 2018-10-04 23:01 UTC (permalink / raw) To: git; +Cc: Josh Steadmon Signed-off-by: Josh Steadmon <steadmon@google.com> --- .gitignore | 2 ++ Makefile | 30 +++++++++++++++++++++++++++++- fuzz-pack-headers.c | 14 ++++++++++++++ 3 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 fuzz-pack-headers.c diff --git a/.gitignore b/.gitignore index 9d1363a1eb..87a28b3115 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ +/fuzz_corpora +/fuzz-pack-headers /GIT-BUILD-OPTIONS /GIT-CFLAGS /GIT-LDFLAGS diff --git a/Makefile b/Makefile index 13e1c52478..10bb82b115 100644 --- a/Makefile +++ b/Makefile @@ -590,6 +590,8 @@ XDIFF_OBJS = VCSSVN_OBJS = GENERATED_H = EXTRA_CPPFLAGS = +FUZZ_OBJS = +FUZZ_PROGRAMS = LIB_OBJS = PROGRAM_OBJS = PROGRAMS = @@ -682,6 +684,10 @@ SCRIPTS = $(SCRIPT_SH_INS) \ ETAGS_TARGET = TAGS +FUZZ_OBJS += fuzz-pack-headers.o + +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) + # Empty... EXTRA_PROGRAMS = @@ -2250,6 +2256,7 @@ TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ + $(FUZZ_OBJS) \ common-main.o \ git.o ifndef NO_CURL @@ -2931,7 +2938,7 @@ profile-clean: cocciclean: $(RM) contrib/coccinelle/*.cocci.patch* -clean: profile-clean coverage-clean cocciclean +clean: profile-clean coverage-clean cocciclean fuzz-clean $(RM) *.res $(RM) $(OBJECTS) $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) @@ -3061,3 +3068,24 @@ cover_db: coverage-report cover_db_html: cover_db cover -report html -outputdir cover_db_html cover_db + +### Fuzz testing +# +.PHONY: fuzz-clean fuzz-objs fuzz-compile + +FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address +FUZZ_LDFLAGS = $(FUZZ_CFLAGS) + + +fuzz-clean: + $(RM) $(FUZZ_PROGRAMS) $(FUZZ_OBJS) + +fuzz-objs: $(FUZZ_OBJS) + +fuzz-compile: + $(MAKE) CC=clang LD=clang CFLAGS="$(FUZZ_CFLAGS)" \ + LDFLAGS="$(FUZZ_LDFLAGS)" all fuzz-objs + +$(FUZZ_PROGRAMS): fuzz-compile + clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ + $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ diff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c new file mode 100644 index 0000000000..99da1d0fd3 --- /dev/null +++ b/fuzz-pack-headers.c @@ -0,0 +1,14 @@ +#include "packfile.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + enum object_type type; + unsigned long len; + + unpack_object_header_buffer((const unsigned char *)data, + (unsigned long)size, &type, &len); + + return 0; +} -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 1/2] fuzz: Add basic fuzz testing target. 2018-10-04 23:01 ` [RFC PATCH 1/2] fuzz: Add basic fuzz testing target Josh Steadmon @ 2018-10-10 2:14 ` Junio C Hamano 2018-10-13 0:59 ` Josh Steadmon 0 siblings, 1 reply; 10+ messages in thread From: Junio C Hamano @ 2018-10-10 2:14 UTC (permalink / raw) To: Josh Steadmon; +Cc: git Josh Steadmon <steadmon@google.com> writes: > +FUZZ_OBJS += fuzz-pack-headers.o > + > +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) > + > ... > +### Fuzz testing > +# > +.PHONY: fuzz-clean fuzz-objs fuzz-compile I take it that you anticipate the fuzz programs in the future all be named fuzz-$(blah), whose source is fuzz-$(blah).o (even though we may grow some common code that may be linked with them, which can be done by tweaking the rule for the $(FUZZ_PROGRAMS) target). Am I reading you correctly? Would fuzz-{clean,objs,compile} risk squatting on nicer names we may want to use for $(blah) down the line? > + ... > +$(FUZZ_PROGRAMS): fuzz-compile > + clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ > + $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ Is the expected usage pattern to know a single fuzz-* program the builder wants to build, to run "make fuzz-pack-headers"? If not, it also would be a good idea to have something like fuzz-build-all:: $(FUZZ_PROGRAMS) .PHONY: fuzz-build-all perhaps? Also, in the final version we unleash to general developer audience, we'd want to support "make V=1" (and "make" that is "$(QUIET)"). ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 1/2] fuzz: Add basic fuzz testing target. 2018-10-10 2:14 ` Junio C Hamano @ 2018-10-13 0:59 ` Josh Steadmon 0 siblings, 0 replies; 10+ messages in thread From: Josh Steadmon @ 2018-10-13 0:59 UTC (permalink / raw) To: Junio C Hamano; +Cc: git On 2018.10.10 11:14, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > +FUZZ_OBJS += fuzz-pack-headers.o > > + > > +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) > > + > > ... > > +### Fuzz testing > > +# > > +.PHONY: fuzz-clean fuzz-objs fuzz-compile > > I take it that you anticipate the fuzz programs in the future all > be named fuzz-$(blah), whose source is fuzz-$(blah).o (even though > we may grow some common code that may be linked with them, which can > be done by tweaking the rule for the $(FUZZ_PROGRAMS) target). Am I > reading you correctly? Would fuzz-{clean,objs,compile} risk squatting > on nicer names we may want to use for $(blah) down the line? Yes, that's correct. I've reworked the rules to be more compatible with how OSS-Fuzz expects to build these targets, and now "fuzz-all" is the only remaining special target. > > + ... > > +$(FUZZ_PROGRAMS): fuzz-compile > > + clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ > > + $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ > > Is the expected usage pattern to know a single fuzz-* program the > builder wants to build, to run "make fuzz-pack-headers"? If not, it > also would be a good idea to have something like > > fuzz-build-all:: $(FUZZ_PROGRAMS) > .PHONY: fuzz-build-all > > perhaps? > > Also, in the final version we unleash to general developer audience, > we'd want to support "make V=1" (and "make" that is "$(QUIET)"). Done and done. ^ permalink raw reply [flat|nested] 10+ messages in thread
* [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices. 2018-10-04 23:01 [RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer Josh Steadmon 2018-10-04 23:01 ` [RFC PATCH 1/2] fuzz: Add basic fuzz testing target Josh Steadmon @ 2018-10-04 23:01 ` Josh Steadmon 2018-10-10 2:19 ` Junio C Hamano 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon 2 siblings, 1 reply; 10+ messages in thread From: Josh Steadmon @ 2018-10-04 23:01 UTC (permalink / raw) To: git; +Cc: Josh Steadmon Signed-off-by: Josh Steadmon <steadmon@google.com> --- .gitignore | 1 + Makefile | 5 ++++- fuzz-pack-idx.c | 13 +++++++++++++ packfile.c | 44 +++++++++++++++++++++++++------------------- packfile.h | 13 +++++++++++++ 5 files changed, 56 insertions(+), 20 deletions(-) create mode 100644 fuzz-pack-idx.c diff --git a/.gitignore b/.gitignore index 87a28b3115..64b3377d40 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ /fuzz_corpora /fuzz-pack-headers +/fuzz-pack-idx /GIT-BUILD-OPTIONS /GIT-CFLAGS /GIT-LDFLAGS diff --git a/Makefile b/Makefile index 10bb82b115..f7e6be8f19 100644 --- a/Makefile +++ b/Makefile @@ -685,6 +685,7 @@ SCRIPTS = $(SCRIPT_SH_INS) \ ETAGS_TARGET = TAGS FUZZ_OBJS += fuzz-pack-headers.o +FUZZ_OBJS += fuzz-pack-idx.o FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) @@ -3071,7 +3072,7 @@ cover_db_html: cover_db ### Fuzz testing # -.PHONY: fuzz-clean fuzz-objs fuzz-compile +.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address FUZZ_LDFLAGS = $(FUZZ_CFLAGS) @@ -3089,3 +3090,5 @@ fuzz-compile: $(FUZZ_PROGRAMS): fuzz-compile clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ + +fuzz-all: $(FUZZ_PROGRAMS) diff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c new file mode 100644 index 0000000000..0c3d777aac --- /dev/null +++ b/fuzz-pack-idx.c @@ -0,0 +1,13 @@ +#include "object-store.h" +#include "packfile.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + struct packed_git p; + + load_idx("fuzz-input", GIT_SHA1_RAWSZ, (void *)data, size, &p); + + return 0; +} diff --git a/packfile.c b/packfile.c index 841b36182f..86074a76e9 100644 --- a/packfile.c +++ b/packfile.c @@ -80,10 +80,8 @@ void pack_report(void) static int check_packed_git_idx(const char *path, struct packed_git *p) { void *idx_map; - struct pack_idx_header *hdr; size_t idx_size; - uint32_t version, nr, i, *index; - int fd = git_open(path); + int fd = git_open(path), ret; struct stat st; const unsigned int hashsz = the_hash_algo->rawsz; @@ -101,16 +99,32 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) idx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0); close(fd); - hdr = idx_map; + ret = load_idx(path, hashsz, idx_map, idx_size, p); + + if (ret) + munmap(idx_map, idx_size); + + return ret; +} + +int load_idx(const char *path, const unsigned int hashsz, void *idx_map, + size_t idx_size, struct packed_git *p) +{ + struct pack_idx_header *hdr = idx_map; + uint32_t version, nr, i, *index; + + if (idx_size < 4 * 256 + hashsz + hashsz) + return error("index file %s is too small", path); + if (idx_map == NULL) + return error("empty data"); + if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) { version = ntohl(hdr->idx_version); - if (version < 2 || version > 2) { - munmap(idx_map, idx_size); + if (version < 2 || version > 2) return error("index file %s is version %"PRIu32 " and is not supported by this binary" " (try upgrading GIT to a newer version)", path, version); - } } else version = 1; @@ -120,10 +134,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) index += 2; /* skip index header */ for (i = 0; i < 256; i++) { uint32_t n = ntohl(index[i]); - if (n < nr) { - munmap(idx_map, idx_size); + if (n < nr) return error("non-monotonic index %s", path); - } nr = n; } @@ -135,10 +147,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) * - hash of the packfile * - file checksum */ - if (idx_size != 4*256 + nr * (hashsz + 4) + hashsz + hashsz) { - munmap(idx_map, idx_size); + if (idx_size != 4 * 256 + nr * (hashsz + 4) + hashsz + hashsz) return error("wrong index v1 file size in %s", path); - } } else if (version == 2) { /* * Minimum size: @@ -157,20 +167,16 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) unsigned long max_size = min_size; if (nr) max_size += (nr - 1)*8; - if (idx_size < min_size || idx_size > max_size) { - munmap(idx_map, idx_size); + if (idx_size < min_size || idx_size > max_size) return error("wrong index v2 file size in %s", path); - } if (idx_size != min_size && /* * make sure we can deal with large pack offsets. * 31-bit signed offset won't be enough, neither * 32-bit unsigned one will be. */ - (sizeof(off_t) <= 4)) { - munmap(idx_map, idx_size); + (sizeof(off_t) <= 4)) return error("pack too large for current definition of off_t in %s", path); - } } p->index_version = version; diff --git a/packfile.h b/packfile.h index 442625723d..6c4037605d 100644 --- a/packfile.h +++ b/packfile.h @@ -164,4 +164,17 @@ extern int has_pack_index(const unsigned char *sha1); */ extern int is_promisor_object(const struct object_id *oid); +/* + * Expose a function for fuzz testing. + * + * load_idx() parses a block of memory as a packfile index and puts the results + * into a struct packed_git. + * + * This function should not be used directly. It is exposed here only so that we + * have a convenient entry-point for fuzz testing. For real uses, you should + * probably use open_pack_index() or parse_pack_index() instead. + */ +extern int load_idx(const char *path, const unsigned int hashsz, void *idx_map, + size_t idx_size, struct packed_git *p); + #endif -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices. 2018-10-04 23:01 ` [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices Josh Steadmon @ 2018-10-10 2:19 ` Junio C Hamano 0 siblings, 0 replies; 10+ messages in thread From: Junio C Hamano @ 2018-10-10 2:19 UTC (permalink / raw) To: Josh Steadmon; +Cc: git Josh Steadmon <steadmon@google.com> writes: > ### Fuzz testing > # > -.PHONY: fuzz-clean fuzz-objs fuzz-compile > +.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all > ... > FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address > ... > + > +fuzz-all: $(FUZZ_PROGRAMS) I guess I read your mind ;-) Please do this in 1/2 instead of adding it at 2/2 as "oops, we'd need it more and more as we add these". ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz 2018-10-04 23:01 [RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer Josh Steadmon 2018-10-04 23:01 ` [RFC PATCH 1/2] fuzz: Add basic fuzz testing target Josh Steadmon 2018-10-04 23:01 ` [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices Josh Steadmon @ 2018-10-13 0:58 ` steadmon 2018-10-13 0:58 ` [PATCH v2 1/2] fuzz: Add basic fuzz testing target steadmon ` (2 more replies) 2 siblings, 3 replies; 10+ messages in thread From: steadmon @ 2018-10-13 0:58 UTC (permalink / raw) To: git; +Cc: gitster, Josh Steadmon From: Josh Steadmon <steadmon@google.com> V2 of this series pulls the compiler flags out of the Makefile, to be provided by the user depending on the combination of compiler and fuzzing engine in use. This also makes it more compatible with OSS-Fuzz's build process. Josh Steadmon (2): fuzz: Add basic fuzz testing target. fuzz: Add fuzz testing for packfile indices. .gitignore | 3 +++ Makefile | 33 +++++++++++++++++++++++++++++++++ fuzz-pack-headers.c | 14 ++++++++++++++ fuzz-pack-idx.c | 13 +++++++++++++ packfile.c | 44 +++++++++++++++++++++++++------------------- packfile.h | 13 +++++++++++++ 6 files changed, 101 insertions(+), 19 deletions(-) create mode 100644 fuzz-pack-headers.c create mode 100644 fuzz-pack-idx.c Range-diff against v1: 1: 9456c41798 ! 1: 446d8081b1 fuzz: Add basic fuzz testing target. @@ -32,6 +32,9 @@ +FUZZ_OBJS += fuzz-pack-headers.o + ++# Always build fuzz objects even if not testing, to prevent bit-rot. ++all:: $(FUZZ_OBJS) ++ +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) + # Empty... @@ -46,14 +49,13 @@ git.o ifndef NO_CURL @@ - cocciclean: - $(RM) contrib/coccinelle/*.cocci.patch* - --clean: profile-clean coverage-clean cocciclean -+clean: profile-clean coverage-clean cocciclean fuzz-clean - $(RM) *.res - $(RM) $(OBJECTS) $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) + $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X + $(RM) $(TEST_PROGRAMS) $(NO_INSTALL) ++ $(RM) $(FUZZ_PROGRAMS) + $(RM) -r bin-wrappers $(dep_dirs) + $(RM) -r po/build/ + $(RM) *.pyc *.pyo */*.pyc */*.pyo command-list.h $(ETAGS_TARGET) tags cscope* @@ cover_db_html: cover_db cover -report html -outputdir cover_db_html cover_db @@ -61,24 +63,24 @@ + +### Fuzz testing +# -+.PHONY: fuzz-clean fuzz-objs fuzz-compile -+ -+FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address -+FUZZ_LDFLAGS = $(FUZZ_CFLAGS) -+ -+ -+fuzz-clean: -+ $(RM) $(FUZZ_PROGRAMS) $(FUZZ_OBJS) -+ -+fuzz-objs: $(FUZZ_OBJS) ++# Building fuzz targets generally requires a special set of compiler flags that ++# are not necessarily appropriate for general builds, and that vary greatly ++# depending on the compiler version used. ++# ++# An example command to build against libFuzzer from LLVM 4.0.0: ++# ++# make CC=clang CXX=clang++ \ ++# CFLAGS="-fsanitize-coverage=trace-pc-guard -fsanitize=address" \ ++# LIB_FUZZING_ENGINE=/usr/lib/llvm-4.0/lib/libFuzzer.a \ ++# fuzz-all ++# ++.PHONY: fuzz-all + -+fuzz-compile: -+ $(MAKE) CC=clang LD=clang CFLAGS="$(FUZZ_CFLAGS)" \ -+ LDFLAGS="$(FUZZ_LDFLAGS)" all fuzz-objs ++$(FUZZ_PROGRAMS): all ++ $(QUIET_LINK)$(CXX) $(CFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) \ ++ $(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@ + -+$(FUZZ_PROGRAMS): fuzz-compile -+ clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ -+ $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ ++fuzz-all: $(FUZZ_PROGRAMS) diff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c new file mode 100644 2: 581eb8f817 ! 2: c7b5a03d81 fuzz: Add fuzz testing for packfile indices. @@ -24,23 +24,8 @@ FUZZ_OBJS += fuzz-pack-headers.o +FUZZ_OBJS += fuzz-pack-idx.o - FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) - -@@ - - ### Fuzz testing - # --.PHONY: fuzz-clean fuzz-objs fuzz-compile -+.PHONY: fuzz-clean fuzz-objs fuzz-compile fuzz-all - - FUZZ_CFLAGS = $(CFLAGS) -fsanitize-coverage=trace-pc-guard -fsanitize=address - FUZZ_LDFLAGS = $(FUZZ_CFLAGS) -@@ - $(FUZZ_PROGRAMS): fuzz-compile - clang++ $(FUZZ_LDFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) $(XDIFF_OBJS) \ - $(EXTLIBS) git.o $@.o /usr/lib/llvm-4.0/lib/libFuzzer.a -o $@ -+ -+fuzz-all: $(FUZZ_PROGRAMS) + # Always build fuzz objects even if not testing, to prevent bit-rot. + all:: $(FUZZ_OBJS) diff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c new file mode 100644 -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v2 1/2] fuzz: Add basic fuzz testing target. 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon @ 2018-10-13 0:58 ` steadmon 2018-10-13 0:58 ` [PATCH v2 2/2] fuzz: Add fuzz testing for packfile indices steadmon 2018-10-16 6:18 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz Junio C Hamano 2 siblings, 0 replies; 10+ messages in thread From: steadmon @ 2018-10-13 0:58 UTC (permalink / raw) To: git; +Cc: gitster, Josh Steadmon From: Josh Steadmon <steadmon@google.com> fuzz-pack-headers.c provides a fuzzing entry point compatible with libFuzzer (and possibly other fuzzing engines). Signed-off-by: Josh Steadmon <steadmon@google.com> --- .gitignore | 2 ++ Makefile | 32 ++++++++++++++++++++++++++++++++ fuzz-pack-headers.c | 14 ++++++++++++++ 3 files changed, 48 insertions(+) create mode 100644 fuzz-pack-headers.c diff --git a/.gitignore b/.gitignore index 9d1363a1eb..87a28b3115 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,5 @@ +/fuzz_corpora +/fuzz-pack-headers /GIT-BUILD-OPTIONS /GIT-CFLAGS /GIT-LDFLAGS diff --git a/Makefile b/Makefile index 13e1c52478..7f5a19b8ab 100644 --- a/Makefile +++ b/Makefile @@ -590,6 +590,8 @@ XDIFF_OBJS = VCSSVN_OBJS = GENERATED_H = EXTRA_CPPFLAGS = +FUZZ_OBJS = +FUZZ_PROGRAMS = LIB_OBJS = PROGRAM_OBJS = PROGRAMS = @@ -682,6 +684,13 @@ SCRIPTS = $(SCRIPT_SH_INS) \ ETAGS_TARGET = TAGS +FUZZ_OBJS += fuzz-pack-headers.o + +# Always build fuzz objects even if not testing, to prevent bit-rot. +all:: $(FUZZ_OBJS) + +FUZZ_PROGRAMS += $(patsubst %.o,%,$(FUZZ_OBJS)) + # Empty... EXTRA_PROGRAMS = @@ -2250,6 +2259,7 @@ TEST_OBJS := $(patsubst %$X,%.o,$(TEST_PROGRAMS)) $(patsubst %,t/helper/%,$(TEST OBJECTS := $(LIB_OBJS) $(BUILTIN_OBJS) $(PROGRAM_OBJS) $(TEST_OBJS) \ $(XDIFF_OBJS) \ $(VCSSVN_OBJS) \ + $(FUZZ_OBJS) \ common-main.o \ git.o ifndef NO_CURL @@ -2937,6 +2947,7 @@ clean: profile-clean coverage-clean cocciclean $(RM) $(LIB_FILE) $(XDIFF_LIB) $(VCSSVN_LIB) $(RM) $(ALL_PROGRAMS) $(SCRIPT_LIB) $(BUILT_INS) git$X $(RM) $(TEST_PROGRAMS) $(NO_INSTALL) + $(RM) $(FUZZ_PROGRAMS) $(RM) -r bin-wrappers $(dep_dirs) $(RM) -r po/build/ $(RM) *.pyc *.pyo */*.pyc */*.pyo command-list.h $(ETAGS_TARGET) tags cscope* @@ -3061,3 +3072,24 @@ cover_db: coverage-report cover_db_html: cover_db cover -report html -outputdir cover_db_html cover_db + +### Fuzz testing +# +# Building fuzz targets generally requires a special set of compiler flags that +# are not necessarily appropriate for general builds, and that vary greatly +# depending on the compiler version used. +# +# An example command to build against libFuzzer from LLVM 4.0.0: +# +# make CC=clang CXX=clang++ \ +# CFLAGS="-fsanitize-coverage=trace-pc-guard -fsanitize=address" \ +# LIB_FUZZING_ENGINE=/usr/lib/llvm-4.0/lib/libFuzzer.a \ +# fuzz-all +# +.PHONY: fuzz-all + +$(FUZZ_PROGRAMS): all + $(QUIET_LINK)$(CXX) $(CFLAGS) $(LIB_OBJS) $(BUILTIN_OBJS) \ + $(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@ + +fuzz-all: $(FUZZ_PROGRAMS) diff --git a/fuzz-pack-headers.c b/fuzz-pack-headers.c new file mode 100644 index 0000000000..99da1d0fd3 --- /dev/null +++ b/fuzz-pack-headers.c @@ -0,0 +1,14 @@ +#include "packfile.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + enum object_type type; + unsigned long len; + + unpack_object_header_buffer((const unsigned char *)data, + (unsigned long)size, &type, &len); + + return 0; +} -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 10+ messages in thread
* [PATCH v2 2/2] fuzz: Add fuzz testing for packfile indices. 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon 2018-10-13 0:58 ` [PATCH v2 1/2] fuzz: Add basic fuzz testing target steadmon @ 2018-10-13 0:58 ` steadmon 2018-10-16 6:18 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz Junio C Hamano 2 siblings, 0 replies; 10+ messages in thread From: steadmon @ 2018-10-13 0:58 UTC (permalink / raw) To: git; +Cc: gitster, Josh Steadmon From: Josh Steadmon <steadmon@google.com> Breaks the majority of check_packed_git_idx() into a separate function, load_idx(). The latter function operates on arbitrary buffers, which makes it suitable as a fuzzing test target. Signed-off-by: Josh Steadmon <steadmon@google.com> --- .gitignore | 1 + Makefile | 1 + fuzz-pack-idx.c | 13 +++++++++++++ packfile.c | 44 +++++++++++++++++++++++++------------------- packfile.h | 13 +++++++++++++ 5 files changed, 53 insertions(+), 19 deletions(-) create mode 100644 fuzz-pack-idx.c diff --git a/.gitignore b/.gitignore index 87a28b3115..64b3377d40 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ /fuzz_corpora /fuzz-pack-headers +/fuzz-pack-idx /GIT-BUILD-OPTIONS /GIT-CFLAGS /GIT-LDFLAGS diff --git a/Makefile b/Makefile index 7f5a19b8ab..1b3d893090 100644 --- a/Makefile +++ b/Makefile @@ -685,6 +685,7 @@ SCRIPTS = $(SCRIPT_SH_INS) \ ETAGS_TARGET = TAGS FUZZ_OBJS += fuzz-pack-headers.o +FUZZ_OBJS += fuzz-pack-idx.o # Always build fuzz objects even if not testing, to prevent bit-rot. all:: $(FUZZ_OBJS) diff --git a/fuzz-pack-idx.c b/fuzz-pack-idx.c new file mode 100644 index 0000000000..0c3d777aac --- /dev/null +++ b/fuzz-pack-idx.c @@ -0,0 +1,13 @@ +#include "object-store.h" +#include "packfile.h" + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); + +int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) +{ + struct packed_git p; + + load_idx("fuzz-input", GIT_SHA1_RAWSZ, (void *)data, size, &p); + + return 0; +} diff --git a/packfile.c b/packfile.c index 841b36182f..86074a76e9 100644 --- a/packfile.c +++ b/packfile.c @@ -80,10 +80,8 @@ void pack_report(void) static int check_packed_git_idx(const char *path, struct packed_git *p) { void *idx_map; - struct pack_idx_header *hdr; size_t idx_size; - uint32_t version, nr, i, *index; - int fd = git_open(path); + int fd = git_open(path), ret; struct stat st; const unsigned int hashsz = the_hash_algo->rawsz; @@ -101,16 +99,32 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) idx_map = xmmap(NULL, idx_size, PROT_READ, MAP_PRIVATE, fd, 0); close(fd); - hdr = idx_map; + ret = load_idx(path, hashsz, idx_map, idx_size, p); + + if (ret) + munmap(idx_map, idx_size); + + return ret; +} + +int load_idx(const char *path, const unsigned int hashsz, void *idx_map, + size_t idx_size, struct packed_git *p) +{ + struct pack_idx_header *hdr = idx_map; + uint32_t version, nr, i, *index; + + if (idx_size < 4 * 256 + hashsz + hashsz) + return error("index file %s is too small", path); + if (idx_map == NULL) + return error("empty data"); + if (hdr->idx_signature == htonl(PACK_IDX_SIGNATURE)) { version = ntohl(hdr->idx_version); - if (version < 2 || version > 2) { - munmap(idx_map, idx_size); + if (version < 2 || version > 2) return error("index file %s is version %"PRIu32 " and is not supported by this binary" " (try upgrading GIT to a newer version)", path, version); - } } else version = 1; @@ -120,10 +134,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) index += 2; /* skip index header */ for (i = 0; i < 256; i++) { uint32_t n = ntohl(index[i]); - if (n < nr) { - munmap(idx_map, idx_size); + if (n < nr) return error("non-monotonic index %s", path); - } nr = n; } @@ -135,10 +147,8 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) * - hash of the packfile * - file checksum */ - if (idx_size != 4*256 + nr * (hashsz + 4) + hashsz + hashsz) { - munmap(idx_map, idx_size); + if (idx_size != 4 * 256 + nr * (hashsz + 4) + hashsz + hashsz) return error("wrong index v1 file size in %s", path); - } } else if (version == 2) { /* * Minimum size: @@ -157,20 +167,16 @@ static int check_packed_git_idx(const char *path, struct packed_git *p) unsigned long max_size = min_size; if (nr) max_size += (nr - 1)*8; - if (idx_size < min_size || idx_size > max_size) { - munmap(idx_map, idx_size); + if (idx_size < min_size || idx_size > max_size) return error("wrong index v2 file size in %s", path); - } if (idx_size != min_size && /* * make sure we can deal with large pack offsets. * 31-bit signed offset won't be enough, neither * 32-bit unsigned one will be. */ - (sizeof(off_t) <= 4)) { - munmap(idx_map, idx_size); + (sizeof(off_t) <= 4)) return error("pack too large for current definition of off_t in %s", path); - } } p->index_version = version; diff --git a/packfile.h b/packfile.h index 442625723d..6c4037605d 100644 --- a/packfile.h +++ b/packfile.h @@ -164,4 +164,17 @@ extern int has_pack_index(const unsigned char *sha1); */ extern int is_promisor_object(const struct object_id *oid); +/* + * Expose a function for fuzz testing. + * + * load_idx() parses a block of memory as a packfile index and puts the results + * into a struct packed_git. + * + * This function should not be used directly. It is exposed here only so that we + * have a convenient entry-point for fuzz testing. For real uses, you should + * probably use open_pack_index() or parse_pack_index() instead. + */ +extern int load_idx(const char *path, const unsigned int hashsz, void *idx_map, + size_t idx_size, struct packed_git *p); + #endif -- 2.19.0.605.g01d371f741-goog ^ permalink raw reply related [flat|nested] 10+ messages in thread
* Re: [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon 2018-10-13 0:58 ` [PATCH v2 1/2] fuzz: Add basic fuzz testing target steadmon 2018-10-13 0:58 ` [PATCH v2 2/2] fuzz: Add fuzz testing for packfile indices steadmon @ 2018-10-16 6:18 ` Junio C Hamano 2 siblings, 0 replies; 10+ messages in thread From: Junio C Hamano @ 2018-10-16 6:18 UTC (permalink / raw) To: steadmon; +Cc: git steadmon@google.com writes: > From: Josh Steadmon <steadmon@google.com> > > V2 of this series pulls the compiler flags out of the Makefile, to be > provided by the user depending on the combination of compiler and > fuzzing engine in use. This also makes it more compatible with > OSS-Fuzz's build process. Thanks, will replace. I think we can merge this to 'next' and down and build incrementally on top. ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2018-10-16 6:18 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-10-04 23:01 [RFC PATCH 0/2] add fuzzing targets for use with LLVM libFuzzer Josh Steadmon 2018-10-04 23:01 ` [RFC PATCH 1/2] fuzz: Add basic fuzz testing target Josh Steadmon 2018-10-10 2:14 ` Junio C Hamano 2018-10-13 0:59 ` Josh Steadmon 2018-10-04 23:01 ` [RFC PATCH 2/2] fuzz: Add fuzz testing for packfile indices Josh Steadmon 2018-10-10 2:19 ` Junio C Hamano 2018-10-13 0:58 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz steadmon 2018-10-13 0:58 ` [PATCH v2 1/2] fuzz: Add basic fuzz testing target steadmon 2018-10-13 0:58 ` [PATCH v2 2/2] fuzz: Add fuzz testing for packfile indices steadmon 2018-10-16 6:18 ` [PATCH v2 0/2] add fuzzing targets for use with OSS-Fuzz Junio C Hamano
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).