What's cooking in git.git (Mar 2012, #03; Mon, 5)

What's cooking in git.git (Mar 2012, #03; Mon, 5)

[Junio C Hamano]

Here are the topics that have been cooking.  Commits prefixed with '-' are
only in 'pu' (proposed updates) while commits prefixed with '+' are in 'next'.

The tip of 'next' as of tonight is more or less how 1.7.10-rc0 will
look like in a few days.  I do not have enough confidence on other
large-ish topics that are not yet in 'next' to include in them in
the upcoming release.

Small, trivially correct topics that have high value/damage ratio
however may have enough time to go in 1.7.10 final even if they are
still in 'pu', if the owners put enough effort in them.  Otherwise
let's throw them into Stalled category and leave them to the next
cycle.

You can find the changes described here in the integration branches of the
repositories listed at

    http://git-blame.blogspot.com/p/git-public-repositories.html

--------------------------------------------------
[New Topics]

* jc/fmt-merge-msg-people (2012-03-05) 1 commit
  (merged to 'next' on 2012-03-05 at 38de349)
 + fmt-merge-msg: show those involved in a merged series

The "fmt-merge-msg" command learns to list the primary contributors
involved in the side topic you are merging.

* nl/http-proxy-more (2012-03-05) 3 commits
 - http: handle proxy authentication failure (error 407)
 - http: handle proxy proactive authentication
 - http: try http_proxy env var when http.proxy config option is not set
 (this branch uses nl/http-proxy-auth.)

The code to talk to http proxies learn to use the same credential
API used to talk to the final http destinations.

* nd/stream-more (2012-03-05) 6 commits
 - fsck: use streaming API for writing lost-found blobs
 - show: use streaming API for showing blobs
 - parse_object: avoid putting whole blob in core
 - cat-file: use streaming API to print blobs
 - Add more large blob test cases
 - streaming: make streaming-write-entry to be more reusable

As I do not think changes in the original larger series that touch
index-pack and unpack-objects will be ready for the upcoming 1.7.10,
I've picked up only bits that deal with "use streaming API instead
of reading things in core" and minimally fixed them up.  These we
should be able to polish in time.

The "test cases" patch needs to be trimmed to cover only these, I
would think.

--------------------------------------------------
[Stalled]

* nd/optim-connected (2012-02-29) 1 commit
 - Perform cheaper connectivity check when pack is used as medium

Cheats the local connectivity check performed by "git fetch"
slightly to gain some performance. This is a bit iffy.

* jc/diff-ignore-mode (2012-03-01) 1 commit
 - diff --ignore-mode-change
 (this branch uses jc/maint-diff-patch-header.)

Will discard.

* nd/columns (2012-02-28) 10 commits
 - tag: add --column
 - column: support piping stdout to external git-column process
 - status: add --column
 - branch: add --column
 - help: reuse print_columns() for help -a
 - column: add column.ui for default column output settings
 - column: add dense layout support
 - column: add columnar layout
 - Stop starting pager recursively
 - Add git-column for columnar display

Rerolled; the configuration handling looked iffy, but otherwise well
explained.

* jc/diff-ignore-case (2012-02-28) 6 commits
 - diff: -i is "--ignore-case" but means a bit more in "log"
 - diff: --ignore-case
 - xdiff: introduce XDF_IGNORE_CASE
 - xdiff: introduce XDF_INEXACT_MATCH
 - xdiff: PATIENCE/HISTOGRAM are not independent option bits
 - xdiff: remove XDL_PATCH_* macros

"git diff" learns "--ignore-case" option.
Will discard, as nobody seems to need this.

* hv/submodule-recurse-push (2012-02-13) 3 commits
 - push: teach --recurse-submodules the on-demand option
 - Refactor submodule push check to use string list instead of integer
 - Teach revision walking machinery to walk multiple times sequencially

The bottom one was not clearly explained and needs a reroll.

* jc/advise-push-default (2011-12-18) 1 commit
 - push: hint to use push.default=upstream when appropriate

Peff had a good suggestion outlining an updated code structure so
that somebody new can try to dip his or her toes in the
development. Any takers?

* jh/trace-use-startup-info (2012-03-02) 1 commit
 - Use startup_info->prefix rather than prefix.

I tend to agree with the doubt of the author of this patch expressed.

* ss/git-svn-prompt-sans-terminal (2012-01-04) 3 commits
 - fixup! 15eaaf4
 - git-svn, perl/Git.pm: extend Git::prompt helper for querying users
 - perl/Git.pm: "prompt" helper to honor GIT_ASKPASS and SSH_ASKPASS

The bottom one has been replaced with a rewrite based on comments
from Ævar. The second one needs more work, both in perl/Git.pm and
prompt.c, to give precedence to tty over SSH_ASKPASS when terminal
is available.

* jc/split-blob (2012-02-23) 7 commits
 - fixup?
 - chunked-object: streaming checkout
 - chunked-object: fallback checkout codepaths
 - bulk-checkin: support chunked-object encoding
 - bulk-checkin: allow the same data to be multiply hashed
 - new representation types in the packstream
 - varint-in-pack: refactor varint encoding/decoding

Not ready.

I finished the streaming checkout codepath, but as explained in
127b177 (bulk-checkin: support chunked-object encoding, 2011-11-30),
these are still early steps of a long and painful journey. At least
pack-objects and fsck need to learn the new encoding for the series
to be usable locally, and then index-pack/unpack-objects needs to
learn it to be used remotely.

Given that I heard a lot of noise that people want large files, and
that I was asked by somebody at GitTogether'11 privately for an
advice on how to pay developers (not me) to help adding necessary
support, I am somewhat dissapointed that the original patch series
that was sent almost two months ago still remains here without much
comments and updates from the developer community. I even made the
interface to the logic that decides where to split chunks easily
replaceable, and I deliberately made the logic in the original patch
extremely stupid to entice others, especially the "bup" fanboys, to
come up with a better logic, thinking that giving people an easy
target to shoot for, they may be encouraged to help out. The plan is
not working :-(.

--------------------------------------------------
[Cooking]

* cn/pull-rebase-message (2012-03-04) 1 commit
  (merged to 'next' on 2012-03-04 at 5a6cd58)
 + Make git-{pull,rebase} message without tracking information friendlier

The advise message given when the user didn't give enough clue on what
to merge was overly long.
Will merge to 'master'.

* dg/test-from-elsewhere (2012-03-04) 2 commits
 - Support out-of-tree Valgrind tests
 - Allow overriding GIT_BUILD_DIR

Better support for out-of-tree test scripts.

* jh/threadable-symlink-check (2012-03-02) 1 commit
  (merged to 'next' on 2012-03-05 at fdd667c)
 + Add threaded versions of functions in symlinks.c.

It probably is "threadble" not "threaded" but in any case this
should not regress the current callers.

* jn/maint-do-not-match-with-unsanitized-searchtext (2012-03-04) 1 commit
 - gitweb: Fix fixed string (non-regexp) project search

"gitweb" did use quotemeta() to prepare search string when asked to
do a fixed-string project search, but did not use it by mistake and
used the user-supplied string instead.

I would love to see this in 1.7.10; what I queued for 'maint' (that
does not have the lazy fill-info stuff), and my conflict resolution
into 'pu' to adjust for lazy fill-info, are both correct, I think,
but please double check.

* nl/http-proxy-auth (2012-03-02) 1 commit
  (merged to 'next' on 2012-03-04 at f368669)
 + http: support proxies that require authentication
 (this branch is used by nl/http-proxy-more.)

Allow curl-based transport to use proxies that require authentication.
Will merge to 'master'.

* sl/modern-t0000 (2012-03-02) 1 commit
  (merged to 'next' on 2012-03-04 at 2cd1508)
 + t0000: modernise style

Will merge to 'master'.

* th/mergetools-deltawalker (2012-03-05) 1 commit
  (merged to 'next' on 2012-03-05 at 167c74f)
 + mergetools: add a plug-in to support DeltaWalker

Add plug-in to support DeltaWalker in difftool/mergetool.  
Will merge to 'master'.

* jl/maint-submodule-relative (2012-03-04) 4 commits
 - submodules: fix ambiguous absolute paths under Windows
 - submodules: refactor computation of relative gitdir path
 - submodules: always use a relative path from gitdir to work tree
 - submodules: always use a relative path to gitdir

Rerolled and resurrected from Stalled category.

I would love to have this in 1.7.10; a couple of Acks from different
platforms and success reports would be very helpful.

* nd/threaded-index-pack (2012-03-02) 2 commits
 - index-pack: support multithreaded delta resolving
 - index-pack: split second pass obj handling into own function

Rerolled to fix a bug that was discovered with a trivial test, and
resurrected from Stalled category, but I am reluctant to advance a
topic that can easily break many people's repositories at this late
in the cycle.

* jc/maint-diff-patch-header (2012-03-01) 3 commits
  (merged to 'next' on 2012-03-02 at 2cd2059)
 + diff -p: squelch "diff --git" header for stat-dirty paths
 + t4011: illustrate "diff-index -p" on stat-dirty paths
 + t4011: modernise style
 (this branch is used by jc/diff-ignore-mode.)

"diff-index" and friends showed "diff --git" header and nothing else
for a path that is only stat-dirty.

Should be safe, but is not urgent.

* tr/maint-bundle-boundary (2012-03-01) 3 commits
  (merged to 'next' on 2012-03-02 at c25692f)
 + bundle: keep around names passed to add_pending_object()
 + t5510: ensure we stay in the toplevel test dir
 + t5510: refactor bundle->pack conversion

"git bundle" did not record boundary commits correctly when there
are many of them.

Will merge to 'master'.

* jc/pickaxe-ignore-case (2012-03-04) 3 commits
  (merged to 'next' on 2012-03-04 at 1ad3a39)
 + ctype.c: Fix a sparse warning
  (merged to 'next' on 2012-03-02 at 73a632b)
 + pickaxe: allow -i to search in patch case-insensitively
 + grep: use static trans-case table

"git log -G/-S" pays attention to the "-i" option.

Will merge to 'master'.

* zj/diff-stat-dyncol (2012-03-01) 10 commits
  (merged to 'next' on 2012-03-01 at 79b4152)
 + diff --stat: add config option to limit graph width
 + diff --stat: enable limiting of the graph part
 + diff --stat: add a test for output with COLUMNS=40
 + diff --stat: use a maximum of 5/8 for the filename part
 + merge --stat: use the full terminal width
 + log --stat: use the full terminal width
 + show --stat: use the full terminal width
 + diff --stat: use the full terminal width
 + diff --stat: tests for long filenames and big change counts
 + Merge branches zj/decimal-width, zj/term-columns and jc/diff-stat-scaler

Rerolled again and looking good.
Will merge to 'master'.

Re: What's cooking in git.git (Mar 2012, #03; Mon, 5)

[Nguyen Thai Ngoc Duy]

On Tue, Mar 6, 2012 at 2:15 PM, Junio C Hamano <gitster@pobox.com> wrote:
> * nd/stream-more (2012-03-05) 6 commits
>  - fsck: use streaming API for writing lost-found blobs
>  - show: use streaming API for showing blobs
>  - parse_object: avoid putting whole blob in core
>  - cat-file: use streaming API to print blobs
>  - Add more large blob test cases
>  - streaming: make streaming-write-entry to be more reusable
>
> As I do not think changes in the original larger series that touch
> index-pack and unpack-objects will be ready for the upcoming 1.7.10,
> I've picked up only bits that deal with "use streaming API instead
> of reading things in core" and minimally fixed them up.  These we
> should be able to polish in time.

11/11 should be safe to go too. It simply reads config files so that
parse_object() respects core.bigfilethreshold.

http://mid.gmane.org/1330919028-6611-12-git-send-email-pclouds@gmail.com

> The "test cases" patch needs to be trimmed to cover only these, I
> would think.

I'd rather leave it as is. All new test cases are test_expect_failure,
they should not interrupt "make test". If I slack off, somebody may be
annoyed enough with those known breakages to give me a little push.
-- 
Duy

Re: What's cooking in git.git (Mar 2012, #03; Mon, 5)

[Junio C Hamano]

Nguyen Thai Ngoc Duy <pclouds@gmail.com> writes:

>> I've picked up only bits that deal with "use streaming API instead
>> of reading things in core" and minimally fixed them up.  These we
>> should be able to polish in time.
>
> 11/11 should be safe to go too. It simply reads config files so that
> parse_object() respects core.bigfilethreshold.

Ok.

> I'd rather leave it as is. All new test cases are test_expect_failure,
> they should not interrupt "make test". If I slack off, somebody may be
> annoyed enough with those known breakages to give me a little push.

That requires us to vet and agree that the conditions marked with
expect_failure are indeed breakages, which I would want avoid. It is
frustrating for people who has to later "fix" things and find out
that the original "bug" was marked without thinking things through.

In preparation for 1.7.10-rc0

[Junio C Hamano]

There are a couple loose ends I would like to tie before we go deep
into the pre-release freeze for the upcoming 1.7.10.

* Subsystems

I recall that I received many months worth of accumulated gitk
updates during the last round from Paul.

Are there updates for this cycle coming soon?  The same question for
git-gui (Pat) and git-svn (Eric)?

* L10N

Immediately after 1.7.10-rc0 gets tagged may be a good time for the
l10n coordinator to pull from me, run "make pot" to update the
po/git.pot file and propagate the result to the language teams (oh,
we would also need to recruit language teams other than Chinese ;-),
so that the translations can be updated (or added).

Thanks.

[PATCH 0/7] nd/stream-more updates

[Nguyễn Thái Ngọc Duy]

As discussed, this series removes test_expect_failure that are still
not fixed in the end. It also adds update-server-info patch. I'll hold
off the rest of the previous series until 1.7.10 comes out.

Junio C Hamano (1):
  streaming: make streaming-write-entry to be more reusable

Nguyễn Thái Ngọc Duy (6):
  Add more large blob test cases
  cat-file: use streaming API to print blobs
  parse_object: avoid putting whole blob in core
  show: use streaming API for showing blobs
  fsck: use streaming API for writing lost-found blobs
  update-server-info: respect core.bigfilethreshold

 builtin/cat-file.c           |   25 +++++++++++++++++++
 builtin/fsck.c               |    8 +----
 builtin/log.c                |   34 +++++++++++++++----------
 builtin/update-server-info.c |    1 +
 entry.c                      |   53 ++++------------------------------------
 object.c                     |   11 ++++++++
 sha1_file.c                  |   42 ++++++++++++++++++++++++++++++-
 streaming.c                  |   55 ++++++++++++++++++++++++++++++++++++++++++
 streaming.h                  |    2 +
 t/t1050-large.sh             |   38 +++++++++++++++++++++++++++-
 wrapper.c                    |   27 ++++++++++++++++++--
 11 files changed, 221 insertions(+), 75 deletions(-)

-- 
1.7.8.36.g69ee2

[PATCH 1/7] streaming: make streaming-write-entry to be more reusable

[Nguyễn Thái Ngọc Duy]

From: Junio C Hamano <gitster@pobox.com>

The static function in entry.c takes a cache entry and streams its blob
contents to a file in the working tree.  Refactor the logic to a new API
function stream_blob_to_fd() that takes an object name and an open file
descriptor, so that it can be reused by other callers.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 entry.c     |   53 +++++------------------------------------------------
 streaming.c |   55 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 streaming.h |    2 ++
 3 files changed, 62 insertions(+), 48 deletions(-)

diff --git a/entry.c b/entry.c
index 852fea1..17a6bcc 100644
--- a/entry.c
+++ b/entry.c
@@ -120,58 +120,15 @@ static int streaming_write_entry(struct cache_entry *ce, char *path,
 				 const struct checkout *state, int to_tempfile,
 				 int *fstat_done, struct stat *statbuf)
 {
-	struct git_istream *st;
-	enum object_type type;
-	unsigned long sz;
 	int result = -1;
-	ssize_t kept = 0;
-	int fd = -1;
-
-	st = open_istream(ce->sha1, &type, &sz, filter);
-	if (!st)
-		return -1;
-	if (type != OBJ_BLOB)
-		goto close_and_exit;
+	int fd;
 
 	fd = open_output_fd(path, ce, to_tempfile);
-	if (fd < 0)
-		goto close_and_exit;
-
-	for (;;) {
-		char buf[1024 * 16];
-		ssize_t wrote, holeto;
-		ssize_t readlen = read_istream(st, buf, sizeof(buf));
-
-		if (!readlen)
-			break;
-		if (sizeof(buf) == readlen) {
-			for (holeto = 0; holeto < readlen; holeto++)
-				if (buf[holeto])
-					break;
-			if (readlen == holeto) {
-				kept += holeto;
-				continue;
-			}
-		}
-
-		if (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)
-			goto close_and_exit;
-		else
-			kept = 0;
-		wrote = write_in_full(fd, buf, readlen);
-
-		if (wrote != readlen)
-			goto close_and_exit;
-	}
-	if (kept && (lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1 ||
-		     write(fd, "", 1) != 1))
-		goto close_and_exit;
-	*fstat_done = fstat_output(fd, state, statbuf);
-
-close_and_exit:
-	close_istream(st);
-	if (0 <= fd)
+	if (0 <= fd) {
+		result = stream_blob_to_fd(fd, ce->sha1, filter, 1);
+		*fstat_done = fstat_output(fd, state, statbuf);
 		result = close(fd);
+	}
 	if (result && 0 <= fd)
 		unlink(path);
 	return result;
diff --git a/streaming.c b/streaming.c
index 71072e1..7e7ee2b 100644
--- a/streaming.c
+++ b/streaming.c
@@ -489,3 +489,58 @@ static open_method_decl(incore)
 
 	return st->u.incore.buf ? 0 : -1;
 }
+
+
+/****************************************************************
+ * Users of streaming interface
+ ****************************************************************/
+
+int stream_blob_to_fd(int fd, unsigned const char *sha1, struct stream_filter *filter,
+		      int can_seek)
+{
+	struct git_istream *st;
+	enum object_type type;
+	unsigned long sz;
+	ssize_t kept = 0;
+	int result = -1;
+
+	st = open_istream(sha1, &type, &sz, filter);
+	if (!st)
+		return result;
+	if (type != OBJ_BLOB)
+		goto close_and_exit;
+	for (;;) {
+		char buf[1024 * 16];
+		ssize_t wrote, holeto;
+		ssize_t readlen = read_istream(st, buf, sizeof(buf));
+
+		if (!readlen)
+			break;
+		if (can_seek && sizeof(buf) == readlen) {
+			for (holeto = 0; holeto < readlen; holeto++)
+				if (buf[holeto])
+					break;
+			if (readlen == holeto) {
+				kept += holeto;
+				continue;
+			}
+		}
+
+		if (kept && lseek(fd, kept, SEEK_CUR) == (off_t) -1)
+			goto close_and_exit;
+		else
+			kept = 0;
+		wrote = write_in_full(fd, buf, readlen);
+
+		if (wrote != readlen)
+			goto close_and_exit;
+	}
+	if (kept && (lseek(fd, kept - 1, SEEK_CUR) == (off_t) -1 ||
+		     write(fd, "", 1) != 1))
+		goto close_and_exit;
+	result = 0;
+
+ close_and_exit:
+	close_istream(st);
+	return result;
+}
diff --git a/streaming.h b/streaming.h
index 589e857..3e82770 100644
--- a/streaming.h
+++ b/streaming.h
@@ -12,4 +12,6 @@ extern struct git_istream *open_istream(const unsigned char *, enum object_type
 extern int close_istream(struct git_istream *);
 extern ssize_t read_istream(struct git_istream *, char *, size_t);
 
+extern int stream_blob_to_fd(int fd, const unsigned char *, struct stream_filter *, int can_seek);
+
 #endif /* STREAMING_H */
-- 
1.7.8.36.g69ee2

[PATCH 2/7] Add more large blob test cases

[Nguyễn Thái Ngọc Duy]

New test cases list commands that should work when memory is
limited. All memory allocation functions (*) learn to reject any
allocation larger than $GIT_ALLOC_LIMIT if set.

(*) Not exactly all. Some places do not use x* functions, but
malloc/calloc directly, notably diff-delta. These code path should
never be run on large blobs.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 t/t1050-large.sh |   38 ++++++++++++++++++++++++++++++++++++--
 wrapper.c        |   27 ++++++++++++++++++++++++---
 2 files changed, 60 insertions(+), 5 deletions(-)

diff --git a/t/t1050-large.sh b/t/t1050-large.sh
index 29d6024..ded66b3 100755
--- a/t/t1050-large.sh
+++ b/t/t1050-large.sh
@@ -6,11 +6,15 @@ test_description='adding and checking out large blobs'
 . ./test-lib.sh
 
 test_expect_success setup '
-	git config core.bigfilethreshold 200k &&
+	# clone does not allow us to pass core.bigfilethreshold to
+	# new repos, so set core.bigfilethreshold globally
+	git config --global core.bigfilethreshold 200k &&
 	echo X | dd of=large1 bs=1k seek=2000 &&
 	echo X | dd of=large2 bs=1k seek=2000 &&
 	echo X | dd of=large3 bs=1k seek=2000 &&
-	echo Y | dd of=huge bs=1k seek=2500
+	echo Y | dd of=huge bs=1k seek=2500 &&
+	GIT_ALLOC_LIMIT=1500 &&
+	export GIT_ALLOC_LIMIT
 '
 
 test_expect_success 'add a large file or two' '
@@ -100,4 +104,34 @@ test_expect_success 'packsize limit' '
 	)
 '
 
+test_expect_success 'diff --raw' '
+	git commit -q -m initial &&
+	echo modified >>large1 &&
+	git add large1 &&
+	git commit -q -m modified &&
+	git diff --raw HEAD^
+'
+
+test_expect_success 'hash-object' '
+	git hash-object large1
+'
+
+test_expect_failure 'cat-file a large file' '
+	git cat-file blob :large1 >/dev/null
+'
+
+test_expect_failure 'cat-file a large file from a tag' '
+	git tag -m largefile largefiletag :large1 &&
+	git cat-file blob largefiletag >/dev/null
+'
+
+test_expect_failure 'git-show a large file' '
+	git show :large1 >/dev/null
+
+'
+
+test_expect_failure 'repack' '
+	git repack -ad
+'
+
 test_done
diff --git a/wrapper.c b/wrapper.c
index 85f09df..6ccd059 100644
--- a/wrapper.c
+++ b/wrapper.c
@@ -9,6 +9,18 @@ static void do_nothing(size_t size)
 
 static void (*try_to_free_routine)(size_t size) = do_nothing;
 
+static void memory_limit_check(size_t size)
+{
+	static int limit = -1;
+	if (limit == -1) {
+		const char *env = getenv("GIT_ALLOC_LIMIT");
+		limit = env ? atoi(env) * 1024 : 0;
+	}
+	if (limit && size > limit)
+		die("attempting to allocate %"PRIuMAX" over limit %d",
+		    (intmax_t)size, limit);
+}
+
 try_to_free_t set_try_to_free_routine(try_to_free_t routine)
 {
 	try_to_free_t old = try_to_free_routine;
@@ -32,7 +44,10 @@ char *xstrdup(const char *str)
 
 void *xmalloc(size_t size)
 {
-	void *ret = malloc(size);
+	void *ret;
+
+	memory_limit_check(size);
+	ret = malloc(size);
 	if (!ret && !size)
 		ret = malloc(1);
 	if (!ret) {
@@ -79,7 +94,10 @@ char *xstrndup(const char *str, size_t len)
 
 void *xrealloc(void *ptr, size_t size)
 {
-	void *ret = realloc(ptr, size);
+	void *ret;
+
+	memory_limit_check(size);
+	ret = realloc(ptr, size);
 	if (!ret && !size)
 		ret = realloc(ptr, 1);
 	if (!ret) {
@@ -95,7 +113,10 @@ void *xrealloc(void *ptr, size_t size)
 
 void *xcalloc(size_t nmemb, size_t size)
 {
-	void *ret = calloc(nmemb, size);
+	void *ret;
+
+	memory_limit_check(size * nmemb);
+	ret = calloc(nmemb, size);
 	if (!ret && (!nmemb || !size))
 		ret = calloc(1, 1);
 	if (!ret) {
-- 
1.7.8.36.g69ee2

[PATCH 3/7] cat-file: use streaming API to print blobs

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/cat-file.c |   25 +++++++++++++++++++++++++
 t/t1050-large.sh   |    4 ++--
 2 files changed, 27 insertions(+), 2 deletions(-)

diff --git a/builtin/cat-file.c b/builtin/cat-file.c
index 8ed501f..36a9104 100644
--- a/builtin/cat-file.c
+++ b/builtin/cat-file.c
@@ -11,6 +11,7 @@
 #include "parse-options.h"
 #include "diff.h"
 #include "userdiff.h"
+#include "streaming.h"
 
 #define BATCH 1
 #define BATCH_CHECK 2
@@ -127,6 +128,8 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name)
 			return cmd_ls_tree(2, ls_args, NULL);
 		}
 
+		if (type == OBJ_BLOB)
+			return stream_blob_to_fd(1, sha1, NULL, 0);
 		buf = read_sha1_file(sha1, &type, &size);
 		if (!buf)
 			die("Cannot read object %s", obj_name);
@@ -149,6 +152,28 @@ static int cat_one_file(int opt, const char *exp_type, const char *obj_name)
 		break;
 
 	case 0:
+		if (type_from_string(exp_type) == OBJ_BLOB) {
+			unsigned char blob_sha1[20];
+			if (sha1_object_info(sha1, NULL) == OBJ_TAG) {
+				enum object_type type;
+				unsigned long size;
+				char *buffer = read_sha1_file(sha1, &type, &size);
+				if (memcmp(buffer, "object ", 7) ||
+				    get_sha1_hex(buffer + 7, blob_sha1))
+					die("%s not a valid tag", sha1_to_hex(sha1));
+				free(buffer);
+			} else
+				hashcpy(blob_sha1, sha1);
+
+			if (sha1_object_info(blob_sha1, NULL) == OBJ_BLOB)
+				return stream_blob_to_fd(1, blob_sha1, NULL, 0);
+			/*
+			 * we attempted to dereference a tag to a blob
+			 * and failed; there may be new dereference
+			 * mechanisms this code is not aware of.
+			 * fall-back to the usual case.
+			 */
+		}
 		buf = read_object_with_reference(sha1, exp_type, &size, NULL);
 		break;
 
diff --git a/t/t1050-large.sh b/t/t1050-large.sh
index ded66b3..f662fef 100755
--- a/t/t1050-large.sh
+++ b/t/t1050-large.sh
@@ -116,11 +116,11 @@ test_expect_success 'hash-object' '
 	git hash-object large1
 '
 
-test_expect_failure 'cat-file a large file' '
+test_expect_success 'cat-file a large file' '
 	git cat-file blob :large1 >/dev/null
 '
 
-test_expect_failure 'cat-file a large file from a tag' '
+test_expect_success 'cat-file a large file from a tag' '
 	git tag -m largefile largefiletag :large1 &&
 	git cat-file blob largefiletag >/dev/null
 '
-- 
1.7.8.36.g69ee2

[PATCH 4/7] parse_object: avoid putting whole blob in core

[Nguyễn Thái Ngọc Duy]

Traditionally, all the callers of check_sha1_signature() first
called read_sha1_file() to prepare the whole object data in core,
and called this function.  The function is used to revalidate what
we read from the object database actually matches the object name we
used to ask for the data from the object database.

Update the API to allow callers to pass NULL as the object data, and
have the function read and hash the object data using streaming API
to recompute the object name, without having to hold everything in
core at the same time.  This is most useful in parse_object() that
parses a blob object, because this caller does not have to keep the
actual blob data around in memory after a "struct blob" is returned.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 object.c    |   11 +++++++++++
 sha1_file.c |   42 ++++++++++++++++++++++++++++++++++++++++--
 2 files changed, 51 insertions(+), 2 deletions(-)

diff --git a/object.c b/object.c
index 6b06297..0498b18 100644
--- a/object.c
+++ b/object.c
@@ -198,6 +198,17 @@ struct object *parse_object(const unsigned char *sha1)
 	if (obj && obj->parsed)
 		return obj;
 
+	if ((obj && obj->type == OBJ_BLOB) ||
+	    (!obj && has_sha1_file(sha1) &&
+	     sha1_object_info(sha1, NULL) == OBJ_BLOB)) {
+		if (check_sha1_signature(repl, NULL, 0, NULL) < 0) {
+			error("sha1 mismatch %s\n", sha1_to_hex(repl));
+			return NULL;
+		}
+		parse_blob_buffer(lookup_blob(sha1), NULL, 0);
+		return lookup_object(sha1);
+	}
+
 	buffer = read_sha1_file(sha1, &type, &size);
 	if (buffer) {
 		if (check_sha1_signature(repl, buffer, size, typename(type)) < 0) {
diff --git a/sha1_file.c b/sha1_file.c
index 4f06a0e..ad314f0 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -19,6 +19,7 @@
 #include "pack-revindex.h"
 #include "sha1-lookup.h"
 #include "bulk-checkin.h"
+#include "streaming.h"
 
 #ifndef O_NOATIME
 #if defined(__linux__) && (defined(__i386__) || defined(__PPC__))
@@ -1146,10 +1147,47 @@ static const struct packed_git *has_packed_and_bad(const unsigned char *sha1)
 	return NULL;
 }
 
-int check_sha1_signature(const unsigned char *sha1, void *map, unsigned long size, const char *type)
+/*
+ * With an in-core object data in "map", rehash it to make sure the
+ * object name actually matches "sha1" to detect object corruption.
+ * With "map" == NULL, try reading the object named with "sha1" using
+ * the streaming interface and rehash it to do the same.
+ */
+int check_sha1_signature(const unsigned char *sha1, void *map,
+			 unsigned long size, const char *type)
 {
 	unsigned char real_sha1[20];
-	hash_sha1_file(map, size, type, real_sha1);
+	enum object_type obj_type;
+	struct git_istream *st;
+	git_SHA_CTX c;
+	char hdr[32];
+	int hdrlen;
+
+	if (map) {
+		hash_sha1_file(map, size, type, real_sha1);
+		return hashcmp(sha1, real_sha1) ? -1 : 0;
+	}
+
+	st = open_istream(sha1, &obj_type, &size, NULL);
+	if (!st)
+		return -1;
+
+	/* Generate the header */
+	hdrlen = sprintf(hdr, "%s %lu", typename(obj_type), size) + 1;
+
+	/* Sha1.. */
+	git_SHA1_Init(&c);
+	git_SHA1_Update(&c, hdr, hdrlen);
+	for (;;) {
+		char buf[1024 * 16];
+		ssize_t readlen = read_istream(st, buf, sizeof(buf));
+
+		if (!readlen)
+			break;
+		git_SHA1_Update(&c, buf, readlen);
+	}
+	git_SHA1_Final(real_sha1, &c);
+	close_istream(st);
 	return hashcmp(sha1, real_sha1) ? -1 : 0;
 }
 
-- 
1.7.8.36.g69ee2

[PATCH 5/7] show: use streaming API for showing blobs

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/log.c    |   34 ++++++++++++++++++++--------------
 t/t1050-large.sh |    2 +-
 2 files changed, 21 insertions(+), 15 deletions(-)

diff --git a/builtin/log.c b/builtin/log.c
index 7d1f6f8..d1702e7 100644
--- a/builtin/log.c
+++ b/builtin/log.c
@@ -20,6 +20,7 @@
 #include "string-list.h"
 #include "parse-options.h"
 #include "branch.h"
+#include "streaming.h"
 
 /* Set a default date-time format for git log ("log.date" config variable) */
 static const char *default_date_mode = NULL;
@@ -381,8 +382,13 @@ static void show_tagger(char *buf, int len, struct rev_info *rev)
 	strbuf_release(&out);
 }
 
-static int show_object(const unsigned char *sha1, int show_tag_object,
-	struct rev_info *rev)
+static int show_blob_object(const unsigned char *sha1, struct rev_info *rev)
+{
+	fflush(stdout);
+	return stream_blob_to_fd(1, sha1, NULL, 0);
+}
+
+static int show_tag_object(const unsigned char *sha1, struct rev_info *rev)
 {
 	unsigned long size;
 	enum object_type type;
@@ -392,16 +398,16 @@ static int show_object(const unsigned char *sha1, int show_tag_object,
 	if (!buf)
 		return error(_("Could not read object %s"), sha1_to_hex(sha1));
 
-	if (show_tag_object)
-		while (offset < size && buf[offset] != '\n') {
-			int new_offset = offset + 1;
-			while (new_offset < size && buf[new_offset++] != '\n')
-				; /* do nothing */
-			if (!prefixcmp(buf + offset, "tagger "))
-				show_tagger(buf + offset + 7,
-					    new_offset - offset - 7, rev);
-			offset = new_offset;
-		}
+	assert(type == OBJ_TAG);
+	while (offset < size && buf[offset] != '\n') {
+		int new_offset = offset + 1;
+		while (new_offset < size && buf[new_offset++] != '\n')
+			; /* do nothing */
+		if (!prefixcmp(buf + offset, "tagger "))
+			show_tagger(buf + offset + 7,
+				    new_offset - offset - 7, rev);
+		offset = new_offset;
+	}
 
 	if (offset < size)
 		fwrite(buf + offset, size - offset, 1, stdout);
@@ -459,7 +465,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)
 		const char *name = objects[i].name;
 		switch (o->type) {
 		case OBJ_BLOB:
-			ret = show_object(o->sha1, 0, NULL);
+			ret = show_blob_object(o->sha1, NULL);
 			break;
 		case OBJ_TAG: {
 			struct tag *t = (struct tag *)o;
@@ -470,7 +476,7 @@ int cmd_show(int argc, const char **argv, const char *prefix)
 					diff_get_color_opt(&rev.diffopt, DIFF_COMMIT),
 					t->tag,
 					diff_get_color_opt(&rev.diffopt, DIFF_RESET));
-			ret = show_object(o->sha1, 1, &rev);
+			ret = show_tag_object(o->sha1, &rev);
 			rev.shown_one = 1;
 			if (ret)
 				break;
diff --git a/t/t1050-large.sh b/t/t1050-large.sh
index f662fef..dd1bb84 100755
--- a/t/t1050-large.sh
+++ b/t/t1050-large.sh
@@ -125,7 +125,7 @@ test_expect_success 'cat-file a large file from a tag' '
 	git cat-file blob largefiletag >/dev/null
 '
 
-test_expect_failure 'git-show a large file' '
+test_expect_success 'git-show a large file' '
 	git show :large1 >/dev/null
 
 '
-- 
1.7.8.36.g69ee2

[PATCH 6/7] fsck: use streaming API for writing lost-found blobs

[Nguyễn Thái Ngọc Duy]

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/fsck.c |    8 ++------
 1 files changed, 2 insertions(+), 6 deletions(-)

diff --git a/builtin/fsck.c b/builtin/fsck.c
index 67eb553..a710227 100644
--- a/builtin/fsck.c
+++ b/builtin/fsck.c
@@ -12,6 +12,7 @@
 #include "parse-options.h"
 #include "dir.h"
 #include "progress.h"
+#include "streaming.h"
 
 #define REACHABLE 0x0001
 #define SEEN      0x0002
@@ -238,13 +239,8 @@ static void check_unreachable_object(struct object *obj)
 			if (!(f = fopen(filename, "w")))
 				die_errno("Could not open '%s'", filename);
 			if (obj->type == OBJ_BLOB) {
-				enum object_type type;
-				unsigned long size;
-				char *buf = read_sha1_file(obj->sha1,
-						&type, &size);
-				if (buf && fwrite(buf, 1, size, f) != size)
+				if (stream_blob_to_fd(fileno(f), obj->sha1, NULL, 1))
 					die_errno("Could not write '%s'", filename);
-				free(buf);
 			} else
 				fprintf(f, "%s\n", sha1_to_hex(obj->sha1));
 			if (fclose(f))
-- 
1.7.8.36.g69ee2

[PATCH 7/7] update-server-info: respect core.bigfilethreshold

[Nguyễn Thái Ngọc Duy]

This command indirectly calls check_sha1_signature() (add_info_ref ->
deref_tag -> parse_object -> ..) , which may put whole blob in memory
if the blob's size is under core.bigfilethreshold. As config is not
read, the threshold is always 512MB. Respect user settings here.

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
---
 builtin/update-server-info.c |    1 +
 t/t1050-large.sh             |    2 +-
 2 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/builtin/update-server-info.c b/builtin/update-server-info.c
index b90dce6..0d63c44 100644
--- a/builtin/update-server-info.c
+++ b/builtin/update-server-info.c
@@ -15,6 +15,7 @@ int cmd_update_server_info(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 
+	git_config(git_default_config, NULL);
 	argc = parse_options(argc, argv, prefix, options,
 			     update_server_info_usage, 0);
 	if (argc > 0)
diff --git a/t/t1050-large.sh b/t/t1050-large.sh
index dd1bb84..4d127f1 100755
--- a/t/t1050-large.sh
+++ b/t/t1050-large.sh
@@ -130,7 +130,7 @@ test_expect_success 'git-show a large file' '
 
 '
 
-test_expect_failure 'repack' '
+test_expect_success 'repack' '
 	git repack -ad
 '
 
-- 
1.7.8.36.g69ee2

Re: [PATCH 0/7] nd/stream-more updates

[Junio C Hamano]

Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:

> As discussed, this series removes test_expect_failure that are still
> not fixed in the end. It also adds update-server-info patch. I'll hold
> off the rest of the previous series until 1.7.10 comes out.

Thanks; all look good.  Will queue and let's aim for 1.7.10-rc1 if
not earlier.