[bug] Stashes lost after out-of-memory situation

[bug] Stashes lost after out-of-memory situation

[Marek Mrva]

Hello, hopefully this is the correct mailing list - apologies if it is not.

After issuing "git stash pop" while being low on memory, the following 
was printed to the console:

       0 [main] git 2061 fhandler_disk_file::fixup_mmap_after_fork: 
requested 0x6FFFC1550000 != 0x0 mem alloc base 0x0, state 0x10000,
size 17545957736448, Win32 error 1455
   36836 [main] git 2061 C:\cygwin64\bin\git.exe: *** fatal error in 
forked process - recreate_mmaps_after_fork_failed
   37523 [main] git 2061 cygwin_exception::open_stackdumpfile: Dumping 
stack trace to git.exe.stackdump
       0 [main] git 2056 dofork: child -1 - forked process 12100 died 
unexpectedly, retry 0, exit code 0x100, errno 11
error: cannot fork() for status: Resource temporarily unavailable
Dropped refs/stash@{0} (06d44ccc5ed2ac93b370100f481147ae4f0065db)
error: cannot fork() for rev-parse: Resource temporarily unavailable

Afterwards, the result of "git stash list" is empty, even though there 
used to be more than 10+ stashes saved.

Obviously while being low on memory, one should not expect commands to 
run properly. Losing all the *other* stashes could hopefully be somehow 
avoided, if possible. It is worth mentioning this happened in a cygwin 
environment on Windows.

Any help would be greatly appreciated! :)


With best regards,
Marek Mrva

Re: [bug] Stashes lost after out-of-memory situation

[René Scharfe]

Am 23.10.20 um 16:53 schrieb Marek Mrva:
> Hello, hopefully this is the correct mailing list - apologies if it is not.

You came to the right place.

> After issuing "git stash pop" while being low on memory, the following was printed to the console:
>
>       0 [main] git 2061 fhandler_disk_file::fixup_mmap_after_fork: requested 0x6FFFC1550000 != 0x0 mem alloc base 0x0, state 0x10000,
> size 17545957736448, Win32 error 1455
>   36836 [main] git 2061 C:\cygwin64\bin\git.exe: *** fatal error in forked process - recreate_mmaps_after_fork_failed
>   37523 [main] git 2061 cygwin_exception::open_stackdumpfile: Dumping stack trace to git.exe.stackdump
>       0 [main] git 2056 dofork: child -1 - forked process 12100 died unexpectedly, retry 0, exit code 0x100, errno 11
> error: cannot fork() for status: Resource temporarily unavailable
> Dropped refs/stash@{0} (06d44ccc5ed2ac93b370100f481147ae4f0065db)
> error: cannot fork() for rev-parse: Resource temporarily unavailable
>
> Afterwards, the result of "git stash list" is empty, even though there used to be more than 10+ stashes saved.

How horrible!

> Obviously while being low on memory, one should not expect commands
> to run properly. Losing all the *other* stashes could hopefully be
> somehow avoided, if possible.

Of course.

> It is worth mentioning this happened in a cygwin environment on Windows.
>
> Any help would be greatly appreciated! :)

Before any repair attempt please make a backup of the whole repository.

You may be able to recover the lost stacks using git fsck, which can
show dangling commits, i.e. commits that are no longer referenced.
They will be cleaned up eventually by git gc, so avoid running that
until you recovered as many of them as possible.

The manpage of git stash says:

  Recovering stash entries that were cleared/dropped erroneously::

  If you mistakenly drop or clear stash entries, they cannot be recovered
  through the normal safety mechanisms.  However, you can try the
  following incantation to get a list of stash entries that are still in
  your repository, but not reachable any more:

  ----------------------------------------------------------------
  git fsck --unreachable |
  grep commit | cut -d\  -f3 |
  xargs git log --merges --no-walk --grep=WIP
  ----------------------------------------------------------------

You are basically looking for merges with two parents and your stash
message as commit message.  See the manpage for some more details.

You could use them to rebuild .git/logs/refs/stash manually, or to
apply them with git cherry-pick -m1.

Good luck!

So why did this happen?  Looks like stash calls rev-parse to see if a
stash pop removed the last stash and in that case proceeds to delete the
stash ref and its reflog.  A failure of rev-parse is interpreted as no
stashes being left.  This can be triggered by other reasons (like OOM),
so this is dangerously fragile.  Let's make that check more precise.

-- >8 --
Subject: [PATCH] stash: simplify reflog emptiness check

Calling rev-parse to check if the drop subcommand removed the last stash
and treating its failure as confirmation is fragile, as the command can
fail for other reasons, e.g. because the system is out of memory.
Directly check if the reflog is empty instead, which is more robust.

Reported-by: Marek Mrva <mrva@eof-studios.com>
Signed-off-by: René Scharfe <l.s.r@web.de>
---
 builtin/stash.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/builtin/stash.c b/builtin/stash.c
index 3f811f3050..24ddb1bffa 100644
--- a/builtin/stash.c
+++ b/builtin/stash.c
@@ -534,11 +534,22 @@ static int apply_stash(int argc, const char **argv, const char *prefix)
 	return ret;
 }

+static int reject_reflog_ent(struct object_id *ooid, struct object_id *noid,
+			     const char *email, timestamp_t timestamp, int tz,
+			     const char *message, void *cb_data)
+{
+	return 1;
+}
+
+static int reflog_is_empty(const char *refname)
+{
+	return !for_each_reflog_ent(refname, reject_reflog_ent, NULL);
+}
+
 static int do_drop_stash(struct stash_info *info, int quiet)
 {
 	int ret;
 	struct child_process cp_reflog = CHILD_PROCESS_INIT;
-	struct child_process cp = CHILD_PROCESS_INIT;

 	/*
 	 * reflog does not provide a simple function for deleting refs. One will
@@ -559,19 +570,7 @@ static int do_drop_stash(struct stash_info *info, int quiet)
 			     info->revision.buf);
 	}

-	/*
-	 * This could easily be replaced by get_oid, but currently it will throw
-	 * a fatal error when a reflog is empty, which we can not recover from.
-	 */
-	cp.git_cmd = 1;
-	/* Even though --quiet is specified, rev-parse still outputs the hash */
-	cp.no_stdout = 1;
-	strvec_pushl(&cp.args, "rev-parse", "--verify", "--quiet", NULL);
-	strvec_pushf(&cp.args, "%s@{0}", ref_stash);
-	ret = run_command(&cp);
-
-	/* do_clear_stash if we just dropped the last stash entry */
-	if (ret)
+	if (reflog_is_empty(ref_stash))
 		do_clear_stash();

 	return 0;
--
2.28.0

Re: [bug] Stashes lost after out-of-memory situation

[Thomas Braun]

On 10/24/2020 7:06 PM, René Scharfe wrote:

[...]

> Looks like stash calls rev-parse to see if a
> stash pop removed the last stash and in that case proceeds to delete the
> stash ref and its reflog

I was a bit suprised to learn that removing the last stash entry also
removes it from the reflog.

Wouldn't it be more convenient if it would be kept in the reflog even
after popping?

So that in cases like

git init
echo 1 > test
git add test
git commit -m "one" test
echo 2 > test
git stash
git checkout .
git stash pop
git checkout .
git reflog -p

my once stashed change would still be in the reflog?

Re: [bug] Stashes lost after out-of-memory situation

[René Scharfe]

Am 25.10.20 um 13:27 schrieb Thomas Braun:
> On 10/24/2020 7:06 PM, René Scharfe wrote:
>
> [...]
>
>> Looks like stash calls rev-parse to see if a
>> stash pop removed the last stash and in that case proceeds to delete the
>> stash ref and its reflog
>
> I was a bit suprised to learn that removing the last stash entry also
> removes it from the reflog.
>
> Wouldn't it be more convenient if it would be kept in the reflog even
> after popping?

The log of the stash ref is separate from the normal reflog.  Popping
a stash doesn't remove anything from the latter.

René