* safer git?
@ 2020-10-24 20:19 hv
2020-10-24 20:21 ` hv
2020-10-24 22:11 ` Randall S. Becker
0 siblings, 2 replies; 6+ messages in thread
From: hv @ 2020-10-24 20:19 UTC (permalink / raw)
To: git; +Cc: hv
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]
Hi, due to a hardware or firmware issue, my machine is currently crashing
rather a lot. If that happens around the time I'm performing any git
activity, it seems quite likely to cause substantial corruption of the
git state.
I've responded both by running sync(1) every 60 seconds and by pushing
work to github more frequently, but the latter isn't appropriate for
everything; twice recently I've lost a fair bit of work as a result.
Q: Is there a mode in which I can run git that would make it a bit more
robust against crashes, at the cost of being a bit slower?
The primary symptom is that files modified shortly before a crash show
up existing but zero-length after the crash. For source files I mostly
know what to do in that situation, but `git fsck` shows a lot of files
under '.git/objects' that are empty, which seems to make things hard
to recover:
% git fsck
error: object file .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
error: unable to mmap .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file or directory
error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533
[... a dozen similar entries ...]
error: object file .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
error: unable to mmap .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or directory
error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5
Checking object directories: 100% (256/256), done.
Checking objects: 100% (1577/1577), done.
error: refs/stash: invalid sha1 pointer 0000000000000000000000000000000000000000
error: bad ref for .git/logs/refs/stash
dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107
%
Last time I checked out the previous state from github in a new
directory and was able to find and copy over most of my work before
continuing. On this occasion I did a `git stash save` shortly before
the crash, and I'm not sure how to get that back. I see René Scharfe's
suggestion of:
git fsck --unreachable |
grep commit | cut -d\ -f3 |
xargs git log --merges --no-walk --grep=WIP
from a recent message, but that is only showing me an older stash item.
Thanks in advance for any suggestions,
Hugo van der Sanden
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: safer git?
2020-10-24 20:19 safer git? hv
@ 2020-10-24 20:21 ` hv
2020-10-24 22:11 ` Randall S. Becker
1 sibling, 0 replies; 6+ messages in thread
From: hv @ 2020-10-24 20:21 UTC (permalink / raw)
To: hv; +Cc: git
Sorry, I should have added: I'm using Ubuntu 18.04.5 (bionic) with
git version 2.17.1.
Hugo
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: safer git?
2020-10-24 20:19 safer git? hv
2020-10-24 20:21 ` hv
@ 2020-10-24 22:11 ` Randall S. Becker
2020-10-25 3:06 ` brian m. carlson
1 sibling, 1 reply; 6+ messages in thread
From: Randall S. Becker @ 2020-10-24 22:11 UTC (permalink / raw)
To: hv, git
On October 24, 2020 4:19 PM, hv@crypt.org wrote:
> Hi, due to a hardware or firmware issue, my machine is currently crashing
> rather a lot. If that happens around the time I'm performing any git
activity,
> it seems quite likely to cause substantial corruption of the git state.
>
> I've responded both by running sync(1) every 60 seconds and by pushing
work
> to github more frequently, but the latter isn't appropriate for
everything;
> twice recently I've lost a fair bit of work as a result.
>
> Q: Is there a mode in which I can run git that would make it a bit more
robust
> against crashes, at the cost of being a bit slower?
>
>
> The primary symptom is that files modified shortly before a crash show up
> existing but zero-length after the crash. For source files I mostly know
what
> to do in that situation, but `git fsck` shows a lot of files under
'.git/objects'
> that are empty, which seems to make things hard to recover:
>
> % git fsck
> error: object file
> .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
> error: unable to mmap
> .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file
> or directory
> error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
> missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533
> [... a dozen similar entries ...]
> error: object file
> .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
> error: unable to mmap
> .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or
> directory
> error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
> missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5
> Checking object directories: 100% (256/256), done.
> Checking objects: 100% (1577/1577), done.
> error: refs/stash: invalid sha1 pointer
> 0000000000000000000000000000000000000000
> error: bad ref for .git/logs/refs/stash
> dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107
> %
>
> Last time I checked out the previous state from github in a new directory
and
> was able to find and copy over most of my work before continuing. On this
> occasion I did a `git stash save` shortly before the crash, and I'm not
sure
> how to get that back. I see René Scharfe's suggestion of:
> git fsck --unreachable |
> grep commit | cut -d\ -f3 |
> xargs git log --merges --no-walk --grep=WIP from a recent message, but
> that is only showing me an older stash item.
I would suggest turning off write-through buffering on your disk. Let writes
complete immediately instead of being deferred to sync. Also, this does feel
like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
Good luck,
Randall
-- Brief whoami:
NonStop developer since approximately 211288444200000000
UNIX developer since approximately 421664400
-- In my real life, I talk too much.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: safer git?
2020-10-24 22:11 ` Randall S. Becker
@ 2020-10-25 3:06 ` brian m. carlson
2020-10-25 12:45 ` hv
2020-10-25 15:17 ` Randall S. Becker
0 siblings, 2 replies; 6+ messages in thread
From: brian m. carlson @ 2020-10-25 3:06 UTC (permalink / raw)
To: Randall S. Becker; +Cc: hv, git
[-- Attachment #1: Type: text/plain, Size: 3811 bytes --]
[I somehow didn't get the original message, so replying inline below.]
On 2020-10-24 at 22:11:54, Randall S. Becker wrote:
> On October 24, 2020 4:19 PM, hv@crypt.org wrote:
> > Q: Is there a mode in which I can run git that would make it a bit
> > more robust
> > against crashes, at the cost of being a bit slower?
> >
> >
> > The primary symptom is that files modified shortly before a crash
> > show up existing but zero-length after the crash. For source files I
> > mostly know what to do in that situation, but `git fsck` shows a lot
> > of files under '.git/objects' that are empty, which seems to make
> > things hard to recover:
> >
> > % git fsck
> > error: object file
> > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
> > error: unable to mmap
> > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such file
> > or directory
> > error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
> > missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533
> > [... a dozen similar entries ...]
> > error: object file
> > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
> > error: unable to mmap
> > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such file or
> > directory
> > error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
> > missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5
> > Checking object directories: 100% (256/256), done.
> > Checking objects: 100% (1577/1577), done.
> > error: refs/stash: invalid sha1 pointer
> > 0000000000000000000000000000000000000000
> > error: bad ref for .git/logs/refs/stash
> > dangling commit 1c0ea4e6159952501957012d2b9db7d68b52d107
> > %
You can try setting core.fsyncObjectFiles to true. That's the only knob
that Git has for that at the moment, although there was some discussion
about adding a new feature for other files[0].
I suspect a lot of the zero-byte files and any files that end up as
all-zeros are due to your file system. The default file system on
Ubuntu is ext4, IIRC, and if that's what you're using, you can set
data=journal instead of data=ordered as a mount option. For the root
file system, you'll need to pass rootflags=data=journal as a kernel
boot option.
That may be significantly slower, but until you get your hardware
problem sorted out, it may very well be worth it for you. I'd try this
option before the one below because it'll have less of an impact on
performance and may solve most or all of your problems.
> > Last time I checked out the previous state from github in a new directory
> and
> > was able to find and copy over most of my work before continuing. On this
> > occasion I did a `git stash save` shortly before the crash, and I'm not
> sure
> > how to get that back. I see René Scharfe's suggestion of:
> > git fsck --unreachable |
> > grep commit | cut -d\ -f3 |
> > xargs git log --merges --no-walk --grep=WIP from a recent message, but
> > that is only showing me an older stash item.
>
> I would suggest turning off write-through buffering on your disk. Let writes
> complete immediately instead of being deferred to sync. Also, this does feel
> like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
Turning off buffering and caching for your disk drive may make things
_really_ slow, but it will definitely improve data integrity.
I know hardware problems are always a hassle, so I hope you get things
figured out and fixed soon.
[0] I admit I am not running the very latest version and the new feature
may have already landed; if so, I apologize for the out-of-date
information and for not keeping up with the list.
--
brian m. carlson (he/him or they/them)
Houston, Texas, US
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 263 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: safer git?
2020-10-25 3:06 ` brian m. carlson
@ 2020-10-25 12:45 ` hv
2020-10-25 15:17 ` Randall S. Becker
1 sibling, 0 replies; 6+ messages in thread
From: hv @ 2020-10-25 12:45 UTC (permalink / raw)
To: brian m. carlson, Randall S. Becker, hv, git
"brian m. carlson" wrote:
:You can try setting core.fsyncObjectFiles to true.
Super, that's exactly the sort of thing I was hoping to find.
:I suspect a lot of the zero-byte files and any files that end up as
:all-zeros are due to your file system. The default file system on
:Ubuntu is ext4, IIRC, and if that's what you're using, you can set
:data=journal instead of data=ordered as a mount option.
It is indeed ext4. I'll consider this option; for now I've turned off
write caching as suggested by Randall, which feels like a lighter-weight
approach that should give almost all of the benefit.
:"Randall S. Becker" wrote:
:> I would suggest turning off write-through buffering on your disk. Let writes
:> complete immediately instead of being deferred to sync. Also, this does feel
:> like a disk issue, so fsck or chkdsk /f (or whatever) on your disk urgently.
fsck doesn't seem to be complaining, but I've set it to run every
20 mounts. What I do see is a handful of "orphaned inodes" being
reclaimed on boot after every crash.
:Turning off buffering and caching for your disk drive may make things
:_really_ slow, but it will definitely improve data integrity.
I haven't noticed a big slowdown so far; I'm rarely doing a _lot_ of
writes.
Thanks greatly to both of you for the suggestions.
Hugo
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: safer git?
2020-10-25 3:06 ` brian m. carlson
2020-10-25 12:45 ` hv
@ 2020-10-25 15:17 ` Randall S. Becker
1 sibling, 0 replies; 6+ messages in thread
From: Randall S. Becker @ 2020-10-25 15:17 UTC (permalink / raw)
To: 'brian m. carlson'; +Cc: hv, git
On October 24, 2020 11:06 PM, brian m. carlson wrote:
> [I somehow didn't get the original message, so replying inline below.]
>
> On 2020-10-24 at 22:11:54, Randall S. Becker wrote:
> > On October 24, 2020 4:19 PM, hv@crypt.org wrote:
> > > Q: Is there a mode in which I can run git that would make it a bit
> > > more robust against crashes, at the cost of being a bit slower?
> > >
> > >
> > > The primary symptom is that files modified shortly before a crash
> > > show up existing but zero-length after the crash. For source files I
> > > mostly know what to do in that situation, but `git fsck` shows a lot
> > > of files under '.git/objects' that are empty, which seems to make
> > > things hard to recover:
> > >
> > > % git fsck
> > > error: object file
> > > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533 is empty
> > > error: unable to mmap
> > > .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533: No such
> file
> > > or directory
> > > error: 0ef31631726cea2e9bf89d7bbe7b924b5282d533: object corrupt or
> > > missing: .git/objects/0e/f31631726cea2e9bf89d7bbe7b924b5282d533
> > > [... a dozen similar entries ...]
> > > error: object file
> > > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5 is empty
> > > error: unable to mmap
> > > .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: No such
> file
> > > or directory
> > > error: f5a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5: object corrupt or
> > > missing: .git/objects/f5/a9d125645e69a0e40f9bf7a8c90b1c1c4a4ea5
> > > Checking object directories: 100% (256/256), done.
> > > Checking objects: 100% (1577/1577), done.
> > > error: refs/stash: invalid sha1 pointer
> > > 0000000000000000000000000000000000000000
> > > error: bad ref for .git/logs/refs/stash dangling commit
> > > 1c0ea4e6159952501957012d2b9db7d68b52d107
> > > %
>
> You can try setting core.fsyncObjectFiles to true. That's the only knob that
> Git has for that at the moment, although there was some discussion about
> adding a new feature for other files[0].
>
> I suspect a lot of the zero-byte files and any files that end up as all-zeros are
> due to your file system. The default file system on Ubuntu is ext4, IIRC, and
> if that's what you're using, you can set data=journal instead of data=ordered
> as a mount option. For the root file system, you'll need to pass
> rootflags=data=journal as a kernel boot option.
>
> That may be significantly slower, but until you get your hardware problem
> sorted out, it may very well be worth it for you. I'd try this option before the
> one below because it'll have less of an impact on performance and may solve
> most or all of your problems.
>
> > > Last time I checked out the previous state from github in a new
> > > directory
> > and
> > > was able to find and copy over most of my work before continuing. On
> > > this occasion I did a `git stash save` shortly before the crash, and
> > > I'm not
> > sure
> > > how to get that back. I see René Scharfe's suggestion of:
> > > git fsck --unreachable |
> > > grep commit | cut -d\ -f3 |
> > > xargs git log --merges --no-walk --grep=WIP from a recent message,
> > > but that is only showing me an older stash item.
> >
> > I would suggest turning off write-through buffering on your disk. Let
> > writes complete immediately instead of being deferred to sync. Also,
> > this does feel like a disk issue, so fsck or chkdsk /f (or whatever) on your
> disk urgently.
>
> Turning off buffering and caching for your disk drive may make things
> _really_ slow, but it will definitely improve data integrity.
>
> I know hardware problems are always a hassle, so I hope you get things
> figured out and fixed soon.
>
> [0] I admit I am not running the very latest version and the new feature may
> have already landed; if so, I apologize for the out-of-date information and
> for not keeping up with the list.
Thanks for pointing out the git knob. I didn't know about that one and it seems helpful in general.
Regards,
Randall
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-10-25 15:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-24 20:19 safer git? hv
2020-10-24 20:21 ` hv
2020-10-24 22:11 ` Randall S. Becker
2020-10-25 3:06 ` brian m. carlson
2020-10-25 12:45 ` hv
2020-10-25 15:17 ` Randall S. Becker
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).