* Git bug report - disk errors on Windows after push @ 2022-04-18 7:47 Attila Csosz 2022-04-18 17:37 ` brian m. carlson 0 siblings, 1 reply; 10+ messages in thread From: Attila Csosz @ 2022-04-18 7:47 UTC (permalink / raw) To: git Thank you for filling out a Git bug report! Please answer the following questions to help us understand your issue. What did you do before the bug happened? (Steps to reproduce your issue) Pushing to a directory directly. What did you expect to happen? (Expected behavior) Good behavior What happened instead? (Actual behavior) Disk errors. After pushing to a directory (my origin field is C:\Work for example) the git repository will be corrupted. Moreover causing disk errors not only in the target git bare repository. I've loosed some other files. Cannot be reproduced exactly. However I have decided to report this problem. I've experienced this problem for several years. What's different between what you expected and what actually happened? Described. Anything else you want to add: Please review the rest of the bug report below. You can delete any lines you don't wish to share. [System Info] git version: git version 2.31.1.windows.1 cpu: x86_64 built from commit: c5f0be26a7e3846e3b6268d1c6c4800d838c6bbb sizeof-long: 4 sizeof-size_t: 8 shell-path: /bin/sh feature: fsmonitor--daemon uname: Windows 10.0 19044 compiler info: gnuc: 10.2 libc info: no libc information available $SHELL (typically, interactive shell): C:\Program Files\Git\usr\bin\bash.exe [Enabled Hooks] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-18 7:47 Git bug report - disk errors on Windows after push Attila Csosz @ 2022-04-18 17:37 ` brian m. carlson 2022-04-19 3:52 ` Attila Csosz 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason 0 siblings, 2 replies; 10+ messages in thread From: brian m. carlson @ 2022-04-18 17:37 UTC (permalink / raw) To: Attila Csosz; +Cc: git [-- Attachment #1: Type: text/plain, Size: 1564 bytes --] Hey, On 2022-04-18 at 07:47:07, Attila Csosz wrote: > Thank you for filling out a Git bug report! > Please answer the following questions to help us understand your issue. > > What did you do before the bug happened? (Steps to reproduce your issue) > Pushing to a directory directly. > > What did you expect to happen? (Expected behavior) > Good behavior > > What happened instead? (Actual behavior) > Disk errors. After pushing to a directory (my origin field is C:\Work for > example) the git repository will be corrupted. > Moreover causing disk errors not only in the target git bare repository. > I've loosed some other files. > Cannot be reproduced exactly. However I have decided to report this problem. > I've experienced this problem for several years. There's nothing that Git should be able to do as a normal operating system that causes disk errors. If you're seeing those kinds of errors, the likelihood is that your disk is bad and you need to replace it. Do note that if you're storing your data in a directory managed by a cloud syncing service (e.g., Dropbox or OneDrive), then it is known that those file systems corrupt Git repositories and you shouldn't use them with Git. That's due to the way they sync data. However, if this is a regular file system (e.g., NTFS), then Git shouldn't be able to cause disk problems, and if it could, that would be a serious defect in Windows, so it's almost certainly going to be a hardware problem. -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-18 17:37 ` brian m. carlson @ 2022-04-19 3:52 ` Attila Csosz 2022-04-20 21:28 ` brian m. carlson 2022-04-20 22:54 ` Neeraj Singh 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason 1 sibling, 2 replies; 10+ messages in thread From: Attila Csosz @ 2022-04-19 3:52 UTC (permalink / raw) To: brian m. carlson, git I've experienced the problem at least 3 different external disk. I'm not using cloud syncing service (e.g., Dropbox or OneDrive) for git. Attila On 4/18/2022 19:37, brian m. carlson wrote: > Hey, > > On 2022-04-18 at 07:47:07, Attila Csosz wrote: >> Thank you for filling out a Git bug report! >> Please answer the following questions to help us understand your issue. >> >> What did you do before the bug happened? (Steps to reproduce your issue) >> Pushing to a directory directly. >> >> What did you expect to happen? (Expected behavior) >> Good behavior >> >> What happened instead? (Actual behavior) >> Disk errors. After pushing to a directory (my origin field is C:\Work for >> example) the git repository will be corrupted. >> Moreover causing disk errors not only in the target git bare repository. >> I've loosed some other files. >> Cannot be reproduced exactly. However I have decided to report this problem. >> I've experienced this problem for several years. > There's nothing that Git should be able to do as a normal operating > system that causes disk errors. If you're seeing those kinds of errors, > the likelihood is that your disk is bad and you need to replace it. > > Do note that if you're storing your data in a directory managed by a > cloud syncing service (e.g., Dropbox or OneDrive), then it is known that > those file systems corrupt Git repositories and you shouldn't use them > with Git. That's due to the way they sync data. > > However, if this is a regular file system (e.g., NTFS), then Git > shouldn't be able to cause disk problems, and if it could, that would be > a serious defect in Windows, so it's almost certainly going to be a > hardware problem. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-19 3:52 ` Attila Csosz @ 2022-04-20 21:28 ` brian m. carlson 2022-04-20 22:54 ` Neeraj Singh 1 sibling, 0 replies; 10+ messages in thread From: brian m. carlson @ 2022-04-20 21:28 UTC (permalink / raw) To: Attila Csosz; +Cc: git [-- Attachment #1: Type: text/plain, Size: 1443 bytes --] [Please avoid top-posting on this list.] On 2022-04-19 at 03:52:23, Attila Csosz wrote: > I've experienced the problem at least 3 different external disk. > I'm not using cloud syncing service (e.g., Dropbox or OneDrive) for git. I don't doubt that's the case. However, it's not possible for Git to corrupt a disk just by writing to it. Git, as a normal unprivileged program, can only open files and perform normal read and write operations on them. Management of the file system, including integrity, is the responsibility of the operating system. If running Git on Windows causes files outside of the repository to be corrupted, that's either a hardware problem or a bug in Windows. If you're seeing this problem, it could be that you have a series of bad disks (say, if you bought a set of cheap flash drives), that the drive isn't being removed properly, that you have some sort of broken driver or malware, that there's some other hardware problem (e.g., a bad dock, USB device, CPU, or memory), or just that there's a previously unknown bug in Windows. None of this is Git's fault, and it's really up to you to figure out which of these it is. You might try isolating it by using a different computer to write to the repository first, to see if something about your current machine is the problem, and then try isolating other causes. -- brian m. carlson (he/him or they/them) Toronto, Ontario, CA [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 262 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-19 3:52 ` Attila Csosz 2022-04-20 21:28 ` brian m. carlson @ 2022-04-20 22:54 ` Neeraj Singh 2022-04-21 2:29 ` Attila Csosz 1 sibling, 1 reply; 10+ messages in thread From: Neeraj Singh @ 2022-04-20 22:54 UTC (permalink / raw) To: Attila Csosz, brian m. carlson, git On 4/18/2022 8:52 PM, Attila Csosz wrote: > I've experienced the problem at least 3 different external disk. > I'm not using cloud syncing service (e.g., Dropbox or OneDrive) for git. > > Attila > Hi Attila, Could you please share a bit more information about the disks that are being corrupted and the specific error messages you're seeing? What is the underlying storage type (e.g. USB key, external hard drive, etc)? Also, which filesystem is the drive formatted as? Could you please save one instance of a volume in the corrupted state so that we can consider further investigations? I believe the output of "fsutil fsinfo volumeinfo <drive letter>" and "fsutil fsinfo drivetype <drive letter>" should give us some useful info about the volume that's being corrupted. Thanks, Neeraj Singh, Windows FileSystem Dev ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-20 22:54 ` Neeraj Singh @ 2022-04-21 2:29 ` Attila Csosz 2022-04-21 20:08 ` Neeraj Singh 0 siblings, 1 reply; 10+ messages in thread From: Attila Csosz @ 2022-04-21 2:29 UTC (permalink / raw) To: neerajsi; +Cc: brian m. carlson, git Please check this link also https://github.community/t/what-could-be-the-reason-my-git-repositories-keep-getting-corrupted-broken/184711 You can found many link on web on 'git corrupted repository'. In my opinion when a git repository is corrupted on Windows there is a good chance to HDD errors. I suggest also run intensive test on Windows (run a bat file commiting and pushing to a bare directory type repository). Check your HDD healthy status after 10-20 run of 'git push'. My git software is come form 'git-scm.com' and I'm using TortoiseGit. My git software is up to date. Last time my laptop HDD is crashed loosing sensitive datas. I will never push to a local repo I'm using online repo-s like github. /// fsutil fsinfo volumeinfo e: Volume Name : Seagate Backup Plus Drive Volume Serial Number : <removed by hand> Max Component Length : 255 File System Name : NTFS Is ReadWrite Not Thinly-Provisioned Supports Case-sensitive filenames Preserves Case of filenames Supports Unicode in filenames Preserves & Enforces ACL's Supports file-based Compression Supports Disk Quotas Supports Sparse files Supports Reparse Points Returns Handle Close Result Information Supports POSIX-style Unlink and Rename Supports Object Identifiers Supports Encrypted File System Supports Named Streams Supports Transactions Supports Hard Links Supports Extended Attributes Supports Open By FileID Supports USN Journal /// Maybe this email not reaching the main mailing list I'm somehow blacklisted. Attila On Thu, Apr 21, 2022 at 12:54 AM Neeraj Singh <nksingh85@gmail.com> wrote: > > On 4/18/2022 8:52 PM, Attila Csosz wrote: > > I've experienced the problem at least 3 different external disk. > > I'm not using cloud syncing service (e.g., Dropbox or OneDrive) for git. > > > > Attila > > > Hi Attila, > Could you please share a bit more information about the disks that are > being corrupted and the specific error messages you're seeing? > > What is the underlying storage type (e.g. USB key, external hard drive, > etc)? Also, which filesystem is the drive formatted as? > > Could you please save one instance of a volume in the corrupted state so > that we can consider further investigations? > > I believe the output of "fsutil fsinfo volumeinfo <drive letter>" and > "fsutil fsinfo drivetype <drive letter>" should give us some useful info > about the volume that's being corrupted. > > Thanks, > Neeraj Singh, > Windows FileSystem Dev ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-21 2:29 ` Attila Csosz @ 2022-04-21 20:08 ` Neeraj Singh 0 siblings, 0 replies; 10+ messages in thread From: Neeraj Singh @ 2022-04-21 20:08 UTC (permalink / raw) To: Attila Csosz, neerajsi; +Cc: nksingh85, brian m. carlson, git On 4/20/2022 7:29 PM, Attila Csosz wrote: > Please check this link also > https://github.community/t/what-could-be-the-reason-my-git-repositories-keep-getting-corrupted-broken/184711 > > You can found many link on web on 'git corrupted repository'. In my > opinion when a git repository is corrupted on Windows there is a good > chance to HDD errors. > > I suggest also run intensive test on Windows (run a bat file commiting > and pushing to a bare directory type repository). Check your HDD > healthy status after 10-20 run of 'git push'. > > My git software is come form 'git-scm.com' and I'm using TortoiseGit. > My git software is up to date. > Last time my laptop HDD is crashed loosing sensitive datas. I will > never push to a local repo I'm using online repo-s like github. > Do you currently have a drive in the bad state? Is write-caching enabled for the drive (https://docs.microsoft.com/en-us/troubleshoot/windows-server/backup-and-storage/turn-disk-write-caching-on-off)? Git hammers the filesystem in a way that's not typically done for a USB drive (especially issuing some "FlushFileBuffers" calls). So it possibly exposes some inherent flakiness in your setup. For instance, your drives might be good, but something could be wrong with your USB cable or the power supply. If you have a drive in the bad state, a "chkdsk /scan /X <drive letter>" might give some hints about what specifically is wrong with that filesystem. I'd recommend preferring a cloud-based backup system for your Git repos (e.g. GitHub or GitLab). Also, for local backups, something solid-state like an SSD might be more reliable in the long run. Thanks, Neeraj ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-18 17:37 ` brian m. carlson 2022-04-19 3:52 ` Attila Csosz @ 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason 2022-04-22 15:04 ` Attila Csosz 2022-04-22 16:13 ` Junio C Hamano 1 sibling, 2 replies; 10+ messages in thread From: Ævar Arnfjörð Bjarmason @ 2022-04-22 9:27 UTC (permalink / raw) To: brian m. carlson; +Cc: Attila Csosz, git On Mon, Apr 18 2022, brian m. carlson wrote: > [[PGP Signed Part:Undecided]] > Hey, > > On 2022-04-18 at 07:47:07, Attila Csosz wrote: >> Thank you for filling out a Git bug report! >> Please answer the following questions to help us understand your issue. >> >> What did you do before the bug happened? (Steps to reproduce your issue) >> Pushing to a directory directly. >> >> What did you expect to happen? (Expected behavior) >> Good behavior >> >> What happened instead? (Actual behavior) >> Disk errors. After pushing to a directory (my origin field is C:\Work for >> example) the git repository will be corrupted. >> Moreover causing disk errors not only in the target git bare repository. >> I've loosed some other files. >> Cannot be reproduced exactly. However I have decided to report this problem. >> I've experienced this problem for several years. > > There's nothing that Git should be able to do as a normal operating > system that causes disk errors. If you're seeing those kinds of errors, > the likelihood is that your disk is bad and you need to replace it. > > Do note that if you're storing your data in a directory managed by a > cloud syncing service (e.g., Dropbox or OneDrive), then it is known that > those file systems corrupt Git repositories and you shouldn't use them > with Git. That's due to the way they sync data. > > However, if this is a regular file system (e.g., NTFS), then Git > shouldn't be able to cause disk problems, and if it could, that would be > a serious defect in Windows, so it's almost certainly going to be a > hardware problem. I think you're most likely right that this is some local HW issue, but the report doesn't have enough information to say either way. But generally as we've been discussing at length in various threads related to git's fsync() behavior we can and will corrupt repositories due only to our own (mis)use of FS APIs, i.e. not doing fsyncing and the like correctly, it's been improved a bit recently. But we're still in a state of not doing it correctly, and/or relying on some implicit assumptions. For something like an external disk that may have its power cable yanked I'd give it about even odds that it's HW v.s. git's own FS syncing logic being at fault. We get away with a lot of shenanigans because we're usually on a journaled FS, which e.g. on a FAT32 external volume we won't benefit from. Attila: I have no idea if this even exists on Windows, but aside from other suggestions in this thread I'd suggest enabling whatever Window's equivalent of "sync" mounting is (if it even exists). I.e. to have every single FS operation by synced to the metal. It will make your repository *much much slower*, but it will be safe as far as any logic errors in git vis-a-vis syncing go. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason @ 2022-04-22 15:04 ` Attila Csosz 2022-04-22 16:13 ` Junio C Hamano 1 sibling, 0 replies; 10+ messages in thread From: Attila Csosz @ 2022-04-22 15:04 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason; +Cc: brian m. carlson, git I'm using git on Windows for 6-7 years sometimes facing this problem (different computers, different HDDs). Last time the problem occured on my internal laptop HDD. So I have decided to report the problem. If unstable to use local bare repos on Windows you should mention in somewhere or completly disable it. Disk errors are unacceptable even the git repo is corrupted. Please check this link also https://github.community/t/what-could-be-the-reason-my-git-repositories-keep-getting-corrupted-broken/184711 Attila On Fri, Apr 22, 2022 at 11:27 AM Ævar Arnfjörð Bjarmason <avarab@gmail.com> wrote: > > > On Mon, Apr 18 2022, brian m. carlson wrote: > > > [[PGP Signed Part:Undecided]] > > Hey, > > > > On 2022-04-18 at 07:47:07, Attila Csosz wrote: > >> Thank you for filling out a Git bug report! > >> Please answer the following questions to help us understand your issue. > >> > >> What did you do before the bug happened? (Steps to reproduce your issue) > >> Pushing to a directory directly. > >> > >> What did you expect to happen? (Expected behavior) > >> Good behavior > >> > >> What happened instead? (Actual behavior) > >> Disk errors. After pushing to a directory (my origin field is C:\Work for > >> example) the git repository will be corrupted. > >> Moreover causing disk errors not only in the target git bare repository. > >> I've loosed some other files. > >> Cannot be reproduced exactly. However I have decided to report this problem. > >> I've experienced this problem for several years. > > > > There's nothing that Git should be able to do as a normal operating > > system that causes disk errors. If you're seeing those kinds of errors, > > the likelihood is that your disk is bad and you need to replace it. > > > > Do note that if you're storing your data in a directory managed by a > > cloud syncing service (e.g., Dropbox or OneDrive), then it is known that > > those file systems corrupt Git repositories and you shouldn't use them > > with Git. That's due to the way they sync data. > > > > However, if this is a regular file system (e.g., NTFS), then Git > > shouldn't be able to cause disk problems, and if it could, that would be > > a serious defect in Windows, so it's almost certainly going to be a > > hardware problem. > > I think you're most likely right that this is some local HW issue, but > the report doesn't have enough information to say either way. > > But generally as we've been discussing at length in various threads > related to git's fsync() behavior we can and will corrupt repositories > due only to our own (mis)use of FS APIs, i.e. not doing fsyncing and the > like correctly, it's been improved a bit recently. > > But we're still in a state of not doing it correctly, and/or relying on > some implicit assumptions. > > For something like an external disk that may have its power cable yanked > I'd give it about even odds that it's HW v.s. git's own FS syncing logic > being at fault. > > We get away with a lot of shenanigans because we're usually on a > journaled FS, which e.g. on a FAT32 external volume we won't benefit > from. > > Attila: I have no idea if this even exists on Windows, but aside from > other suggestions in this thread I'd suggest enabling whatever Window's > equivalent of "sync" mounting is (if it even exists). I.e. to have every > single FS operation by synced to the metal. > > It will make your repository *much much slower*, but it will be safe as > far as any logic errors in git vis-a-vis syncing go. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Git bug report - disk errors on Windows after push 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason 2022-04-22 15:04 ` Attila Csosz @ 2022-04-22 16:13 ` Junio C Hamano 1 sibling, 0 replies; 10+ messages in thread From: Junio C Hamano @ 2022-04-22 16:13 UTC (permalink / raw) To: Ævar Arnfjörð Bjarmason Cc: brian m. carlson, Attila Csosz, git Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes: > For something like an external disk that may have its power cable yanked > I'd give it about even odds that it's HW v.s. git's own FS syncing logic > being at fault. Given that the original report had this: Disk errors. After pushing to a directory (my origin field is C:\Work for example) the git repository will be corrupted. Moreover causing disk errors not only in the target git bare repository. I've loosed some other files. that clearly stated that the corruption spreads outside repositories managed (and written and fsynced) by Git, it does not seem likely that Git's relying on "closing is enough as sync daemon will eventually flush them" is the primary source of corruption. The recent "are we fsyncing correctly?" topic is about how likely we would survive in the presense of actions like unplugging without unmounting. It does not help a bit against flaky hardware, and even when it helps, it is merely making the damage caused by such actions less severe. ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2022-04-22 16:14 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-04-18 7:47 Git bug report - disk errors on Windows after push Attila Csosz 2022-04-18 17:37 ` brian m. carlson 2022-04-19 3:52 ` Attila Csosz 2022-04-20 21:28 ` brian m. carlson 2022-04-20 22:54 ` Neeraj Singh 2022-04-21 2:29 ` Attila Csosz 2022-04-21 20:08 ` Neeraj Singh 2022-04-22 9:27 ` Ævar Arnfjörð Bjarmason 2022-04-22 15:04 ` Attila Csosz 2022-04-22 16:13 ` Junio C Hamano
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).