* Windows performance / threading file access
@ 2013-10-10 18:18 Stefan Zager
2013-10-10 20:19 ` Sebastian Schuberth
0 siblings, 1 reply; 12+ messages in thread
From: Stefan Zager @ 2013-10-10 18:18 UTC (permalink / raw)
To: git
Hi folks,
I don't follow the mailing list carefully, so forgive me if this has
been discussed before, but:
I've noticed that when working with a very large repository using msys
git, the initial checkout of a cloned repository is excruciatingly
slow (80%+ of total clone time). The root cause, I think, is that git
does all the file access serially, and that's really slow on Windows.
Has anyone considered threading file access to speed this up? In
particular, I've got my eye on this loop in unpack-trees.c:
static struct checkout state;
static int check_updates(struct unpack_trees_options *o)
{
unsigned cnt = 0, total = 0;
struct progress *progress = NULL;
struct index_state *index = &o->result;
int i;
int errs = 0;
...
for (i = 0; i < index->cache_nr; i++) {
struct cache_entry *ce = index->cache[i];
if (ce->ce_flags & CE_UPDATE) {
display_progress(progress, ++cnt);
ce->ce_flags &= ~CE_UPDATE;
if (o->update && !o->dry_run) {
errs |= checkout_entry(ce, &state, NULL);
}
}
}
stop_progress(&progress);
if (o->update)
git_attr_set_direction(GIT_ATTR_CHECKIN, NULL);
return errs != 0;
}
Any thoughts on adding threading around the call to checkout_entry?
Thanks in advance,
Stefan
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: Windows performance / threading file access 2013-10-10 18:18 Windows performance / threading file access Stefan Zager @ 2013-10-10 20:19 ` Sebastian Schuberth 2013-10-11 0:51 ` Karsten Blees 0 siblings, 1 reply; 12+ messages in thread From: Sebastian Schuberth @ 2013-10-10 20:19 UTC (permalink / raw) To: Stefan Zager, git; +Cc: msysGit, Karsten Blees Please keep in mind to CC the msysgit mailing list for Windows-specific stuff. I'm also CC'ing Karsten who has worked on performance improvements for Git for Windows in the past. Thanks for bringing this up! -- Sebastian Schuberth > Hi folks, > > I don't follow the mailing list carefully, so forgive me if this has > been discussed before, but: > > I've noticed that when working with a very large repository using msys > git, the initial checkout of a cloned repository is excruciatingly > slow (80%+ of total clone time). The root cause, I think, is that git > does all the file access serially, and that's really slow on Windows. > > Has anyone considered threading file access to speed this up? In > particular, I've got my eye on this loop in unpack-trees.c: > > static struct checkout state; > static int check_updates(struct unpack_trees_options *o) > { > unsigned cnt = 0, total = 0; > struct progress *progress = NULL; > struct index_state *index = &o->result; > int i; > int errs = 0; > > ... > > for (i = 0; i < index->cache_nr; i++) { > struct cache_entry *ce = index->cache[i]; > > if (ce->ce_flags & CE_UPDATE) { > display_progress(progress, ++cnt); > ce->ce_flags &= ~CE_UPDATE; > if (o->update && !o->dry_run) { > errs |= checkout_entry(ce, &state, NULL); > } > } > } > stop_progress(&progress); > if (o->update) > git_attr_set_direction(GIT_ATTR_CHECKIN, NULL); > return errs != 0; > } > > > Any thoughts on adding threading around the call to checkout_entry? > > > Thanks in advance, > > Stefan -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-10 20:19 ` Sebastian Schuberth @ 2013-10-11 0:51 ` Karsten Blees 2013-10-11 5:28 ` Stefan Zager 2013-10-11 5:35 ` Stefan Zager 0 siblings, 2 replies; 12+ messages in thread From: Karsten Blees @ 2013-10-11 0:51 UTC (permalink / raw) To: Sebastian Schuberth; +Cc: Stefan Zager, git, msysGit Am 10.10.2013 22:19, schrieb Sebastian Schuberth: > Please keep in mind to CC the msysgit mailing list for Windows-specific stuff. I'm also CC'ing Karsten who has worked on performance improvements for Git for Windows in the past. > Thanks > Thanks for bringing this up! > > -- > Sebastian Schuberth > > >> Hi folks, >> >> I don't follow the mailing list carefully, so forgive me if this has >> been discussed before, but: >> >> I've noticed that when working with a very large repository using msys >> git, the initial checkout of a cloned repository is excruciatingly >> slow (80%+ of total clone time). The root cause, I think, is that git >> does all the file access serially, and that's really slow on Windows. >> What exactly do you mean by "excruciatingly slow"? I just ran a few tests with a big repo (WebKit, ~2GB, ~200k files). A full checkout with git 1.8.4 on my SSD took 52s on Linux and 81s on Windows. Xcopy /s took ~4 minutes (so xcopy is much slower than git). On a 'real' HD (WD Caviar Green) the Windows checkout took ~9 minutes. That's not so bad I think, considering that we read from pack files and write both files and directory structures, so there's a lot of disk seeking involved. If your numbers are much slower, check for overeager virus scanners and probably the infamous "User Account Control" (On Vista/7 (8?), the luafv.sys driver slows down things on the system drive even with UAC turned off in control panel. The driver can be disabled with "sc config luafv start= disabled" + reboot. Reenable with "sc config luafv start= auto"). >> Has anyone considered threading file access to speed this up? In >> particular, I've got my eye on this loop in unpack-trees.c: >> Its probably worth a try, however, in my experience, doing disk IO in parallel tends to slow things down due to more disk seeks. I'd rather try to minimize seeks, e.g.: * read the blob data for a block of cache_entries, then write out the files, repeat (this would require lots of memory, though) * index->cache is typically sorted by name and pack files by size, right? Perhaps its faster to iterate cache_entries by size so that we read the pack file sequentially (but then we'd write files/directories in random order...) If you want to measure exactly which part of checkout eats the performance, check out this: https://github.com/kblees/git/commits/kb/performance-tracing-v3 Bye, Karsten -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-11 0:51 ` Karsten Blees @ 2013-10-11 5:28 ` Stefan Zager 2013-10-11 5:35 ` Stefan Zager 1 sibling, 0 replies; 12+ messages in thread From: Stefan Zager @ 2013-10-11 5:28 UTC (permalink / raw) To: Karsten Blees; +Cc: Sebastian Schuberth, git, msysGit [-- Attachment #1: Type: text/plain, Size: 3451 bytes --] On Thu, Oct 10, 2013 at 5:51 PM, Karsten Blees <karsten.blees@gmail.com>wrote: > >> I've noticed that when working with a very large repository using msys > >> git, the initial checkout of a cloned repository is excruciatingly > >> slow (80%+ of total clone time). The root cause, I think, is that git > >> does all the file access serially, and that's really slow on Windows. > >> > > What exactly do you mean by "excruciatingly slow"? > > I just ran a few tests with a big repo (WebKit, ~2GB, ~200k files). A full > checkout with git 1.8.4 on my SSD took 52s on Linux and 81s on Windows. > Xcopy /s took ~4 minutes (so xcopy is much slower than git). On a 'real' HD > (WD Caviar Green) the Windows checkout took ~9 minutes. > I'm using blink for my test, which should be more or less indistinguishable from WebKit. I'm using a standard spinning disk, no SSD. For my purposes, I need to optimize this for "standard" hardware, not for best-in-class. For my test, I first run 'git clone -n <repo>', and then measure the running time of 'git checkout --force HEAD'. On linux, the checkout command runs in 0:12; on Windows, it's about 3:30. > If your numbers are much slower, check for overeager virus scanners and > probably the infamous "User Account Control" (On Vista/7 (8?), the > luafv.sys driver slows down things on the system drive even with UAC turned > off in control panel. The driver can be disabled with "sc config luafv > start= disabled" + reboot. Reenable with "sc config luafv start= auto"). > I confess that I am pretty ignorant about Windows, so I'll have to research these. >> Has anyone considered threading file access to speed this up? In > >> particular, I've got my eye on this loop in unpack-trees.c: > >> > > Its probably worth a try, however, in my experience, doing disk IO in > parallel tends to slow things down due to more disk seeks. > I'd rather try to minimize seeks, ... > In my experience, modern disk controllers are very very good at this; it rarely, if ever, makes sense to try and outsmart them. But, from talking to Windows-savvy people, I believe the issue is not disk seek time, but rather the fact that Windows doesn't cache file stat information. Instead, it goes all the way to the source of truth (i.e., the physical disk) every time it stats a file or directory. That's what causes the checkout to be so slow: all those file stats run serially. Does that sound right? I'm prepared to be wrong about this; but if no one has tried it, then it's probably at least worth an experiment. Thanks, Stefan -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. [-- Attachment #2: Type: text/html, Size: 4940 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-11 0:51 ` Karsten Blees 2013-10-11 5:28 ` Stefan Zager @ 2013-10-11 5:35 ` Stefan Zager 2013-10-11 5:48 ` Duy Nguyen 2013-10-15 22:22 ` pro-logic 1 sibling, 2 replies; 12+ messages in thread From: Stefan Zager @ 2013-10-11 5:35 UTC (permalink / raw) To: Karsten Blees; +Cc: Sebastian Schuberth, git, msysGit On Thu, Oct 10, 2013 at 5:51 PM, Karsten Blees <karsten.blees@gmail.com>wrote: > >> I've noticed that when working with a very large repository using msys > >> git, the initial checkout of a cloned repository is excruciatingly > >> slow (80%+ of total clone time). The root cause, I think, is that git > >> does all the file access serially, and that's really slow on Windows. > >> > > What exactly do you mean by "excruciatingly slow"? > > I just ran a few tests with a big repo (WebKit, ~2GB, ~200k files). A full > checkout with git 1.8.4 on my SSD took 52s on Linux and 81s on Windows. > Xcopy /s took ~4 minutes (so xcopy is much slower than git). On a 'real' HD > (WD Caviar Green) the Windows checkout took ~9 minutes. I'm using blink for my test, which should be more or less indistinguishable from WebKit. I'm using a standard spinning disk, no SSD. For my purposes, I need to optimize this for "standard"-ish hardware, not best-in-class. For my test, I first run 'git clone -n <repo>', and then measure the running time of 'git checkout --force HEAD'. On linux, the checkout command runs in 0:12; on Windows, it's about 3:30. > If your numbers are much slower, check for overeager virus scanners and > probably the infamous "User Account Control" (On Vista/7 (8?), the > luafv.sys driver slows down things on the system drive even with UAC turned > off in control panel. The driver can be disabled with "sc config luafv > start= disabled" + reboot. Reenable with "sc config luafv start= auto"). I confess that I am pretty ignorant about Windows, so I'll have to research these. >> Has anyone considered threading file access to speed this up? In > >> particular, I've got my eye on this loop in unpack-trees.c: > >> > > Its probably worth a try, however, in my experience, doing disk IO in > parallel tends to slow things down due to more disk seeks. > I'd rather try to minimize seeks, ... > In my experience, modern disk controllers are very very good at this; it rarely, if ever, makes sense to try and outsmart them. But, from talking to Windows-savvy people, I believe the issue is not disk seek time, but rather the fact that Windows doesn't cache file stat information. Instead, it goes all the way to the source of truth (i.e., the physical disk) every time it stats a file or directory. That's what causes the checkout to be so slow: all those file stats run serially. Does that sound right? I'm prepared to be wrong about this; but if no one has tried it, then it's probably at least worth an experiment. Thanks, Stefan ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-11 5:35 ` Stefan Zager @ 2013-10-11 5:48 ` Duy Nguyen 2013-10-15 22:22 ` pro-logic 1 sibling, 0 replies; 12+ messages in thread From: Duy Nguyen @ 2013-10-11 5:48 UTC (permalink / raw) To: Stefan Zager Cc: Karsten Blees, Sebastian Schuberth, Git Mailing List, msysGit On Fri, Oct 11, 2013 at 12:35 PM, Stefan Zager <szager@google.com> wrote: > For my test, I first run 'git clone -n <repo>', and then measure the > running time of 'git checkout --force HEAD'. On linux, the checkout > command runs in 0:12; on Windows, it's about 3:30. try git read-tree HEAD git ls-files | xargs -P=XXX -n=YYYY git checkout-index That should give you a rough idea how much gain (or loss) by parallelization -- Duy -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-11 5:35 ` Stefan Zager 2013-10-11 5:48 ` Duy Nguyen @ 2013-10-15 22:22 ` pro-logic 2013-10-17 16:50 ` Karsten Blees 1 sibling, 1 reply; 12+ messages in thread From: pro-logic @ 2013-10-15 22:22 UTC (permalink / raw) To: msysgit; +Cc: Karsten Blees, Sebastian Schuberth, git, szager [-- Attachment #1: Type: text/plain, Size: 1920 bytes --] I also get fairly slow performance out of the checkout / reset operations on windows. This discussion got me trying to work out what's taking so long on windows. To help I used killcache [1] to flush the HDD cache and Very Sleepy [2] to profile the code. I couldn't use the GIT_TRACE_PERFORMANCE [3] patch as that seems to only work on script commands, and in my case I just get a result of "335 seconds git reset --hard head" from the log. After running killcache I ran very sleepy connected to git, and according to the profile: 95.5% of the time is spent in do_lstat (mingw.c) / NtQueryFullAttributeFile (ntdll) For fun, not knowing if I would break anything or not (it probably does), I wrapped the entire unpack_trees method in the fscache [4] and the total git reset --hard head time fell from 335 seconds to 28 seconds, a 11x improvement. Albert [1] https://github.com/sgraham/killcache [2] http://www.codersnotes.com/sleepy [3] https://github.com/msysgit/git/pull/38 [4] https://github.com/msysgit/git/pull/94 -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. [-- Attachment #2: Type: text/html, Size: 2492 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-15 22:22 ` pro-logic @ 2013-10-17 16:50 ` Karsten Blees 2013-10-21 22:58 ` pro-logic 0 siblings, 1 reply; 12+ messages in thread From: Karsten Blees @ 2013-10-17 16:50 UTC (permalink / raw) To: pro-logic, msysgit; +Cc: Sebastian Schuberth, git, szager Am 16.10.2013 00:22, schrieb pro-logic: > I also get fairly slow performance out of the checkout / reset > operations on windows. > > This discussion got me trying to work out what's taking so long on > windows. To help I used killcache [1] to flush the HDD cache and > Very Sleepy [2] to profile the code. I couldn't use the > GIT_TRACE_PERFORMANCE [3] patch as that seems to only work on script > commands, and in my case I just get a result of "335 seconds git > reset --hard head" from the log. The trace_performance functions require manual instrumentation of the code sections you want to measure, e.g. like this [1]. Output looks like this for a full WebKit checkout (Win7 x64, Core i7 860, WD VelociRaptor 300, NTFS, no virus scanner, no luafv, no defragger): trace: at entry.c:128, time: 135.786 s: write_entry::create trace: at entry.c:129, time: 101.6 s: write_entry::stream trace: at entry.c:130, time: 0 s: write_entry::read trace: at entry.c:131, time: 0 s: write_entry::convert trace: at entry.c:132, time: 0 s: write_entry::write trace: at entry.c:133, time: 4.71825 s: write_entry::close trace: at compat/mingw.c:2150, time: 5.68786 s: mingw_lstat (called 661660 times) trace: at compat/mingw.c:2151, time: 259.219 s: command: c:\git\msysgit\git\git-checkout.exe -f HEAD > After running killcache I ran very sleepy connected to git, and > according to the profile: 95.5% of the time is spent in do_lstat > (mingw.c) / NtQueryFullAttributeFile (ntdll) Very Sleepy confirmed my numbers from above: lstat was always much smaller than create/stream/read/write. Could you post details about your test setup? Are you still using WebKit for your tests? > For fun, not knowing if I would break anything or not (it probably > does), I wrapped the entire unpack_trees method in the fscache [4] > and the total git reset --hard head time fell from 335 seconds to 28 > seconds, a 11x improvement. Hmmm...this doesn't work for me at all. Fscache isn't updated during checkout, so lstat-checks whether creating a file or directory succeeded will fail. $ git config core.fscache true $ time git checkout -f HEAD Unlink of file 'Examples' failed. Should I try again? (y/n) n warning: unable to unlink Examples: Permission denied fatal: cannot create directory at 'Examples': Permission denied Karsten [1] https://github.com/kblees/git/commit/b8eca278 -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Windows performance / threading file access 2013-10-17 16:50 ` Karsten Blees @ 2013-10-21 22:58 ` pro-logic 2013-10-22 14:30 ` Karsten Blees 0 siblings, 1 reply; 12+ messages in thread From: pro-logic @ 2013-10-21 22:58 UTC (permalink / raw) To: msysgit; +Cc: pro-logic, Sebastian Schuberth, git, szager [-- Attachment #1: Type: text/plain, Size: 3882 bytes --] > The trace_performance functions require manual instrumentation of the code sections you want to measure Ahh a case of RTFM :) > Could you post details about your test setup? Are you still using WebKit for your tests? I'm on Win7 x64, Core i5 M560, WD 7200 Laptop HDD, NTSF, no virus scanner, truecrypt, no defragger. I've tried to be a bit smarter with the intent of my code, and this is what I came up with. diff --git a/cache.h b/cache.h index 4bf19e3..2e9fb1f 100644 --- a/cache.h +++ b/cache.h @@ -294,7 +294,7 @@ extern void free_name_hash(struct index_state *istate); #define active_cache_changed (the_index.cache_changed) #define active_cache_tree (the_index.cache_tree) -#define read_cache() read_index(&the_index) +#define read_cache() read_index_preload(&the_index, NULL) #define read_cache_from(path) read_index_from(&the_index, (path)) #define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec)) #define is_cache_unborn() is_index_unborn(&the_index) diff --git a/read-cache.c b/read-cache.c index c3d5e35..5fb2788 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1866,7 +1866,7 @@ int read_index_unmerged(struct index_state *istate) int i; int unmerged = 0; - read_index(istate); + read_index_preload(istate, NULL); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; struct cache_entry *new_ce; -- Interestingly when I run on a cleanly checked out blink repo my changes seem to make matters worse in terms of performance, but when working on a repo with ignored files in it it seems to work better. So for point of comparison I decided to run it on a comparison on a repo with working ignored files in it in this case msysgit/git after a 'make install'. When I get a few hours I'll try to build blink and re-run the numbers on a much much larger repo. This comparison is a average of 3 cold cache runs of the kb/fscache-v4 [a] vs kb/fscache-v4 with my above changes applied [b], with preloadindex and fscache set to true. For comparison git status -s [a] 3.02s [b] 2.92s git reset --hard head [a] 3.67s [b] 3.09s git add -u [a] 2.89s [b] 2.08s I noticed something interesting. Preload index uses 20 threads to do the work. When I was keeping an eye on them in task manager some threads will finish quite quickly, while others will run a lot longer. The way I understand the code at the moment the threads get equal chunks of work to perform. It's quite lilkely that even more performance could be obtained out of preload if the work splitting was 'smarter'. My currently best idea would be to use something like a lock-free queue to queue up the work and let the threads get the work of the queue. That way all threads are busy with work for longer. A candidate for the implementation would be libfds [1] queue. However my issue with this library and the reason I haven't tried to integrate is simply because the code expressly has no license. [1] http://www.liblfds.org/ -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. [-- Attachment #2: Type: text/html, Size: 5406 bytes --] ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: Re: Windows performance / threading file access 2013-10-21 22:58 ` pro-logic @ 2013-10-22 14:30 ` Karsten Blees 2013-10-22 14:49 ` Sebastian Schuberth 0 siblings, 1 reply; 12+ messages in thread From: Karsten Blees @ 2013-10-22 14:30 UTC (permalink / raw) To: pro-logic, msysgit; +Cc: Sebastian Schuberth, git, szager Am 22.10.2013 00:58, schrieb pro-logic: >> The trace_performance functions require manual instrumentation of >> the code sections you want to measure > Ahh a case of RTFM :) > >> Could you post details about your test setup? Are you still using >> WebKit for your tests? > I'm on Win7 x64, Core i5 M560, WD 7200 Laptop HDD, NTSF, no virus > scanner, truecrypt, no defragger. > OK, so truecrypt and luafv may screw things up for you (according to my measurements, luafv roughly doubles lstat times on C:). > I've tried to be a bit smarter with the intent of my code, and this > is what I came up with. > > diff --git a/cache.h b/cache.h > index 4bf19e3..2e9fb1f 100644 > --- a/cache.h > +++ b/cache.h > @@ -294,7 +294,7 @@ extern void free_name_hash(struct index_state *istate); > #define active_cache_changed (the_index.cache_changed) > #define active_cache_tree (the_index.cache_tree) > > -#define read_cache() read_index(&the_index) > +#define read_cache() read_index_preload(&the_index, NULL) > #define read_cache_from(path) read_index_from(&the_index, (path)) > #define read_cache_preload(pathspec) read_index_preload(&the_index, (pathspec)) > #define is_cache_unborn() is_index_unborn(&the_index) > diff --git a/read-cache.c b/read-cache.c > index c3d5e35..5fb2788 100644 > --- a/read-cache.c > +++ b/read-cache.c > @@ -1866,7 +1866,7 @@ int read_index_unmerged(struct index_state *istate) > int i; > int unmerged = 0; > > -read_index(istate); > +read_index_preload(istate, NULL); > for (i = 0; i < istate->cache_nr; i++) { > struct cache_entry *ce = istate->cache[i]; > struct cache_entry *new_ce; > -- > Ahh, I thought that you had enabled fscache during the entire checkout. > Interestingly when I run on a cleanly checked out blink repo my > changes seem to make matters worse in terms of performance, but when > working on a repo with ignored files in it it seems to work better. > So for point of comparison I decided to run it on a comparison on a > repo with working ignored files in it in this case msysgit/git after > a 'make install'. When I get a few hours I'll try to build blink and > re-run the numbers on a much much larger repo. > > This comparison is a average of 3 cold cache runs of the > kb/fscache-v4 [a] vs kb/fscache-v4 with my above changes applied [b], > with preloadindex and fscache set to true. > > For comparison > git status -s > [a] 3.02s > [b] 2.92s > > git reset --hard head > [a] 3.67s > [b] 3.09s > These numbers look far too good, so you don't actually do a fresh checkout, do you? I mean, delete all files except .git; killcache; git reset --hard / git checkout -f? That would also explain your 95% lstat times, if there's nothing to do... > git add -u > [a] 2.89s > [b] 2.08s > > > I noticed something interesting. Preload index uses 20 threads to do > the work. When I was keeping an eye on them in task manager some > threads will finish quite quickly, while others will run a lot > longer. The way I understand the code at the moment the threads get > equal chunks of work to perform. It's quite lilkely that even more > performance could be obtained out of preload if the work splitting > was 'smarter'. My currently best idea would be to use something like > a lock-free queue to queue up the work and let the threads get the > work of the queue. That way all threads are busy with work for > longer. A candidate for the implementation would be libfds [1] queue. > However my issue with this library and the reason I haven't tried to > integrate is simply because the code expressly has no license. > As cache/cache_nr are not modified by the threads, you actually don't need a lock-free queue. An atomic counter shared by all threads should suffice (i.e. pthread's equivalent to InterlockedIncrement/InterlockedAdd). Karsten -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: Windows performance / threading file access 2013-10-22 14:30 ` Karsten Blees @ 2013-10-22 14:49 ` Sebastian Schuberth 2013-10-22 15:40 ` Karsten Blees 0 siblings, 1 reply; 12+ messages in thread From: Sebastian Schuberth @ 2013-10-22 14:49 UTC (permalink / raw) To: Karsten Blees; +Cc: pro-logic, msysGit Mailinglist, Git Mailing List, szager On Tue, Oct 22, 2013 at 4:30 PM, Karsten Blees <karsten.blees@gmail.com> wrote: >>> Could you post details about your test setup? Are you still using >>> WebKit for your tests? >> I'm on Win7 x64, Core i5 M560, WD 7200 Laptop HDD, NTSF, no virus >> scanner, truecrypt, no defragger. >> > > OK, so truecrypt and luafv may screw things up for you (according to my measurements, luafv roughly doubles lstat times on C:). Aren't we disabling UAC / LUAFV on a per-executable basis using manifests? At least the blog article at [1] suggests that we are in fact doing it the right way using our script to genera the manifests [2]. Oh but wait, we're not generating a manifest for git.exe itself, only for executables that contain "setup", "install", "update", "patch" etc. So maybe having a manifest for git.exe, too, would improve performance? [1] http://blogs.msdn.com/b/alexcarp/archive/2009/06/25/the-deal-with-luafv-sys.aspx [2] https://github.com/msysgit/msysgit/blob/master/share/msysGit/make-manifests.sh -- Sebastian Schuberth -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: Windows performance / threading file access 2013-10-22 14:49 ` Sebastian Schuberth @ 2013-10-22 15:40 ` Karsten Blees 0 siblings, 0 replies; 12+ messages in thread From: Karsten Blees @ 2013-10-22 15:40 UTC (permalink / raw) To: Sebastian Schuberth Cc: pro-logic, msysGit Mailinglist, Git Mailing List, szager Am 22.10.2013 16:49, schrieb Sebastian Schuberth: > On Tue, Oct 22, 2013 at 4:30 PM, Karsten Blees <karsten.blees@gmail.com> wrote: > >>>> Could you post details about your test setup? Are you still using >>>> WebKit for your tests? >>> I'm on Win7 x64, Core i5 M560, WD 7200 Laptop HDD, NTSF, no virus >>> scanner, truecrypt, no defragger. >>> >> >> OK, so truecrypt and luafv may screw things up for you (according to my measurements, luafv roughly doubles lstat times on C:). > > Aren't we disabling UAC / LUAFV on a per-executable basis using > manifests? At least the blog article at [1] suggests that we are in > fact doing it the right way using our script to genera the manifests > [2]. > > Oh but wait, we're not generating a manifest for git.exe itself, only > for executables that contain "setup", "install", "update", "patch" > etc. So maybe having a manifest for git.exe, too, would improve > performance? > Even with UAC disabled in control panel, the luafv.sys driver slows things down on C: (no impact on non-system drives). Procmon shows that with disabled luafv, GetFileAttributesEx is a single FASTIO call. With luafv running, FASTIO fails and is followed by three IRP calls (open, query, close). I haven't tried with UAC enabled, or if disabling virtualization for git.exe has an impact. Karsten -- -- *** Please reply-to-all at all times *** *** (do not pretend to know who is subscribed and who is not) *** *** Please avoid top-posting. *** The msysGit Wiki is here: https://github.com/msysgit/msysgit/wiki - Github accounts are free. You received this message because you are subscribed to the Google Groups "msysGit" group. To post to this group, send email to msysgit@googlegroups.com To unsubscribe from this group, send email to msysgit+unsubscribe@googlegroups.com For more options, and view previous threads, visit this group at http://groups.google.com/group/msysgit?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "msysGit" group. To unsubscribe from this group and stop receiving emails from it, send an email to msysgit+unsubscribe@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-10-22 15:40 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-10-10 18:18 Windows performance / threading file access Stefan Zager 2013-10-10 20:19 ` Sebastian Schuberth 2013-10-11 0:51 ` Karsten Blees 2013-10-11 5:28 ` Stefan Zager 2013-10-11 5:35 ` Stefan Zager 2013-10-11 5:48 ` Duy Nguyen 2013-10-15 22:22 ` pro-logic 2013-10-17 16:50 ` Karsten Blees 2013-10-21 22:58 ` pro-logic 2013-10-22 14:30 ` Karsten Blees 2013-10-22 14:49 ` Sebastian Schuberth 2013-10-22 15:40 ` Karsten Blees
Code repositories for project(s) associated with this public inbox https://80x24.org/mirrors/git.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).