From: Jeff King <peff@peff.net>
To: Jonathan Tan <jonathantanmy@google.com>
Cc: matheus.bernardino@usp.br, git@vger.kernel.org,
christian.couder@gmail.com, olyatelezhnaya@gmail.com,
pclouds@gmail.com, gitster@pobox.com, jrnieder@gmail.com,
stefanbeller@gmail.com
Subject: Re: [PATCH v2 05/11] object-store: allow threaded access to object reading
Date: Thu, 14 Nov 2019 23:12:15 -0500 [thread overview]
Message-ID: <20191115041215.GB21654@sigill.intra.peff.net> (raw)
In-Reply-To: <20191114181552.137071-1-jonathantanmy@google.com>
On Thu, Nov 14, 2019 at 10:15:52AM -0800, Jonathan Tan wrote:
> > > > A pthread_rwlock would work, but it would be the first use in Git. I
> > > > think we'd need to find an equivalent for compat/win32/pthread.h.
> > >
> > > These[1][2] seems to be the equivalent options on Windows. I'll have
> > > to read these docs more carefully, but [2] seems to be more
> > > interesting in terms of speed. Also, the extra features of [1] are not
> > > really needed for our use case here.
> > >
> > > [1]: https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/reader-writer-spin-locks
> > > [2]: https://docs.microsoft.com/en-us/windows/win32/sync/slim-reader-writer--srw--locks
> >
> > Yeah, looks like it, but I don't have any expertise there (nor a Windows
> > system to test on).
>
> One thing to note is that that if we do this, we'll be having one rwlock
> per pack window. I couldn't find out what the Windows limits were, but
> it seems that pthreads does not mandate having no limit [1]:
Yeah, interesting point. We shouldn't _usually_ have too many windows at
once, but I think this would be the first place where we allocate a
non-constant number of thread mechanisms. And there are degenerate cases
where you might have tens of thousands of packs.
I suspect it's a non-issue in practice, though. Any limits there are
likely related to kernel resources like descriptors. And those
degenerate cases already run into issues there (did you know that Linux
systems limit the number of mmaps a single process can have? I didn't
until I tried repacking a repo with 35,000 packs).
-Peff
next prev parent reply other threads:[~2019-11-15 4:12 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-10 20:27 [GSoC][PATCH 0/4] grep: re-enable threads when cached, w/ parallel inflation Matheus Tavares
2019-08-10 20:27 ` [GSoC][PATCH 1/4] object-store: add lock to read_object_file_extended() Matheus Tavares
2019-08-10 20:27 ` [GSoC][PATCH 2/4] grep: allow locks to be enabled individually Matheus Tavares
2019-08-10 20:27 ` [GSoC][PATCH 3/4] grep: disable grep_read_mutex when possible Matheus Tavares
2019-08-10 20:27 ` [GSoC][PATCH 4/4] grep: re-enable threads in some non-worktree cases Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 00/11] grep: improve threading and fix race conditions Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 01/11] grep: fix race conditions on userdiff calls Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 02/11] grep: fix race conditions at grep_submodule() Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 03/11] grep: fix racy calls in grep_objects() Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 04/11] replace-object: make replace operations thread-safe Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 05/11] object-store: allow threaded access to object reading Matheus Tavares
2019-11-12 2:54 ` Jonathan Tan
2019-11-13 5:20 ` Jeff King
2019-11-14 5:57 ` Matheus Tavares Bernardino
2019-11-14 6:01 ` Jeff King
2019-11-14 18:15 ` Jonathan Tan
2019-11-15 4:12 ` Jeff King [this message]
2019-12-19 22:27 ` Matheus Tavares Bernardino
2020-01-09 22:02 ` Matheus Tavares Bernardino
2020-01-10 19:07 ` Christian Couder
2019-09-30 1:50 ` [PATCH v2 06/11] grep: replace grep_read_mutex by internal obj read lock Matheus Tavares
2019-10-01 19:23 ` [PATCH] squash! " Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 07/11] submodule-config: add skip_if_read option to repo_read_gitmodules() Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 08/11] grep: allow submodule functions to run in parallel Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 09/11] grep: protect packed_git [re-]initialization Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 10/11] grep: re-enable threads in non-worktree case Matheus Tavares
2019-09-30 1:50 ` [PATCH v2 11/11] grep: move driver pre-load out of critical section Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 00/12] grep: improve threading and fix race conditions Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 01/12] grep: fix race conditions on userdiff calls Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 02/12] grep: fix race conditions at grep_submodule() Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 03/12] grep: fix racy calls in grep_objects() Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 04/12] replace-object: make replace operations thread-safe Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 05/12] object-store: allow threaded access to object reading Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 06/12] grep: replace grep_read_mutex by internal obj read lock Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 07/12] submodule-config: add skip_if_read option to repo_read_gitmodules() Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 08/12] grep: allow submodule functions to run in parallel Matheus Tavares
2020-01-29 11:26 ` SZEDER Gábor
2020-01-29 18:49 ` Junio C Hamano
2020-01-29 18:57 ` Junio C Hamano
2020-01-29 20:42 ` Matheus Tavares Bernardino
2020-01-30 13:28 ` Philippe Blain
2020-01-16 2:39 ` [PATCH v3 09/12] grep: protect packed_git [re-]initialization Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 10/12] grep: re-enable threads in non-worktree case Matheus Tavares
2020-01-16 2:39 ` [PATCH v3 11/12] grep: move driver pre-load out of critical section Matheus Tavares
2020-01-16 2:40 ` [PATCH v3 12/12] grep: use no. of cores as the default no. of threads Matheus Tavares
2020-01-16 13:11 ` Victor Leschuk
2020-01-16 14:47 ` [PATCH] " Matheus Tavares
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: http://vger.kernel.org/majordomo-info.html
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191115041215.GB21654@sigill.intra.peff.net \
--to=peff@peff.net \
--cc=christian.couder@gmail.com \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=jonathantanmy@google.com \
--cc=jrnieder@gmail.com \
--cc=matheus.bernardino@usp.br \
--cc=olyatelezhnaya@gmail.com \
--cc=pclouds@gmail.com \
--cc=stefanbeller@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).