git@vger.kernel.org list mirror (unofficial, one of many)
 help / color / mirror / Atom feed
* [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
@ 2013-07-06  3:44 Kyle McKay
  2013-07-07  0:24 ` Jonathan Nieder
  0 siblings, 1 reply; 8+ messages in thread
From: Kyle McKay @ 2013-07-06  3:44 UTC (permalink / raw)
  To: git; +Cc: David Rothenberger, Petr Baudis, Jonathan Nieder, Daniel Shahaf

When attempting to git-svn fetch files from an svn https?: url using
the serf library (the only choice starting with svn 1.8) the following
errors can occur:

Temp file with moniker 'svn_delta' already in use at Git.pm line 1250
Temp file with moniker 'git_blob' already in use at Git.pm line 1250

David Rothenberger <daveroth@acm.org> has determined the cause to
be that ra_serf does not drive the delta editor in a depth-first
manner [...]. Instead, the calls come in this order:

1. open_root
2. open_directory
3. add_file
4. apply_textdelta
5. add_file
6. apply_textdelta

This change causes a new temp file moniker to be generated if the
one that would otherwise have been used is currently locked.

Signed-off-by: Kyle J. McKay <mackyle@gmail.com>
---
perl/Git/SVN/Fetcher.pm | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/perl/Git/SVN/Fetcher.pm b/perl/Git/SVN/Fetcher.pm
index bd17418..10edb27 100644
--- a/perl/Git/SVN/Fetcher.pm
+++ b/perl/Git/SVN/Fetcher.pm
@@ -315,11 +315,13 @@ sub change_file_prop {
sub apply_textdelta {
	my ($self, $fb, $exp) = @_;
	return undef if $self->is_path_ignored($fb->{path});
-	my $fh = $::_repository->temp_acquire('svn_delta');
+	my $suffix = 0;
+	++$suffix while $::_repository->temp_is_locked("svn_delta_${$}_ 
$suffix");
+	my $fh = $::_repository->temp_acquire("svn_delta_${$}_$suffix");
	# $fh gets auto-closed() by SVN::TxDelta::apply(),
	# (but $base does not,) so dup() it for reading in close_file
	open my $dup, '<&', $fh or croak $!;
-	my $base = $::_repository->temp_acquire('git_blob');
+	my $base = $::_repository->temp_acquire("git_blob_${$}_$suffix");

	if ($fb->{blob}) {
		my ($base_is_link, $size);
-- 
1.8.3

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-06  3:44 [PATCH 2/2] git-svn: allow git-svn fetching to work using serf Kyle McKay
@ 2013-07-07  0:24 ` Jonathan Nieder
  2013-07-07  2:13   ` Kyle McKay
  0 siblings, 1 reply; 8+ messages in thread
From: Jonathan Nieder @ 2013-07-07  0:24 UTC (permalink / raw)
  To: Kyle McKay; +Cc: git, David Rothenberger, Petr Baudis, Daniel Shahaf, Eric Wong

(cc-ing Eric Wong, who wrote this code)
Hi,

Kyle McKay wrote:

> Temp file with moniker 'svn_delta' already in use at Git.pm line 1250
> Temp file with moniker 'git_blob' already in use at Git.pm line 1250
>
> David Rothenberger <daveroth@acm.org> has determined the cause to
> be that ra_serf does not drive the delta editor in a depth-first
> manner [...]. Instead, the calls come in this order:
[...]
> --- a/perl/Git/SVN/Fetcher.pm
> +++ b/perl/Git/SVN/Fetcher.pm
> @@ -315,11 +315,13 @@ sub change_file_prop {
> sub apply_textdelta {
> 	my ($self, $fb, $exp) = @_;
> 	return undef if $self->is_path_ignored($fb->{path});
> -	my $fh = $::_repository->temp_acquire('svn_delta');
> +	my $suffix = 0;
> +	++$suffix while $::_repository->temp_is_locked("svn_delta_${$}_$suffix");
> +	my $fh = $::_repository->temp_acquire("svn_delta_${$}_$suffix");
> 	# $fh gets auto-closed() by SVN::TxDelta::apply(),
> 	# (but $base does not,) so dup() it for reading in close_file
> 	open my $dup, '<&', $fh or croak $!;
> -	my $base = $::_repository->temp_acquire('git_blob');
> +	my $base = $::_repository->temp_acquire("git_blob_${$}_$suffix");

Thanks for your work tracking this down.

I'm a bit confused.  Are you saying that apply_textdelta gets called
multiple times in a row without an intervening close_file?

Puzzled,
Jonathan

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07  0:24 ` Jonathan Nieder
@ 2013-07-07  2:13   ` Kyle McKay
  2013-07-07  2:23     ` Jonathan Nieder
  0 siblings, 1 reply; 8+ messages in thread
From: Kyle McKay @ 2013-07-07  2:13 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: git, David Rothenberger, Petr Baudis, Daniel Shahaf, Eric Wong

On Jul 6, 2013, at 17:24, Jonathan Nieder wrote:
> (cc-ing Eric Wong, who wrote this code)
> Hi,
>
> Kyle McKay wrote:
>
>> Temp file with moniker 'svn_delta' already in use at Git.pm line 1250
>> Temp file with moniker 'git_blob' already in use at Git.pm line 1250
>>
>> David Rothenberger <daveroth@acm.org> has determined the cause to
>> be that ra_serf does not drive the delta editor in a depth-first
>> manner [...]. Instead, the calls come in this order:
> [...]
>> --- a/perl/Git/SVN/Fetcher.pm
>> +++ b/perl/Git/SVN/Fetcher.pm
>> @@ -315,11 +315,13 @@ sub change_file_prop {
>> sub apply_textdelta {
>> 	my ($self, $fb, $exp) = @_;
>> 	return undef if $self->is_path_ignored($fb->{path});
>> -	my $fh = $::_repository->temp_acquire('svn_delta');
>> +	my $suffix = 0;
>> +	++$suffix while $::_repository->temp_is_locked("svn_delta_${$}_ 
>> $suffix");
>> +	my $fh = $::_repository->temp_acquire("svn_delta_${$}_$suffix");
>> 	# $fh gets auto-closed() by SVN::TxDelta::apply(),
>> 	# (but $base does not,) so dup() it for reading in close_file
>> 	open my $dup, '<&', $fh or croak $!;
>> -	my $base = $::_repository->temp_acquire('git_blob');
>> +	my $base = $::_repository->temp_acquire("git_blob_${$}_$suffix");
>
> Thanks for your work tracking this down.
>
> I'm a bit confused.  Are you saying that apply_textdelta gets called
> multiple times in a row without an intervening close_file?

Unless bulk updates are disabled when using the serf access method  
(the only one available with svn 1.8) for https?: urls,  
apply_textdelta does indeed get called multiple times in a row without  
an intervening temp_release.

Two temp files are created for each apply_textdelta ('svn_delta...'  
and 'git_blob...').  In my tests it seems that most of the time the  
two temp files are enough, but occasionally as many as six will be  
open at the same time.

I suspect this maximum number is related to the maximum number of  
simultaneous connections the serf access method will use which  
defaults to 4.  Therefore I would expect to see as many as 8 temp  
files (4 each for 'svn_delta...' and 'git_blob...'), but I have only  
been able to trigger creation of 6 temp files so far.

Kyle

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07  2:13   ` Kyle McKay
@ 2013-07-07  2:23     ` Jonathan Nieder
  2013-07-07  2:46       ` Kyle McKay
  0 siblings, 1 reply; 8+ messages in thread
From: Jonathan Nieder @ 2013-07-07  2:23 UTC (permalink / raw)
  To: Kyle McKay; +Cc: git, David Rothenberger, Petr Baudis, Daniel Shahaf, Eric Wong

Kyle McKay wrote:

> Unless bulk updates are disabled when using the serf access method
> (the only one available with svn 1.8) for https?: urls,
> apply_textdelta does indeed get called multiple times in a row
> without an intervening temp_release.

You mean "Unless bulk updates are enabled" and "without an intervening
close_file", right?

Unlike the non-depth-first thing, that sounds basically broken ---
what would be stopping subversion from calling the editor's close
method when done with each file?  I can't see much reason unless it is
calling apply_textdelta multiple times in parallel --- is it doing
that, and if so is git-svn able to cope with that?

This sounds like something that should be fixed in ra_serf.

But if the number of overlapping open text nodes is bounded by a low
number, the workaround of using multiple temp files sounds ok as a way
of dealing with unfixed versions of Subversion.

Jonathan

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07  2:23     ` Jonathan Nieder
@ 2013-07-07  2:46       ` Kyle McKay
  2013-07-07 13:39         ` Daniel Shahaf
  0 siblings, 1 reply; 8+ messages in thread
From: Kyle McKay @ 2013-07-07  2:46 UTC (permalink / raw)
  To: Jonathan Nieder
  Cc: git, David Rothenberger, Petr Baudis, Daniel Shahaf, Eric Wong

On Jul 6, 2013, at 19:23, Jonathan Nieder wrote:
> Kyle McKay wrote:
>
>> Unless bulk updates are disabled when using the serf access method
>> (the only one available with svn 1.8) for https?: urls,
>> apply_textdelta does indeed get called multiple times in a row
>> without an intervening temp_release.
>
> You mean "Unless bulk updates are enabled" and "without an intervening
> close_file", right?

The problem seems to be skelta mode although it may just be the fact  
that ra_serf has multiple connections outstanding and since ra_neon  
only ever has one it can't happen over ra_neon.

If the server disables bulk updates (SVNAllowBulkUpdates Off) all  
clients are forced to use skelta mode, even ra_neon clients.

> This sounds like something that should be fixed in ra_serf.

Yes, but apparently it will not be.

> But if the number of overlapping open text nodes is bounded by a low
> number, the workaround of using multiple temp files sounds ok as a way
> of dealing with unfixed versions of Subversion.

I believe it will never exceed twice ('svn_delta...' and  
'git_blob...') the maximum number of serf connections allowed.  Four  
by default (hard-coded prior to svn 1.8).  Limited to between 1 and 8  
on svn 1.8.  Actually it looks like from my testing that it won't ever  
exceed twice the (max number of serf connections - 1).

Kyle

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07  2:46       ` Kyle McKay
@ 2013-07-07 13:39         ` Daniel Shahaf
  2013-07-07 16:18           ` David Rothenberger
  2013-07-07 18:27           ` Kyle McKay
  0 siblings, 2 replies; 8+ messages in thread
From: Daniel Shahaf @ 2013-07-07 13:39 UTC (permalink / raw)
  To: Kyle McKay
  Cc: Jonathan Nieder, git, David Rothenberger, Petr Baudis, Eric Wong

Kyle McKay wrote on Sat, Jul 06, 2013 at 19:46:40 -0700:
> On Jul 6, 2013, at 19:23, Jonathan Nieder wrote:
>> Kyle McKay wrote:
>>
>>> Unless bulk updates are disabled when using the serf access method
>>> (the only one available with svn 1.8) for https?: urls,
>>> apply_textdelta does indeed get called multiple times in a row
>>> without an intervening temp_release.
>>
>> You mean "Unless bulk updates are enabled" and "without an intervening
>> close_file", right?
>
> The problem seems to be skelta mode although it may just be the fact  
> that ra_serf has multiple connections outstanding and since ra_neon only 
> ever has one it can't happen over ra_neon.
>
> If the server disables bulk updates (SVNAllowBulkUpdates Off) all  
> clients are forced to use skelta mode, even ra_neon clients.

As Brane and I have pointed out, git-svn can instruct libsvn_* to use
bulk updates regardless of the server version, by setting
SVN_CONFIG_OPTION_HTTP_BULK_UPDATES (new in 1.8).

If you have questions about that, though, please address them to
users@subversion.apache.org (the proper list for API usage questions),
not to me personally.

Cheers,

Daniel

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07 13:39         ` Daniel Shahaf
@ 2013-07-07 16:18           ` David Rothenberger
  2013-07-07 18:27           ` Kyle McKay
  1 sibling, 0 replies; 8+ messages in thread
From: David Rothenberger @ 2013-07-07 16:18 UTC (permalink / raw)
  To: users; +Cc: Kyle McKay, Jonathan Nieder, git, Petr Baudis, Eric Wong

On 7/7/2013 6:39 AM, Daniel Shahaf wrote:
> Kyle McKay wrote on Sat, Jul 06, 2013 at 19:46:40 -0700:
>> On Jul 6, 2013, at 19:23, Jonathan Nieder wrote:
>>> Kyle McKay wrote:
>>>
>>>> Unless bulk updates are disabled when using the serf access method
>>>> (the only one available with svn 1.8) for https?: urls,
>>>> apply_textdelta does indeed get called multiple times in a row
>>>> without an intervening temp_release.
>>>
>>> You mean "Unless bulk updates are enabled" and "without an intervening
>>> close_file", right?
>>
>> The problem seems to be skelta mode although it may just be the fact  
>> that ra_serf has multiple connections outstanding and since ra_neon only 
>> ever has one it can't happen over ra_neon.
>>
>> If the server disables bulk updates (SVNAllowBulkUpdates Off) all  
>> clients are forced to use skelta mode, even ra_neon clients.
> 
> As Brane and I have pointed out, git-svn can instruct libsvn_* to use
> bulk updates regardless of the server version, by setting
> SVN_CONFIG_OPTION_HTTP_BULK_UPDATES (new in 1.8).

According to the table in the release notes [1], Skelta mode will be
used if the 1.7 or 1.8 server sets SVNAllowBulkUpdates to Off,
regardless of what the client sets in the configuration.

Is that not true?

[1] https://subversion.apache.org/docs/release-notes/1.8.html#neon-deleted

-- 
David Rothenberger  ----  daveroth@acm.org

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [PATCH 2/2] git-svn: allow git-svn fetching to work using serf
  2013-07-07 13:39         ` Daniel Shahaf
  2013-07-07 16:18           ` David Rothenberger
@ 2013-07-07 18:27           ` Kyle McKay
  1 sibling, 0 replies; 8+ messages in thread
From: Kyle McKay @ 2013-07-07 18:27 UTC (permalink / raw)
  To: git; +Cc: Jonathan Nieder, David Rothenberger, Petr Baudis, Eric Wong

I forwarded the "SVNAllowBulkUpdates Off" question to the users@subversion.apache.org 
  list and here's the reply:

On Jul 7, 2013, at 11:11, Lieven Govaerts wrote:
> On Sun, Jul 7, 2013 at 4:48 PM, Kyle McKay <mackyle@gmail.com> wrote:
>> On Jul 7, 2013, at 06:39, Daniel Shahaf wrote:
>>>
>>> Kyle McKay wrote on Sat, Jul 06, 2013 at 19:46:40 -0700:
>>>>
>>>> On Jul 6, 2013, at 19:23, Jonathan Nieder wrote:
>>>>>
>>>>> Kyle McKay wrote:
>>>>>
>>>>>> Unless bulk updates are disabled when using the serf access  
>>>>>> method
>>>>>> (the only one available with svn 1.8) for https?: urls,
>>>>>> apply_textdelta does indeed get called multiple times in a row
>>>>>> without an intervening temp_release.
>>>>>
>>>>>
>>>>> You mean "Unless bulk updates are enabled" and "without an  
>>>>> intervening
>>>>> close_file", right?
>>>>
>>>>
>>>> The problem seems to be skelta mode although it may just be the  
>>>> fact
>>>> that ra_serf has multiple connections outstanding and since  
>>>> ra_neon only
>>>> ever has one it can't happen over ra_neon.
>>>>
>>>> If the server disables bulk updates (SVNAllowBulkUpdates Off) all
>>>> clients are forced to use skelta mode, even ra_neon clients.
>>>
>>>
>>> As Brane and I have pointed out, git-svn can instruct libsvn_* to  
>>> use
>>> bulk updates regardless of the server version, by setting
>>> SVN_CONFIG_OPTION_HTTP_BULK_UPDATES (new in 1.8).
>>>
>>> If you have questions about that, though, please address them to
>>> users@subversion.apache.org (the proper list for API usage  
>>> questions),
>>> not to me personally.
>>
>>
>> According to the table at
>> <http://subversion.apache.org/docs/release-notes/1.8.html#serf-skelta-default 
>> >,
>> if the server sets SVNAllowBulkUpdates Off, the client will be  
>> forced to use
>> skelta no matter what the client setting is.
>
> Indeed, the server admin has the final say in which mode is actually
> used. SVNAllowBulkUpdates Off is only advised if the server admin
> wants a log line per accessed resource. I doubt it's used a lot, but
> the option is there.
>
>>
>> Is that table incorrect?
>
> No, that table is correct.
>
> Lieven

So the final say so on whether or not bulk updates are allowed is on  
the server side which means git-svn really needs to handle skelta mode  
on the client side properly when using ra-serf to guarantee  
functionality with all subversion server configurations.

Kyle

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2013-07-07 18:27 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-07-06  3:44 [PATCH 2/2] git-svn: allow git-svn fetching to work using serf Kyle McKay
2013-07-07  0:24 ` Jonathan Nieder
2013-07-07  2:13   ` Kyle McKay
2013-07-07  2:23     ` Jonathan Nieder
2013-07-07  2:46       ` Kyle McKay
2013-07-07 13:39         ` Daniel Shahaf
2013-07-07 16:18           ` David Rothenberger
2013-07-07 18:27           ` Kyle McKay

git@vger.kernel.org list mirror (unofficial, one of many)

This inbox may be cloned and mirrored by anyone:

	git clone --mirror https://public-inbox.org/git
	git clone --mirror http://ou63pmih66umazou.onion/git
	git clone --mirror http://czquwvybam4bgbro.onion/git
	git clone --mirror http://hjrcffqmbrq6wope.onion/git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V1 git git/ https://public-inbox.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors.
Newsgroups are available over NNTP:
	nntp://news.public-inbox.org/inbox.comp.version-control.git
	nntp://ou63pmih66umazou.onion/inbox.comp.version-control.git
	nntp://czquwvybam4bgbro.onion/inbox.comp.version-control.git
	nntp://hjrcffqmbrq6wope.onion/inbox.comp.version-control.git
	nntp://news.gmane.io/gmane.comp.version-control.git
 note: .onion URLs require Tor: https://www.torproject.org/

code repositories for the project(s) associated with this inbox:

	https://80x24.org/mirrors/git.git

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git