[PATCH v1 0/2] git-p4: fix Git LFS pointer parsing

[PATCH v1 0/2] git-p4: fix Git LFS pointer parsing

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Travis-CI uses 'brew' to always install the latest available version of
Git LFS on the OS X build machines (on Linux the version is sticky).
A change in Git LFS 1.2.0 [1] breaks the git-p4 LFS integration [2].

This mini series updates Travis-CI to the latest Git-LFS and Perforce
versions on Linux and Mac and makes git-p4 work with the latest Git LFS
again.

Thanks,
Lars

[1] https://github.com/github/git-lfs/pull/1105
[2] https://travis-ci.org/git/git/jobs/124047942

Lars Schneider (2):
  travis-ci: update Git-LFS and P4 to the latest version
  git-p4: fix Git LFS pointer parsing

 .travis.yml              | 4 ++--
 Documentation/git-p4.txt | 3 ++-
 git-p4.py                | 6 +++---
 3 files changed, 7 insertions(+), 6 deletions(-)

--
2.5.1

[PATCH v1 1/2] travis-ci: update Git-LFS and P4 to the latest version

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 78e433b..4acf617 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,8 @@ addons:
 env:
   global:
     - DEVELOPER=1
-    - P4_VERSION="15.2"
-    - GIT_LFS_VERSION="1.1.0"
+    - P4_VERSION="16.1"
+    - GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
     - GIT_TEST_OPTS="--verbose --tee"
-- 
2.5.1

[PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
command [1] which broke the parsing of this output. Adjust the parser
to the new output and add minimum Git LFS version to the docs.

[1] https://github.com/github/git-lfs/pull/1105

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 Documentation/git-p4.txt | 3 ++-
 git-p4.py                | 6 +++---
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
index 88ba42b..b862cb9 100644
--- a/Documentation/git-p4.txt
+++ b/Documentation/git-p4.txt
@@ -522,7 +522,8 @@ git-p4.largeFileSystem::
 	that large file systems do not support the 'git p4 submit' command.
 	Only Git LFS is implemented right now (see https://git-lfs.github.com/
 	for more information). Download and install the Git LFS command line
-	extension to use this option and configure it like this:
+	extension (minimum version 1.2.0) to use this option and configure it
+	like this:
 +
 -------------
 git config       git-p4.largeFileSystem GitLFS
diff --git a/git-p4.py b/git-p4.py
index 527d44b..d2be574 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
         if pointerProcess.wait():
             os.remove(contentFile)
             die('git-lfs pointer command failed. Did you install the extension?')
-        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
-        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
+        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
+        oid = oidEntry[0].split(' ')[1].split(':')[1]
         localLargeFile = os.path.join(
             os.getcwd(),
             '.git', 'lfs', 'objects', oid[:2], oid[2:4],
@@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
         )
         # LFS Spec states that pointer files should not have the executable bit set.
         gitMode = '100644'
-        return (gitMode, pointerContents, localLargeFile)
+        return (gitMode, pointerFile, localLargeFile)

     def pushFile(self, localLargeFile):
         uploadProcess = subprocess.Popen(
--
2.5.1

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Junio C Hamano]

larsxschneider@gmail.com writes:

> From: Lars Schneider <larsxschneider@gmail.com>
>
> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
> command [1] which broke the parsing of this output. Adjust the parser
> to the new output and add minimum Git LFS version to the docs.

Hmph, adjust to operate with both, or drop the support for the old
one?



>
> [1] https://github.com/github/git-lfs/pull/1105
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
> ---
>  Documentation/git-p4.txt | 3 ++-
>  git-p4.py                | 6 +++---
>  2 files changed, 5 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
> index 88ba42b..b862cb9 100644
> --- a/Documentation/git-p4.txt
> +++ b/Documentation/git-p4.txt
> @@ -522,7 +522,8 @@ git-p4.largeFileSystem::
>  	that large file systems do not support the 'git p4 submit' command.
>  	Only Git LFS is implemented right now (see https://git-lfs.github.com/
>  	for more information). Download and install the Git LFS command line
> -	extension to use this option and configure it like this:
> +	extension (minimum version 1.2.0) to use this option and configure it
> +	like this:
>  +
>  -------------
>  git config       git-p4.largeFileSystem GitLFS
> diff --git a/git-p4.py b/git-p4.py
> index 527d44b..d2be574 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
>          if pointerProcess.wait():
>              os.remove(contentFile)
>              die('git-lfs pointer command failed. Did you install the extension?')
> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
> +        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
> +        oid = oidEntry[0].split(' ')[1].split(':')[1]
>          localLargeFile = os.path.join(
>              os.getcwd(),
>              '.git', 'lfs', 'objects', oid[:2], oid[2:4],
> @@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
>          )
>          # LFS Spec states that pointer files should not have the executable bit set.
>          gitMode = '100644'
> -        return (gitMode, pointerContents, localLargeFile)
> +        return (gitMode, pointerFile, localLargeFile)
>
>      def pushFile(self, localLargeFile):
>          uploadProcess = subprocess.Popen(
> --
> 2.5.1

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Lars Schneider]

> On 19 Apr 2016, at 22:30, Junio C Hamano <gitster@pobox.com> wrote:
> 
> larsxschneider@gmail.com writes:
> 
>> From: Lars Schneider <larsxschneider@gmail.com>
>> 
>> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
>> command [1] which broke the parsing of this output. Adjust the parser
>> to the new output and add minimum Git LFS version to the docs.
> 
> Hmph, adjust to operate with both, or drop the support for the old
> one?
I dropped the support for the older version to keep the code as simple as possible (plus it would be cumbersome to test with an outdated Git LFS version). Since it is probably a niche feature I thought that might be acceptable.

Thanks,
Lars

>> 
>> [1] https://github.com/github/git-lfs/pull/1105
>> 
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
>> ---
>> Documentation/git-p4.txt | 3 ++-
>> git-p4.py                | 6 +++---
>> 2 files changed, 5 insertions(+), 4 deletions(-)
>> 
>> diff --git a/Documentation/git-p4.txt b/Documentation/git-p4.txt
>> index 88ba42b..b862cb9 100644
>> --- a/Documentation/git-p4.txt
>> +++ b/Documentation/git-p4.txt
>> @@ -522,7 +522,8 @@ git-p4.largeFileSystem::
>>    that large file systems do not support the 'git p4 submit' command.
>>    Only Git LFS is implemented right now (see https://git-lfs.github.com/
>>    for more information). Download and install the Git LFS command line
>> -    extension to use this option and configure it like this:
>> +    extension (minimum version 1.2.0) to use this option and configure it
>> +    like this:
>> +
>> -------------
>> git config       git-p4.largeFileSystem GitLFS
>> diff --git a/git-p4.py b/git-p4.py
>> index 527d44b..d2be574 100755
>> --- a/git-p4.py
>> +++ b/git-p4.py
>> @@ -1064,8 +1064,8 @@ class GitLFS(LargeFileSystem):
>>         if pointerProcess.wait():
>>             os.remove(contentFile)
>>             die('git-lfs pointer command failed. Did you install the extension?')
>> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
>> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
>> +        oidEntry = [i for i in pointerFile.split('\n') if i.startswith('oid')]
>> +        oid = oidEntry[0].split(' ')[1].split(':')[1]
>>         localLargeFile = os.path.join(
>>             os.getcwd(),
>>             '.git', 'lfs', 'objects', oid[:2], oid[2:4],
>> @@ -1073,7 +1073,7 @@ class GitLFS(LargeFileSystem):
>>         )
>>         # LFS Spec states that pointer files should not have the executable bit set.
>>         gitMode = '100644'
>> -        return (gitMode, pointerContents, localLargeFile)
>> +        return (gitMode, pointerFile, localLargeFile)
>> 
>>     def pushFile(self, localLargeFile):
>>         uploadProcess = subprocess.Popen(
>> --
>> 2.5.1

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Junio C Hamano]

Lars Schneider <larsxschneider@gmail.com> writes:

>> On 19 Apr 2016, at 22:30, Junio C Hamano <gitster@pobox.com> wrote:
>> 
>> larsxschneider@gmail.com writes:
>> 
>>> From: Lars Schneider <larsxschneider@gmail.com>
>>> 
>>> Git LFS 1.2.0 removed a line from the output of the 'git lfs pointer'
>>> command [1] which broke the parsing of this output. Adjust the parser
>>> to the new output and add minimum Git LFS version to the docs.
>> 
>> Hmph, adjust to operate with both, or drop the support for the old
>> one?

> I dropped the support for the older version to keep the code as
> simple as possible (plus it would be cumbersome to test with an
> outdated Git LFS version). Since it is probably a niche feature I
> thought that might be acceptable.

It is bad enough that clients need to be adjusted at all in the
first place, but I would have found it very troubling if the
problematic change to LFS thing were made in such a way that it
makes backward compatible adjustment on the client code impossible.

But it seems that you could read their output and strip a line that
begins with a known substring to make it compatible with both?

"git P4" itself may be niche and using it with the LFS thing may
even be more so, but in Git land, traditionally we take the backward
compatibility seriously.  If it is not too much work, I'd prefer to
see this done the right way.

Thanks.

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Sebastian Schuberth]

On Tue, Apr 19, 2016 at 11:04 PM, Junio C Hamano <gitster@pobox.com> wrote:

>> I dropped the support for the older version to keep the code as
>> simple as possible (plus it would be cumbersome to test with an
>> outdated Git LFS version). Since it is probably a niche feature I
>> thought that might be acceptable.
>
> It is bad enough that clients need to be adjusted at all in the
> first place, but I would have found it very troubling if the
> problematic change to LFS thing were made in such a way that it
> makes backward compatible adjustment on the client code impossible.

If clients rely on output targeted at human consumption it's not
surprising that these clients need to be adjusted from time to time.
What's troubling is not the change to git-lfs, but the very un-generic
way git-p4 is implemented.

-- 
Sebastian Schuberth

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Junio C Hamano]

Sebastian Schuberth <sschuberth@gmail.com> writes:

> If clients rely on output targeted at human consumption it's not
> surprising that these clients need to be adjusted from time to time.
> What's troubling is not the change to git-lfs, but the very un-generic
> way git-p4 is implemented.

Sounds like the subcommand they are using is not meant for
scripting?  What is the kosher way to get at the information they
can use that is a supported interface for scripters?

Re: [PATCH v1 2/2] git-p4: fix Git LFS pointer parsing

[Sebastian Schuberth]

On Wed, Apr 20, 2016 at 5:30 PM, Junio C Hamano <gitster@pobox.com> wrote:

>> If clients rely on output targeted at human consumption it's not
>> surprising that these clients need to be adjusted from time to time.
>> What's troubling is not the change to git-lfs, but the very un-generic
>> way git-p4 is implemented.
>
> Sounds like the subcommand they are using is not meant for
> scripting?  What is the kosher way to get at the information they
> can use that is a supported interface for scripters?

The "pointer" subcommand indeed is listed under "Low level commands"
by "git lfs" (without any arguments), and as such it probably can be
considered for scripting use. However, before my fix in [1] the
subcommand was printing both output targeted at humans and output
targeted at scripts to stdout. After my fix, only output targeted at
script goes to stdout, and output targeted at humans goes to stderr.

[1] https://github.com/github/git-lfs/pull/1105

-- 
Sebastian Schuberth

[PATCH] t9824: fix broken &&-chain in a subshell

[SZEDER Gábor]

Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
---

As far as I can tell after eyeballing the test script, this was the
only case where the &&-chain was broken.

 t/t9824-git-p4-git-lfs.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
index 0b664a377c45..64f8d18216d4 100755
--- a/t/t9824-git-p4-git-lfs.sh
+++ b/t/t9824-git-p4-git-lfs.sh
@@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and store files in LFS based on compr
 		# We only import HEAD here ("@all" is missing!)
 		git p4 clone --destination="$git" //depot &&
 
-		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
+		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
 		test_file_count_in_dir ".git/lfs/objects" 1 &&
 
 		cat >expect <<-\EOF &&
-- 
2.8.1.99.g5d5236f

Re: [PATCH] t9824: fix broken &&-chain in a subshell

[Lars Schneider]

On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:

> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
> ---
> 
> As far as I can tell after eyeballing the test script, this was the
> only case where the &&-chain was broken.
> 
> t/t9824-git-p4-git-lfs.sh | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
> index 0b664a377c45..64f8d18216d4 100755
> --- a/t/t9824-git-p4-git-lfs.sh
> +++ b/t/t9824-git-p4-git-lfs.sh
> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and store files in LFS based on compr
> 		# We only import HEAD here ("@all" is missing!)
> 		git p4 clone --destination="$git" //depot &&
> 
> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
Thanks! I believe I remember some script/flag to detect broken &&-chains.
Do I imagine that or can anyone point me to it?

Thanks,
Lars


> 		test_file_count_in_dir ".git/lfs/objects" 1 &&
> 
> 		cat >expect <<-\EOF &&
> -- 
> 2.8.1.99.g5d5236f
> 

Re: [PATCH] t9824: fix broken &&-chain in a subshell

[SZEDER Gábor]

Quoting Lars Schneider <larsxschneider@gmail.com>:

> On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:
>
>> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
>> ---
>>
>> As far as I can tell after eyeballing the test script, this was the
>> only case where the &&-chain was broken.
>>
>> t/t9824-git-p4-git-lfs.sh | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
>> index 0b664a377c45..64f8d18216d4 100755
>> --- a/t/t9824-git-p4-git-lfs.sh
>> +++ b/t/t9824-git-p4-git-lfs.sh
>> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and  
>> store files in LFS based on compr
>> 		# We only import HEAD here ("@all" is missing!)
>> 		git p4 clone --destination="$git" //depot &&
>>
>> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
>> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
> Thanks! I believe I remember some script/flag to detect broken &&-chains.
> Do I imagine that or can anyone point me to it?

It was introduced in bb79af9d0970 (t/test-lib: introduce --chain-lint
option, 2015-03-20) and is enabled by default.  Unfortunately, it
can't "see" inside subshells, that's why it couldn't detect this case.


>> 		test_file_count_in_dir ".git/lfs/objects" 1 &&
>>
>> 		cat >expect <<-\EOF &&
>> --
>> 2.8.1.99.g5d5236f
>>

Re: [PATCH] t9824: fix broken &&-chain in a subshell

[Lars Schneider]

On 24 Apr 2016, at 18:37, SZEDER Gábor <szeder@ira.uka.de> wrote:

> 
> Quoting Lars Schneider <larsxschneider@gmail.com>:
> 
>> On 24 Apr 2016, at 13:50, SZEDER Gábor <szeder@ira.uka.de> wrote:
>> 
>>> Signed-off-by: SZEDER Gábor <szeder@ira.uka.de>
>>> ---
>>> 
>>> As far as I can tell after eyeballing the test script, this was the
>>> only case where the &&-chain was broken.
>>> 
>>> t/t9824-git-p4-git-lfs.sh | 2 +-
>>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>> 
>>> diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
>>> index 0b664a377c45..64f8d18216d4 100755
>>> --- a/t/t9824-git-p4-git-lfs.sh
>>> +++ b/t/t9824-git-p4-git-lfs.sh
>>> @@ -265,7 +265,7 @@ test_expect_success 'Add big files to repo and
>>> store files in LFS based on compr
>>> 		# We only import HEAD here ("@all" is missing!)
>>> 		git p4 clone --destination="$git" //depot &&
>>> 
>>> -		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ"
>>> +		test_file_in_lfs file6.bin 13 "content 6 bin 39 bytes XXXXXYYYYYZZZZZ" &&
>> Thanks! I believe I remember some script/flag to detect broken &&-chains.
>> Do I imagine that or can anyone point me to it?
> 
> It was introduced in bb79af9d0970 (t/test-lib: introduce --chain-lint
> option, 2015-03-20) and is enabled by default.  Unfortunately, it
> can't "see" inside subshells, that's why it couldn't detect this case.

That explains it. Thank you!

- Lars