[PATCH v3 0/3] git-p4: fix Git LFS pointer parsing

[PATCH v3 0/3] git-p4: fix Git LFS pointer parsing

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

v1: $gmane/291917
v2: $gmane/291991

diff to v2:
 * use regex to parse "oid" instead of a complicated split logic
   ($gmane/291995, thanks Sebastian)
 * add a note to the commit message emphasizing the change from array
   to string in the second return parameter ($gmane/292077, thanks Junio)
 * detect the Git-LFS preamble by the constant first part of the string
   ($gmane/292079, thanks Ben)
 * change the name of the Git-LFS and Perforce version variables to
   express that they are only used in the Linux build

Thanks,
Lars

Lars Schneider (3):
  travis-ci: update Git-LFS and P4 to the latest version
  travis-ci: express Linux/OS X dependency versions more clearly
  git-p4: fix Git LFS pointer parsing

 .travis.yml               | 17 ++++++++++-------
 git-p4.py                 | 13 ++++++++++---
 t/t9824-git-p4-git-lfs.sh |  4 ++++
 3 files changed, 24 insertions(+), 10 deletions(-)

--
2.5.1

[PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 78e433b..4acf617 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,8 @@ addons:
 env:
   global:
     - DEVELOPER=1
-    - P4_VERSION="15.2"
-    - GIT_LFS_VERSION="1.1.0"
+    - P4_VERSION="16.1"
+    - GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
     - GIT_TEST_OPTS="--verbose --tee"
-- 
2.5.1

[PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

The Git Travis CI OSX build always installs the latest versions of Git LFS and
Perforce via brew and the Linux build installs fixed versions. Consequently new
LFS/Perforce versions can brake the OS X build even if there is no change in
Git.

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
 .travis.yml | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/.travis.yml b/.travis.yml
index 4acf617..1fdcec8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,11 @@ addons:
 env:
   global:
     - DEVELOPER=1
-    - P4_VERSION="16.1"
-    - GIT_LFS_VERSION="1.2.0"
+    # The Linux build installs the defined dependency versions below.
+    # The OS X build installs the latest available versions. Keep that
+    # in mind when you encounter a broken OS X build!
+    - LINUX_P4_VERSION="16.1"
+    - LINUX_GIT_LFS_VERSION="1.2.0"
     - DEFAULT_TEST_TARGET=prove
     - GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
     - GIT_TEST_OPTS="--verbose --tee"
@@ -38,17 +41,17 @@ before_install:
     linux)
       mkdir --parents custom/p4
       pushd custom/p4
-        wget --quiet http://filehost.perforce.com/perforce/r$P4_VERSION/bin.linux26x86_64/p4d
-        wget --quiet http://filehost.perforce.com/perforce/r$P4_VERSION/bin.linux26x86_64/p4
+        wget --quiet http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION/bin.linux26x86_64/p4d
+        wget --quiet http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION/bin.linux26x86_64/p4
         chmod u+x p4d
         chmod u+x p4
         export PATH="$(pwd):$PATH"
       popd
       mkdir --parents custom/git-lfs
       pushd custom/git-lfs
-        wget --quiet https://github.com/github/git-lfs/releases/download/v$GIT_LFS_VERSION/git-lfs-linux-amd64-$GIT_LFS_VERSION.tar.gz
-        tar --extract --gunzip --file "git-lfs-linux-amd64-$GIT_LFS_VERSION.tar.gz"
-        cp git-lfs-$GIT_LFS_VERSION/git-lfs .
+        wget --quiet https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz
+        tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
+        cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
         export PATH="$(pwd):$PATH"
       popd
       ;;
-- 
2.5.1

Re: [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly

[Eric Sunshine]

On Sunday, April 24, 2016, <larsxschneider@gmail.com> wrote:
> The Git Travis CI OSX build always installs the latest versions of Git LFS and
> Perforce via brew and the Linux build installs fixed versions. Consequently new
> LFS/Perforce versions can brake the OS X build even if there is no change in

s/brake/break/

> Git.
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>

Re: [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly

[Lars Schneider]

On 25 Apr 2016, at 06:40, Eric Sunshine <sunshine@sunshineco.com> wrote:

> On Sunday, April 24, 2016, <larsxschneider@gmail.com> wrote:
>> The Git Travis CI OSX build always installs the latest versions of Git LFS and
>> Perforce via brew and the Linux build installs fixed versions. Consequently new
>> LFS/Perforce versions can brake the OS X build even if there is no change in
> 
> s/brake/break/

Thank you!

> 
>> Git.
>> 
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>

[PATCH v3 3/3] git-p4: fix Git LFS pointer parsing

[larsxschneider]

From: Lars Schneider <larsxschneider@gmail.com>

Git LFS 1.2.0 removed a preamble from the output of the 'git lfs pointer'
command [1] which broke the parsing of this output. Adjust the parser
to support the old and the new format.

Please note that this patch slightly changes the second return parameter
from a list of LF terminated strings to a single string that contains
a number of LF characters.

[1] https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43

Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Helped-by: Sebastian Schuberth <sschuberth@gmail.com>
Helped-by: Ben Woosley <ben.woosley@gmail.com>
---
 git-p4.py                 | 13 ++++++++++---
 t/t9824-git-p4-git-lfs.sh |  4 ++++
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index 527d44b..f92a4bc 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
         if pointerProcess.wait():
             os.remove(contentFile)
             die('git-lfs pointer command failed. Did you install the extension?')
-        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
-        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
+
+        # Git LFS removed the preamble in the output of the 'pointer' command
+        # starting from version 1.2.0. Check for the preamble here to support
+        # earlier versions.
+        # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
+        if pointerFile.startswith('Git LFS pointer for'):
+            re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
+
+        oid = re.search(r'^oid \w+:(\w+)', pointerFile, re.MULTILINE).group(1)
         localLargeFile = os.path.join(
             os.getcwd(),
             '.git', 'lfs', 'objects', oid[:2], oid[2:4],
@@ -1073,7 +1080,7 @@ class GitLFS(LargeFileSystem):
         )
         # LFS Spec states that pointer files should not have the executable bit set.
         gitMode = '100644'
-        return (gitMode, pointerContents, localLargeFile)
+        return (gitMode, pointerFile, localLargeFile)
 
     def pushFile(self, localLargeFile):
         uploadProcess = subprocess.Popen(
diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
index 0b664a3..ca93ac8 100755
--- a/t/t9824-git-p4-git-lfs.sh
+++ b/t/t9824-git-p4-git-lfs.sh
@@ -13,6 +13,10 @@ test_file_in_lfs () {
 	FILE="$1" &&
 	SIZE="$2" &&
 	EXPECTED_CONTENT="$3" &&
+	sed -n '1,1 p' "$FILE" | grep "^version " &&
+	sed -n '2,2 p' "$FILE" | grep "^oid " &&
+	sed -n '3,3 p' "$FILE" | grep "^size " &&
+	test_line_count = 3 "$FILE" &&
 	cat "$FILE" | grep "size $SIZE" &&
 	HASH=$(cat "$FILE" | grep "oid sha256:" | sed -e "s/oid sha256://g") &&
 	LFS_FILE=".git/lfs/objects/$(echo "$HASH" | cut -c1-2)/$(echo "$HASH" | cut -c3-4)/$HASH" &&
-- 
2.5.1

Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing

[Sebastian Schuberth]

On Sun, Apr 24, 2016 at 8:58 PM,  <larsxschneider@gmail.com> wrote:

> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
>          if pointerProcess.wait():
>              os.remove(contentFile)
>              die('git-lfs pointer command failed. Did you install the extension?')
> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
> +
> +        # Git LFS removed the preamble in the output of the 'pointer' command
> +        # starting from version 1.2.0. Check for the preamble here to support
> +        # earlier versions.
> +        # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
> +        if pointerFile.startswith('Git LFS pointer for'):
> +            re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)

I liked the code from v2 better. I know Ben said "there could be
expansions or other modifications applied by git-lfs between input and
output", but I believe it's better to be too strict than too lenient
if you're omitting lines from the output. Also, the regex matches
against the whole multi-line string. That is, if the file for some
reason was ending in '\n\n' instead of just '\n', the '.*' would match
almost all content of the pointer file, not just the remains of the
preamble. One way to fix this would be to use

re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)

instead.

Regards,
Sebastian

Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing

[Lars Schneider]

On 24 Apr 2016, at 21:16, Sebastian Schuberth <sschuberth@gmail.com> wrote:

> On Sun, Apr 24, 2016 at 8:58 PM,  <larsxschneider@gmail.com> wrote:
> 
>> --- a/git-p4.py
>> +++ b/git-p4.py
>> @@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
>>         if pointerProcess.wait():
>>             os.remove(contentFile)
>>             die('git-lfs pointer command failed. Did you install the extension?')
>> -        pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
>> -        oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
>> +
>> +        # Git LFS removed the preamble in the output of the 'pointer' command
>> +        # starting from version 1.2.0. Check for the preamble here to support
>> +        # earlier versions.
>> +        # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
>> +        if pointerFile.startswith('Git LFS pointer for'):
>> +            re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
> 
> I liked the code from v2 better. I know Ben said "there could be
> expansions or other modifications applied by git-lfs between input and
> output", but I believe it's better to be too strict than too lenient
> if you're omitting lines from the output. Also, the regex matches
> against the whole multi-line string. That is, if the file for some
> reason was ending in '\n\n' instead of just '\n', the '.*' would match
> almost all content of the pointer file, not just the remains of the
> preamble. One way to fix this would be to use
> 
> re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)
> 
> instead.

In general you are right as "*" is greedy. However, in Python "." matches any 
character except a newline [1]. Therefore I think the regex is correct.

Nevertheless... thanks for making me read the line again. I forgot to
assign the pointerFile variable in the version I sent around :-(

This is how it should be:

pointerFile = re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)

Thanks,
Lars


[1] https://docs.python.org/2/library/re.html

Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing

[Sebastian Schuberth]

On Mon, Apr 25, 2016 at 9:33 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:

>> if you're omitting lines from the output. Also, the regex matches
>> against the whole multi-line string. That is, if the file for some
>> reason was ending in '\n\n' instead of just '\n', the '.*' would match
>> almost all content of the pointer file, not just the remains of the
>> preamble. One way to fix this would be to use
>>
>> re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)
>>
>> instead.
>
> In general you are right as "*" is greedy. However, in Python "." matches any
> character except a newline [1]. Therefore I think the regex is correct.

Ah, thanks for pointing that out. Looks ok to me then.

> Nevertheless... thanks for making me read the line again. I forgot to
> assign the pointerFile variable in the version I sent around :-(
>
> This is how it should be:
>
> pointerFile = re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)

Right. Good you've catched that!

-- 
Sebastian Schuberth