* [PATCH v3 0/3] git-p4: fix Git LFS pointer parsing
@ 2016-04-24 18:58 larsxschneider
2016-04-24 18:58 ` [PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version larsxschneider
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: larsxschneider @ 2016-04-24 18:58 UTC (permalink / raw)
To: git; +Cc: sschuberth, gitster, ben.woosley, Lars Schneider
From: Lars Schneider <larsxschneider@gmail.com>
v1: $gmane/291917
v2: $gmane/291991
diff to v2:
* use regex to parse "oid" instead of a complicated split logic
($gmane/291995, thanks Sebastian)
* add a note to the commit message emphasizing the change from array
to string in the second return parameter ($gmane/292077, thanks Junio)
* detect the Git-LFS preamble by the constant first part of the string
($gmane/292079, thanks Ben)
* change the name of the Git-LFS and Perforce version variables to
express that they are only used in the Linux build
Thanks,
Lars
Lars Schneider (3):
travis-ci: update Git-LFS and P4 to the latest version
travis-ci: express Linux/OS X dependency versions more clearly
git-p4: fix Git LFS pointer parsing
.travis.yml | 17 ++++++++++-------
git-p4.py | 13 ++++++++++---
t/t9824-git-p4-git-lfs.sh | 4 ++++
3 files changed, 24 insertions(+), 10 deletions(-)
--
2.5.1
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version
2016-04-24 18:58 [PATCH v3 0/3] git-p4: fix Git LFS pointer parsing larsxschneider
@ 2016-04-24 18:58 ` larsxschneider
2016-04-24 18:58 ` [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly larsxschneider
2016-04-24 18:58 ` [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing larsxschneider
2 siblings, 0 replies; 9+ messages in thread
From: larsxschneider @ 2016-04-24 18:58 UTC (permalink / raw)
To: git; +Cc: sschuberth, gitster, ben.woosley, Lars Schneider
From: Lars Schneider <larsxschneider@gmail.com>
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
.travis.yml | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 78e433b..4acf617 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,8 @@ addons:
env:
global:
- DEVELOPER=1
- - P4_VERSION="15.2"
- - GIT_LFS_VERSION="1.1.0"
+ - P4_VERSION="16.1"
+ - GIT_LFS_VERSION="1.2.0"
- DEFAULT_TEST_TARGET=prove
- GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
- GIT_TEST_OPTS="--verbose --tee"
--
2.5.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly
2016-04-24 18:58 [PATCH v3 0/3] git-p4: fix Git LFS pointer parsing larsxschneider
2016-04-24 18:58 ` [PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version larsxschneider
@ 2016-04-24 18:58 ` larsxschneider
2016-04-25 4:40 ` Eric Sunshine
2016-04-24 18:58 ` [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing larsxschneider
2 siblings, 1 reply; 9+ messages in thread
From: larsxschneider @ 2016-04-24 18:58 UTC (permalink / raw)
To: git; +Cc: sschuberth, gitster, ben.woosley, Lars Schneider
From: Lars Schneider <larsxschneider@gmail.com>
The Git Travis CI OSX build always installs the latest versions of Git LFS and
Perforce via brew and the Linux build installs fixed versions. Consequently new
LFS/Perforce versions can brake the OS X build even if there is no change in
Git.
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
---
.travis.yml | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/.travis.yml b/.travis.yml
index 4acf617..1fdcec8 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -22,8 +22,11 @@ addons:
env:
global:
- DEVELOPER=1
- - P4_VERSION="16.1"
- - GIT_LFS_VERSION="1.2.0"
+ # The Linux build installs the defined dependency versions below.
+ # The OS X build installs the latest available versions. Keep that
+ # in mind when you encounter a broken OS X build!
+ - LINUX_P4_VERSION="16.1"
+ - LINUX_GIT_LFS_VERSION="1.2.0"
- DEFAULT_TEST_TARGET=prove
- GIT_PROVE_OPTS="--timer --jobs 3 --state=failed,slow,save"
- GIT_TEST_OPTS="--verbose --tee"
@@ -38,17 +41,17 @@ before_install:
linux)
mkdir --parents custom/p4
pushd custom/p4
- wget --quiet http://filehost.perforce.com/perforce/r$P4_VERSION/bin.linux26x86_64/p4d
- wget --quiet http://filehost.perforce.com/perforce/r$P4_VERSION/bin.linux26x86_64/p4
+ wget --quiet http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION/bin.linux26x86_64/p4d
+ wget --quiet http://filehost.perforce.com/perforce/r$LINUX_P4_VERSION/bin.linux26x86_64/p4
chmod u+x p4d
chmod u+x p4
export PATH="$(pwd):$PATH"
popd
mkdir --parents custom/git-lfs
pushd custom/git-lfs
- wget --quiet https://github.com/github/git-lfs/releases/download/v$GIT_LFS_VERSION/git-lfs-linux-amd64-$GIT_LFS_VERSION.tar.gz
- tar --extract --gunzip --file "git-lfs-linux-amd64-$GIT_LFS_VERSION.tar.gz"
- cp git-lfs-$GIT_LFS_VERSION/git-lfs .
+ wget --quiet https://github.com/github/git-lfs/releases/download/v$LINUX_GIT_LFS_VERSION/git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz
+ tar --extract --gunzip --file "git-lfs-linux-amd64-$LINUX_GIT_LFS_VERSION.tar.gz"
+ cp git-lfs-$LINUX_GIT_LFS_VERSION/git-lfs .
export PATH="$(pwd):$PATH"
popd
;;
--
2.5.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing
2016-04-24 18:58 [PATCH v3 0/3] git-p4: fix Git LFS pointer parsing larsxschneider
2016-04-24 18:58 ` [PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version larsxschneider
2016-04-24 18:58 ` [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly larsxschneider
@ 2016-04-24 18:58 ` larsxschneider
2016-04-24 19:16 ` Sebastian Schuberth
2 siblings, 1 reply; 9+ messages in thread
From: larsxschneider @ 2016-04-24 18:58 UTC (permalink / raw)
To: git; +Cc: sschuberth, gitster, ben.woosley, Lars Schneider
From: Lars Schneider <larsxschneider@gmail.com>
Git LFS 1.2.0 removed a preamble from the output of the 'git lfs pointer'
command [1] which broke the parsing of this output. Adjust the parser
to support the old and the new format.
Please note that this patch slightly changes the second return parameter
from a list of LF terminated strings to a single string that contains
a number of LF characters.
[1] https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
Helped-by: Sebastian Schuberth <sschuberth@gmail.com>
Helped-by: Ben Woosley <ben.woosley@gmail.com>
---
git-p4.py | 13 ++++++++++---
t/t9824-git-p4-git-lfs.sh | 4 ++++
2 files changed, 14 insertions(+), 3 deletions(-)
diff --git a/git-p4.py b/git-p4.py
index 527d44b..f92a4bc 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
if pointerProcess.wait():
os.remove(contentFile)
die('git-lfs pointer command failed. Did you install the extension?')
- pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
- oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
+
+ # Git LFS removed the preamble in the output of the 'pointer' command
+ # starting from version 1.2.0. Check for the preamble here to support
+ # earlier versions.
+ # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
+ if pointerFile.startswith('Git LFS pointer for'):
+ re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
+
+ oid = re.search(r'^oid \w+:(\w+)', pointerFile, re.MULTILINE).group(1)
localLargeFile = os.path.join(
os.getcwd(),
'.git', 'lfs', 'objects', oid[:2], oid[2:4],
@@ -1073,7 +1080,7 @@ class GitLFS(LargeFileSystem):
)
# LFS Spec states that pointer files should not have the executable bit set.
gitMode = '100644'
- return (gitMode, pointerContents, localLargeFile)
+ return (gitMode, pointerFile, localLargeFile)
def pushFile(self, localLargeFile):
uploadProcess = subprocess.Popen(
diff --git a/t/t9824-git-p4-git-lfs.sh b/t/t9824-git-p4-git-lfs.sh
index 0b664a3..ca93ac8 100755
--- a/t/t9824-git-p4-git-lfs.sh
+++ b/t/t9824-git-p4-git-lfs.sh
@@ -13,6 +13,10 @@ test_file_in_lfs () {
FILE="$1" &&
SIZE="$2" &&
EXPECTED_CONTENT="$3" &&
+ sed -n '1,1 p' "$FILE" | grep "^version " &&
+ sed -n '2,2 p' "$FILE" | grep "^oid " &&
+ sed -n '3,3 p' "$FILE" | grep "^size " &&
+ test_line_count = 3 "$FILE" &&
cat "$FILE" | grep "size $SIZE" &&
HASH=$(cat "$FILE" | grep "oid sha256:" | sed -e "s/oid sha256://g") &&
LFS_FILE=".git/lfs/objects/$(echo "$HASH" | cut -c1-2)/$(echo "$HASH" | cut -c3-4)/$HASH" &&
--
2.5.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing
2016-04-24 18:58 ` [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing larsxschneider
@ 2016-04-24 19:16 ` Sebastian Schuberth
2016-04-25 7:33 ` Lars Schneider
0 siblings, 1 reply; 9+ messages in thread
From: Sebastian Schuberth @ 2016-04-24 19:16 UTC (permalink / raw)
To: larsxschneider; +Cc: Git Mailing List, Junio C Hamano, ben.woosley
On Sun, Apr 24, 2016 at 8:58 PM, <larsxschneider@gmail.com> wrote:
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
> if pointerProcess.wait():
> os.remove(contentFile)
> die('git-lfs pointer command failed. Did you install the extension?')
> - pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
> - oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
> +
> + # Git LFS removed the preamble in the output of the 'pointer' command
> + # starting from version 1.2.0. Check for the preamble here to support
> + # earlier versions.
> + # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
> + if pointerFile.startswith('Git LFS pointer for'):
> + re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
I liked the code from v2 better. I know Ben said "there could be
expansions or other modifications applied by git-lfs between input and
output", but I believe it's better to be too strict than too lenient
if you're omitting lines from the output. Also, the regex matches
against the whole multi-line string. That is, if the file for some
reason was ending in '\n\n' instead of just '\n', the '.*' would match
almost all content of the pointer file, not just the remains of the
preamble. One way to fix this would be to use
re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)
instead.
Regards,
Sebastian
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly
2016-04-24 18:58 ` [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly larsxschneider
@ 2016-04-25 4:40 ` Eric Sunshine
2016-04-25 7:05 ` Lars Schneider
0 siblings, 1 reply; 9+ messages in thread
From: Eric Sunshine @ 2016-04-25 4:40 UTC (permalink / raw)
To: larsxschneider@gmail.com
Cc: git@vger.kernel.org, sschuberth@gmail.com, gitster@pobox.com,
ben.woosley@gmail.com
On Sunday, April 24, 2016, <larsxschneider@gmail.com> wrote:
> The Git Travis CI OSX build always installs the latest versions of Git LFS and
> Perforce via brew and the Linux build installs fixed versions. Consequently new
> LFS/Perforce versions can brake the OS X build even if there is no change in
s/brake/break/
> Git.
>
> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly
2016-04-25 4:40 ` Eric Sunshine
@ 2016-04-25 7:05 ` Lars Schneider
0 siblings, 0 replies; 9+ messages in thread
From: Lars Schneider @ 2016-04-25 7:05 UTC (permalink / raw)
To: Eric Sunshine
Cc: git@vger.kernel.org, sschuberth@gmail.com, gitster@pobox.com,
ben.woosley@gmail.com
On 25 Apr 2016, at 06:40, Eric Sunshine <sunshine@sunshineco.com> wrote:
> On Sunday, April 24, 2016, <larsxschneider@gmail.com> wrote:
>> The Git Travis CI OSX build always installs the latest versions of Git LFS and
>> Perforce via brew and the Linux build installs fixed versions. Consequently new
>> LFS/Perforce versions can brake the OS X build even if there is no change in
>
> s/brake/break/
Thank you!
>
>> Git.
>>
>> Signed-off-by: Lars Schneider <larsxschneider@gmail.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing
2016-04-24 19:16 ` Sebastian Schuberth
@ 2016-04-25 7:33 ` Lars Schneider
2016-04-25 7:35 ` Sebastian Schuberth
0 siblings, 1 reply; 9+ messages in thread
From: Lars Schneider @ 2016-04-25 7:33 UTC (permalink / raw)
To: Sebastian Schuberth; +Cc: Git Mailing List, Junio C Hamano, ben.woosley
On 24 Apr 2016, at 21:16, Sebastian Schuberth <sschuberth@gmail.com> wrote:
> On Sun, Apr 24, 2016 at 8:58 PM, <larsxschneider@gmail.com> wrote:
>
>> --- a/git-p4.py
>> +++ b/git-p4.py
>> @@ -1064,8 +1064,15 @@ class GitLFS(LargeFileSystem):
>> if pointerProcess.wait():
>> os.remove(contentFile)
>> die('git-lfs pointer command failed. Did you install the extension?')
>> - pointerContents = [i+'\n' for i in pointerFile.split('\n')[2:][:-1]]
>> - oid = pointerContents[1].split(' ')[1].split(':')[1][:-1]
>> +
>> + # Git LFS removed the preamble in the output of the 'pointer' command
>> + # starting from version 1.2.0. Check for the preamble here to support
>> + # earlier versions.
>> + # c.f. https://github.com/github/git-lfs/commit/da2935d9a739592bc775c98d8ef4df9c72ea3b43
>> + if pointerFile.startswith('Git LFS pointer for'):
>> + re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
>
> I liked the code from v2 better. I know Ben said "there could be
> expansions or other modifications applied by git-lfs between input and
> output", but I believe it's better to be too strict than too lenient
> if you're omitting lines from the output. Also, the regex matches
> against the whole multi-line string. That is, if the file for some
> reason was ending in '\n\n' instead of just '\n', the '.*' would match
> almost all content of the pointer file, not just the remains of the
> preamble. One way to fix this would be to use
>
> re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)
>
> instead.
In general you are right as "*" is greedy. However, in Python "." matches any
character except a newline [1]. Therefore I think the regex is correct.
Nevertheless... thanks for making me read the line again. I forgot to
assign the pointerFile variable in the version I sent around :-(
This is how it should be:
pointerFile = re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
Thanks,
Lars
[1] https://docs.python.org/2/library/re.html
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing
2016-04-25 7:33 ` Lars Schneider
@ 2016-04-25 7:35 ` Sebastian Schuberth
0 siblings, 0 replies; 9+ messages in thread
From: Sebastian Schuberth @ 2016-04-25 7:35 UTC (permalink / raw)
To: Lars Schneider; +Cc: Git Mailing List, Junio C Hamano, ben.woosley
On Mon, Apr 25, 2016 at 9:33 AM, Lars Schneider
<larsxschneider@gmail.com> wrote:
>> if you're omitting lines from the output. Also, the regex matches
>> against the whole multi-line string. That is, if the file for some
>> reason was ending in '\n\n' instead of just '\n', the '.*' would match
>> almost all content of the pointer file, not just the remains of the
>> preamble. One way to fix this would be to use
>>
>> re.sub(r'Git LFS pointer for [^\n]+\n\n', '', pointerFile)
>>
>> instead.
>
> In general you are right as "*" is greedy. However, in Python "." matches any
> character except a newline [1]. Therefore I think the regex is correct.
Ah, thanks for pointing that out. Looks ok to me then.
> Nevertheless... thanks for making me read the line again. I forgot to
> assign the pointerFile variable in the version I sent around :-(
>
> This is how it should be:
>
> pointerFile = re.sub(r'Git LFS pointer for.*\n\n', '', pointerFile)
Right. Good you've catched that!
--
Sebastian Schuberth
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-04-25 7:35 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-04-24 18:58 [PATCH v3 0/3] git-p4: fix Git LFS pointer parsing larsxschneider
2016-04-24 18:58 ` [PATCH v3 1/3] travis-ci: update Git-LFS and P4 to the latest version larsxschneider
2016-04-24 18:58 ` [PATCH v3 2/3] travis-ci: express Linux/OS X dependency versions more clearly larsxschneider
2016-04-25 4:40 ` Eric Sunshine
2016-04-25 7:05 ` Lars Schneider
2016-04-24 18:58 ` [PATCH v3 3/3] git-p4: fix Git LFS pointer parsing larsxschneider
2016-04-24 19:16 ` Sebastian Schuberth
2016-04-25 7:33 ` Lars Schneider
2016-04-25 7:35 ` Sebastian Schuberth
Code repositories for project(s) associated with this public inbox
https://80x24.org/mirrors/git.git
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).