git@vger.kernel.org mailing list mirror (one of many)
 help / color / mirror / code / Atom feed
From: Yang Zhao <yang.zhao@skyboxlabs.com>
To: git@vger.kernel.org
Cc: Yang Zhao <yang.zhao@skyboxlabs.com>,
	luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com
Subject: [PATCH v2 05/14] git-p4: encode/decode communication with git for python3
Date: Fri, 13 Dec 2019 15:52:39 -0800	[thread overview]
Message-ID: <20191213235247.23660-7-yang.zhao@skyboxlabs.com> (raw)
In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com>

Under python3, calls to write() on the stream to `git fast-import` must
be encoded.  This patch wraps the IO object such that this encoding is
done transparently.

Conversely, any text data read from subprocesses must also be decoded
before running through the rest of the pipeline.

Signed-off-by: Yang Zhao <yang.zhao@skyboxlabs.com>
Reviewed-by: Ben Keene <seraphire@gmail.com>
---
 git-p4.py | 21 +++++++++++++++++----
 1 file changed, 17 insertions(+), 4 deletions(-)

diff --git a/git-p4.py b/git-p4.py
index ca891e3d5d..d62fb05989 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -183,10 +183,12 @@ def read_pipe_full(c):
     (out, err) = p.communicate()
     return (p.returncode, out, decode_text_stream(err))
 
-def read_pipe(c, ignore_error=False):
+def read_pipe(c, ignore_error=False, raw=False):
     """ Read output from  command. Returns the output text on
         success. On failure, terminates execution, unless
         ignore_error is True, when it returns an empty string.
+
+        If raw is True, do not attempt to decode output text.
     """
     (retcode, out, err) = read_pipe_full(c)
     if retcode != 0:
@@ -194,6 +196,8 @@ def read_pipe(c, ignore_error=False):
             out = ""
         else:
             die('Command failed: %s\nError: %s' % (str(c), err))
+    if not raw:
+        out = decode_text_stream(out)
     return out
 
 def read_pipe_text(c):
@@ -220,7 +224,6 @@ def read_pipe_lines(c):
     val = [decode_text_stream(line) for line in pipe.readlines()]
     if pipe.close() or p.wait():
         die('Command failed: %s' % str(c))
-
     return val
 
 def p4_read_pipe_lines(c):
@@ -616,7 +619,8 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False,
             stdin_file.write(stdin)
         else:
             for i in stdin:
-                stdin_file.write(i + '\n')
+                stdin_file.write(encode_text_stream(i))
+                stdin_file.write(b'\n')
         stdin_file.flush()
         stdin_file.seek(0)
 
@@ -1245,7 +1249,7 @@ def generatePointer(self, contentFile):
             ['git', 'lfs', 'pointer', '--file=' + contentFile],
             stdout=subprocess.PIPE
         )
-        pointerFile = pointerProcess.stdout.read()
+        pointerFile = decode_text_stream(pointerProcess.stdout.read())
         if pointerProcess.wait():
             os.remove(contentFile)
             die('git-lfs pointer command failed. Did you install the extension?')
@@ -3538,6 +3542,15 @@ def openStreams(self):
         self.gitStream = self.importProcess.stdin
         self.gitError = self.importProcess.stderr
 
+        if bytes is not str:
+            # Wrap gitStream.write() so that it can be called using `str` arguments
+            def make_encoded_write(write):
+                def encoded_write(s):
+                    return write(s.encode() if isinstance(s, str) else s)
+                return encoded_write
+
+            self.gitStream.write = make_encoded_write(self.gitStream.write)
+
     def closeStreams(self):
         self.gitStream.close()
         if self.importProcess.wait() != 0:
-- 
2.21.0.windows.1


  parent reply	other threads:[~2019-12-13 23:53 UTC|newest]

Thread overview: 22+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20191213235247.23660-1-yang.zhao@skyboxlabs.com>
2019-12-13 23:52 ` [PATCH v2 00/14] git-p4: python3 compatibility Yang Zhao
2020-01-17 22:00   ` Yang Zhao
2020-01-24 20:14     ` Luke Diamand
2020-01-30 13:35       ` Luke Diamand
2020-02-03 12:54         ` Luke Diamand
2020-02-03 18:11           ` Yang Zhao
2020-02-04  1:35             ` Yang Zhao
2019-12-13 23:52 ` [PATCH v2 01/14] git-p4: make python2.7 the oldest supported version Yang Zhao
2019-12-13 23:52 ` [PATCH v2 02/14] git-p4: change the expansion test from basestring to list Yang Zhao
2019-12-13 23:52 ` [PATCH v2 03/14] git-p4: remove string type aliasing Yang Zhao
2019-12-13 23:52 ` [PATCH v2 04/14] git-p4: encode/decode communication with p4 for python3 Yang Zhao
2019-12-17 22:51   ` Junio C Hamano
2019-12-13 23:52 ` Yang Zhao [this message]
2019-12-13 23:52 ` [PATCH v2 06/14] git-p4: convert path to unicode before processing them Yang Zhao
2019-12-13 23:52 ` [PATCH v2 07/14] git-p4: open .gitp4-usercache.txt in text mode Yang Zhao
2019-12-13 23:52 ` [PATCH v2 08/14] git-p4: use marshal format version 2 when sending to p4 Yang Zhao
2019-12-13 23:52 ` [PATCH v2 09/14] git-p4: fix freezing while waiting for fast-import progress Yang Zhao
2019-12-13 23:52 ` [PATCH v2 10/14] git-p4: use functools.reduce instead of reduce Yang Zhao
2019-12-13 23:52 ` [PATCH v2 11/14] git-p4: use dict.items() iteration for python3 compatibility Yang Zhao
2019-12-13 23:52 ` [PATCH v2 12/14] git-p4: simplify regex pattern generation for parsing diff-tree Yang Zhao
2019-12-13 23:52 ` [PATCH v2 13/14] git-p4: use python3's input() everywhere Yang Zhao
2019-12-13 23:52 ` [RFC PATCH v2 14/14] ci: also run linux-gcc pipeline with python3.5 environment Yang Zhao

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: http://vger.kernel.org/majordomo-info.html

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191213235247.23660-7-yang.zhao@skyboxlabs.com \
    --to=yang.zhao@skyboxlabs.com \
    --cc=git@vger.kernel.org \
    --cc=liu.denton@gmail.com \
    --cc=luke@diamand.org \
    --cc=seraphire@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://80x24.org/mirrors/git.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).