Re: [PATCH] git-p4: close temporary file before removing

[Andrey <ahippo@yandex.ru>]

31.07.2019, 17:52, "Philip McGraw" <philip.mcgraw@bentley.com>:
> 2019.07.31 10:09 Andrey <ahippo@yandex.ru>
>> 31.07.2019, 09:53, "Philip McGraw" <philip.mcgraw@bentley.com>:
>>>>   30.07.2019, 13:37, "Philip McGraw" <philip.mcgraw@bentley.com>:
>>>>   > python os.remove() throws exceptions on Windows platform when attempting
>>>>   > to remove file while it is still open. Need to grab filename while file open,
>>>>   > close file handle, then remove by name. Apparently other platforms are more
>>>>   > permissive of removing files while busy.
>>>>   > reference:
>>>>   > ---
>>>>   >  git-p4.py | 4 +++-
>>>>   >  1 file changed, 3 insertions(+), 1 deletion(-)
>>>>   >
>>>>   > diff --git a/git-p4.py b/git-p4.py
>>>>   > index c71a6832e2..6b9d2a8317 100755
>>>>   > --- a/git-p4.py
>>>>   > +++ b/git-p4.py
>>>>   > @@ -1161,12 +1161,14 @@ def exceedsLargeFileThreshold(self, relPath, contents):
>>>>   >                  return False
>>>>   >              contentTempFile = self.generateTempFile(contents)
>>>>   >              compressedContentFile = tempfile.NamedTemporaryFile(prefix='git-p4-large-file', delete=False)
>>>>   > + compressedContentFileName = compressedContentFile.name
>>>>   >              zf = zipfile.ZipFile(compressedContentFile.name, mode='w')
>>>>   >              zf.write(contentTempFile, compress_type=zipfile.ZIP_DEFLATED)
>>>>   >              zf.close()
>>>>   >              compressedContentsSize = zf.infolist()[0].compress_size
>>>>   >              os.remove(contentTempFile)
>>>>   > - os.remove(compressedContentFile.name)
>>>>   > + compressedContentFile.close()
>>>>   > + os.remove(compressedContentFileName)
>>>>
>>>>   I'm not sure why NamedTemporaryFile() is called with delete=False above,
>>>>   but it appears to me that it can have delete=True instead,
>>>>   so that there is no need to call os.remove() explicitly
>>>>   and thus worry about remove vs close ordering at all.
>>>>
>>>>   >              if compressedContentsSize > gitConfigInt('git-p4.largeFileCompressedThreshold'):
>>>>   >                  return True
>>>>   >          return False
>>>>   > --
>>>>   > 2.21.0.windows.1
>>>>
>>>>   Thank you,
>>>>   Andrey.
>>>
>>>  Thanks Andrey; simpler is certainly better! I will test and re-submit v2 of patch with that approach.
>>
>> Thank you, that would be great!
>>
>> --
>> Andrey.
>
> Unfortunately it wasn't as simple it seemed: upon testing with only changing delete=True,
> found that the problem was not solved. Upon further debugging, recoded/refactored slightly adding
> allocateTempFileName() locally scoped function to try to clarify how the NamedTemporaryFile()
> was actually being used.
>
> We can't depend on the delete-on-close because the NamedTemporaryFile() is merely allocating
> a temporary name for real use by the zipfile open-for-write which fails (on Windows) if file
> was not explicitly closed first.

Oh, sorry for misguiding you!
I didn't think of this aspect.

> Hopefully the new patch (https://github.com/gitgitgadget/git/pull/301) will make this more clear.

The new changeset looks good to me.
(I'll post a reply in that thread too)

> Open to other suggestions if still not clear.

Just as a thought, ZipFile() can take a file-like object instead of a file name,
so can be passed the NamedTemporaryFile() object directly instead of its file name.
This should hopefully avoid double-open issue on Windows.

However, I'm good with your allocateTempFileName() changeset,
so it's up to you to give it a try or not.

> Thanks again,
> Philip

Thank you,
Andrey.