[ruby-core:62257] [ruby-trunk - Bug #9790] [Open] Zlib::GzipReader only decompressed the first of concatenated files

[ruby-core:62257] [ruby-trunk - Bug #9790] [Open] Zlib::GzipReader only decompressed the first of concatenated files

[quainjn]

Issue #9790 has been reported by Jake Quain.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790

* Author: Jake Quain
* Status: Open
* Priority: Normal
* Assignee: 
* Category: 
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```




-- 
https://bugs.ruby-lang.org/

[ruby-core:62271] [ruby-trunk - Bug #9790] [Assigned] Zlib::GzipReader only decompressed the first of concatenated files

[drbrain]

Issue #9790 has been updated by Eric Hodel.

Category set to ext
Status changed from Open to Assigned
Assignee set to Eric Hodel
Target version set to current: 2.2.0

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-46434

* Author: Jake Quain
* Status: Assigned
* Priority: Normal
* Assignee: Eric Hodel
* Category: ext
* Target version: current: 2.2.0
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```




-- 
https://bugs.ruby-lang.org/

[ruby-core:68286] [Ruby trunk - Bug #9790] Zlib::GzipReader only decompressed the first of concatenated files

[akostadinov]

Issue #9790 has been updated by Aleksandar Kostadinov.


Because gzip format allows multiple entries with filename I'd suggest to support a method like Java's ZipInputStream `getNextEntry()` [1]. This way programmer can choose to read everything as one chunk of data or multiple chunks each with its own name. This would allow storing and then retrieving multiple files in/from one gz. 

On the other hand the command line gzip utility only supports reading the whole thing as one. So a convenience method to read everything in one go, would also be nice.

[1] http://docs.oracle.com/javase/7/docs/api/java/util/zip/ZipInputStream.html

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-51642

* Author: Jake Quain
* Status: Assigned
* Priority: Normal
* Assignee: Eric Hodel
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```




-- 
https://bugs.ruby-lang.org/

[ruby-core:68303] [Ruby trunk - Bug #9790] Zlib::GzipReader only decompressed the first of concatenated files

[duerst]

Issue #9790 has been updated by Martin Dürst.


Aleksandar Kostadinov wrote:
> Because gzip format allows multiple entries with filename I'd suggest to support a method like Java's ZipInputStream `getNextEntry()` [1]. This way programmer can choose to read everything as one chunk of data or multiple chunks each with its own name. This would allow storing and then retrieving multiple files in/from one gz. 

Good idea, but it should be more Ruby-like, such as .each_file or so.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-51657

* Author: Jake Quain
* Status: Assigned
* Priority: Normal
* Assignee: Eric Hodel
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```




-- 
https://bugs.ruby-lang.org/

[ruby-core:69362] [Ruby trunk - Bug #9790] Zlib::GzipReader only decompressed the first of concatenated files

[exaspark]

Issue #9790 has been updated by Evgeny Li.


Hey guys, is there any updates?

I have created a small gem yesterday to make it able to read multiple files https://github.com/exAspArk/multiple_files_gzip_reader

~~~ruby
> MultipleFilesGzipReader.open("3.txt.gz") do |gz|
>   puts gz.read
> end

# 1
# 2
# => nil
~~~

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-52636

* Author: Jake Quain
* Status: Assigned
* Priority: Normal
* Assignee: Eric Hodel
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```




-- 
https://bugs.ruby-lang.org/

[ruby-core:95987] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[merch-redmine]

Issue #9790 has been updated by jeremyevans0 (Jeremy Evans).

File zlib-gzreader-each_file-9790.patch added

Attached is a patch that adds `Zlib::GzipReader.each_file` will which handle multiple concatenated gzip streams in the same file, similar to common tools that operate on `.gz` files.  `Zlib::GzipReader.each_file` yields one `Zlib::GzipReader` instance per gzip stream in the file.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-82820

* Author: quainjn (Jake Quain)
* Status: Assigned
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96832] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[ko1]

Issue #9790 has been updated by ko1 (Koichi Sasada).

Status changed from Assigned to Feedback

do you have real (popular) usecases?

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83833

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96833] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[merch-redmine]

Issue #9790 has been updated by jeremyevans0 (Jeremy Evans).


ko1 (Koichi Sasada) wrote:
> do you have real (popular) usecases?

For real but not necessarily popular, but at least OpenBSD's package format uses this.  I'm not sure if package formats for other operating systems use it, though.  #14804 pointed out it was used by this file: http://commondatastorage.googleapis.com/freebase-public/rdf/freebase-rdf-latest.gz

There are 3 bug reports for this, and I think while the need isn't common, it's something worth supporting via a new method.  However, if we decide we don't want to support this, I'm fine closing the 3 bug reports.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83834

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96898] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[ko1]

Issue #9790 has been updated by ko1 (Koichi Sasada).


* mame: can each_file return an Enumerator?  Seems difficult to implement it
* matz: How about always behaving like zcat?  Is an option to keep the old behaviors really needed?
* akr: The traditional behavior should be kept
* akr: gzip(1) describes concatenation of gzip files in ADVANCED USAGE. https://www.gnu.org/software/gzip/manual/gzip.html#Advanced-usage

Conclusion:

* matz: it should behave like zcat.  Handling each member should be deleted.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83912

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96910] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[daniel]

Issue #9790 has been updated by Dan0042 (Daniel DeLorme).


> matz: it should behave like zcat.  Handling each member should be deleted.

Really? I agree that `Zlib::GzipReader.open` should behave like zcat, but `each_file` is a useful additional feature to have.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83928

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96922] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[shyouhei]

Issue #9790 has been updated by shyouhei (Shyouhei Urabe).


Dan0042 (Daniel DeLorme) wrote:
> > matz: it should behave like zcat.  Handling each member should be deleted.
> 
> Really? I agree that `Zlib::GzipReader.open` should behave like zcat, but `each_file` is a useful additional feature to have. And it's already implemented.

I was at the meeting @ko1 is talking about.  Devs at the meeting were not sure if "`each_file` is a useful additional feature to have" you say is real or not.

Do you know if there are cases when taking a random member of such gzip file is actually useful?

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83941

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96927] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[daniel]

Issue #9790 has been updated by Dan0042 (Daniel DeLorme).


I can't think of taking a random member but I can imagine wanting to extract to separate files.

```ruby
Zlib::GzipReader.each_file(filename).with_index do |gzip,i|
  File.open("output#{i}","w"){ |f| f.write(gzip.read) }
end
```

But I admit I haven't needed this personally, so maybe this is not so useful.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83950

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96929] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[merch-redmine]

Issue #9790 has been updated by jeremyevans0 (Jeremy Evans).


shyouhei (Shyouhei Urabe) wrote:
> Do you know if there are cases when taking a random member of such gzip file is actually useful?

It's not taking a random member, it's processing the separate gzip streams in order, which is actually useful.  For example, in OpenBSD's packages, the first gzip stream in the package contains the package metadata.  That is read completely, and after processing the metadata, the programs that handle packages determine whether it needs to read the actual package data (stored in the second gzip stream).  If it doesn't need to process the package data, it can then stop at that point without having read any more than it needs to.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83956

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96933] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[nobu]

Issue #9790 has been updated by nobu (Nobuyoshi Nakada).


jeremyevans0 (Jeremy Evans) wrote:
> For example, in OpenBSD's packages, the first gzip stream in the package contains the package metadata.  That is read completely, and after processing the metadata, the programs that handle packages determine whether it needs to read the actual package data (stored in the second gzip stream).  If it doesn't need to process the package data, it can then stop at that point without having read any more than it needs to.

It is an interesting usage.
Does that metadata need to be a separate member, not the leading part of a large stream?

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83960

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/

[ruby-core:96934] [Ruby master Bug#9790] Zlib::GzipReader only decompressed the first of concatenated files

[merch-redmine]

Issue #9790 has been updated by jeremyevans0 (Jeremy Evans).


nobu (Nobuyoshi Nakada) wrote:
> It is an interesting usage.
> Does that metadata need to be a separate member, not the leading part of a large stream?

It's easier and more efficient if the metadata is a separate member.  I'm guessing it could be changed to use a single large stream, but there is no reason to do so, and doing so would break existing tooling.

----------------------------------------
Bug #9790: Zlib::GzipReader only decompressed the first of concatenated files
https://bugs.ruby-lang.org/issues/9790#change-83961

* Author: quainjn (Jake Quain)
* Status: Feedback
* Priority: Normal
* Assignee: drbrain (Eric Hodel)
* Target version: 
* ruby -v: 2.1.1
* Backport: 2.0.0: UNKNOWN, 2.1: UNKNOWN
----------------------------------------
There is a similar old issue in Node that I came across that perfectly describes the situation in ruby:

https://github.com/joyent/node/issues/6032

In ruby given the following setup:

```
echo "1" > 1.txt
echo "2" > 2.txt
gzip 1.txt
gzip 2.txt
cat 1.txt.gz 2.txt.gz > 3.txt.gz
```

Calling:
    
```
Zlib::GzipReader.open("3.txt.gz") do |gz|
  print gz.read
end
```

would just print:

```
1
```


---Files--------------------------------
zlib-gzreader-each_file-9790.patch (3.47 KB)


-- 
https://bugs.ruby-lang.org/