Segmentation fault when calling `inspect` on `finish` of `ZStream`

[ndossche]

The following test case causes a segmentation fault (when added to test/zlib/test_zlib.rb, under test_inflate_buffer_partial_input):

    def test_inflate_finish_inspect
      deflated = Zlib::Deflate.deflate "\0"

      z = Zlib::Inflate.new

      buf = String.new
      deflated.each_char do |byte|
        z.inflate(byte, buffer: buf)
      end

      z.finish.inspect # segfault!
    end

Commit: 5c3f2ec
Zlib version: 1.3
Ruby version: 3.2.0

Output:

Loaded suite /var/lib/gems/3.2.0/gems/rake-13.4.2/lib/rake/rake_test_loader
Started
O
==============================================================================================================================================================
Omission: O_TMPFILE not supported (EOPNOTSUPP) [test_path_tmpfile(TestZlibGzipFile)]
/work/ruby-zlib/test/zlib/test_zlib.rb:844:in `rescue in test_path_tmpfile'
==============================================================================================================================================================
//work/ruby-zlib/test/zlib/test_zlib.rb:411: [BUG] Segmentation fault at 0x0000000000000000
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux-gnu]

-- Control frame information -----------------------------------------------
c:0039 p:0037 s:0223 e:000221 METHOD /work/ruby-zlib/test/zlib/test_zlib.rb:411
c:0038 p:0138 s:0215 e:000214 METHOD /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/testcase.rb:969
c:0037 p:0004 s:0207 e:000206 BLOCK  /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/testcase.rb:634
c:0036 p:0025 s:0204 e:000203 BLOCK  /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/fixture.rb:276
c:0035 p:0025 s:0200 e:000199 BLOCK  /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/fixture.rb:276
c:0034 p:0052 s:0196 e:000195 METHOD /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/fixture.rb:257
c:0033 p:0009 s:0187 e:000186 METHOD /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/fixture.rb:292
c:0032 p:0005 s:0182 E:001998 BLOCK  /var/lib/gems/3.2.0/gems/test-unit-3.7.7/lib/test/unit/testcase.rb:632 [FINISH]
.........

-- Machine register context ------------------------------------------------
 RIP: 0x000070000a79411f RBP: 0x00007ffe601134f0 RSP: 0x00007ffe60113480
 RAX: 0x000070000a8e7ea0 RBX: 0x0000600b13216160 RCX: 0x0000000000000000
 RDX: 0x0000000000000000 RDI: 0x0000000000000000 RSI: 0x00000ad100100001
  R8: 0x0000000000000000  R9: 0x0000000000000001 R10: 0x0000000000008000
 R11: 0x0000000000000000 R12: 0x000070000a168650 R13: 0x0000600b13927bb0
 R14: 0x00007000057fa808 R15: 0x000070000a168650 EFL: 0x0000000000010246

-- C level backtrace information -------------------------------------------
/lib/x86_64-linux-gnu/libruby-3.2.so.3.2(0x70000a7adcee) [0x70000a7adcee]
/lib/x86_64-linux-gnu/libruby-3.2.so.3.2(0x70000a5f2bdc) [0x70000a5f2bdc]
/lib/x86_64-linux-gnu/libruby-3.2.so.3.2(0x70000a71b116) [0x70000a71b116]
/lib/x86_64-linux-gnu/libc.so.6(0x70000a1b2330) [0x70000a1b2330]

.........

[kpumuk]

Started reporting to Ruby bug tracker when found this same issue is already reported here :-)

Zlib is crashing when used with an external buffer:

require 'zlib'

string = 'A' * 1000
buffer = ''

compressed = Zlib::Deflate.deflate(string)
inflator = Zlib::Inflate.new
inflator.inflate(compressed, buffer:)
inflator.finish.bytesize

This seem to be related to ZSTREAM_REUSE_BUFFER, which is set by inflate, but not reset by the finish. This leads to zstream_detach_buffer skipping rb_obj_reveal and Ruby receives a String-like object instead of a String, which crashes Ruby on any method call to that object.

The crash does not happen when inflate is called with nil and the same buffer:

inflator.inflate(compressed, buffer:)
inflator.inflate(nil, buffer:).bytesize

Reproduced on 3.4 and current master.