From dd9c2f6aa80d18ccdfe8373abe28ec1d87e4f013 Mon Sep 17 00:00:00 2001 From: Shizuo Fujita Date: Wed, 8 Jul 2026 11:33:33 +0900 Subject: [PATCH] test_buffer: make gzip chunk overflow test independent of zlib implementation (#5415) **Which issue(s) this PR fixes**: Fixes # **What this PR does / why we need it**: The gzipped size of a record varies by a few bytes depending on the zlib implementation. zlib-ng, which is used by default on some distributions, produces slightly larger output for tiny records than stock zlib. The test hard-coded chunk_limit_size to 70 and implicitly assumed the small "aaa"/"bbb" records stay under that limit, which holds for stock zlib but not for zlib-ng where they compress to 72 bytes. As a result, those records also triggered BufferChunkOverflowError locally and the test failed, while it kept passing on CI (stock zlib). Derive chunk_limit_size from the actual gzipped size of the small records so they always fit into (and fill up) a chunk, and enlarge the overflowing record so it exceeds the limit by a wide margin regardless of the zlib implementation. **Docs Changes**: N/A **Release Note**: N/A Signed-off-by: Shizuo Fujita Signed-off-by: github-actions[bot] --- test/plugin/test_buffer.rb | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/test/plugin/test_buffer.rb b/test/plugin/test_buffer.rb index 4e3f5714ca..6c62c3a542 100644 --- a/test/plugin/test_buffer.rb +++ b/test/plugin/test_buffer.rb @@ -1451,20 +1451,31 @@ def create_chunk_es(metadata, es) end test '#write compressed data which exceeds chunk_limit_size, it raises BufferChunkOverflowError' do - @p = create_buffer({'compress' => 'gzip', 'chunk_limit_size' => 70}) + # The gzipped size of a record varies by a few bytes depending on the zlib implementation (e.g. zlib vs zlib-ng), + # so a hard-coded chunk_limit_size makes this test environment-dependent. + # To keep it stable, derive chunk_limit_size from the actual gzipped size of the small records + # so that they always fit into a chunk, while the overflowing record is chosen to always exceed the limit by a wide + # margin regardless of the zlib implementation. timestamp = event_time('2016-04-11 16:00:02 +0000') - es = Fluent::ArrayEventStream.new([[timestamp, {"message" => "012345"}], # overflow + small_chunk = @p.generate_chunk(create_metadata) + small_chunk.append(Fluent::ArrayEventStream.new([[timestamp, {"message" => "aaa"}]]), compress: :gzip) + chunk_limit_size = small_chunk.bytesize + + @p = create_buffer({'compress' => 'gzip', 'chunk_limit_size' => chunk_limit_size}) + overflow_record = "The quick brown fox jumps over the lazy dog 123456" + es = Fluent::ArrayEventStream.new([[timestamp, {"message" => overflow_record}], [timestamp, {"message" => "aaa"}], [timestamp, {"message" => "bbb"}]]) assert_equal [], @p.queue.map(&:metadata) - assert_equal 70, @p.chunk_limit_size + assert_equal chunk_limit_size, @p.chunk_limit_size # calculate the actual boundary value. it varies on machine c = @p.generate_chunk(create_metadata) - c.append(Fluent::ArrayEventStream.new([[timestamp, {"message" => "012345"}]]), compress: :gzip) + c.append(Fluent::ArrayEventStream.new([[timestamp, {"message" => overflow_record}]]), compress: :gzip) overflow_bytes = c.bytesize + assert_operator overflow_bytes, :>, chunk_limit_size - messages = "concatenated/appended a #{overflow_bytes} bytes record (nth: 0) is larger than buffer chunk limit size (70)" + messages = "concatenated/appended a #{overflow_bytes} bytes record (nth: 0) is larger than buffer chunk limit size (#{chunk_limit_size})" assert_raise Fluent::Plugin::Buffer::BufferChunkOverflowError.new(messages) do # test format == nil && compress == :gzip @p.write({@dm0 => es})