Skip to content

Error during CRC check when uploading an open + unlinked file #34987

Description

@rolftimmermans

Context

The google-cloud-storage library attempts to reopen any File that is attempted to be uploaded. See this line:
https://github.com/shubhangi-google/google-cloud-ruby/blob/042468dcccc83a802ae4993e76ebea203827d3cb/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb#L84

This is a problem when the file no longer exists.

When would that actually happen? The puma web server notably unlinks files immediately after storing, while keeping the file descriptor open:
https://github.com/puma/puma/blob/v8.0.2/lib/puma/client.rb#L507-L511

Although this trips people up, it is very much intentional and in fact recommended by the Ruby documentation.
puma/puma#3597
aws/aws-sdk-ruby#3163
aws/aws-sdk-ruby#3164
https://docs.ruby-lang.org/en/3.4/Tempfile.html#class-Tempfile-label-Unlink+after+creation

It would be tremendously helpful if the google-cloud-storage client would not assume that a File or Tempfile is still linked when it is being uploaded.

Environment details

  • OS: Alpine Linux, Debian Linux
  • Ruby version: 3.4
  • Gem name and version: google-cloud-storage (1.62.0)

Steps to reproduce

  1. Use Puma web server
  2. Upload files above configured memory limit
  3. Observe that google-cloud-storage fails to compute digest

Code example

This is a minimal example that mimics Puma's behaviour but it is a much simpler reproduction.

source "https://rubygems.org"

gem "google-cloud-storage", "1.62.0"
require "google/cloud/storage"
require "tempfile"

storage = Google::Cloud::Storage.anonymous
bucket = storage.bucket("unused", skip_lookup: true)

Tempfile.create("gcs-repro", binmode: true) do |file|
  file.write("still readable")
  file.rewind

  path = file.path
  File.unlink(path)

  puts "Path exists: #{File.exist?(path)}"
  puts "Contents: #{file.read.inspect}"
  file.rewind

  bucket.create_file(file, "test")
end

Full backtrace

3.4-slim: Pulling from library/ruby
b16cb8fde57a: Pulling fs layer
ead741ce7b27: Pulling fs layer
59f54fbcd984: Pulling fs layer
b160435c969d: Pulling fs layer
52a2f7711503: Pulling fs layer
b16cb8fde57a: Download complete
ead741ce7b27: Download complete
b160435c969d: Download complete
4371fa0b2e33: Download complete
79dd7f77afe3: Download complete
52a2f7711503: Download complete
59f54fbcd984: Download complete
59f54fbcd984: Pull complete
ead741ce7b27: Pull complete
b160435c969d: Pull complete
b16cb8fde57a: Pull complete
52a2f7711503: Pull complete
Digest: sha256:614edae6a80eb2a7cf1984f03a6d814f523a48355e8f96deb5ddd25faa86353e
Status: Downloaded newer image for ruby:3.4-slim
/usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'File#initialize': No such file or directory @ rb_sysopen - /tmp/gcs-repro20260721-29-v1priq (Errno::ENOENT)
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'IO.open'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'Google::Cloud::Storage::File::Verifier._digest_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:59:in 'Google::Cloud::Storage::File::Verifier.crc32c_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:3507:in 'Google::Cloud::Storage::Bucket#crc32c_for'
	from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:2007:in 'Google::Cloud::Storage::Bucket#create_file'
	from -e:1:in 'block in <main>'
	from /usr/local/lib/ruby/3.4.0/tempfile.rb:576:in 'Tempfile.create_with_filename'
	from /usr/local/lib/ruby/3.4.0/tempfile.rb:562:in 'Tempfile.create'
	from -e:1:in '<main>'
path exists: false
contents: still readable

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions