Context
The google-cloud-storage library attempts to reopen any File that is attempted to be uploaded. See this line:
https://github.com/shubhangi-google/google-cloud-ruby/blob/042468dcccc83a802ae4993e76ebea203827d3cb/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb#L84
This is a problem when the file no longer exists.
When would that actually happen? The puma web server notably unlinks files immediately after storing, while keeping the file descriptor open:
https://github.com/puma/puma/blob/v8.0.2/lib/puma/client.rb#L507-L511
Although this trips people up, it is very much intentional and in fact recommended by the Ruby documentation.
puma/puma#3597
aws/aws-sdk-ruby#3163
aws/aws-sdk-ruby#3164
https://docs.ruby-lang.org/en/3.4/Tempfile.html#class-Tempfile-label-Unlink+after+creation
It would be tremendously helpful if the google-cloud-storage client would not assume that a File or Tempfile is still linked when it is being uploaded.
Environment details
- OS: Alpine Linux, Debian Linux
- Ruby version: 3.4
- Gem name and version: google-cloud-storage (1.62.0)
Steps to reproduce
- Use Puma web server
- Upload files above configured memory limit
- Observe that google-cloud-storage fails to compute digest
Code example
This is a minimal example that mimics Puma's behaviour but it is a much simpler reproduction.
source "https://rubygems.org"
gem "google-cloud-storage", "1.62.0"
require "google/cloud/storage"
require "tempfile"
storage = Google::Cloud::Storage.anonymous
bucket = storage.bucket("unused", skip_lookup: true)
Tempfile.create("gcs-repro", binmode: true) do |file|
file.write("still readable")
file.rewind
path = file.path
File.unlink(path)
puts "Path exists: #{File.exist?(path)}"
puts "Contents: #{file.read.inspect}"
file.rewind
bucket.create_file(file, "test")
end
Full backtrace
3.4-slim: Pulling from library/ruby
b16cb8fde57a: Pulling fs layer
ead741ce7b27: Pulling fs layer
59f54fbcd984: Pulling fs layer
b160435c969d: Pulling fs layer
52a2f7711503: Pulling fs layer
b16cb8fde57a: Download complete
ead741ce7b27: Download complete
b160435c969d: Download complete
4371fa0b2e33: Download complete
79dd7f77afe3: Download complete
52a2f7711503: Download complete
59f54fbcd984: Download complete
59f54fbcd984: Pull complete
ead741ce7b27: Pull complete
b160435c969d: Pull complete
b16cb8fde57a: Pull complete
52a2f7711503: Pull complete
Digest: sha256:614edae6a80eb2a7cf1984f03a6d814f523a48355e8f96deb5ddd25faa86353e
Status: Downloaded newer image for ruby:3.4-slim
/usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'File#initialize': No such file or directory @ rb_sysopen - /tmp/gcs-repro20260721-29-v1priq (Errno::ENOENT)
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'IO.open'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:84:in 'Google::Cloud::Storage::File::Verifier._digest_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/file/verifier.rb:59:in 'Google::Cloud::Storage::File::Verifier.crc32c_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:3507:in 'Google::Cloud::Storage::Bucket#crc32c_for'
from /usr/local/bundle/gems/google-cloud-storage-1.62.0/lib/google/cloud/storage/bucket.rb:2007:in 'Google::Cloud::Storage::Bucket#create_file'
from -e:1:in 'block in <main>'
from /usr/local/lib/ruby/3.4.0/tempfile.rb:576:in 'Tempfile.create_with_filename'
from /usr/local/lib/ruby/3.4.0/tempfile.rb:562:in 'Tempfile.create'
from -e:1:in '<main>'
path exists: false
contents: still readable
Context
The
google-cloud-storagelibrary attempts to reopen anyFilethat is attempted to be uploaded. See this line:https://github.com/shubhangi-google/google-cloud-ruby/blob/042468dcccc83a802ae4993e76ebea203827d3cb/google-cloud-storage/lib/google/cloud/storage/file/verifier.rb#L84
This is a problem when the file no longer exists.
When would that actually happen? The
pumaweb server notably unlinks files immediately after storing, while keeping the file descriptor open:https://github.com/puma/puma/blob/v8.0.2/lib/puma/client.rb#L507-L511
Although this trips people up, it is very much intentional and in fact recommended by the Ruby documentation.
puma/puma#3597
aws/aws-sdk-ruby#3163
aws/aws-sdk-ruby#3164
https://docs.ruby-lang.org/en/3.4/Tempfile.html#class-Tempfile-label-Unlink+after+creation
It would be tremendously helpful if the
google-cloud-storageclient would not assume that aFileorTempfileis still linked when it is being uploaded.Environment details
Steps to reproduce
Code example
This is a minimal example that mimics Puma's behaviour but it is a much simpler reproduction.
Full backtrace