Repository navigation
Accept GLACIER/DEEP_ARCHIVE as cold locations, advertise the Veeam archive tier - #6329
francoisferrand wants to merge 2 commits into
Conversation
Hello francoisferrand,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| buildAttributesXml(v, v.userMetadata, listParams.optionalAttributes, xml); | ||
|
|
||
| return xml.push(`<StorageClass>${v.StorageClass}</StorageClass>`, '</Contents>'); | ||
| return xml.push(`<StorageClass>${formatStorageClass(v.StorageClass)}</StorageClass>`, '</Contents>'); |
There was a problem hiding this comment.
ListParts was missed: lib/api/listParts.js:306 still returns mpuOverviewObj.storageClass as stored. An MPU started with GLACIER will show <StorageClass>glacier</StorageClass> there, while HeadObject and ListObjects show GLACIER. Wrap it in formatStorageClass() too (in listParts.js, outside this diff).
There was a problem hiding this comment.
Good catch, fixed: ListParts now goes through formatStorageClass(), and so does ListMultipartUploads, which had the same issue (arsenal emits the stored x-amz-storage-class per <Upload>). Both are covered by unit tests in directToCold.js (MPU initiated with GLACIER, listing reports GLACIER).
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files
... and 2 files with indirect coverage changes @@ Coverage Diff @@
## development/9.5 #6329 +/- ##
===================================================
- Coverage 87.89% 87.88% -0.01%
===================================================
Files 206 206
Lines 14193 14209 +16
===================================================
+ Hits 12475 12488 +13
- Misses 1718 1721 +3
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Request integration branchesWaiting for integration branch creation to be requested by the user. To request integration branches, please comment on this pull request with the following command: Alternatively, the |
Standard S3 clients (AWS CLI, Veeam...) only accept the official storage class names, so they cannot write directly to a cold location: the location names are lower case, and cannot be created with an upper case name anyway. GLACIER and DEEP_ARCHIVE in x-amz-storage-class are now mapped to a location of the same (lower case) name, when the header is validated by the object write APIs. Policies are evaluated beforehand, so they apply to the storage class as sent by the client. Conversely, an object stored in a location named after one of these classes is reported with the AWS name in HeadObject, GetObject, ListObjects(Versions) and GetObjectAttributes. Issue: CLDSRV-988
Veeam only offers a bucket as an on-premises archive tier when its system.xml declares the EnableOnPremArchiveTier capability. The field is now part of the SOSAPI schema, and defaults to whether the archive tier can actually be used: Veeam writes to it with the GLACIER storage class, which needs direct-to-cold writes and a cold location named glacier. The UI does not need to know about it, while an explicit value still wins. Issue: CLDSRV-988
6957fbd to
6b5a944
Compare
| const location = normalizeStorageClass(storageClass); | ||
| if (config.enableDirectToCold && isColdStorageClass(location)) { | ||
| // eslint-disable-next-line no-param-reassign | ||
| headers['x-amz-storage-class'] = location; |
There was a problem hiding this comment.
Bucket policies are checked after this rewrite (objectPut/objectCopy/initiateMultipartUpload call validateStorageClass before standardMetadataValidateBucket(AndObj), and checkBucketPolicy builds its RequestContext from request.headers). So bucket policies see glacier while IAM saw GLACIER. A bucket policy Deny with StringEquals s3:x-amz-storage-class: GLACIER will never match, so the write goes through. That doesn't fit "policies apply to the storage class as sent by the client". Keep the original header and pass the resolved location separately, e.g. return it or store it on request, instead of mutating headers.
| CapacityInfo: joi.boolean().required(), | ||
| UploadSessions: joi.boolean().required(), | ||
| IAMSTS: joi.boolean().default(false), | ||
| EnableOnPremArchiveTier: joi.boolean().default(supportsOnPremArchiveTier), |
There was a problem hiding this comment.
This default is computed once, when system.xml is PUT, and then saved in bucketMd._capabilities (veeam/put.js). If someone enables enableDirectToCold or creates or removes the glacier location later, existing buckets keep the old value. They then advertise the archive tier wrongly until Veeam rewrites system.xml. Store the value only when the client sent it, and work out the default when serving GET.
This branch has not been deployed
2 participants
AWS CLI (and Veeam) only accept the official S3 storage class names, so they could not write "direct to cold": location names are lower case, and the UI does not allow creating upper case ones anyway.
Storage class mapping —
GLACIER/DEEP_ARCHIVEinx-amz-storage-classnow map to a location of the same lower case name (glacier/deep_archive). This is done invalidateStorageClass(), i.e. after auth, so policies apply to the storage class as sent by the client; the usualSTANDARDpath is untouched, and the header is only rewritten when the value resolves to an accepted cold location. Conversely, objects stored in one of those locations are reported with the AWS name in HeadObject/GetObject, ListObjects(Versions) and GetObjectAttributes. Metadata keeps the canonical location name, so nothing changes for backbeat.Veeam —
system.xmlnow acceptsEnableOnPremArchiveTierinProtocolCapabilities. When not provided (the zenko-ui template does not set it), it defaults to whether the archive tier is actually usable:enableDirectToColdand a cold location namedglacier.Note: an object in a
glacierlocation is always reported asGLACIER, whichever spelling was used on write.Issue: CLDSRV-988