You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Bug report — integrity. Split out of tracking issue #2031 (CAS-021, prev CAS-035). Static analysis of PR #2159 / cas-code-only-strip; not yet reproduced end-to-end.
Describe the situation
When a conditional object-store write (create-if-absent / If-Match overwrite) does not return a definite ACK, the CAS request controller guesses a definite outcome from a later GET/HEAD.
Two complementary lies:
Content equality is treated as proof that this attempt committed. After 412, 500, timeout, or a lost 200, putOverwriteControlled / putIfAbsentControlledMutable GET the key and, if got->bytes == bytes_s, return Committed with got->token — the current object's etag, which may belong to another writer.
A landed-then-timed-out write is reported as another writer's object. After the same ambiguity, conditionalCreateControlled / slotOccupy see that the key exists and return Occupied with unresolved_reason = NotUnresolved.
Content equality answers “is the durable value the value I intended?” It does not answer “did this client's conditional PUT succeed?” That second question is authorship. It is only recoverable from this PUT's 200+etag, or from a writer/attempt id embedded in the object. Blob freshness meta is generic {state, condemn_round, size} — no owner.
This issue:
Erases compare-and-set exclusivity on mutable keys (casMeta / putMetaIfAbsent)
Hands the loser the winner's etag, so a later If-Match can splice the CAS chain
Lets GC treat a false Committed as “I confirmed the condemn marker,” which licenses deleteExact of the blob body
Lets a lost ACK of our own ref-log append be classified as foreign occupancy and fault the mount as impossible interference
The same codebase already does the correct thing for the server-root owner anchor: claimOwnerOrThrow rereads the object and accepts only if server_uuid == our_uuid.
Build type: n/a (code-path finding; trigger is a lost ACK / 5xx / 412 on a conditional PUT)
Steps (logical)
The controller never distinguishes “PUT applied, ACK lost” from “PUT refused” from “PUT never applied.” Any of: dropped 200, HTTP 500/503, timeout after send, or a definite 412 followed by GET.
Direction A — false Committed (generic mutable key)
Two writers issue casMeta / putOverwriteControlled of identical bytes to one blob-meta key (GC condemn of {Condemned, round N, size S} is the realistic pair: two GC actors, or a retry).
Winner's If-Match succeeds. Loser's If-Match is 412, or either writer times out after the store applied the PUT.
Loser (or the timed-out winner) GETs, sees matching bytes, returns {Committed, got->token}.
Direction B — false Occupied (lost ACK of our own create)
Writer does putIfAbsent / slotOccupy / conditionalCreateControlled. Store applies it; ACK is lost.
Controller HEAD/GETs, key exists, returns Occupied and claims the outcome is not unresolved.
Expected behavior
An ambiguous conditional-write result stays Unresolved. Callers already have a wedge/retry for that (ref-log append). Do not return Committed. Do not return Occupied/Conflict. Do not adopt got->token as ours.
Ownership of a mutable key is only:
the etag from this PUT's 200, or
a writer-unique attempt id inside the payload (as claimOwnerOrThrow already checks server_uuid)
Byte equality of generic content is not authorship.
Actual behavior
False Committed + stolen token
CasRequestControl.cppputOverwriteControlled — any non-Done PUT, including definite 412, falls into GET; equal bytes → Committed with the GET's token:
classifyConditionalWriteResult correctly maps 500 / timeout / lost 200 to Unresolved (only malformed / entity-too-large / access-denied are DefiniteFailure). The bug is the resolution ladder after that: a later GET/HEAD of a shared key is treated as the PUT's result.
casMeta / putMetaIfAbsent feed that encoding into the controller. Two processes that want the same {Condemned, N, S} write identical bytes, so the equality branch fires even on a definite 412.
Consequence chain (GC)
Gc::writeCondemnedMeta treats outcome == Committed as “this process won the condemn CAS”:
if (writeCondemnedMeta(*store, ref, condemn_round, size))
noteCondemnMarkerDurable(ref, token);
That in-process set is later trusted without re-reading meta:
if (condemnMarkerConfirmedInProcess(entry.ref, entry.token))
returntrue;
The blob is graduated to delete_pending and deleteExact(blobKey, token) runs.
A part writer uses the same controller to move meta the other way (writeResurrectMetaClean: casMeta(..., Clean)). Both sides can take Committed from equality of generic {Clean,…} / {Condemned,…}. GC's local confirmation can then delete a body a writer has already re-admitted.
Consequence chain (Occupied → mount fault)
Wedge resolution classifies slotOccupy's Occupied payload. Equal bytes → Ours (accidentally OK for unique ref-log keys). Anything else, including a decode failure, is Foreign → lane Faulted, mount fenced, CORRUPTED_DATA “impossible foreign interference.”
Blob create maps Occupied to PreconditionFailed and adopts the object as a peer's live incarnation (PartWriteTxn::uploadFromSource), including when the occupant is our own lost-ACK write.
What is not this bug
Content-addressed blob keys (key = hash of bytes): “occupied at blobs/<hash>” means those bytes exist; adopting them does not claim exclusive create. The defect is using that reasoning on mutable slots (blob meta, mount/owner, ref-log occupy) whose payload does not name the writer.
Leave ambiguous results Unresolved. Do not promote GET/HEAD to Committed/Occupied. Do not return a GET etag as ours. If a protocol needs “I won this slot,” put a writer-unique attempt id in the payload and only treat a 200 of this PUT as Committed.
Update (2026-08-14) — narrowed after design review
The original report and the follow-up comment mixed different object types that the protocol treats differently on purpose. That overstated CAS-021. This update is the corrected scope.
Filimonov’s split is the right frame:
Blob bodies have no owner. There are only edges in the graph (ref → manifest → blob). It does not matter who uploaded the bytes. Occupied at blobs/<hash> means those bytes exist; adopting them is the design, not a bug.
Refs and manifests have exactly one writer per server_root_id, via the mount lease. The realistic race is this process against its own retry after a lost ACK, not two owners. Byte equality after GET on a lease-unique key (writer_epoch, seq / ManifestId) is “my attempt is durable.” Foreign is meant to be impossible and fail-closed.
The original “you must never infer authorship from content” is too wide here. Under a healthy mount lease, GET-after-ambiguity on those unique keys is a self-retry resolver, not an ownership oracle.
GC is not that writer. It does not take the mount lease. It takes a separate lease on gc/state (one leader per pool). That is a second actor, but it does not make blob bodies owned.
What the original report got wrong
Claim in the original / comment
Reality
Blob create Occupied → adopt is an integrity failure
Not a bug. Hash key, no owner.
Ref-log / manifest / snapshot Committed via byte equality after lost ACK
By design under the mount lease (self-retry of a unique id).
Wedge Occupied + equal bytes → Ours
By design (same attempt). Foreign is the invariant alarm if the lease held.
Two writers of opposite blob-meta (Clean vs Condemned) both “win” via equality
Does not happen. Different payloads → Conflict → retry.
Stolen GET etag splices the CAS chain today
Loaded gun, not a current firing path. GC and resurrect check .outcome only and discard the token.
The six controller exits are six extra bugs
One API. Most callers are consistent with the design split above.
The long consumer map in the first comment is still a useful inventory of call sites. It is not a list of distinct integrity holes.
What is still real (narrower)
1. The controller API names the wrong thing.
After a PUT that is not a clean Done, putOverwriteControlled / putIfAbsentControlledMutable / resolveByExactGet return Committed + the GET etag when bytes match — including on a definite 412. conditionalCreateControlled / slotOccupy return Occupied with unresolved_reason = NotUnresolved when the key exists.
classifyConditionalWriteResult already maps timeout / 500 / lost 200 to Unresolved. The GET/HEAD ladder then promotes that to a definite win/loss. For blob bodies and lease-unique refs that promotion is often the intended end state (“bytes I wanted are there” / “my unique slot is durable”). Returning Committed still means “this attempt won,” which is a lie when the PUT was 412 or never ACKed. That is API hygiene / a footgun, not proof of today’s data loss.
2. Blob meta is not a blob.
BlobMeta is {state, condemn_round, size} — a mutable flag between GC and writers, not content-addressed data and not covered by the mount lease.
Two GCs writing the same{Condemned, round N, size}: equality → both see Committed. That is the same fact as writeCondemnedMeta’s if already Condemned return true. Idempotent. Who wrote the marker does not matter if the next delete re-reads meta.
GC vs INSERT write opposite payloads → Conflict, not false Committed.
What can go stale: in-process condemn_markers_confirmed is trusted without a meta re-read (confirm_condemned_marker). That is a GC confirmation shortcut, not “blobs have owners.”
3. Call sites that ignore the CAS result (survive a controller-only fix).
claimMount: putIfAbsent / putOverwrite!= Done → LiveDoubleStart with no GET. Lost ACK of our mint can refuse the disk. Liveness, not blob ownership. When the slot already exists, claimMountdoes compare server_uuid (correct).
4. Not this issue
CAS-003 (CAS — consolidated static analysis audit findings (tracking) #2031): two GC leaders overlapping destructive phases (lease has no TTL; heartbeat casPut result discarded; blob deleteExact not revalidated). That is the serious “GC isn’t one” question. Separate from CAS-021.
Do not treat blob-body Occupied or lease-unique self-retry as bugs.
Do not steal a live GC lease to “fix” this (see CAS-003).
Still worth doing:
Stop returning Committed on a definite 412 just because GET bytes match; that is “desired state present,” not “this attempt won.” Keep a distinct outcome or stay Unresolved and let the caller re-read.
Do not return a GET etag as if it were this PUT’s token unless the PUT returned Done.
observeAndAdmit must honor putMetaIfAbsent / casMeta (Conflict / condemned).
Optionally: claimMount should GET and compare server_uuid on non-Done, not assume another live server.
A stress test that only checks “who uploaded the blob” will confirm Filimonov and miss the remainder. The interesting test is GC vs concurrent INSERT on the same hash (condemn vs adopt/resurrect), with injected 412/timeout on the meta PUT — and, separately, CAS-003 (two GC actors).
✅ I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.
Type of problem
Bug report — integrity. Split out of tracking issue #2031 (CAS-021, prev CAS-035). Static analysis of PR #2159 /
cas-code-only-strip; not yet reproduced end-to-end.Describe the situation
When a conditional object-store write (create-if-absent / If-Match overwrite) does not return a definite ACK, the CAS request controller guesses a definite outcome from a later GET/HEAD.
Two complementary lies:
putOverwriteControlled/putIfAbsentControlledMutableGET the key and, ifgot->bytes == bytes_s, returnCommittedwithgot->token— the current object's etag, which may belong to another writer.conditionalCreateControlled/slotOccupysee that the key exists and returnOccupiedwithunresolved_reason = NotUnresolved.Content equality answers “is the durable value the value I intended?” It does not answer “did this client's conditional PUT succeed?” That second question is authorship. It is only recoverable from this PUT's 200+etag, or from a writer/attempt id embedded in the object. Blob freshness meta is generic
{state, condemn_round, size}— no owner.This issue:
casMeta/putMetaIfAbsent)If-Matchcan splice the CAS chainCommittedas “I confirmed the condemn marker,” which licensesdeleteExactof the blob bodyThe same codebase already does the correct thing for the server-root owner anchor:
claimOwnerOrThrowrereads the object and accepts only ifserver_uuid == our_uuid.How to reproduce the behavior
Environment
feature/antalya-26.6/CAS, audited oncas-code-only-stripat842f2b37b8f)Steps (logical)
The controller never distinguishes “PUT applied, ACK lost” from “PUT refused” from “PUT never applied.” Any of: dropped 200, HTTP 500/503, timeout after send, or a definite 412 followed by GET.
Direction A — false
Committed(generic mutable key)casMeta/putOverwriteControlledof identical bytes to one blob-meta key (GC condemn of{Condemned, round N, size S}is the realistic pair: two GC actors, or a retry).{Committed, got->token}.Direction B — false
Occupied(lost ACK of our own create)putIfAbsent/slotOccupy/conditionalCreateControlled. Store applies it; ACK is lost.Occupiedand claims the outcome is not unresolved.Expected behavior
An ambiguous conditional-write result stays
Unresolved. Callers already have a wedge/retry for that (ref-log append). Do not returnCommitted. Do not returnOccupied/Conflict. Do not adoptgot->tokenas ours.Ownership of a mutable key is only:
claimOwnerOrThrowalready checksserver_uuid)Byte equality of generic content is not authorship.
Actual behavior
False
Committed+ stolen tokenCasRequestControl.cppputOverwriteControlled— any non-DonePUT, including definite 412, falls into GET; equal bytes →Committedwith the GET's token:Same shape in
putIfAbsentControlledMutable.putIfAbsentControlledusesresolveByExactGet, which also maps equal bytes →Committed.False
OccupiedconditionalCreateControlled: after a thrown/non-Done attempt,head(key).exists→{Occupied, {}}with no token and no unresolved reason.slotOccupy: GET exists →Kind::Occupied,unresolved_reason = NotUnresolved,occupant_bytespossibly our own.Root cause analysis
classifyConditionalWriteResultcorrectly maps 500 / timeout / lost 200 toUnresolved(only malformed / entity-too-large / access-denied areDefiniteFailure). The bug is the resolution ladder after that: a later GET/HEAD of a shared key is treated as the PUT's result.Blob meta has no owner id:
casMeta/putMetaIfAbsentfeed that encoding into the controller. Two processes that want the same{Condemned, N, S}write identical bytes, so the equality branch fires even on a definite 412.Consequence chain (GC)
Gc::writeCondemnedMetatreatsoutcome == Committedas “this process won the condemn CAS”:That in-process set is later trusted without re-reading meta:
The blob is graduated to
delete_pendinganddeleteExact(blobKey, token)runs.A part writer uses the same controller to move meta the other way (
writeResurrectMetaClean:casMeta(..., Clean)). Both sides can takeCommittedfrom equality of generic{Clean,…}/{Condemned,…}. GC's local confirmation can then delete a body a writer has already re-admitted.Consequence chain (Occupied → mount fault)
Wedge resolution classifies
slotOccupy'sOccupiedpayload. Equal bytes →Ours(accidentally OK for unique ref-log keys). Anything else, including a decode failure, isForeign→ laneFaulted, mount fenced,CORRUPTED_DATA“impossible foreign interference.”Blob create maps
OccupiedtoPreconditionFailedand adopts the object as a peer's live incarnation (PartWriteTxn::uploadFromSource), including when the occupant is our own lost-ACK write.What is not this bug
Content-addressed blob keys (key = hash of bytes): “occupied at
blobs/<hash>” means those bytes exist; adopting them does not claim exclusive create. The defect is using that reasoning on mutable slots (blob meta, mount/owner, ref-log occupy) whose payload does not name the writer.Additional context
Anchors (
CA/=src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/)CA/Backend/CasRequestControl.cpp:427-435,:498-506— equality →Committed+ stolen tokenCA/Backend/CasRequestControl.cpp:357-368,:543-562— existence →OccupiedCA/Backend/CasRequestControl.cpp:229-233—resolveByExactGetsame equality ruleCA/Backend/CasObjectStorageBackend.cpp:109-124—NoSuchKeymapped toPreconditionFailedCA/Pool/CasBlobMeta.cpp/Formats/CasBlobMetaFormat.h— generic meta, no ownerCA/Gc/CasGc.cppwriteCondemnedMeta/noteCondemnMarkerDurable/deleteExactCA/Pool/CasPartWriteTxn.cppwriteResurrectMetaClean,uploadFromSourceOccupied → adoptCA/Pool/CasRefLedger.cppclassifyRefLogOccupant, wedgeForeign→ FaultedCA/Pool/CasServerRoot.cppclaimOwnerOrThrowcomparesserver_uuidTracking
NEW-FINDINGS.mdCAS-021;reports/tier2.mdtier2-4 / tier2-5Suggested fix direction
Leave ambiguous results
Unresolved. Do not promote GET/HEAD toCommitted/Occupied. Do not return a GET etag as ours. If a protocol needs “I won this slot,” put a writer-unique attempt id in the payload and only treat a 200 of this PUT asCommitted.Update (2026-08-14) — narrowed after design review
The original report and the follow-up comment mixed different object types that the protocol treats differently on purpose. That overstated CAS-021. This update is the corrected scope.
Filimonov’s split is the right frame:
ref → manifest → blob). It does not matter who uploaded the bytes. Occupied atblobs/<hash>means those bytes exist; adopting them is the design, not a bug.server_root_id, via the mount lease. The realistic race is this process against its own retry after a lost ACK, not two owners. Byte equality after GET on a lease-unique key (writer_epoch,seq/ManifestId) is “my attempt is durable.”Foreignis meant to be impossible and fail-closed.The original “you must never infer authorship from content” is too wide here. Under a healthy mount lease, GET-after-ambiguity on those unique keys is a self-retry resolver, not an ownership oracle.
GC is not that writer. It does not take the mount lease. It takes a separate lease on
gc/state(one leader per pool). That is a second actor, but it does not make blob bodies owned.What the original report got wrong
Occupied→ adopt is an integrity failureCommittedvia byte equality after lost ACKOccupied+ equal bytes →OursForeignis the invariant alarm if the lease held.CleanvsCondemned) both “win” via equalityConflict→ retry..outcomeonly and discard the token.The long consumer map in the first comment is still a useful inventory of call sites. It is not a list of distinct integrity holes.
What is still real (narrower)
1. The controller API names the wrong thing.
After a PUT that is not a clean
Done,putOverwriteControlled/putIfAbsentControlledMutable/resolveByExactGetreturnCommitted+ the GET etag when bytes match — including on a definite 412.conditionalCreateControlled/slotOccupyreturnOccupiedwithunresolved_reason = NotUnresolvedwhen the key exists.classifyConditionalWriteResultalready maps timeout / 500 / lost 200 toUnresolved. The GET/HEAD ladder then promotes that to a definite win/loss. For blob bodies and lease-unique refs that promotion is often the intended end state (“bytes I wanted are there” / “my unique slot is durable”). ReturningCommittedstill means “this attempt won,” which is a lie when the PUT was 412 or never ACKed. That is API hygiene / a footgun, not proof of today’s data loss.2. Blob meta is not a blob.
BlobMetais{state, condemn_round, size}— a mutable flag between GC and writers, not content-addressed data and not covered by the mount lease.{Condemned, round N, size}: equality → both seeCommitted. That is the same fact aswriteCondemnedMeta’sif already Condemned return true. Idempotent. Who wrote the marker does not matter if the next delete re-reads meta.Conflict, not falseCommitted.condemn_markers_confirmedis trusted without a meta re-read (confirm_condemned_marker). That is a GC confirmation shortcut, not “blobs have owners.”3. Call sites that ignore the CAS result (survive a controller-only fix).
observeAndAdmitdiscardsputMetaIfAbsent(Clean). If GC already putCondemned, the writer can still adopt. That is “ignore the outcome,” adjacent to CAS — consolidated static analysis audit findings (tracking) #2031 CAS-002, not content-as-authorship.claimMount:putIfAbsent/putOverwrite!= Done→LiveDoubleStartwith no GET. Lost ACK of our mint can refuse the disk. Liveness, not blob ownership. When the slot already exists,claimMountdoes compareserver_uuid(correct).4. Not this issue
casPutresult discarded; blobdeleteExactnot revalidated). That is the serious “GC isn’t one” question. Separate from CAS-021.SYSTEM CAS GC RUNon a follower silently no-ops. Operator UX, not this protocol.Expected behavior (revised)
Do not treat blob-body Occupied or lease-unique self-retry as bugs.
Do not steal a live GC lease to “fix” this (see CAS-003).
Still worth doing:
Committedon a definite 412 just because GET bytes match; that is “desired state present,” not “this attempt won.” Keep a distinct outcome or stayUnresolvedand let the caller re-read.Done.observeAndAdmitmust honorputMetaIfAbsent/casMeta(Conflict/ condemned).claimMountshould GET and compareserver_uuidon non-Done, not assume another live server.A stress test that only checks “who uploaded the blob” will confirm Filimonov and miss the remainder. The interesting test is GC vs concurrent INSERT on the same hash (condemn vs adopt/resurrect), with injected 412/timeout on the meta PUT — and, separately, CAS-003 (two GC actors).
Not reproduced end-to-end. Static / logical only.