Skip to content

CAS-021: Ambiguous conditional-write outcomes are reported as definite ownership #2207

Description

@alsugiliazova

I checked the Altinity Stable Builds lifecycle table, and the Altinity Stable Build version I'm using is still supported.

Type of problem

Bug report — integrity. Split out of tracking issue #2031 (CAS-021, prev CAS-035). Static analysis of PR #2159 / cas-code-only-strip; not yet reproduced end-to-end.

Describe the situation

When a conditional object-store write (create-if-absent / If-Match overwrite) does not return a definite ACK, the CAS request controller guesses a definite outcome from a later GET/HEAD.

Two complementary lies:

  1. Content equality is treated as proof that this attempt committed. After 412, 500, timeout, or a lost 200, putOverwriteControlled / putIfAbsentControlledMutable GET the key and, if got->bytes == bytes_s, return Committed with got->token — the current object's etag, which may belong to another writer.
  2. A landed-then-timed-out write is reported as another writer's object. After the same ambiguity, conditionalCreateControlled / slotOccupy see that the key exists and return Occupied with unresolved_reason = NotUnresolved.

Content equality answers “is the durable value the value I intended?” It does not answer “did this client's conditional PUT succeed?” That second question is authorship. It is only recoverable from this PUT's 200+etag, or from a writer/attempt id embedded in the object. Blob freshness meta is generic {state, condemn_round, size} — no owner.

This issue:

  • Erases compare-and-set exclusivity on mutable keys (casMeta / putMetaIfAbsent)
  • Hands the loser the winner's etag, so a later If-Match can splice the CAS chain
  • Lets GC treat a false Committed as “I confirmed the condemn marker,” which licenses deleteExact of the blob body
  • Lets a lost ACK of our own ref-log append be classified as foreign occupancy and fault the mount as impossible interference

The same codebase already does the correct thing for the server-root owner anchor: claimOwnerOrThrow rereads the object and accepts only if server_uuid == our_uuid.


How to reproduce the behavior

Environment

Steps (logical)

The controller never distinguishes “PUT applied, ACK lost” from “PUT refused” from “PUT never applied.” Any of: dropped 200, HTTP 500/503, timeout after send, or a definite 412 followed by GET.

Direction A — false Committed (generic mutable key)

  1. Two writers issue casMeta / putOverwriteControlled of identical bytes to one blob-meta key (GC condemn of {Condemned, round N, size S} is the realistic pair: two GC actors, or a retry).
  2. Winner's If-Match succeeds. Loser's If-Match is 412, or either writer times out after the store applied the PUT.
  3. Loser (or the timed-out winner) GETs, sees matching bytes, returns {Committed, got->token}.

Direction B — false Occupied (lost ACK of our own create)

  1. Writer does putIfAbsent / slotOccupy / conditionalCreateControlled. Store applies it; ACK is lost.
  2. Controller HEAD/GETs, key exists, returns Occupied and claims the outcome is not unresolved.

Expected behavior

An ambiguous conditional-write result stays Unresolved. Callers already have a wedge/retry for that (ref-log append). Do not return Committed. Do not return Occupied/Conflict. Do not adopt got->token as ours.

Ownership of a mutable key is only:

  • the etag from this PUT's 200, or
  • a writer-unique attempt id inside the payload (as claimOwnerOrThrow already checks server_uuid)

Byte equality of generic content is not authorship.


Actual behavior

False Committed + stolen token

CasRequestControl.cpp putOverwriteControlledany non-Done PUT, including definite 412, falls into GET; equal bytes → Committed with the GET's token:

else if (got && got->bytes == bytes_s)
{
    ...
    return {CasOverwriteOutcome::Committed, got->token};
}

Same shape in putIfAbsentControlledMutable. putIfAbsentControlled uses resolveByExactGet, which also maps equal bytes → Committed.

False Occupied

conditionalCreateControlled: after a thrown/non-Done attempt, head(key).exists{Occupied, {}} with no token and no unresolved reason.

slotOccupy: GET exists → Kind::Occupied, unresolved_reason = NotUnresolved, occupant_bytes possibly our own.


Root cause analysis

classifyConditionalWriteResult correctly maps 500 / timeout / lost 200 to Unresolved (only malformed / entity-too-large / access-denied are DefiniteFailure). The bug is the resolution ladder after that: a later GET/HEAD of a shared key is treated as the PUT's result.

Blob meta has no owner id:

struct BlobMeta {
    uint8_t version = 1;
    MetaState state = MetaState::Clean;  // clean | condemned
    uint64_t condemn_round = 0;
    uint64_t size = 0;
};

casMeta / putMetaIfAbsent feed that encoding into the controller. Two processes that want the same {Condemned, N, S} write identical bytes, so the equality branch fires even on a definite 412.

Consequence chain (GC)

Gc::writeCondemnedMeta treats outcome == Committed as “this process won the condemn CAS”:

if (writeCondemnedMeta(*store, ref, condemn_round, size))
    noteCondemnMarkerDurable(ref, token);

That in-process set is later trusted without re-reading meta:

if (condemnMarkerConfirmedInProcess(entry.ref, entry.token))
    return true;

The blob is graduated to delete_pending and deleteExact(blobKey, token) runs.

A part writer uses the same controller to move meta the other way (writeResurrectMetaClean: casMeta(..., Clean)). Both sides can take Committed from equality of generic {Clean,…} / {Condemned,…}. GC's local confirmation can then delete a body a writer has already re-admitted.

Consequence chain (Occupied → mount fault)

Wedge resolution classifies slotOccupy's Occupied payload. Equal bytes → Ours (accidentally OK for unique ref-log keys). Anything else, including a decode failure, is Foreign → lane Faulted, mount fenced, CORRUPTED_DATA “impossible foreign interference.”

Blob create maps Occupied to PreconditionFailed and adopts the object as a peer's live incarnation (PartWriteTxn::uploadFromSource), including when the occupant is our own lost-ACK write.

What is not this bug

Content-addressed blob keys (key = hash of bytes): “occupied at blobs/<hash>” means those bytes exist; adopting them does not claim exclusive create. The defect is using that reasoning on mutable slots (blob meta, mount/owner, ref-log occupy) whose payload does not name the writer.


Additional context

Anchors (CA/ = src/Disks/DiskObjectStorage/MetadataStorages/ContentAddressed/)

  • CA/Backend/CasRequestControl.cpp:427-435, :498-506 — equality → Committed + stolen token
  • CA/Backend/CasRequestControl.cpp:357-368, :543-562 — existence → Occupied
  • CA/Backend/CasRequestControl.cpp:229-233resolveByExactGet same equality rule
  • CA/Backend/CasObjectStorageBackend.cpp:109-124NoSuchKey mapped to PreconditionFailed
  • CA/Pool/CasBlobMeta.cpp / Formats/CasBlobMetaFormat.h — generic meta, no owner
  • CA/Gc/CasGc.cpp writeCondemnedMeta / noteCondemnMarkerDurable / deleteExact
  • CA/Pool/CasPartWriteTxn.cpp writeResurrectMetaClean, uploadFromSource Occupied → adopt
  • CA/Pool/CasRefLedger.cpp classifyRefLogOccupant, wedge Foreign → Faulted
  • Contrast (correct): CA/Pool/CasServerRoot.cpp claimOwnerOrThrow compares server_uuid

Tracking

Suggested fix direction

Leave ambiguous results Unresolved. Do not promote GET/HEAD to Committed/Occupied. Do not return a GET etag as ours. If a protocol needs “I won this slot,” put a writer-unique attempt id in the payload and only treat a 200 of this PUT as Committed.

Update (2026-08-14) — narrowed after design review

The original report and the follow-up comment mixed different object types that the protocol treats differently on purpose. That overstated CAS-021. This update is the corrected scope.

Filimonov’s split is the right frame:

  • Blob bodies have no owner. There are only edges in the graph (ref → manifest → blob). It does not matter who uploaded the bytes. Occupied at blobs/<hash> means those bytes exist; adopting them is the design, not a bug.
  • Refs and manifests have exactly one writer per server_root_id, via the mount lease. The realistic race is this process against its own retry after a lost ACK, not two owners. Byte equality after GET on a lease-unique key (writer_epoch, seq / ManifestId) is “my attempt is durable.” Foreign is meant to be impossible and fail-closed.

The original “you must never infer authorship from content” is too wide here. Under a healthy mount lease, GET-after-ambiguity on those unique keys is a self-retry resolver, not an ownership oracle.

GC is not that writer. It does not take the mount lease. It takes a separate lease on gc/state (one leader per pool). That is a second actor, but it does not make blob bodies owned.


What the original report got wrong

Claim in the original / comment Reality
Blob create Occupied → adopt is an integrity failure Not a bug. Hash key, no owner.
Ref-log / manifest / snapshot Committed via byte equality after lost ACK By design under the mount lease (self-retry of a unique id).
Wedge Occupied + equal bytes → Ours By design (same attempt). Foreign is the invariant alarm if the lease held.
Two writers of opposite blob-meta (Clean vs Condemned) both “win” via equality Does not happen. Different payloads → Conflict → retry.
Stolen GET etag splices the CAS chain today Loaded gun, not a current firing path. GC and resurrect check .outcome only and discard the token.
The six controller exits are six extra bugs One API. Most callers are consistent with the design split above.

The long consumer map in the first comment is still a useful inventory of call sites. It is not a list of distinct integrity holes.


What is still real (narrower)

1. The controller API names the wrong thing.

After a PUT that is not a clean Done, putOverwriteControlled / putIfAbsentControlledMutable / resolveByExactGet return Committed + the GET etag when bytes match — including on a definite 412. conditionalCreateControlled / slotOccupy return Occupied with unresolved_reason = NotUnresolved when the key exists.

classifyConditionalWriteResult already maps timeout / 500 / lost 200 to Unresolved. The GET/HEAD ladder then promotes that to a definite win/loss. For blob bodies and lease-unique refs that promotion is often the intended end state (“bytes I wanted are there” / “my unique slot is durable”). Returning Committed still means “this attempt won,” which is a lie when the PUT was 412 or never ACKed. That is API hygiene / a footgun, not proof of today’s data loss.

2. Blob meta is not a blob.

BlobMeta is {state, condemn_round, size} — a mutable flag between GC and writers, not content-addressed data and not covered by the mount lease.

  • Two GCs writing the same {Condemned, round N, size}: equality → both see Committed. That is the same fact as writeCondemnedMeta’s if already Condemned return true. Idempotent. Who wrote the marker does not matter if the next delete re-reads meta.
  • GC vs INSERT write opposite payloads → Conflict, not false Committed.
  • What can go stale: in-process condemn_markers_confirmed is trusted without a meta re-read (confirm_condemned_marker). That is a GC confirmation shortcut, not “blobs have owners.”

3. Call sites that ignore the CAS result (survive a controller-only fix).

  • observeAndAdmit discards putMetaIfAbsent(Clean). If GC already put Condemned, the writer can still adopt. That is “ignore the outcome,” adjacent to CAS — consolidated static analysis audit findings (tracking) #2031 CAS-002, not content-as-authorship.
  • claimMount: putIfAbsent / putOverwrite != DoneLiveDoubleStart with no GET. Lost ACK of our mint can refuse the disk. Liveness, not blob ownership. When the slot already exists, claimMount does compare server_uuid (correct).

4. Not this issue


Expected behavior (revised)

Do not treat blob-body Occupied or lease-unique self-retry as bugs.

Do not steal a live GC lease to “fix” this (see CAS-003).

Still worth doing:

  • Stop returning Committed on a definite 412 just because GET bytes match; that is “desired state present,” not “this attempt won.” Keep a distinct outcome or stay Unresolved and let the caller re-read.
  • Do not return a GET etag as if it were this PUT’s token unless the PUT returned Done.
  • observeAndAdmit must honor putMetaIfAbsent / casMeta (Conflict / condemned).
  • Optionally: claimMount should GET and compare server_uuid on non-Done, not assume another live server.

A stress test that only checks “who uploaded the blob” will confirm Filimonov and miss the remainder. The interesting test is GC vs concurrent INSERT on the same hash (condemn vs adopt/resurrect), with injected 412/timeout on the meta PUT — and, separately, CAS-003 (two GC actors).

Not reproduced end-to-end. Static / logical only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions