Skip to content

Set CKRecord system fields in MockCloudDatabase - #573

Closed
lukaskubanek wants to merge 5 commits into
pointfreeco:mainfrom
structuredpath:mock-record-system-fields
Closed

lukaskubanek wants to merge 5 commits into
pointfreeco:mainfrom
structuredpath:mock-record-system-fields

Conversation

@lukaskubanek

@lukaskubanek lukaskubanek commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #411, which populated CKRecord.modificationDate in MockCloudDatabase via swizzling. Motivation and test impact are documented there. This PR replaces the mechanism, as requested in review, and also includes the planned follow-up for handling recordChangeTag.

Beyond aligning MockCloudDatabase with the actual server behavior, this is a prerequisite for the conflict resolution work, which will rely on these system fields to detect and handle conflicts.

Approaches Tried

  • Subclassing CKRecord: CloudKit logs BUG IN CLIENT OF CLOUDKIT: CKRecord should not be subclassed, so subclassing doesn’t seem to be supported.
  • Underscored property with fallback: Two sources of truth, and every call site must remember to use it. Otherwise tests exercise different behavior than production.
  • Swizzling: Works, but requires quite a bit of plumbing (e.g. copy(with:) must be swizzled too so that associated objects survive). Concerns regarding static analysis were also raised in the review of the initial PR.
  • Archive roundtrip: The approach used in this PR.

Changes

  • Add CKRecord.copyWithMockSystemFields(recordChangeTag:modificationDate:), which creates a new CKRecord instance with the system fields set to the provided values. It archives the receiver and unarchives it through a custom NSKeyedUnarchiver subclass that substitutes the two archived keys. The values land in the real system fields, so they are returned through the regular properties, survive copy() and persist to SQLite with no mock-specific plumbing.
  • Assign modificationDate and recordChangeTag in MockCloudDatabase on every save and when touching a parent record. Change tags are now zero-based base-36 strings, matching CloudKit’s format.
  • Order change tags in MockSyncEngine with isNewerChangeTag(than:), which compares by length before lexicographic order, since base-36 strings such as "10" and "z" don’t sort correctly on their own.
  • Read the actual recordChangeTag in MockCloudDatabase and MockSyncEngine, and remove _recordChangeTag, its isTesting persistence branches and the CustomDump filter that hid the mock field in snapshots.

Naming

I went back and forth between settingMockSystemFields(…), which follows the Swift naming convention for non-mutating variants, and copyWithMockSystemFields(…). I settled on the latter because CKRecord is an Objective-C class, the name sits naturally next to copy(with:), and the method takes the place of explicit copy() calls in the mock. Happy to hear your take.

Impact on Tests

Mainly still the same change as in #411: isCompleted🗓️: 30 becomes 60 in one merge conflict test. There is much more to untangle regarding storing the in-flight record as the last-known server record on send, but I’d like to discuss that separately. The key point here is that the test is now exercising the same behavior as production.


Disclaimer: I used Claude Code to help me write the code and work through the large amount of prep work I accumulated over the past months.

`CKRecord.copyWithMockSystemFields(…)` returns a copy with the given values
stored in the real system fields, so they behave like server-assigned values
without swizzling. Supports `modificationDate` only for now and is not yet
used by `MockCloudDatabase`.
Records saved in a batch now receive an incrementing modification date,
mirroring the CloudKit server. The sync engine's modification date checks
therefore run in tests as they do in production, which changes an
expectation in one merge conflict test.
`MockCloudDatabase` now stores the change tag in the real system field as
well, while the sync machinery still reads `_recordChangeTag` for now.
`MockCloudDatabase` and `MockSyncEngine` now read `CKRecord.recordChangeTag`
from the record's system field instead of the `_recordChangeTag` mock field,
which is removed along with its `isTesting` persistence branches. Change tags
become strings but remain digit-based for now, to be replaced later.
Matches the format CloudKit uses for `CKRecord.recordChangeTag`.
@mbrandonw

Copy link
Copy Markdown
Member

Hey @lukaskubanek, Stephen and I chatted about this quite a bit today and we're still trying to wrap our heads around a number of things.

First, and easiest to discuss, it does not seem necessary to make any changes to _recordChangeTag because as far as I can tell there is no deviation between how it works in tests and how it would work in production. Even the change to make it a base-36 value is purely cosmetic and does not actually change any functionality. Would you agree with this?

Second, I still don't see why something like this:

extension CKRecord {
  package var _modificationDate: Date? {
    get { modificationDate ?? self[#function]  }
    set { self[#function] = newValue }
  }
}

…couldn't work to simulate CKRecord.modificationDate in tests. It would allow the mock database to set the modificationDate in tests while in production it would always fallback to modificationDate.

Third, and finally, we definitely agree that the mock database really should be simulating modificationDate, and so we do want to update the mock database. And you saying that the act of updating the mock database to set modificationDates causes the 30 -> 60 timestamp test failure, right? If so, then I think we would interpret that as a bug in our code that needs fixing to ensure the timestamp stays 30.

I understand your reasoning about 3-way merging, but consider the following. There are 3 devices (A, B, C), and A makes a change at t=1, B makes a change at t=2 and C makes a change at t=3. Further suppose that A and B sync at t=4 so that their conflicts are merged and the timestamp is moved to t=4 with your suggestion. And finally C syncs at t>4, but because its change was at t=3 its edit is discarded. This is happening even though C made its change after A and B, and the reason it is happening is because the conflict resolution is updating the timestamp to be the newest time.

So it really does seem we would like that test to stay at 30 and for the modificationDate to be handled correctly in the mock database. We are investigating things a bit more and will share what we find, but please let us know if you have any comments about the above.

@mbrandonw

Copy link
Copy Markdown
Member

Oh, and one more thing. It also seems like it would be ideal to use \.currentTime for modificationDate so that all conflict resolution is using the same notion of time. I believe you had comments about that but I cannot find them now.

@mbrandonw

Copy link
Copy Markdown
Member

Hey @lukaskubanek, here is our approach to modificationDate: #574

FWIW, the timestamp test failure was actually a legitimate bug in the sync engine. The place where we compared modificationDate we used < instead of <=, but it is absolutely the case that we can refresh last known server records with the same modificationDate. This happens when we grab the existing last known server record and make some updates to it since that does not change the modificationDate.

Would love to get your thoughts on this approach.

@lukaskubanek

lukaskubanek commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor Author

@mbrandonw Thanks for digging in! Here are some initial thoughts before going to bed. Will have more to say tomorrow.

I still don’t see why something like this […] couldn’t work to simulate CKRecord.modificationDate in tests.

It works, but it requires all (current and future) call sites in the library to go through the underscored property. This makes the library aware of and coupled to the mock infrastructure.

Another downside is that the mock value is stored as a regular record field, so it leaks into allKeys() and needs special handling in dumps and in the SQLite representations.

The approach presented in this PR, unlike the one from #574, doesn’t come with these downsides and is self-contained. Its cost is relying on CloudKit’s private archive keys, which seem to be pretty stable and only ever touched in tests.

it does not seem necessary to make any changes to _recordChangeTag

When the change tag is accessed from the library, which I think is needed for properly detecting conflicts, it’s in the same situation as modificationDate. So, should you favor the underscored properties, all call sites would need to go through _recordChangeTag as well.

Even the change to make it a base-36 value is purely cosmetic and does not actually change any functionality. Would you agree with this?

Yes, this change is purely cosmetic. The idea, which I hope you share, is to bring the mock cloud database as close to the actual behavior as possible.

FWIW, the timestamp test failure was actually a legitimate bug in the sync engine.

I agree and I’ve found even more of them and have fixes in my fork. The thing is that I wanted to first get tests ready that document the current behavior including the bugs (see also #414) and then submit the fixes. That’s why I’ve been focusing on the test infrastructure. But it’s really hard to pick the exact amount of changes to submit.

I’ll reply regarding the remaining topics tomorrow.

@mbrandonw

Copy link
Copy Markdown
Member

Hey @lukaskubanek, thanks for the clarifications!

It works, but it requires all (current and future) call sites in the library to go through the underscored property. This makes the library aware of and coupled to the mock infrastructure.

I think we are willing to take this downside to not resort to swizzling or encoding/decoding tricks. Further, we could technically swizzle in just debug mode to reportIssue if modificationDate is ever used directly. But now we would not want to swizzle only in debug mode in general because then we could not run tests in release mode.

Another downside is that the mock value is stored as a regular record field, so it leaks into allKeys() and needs special handling in dumps and in the SQLite representations.

True, but that's easy enough to handle.

The approach presented in this PR, unlike the one from #574, doesn’t come with these downsides and is self-contained. Its cost is relying on CloudKit’s private archive keys, which seem to be pretty stable and only ever touched in tests.

Your approach has a similar downside in that you have to make sure to use the special copyWithMockSystemFields() everywhere instead of copy().

So, should you favor the underscored properties, all call sites would need to go through _recordChangeTag as well.

Yep, but we are willing to accept that tradeoff.

Yes, this change is purely cosmetic. The idea, which I hope you share, is to bring the mock cloud database as close to the actual behavior as possible.

Ok cool, just wanted to make sure I understand. We certainly can take that base-36 change as a self-contained PR if you have the time, or I can get to it sometime soon.

I agree and I’ve found even more of them and have fixes in my fork. The thing is that I wanted to first get tests ready that document the current behavior including the bugs (see also #414) and then submit the fixes. That’s why I’ve been focusing on the test infrastructure. But it’s really hard to pick the exact amount of changes to submit.

Yeah sorry it's been difficult to land things. I would be curious to see if you have one (or a few) failing tests that demonstrate some behavior we are getting wrong (ideally based on some of the PRs we will be merging soon) so that we can understand a single issue at a time.

@lukaskubanek

Copy link
Copy Markdown
Contributor Author

@mbrandonw OK, so my understanding is that you’re going forward with the underscore approach in #574. While not my favored approach, I’m happy to see the modificationDate populated in the mock cloud database, which helped surface and fix one of the bugs. That’s a step forward!

to not resort to swizzling or encoding/decoding tricks

Just so we’re on the same page. I moved away from swizzling (in #411) to the encoding/decoding trick in this PR. So I considered swizzling ruled out already as you’ve requested before.

Your approach has a similar downside in that you have to make sure to use the special copyWithMockSystemFields() everywhere instead of copy().

Allow me one last comment on this and then I’ll sign off. The difference I see with this is that you’d be nudged to use the former when you want to set the mock values, which happens only in the MockCloudDatabase (and one test that simulates remote change). There’d literally be no other way around it. On the other hand, the underscored properties coexist with the regular ones and one needs to remember this, as otherwise the tests would exercise wrong logic. But fortunately, it’s not that many instances, so it’s fine.

It also seems like it would be ideal to use \.currentTime for modificationDate so that all conflict resolution is using the same notion of time.

That makes more sense, yes.

I understand your reasoning about 3-way merging, but consider the following. There are 3 devices (A, B, C), and A makes a change at t=1, B makes a change at t=2 and C makes a change at t=3. Further suppose that A and B sync at t=4 so that their conflicts are merged and the timestamp is moved to t=4 with your suggestion. And finally C syncs at t>4, but because its change was at t=3 its edit is discarded. This is happening even though C made its change after A and B, and the reason it is happening is because the conflict resolution is updating the timestamp to be the newest time.

You’re likely referring to the section “Impact on Tests” from #411. It’s been a while since I wrote that and I since gained a better understanding of what’s going on. So, let me give an updated stance on this matter.

I agree that the described scenario is problematic and that a merge of A and B should not promote the field timestamp to t=4 and thus ignore the change from C at t=3. I think a better approach is to use the max of the per-field timestamps, i.e. t=2 from B, and leave other untouched fields alone. And that is regardless of the merge policy, be it the current last edit wins or a potential customizable one in the future.

So, the flip in the test was wrong. In hindsight, I should have used withKnownIssue for the flipped value in the test to mark it as needing attention rather than simply running with the changed value, so that one is definitely on me.

I would be curious to see if you have one (or a few) failing tests that demonstrate some behavior we are getting wrong so that we can understand a single issue at a time.

As referenced above, I have a draft of rewritten merge conflict tests ready that show all possible scenarios, document the current behavior, and uncover two more categories of bugs. I know it goes against your requirement of presenting a single issue at a time, but I feel this needs to be considered more holistically. And since these are tests, not production code, I wanted to ask if there is a remote chance of #414 being accepted in its current shape (assuming rebased onto your latest changes)? I spent quite a lot of time distilling these scenarios and I’d like to contribute them to improve the library, but I’m totally lost when it comes to the approach I should take.

Ok cool, just wanted to make sure I understand. We certainly can take that base-36 change as a self-contained PR if you have the time, or I can get to it sometime soon.

I mean I’d still be willing to, but I’m not sure if it’s helpful. Since you’re already in that code it might be easier if it comes from your side. The implementation is simple enough and it can be found in this PR.


Feel free to close this PR. I’d still be glad to hear your thoughts on my comments above, though.

@mbrandonw

Copy link
Copy Markdown
Member

Just so we’re on the same page. I moved away from swizzling (in #411) to the encoding/decoding trick in this PR. So I considered swizzling ruled out already as you’ve requested before.

Yeah, sorry, I understand you moved away from swizzling. I was just re-iterating all the possible solutions we had chatted about in the past.

On the other hand, the underscored properties coexist with the regular ones and one needs to remember this, as otherwise the tests would exercise wrong logic.

Point definitely taken, and we can always switch to something more robust in the future!

And since these are tests, not production code, I wanted to ask if there is a remote chance of #414 being accepted in its current shape (assuming rebased onto your latest changes)?

Yeah definitely. Before rebasing let me and Stephen chat about it first and we will get back to you.

I mean I’d still be willing to, but I’m not sure if it’s helpful. Since you’re already in that code it might be easier if it comes from your side. The implementation is simple enough and it can be found in this PR.

Yep, no problem!

@lukaskubanek

Copy link
Copy Markdown
Contributor Author

Before rebasing let me and Stephen chat about it first and we will get back to you.

I actually went ahead and rebased #414 onto main, as it contained the baggage from my attempts at mocking the modificationDate, which made it harder to review the important bits. Since #574 landed, I was able to submit #414 for review. Happy to take advice and make changes over there.

We certainly can take that base-36 change as a self-contained PR if you have the time, or I can get to it sometime soon.

Since the base-36 format for change tags didn’t make it to the latest release, I went ahead, extracted those changes from my branch, and submitted them as #580.


With that, I’m closing this PR in favor of #574.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants