Skip to content

CurrentBehavior.md

Latest commit

 

History

History
211 lines (158 loc) · 27.1 KB

CurrentBehavior.md

File metadata and controls

211 lines (158 loc) · 27.1 KB
Raw

Current CloudKit Sync Behavior in SQLiteData

This document captures how CloudKit sync currently behaves in SQLiteData as of version 1.8.11, focusing on the implementation of the built-in “field-wise last edit wins” conflict resolution strategy. In particular, it documents how last-known server records and timestamps are handled today. The goal is to provide a clear reference for the current semantics as a first step towards making the conflict resolution customizable per the discussion in #272 and to highlight areas that will need to change in order to support a proper three-way merge and custom merge strategies.

1. Core Concepts

1.1. General Sync Flow

At a high level, SQLiteData’s CloudKit sync operates in two directions, both mediated by an underlying CKSyncEngine:

  • Client → Server: Changes are first written to the local SQLite database and later picked up by the sync engine, translated into CKRecords, and sent to the server. When an upload succeeds, the local database already reflects the desired state, so sync primarily updates metadata.
  • Server → Client: Changes arriving from the server are applied to the local database using upsert logic, which inserts new rows or updates existing ones. In this direction, incoming server state is reconciled with both the local database state and any pending client changes using the built-in “field-wise last edit wins” strategy. Sync metadata is updated accordingly.

Conflicts arise when client-side and server-side changes affect the same record concurrently. They fall into two families.2 A merge conflict occurs for a previously synchronized record, where the last-known server record serves as the ancestor for a three-way merge. A reconciliation conflict occurs when no last-known server record exists, such as for independently created records, leaving only a two-way comparison. In both families, a conflict-on-send scenario is handled explicitly by reacting to a .serverRecordChanged error and performing an upsert, while a conflict-on-fetch scenario is handled implicitly by applying the same upsert logic without an explicit record-level conflict detection.3

The following sections describe the data structures and timestamps that underpin this flow, before diving into concrete send, fetch, and conflict scenarios.

1.2. SyncMetadata Fields

SQLiteData with syncing enabled manages a SyncMetadata table in the metadatabase with a row for each row in the user tables. Data stored in that table is used when interfacing with CloudKit. Among others, the table contains the following columns:

  • lastKnownServerRecord: The last known CKRecord received from the server serialized via CKRecord.encodeSystemFields(with:) and thus containing only the system fields.
  • _lastKnownServerRecordAllFields: The last known CKRecord received from the server serialized via CKRecord.encode(with:) and thus containing all fields, including per-field modification timestamps (explained in the next section). As it stands, the last-known server record doesn’t necessarily reflect the server state all the time. There are some nuances that are explained later in the document.
  • userModificationTime: A timestamp indicating when the user last modified the record. This value gets updated whenever the row in the user table is updated, be it by a manual edit or due to a change coming from the sync engine.

1.3. Per-Field Timestamps

SQLiteData’s built-in “field-wise last edit wins” strategy employed during conflict resolution relies on keeping track of modification timestamps for individual fields. This allows SQLiteData to reason about which values should win when the same record has been edited concurrently.

These timestamps are stored directly on the CKRecord. Each field is accompanied by a corresponding modification timestamp, which SQLiteData reads from and writes to using the encryptedValues[at: key] API. In addition, the record carries an overall userModificationTime that reflects the maximum of all per-field timestamps. This should not be confused with CKRecord.modificationDate, which contains the time that CloudKit persisted the record to the server and has no equivalent on the client.

A simplified snapshot of a record with per-field timestamps looks like this:

CKRecord(
  recordType: "reminders",
  title: "Buy milk",
  isCompleted: 1,
  sqlitedata_icloud_userModificationTime_title: 60,
  sqlitedata_icloud_userModificationTime_isCompleted: 30,
  sqlitedata_icloud_userModificationTime: 60
)

2. Sync Scenarios

This section walks through concrete sync scenarios to illustrate how SQLiteData behaves in practice and where assumptions start to break down.

2.1. Sending Client Change

This scenario describes a local change to a row that is sent to the server without any concurrent server-side edits.

  1. The record is fully in sync. The local database row and the last-known server record reflect the same server state.
  2. The user modifies the record locally.
    • The change is written to the local database.
    • SyncMetadata.userModificationTime is updated via a trigger.
    • In a trigger, the modification is recorded with the sync engine as pending for upload.
  3. The sync engine picks up the pending change in SyncEngine.nextRecordZoneChangeBatch(…) and prepares the record to send.
    • The CKRecord is created from SyncMetadata._lastKnownServerRecordAllFields. If no last-known server record exists, a fresh CKRecord is created instead.
    • The local row is then applied to this record using CKRecord.update(with:userModificationTime:), stamping each modified field with the current userModificationTime while leaving unchanged fields with their existing timestamps. Even if individual fields were changed at different times, only the latest timestamp is used, so all modified fields end up sharing a single modification time despite timestamps being tracked per field in records.
  4. Even before the upload is confirmed, SQLiteData attempts to persist the constructed CKRecord instance as the last-known server record by calling refreshLastKnownServerRecord(…). This update only succeeds if no previous last-known server record exists, i.e. when uploading a record for the first time. For previously synced records, the stored last-known server record and the constructed upload record share the same modificationDate, causing the refresh to be skipped.

Warning

While this behavior does not cause immediate issues in this scenario, it is conceptually incorrect to treat a pending upload record as a record known to the server. To support conflict resolution via a proper three-way merge, the last-known server record must reflect server-acknowledged state, not client intent.

This issue becomes visible in tests, where the mock server does not populate modificationDate when confirming a sent record. As a result, the last-known server record gets updated with an in-flight record, which then breaks when a conflict is encountered. But really, a much bigger problem is that the test setup diverges from production behavior.

  1. The record is sent to the server and the result is reported in SyncEngine.handleSentRecordZoneChanges(…).
    • Since there are no concurrent server-side changes in this scenario, the upload succeeds without conflict.
    • The confirmed server record is stored as the new last-known server record, now reflecting state that has been acknowledged by the server.
    • SyncMetadata.userModificationTime is updated from the server-confirmed record, but effectively retains the same value that was used when constructing the upload.
    • No changes are applied to the local database row, as it already reflects the desired state.

2.2. Fetching Server Change

This scenario describes a server-side change that is received and applied locally when there are no pending client-side modifications for the record.

  1. The record is fully in sync. The local database row and the last-known server record reflect the same server state.
  2. The record is modified on the server from another device and now contains updated field values, updated per-field modification timestamps, and an updated overall user modification timestamp.
  3. The updated record is delivered to the client and processed in SyncEngine.handleFetchedRecordZoneChanges(…) as a modification, which is routed to SyncEngine.upsertFromServerRecord(…) to apply the upsert logic.
    • A corresponding SyncMetadata row is ensured to exist, with a populated last-known server record.
    • The server record’s overall userModificationTime is combined with the locally stored value from SyncMetadata. Since the property setter applies the maximum of both values, the newer of the two timestamps is kept. In this scenario, the incoming server-provided timestamp is newer and is therefore retained.
    • The server record is reconciled with both the last-known server record and the current row fetched from the database in CKRecord.update(with:row:columnNames:parentForeignKey:). This method restores field values from the last-known server record when they (a) differ and (b) their timestamp is newer or equal to those in the incoming server record. It also narrows down the set of columns to be written by excluding columns whose values were restored (didSet flag) as well as columns with pending local edits (isRowValueModified flag). In this scenario, however, there are no local edits and the last-known server record does not contain any newer values, so no columns are excluded.
    • Next, the selected columns are updated on the local database row using values from the server record.
    • After this update, a trigger fires and sets SyncMetadata.userModificationTime to the current time.
    • The (potentially mutated) server record is then persisted as the new last-known server record.
    • Finally, SyncMetadata.userModificationTime is raised to the record’s overall userModificationTime if that value is newer. Since the trigger has just set it to the current time, it effectively keeps that value.

2.3. Merge Conflict on Send

This scenario describes a local change to a row that is sent to the server while a concurrent server-side edit exists, resulting in a .serverRecordChanged error.

  1. The record is fully in sync. The local database row and the last-known server record reflect the same server state.
  2. The record is modified on the server from another device and now contains updated field values, updated per-field modification timestamps, and an updated overall user modification timestamp.
  3. The user modifies the record locally, potentially editing different fields than the server change.
    • The change is written to the local database.
    • SyncMetadata.userModificationTime is updated via a trigger.
    • In a trigger, the modification is recorded with the sync engine as pending for upload.
  4. The sync engine picks up the pending change in SyncEngine.nextRecordZoneChangeBatch(…) and prepares the record to send.
    • The CKRecord is created from SyncMetadata._lastKnownServerRecordAllFields.
    • The local row is then applied to this record using CKRecord.update(with:userModificationTime:), stamping each modified field with the current userModificationTime while leaving unchanged fields with their existing timestamps.
    • refreshLastKnownServerRecord(…) is then invoked with the constructed record. In production, this call is skipped because the existing last-known server record has the same modificationDate. In tests, however, the check succeeds, causing the client-constructed upload record to be incorrectly persisted as the last-known server record (see 2.1).
  5. The result is reported in SyncEngine.handleSentRecordZoneChanges(…) where the send fails with a .serverRecordChanged error that includes the latest server record. This error is handled by routing the server record into SyncEngine.upsertFromServerRecord(…), which applies the same upsert logic as in section 2.2, but this time, it resolves the conflict between the server state and the pending client change.
    • The server record’s overall userModificationTime is combined with the client’s value from SyncMetadata, keeping the newer of the two timestamps.
    • The server record is reconciled with both the last-known server record and the current row fetched from the database in CKRecord.update(with:row:columnNames:parentForeignKey:).
    • Inside this method, for each field, two flags are computed:
      • didSet: Attempts to restore the last-known server record’s value and timestamp to the incoming server record. Returns true only if (a) the values differ and (b) the last-known server record’s timestamp for this field is greater than or equal to the server’s timestamp. In production with a clean last-known server record, this typically returns false because the last-known server record has older timestamps, so the incoming server’s value is kept.
      • isRowValueModified: Compares the current database row value to the last-known server record’s value, ignoring timestamps. Returns true if they differ, indicating a pending client edit exists for this field.
    • If either didSet or isRowValueModified is true, the field is excluded from the set of columns to be written into the database.
    • The marked columns are updated for the database row with values from the (potentially mutated) server record.
    • After this update, a trigger fires and sets SyncMetadata.userModificationTime to the current time.
    • The (potentially mutated) server record is saved as the new last-known server record, and SyncMetadata.userModificationTime is raised to the record’s overall userModificationTime if that value is newer.

Warning

In tests, the last-known server record can be polluted with client-constructed state before uploads are confirmed (see section 2.1), causing didSet equal to true to occur more frequently during conflict resolution.

It was observed that didSet being equal to true can occur in production when concurrent edits happen at the exact same timestamp. Because the comparison uses <= (less than or equal), equal timestamps cause the last-known server record’s value to win. This leads to non-deterministic outcomes depending on which device’s change was processed first.

Warning

It’s conceptually inappropriate to save a mutated server record as the last-known one acknowledged by the server. It’s optimistically counting on the server to accept on the next send, but another conflict might be encountered with no access to a clean last-known server record.

Unlike the noop last-known server record refresh on send described in section 2.1, this premature refresh happens in both production and test environments.

  1. After the conflict is resolved, the failed record is re-queued for upload.
  2. The sync engine picks up the re-queued record for upload in SyncEngine.nextRecordZoneChangeBatch(…) using the same logic as step 4. It constructs a CKRecord from the updated last-known server record and applies the local row to it.
    • A field is applied only when the row’s value differs from the record’s value and the record’s timestamp for that field is lower than or equal to the current SyncMetadata.userModificationTime, which then becomes the field’s new timestamp. Because the metadata timestamp was raised in step 5, this check passes even for fields whose server-side edit was newer than the client’s edit.

Warning

This scenario is subject to the same violation described in section 2.4. Through the value-based exclusion in step 5 and the raised metadata timestamp in step 7, the pending client edit wins over the server’s newer edit (see #354).

In tests, however, the outcome flips. The polluted last-known server record (see the first warning above) causes the pending client edit to no longer register as a modification, so the server value is written locally and wins instead. The test suite therefore does not reflect the production outcome of this scenario.

  1. The sync engine reports the result in SyncEngine.handleSentRecordZoneChanges(…). If the save succeeds, the sync engine updates the metadata by storing the reported record as the last-known server record.

2.4. Merge Conflict on Fetch

This scenario describes a server-side change that is received while a local change to the same record is still pending for upload. Unlike in the conflict-on-send scenario, no error signals the conflict. The fetched record is applied through the regular upsert logic, which resolves the conflict implicitly.

  1. The record is fully in sync. The local database row and the last-known server record reflect the same server state.
  2. The user modifies the record locally.
    • The change is written to the local database.
    • SyncMetadata.userModificationTime is updated via a trigger.
    • In a trigger, the modification is recorded with the sync engine as pending for upload.
  3. The record is modified on the server from another device and now contains updated field values, updated per-field modification timestamps, and an updated overall user modification timestamp.
  4. Before the pending local change is sent, the updated record is delivered to the client and processed in SyncEngine.handleFetchedRecordZoneChanges(…), which routes it to SyncEngine.upsertFromServerRecord(…). The same upsert logic as in sections 2.2 and 2.3 is applied, this time resolving the conflict between the incoming server state and the pending client change.
    • The server record’s overall userModificationTime is combined with the locally stored value from SyncMetadata. Since the property setter applies the maximum of both values, the newer of the two timestamps is kept.
    • The server record is reconciled with both the last-known server record and the current row fetched from the database in CKRecord.update(with:row:columnNames:parentForeignKey:), computing the didSet and isRowValueModified flags for each field as described in section 2.3.
    • For fields changed only on the server, both flags are false, so the server values are written to the local database row.
    • For fields with pending client edits, isRowValueModified is true, so they are excluded from the write and the local values are preserved. This exclusion is purely value-based, so the client value survives locally even when the server’s edit to the same field is newer (see the warning below).
    • After the row update, a trigger fires and sets SyncMetadata.userModificationTime to the current time.
    • The (potentially mutated) server record is persisted as the new last-known server record, and SyncMetadata.userModificationTime is raised to the record’s overall userModificationTime if that value is newer.
  5. The pending local change remains queued and is later picked up in SyncEngine.nextRecordZoneChangeBatch(…) using the same logic as in section 2.1. The record is constructed from the updated last-known server record and the local row is applied to it, stamping each differing field with the current SyncMetadata.userModificationTime.
    • A field value is only applied when its existing timestamp on the record is lower than or equal to the modification time being used. Because the metadata timestamp was raised in step 4, this check passes even for fields whose server-side edit was newer than the client’s edit.

Warning

When the client and the server edit the same field concurrently, the pending client edit always wins, even when the server’s edit has a newer timestamp (see #354). Locally, the field is excluded from the upsert based on value comparison alone: the isRowValueModified flag never consults per-field timestamps, even though its doc comment claims to detect a row value that was “modified more recently than the last known record”. On the subsequent upload, the client value passes the timestamp check because SyncMetadata.userModificationTime was previously raised to the server record’s newer overall timestamp. As a result, the client’s older edit is re-stamped with a newer modification time it never had and overwrites the server’s newer value on the server, violating the “field-wise last edit wins” strategy.

  1. The record is sent and the result is reported in SyncEngine.handleSentRecordZoneChanges(…). On success, the confirmed record is stored as the new last-known server record. If the server changed again in the meantime, the send fails with a .serverRecordChanged error and the flow continues as in section 2.3.

2.5. Reconciliation Conflict on Send

The previous conflict scenarios rely on the last-known server record as the baseline for a three-way merge. The remaining two scenarios describe conflicts without that baseline, where the same record exists on both the client and the server without ever having been synchronized, e.g. when both sides independently create a row with the same primary key. A similar state arises when the sync metadata is lost while the user data is kept, such as after signing out of and back into iCloud.4

In this scenario, the pending local change is sent before the server record is fetched.

  1. The user creates a row locally.
    • The row is written to the local database.
    • A SyncMetadata row is created via a trigger, with no last-known server record.
    • In a trigger, the creation is recorded with the sync engine as pending for upload.
  2. A record with the same primary key already exists on the server, created independently by another device.
  3. The sync engine picks up the pending change in SyncEngine.nextRecordZoneChangeBatch(…). With no last-known server record, a fresh CKRecord without a change tag is constructed from the local row, and the premature refresh described in section 2.1 persists it as the last-known server record.
  4. The send fails because the record already exists on the server. The failure surfaces as a .serverRecordChanged error, which is handled with the same conflict resolution as in section 2.3.
    • Since the premature refresh from step 3 installed the client-constructed upload record as the last-known server record, the client’s own record acts as the ancestor.
    • No field registers as a pending client edit, as the database row matches the ancestor.
    • The client’s values are written back onto the incoming server record whenever their timestamps are greater than or equal to the server’s. Otherwise, the server values are written to the local database row. The outcome therefore depends on the creation timestamps of the individual fields.
  5. The record is re-queued and sent again with the resolved state, as in section 2.3.

Warning

While this flow effectively performs a correct “field-wise last edit wins” reconciliation between the two independently created versions, with ties favoring the client, the behavior is fragile and limited to production. It arises accidentally from the premature refresh criticized in section 2.1 and applies only in this ordering, as the fetch-first counterpart in section 2.6 performs no resolution at all.

Warning

In the test environment, the mock server reports a .serverRejectedRequest error instead, interpreting the save of a record without a change tag as an attempt to create a new record identity. This error is handled by only clearing the last-known server record, undoing the premature refresh from step 3. No conflict resolution is performed and the record is not re-queued for upload, even though the error includes the latest server record. The later fetch then applies the wholesale overwrite described in section 2.6, so in tests the server always wins.

2.6. Reconciliation Conflict on Fetch

This scenario describes the counterpart to section 2.5, in which the server record is fetched before the pending local change is sent. The starting state is the same, with the client and the server holding independently created records with the same primary key and no last-known server record.

  1. The user creates a row locally, as in section 2.5.
  2. A record with the same primary key already exists on the server, created independently by another device.
  3. Before the pending local change is sent, the server record is delivered to the client and processed in SyncEngine.handleFetchedRecordZoneChanges(…), which routes it to SyncEngine.upsertFromServerRecord(…). Since no last-known server record exists, the reconciliation step is skipped entirely. No flags are computed, and all writable columns are written to the local database row, wholesale overwriting the locally created values pending upload. The server record is persisted as the new last-known server record.
  4. The upload registered in step 1 is still queued with the sync engine and is eventually processed, even though the local values it was meant to carry no longer exist. The record is constructed from the freshly stored last-known server record. Since the overwrite in step 3 left the local row with exactly the values of that record, no fields are applied, and the unchanged record is sent and accepted by the server without effect.

Warning

In this scenario, no conflict resolution takes place at all. The server values always win, and the locally created values are silently discarded without consulting timestamps, even when they are newer. This violates the “field-wise last edit wins” strategy and is the motivating case for a proper two-way reconciliation.

Footnotes

  1. This document was originally written for version 1.4 and later updated for version 1.8.1. Both versions behave the same in all documented scenarios, with one exception. Since #386, SyncMetadata.userModificationTime is raised via a maximum instead of being overwritten when storing a last-known server record. Because applying a server change to the row fires a trigger that sets the value to the current time, the metadata effectively keeps that current time, rather than being lowered to the record’s timestamp as before this PR. ↩

  2. The terms merge conflict and reconciliation conflict are used by the author of this document to distinguish the two situations. They are not established terminology in the SQLiteData codebase, but are intended to be carried over into the further development of conflict resolution. ↩

  3. In contrast to conflict-on-send, conflict-on-fetch scenarios are not explicitly signaled by CloudKit. Detecting such conflicts would require SQLiteData to infer them based on the presence of pending local changes for the record. ↩

  4. The two versions do not necessarily differ in content. When the values coincide, such as when neither side has changed since the original write, the conflict still arises structurally, but resolving it leaves the row values unchanged and only updates the sync metadata. ↩