Regression Testing iOS Keychain Migrations on a Cloud Mac

A routine iOS update can suddenly make a login token appear to have vanished. The new version may not crash, and the API may still work normally. The real change is often hidden in the Keychain item’s service, account, access group, or protection attributes. To detect these issues reliably on a cloud Mac, testing must begin with “write using the old version, read using the new version,” rather than covering only clean installations.

Define the Migration Contract First

Treat Keychain items as data structures that require backward compatibility. Record at least five attributes: the item class, service, account, access group, and kSecAttrAccessible. If the new version renames anything, do not simply replace the query criteria. Read the item using the old criteria first, write it using the new structure, confirm that the write succeeded, and only then delete the old item.

Assign an integer version to each migration, such as keychainSchemaVersion = 2, and store the completion marker in regular preferences. This marker exists only to prevent the migration from running repeatedly; it does not prove that the Keychain data still exists. The app should continue checking for the target item at startup so it can handle cases where the marker exists but the data has been cleared.

A migration is successful not merely when its function returns success, but when the new version can read the old data, repeated execution does not corrupt the data, and the original recoverable item remains intact after a failure.

Start with this minimum acceptance matrix:

Build Old- and New-Version Test Fixtures

Keep two installable simulator .app bundles: the old version seeds the data, while the new version performs and verifies the migration. Do not run erase or uninstall the app between the two builds, because doing so changes the test preconditions. Add launch arguments that are enabled only in automation builds, such as -UITestSeedLegacyKeychain and -UITestVerifyKeychainV2. Production builds must not respond to these arguments.

set -euo pipefail

UDID="${SIMULATOR_UDID:?missing simulator id}"
OLD_APP="${OLD_APP_PATH:?missing old app path}"
NEW_APP="${NEW_APP_PATH:?missing new app path}"
BUNDLE_ID="com.example.product"

xcrun simctl bootstatus "$UDID" -b
xcrun simctl install "$UDID" "$OLD_APP"
xcrun simctl launch --terminate-running-process \
  "$UDID" "$BUNDLE_ID" -UITestSeedLegacyKeychain

xcrun simctl install "$UDID" "$NEW_APP"
xcrun simctl launch --terminate-running-process \
  "$UDID" "$BUNDLE_ID" -UITestVerifyKeychainV2

The seeding operation must be verifiable. After the old version writes the item, the test UI can produce a status file that contains no sensitive values and records only the item version, write result, and a random test-case identifier. The new version should write a separate result after verification. CI compares only these status files and never prints the token contents.

Prevent Cross-Test Contamination from Concurrency

Do not run multiple groups of Keychain tests concurrently on the same simulator. Create a dedicated simulator for each job, or run the test suites serially. Add a random test-case suffix to service and account so that data left behind by a previously interrupted run cannot cause the next test to report a false success.

Preserve Diagnostic Results in Code

Do not convert every failure into nil. Preserve the OSStatus value so that not-found, permission, and invalid-parameter errors remain distinguishable. A query function can return an explicit result type:

enum KeychainReadResult {
    case found(Data)
    case missing
    case failed(OSStatus)
}

func readLegacyToken(service: String, account: String) -> KeychainReadResult {
    let query: [CFString: Any] = [
        kSecClass: kSecClassGenericPassword,
        kSecAttrService: service,
        kSecAttrAccount: account,
        kSecReturnData: true,
        kSecMatchLimit: kSecMatchLimitOne
    ]

    var item: CFTypeRef?
    let status = SecItemCopyMatching(query as CFDictionary, &item)

    if status == errSecItemNotFound {
        return .missing
    }
    guard status == errSecSuccess, let data = item as? Data else {
        return .failed(status)
    }
    return .found(data)
}

During migration, add the new-version item first, read it back and compare a digest of its contents, and delete the old item only afterward. Never delete before writing. If the write fails because of permissions or invalid parameters, the user’s data would become unrecoverable. Logs should contain only the status code, migration version, and test-case identifier—not the original credentials.

Verify Signing and Access Groups

If the code has not changed but queries suddenly return errSecItemNotFound, compare the entitlements of the old and new artifacts. A change to the access-group prefix, application identifier, or Keychain access groups can make the original item invisible to the new version.

codesign -d --entitlements :- "$OLD_APP_PATH" > old-entitlements.plist
codesign -d --entitlements :- "$NEW_APP_PATH" > new-entitlements.plist
diff -u old-entitlements.plist new-entitlements.plist

Save the diff as a CI artifact, but do not copy signing material into the build logs. If the access group genuinely needs to change, create a transitional version that can access both the old and new groups and use it to move the item. Do not assume that the new version can directly read an arbitrary old group.

The simulator is suitable for testing migration logic and error branches, but it cannot reproduce every signing and access-control behavior. Before release, run the same old-version installation, data seeding, and in-place upgrade flow once on a controlled physical device. Record those results separately from the simulator tests.

Cleanup, Failure Forensics, and Release Gates

After a successful test, use an in-app testing interface to delete the test item by its exact service and account. Do not issue a broad deletion query, as it could remove unrelated data from the same test device. For failed tests, preserve the simulator first and collect the new- and old-version entitlements, OSStatus, migration steps, and application logs before deciding whether to destroy it.

The release gate should verify three conditions: the new version can read old-version data; running the migration twice in succession produces the same result; and malformed input does not delete the old item. Failure of any condition must prevent the artifact from advancing to the next distribution stage.

Finally, keep the migration script’s inputs fixed: the old build identifier, new build identifier, simulator OS version, and test-case identifier. This makes it possible to reconstruct the same upgrade path from archived artifacts even after the cloud Mac has been reinitialized. The hardest part of a Keychain issue is not writing data but reproducing the old state. Once the old-version fixture, entitlement differences, and error codes are reproducible, migration risk in production becomes an ordinary regression test.

Frequently asked questions