Audit XCTestPlan Drift on a Cloud Mac

When the same commit runs the full test suite on a developer’s machine but executes only part of it on a cloud Mac, the test code is rarely the most likely cause. More often, the .xctestplan has drifted unnoticed: someone temporarily skipped tests in Xcode, disabled diagnostics, or saved machine-specific environment variables in the plan. The solution is to treat XCTestPlan as a build input rather than an auxiliary GUI setting.

Define the test contract boundary first

A reproducible test entry point must include, at minimum, a shared Scheme, an XCTestPlan, an execution destination, and command-line arguments. First, verify that the Scheme and test plan are visible from the command line:

WORKSPACE="${WORKSPACE:-App.xcworkspace}"
SCHEME="${SCHEME:-App}"

xcodebuild \
  -workspace "$WORKSPACE" \
  -scheme "$SCHEME" \
  -showTestPlans

If the expected plan does not appear in the output, check whether the Scheme is marked as Shared, whether the plan file is committed to the repository, and whether the Scheme’s Test action references it. Do not use CI arguments to substitute a plan that does not exist. That only allows the local entry point and the pipeline to diverge further.

The following items should be reviewed explicitly in the plan:

Skipping a test is not a fix. Every newly skipped test should be reviewed like a code change and include the conditions for restoring it.

Generate a readable normalized diff

XCTestPlan is a structured file. Reviewing its raw text directly can be noisy because of field ordering and automatically generated identifiers. Keep a normalized baseline in the repository that removes only configuration identifiers that do not affect execution semantics:

PLAN="App.xctestplan"
CURRENT=".ci/xctestplan.current.json"
BASELINE=".ci/xctestplan.baseline.json"

mkdir -p .ci

plutil -convert json -o - "$PLAN" |
  jq -S 'del(.configurations[]?.id)' > "$CURRENT"

diff -u "$BASELINE" "$CURRENT"

During initial setup, review $CURRENT manually, then copy it to the baseline and commit it. Subsequent pipeline runs should generate only the current file and run diff. Targets, skipped tests, environment variables, arguments, and diagnostic options must all remain intact. Do not remove business-relevant fields merely to produce a “clean diff.”

The normalization script is also part of the test infrastructure. Changes to the script and the plan should appear in the same merge request. Otherwise, a single relaxation of the filtering rules could make all subsequent drift invisible.

Block sensitive values and host dependencies

A plan file is suitable for storing variable names, but not tokens, passwords, private-key contents, or developer-specific directories. Start by recursively extracting enabled environment variables:

plutil -convert json -o - App.xctestplan |
  jq -r '
    .. |
    objects |
    .environmentVariableEntries? // empty |
    .[]? |
    select(.enabled == true) |
    [.key, .value] |
    @tsv
  '

When reviewing the output, block three categories in particular: long strings that resemble secrets, absolute paths in the form /Users/someone/, and tool paths available only in an interactive Shell. Actual sensitive values should be injected at runtime through the controlled CI environment. Test code should read only the variable names and report a clear error when a required value is missing.

Keep override precedence under control

XCTestPlan, Scheme, xcodebuild arguments, and test code can all define launch arguments. Establish a single precedence policy: the plan stores stable defaults, CI injects only sensitive values and identifiers for the current run, and test code does not modify process-level configuration. If the pipeline uses -only-testing or -skip-testing, place those arguments in a reviewable script rather than hiding them in a temporary input field in the job configuration panel.

Run against a fixed destination and preserve evidence

Use xcrun simctl list devices available to select a device installed on the current node, then pass its UDID into the pipeline. Compared with vaguely requesting the “latest OS,” pinning the device and runtime makes discrepancies easier to explain.

DEVICE_UDID="${DEVICE_UDID:?Set DEVICE_UDID from simctl}"
RESULT_PATH="${RESULT_PATH:-artifacts/CI.xcresult}"

rm -rf "$RESULT_PATH"
mkdir -p "$(dirname "$RESULT_PATH")"

xcodebuild test \
  -workspace "$WORKSPACE" \
  -scheme "$SCHEME" \
  -testPlan CI \
  -destination "platform=iOS Simulator,id=$DEVICE_UDID" \
  -resultBundlePath "$RESULT_PATH"

Whether the run succeeds or fails, preserve the .xcresult, the full command, the commit identifier, the Xcode version, and the selected device UDID. Do not retain only the last few dozen log lines. Tests that never ran, process crashes, and assertion failures require different evidence, while the result bundle preserves the test hierarchy, attachments, and diagnostic information.

Start parallel execution with conservative settings. Test targets that depend on the same database, a fixed port, or shared files should not be parallelized immediately. Fully isolate their state first, then enable parallelism one target at a time instead of using retries to conceal race conditions.

Turn the audit into a merge gate

The final gate should run in a fixed order: verify that the Scheme can discover the plan, generate the normalized file, compare it with the baseline, scan for sensitive values and absolute paths, validate the skipped-test list, and only then execute the tests. This allows configuration errors to fail before a simulator starts, reducing queue and diagnostic time.

Use the following checklist before merging:

• The Scheme is shared, and the plan file is under version control.
• New test targets have been added to the correct plan.
• Every skipped test has a reason, an owner, and conditions for restoration.
• The plan contains no actual sensitive values or personal directories.
• CI has no hidden arguments that override the test scope.
• The device UDID, Xcode version, and result bundle are all recorded.
• Normalization rules do not remove fields that affect execution semantics.
• Failed runs still archive the .xcresult.

When every XCTestPlan change is understandable during code review, “everything passes locally, but the cloud skipped tests” stops being an intermittent mystery. It becomes a configuration difference that can be blocked before execution.

Frequently asked questions