Set up full test suite in CI and Xcode Cloud (#447)

* Set up full test suite in CI and Xcode Cloud

Add build-ghosttykit.yml workflow to pre-build and publish
GhosttyKit.xcframework as a GitHub release on manaflow-ai/ghostty,
keyed by submodule SHA. Add ci_scripts/ci_post_clone.sh for Xcode
Cloud to download the pre-built xcframework with retry logic. Create
cmux-ci scheme that runs both cmuxTests and cmuxUITests. Switch the
CI tests job from running a single UI test class to the full suite.

* Run unit tests + single UI test class on self-hosted runner

The self-hosted runner can't launch the full app for UI tests (no GUI
session), so run all unit tests via cmux-unit scheme and keep the
original UpdatePillUITests as a smoke test. Full UI test suite runs
on Xcode Cloud which has proper macOS GUI support.

* Handle expected test failures in unit tests step

xcodebuild returns exit code 65 even for expected failures
(XCTExpectFailure). Parse the summary line to only fail the CI job
when there are unexpected failures.
This commit is contained in:
Lawrence Chen 2026-02-27 04:18:26 -08:00 committed by GitHub
parent 8968f787ca
commit ae3bcd7eea
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
5 changed files with 211 additions and 8 deletions

View file

@ -37,7 +37,7 @@ jobs:
- name: Typecheck
run: bun tsc --noEmit
ui-tests:
tests:
# Never run self-hosted jobs for fork pull requests.
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
runs-on: self-hosted
@ -93,8 +93,28 @@ jobs:
# Remove stale build cache to avoid incremental build errors
rm -rf ~/Library/Developer/Xcode/DerivedData/GhosttyTabs-*
- name: Run unit tests
run: |
set -euo pipefail
# xcodebuild exits 65 even for expected failures (XCTExpectFailure).
# Capture output and fail only if there are unexpected failures.
set +e
OUTPUT=$(xcodebuild -project GhosttyTabs.xcodeproj -scheme cmux-unit -configuration Debug \
-destination "platform=macOS" test 2>&1)
EXIT_CODE=$?
set -e
echo "$OUTPUT"
if [ "$EXIT_CODE" -ne 0 ]; then
SUMMARY=$(echo "$OUTPUT" | grep "Executed.*tests.*with.*failures" | tail -1)
if echo "$SUMMARY" | grep -q "(0 unexpected)"; then
echo "All failures are expected, treating as pass"
else
echo "Unexpected test failures detected"
exit 1
fi
fi
- name: Run UI tests
run: |
set -euo pipefail
# Run directly on the self-hosted macOS runner.
xcodebuild -project GhosttyTabs.xcodeproj -scheme cmux -configuration Debug -destination "platform=macOS" -only-testing:cmuxUITests/UpdatePillUITests test