RFC 0001 — End-to-end testing strategy

Status: draft · Date: 2026-07-28 · Author: Matt · Decision needed by: before next release cut

Context

The app has grown enough flows — auth, quote list, draft create/edit, offline draft + sync, send with token link, web accept — that manual regression testing no longer covers them. Recent offline work (ADR 0001) touched draft persistence and snapshot sync, exactly the kind of change where a silent regression in an untouched flow is easy to ship.

We have no automated end-to-end coverage today. Unit/typecheck (bun run check) catches type and lint breakage but nothing about behaviour — a broken send button, a draft that fails to persist offline, an accept link that 500s.

Two surfaces need driving:

• /app — Expo SDK 57 / React Native 0.86 tradesperson app. The bulk of the product and the bulk of the flows. Primary priority.
• /web — Bun.serve + React 19 client page, /q/:token accept flow. Smaller surface, but it’s the customer-facing half of the vertical slice and regression here is invisible to us until a customer hits it.

Neither has a natural home in a JS unit runner — both are real user journeys crossing the network boundary into Supabase.

Goals

1. Catch cross-flow regressions before release, not in the field.
2. Readable tests a solo team can maintain — low ceremony, no native-build babysitting.
3. Run locally and in CI against an ephemeral seeded Supabase (reuse the supabase-ci.yaml pattern).
4. Deterministic. Seeded data, stubbed external side-effects, no flaky sleeps.

Non-goals

• Full cross-boundary orchestration (app sends → web accepts → push fires) in one test. Deferred — push Lambda isn’t built (/api unscaffolded).
• Visual regression / screenshot diffing. Later, if needed.
• Load / performance testing. Out of scope.

Decision

Adopt two complementary frameworks, one per surface:

Surface	Framework	Why
/app (Expo/RN)	Maestro	Built for RN and supported by Expo’s E2E tooling. YAML flows, accessibility-layer selectors, and automatic waiting. Start on iOS Simulator with a cached Debug build and Metro locally; retain standalone Release builds for CI/parity.
/web (React)	Playwright	Best-in-class browser e2e. Real RPC coverage for the core journey, optional network interception for frontend-only error states, and trace viewer for debugging regressions.

Mobile is the priority build and lands first. Web follows once the app suite has a foothold.

Why Maestro over Detox (app)

Detox is the older RN incumbent. Rejected:

• Requires a native debug build wired into the test harness — heavy setup, breaks on Expo config changes.
• Brittle synchronisation model; slower to keep green.
• Maestro selectors are text/testID with built-in automatic waiting — no manual waitFor chains. Stable testIDs are the default for controls; visible text is reserved for user-facing assertions.

Maestro flow shape:

appId: com.sawdust.app
---
- launchApp:
    clearState: true
    clearKeychain: true
- tapOn:
    id: "e2e.sign-in.email"
- inputText: "rhys@example.com"
- tapOn:
    id: "e2e.sign-in.password"
- inputText: "password123"
- tapOn:
    id: "e2e.sign-in.submit"
- assertVisible: "Owed to you"
- tapOn:
    id: "e2e.home.quotes"
- tapOn:
    id: "e2e.quotes.new"
- tapOn:
    id: "e2e.new-quote.new-client"
- tapOn:
    id: "e2e.new-quote.client-name"
- inputText: "Jane Smith"
- tapOn:
    id: "e2e.new-quote.client-email"
- inputText: "jane@example.com"
- tapOn:
    id: "e2e.new-quote.save"
- assertVisible: "Status: draft"

Trade-off accepted: Maestro is a separate Homebrew-installed binary (not an npm dep) and its cloud runner is paid. Local CLI execution is free. The CI runner shape remains open until the local iOS flow is proven.

Why Playwright over Cypress (web)

Playwright wins on real-browser RPC coverage, optional network interception for frontend-only failure states, true multi-browser support, and the trace viewer. Cypress’s in-browser model makes cross-origin and network stubbing more awkward. No strong reason to prefer Cypress here.

Test architecture

Both suites hit a local, seeded Supabase (supabase start), not a shared/remote instance. Reset through bun --filter='@sawdust/db' run reset, which recreates login-capable users through seed-auth.sh before loading seed.sql. The existing seed already gives us:

• User rhys@example.com / password123
• Quote token aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

Determinism rules:

• Seed is the fixture. Every run starts from the repository reset script. Mutating web tests use one worker initially and must move to unique per-test tokens before enabling retries, parallelism, or multiple browser projects.
• Device state is a fixture too. Maestro clears app state and the iOS Keychain before the flow so SecureStore, SQLite, the draft outbox, and auth cannot leak between runs. General-purpose E2E builds bypass the biometric/device-PIN lock with EXPO_PUBLIC_E2E=1; lock-screen behaviour gets a dedicated flow with that bypass disabled.
• External side-effects stubbed. Stripe, Expo push, Claude vision — never hit real services in e2e. Assert the DB state or the intercepted request instead.
• Analytics disabled. The E2E app build sets EXPO_PUBLIC_ANALYTICS_DISABLED=1.
• No wall-clock sleeps. Maestro auto-waits; Playwright uses web-first assertions.

iOS offline strategy

iOS Simulator does not expose a controllable airplane mode to Maestro. The E2E build therefore keeps the host network connected and enables the existing Dev tools Simulate offline flag with EXPO_PUBLIC_E2E=1. That app-level state is authoritative for draft creation, pauses online query behaviour, and drains queued work when switched back online.

The fast local loop uses an unsigned Debug simulator build with Metro (expo run:ios --configuration Debug --no-bundler). The native shell is built once; subsequent JavaScript changes are served by Metro without recompiling the React Native dependency graph. A standalone Release build (expo run:ios --configuration Release) remains the production-parity and CI target. Both can safely clear app state, and neither requires an Apple certificate. Expo SDK 57 requires Xcode 26.4 or newer.

CI shape

Reuse the migration/reset concepts from supabase-ci.yaml, but run the full Supabase stack — the existing workflow’s vanilla Postgres plus auth stubs cannot serve GoTrue/PostgREST traffic:

1. Spin up full local Supabase, apply migrations, create auth users, and load seed.
2. Web job: build /web, run Playwright headless against it. Fast — runs on every PR.
3. App job: boot iOS Simulator, build/install the E2E Release app, run Maestro flows, and retain failure artifacts. Slower/heavier — gate behind label or nightly initially to keep PR feedback fast, promote to per-PR once stable.

The app job cannot run on Ubuntu because Xcode and iOS Simulator require macOS. GitHub-hosted Apple-silicon macOS runners can provide Xcode and the simulator, but do not support the nested virtualization required to run the Docker-backed local Supabase fixture. The initial checked-in workflow is therefore manual and targets a self-hosted Apple-silicon macOS runner with Docker Desktop. A future hosted-runner topology must use a network-reachable test Supabase project (or Maestro Cloud with the same remote fixture).

Expected wall time is 10–16 minutes from a cold runner and 4–7 minutes with warm Xcode caches: one Release build, one Supabase reset, then all Maestro flows against that installed app.

Rollout plan

Phased, iOS-first per priority:

1. Phase 0 — app testability + harness. Wire local seeded Supabase, add stable selectors, add the unsigned iOS E2E build, and document commands.
2. Phase 1 — app foundation. Sign-up; authenticated navigation/sign-out smoke coverage; create a server-backed quote draft; enable simulated offline → create local draft → reconnect → assert server sync. Covers both normal usage and the highest-risk ADR 0001 surface.
3. Phase 2 — app send. Auth → quote list → create server draft → send quote → token generated → status sent.
4. Phase 3 — web (Playwright). One serial Chromium suite using real RPCs: /q/:token load → accept → visible confirmation → assert quote flips to accepted and its draft invoice is created.
5. Phase 4 — CI promotion. Move app suite from nightly/label to per-PR once green and fast enough.
6. Later — cross-boundary. Once /api push Lambda exists, a single orchestrated journey (app send → web accept → push assert).

Open questions

• Hosted CI topology after the initial self-hosted workflow: dedicated remote Supabase on GitHub-hosted macos-26 versus Maestro Cloud with the same remote fixture.
• EAS project linkage: local runs use expo run:ios; the checked-in EAS e2e profile is for later CI use once the repository is linked to an Expo project.
• Where do e2e specs live — per-package (projects/app/e2e, projects/web/e2e) or a top-level projects/e2e? Lean per-package to keep ownership local.
• Seed drift: as schema grows, seed must stay representative. Owner of that upkeep?

Alternatives considered

• Detox (app) — rejected, see above.
• Cypress (web) — rejected, see above.
• Appium (app) — general-purpose mobile automation, far heavier than Maestro for RN, no Expo affinity. Rejected.
• Do nothing / manual QA — status quo. Rejected; the flow count already exceeds what manual regression covers reliably.