Device Farms in CI

A device farm (BrowserStack, Sauce Labs, AWS Device Farm, or your own rack) is not a new testing model — it is your existing Appium server, hosted somewhere with a rack of phones behind it. Because Appium speaks W3C WebDriver over HTTP, the migration is an endpoint change plus capabilities, and suites that were written with the discipline from Mobile Automation with Appium — gestures, network ladders, and offline flows from Gestures, Network, and Offline Testing — move over untouched:

const driver = await remote({
  hostname: "hub-cloud.browserstack.com",
  port: 443,
  protocol: "https",
  capabilities: {
    platformName: "Android",
    "appium:automationName": "UiAutomator2",
    "appium:deviceName": "Google Pixel 7",
    "appium:platformVersion": "14.0",
    "appium:app": "bs://a1b2c3d4e5f6",          // uploaded once, referenced by id
    "appium:autoGrantPermissions": true,
    "bstack:options": {
      projectName: "shop-android",
      buildName: "2.9.4+30014",                  // ties the session to your commit
      sessionName: "checkout-smoke::pixel7::shard1",
      debug: true,                                // enhanced logs + video
    },
  },
});

The vendor-specific options block (bstack:options, sauce:options, testobject:options) carries metadata that does not belong in the spec — build, session name, video toggles — and it is what makes sessions findable in the vendor's dashboard later. What genuinely changes compared to a local emulator:

Three operational rules keep the bill sane. Upload the app once per build and reuse the storage id across shards — re-uploading per session multiplies transfer time and breaks the link between a failure and the build that caused it:

curl -u "$BS_USER:$BS_KEY" -X POST "https://api-cloud.browserstack.com/app-upload" \
  -F "file=@app-release.apk" -F "custom_id=build-30014"
# → { "app_url": "bs://a1b2c3d4e5f6", "custom_id": "build-30014" }

Set maxDuration and idleTimeout (both vendors enforce them) so a hung test is killed rather than billed for an hour; a suite that occasionally stalls is exactly how mobile bills double overnight. Label sessions with build + shard + device so a red run can be joined to a commit without opening fifty tabs — session metadata is the join key between your CI and their dashboard, and it is what makes the flakiness analysis in the third chapter possible at all.

One selection decision comes before any of that: virtual versus real devices on the farm. Vendors sell emulated/synthetic devices (cheaper, always available, good enough for logic and layout) alongside physical handsets (required for sensors, OEM battery managers, camera, and radio behavior). Route by risk — synthetic for the broad nightly matrix, physical for your top models and anything touching hardware — because real-device minutes are the line item that grows fastest, and a suite that runs every test on physical flagships spends money answering questions an emulator already answered.

The number that matters is agreed, not hoped: pull-request feedback arrives in under 10 minutes; the nightly matrix finishes in 45. A 120-test mobile suite at a realistic 40 seconds per test is 80 minutes serial — nine times the PR budget — so sharding is not an optimization, it is the precondition for running mobile tests in CI at all.

Start with the arithmetic: serial_time / (devices × tests_per_device) ≈ parallel_time, then discount honestly for session setup. A farm session costs 20–40 seconds of boot before the first test, so shards should carry enough tests that setup is a minority of the shard's runtime — 10 tests per shard at 40 s each gives ~7 minutes of tests plus ~30 s of boot, while 2 tests per shard spends a quarter of its life booting.

# .github/workflows/mobile-smoke.yml
jobs:
  smoke:
    strategy:
      fail-fast: false          # one red shard must not cancel the others
      matrix:
        device: [pixel7-api34, s22-api33, iphone15-ios17]
        shard: [0, 1, 2]        # 3 devices x 3 shards = 9 concurrent sessions
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx wdio run ./config/wdio.conf.js
        env:
          DEVICE: ${{ matrix.device }}
          SHARD_INDEX: ${{ matrix.shard }}
          SHARD_TOTAL: 3

Two sharding strategies, with different failure modes. Static sharding (split a manifest at CI time: shard = index % total) is trivial and deterministic — but a shard that draws all the slow login-and-payment tests runs twice as long as its neighbours, and the slowest shard defines your feedback time. Dynamic sharding (each worker pulls the next test from a queue — what vendor parallel runners provide) self-balances, at the cost of harder per-test reporting and a scheduling layer to maintain. The static version costs three lines:

const manifest = JSON.parse(readFileSync("suite-manifest.json", "utf8"));
const total = Number(process.env.SHARD_TOTAL ?? 1);
const index = Number(process.env.SHARD_INDEX ?? 0);
const mine = manifest.filter((_, i) => i % total === index);   // round-robin, deterministic
runSuite(mine);                                                 // or --spec=${mine.join(",")}

Round-robin beats "first third, middle third": a manifest grouped by feature parks all the slow payment tests in one shard, while interleaving spreads them — and if you sort the manifest by expected duration before interleaving, shard runtimes converge to within a minute of each other.

A third option deserves consideration for the PR lane: shard by risk, not by count — a 6-test critical-path smoke on one device finishes in ~4 minutes with zero coordination overhead, while the full suite shards across nine sessions nightly. The most common mistake is sharding the whole suite on every PR; the budget is met more cheaply by choosing a smaller suite than by adding workers to a large one.

Whatever the strategy, make the budget a measured value rather than an assumption: publish duration per shard, alert when p95 exceeds the agreed number, and treat session boot time and queue wait as part of it — teams that measure only test runtime are surprised when a plan's concurrency limit quietly adds six minutes of queueing to every run. Failures, once they happen, are invisible without artifacts; that is the next chapter's job.

A cloud device failed at 03:12 on a phone you will never touch. If the session hands you only AssertionError: element not found, the next step is a local reproduction attempt that will not reproduce — so the pipeline must carry the evidence out of the farm and into your report, automatically.

Every major farm produces the same artifact set per session: a video of the whole run, the device log (logcat / iOS console), the Appium server log with each command and its response, and sometimes a network waterfall. The discipline is twofold — capture selectively and attach by URL:

// teardown: fetch artifacts only for failed sessions (video costs storage on every session)
if (failed) {
  const session = await farm.getSession(driver.sessionId);   // videoUrl, deviceLogUrl, appiumLogUrl
  allure.attachment("Session video", session.videoUrl, "url");
  allure.attachment("Device log", await session.deviceLog(), "text/plain");
}
await driver.deleteSession();

Store links rather than blobs in the report: a 90-second MP4 inlined into CI artifacts bloats every download, while a vendor URL with session-token expiry stays one click away in the failing test's entry — the exact wiring Test Reporting and Dashboards formalizes. Retention is the cost lever attached to this: keep videos for failed sessions for 14 days and turn recording off for passed sessions, because storage and bandwidth on a farm are metered like everything else, and nobody has ever re-watched a green run.

Join the two systems deliberately instead of eyeballing dashboards: the vendor's session API lists every session for a build, your pipeline already knows pass/fail and duration, and appending deviceModel, osVersion, and sessionStatus to the same results file means per-model rates come from one GROUP BY query rather than from fifty browser tabs. Read them in order: video answers what the user saw (usually: a spinner, a dialog, a half-rendered list), device log answers what the OS did (ANR, low-memory kill, OEM battery manager stopping your service), Appium log answers what the automation did (a stale element, a coordinate that missed, a 40-second command timeout). That order resolves most failures in under two minutes.

Then the analytical piece, and the reason session metadata matters: compute failure rate per device model.

The middle row is the one that corrupts metrics: session-level infrastructure errors are not test failures, and counting them as such trains everyone to distrust the dashboard. Tag them (session_status: infra_error), retry the session once, and report them separately. On the flaky side, keep a per-model quarantine with an owner and an expiry — a test known broken on Galaxy A13 runs everywhere else while a ticket ages — because the alternative, deleting the test or globally skipping it, quietly removes coverage from every device. With per-model rates in hand, "mobile CI is flaky" decomposes into a short, actionable list: which test, on which hardware, failing for which of the five reasons above. Wiring those results into the pull-request pipeline itself is the subject of Integrating Tests into CI.