Mobile Testing Approaches

A checkout crash shows up on a two-year-old Samsung running Android 12, reproduces on no emulator in the office, and disappears on the flagship everyone tests with. The instinct is "test on more real devices"; the better habit is to ask, for each property under test, what is the cheapest environment that can observe it at all? Because a wrong answer in either direction is expensive: five physical phones running your full suite will not finish before lunch, and a simulator will happily certify battery life it does not have.

Three environments, three different illusions:

The iOS distinction is the one that trips teams: an iOS simulator runs the same kernel and libraries as the device, so it boots in seconds and gives correct behavior — but it has no modem, no battery controller, and no thermal envelope, which is why a "100% passing on simulator" suite can ship an app that overheats. The Android emulator is closer to real (it runs an actual system image and, on x86 hosts, translates ARM libraries), yet it still reports a synthetic battery that never drains and a radio that never drops to one bar.

So split the suite by what it claims to prove. Functional flows run headless in the emulator on every pull request — cheap enough to parallelize, and cheap enough that nobody hesitates before running them:

avdmanager create avd -n ci_pixel7 -k "system-images;android-34;google_apis;x86_64"
emulator -avd ci_pixel7 -no-window -no-audio -gpu swiftshader_indirect &
adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 2; done'
adb install -r app-debug.apk

A cold boot lands around 45–90 seconds; the same suite on shared physical hardware costs minutes per device plus queueing, and a shared farm slot is billed while your tests wait for the previous job. Everything the environment fakes runs on real hardware on a schedule: battery drain over a 30-minute playback, thermal throttling during a 10-minute map session, GPS movement, camera permissions in daylight, background sync surviving Doze — nightly or weekly, because that is the budget they need. Reserve owned devices for the paths that hurt most: your top three models by install count, running a 15-minute smoke of login, purchase, and notification receipt. The rule of thumb interviewers accept: emulators for breadth of logic, real devices for truth about hardware and the OS vendor's power management. That split is also what makes automation worth writing at all, which is Mobile Automation with Appium.

Most mobile regressions are not in your screens — they are at the edges of the app's life: the install, the upgrade, the permission dialog, and the moment something interrupts. A suite that launches the app and taps through three flows proves none of it, which is why these four groups get named, scheduled, and automated first.

Clean install. Fresh install, first launch, no account, no cached state: onboarding must complete, and the app must not crash when the database is created for the first time. Upgrade in place is the higher-risk sibling and the one teams skip:

adb install -r old-2.9.4.apk     # install the previous release, run its onboarding
# create data: log in, save a draft, change a setting
adb install -r new-3.0.0.apk     # -r keeps user data — this is a real upgrade, not a reinstall
# assert: still logged in, draft intact, settings preserved, no crash on first launch

If the new build migrates the local database, this is the only test that will notice the migration fails — and users who lose their drafts on update do not file bugs, they leave one-star reviews. Test upgrade from every still-supported release, not just the latest one, because a user on 2.7 who jumps straight to 3.0 exercises a different migration path than one on 2.9.

Permissions. Four states, four tests: granted, denied, denied-permanently ("don't ask again" — the app must handle the dead end and route to settings), and first-ask. Drive them without touching the screen:

adb shell pm grant com.acme.shop android.permission.CAMERA
adb shell pm revoke com.acme.shop android.permission.CAMERA
adb shell cmd appops set com.acme.shop CAMERA deny

The classic defect is assuming the grant: the camera screen calls the camera API unguarded and crashes the first time a user says no. Denial must degrade to an explanation, not an exception.

Interruptions. An app on a phone lives in the worst environment in computing — it gets killed, paused, and talked over. Name the scenarios and script them:

Each row is a contract with the user: what the app promises to preserve when the platform interrupts it. Notice that half of these are exactly the scenarios automation later has to drive — which is the bridge to the next topic. Finally, none of this is reportable without evidence; that is the third chapter.

"Test on all devices" is not a strategy; there are tens of thousands of Android models. The product's support matrix is a small, argued document that answers which combinations get tested, and it is derived from exactly three inputs: the declared minimum OS, the newest OS the week it ships, and where your users actually are — the same compatibility reasoning as API Versioning and Compatibility, applied to hardware and OS builds instead of endpoints.

The reasoning is worth stating in an interview: the minimum OS catches API misuse the newest SDK forgives, and the newest OS catches platform deprecations the minimum never sees. Neither substitutes for the other, and a matrix containing only flagships on the latest OS tests the environment your developers already own. Pull real numbers into the decision — store console telemetry or analytics showing OS-version and model distribution — so "why do we still test Android 12?" is answered with "18% of sessions", not a preference.

Against that matrix, the bug report is a contractual artifact. The one rule: a mobile finding without device model, OS build, and app version is not a finding — it is a rumor, and it will bounce back to triage within a day.

Device:    Samsung Galaxy A13 (SM-A135F)   ← not "my Samsung"
OS:        Android 12, One UI 4.1, build SP1A.210812.016
App:       3.0.0 (build 30014), installed via APK upgrade from 2.9.4
Network:   Wi-Fi → Wi-Fi off (airplane mode) during upload
Observed:  upload spinner never resolves; app killed after 30s, no error shown
Expected:  offline queue message, retried on reconnect
Evidence:  logcat.txt, screenrecording.mp4, adb bugreport, screenshot attached

Capture the evidence mechanically, not by hand: adb logcat -d > logcat.txt, adb shell screencap -p /sdcard/f.png && adb pull /sdcard/f.png, adb shell screenrecord /sdcard/flow.mp4, and adb bugreport bugreport.zip for the deep cases. Two details separate usable from useless reports: the app build (versionName and versionCode/build number — "happens on 3.0" is ambiguous across three internal builds) and the path to the bug (install history and whether it reproduced on a clean install or only after an upgrade).

Pair the report with one quantitative measurement where the defect is performance-shaped, because "feels laggy" cannot be triaged: startup time via adb shell am start -W -n com.acme.shop/.MainActivity (reports TotalTime in milliseconds), frame jank via adb shell dumpsys gfxinfo com.acme.shop framestats, and memory via adb shell dumpsys meminfo com.acme.shop. Quote the number, the device, and the comparison baseline — "2.4 s cold start on the A13 versus 0.9 s on the Pixel 7" assigns the bug to the low-end hardware path immediately, instead of starting an argument about opinions. Numbers attached to a named device close arguments; adjectives start them.

A matrix that says where to test and a report format that says what happened together convert mobile testing from folklore into a repeatable practice — one a rotating team member can pick up without a week of context.