Structuring an Automation Suite

"The suite takes 50 minutes" is not a performance problem — it is a structure problem. A suite with no internal boundaries forces every context to run everything: a pull request waits 50 minutes for confidence it needed in five, and a nightly run that could catch deep regressions finishes too late to matter. The fix is partitioning tests into named suites, each with a job, a stage, and a time budget.

Smoke is the smallest set of tests that proves the deployment works at all: app loads, login succeeds, the core record can be created and read. Think 5–15 tests, 2–5 minutes, run on every commit and as a gate before any deeper stage. If smoke fails, later stages are pointless — the build is broken, and the pipeline should stop with a clear signal rather than burn 45 minutes discovering the same fact repeatedly.

Regression is the broad suite of known-critical behavior: every flow that has broken before, plus the paths that cost the most when they break (checkout, auth, permissions, data writes). Budget 10–20 minutes, run on every pull request or every merge to main. This is the suite whose runtime teams argue about, so it grows deliberately — each addition earns its place by the stability × repetition × value filter from Automation Concepts and Frameworks.

Nightly is where the expensive coverage lives: full browser matrix from your Cross-Browser and Device Coverage tiers, visual regression snapshots, slow integrations, negative-path deep dives, and retries-once passes that surface latent flakiness. Budget 30–60 minutes because nobody waits on it; the payoff is a report ready when the team arrives.

How you declare membership matters as much as the tiers. Playwright's approach — tags plus --grep:

// tests/checkout.spec.ts
test('@smoke guest can complete checkout', ...);
test('@regression saved cart survives reload', ...);
test('@nightly checkout with 40 items in cart', ...);

npx playwright test --grep @smoke        # PR gate
npx playwright test --grep-invert @nightly  # local full run

JUnit 5 expresses the same idea with @Tag("smoke") and @SelectPackages; pytest with marker decorators and -m smoke. Tags beat file-location-only schemes (a smoke/ directory duplicates tests across tiers) and beat renaming files, because one test can legitimately belong to two tiers — a checkout test tagged both @smoke and @regression runs in the fast gate and the broad suite without being written twice.

A concrete pipeline mapping teams ship — each line names the suite it invokes, so the pipeline file stays a list of invocations rather than a list of test logic:

push (PR):     lint → build → unit → smoke (5m) → regression (15m)
merge (main):  smoke + regression + API tests → deploy staging
nightly 02:00: full matrix × visual × slow tests → report to Slack/email
release:       nightly artifacts + trace review + human exploratory session

Two rules keep the tiers honest. First, the pipeline owns the tags, not the developer's memory — CI invokes named suites via scripts in package.json or the Makefile (npm run test:smoke), so adding a test to the right tier is a label, not a pipeline edit. Second, fail fast ordering: within a suite, order tests so cheap, high-signal checks run first (Playwright and JUnit both allow declaration order or priorities), so a catastrophically broken build reports in 90 seconds instead of 12 minutes. A suite without tiers is a queue; a suite with tiers and budgets is a feedback system — and its second component, the configuration that lets the same tests point at local, staging, and CI environments, is next.

The failure this chapter prevents has a recognizable smell: base_url_local.ts, base_url_local_backup.ts, a URL commented out "for CI", and a wiki page titled "How to run tests" that is wrong. Every environment difference handled by copy-paste eventually gets handled wrongly. The design that survives is one checked-in configuration with a defined precedence chain, where environment variables — never code edits — supply the differences.

Layer the config. A working scheme for a web suite:

1. defaults in config file        baseURL, timeouts, browser, headless flag
2. environment file (committed)   .env.test / .env.staging — non-secret values
3. environment variables (CI)     secrets, run-specific overrides
4. CLI flags                       one-off debugging (--headed, --grep)

Playwright reads this natively — use: { baseURL: process.env.BASE_URL ?? 'http://localhost:3000' } — and a committed .env.test with dotenv gives every developer identical defaults on day one. In CI, the variable is exported by the pipeline YAML:

# GitHub Actions sketch
- name: Run regression
  env:
    BASE_URL: https://staging.example.com
    AUTH_EMAIL: ${{ secrets.QA_ACCOUNT }}
    AUTH_TOKEN: ${{ secrets.QA_TOKEN }}
  run: npm run test:regression

Secrets never enter the repo. A credential committed in .env is a credential in git history forever — rotate immediately if it happens, and add secret scanning (gitleaks, trufflehog, or the platform's push protection) so it cannot happen again quietly. The test suite is a high-privilege consumer of data: QA accounts can create orders and read customer records, which makes leaked test credentials a genuine incident, not an inconvenience.

One rule for hard-coding. Search your suite for http:// and for localhost — anything outside the config layer is a future failure. The same applies to credentials, timeouts, and file paths: if changing an environment requires editing test code, the config layer has been bypassed. Enforce it with a trivial lint rule or code review checklist; the payoff is the moment staging moves domains and your entire response is one variable.

Distinguish three kinds of value so nobody confuses them:

Environment parity deserves an honest note. Tests against localhost and tests against staging are different tests: different reverse proxies, different caching, different data. Keep one config shape for all environments, and when a CI-only failure appears, first diff the configuration (URL, flags, browser version) before diffing behavior — configuration drift between environments is the most common cause of "passes locally, fails in CI", and it is cheap to rule out.

Make the config self-documenting. A config/index.ts that exports a typed object gives every consumer autocomplete and turns a missing variable into a startup failure with a named message — BASE_URL is required; copy .env.test to .env — instead of a mysterious undefined mid-test. Fail fast at boot, not at test 47. The third pillar of suite structure — beyond tiers and config — is the shared helpers layer, where auth setup and navigation stop being copy-pasted.

Two months after the first tests, the same eight lines appear in thirty files: navigate to base URL, open login, fill credentials, submit, wait for the avatar. Copy-paste works until the login flow changes a field — thirty edits, and one file missed. The helpers layer is where that repetition dies; the danger it introduces is helpers that swallow the very intent the test was supposed to express.

Start with names — the cheapest structure there is. A test title should state the behavior and the condition, because the title is the first line of every failure report, CI notification, and flaky-test dashboard entry:

✗ test_login_2                       # says nothing about intent or condition
✗ login.spec.ts > case 7            # requires opening the file to understand
✓ 'locked-out account shows an unlock message instead of an error loop'
✓ 'admin sees the audit tab; viewer does not'

Behavior-plus-condition names also do triage work: when a nightly email lists three failures, the titles alone should tell you whether one feature broke or three unrelated things did. Keep them independent of implementation — the name must survive a CSS redesign.

Then extract helpers by intent, not by line count. The litmus test for whether something belongs in the helpers layer: would a reader of the test be misled if they saw these steps in detail? Logging in is setup — a test about password reset should not narrate the login ceremony — so login becomes one call:

// helpers/auth.ts
export async function loginAs(page: Page, role: 'admin' | 'viewer' = 'viewer') {
  const creds = await getCredentials(role);        // from fixtures/env
  await page.goto('/login');
  await page.getByLabel('Email').fill(creds.email);
  await page.getByLabel('Password').fill(creds.password);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByTestId('user-menu')).toBeVisible();  // verify inside the helper
}

// the test now reads at its own altitude:
test('viewer cannot open the audit tab', async ({ page }) => {
  await loginAs(page, 'viewer');
  await page.goto('/settings');
  await expect(page.getByRole('link', { name: 'Audit' })).toHaveCount(0);
});

Note the assertion inside loginAs: a helper that silently produces a half-logged-in state turns every downstream failure into a misleading mystery. Setup helpers must verify their own outcome — "arrange until verified" — so a broken login reports as a login failure, not as forty tests mysteriously missing an avatar.

The limits. Helpers should do one thing a domain expert would name ("log in as role"). A setupEverything() that hides ten implicit steps fails the altitude test: the test no longer says what it does, and debugging requires reading the helper first. Similarly, never put the behavior under test in a helper — if the checkout steps live in completeCheckout(), the checkout test asserts nothing but a return value. The rule: setup and navigation may be abstracted; the assertion and the action under test must stay in the test body.

Where things live is the last structural habit:

tests/
  smoke/            regression/         nightly/
  helpers/          (auth, navigation)
  fixtures/         (data factories — preview of the next phase)
  .env.test         playwright.config.ts (at repo root)

With tiers, config, and helpers settled, every test you write from here lands in a known place with known dependencies — and the only remaining source of everyday breakage is how the tests find elements, which is where CSS and XPath Locator Basics begins.