End-to-End Tests Against a Real Deployment

A suite meant to run only on demand was marked opt-in by adding testIgnore to the other projects.

// The bad way: opt-in by exclusion. The project is still declared.
projects: [
  { name: 'chromium', testIgnore: '**/dev-backend/**' },
  { name: 'dev-backend', testMatch: '**/dev-backend/**' },   // runs on a bare invocation
]

A bare npx playwright test runs every declared project, so the suite that mints real records into a shared tenant ran whenever anyone ran the tests. The good way is to declare the project only when its variable is set: a project that does not exist cannot be selected. Measured on bring-up, a bare run collected 2 dev-backend tests before the change and 0 after.

That is the shape of every rule below: on a live deployment, the default that is merely inconvenient in a mock suite is the one that creates real damage.

A suite that runs against a real deployment and creates real data on a shared tenant needs four properties an ordinary mock suite does not. Each one below is a failure that happened first and a rule that followed.

The worked example is arda-frontend-app’s dev-backend project, brought up in #1305. The steps generalize to any Playwright suite pointed at a live environment.

Make opt-in mean “not declared”

As the opening showed, testIgnore on the other projects does not make a live-backend project opt-in — it makes the other projects skip those files, which is a different thing and leaves the live project running by default.

Spread the project into the projects array only when its variable is set:

projects: [
  ...standardProjects,
  ...(process.env.E2E_DEV_BACKEND ? [devBackendProject] : []),
],

A project that does not exist cannot be selected. Naming it without the variable then fails loudly rather than silently running the specs against MSW.

Verify by collection count, not by reading the config. The config’s intent and the collector’s answer are different facts, and only one of them is what runs.

Give the run its own server

A non-empty E2E_BASE_URL makes Playwright skip its webServer block entirely. reuseExistingServer is never consulted, and whatever happens to be listening on the port gets tested — including a server someone left running in mock mode.

Default the variable to empty so Playwright owns the server, and turn reuseExistingServer off for this project so an occupied port fails the run instead of silently substituting for it.

Scope that setting to the project’s own variable rather than to CI. CI also switches on retries, and a retry in a suite that mints real records multiplies residue rather than recovering the run. Read what a global switch already does before inheriting it — the dev-backend project inherited retries: CI ? 2 silently, and nobody had asked.

Measure residue, do not assert it

Publishing an item auto-mints a card: operations ServiceImpl.kt:78 calls addDefaultCardFor. Two spec header comments stated their residue from belief, and both were wrong by exactly that card.

Read the tenant at the end of the run and log the ids. State what the run left behind from what you observed, not from what you intended to create.

Delete nothing. A card cannot be deleted, and whether one with a live cycle may be is an open question. On a shared tenant that is a bigger decision than a test should take by itself. Name the residue instead — E2E-DEV-<pr>-<UTC timestamp> — so a human can find and clear it.

Identify the subject, do not take the first one

.first() in a cards panel finds the item’s auto-minted card, not the one the spec created. The assertions pass either way, because an AVAILABLE card sends the same verbs whichever one it is. The spec proves nothing and reports a pass.

Find the subject by a value the spec chose — the serial in the row — and assert the locator resolves to exactly one match.

Resolve credentials without forcing a prompt

Run Playwright directly when the credentials are already in the environment, and fall back to 1Password only when they are not:

test-e2e-dev-backend:
  @if [ -n "$$E2E_USER_EMAIL" ] && [ -n "$$E2E_USER_PASSWORD" ]; then \
    npx playwright test --project=dev-backend; \
  else \
    op run --env-file 1Password/e2e-dev.env -- npx playwright test --project=dev-backend; \
  fi

This is the same shape as .npmrc reading ${GITHUB_TOKEN}. A caller that already holds the values is not made to answer a biometric prompt for them, and an unattended run does not stall on an approval nobody is there to give.

Poll to a landing state, not to a match

expect.poll samples until one sample passes. It never settles.

Polling a growing list to equality proves a lower bound only: a sixth event arriving after the passing sample is never seen, and the assertion that read “the client sends exactly these five verbs” actually read “at least these five, at some moment.” To bound a sequence, await the terminal state — the response, the settled UI — and then read once.

This one was got wrong twice on the same assertion: first as a slice(0, 5) prefix comparison, then as a poll-to-equality described in its own comment as “settling.” It belongs to the family in Checks That Cannot Fail.

Know what the run does not cover

State the limits with the results. A suite like this earns its credibility from what it exercises against a real backend, and loses it the moment a reader over-credits a number.

Two limits are structural rather than incidental. Files under e2e/ are excluded from the unit suite by testPathIgnorePatterns, so nothing in the suite is itself unit-tested. And a suite gated on credentials a person must approve does not run in CI, so a change to it ships on the last manual run — name that run and its commit.

Prerequisite for local runs

arda-frontend-app needs a local .env.local carrying placeholder AMAZON_CREATORS_* values. src/lib/env.ts validates them eagerly on first import, and Next pulls that module into nearly every route’s import graph, so without them every authenticated route returns 500 — including routes with nothing to do with Amazon. See knowledge-base/amazon-creators-local-env.md in that repository.

Sources

Brought up on #1305, stacked on #1290, across thirteen runs against a dev deployment. Server ownership landed in 1acd2fb9; the opt-in, residue and subject-identity corrections in 310167fd; the retries inheritance in 10fd7376; the expect.poll correction in 468783b5 on #1290. The submission this page was written from is /workbooks/notebooks/domain-ontology/streams/storefront/parmandil-submission-c4.md.