Skip to content

Decision Log: Operations API Tests

Decision log for the Operations API Tests project. Captures the choices made while defining a TypeScript-based API test suite that re-implements and extends the operations component’s Bruno tests. See the goal for scope.

#QuestionStatusDecisionRound
DQ-001Where do the project docs live (roadmap area)?Decidedroadmap/engineering/quality/operations-api-tests/R1
DQ-002Which repository hosts the new test suite?DecidedCo-located in operationsR1
DQ-003What is the role of the operations repo?DecidedDeploy target and branch for test-driven fixesR1
DQ-004Directory for the suite within operations?Decidedoperations/api-tests (top-level)R1
DQ-005Test runner / framework?DecidedVitest 4R2
DQ-006HTTP client and type-safety strategy?Decidedfetch + openapi-typescript from served OpenAPIR2
DQ-007Lint / format tooling?DecidedESLint flat config + PrettierR2
DQ-008Runtime, build, and packaging?DecidedESM, strict TS, no bundle, tsc gate, Node 22 LTS, npm, MakefileR2
DQ-009Coverage strategy?DecidedEndpoint/scenario checklist vs OpenAPI (no Istanbul)R2
DQ-010OAuth2 / JWT auth library?Decidedjose + openid-client (panva)R2
DQ-011CI execution target for the suite?OpenPlatform = GitHub Actions; both A + B likely needed — nuanced setup, follow-up this sessionR3
DQ-012Package/workspace structure and DAG enforcement?Decidednpm workspaces + TS project references + dependency-cruiser; tests-are-leavesR4
DQ-013Test-tree organization and shared visibility?Decided<domain>/<module>/<endpoint> + cross-workflows/ per level; shared visible to siblings + descendantsR4
DQ-014Test categories and quarantine?DecidedFilename-suffix categories (probesanityacceptance⊆ALL + functional/stress/bug/extra); quarantine = typed statusR5
DQ-015SUT config, capabilities, and runner interface?Decidedsut-config.ts at test root (op:// refs only); capability-gating; tsx CLI over vitest runR5
DQ-016Packaging and execution of the suite?DecidedManual local-shell V1 (no container, no GHA yet); npm-bin run-suite; 1Password service-account + interactive fallbackR6
DQ-017Walking skeleton: seed, SUT, auth, data lifecycle?DecidedSeed item on local; static Bearer auth; per-run minted tenant ID (clean space + tenant-isolation)R6
DQ-018Are generated proxies committed, or generated per-run?DecidedGenerated per-run against the SUT under test (git-ignored build artifact); compile-after-generate is the contract checkR7

Round 1: Placement and Repository Structure

Section titled “Round 1: Placement and Repository Structure”

Context: A recent convention change moved project docs from roadmap/in-progress/<project>/ to roadmap/engineering/<sub-area>/<project>/.

Decision: roadmap/engineering/quality/operations-api-tests/ — an API test suite is cross-cutting engineering work under the quality sub-area (“coverage tooling, acceptance-test health”). Registered in the quality index; an index.md entry point sits beside goal.md.

Applied to: index.md, goal.md, quality/index.md § Projects.


DQ-002: Repository that hosts the test suite

Section titled “DQ-002: Repository that hosts the test suite”

Context: The suite could live in a new repo, the existing api-test repo (Bruno), or co-located in operations.

OptionDescriptionTrade-offs
ACo-locate in operationsTests version with the API they cover; single PR for API-fix + test. Foreign toolchain in a Gradle repo.
Bapi-test repo (existing TS/Node project)Already TS; holds the Bruno tests. But decouples tests from the API version; two-repo changes.
CNew dedicated repoCleanest separation. Heaviest setup (repo, CI, changelog, infra).

Recommendation: Option A.

Decision: Option A — co-locate in operations. Tests travel with the API version; a bug found by a test can be fixed and re-verified in one branch/PR.

Applied to: goal.md § Repositories.


Decision: Both the deploy target (make localInstall runs the API under test) and a code-change branch where API bugs surfaced by the new tests are fixed (test-driven).

Applied to: goal.md § Repositories, § Scope.


DQ-004: Directory for the suite within operations

Section titled “DQ-004: Directory for the suite within operations”

Context: Two candidate paths were considered: operations/src/test/typescript and a top-level operations/api-tests.

OptionDescriptionTrade-offs
ATop-level operations/api-testsOutside all Gradle source sets; matches the existing scripts/bulk_create_items Node subproject precedent; easy CI path filter. Name echoes the standalone api-test repo.
Boperations/src/test/typescriptsrc/test is the JVM unit-test source set (tasks.test, Kover). Placing TS there misleads Gradle/IntelliJ/Kover and pollutes the unit-test tree with node_modules/dist.

Recommendation: Option A, named api-tests (plural, matching the project).

Decision: Option A — operations/api-tests. Self-contained Node/TS project (own package.json, tsconfig, node_modules); node_modules/dist/coverage gitignored; the Gradle build stays unaware of it. These are black-box API/E2E tests (a separate tier from in-process unit tests in src/test/kotlin).

Applied to: operations/api-tests/ (to be created during implementation).


Non-negotiables for this round: (1) strongly-typed TypeScript; (2) runs both locally and in GitHub Actions efficiently and with minimal friction. Reference is arda-frontend-app, with deviation allowed where justified.

Guidance consulted:

  • typescript-coding skill — the intended workspace TS standard is Vitest + Istanbul, ESLint flat config, ESM, strict mode, zero-dependency (the backing doc page is not yet written; this is the skill’s stated direction).
  • tooling.md — TS strict mode; ESLint + Prettier; a GitHub Actions workflow plus a Makefile target for local runs; keep config free of Vercel-only assumptions.
  • run-operations-api-test — the local flow the suite must slot into (make buildmake localInstall → pod check → run → check results); the workspace already runs standalone TS via npx tsx.
  • Reference repos: arda-frontend-app (Jest + Playwright, ESLint flat config, tsx, nyc); api-test (Jest/ts-jest, Biome + ESLint flat, esbuild, ESM, Node 24); ux-prototype — the newest TS repo — Vitest 4 + @vitest/coverage-v8 + ESLint 10 flat + Prettier + ESM + tsc typecheck + Makefile check/ci, Playwright scoped to VRT only, Node >=20.19.0.
  • Convergence: Vitest + ESLint-flat + Prettier + ESM + tsc typecheck + Makefile check/ci is consistent across ux-prototype, arda-frontend-app, and the stated standard; only the older api-test diverges (Biome, Jest).

Context: The runner sets the DX and CI speed for the whole suite. These are black-box HTTP tests against a deployed operations instance (no in-process app handle).

OptionDescriptionTrade-offs
AVitestNative ESM + TS via esbuild (no ts-jest transform); Jest-compatible API (low friction from FE/api-test); fast startup; built-in JUnit reporter + coverage; matches the workspace TS standard. Pulls in Vite as a dep.
BJest + ts-jest (or @swc/jest)What api-test/arda-frontend-app use — familiar. ts-jest is slow (type-checks on transform); Jest ESM support is still awkward and api-test is ESM. @swc/jest restores speed but drops type-checking in-test.
Cnode:test + tsxZero heavy deps, built into Node, very fast. Sparse assertions (node:assert), thinner reporter/mocking ecosystem, less familiar.
DPlaywright Test (request fixture)Excellent API-testing ergonomics (retries, tracing, JUnit/HTML, parallel); arda-frontend-app already uses Playwright. Heavier and browser-centric for a pure-HTTP suite.

Recommendation: Option A (Vitest). Best balance of strong typing, speed, CI reporters, and minimal-friction Jest-style API for a fresh Node/ESM project. Playwright Test (D) is the strong runner-up if request tracing / shared FE e2e infrastructure becomes valuable.

Decision: Vitest 4 — ratified. Confirmed by convergence with ux-prototype (newest TS repo) and the stated workspace standard.

Applied to: operations/api-tests (tooling, to be created).


DQ-006: HTTP client and type-safety strategy

Section titled “DQ-006: HTTP client and type-safety strategy”

Context: The “strongly typed” non-negotiable is best served at the wire boundary. Each operations module already serves OpenAPI at /v1/<module>/docs/openApi.json.

OptionDescriptionTrade-offs
ANative fetch (undici) + thin typed wrapperZero deps, built into Node; full control of headers/bitemporal params. Types are hand-written.
Bopenapi-typescript codegen + openapi-fetchTypes generated from the server’s own OpenAPI → contract-accurate, self-updating typing. Adds a codegen step and a small client dep.
CsupertestFamiliar (in api-test devDeps). Designed to bind an in-process app; awkward for black-box against a deployed URL.
DaxiosErgonomic, interceptors. Extra dep; weaker type-inference than openapi-fetch; redundant with fetch.

Recommendation: Option A now (thin typed fetch client encapsulating auth, X-Tenant-Id/X-Request-ID/X-Author, and effectiveasof/recordedasof), adopting Option B (openapi-typescript) to generate request/response types from the served OpenAPI — this most directly satisfies the strong-typing goal and keeps tests honest against the contract. Avoid C (wrong shape) and D (redundant).

Decision: Native fetch + thin typed client, with openapi-typescript generating request/response types from each module’s served OpenAPI — ratified.

Applied to: operations/api-tests (client + codegen step, to be created).


OptionDescriptionTrade-offs
ABiome (lint + format, one binary)Extremely fast, near-zero config, single tool; matches api-test. TS lint rules are good but less extensive than typescript-eslint’s type-aware set.
BESLint flat config + PrettierMatches arda-frontend-app and the workspace TS standard; richest type-aware rules. Two tools, slower, more config.
CESLint flat config only (no Prettier)One tool; formatting via ESLint stylistic rules. Slower than Biome; formatting-as-lint is fiddly.

Recommendation: Option B (ESLint flat config + Prettier). After checking ux-prototype, the signal is no longer split: the newest TS repo, plus arda-frontend-app and the stated standard, all use ESLint-flat + Prettier. Biome is the outlier (older api-test only). Choose B for consistency and type-aware rules; use ux-prototype’s eslint.config.mjs as the starting point (drop the Storybook/React plugins, keep eslint-plugin-unicorn).

Decision: ESLint flat config + Prettier — ratified.

Applied to: operations/api-tests (eslint.config.mjs, Prettier, to be created).


DimensionRecommendationNotes
Module systemESM ("type": "module")Matches api-test; modern default.
TS strictnessstrict: true (+ noUncheckedIndexedAccess)Serves the strong-typing non-negotiable.
Build/bundlingNone — Vitest/tsx run TS directly; tsc --noEmit as the typecheck gateNo shipped artifact; a test suite doesn’t need bundling.
Node versionNode 22 LTS (pin via .nvmrc + actions/setup-node)api-test pins v24.2.0 (current, non-LTS); 22 LTS is safer for CI reproducibility.
Package managernpm (npm ci), with GITHUB_TOKEN for @arda-cards packagesReference repos use npm; matches workspace auth convention.
Task entryMakefile with check (lint + typecheck) and ci/test targetsMirrors ux-prototype’s make check/make ci for local≈CI parity.

Recommendation: As tabulated. Node 22 LTS sits above ux-prototype’s >=20.19.0 floor and below api-test’s 24; the Makefile check/ci convention directly serves the “runs locally and in GHA with minimal friction” non-negotiable.

Decision: As tabulated — ratified. ESM, strict + noUncheckedIndexedAccess, no bundling, tsc --noEmit gate, Node 22 LTS, npm, Makefile check/ci.

Applied to: operations/api-tests (package.json, tsconfig, Makefile, to be created).


Context: arda-frontend-app measures app code coverage with nyc/Istanbul. That model does not transfer: this suite exercises the Kotlin operations service over HTTP — its code coverage is measured by Kover on the JVM side, not by the TypeScript runner. Measuring coverage of the test files themselves is meaningless.

OptionDescriptionTrade-offs
ATrack endpoint/scenario coverage as a checklist vs the OpenAPIMeasures what matters (API surface exercised); no istanbul plumbing.
BAdd Istanbul/nyc to the TS suiteMatches the frontend mechanically but measures the wrong thing here.

Recommendation: Option A — endpoint/scenario coverage against the served OpenAPI (deliberate deviation from arda-frontend-app/ux-prototype, which measure their own code). If a coverage number is ever wanted for the suite’s own helpers/client, Vitest’s built-in @vitest/coverage-v8 (as in ux-prototype) provides it for free — no Istanbul/nyc.

Decision: Endpoint/scenario checklist vs the served OpenAPI — ratified. No Istanbul/nyc; @vitest/coverage-v8 available if suite-helper coverage is ever wanted.

Applied to: verification approach (planning phase).


DQ-010: OAuth2 / JWT authentication library

Section titled “DQ-010: OAuth2 / JWT authentication library”

Context: The operations API is moving from a static bearer API key toward OIDC JWT auth. Identity is AWS Cognito (COGNITO_USER_POOL_ID, BFF_CLIENT_ID/SECRET), and the server already verifies JWTs — migrations add oidc_sub (“verified JWT subject of the author”) across facility, kanban, item, orders, business-affiliate, and station. The suite must acquire Cognito OAuth2 tokens (primarily client-credentials via /oauth2/token), and decode / verify / inspect JWT claims — possibly minting JWTs for fixtures. Rolling our own is strongly discouraged.

ConcernOptionTrade-offs
JWTA. jose (panva)Zero-dep, ESM-native, fully typed, WebCrypto; verifies against Cognito JWKS and mints test tokens. Modern successor to jsonwebtoken.
JWTB. jsonwebtokenWhat api-test uses; CJS, callback-style, needs @types, dated.
JWTC. aws-jwt-verifyServer-side Cognito validator — wrong side (we are the client).
OAuth2D. openid-client (panva, built on jose)OpenID-certified, ESM, typed; Cognito discovery + client-credentials + auth-code/PKCE + refresh. Future-proofs the OAuth2 direction.
OAuth2E. oauth4webapi (panva)Lighter, function-style, same author; ideal if flows stay client-credentials-only. Less batteries than D.
OAuth2F. amazon-cognito-identity-jsCognito-locked, SRP/user-pool focused, drags AWS SDK; only needed for real user-SRP login.
OAuth2G. Roll-your-ownDiscouraged; only the single client-credentials POST is trivial.

Recommendation: A + Djose for JWT, openid-client for OAuth2. One coherent, certified, ESM-first, minimal-dep family (both by panva) that satisfies strong-typing, minimal-friction, and no-custom-auth at once. Down-scope to oauth4webapi (E) if only client-credentials is ever needed. amazon-cognito-identity-js (F) enters only if persona tests must perform real Cognito SRP user login.

Decision: jose + openid-client (panva).

Applied to: operations/api-tests (auth helper, to be created).


Context: “Runs in GitHub Actions efficiently” depends on what the tests hit in CI — this is not answered by any reference repo (they test in-process or against Storybook, not a deployed backend). It shapes auth, speed, and flakiness more than any tool choice.

OptionDescriptionTrade-offs
ASpin operations in-workflow (kind/k3d cluster or a service container)Hermetic, no shared-env contention; every PR self-contained. Slower CI, more setup, needs DB/LocalStack + secrets in CI.
BRun against an existing dev/stage environmentFast to start, realistic. Shared-env flakiness, data pollution, requires network + partition credentials; not hermetic.
CBoth — local + PR against ephemeral (A); scheduled/nightly against dev (B)Best coverage; most pipeline to build/maintain.

Recommendation: Soft — mirror api-test’s model as a local vs CI split rather than a single A/B pick: local = A (already the dev flow, make localInstall → run, exercises the branch’s code); CI initial = B against dev (minimal plumbing, fastest path to running the ported tests); add A2 in CI later for PR-gating of operations code changes if it proves valuable. Note: even A is not hermetic for auth — Cognito is external, so tokens need a real dev pool or a stubbed issuer.

Decision: Platform confirmed = GitHub Actions. Direction: both A and B will likely be needed (hermetic per-PR gating and a run against a live environment), but the setup is nuanced. Open point — follow-up discussion this session once we have sample tests to reason about data-cleanup and auth.

Applied to: (pending — CI workflow, implementation phase).


Full detail in the design document; these entries record the decisions in summary.

DQ-012: Package/workspace structure and DAG enforcement

Section titled “DQ-012: Package/workspace structure and DAG enforcement”

Context: The suite separates four code areas (util/main, client-proxies, util/test, tests) with a strict one-directional dependency matrix that must form a DAG at every level of aggregation, and each non-test area must be promotable to a standalone npm package.

Decision: Structure the suite as a single npm workspace where each area is a package (@arda-cards/api-util, @arda-cards/api-client-proxies, @arda-cards/api-test-util, plus a private tests package). Enforce the DAG and layer boundaries with three mechanisms in make check/CI: (1) workspace package boundaries, (2) TypeScript project references, (3) dependency-cruiser (no-circular = the DAG guarantee, plus layer + shared-visibility + tests-are-leaves rules). client-proxies splits into generated/ (openapi-typescript, do-not-edit) + a hand-authored façade. Auth: TokenProvider interface in api-util, concrete impl in api-test-util (per DQ-010).

Applied to: design.md § Repository Layout, § Enforcement, § Auth Layering.


DQ-013: Test-tree organization and shared visibility

Section titled “DQ-013: Test-tree organization and shared visibility”

Context: Tests organize by <domain>/<module>/<endpoint>, with cross-element suites and per-level shared code, keeping DRY without over-generalizing.

Decision: Cross-element suites live in cross-workflows/ at each level (renamed from integration/ to avoid the test-tier collision): cross-endpoint at module level, cross-module at domain level, cross-domain at system level. Each level has an optional shared/ visible to its siblings and descendants (its enclosing subtree) and nowhere else; shared may depend on util/*, client-proxies, and ancestor shared, never on tests. Tests are leaves (no test→test imports); reusable workflow fragments live in shared/util/test.

Applied to: design.md § Test Tree, § shared Visibility Rule.


Round 5: Categories, SUT Characterization, and the Runner

Section titled “Round 5: Categories, SUT Characterization, and the Runner”

Full detail in the design document § Test Categories and § SUT Characterization and the Test Runner; these entries record the decisions in summary.

Decision: Categories are declared by filename suffix (a selection concern). Cumulative progression probe ⊆ sanity ⊆ acceptance ⊆ ALL (access renamed to probe to avoid the authorization connotation), plus orthogonal sets functional, stress, bug (pdev-<n>-<slug>.bug.test.ts), extra. Quarantine is an orthogonal status, not a category: a typed option on apiTest ({ quarantine: { until, ticket } }), always runs, ticket required by the type. Governance (expiry + existence budget) validated by a TypeScript (tsx) validator over Vitest collection in make check — mirrors arda-frontend-app’s expiry/ticket/budget practice, typed instead of grep’d.

Applied to: design.md § Test Categories, § SUT Characterization.


DQ-015: SUT configuration, capabilities, and runner interface

Section titled “DQ-015: SUT configuration, capabilities, and runner interface”

Decision: A prominent sut-config.ts at the test root declares each target (keys local, alpha002-dev, alpha002-stage, alpha001-demo, alpha001-prod), holding only 1Password op:// references (no secrets → one typed file, no .env sprawl), two credential tiers (API + OAM), OAM coordinates, and a closed Capability enum shared by config and each test’s requires. A test runs iff it matches required-tests and requires ⊆ SUT.capabilities, else skipped-(capability) and reported. Capability discovery is best-effort (declaration is source of truth). A dynamic SUT descriptor (probed at runtime) carries provenance: component deployment version = repository tag, with a local-untagged allowance; per-module API versions. The runner is a thin tsx CLI over vitest run taking required-tests, SUT name, --quarantine-budget N (0 = fail on any quarantined failure), --missing-capabilities-fail (else skip), and instrumentation (V1 test logger; Sentry + SUT-log tailing V2). probe failure fast-aborts. Report = the non-quarantined/quarantined × skipped/pass/fail matrix (+ JUnit/JSON; k8s log excerpts V2).

Applied to: design.md § SUT Characterization and the Test Runner.

Open: quarantine-budget semantics (runtime failures vs existence) — assumed split, to confirm. Qualified capabilities (production/mock) deferred.


Decision: V1 targets manual execution from a local shell — no container, no GitHub Actions automation yet. Suite = the npm workspace (npm ci, GITHUB_TOKEN for @arda-cards, Node 22 pinned); runner exposed as npm-bin run-suite. Secrets resolve via a 1Password service-account token (OP_SERVICE_ACCOUNT_TOKEN env / GH secret) with fallback to interactive op locally. Test-suite changes take an operations CHANGELOG entry (usually Fixed, patch); the single per-repo version means a SUT’s deployed tag identifies the matching suite revision. Container and package publishing deferred.

Applied to: design.md § Packaging and Execution.


DQ-017: Walking skeleton — seed, SUT, auth, data lifecycle

Section titled “DQ-017: Walking skeleton — seed, SUT, auth, data lifecycle”

Decision: First slice seeds the item module against the local SUT (make localInstall, disposable Postgres). Tests: a probe (configuration / oam/version) and a sanity item flow (query, create → get). Auth is the static Bearer token via a StaticApiKey TokenProvider (OAuth2/Cognito provider built behind the same interface, dormant). Test data: each run mints a fresh tenant ID — clean namespace and tenant-isolation coverage by default; no teardown.

Applied to: design.md § Walking Skeleton (V1 first slice).


DQ-018: Committed proxies vs generate-per-run

Section titled “DQ-018: Committed proxies vs generate-per-run”

Context: An early framing committed the openapi-typescript output as a “reified control point” and regenerated occasionally. But in every supported workflow (local dev, local verification, post-deploy CI) the proxies are generated against a live SUT, and the harness is type-checked against that fresh output.

Decision: The generated proxies are a build artifact, not source — they are git-ignored and regenerated per-run against the SUT under test, via make verify (resolve base URL → gen-proxies → tsc → vitest). The explicit tsc step is the gate: because Vitest transpiles without type-checking, a compile failure after regeneration is the suite’s cheap contract / backwards-compatibility check (removed/renamed schema, new required field, etc.). Committing a snapshot is not merely unnecessary but counterproductive — it compiles green while the live API drifts, masking exactly what the check exists to catch. Specs are fetched once, canonicalized (sorted keys — the SUT serializes non-deterministically), cached, and skipped when unchanged, so the dev loop only recompiles on real contract changes.

Trade-off accepted: no offline typecheck of proxy-facing code — the only typecheck is against a live SUT (post-deploy in CI). The util layer, lint, quarantine, and import type-only unit tests stay hermetic. A spec differ (oasdiff) for direction-aware major/minor CHANGELOG classification is a planned companion, not a prerequisite.

Supersedes: the “reified control point / committed output” framing in earlier design prose (DQ-006 tooling choice is unchanged — openapi-typescript from served OpenAPI; only the baseline-vs-per-run question is settled here).

Applied to: design.md § client-proxies Generation.



Copyright: (c) Arda Systems 2025-2026, All rights reserved