Design: Operations API Tests — Structure and Layering

Provenance. The original structural design, superseded by the canonical design set. Kept for historical context.

The original structural design for the operations/api-tests TypeScript suite. It defines the four code areas, their allowed dependencies, the intra-test directory tree, and — critically — how the “must be a DAG at every level of aggregation” invariant is enforced by the build, not merely documented. See the goal for scope and the decision log for the tooling choices this design builds on (Vitest, openapi-typescript, ESLint flat, ESM, jose/openid-client).

Principles

1. Design for extraction. Each non-test area is structured as a standalone, independently publishable package. Promotion to a shared npm package (for arda-frontend-app, accounts, or a future system-api-tests repo) must be a move-and-publish, not a refactor. Promotion itself is out of scope; the structure that enables it is in scope.
2. Strict layered DAG. Dependencies flow one direction through the four areas. Cycles are prohibited at every level of aggregation and are caught by the build.
3. DRY with a promotion gradient. Common elements live at the lowest level that serves all their consumers: a level’s shared → util/test → util/main. Duplication is a defect; over-generalization (promoting too early) is also a defect.
4. Tests are leaves. Test files never import other test files. Reusable workflow fragments live in shared or util/test as composable builders/fixtures. This keeps the intra-test dependency graph trivially acyclic.
5. The wire is a compiler-enforced control point. All protocol access goes through client-proxies; a contract change breaks the TypeScript build.

The Four Areas

Area	Package (intended)	Depends on (internal)	Responsibility
util/main	@arda-cards/api-util	— (none)	Generic, protocol-agnostic client utilities: retry/backoff, pagination, bitemporal (effectiveasof/recordedasof) helpers, clock, ID/correlation, config parsing, and the TokenProvider interface + a generic OAuth2/OIDC client (built on openid-client/jose). Nothing here references wire types.
client-proxies	@arda-cards/api-client-proxies	util/main	The reified wire layer. openapi-typescript output in generated/ (do-not-edit) plus a hand-authored, stable façade per module that tests target. Accepts a TokenProvider (from util/main) and attaches auth; auth-mechanism-agnostic.
util/test	@arda-cards/api-test-util	util/main, client-proxies	Test-only infrastructure: harnesses, mocks, fixtures/factories, data cleanup, and the concrete credential/persona-backed TokenProvider implementation injected into the proxies.
tests	(private, unpublished)	util/main, util/test, client-proxies	The test suites themselves (see Test Tree). Top of the DAG; consumes everything, is consumed by nothing.

The four-area dependency DAG (foundation at the bottom):

Repository Layout

A single npm workspace (monorepo). Each area is a workspace package; the four package boundaries enforce the four-area layering, and TypeScript project references encode the build-order DAG.

operations/api-tests/
├── package.json                 # private workspace root; workspaces: [packages/*, tests]
├── tsconfig.base.json           # shared strict/ESM compiler options
├── tsconfig.json                # solution file: references all packages + tests
├── vitest.workspace.ts          # Vitest projects across packages + tests
├── eslint.config.mjs            # ESLint flat + Prettier (from ux-prototype baseline)
├── .dependency-cruiser.cjs      # DAG + layer + shared-visibility rules
├── Makefile                     # check / ci / test / gen-proxies
├── .nvmrc                       # Node 22 LTS
├── packages/
│   ├── api-util/                # util/main
│   │   ├── package.json         # @arda-cards/api-util  (no internal deps)
│   │   ├── tsconfig.json
│   │   └── src/
│   ├── api-client-proxies/      # client-proxies
│   │   ├── package.json         # @arda-cards/api-client-proxies (deps: api-util)
│   │   ├── tsconfig.json
│   │   └── src/
│   │       ├── generated/       # openapi-typescript output — DO NOT EDIT
│   │       └── <domain>/<module>/   # hand-authored façade (stable surface)
│   └── api-test-util/           # util/test
│       ├── package.json         # @arda-cards/api-test-util (deps: api-util, api-client-proxies)
│       ├── tsconfig.json
│       └── src/
└── tests/                       # tests area (private package)
    ├── package.json             # deps: api-util, api-test-util, api-client-proxies
    ├── tsconfig.json
    └── src/
        ├── cross-workflows/     # SYSTEM level: cross-domain scenarios
        ├── shared/              # SYSTEM-level shared (visible suite-wide)
        └── <domain>/
            ├── cross-workflows/ # DOMAIN level: cross-module scenarios
            ├── shared/          # DOMAIN-level shared
            └── <module>/
                ├── cross-workflows/  # MODULE level: cross-endpoint scenarios
                ├── shared/           # MODULE-level shared
                └── <endpoint>/
                    └── *.test.ts     # endpoint tests (leaves)

Test Tree

Tests are organized by <domain>/<module>/<endpoint>. Cross-element suites live in a cross-workflows/ directory at each level (renamed from integration/ to avoid colliding with the “integration test tier” meaning):

• Module level — cross-workflows/ holds cross-endpoint scenarios (e.g. create → query → print for one module).
• Domain level — cross-workflows/ holds cross-module scenarios (e.g. item → kanban within the same domain).
• System level (top of tests/) — cross-workflows/ holds cross-domain scenarios spanning the whole component.

Each level also has an optional shared/ for common elements that are not general enough to promote into util/test or util/main.

shared Visibility Rule

A shared/ package at a given level is importable by its siblings and descendants — i.e. anything within its enclosing subtree — and by nothing outside it.

From (importer)	May import shared/ at…
A module’s endpoint tests / cross-workflows	that module’s shared, its domain’s shared, the system shared
A domain’s cross-workflows	that domain’s shared, the system shared (not a child module’s shared)
System cross-workflows	the system shared only

shared may depend on util/main, client-proxies, and more-general (ancestor) shared; it must never depend on tests. This preserves the downward-only flow and keeps every level acyclic.

Enforcement

The DAG and layer boundaries are guaranteed by three mechanisms wired into make check and CI — a violation is a red build, not a review catch:

1. npm workspace package boundaries. An area can only import a package it declares in its package.json. client-proxies cannot import util/test because it isn’t a dependency — the four-area matrix is enforced by construction.
2. TypeScript project references. Each package’s tsconfig lists only its allowed references; cross-layer type errors and illegal imports surface at tsc time and the build order encodes the DAG.
3. dependency-cruiser (.dependency-cruiser.cjs, run in make check):
   • no-circular — the global guarantee that the graph is a DAG at every level of aggregation.
   • Layer rules re-asserting the four-area matrix (defense in depth with 1–2).
   • shared-visibility rules encoding the table above (an import of **/shared/** from outside its enclosing subtree is an error).
   • A tests-are-leaves rule: no module under tests/**/*.test.ts may be imported by another test.

ESLint (eslint-plugin-import / import/no-restricted-paths) provides in-editor feedback for the same rules; dependency-cruiser is the authoritative gate and can emit the dependency graph for this document.

Auth Layering

Consistent with DQ-010:

• util/main defines a TokenProvider interface and a generic OAuth2/OIDC client (openid-client + jose).
• client-proxies accepts a TokenProvider and attaches the Authorization header — with no knowledge of how the token was obtained.
• util/test supplies the concrete credential/persona-backed TokenProvider (Cognito client-credentials for machine-to-machine; persona flows as needed) and injects it into the proxies via a test harness.

This lets a future system-api-tests repo reuse client-proxies unchanged with its own provider — reinforcing the extraction path.

client-proxies Generation

openapi-typescript generates raw types into packages/api-client-proxies/src/generated/ (one module set per operations module, sourced from /v1/<module>/docs/openApi.json). These are never hand-edited. A thin, hand-authored façade under src/<domain>/<module>/ re-exports a stable, ergonomic surface that tests target, so regeneration churn is absorbed in one layer.

The generated types are build artifacts, not source: they are git-ignored and regenerated against the SUT under test before every typecheck/test run. make verify SUT=<name> is the canonical entrypoint — it resolves the SUT’s base URL (discovering the local NodePort via kubectl for local), runs gen-proxies, then the static gate (including tsc), then the tests:

resolve base URL → gen-proxies → tsc (contract gate) → vitest run

The tsc step is load-bearing and explicit: Vitest transpiles via esbuild and does not type-check, so without it a drifted contract would be silently stripped. A tsc failure after regeneration is therefore the suite’s built-in contract / backwards-compatibility check — the harness binds request DTOs (and, as coverage grows, responses/paths) from generated, so a removed/renamed schema or a new required field breaks compilation. This is why the output is not committed: a baselined snapshot compiles green even when the live API has moved, masking exactly the drift the check exists to catch.

Specs are fetched once per run to a git-ignored .cache/, canonicalized (object keys sorted — the SUT serializes schema maps in non-deterministic order per request), and openapi-typescript runs against the cache file. Generation is skipped when the canonical spec is unchanged and the output exists, so the inner dev loop pays the compile cost only when the contract actually changed.

Consequence, accepted deliberately: there is no offline typecheck of the proxy-facing code — the only typecheck is against a live SUT (post-deploy in CI). The util layer (api-util, api-test-util), which does not import generated, plus lint, quarantine, and the import type-only unit tests, remain hermetic. For a direction-aware breaking/non-breaking classification (mapping to the major/minor CHANGELOG decision), a spec differ such as oasdiff is the natural companion to the compile gate — planned as a follow-up, not a prerequisite.

Test Categories

Every test declares a category via a filename suffix — a selection concern (which files a run loads). Categories split into a cumulative progression plus orthogonal additional sets:

Category	Suffix	Meaning
probe	*.probe.test.ts	Minimal reachability + presence of a minimal endpoint set (e.g. the configuration route). Renamed from access to avoid the authorization connotation.
sanity	*.sanity.test.ts	Basic happy-path workflows — the SUT has core capabilities and no major regression.
acceptance	*.acceptance.test.ts	The acceptance criteria to release a version to production.
functional	*.functional.test.ts	Extensive; each focused on a specific workflow, including business-valid non-happy paths.
stress	*.stress.test.ts	Robustness at the edges (invalid inputs, break attempts).
bug	pdev-<n>-<slug>.bug.test.ts	Reproduces a reported bug; the ticket number is in the filename.
extra	*.extra.test.ts	Specialized / ad-hoc.

Progression (cumulative): probe ⊆ sanity ⊆ acceptance ⊆ ALL. Gates map to glob unions (e.g. the acceptance gate loads probe + sanity + acceptance suffixes). functional, stress, bug, extra are additional sets, wired into different gates/processes. Category overlap needs no multi-tagging: because tests are leaves (DQ-013), a workflow needed in two categories lives once in shared/util/test and is composed by two thin leaf files.

SUT Characterization and the Test Runner

SUT configuration (prominent, at the test root)

A single committed sut-config.ts at the root of operations/api-tests declares every deployment target. It is prominent by design — util/main supplies the types (SutConfig, the Capability enum) and helpers, but the config data is not buried in util. Because all secrets live in 1Password, the file holds only op://Arda-{Env}OAM/... references, never values — so one typed file replaces per-environment .env files.

Per-SUT fields: key, kind (local | aws), infra.partition, base URL (or a discovery function for local — dynamic port-forward/NodePort), token/Cognito endpoint, capabilities: Capability[], secret references (two tiers: API creds and OAM creds), and OAM coordinates (aws profile, kubectl context, log group, namespace — carried now even though SUT-log/Sentry monitoring is V2).

SUT keys: local, alpha002-dev, alpha002-stage, alpha001-demo, alpha001-prod. Modeled single-component (operations) with a multi-component-ready shape for a future system-api-tests, avoiding footguns but not over-building.

SUT descriptor (probed at runtime)

Resolved once (globalSetup) for provenance and optional gating: component deployment version = the repository tag the target was deployed from (AWS deploys are GHA-against-a-tag), with an allowance for local, untagged execution (git sha / local-untagged); per-module API versions (OpenAPI info.version); present modules. The true repo tag may require the OAM path (kubectl/helm release) with /operations/oam/version as the HTTP fallback (the operations component version is only indicative — see the component-version skill). The probe tier reconciles declared capabilities against what is reachable, best-effort.

Capabilities

Capabilities are declared in sut-config.ts as the source of truth (a closed Capability enum shared by the config and each test’s requires, so an unknown label is a compile error). A test runs only if requires ⊆ SUT.capabilities; otherwise it is skipped (capability) and reported. Discovery/reconciliation is best-effort only — some capabilities are uncheckable and rely on declaration; diagnose from failures. Qualified capabilities (e.g. production vs mock) are a deliberate future extension, not built now.

Runner (a thin CLI over vitest run)

The runner is a small tsx CLI (run-suite.ts) that resolves the SUT config, sets env (base URL, runtime-resolved secrets, SUT capability set, policy, instrumentation), maps the requested category to include-globs, and invokes vitest run with a custom reporter. It is not a new runner.

Inputs:

• required-tests — a category or an explicit list/glob.
• SUT name — key into sut-config.ts.
• --quarantine-budget N — tolerate up to N quarantined failures; 0 = any quarantined failure fails the run (replaces a separate --fail-on-quarantined).
• --missing-capabilities-fail — fast-abort the whole run if any capability required by a selected test is absent; when absent, those tests are skipped.
• instrumentation — test logger levels (error|warn|info|debug|trace) in V1; Sentry event monitoring and SUT log tailing (kubectl/aws) are V2, with their coordinates already carried in the config.

Selection: a test runs iff it matches required-tests and requires ⊆ SUT.capabilities.

Fail policy:

Outcome	Non-quarantined	Quarantined (always runs)
skipped (capability)	reported, never fails*	reported, never fails*
pass	✓	✓ + flagged “un-quarantine candidate”
fail	fails the run	counts against --quarantine-budget; over budget → fails

* unless --missing-capabilities-fail (fast-abort). A probe failure fast-aborts the run, but a probe test skipped by capability does not. Expired quarantine (past until) and exceeding the existence budget (maxQuarantined, validated by the tsx quarantine validator in make check) fail independently of runtime budget.

Quarantine is a typed option — apiTest('…', { category, requires, quarantine: { until: '<ISO date>', ticket: 'PDEV-####' } }, fn). ticket is required by the type (compile-time governance); the tsx validator adds expiry/too-far/budget checks over Vitest’s collected metadata (no SUT needed).

Report (custom Vitest reporter): the matrix above, with diagnostics for failures (k8s log excerpts when requested — V2), plus JUnit/JSON for GitHub Actions. Quarantined passes surface as un-quarantine candidates.

Packaging and Execution

V1 focus is manual execution from a local shell; GitHub Actions automation comes after. The suite is packaged as the npm workspace, installed with npm ci (+ GITHUB_TOKEN for @arda-cards deps), Node 22 pinned via .nvmrc. The runner is an npm-bin CLI (run-suite) so every context uses one entry point. No container in V1 (revisit only when the hermetic CI target in DQ-011 is built).

Secrets resolve via a 1Password service-account token read from an env var (OP_SERVICE_ACCOUNT_TOKEN, supplied in CI as a GitHub secret), with fallback to interactive op (local biometric) when the service-account token is absent or inaccessible. Locally, make secrets runs op inject over the op:// references declared in sut-config.ts and writes a git-ignored .env in a single authorization (values go to the file via -o, never to the transcript); make run sources .env, so subsequent runs need no further prompt.

CHANGELOG / versioning. The suite lives in operations (direct-edit CHANGELOG), is not in the Gradle build or Helm chart, and does not affect the deployable. Test changes take an operations CHANGELOG entry — usually under Fixed (patch). This co-versioning is a feature: because there is one version per repo, a SUT’s deployed repository tag identifies the exact suite revision that matches it (reinforcing the co-location choice, DQ-002, and the SUT descriptor’s version = repo tag). Publishing the promotable packages is deferred (structure only).

Walking Skeleton (V1 first slice)

The first slice exercises every layer once (config → runner → capability gate → proxy/client → assertion → report). It is built and green against the alpha002-dev SUT.

• Seed module: item (the richest, most-covered Bruno module).
• SUT: alpha002-dev (https://dev.alpha002.io.arda.cards). Each module serves its OpenAPI publicly (/v1/<module>/docs/openApi.json → 200), while data and OAM routes require auth (401). So the no-auth surface is enough to bootstrap. local remains available for authenticated/hermetic runs.
• probe (done): asserts the item module serves valid OpenAPI and declares the core /v1/item/item paths — runs with no credentials.
• Proxies (done): make gen-proxies --sut alpha002-dev generates item.ts from dev’s live spec into api-client-proxies/src/generated/ (regenerated per-run, git-ignored; compiles under strict TS).
• sanity (done): item create → read-back by rId, authenticated, in a freshly minted UUID tenant — green against dev. Write endpoints require Authorization + X-Author + X-Tenant-Id (a bare UUID) — enforced by the server but not in the OpenAPI params; the authed client supplies all three.
• Auth: static Bearer token via StaticApiKeyProvider; a NoAuthProvider covers public endpoints. The openid-client/Cognito provider is built behind the same TokenProvider interface but dormant until required.
• Test data: each run mints a fresh tenant ID, giving a clean namespace and exercising tenant isolation by default — so no teardown is needed. Cleanup discipline is revisited only if/when a shared environment is targeted.

OAuth2 Authentication

Operations has dual auth (verified live against dev): it accepts either the static API key or a Cognito access token as the bearer (the id token is rejected). So the OAuth2 path is exercised by swapping the TokenProvider — the same test runs under either. sutOAuthedClient() selects the provider from the SUT’s auth.oauth.mode:

SUT	oauth.mode	Provider	Token source
AWS (alpha002-dev, …)	cognito	CognitoPasswordTokenProvider	real Cognito InitiateAuth / USER_PASSWORD_AUTH (the flow api-test + arda-frontend-app use); SECRET_HASH = base64(HMAC-SHA256(clientSecret, username+clientId)); the access token is the bearer
local	local-jwt	LocalJwtTokenProvider	a self-minted RS256 JWT — no Cognito

Local OAuth (no Cognito)

Local deployments don’t use Cognito. The Helm chart sets auth.jwt.issuer = http://caddy/ and runs a Caddy sidecar that serves a static public JWKS (config/local/.well-known/jwks.json, kid: local-bruno-test-key). Local operations validates any bearer JWT against that JWKS. So the suite mints its own JWT signed with the matching dev private key, with iss: http://caddy/, aud: operations, token_use: access, and sub / custom:tenant it controls — fully hermetic, no network. This mirrors how api-test authenticates locally (create-user.bru signs a JWT with iss: http://caddy/).

The dev private key is never committed: LocalJwtTokenProvider takes it injected, resolved at runtime from a 1Password reference (base64) into E2E_JWT_PRIVATE_KEY_B64 via the secrets flow. Only the public JWKS lives in the chart. An offline unit test (local-jwt-token-provider.test.ts) generates an ephemeral keypair and proves the minted token verifies against its JWKS with the expected claims — so the minting logic is validated without any key material or a running SUT.

Caddy must be deployed locally. The chart defaults featureFlag.deploysCaddy: false; values-local.yaml now sets it true so the Caddy issuer runs (http://caddy/, serving the committed config/local/.well-known/jwks.json). Without it, local operations is configured with auth.jwt.issuer=http://caddy/ but has no issuer, so JWT auth fails. The api-key path needs none of this.

Two local-specific JWT details: local operations requires an email claim on writes (the AWS Cognito access-token path does not), which LocalJwtTokenProvider mints; and the dev private key is injected at runtime as E2E_JWT_PRIVATE_KEY_B64 (never committed).

Verified against local (docker-desktop)

The full suite runs green against a local operations over both auth modes — 28 passed, 2 capability-skipped (pdf-printing, csv-upload; local declares only baseline) each. This confirms base-URL discovery (SUT_BASE_URL from the kubectl-discovered NodePort), api-key auth (sandbox0-test-key), the local-JWT OAuth path (Caddy issuer + self-minted JWT), and live capability gating. A vitest retry:1 absorbs transient load flakiness on the single local pod. Combined with dev (30/30 over both api-key and OAuth), all four auth×SUT quadrants are green.

Credentials

• AWS: a dedicated e2e Cognito user (operations-e2e-tests@arda.com) created via SignUp and stored in Arda-DevOAM (operations-e2e-oauth-user); the app-client is op://Arda-DevOAM/BFF Client/*. Resolved into E2E_* env by make secrets.
• Local: the local signing key (base64) in Arda-LocalOAM, resolved into E2E_JWT_PRIVATE_KEY_B64.

Selecting auth per run, and auth-specific tests

Tests call the auth-neutral sutClient(), which dispatches on the runner’s --auth switch (SUT_AUTH_MODE): --auth api-key (default) uses the static key, --auth oauth uses the SUT’s OAuth provider. So the whole suite runs under either auth with no code change (make run ARGS="… --auth oauth").

Some tests are auth-specific — e.g. the Postmark webhook, whose bearer is always the component API key, never OAuth. They declare it in the meta: apiTest('…', { auth: 'api-key' }, …) (or auth: ['api-key', 'oauth'], or 'oauth'). When the run’s mode isn’t among the declared modes the test is skipped (reported, name-tagged [auth:…]), or fails if the runner passes --auth-mismatch-fail. This mirrors capability gating, but on the auth axis.

Open Points

• Domain/module taxonomy. Names beyond the item seed are settled as more modules are ported.
• CI execution target — see DQ-011 (both hermetic and live-environment runs likely needed; follow-up this session).
• Quarantine budget semantics — assumed: --quarantine-budget bounds runtime quarantined failures; maxQuarantined (config) bounds existence. To confirm.

---

Copyright: (c) Arda Systems 2025-2026, All rights reserved