Skip to content

Verification: PDEV-490 Operations Performance Improvements

Author: Claude Opus for jmpicnic | Date: 2026-05-19 | Status: Draft

Verification: PDEV-490 Operations Performance Improvements

Section titled “Verification: PDEV-490 Operations Performance Improvements”

Bidirectional traceability between the requirements and the acceptance criteria that verify them. Each acceptance criterion specifies a verification method (code review, unit test, integration test, EXPLAIN ANALYZE, pg_stat_statements comparison, Sentry rolling-window measurement, or the dev synthetic-failover gate). The Status column tracks verification progress during implementation; the implementer updates each row from Pending to Verified (or Blocked with a note) as each criterion is exercised.

Note on AC numbering: acceptance criteria are numbered sequentially within sections (AC-JDBC-1, AC-IDX-1, etc.) so the prefix carries the domain. Where an AC covers more than one requirement, multiple REQs are listed.


ACREQAcceptance CriterionVerification MethodStatus
AC-JDBC-1REQ-PDEV490-001common-module/gradle/libs.versions.toml declares the aws-jdbc-wrapper version (4.0.1) and library; the persistence module’s build.gradle.kts declares api(libs.aws.jdbc.wrapper).Code review of gradle/libs.versions.toml + build.gradle.kts.Pending
AC-JDBC-2REQ-PDEV490-002common-module/lib/src/main/kotlin/cards/arda/common/lib/persistence/DataSource.kt reads dbConfig.url as authoritative and computes an isWrapperScheme flag via dbConfig.url.toString().startsWith("jdbc:aws-wrapper:"). URL construction is not performed inside common-module.Code review.Pending
AC-JDBC-3REQ-PDEV490-003DataSource.kt sets driverClassName = "software.amazon.jdbc.Driver" if and only if isWrapperScheme is true. A unit test constructs DataSource with each scheme and asserts the conditional.Code review + unit test.Pending
AC-JDBC-3bREQ-PDEV490-002, REQ-PDEV490-003, REQ-PDEV490-004, REQ-PDEV490-005, REQ-PDEV490-006A unit test (DataSourceTest) constructs DataSource with a legacy jdbc:postgresql: URL and asserts: no driverClassName override, no wrapperPlugins property, no exceptionOverrideClassName override, no Aurora tuning properties. Confirms strict additivity — zero behavior change for legacy consumers.Unit test.Pending
AC-JDBC-4REQ-PDEV490-004DataSource.kt sets wrapperPlugins = "auroraInitialConnection,failover2,efm2,readWriteSplitting" on the HikariCP dataSourceProperties if and only if isWrapperScheme is true.Code review + unit test + pod startup log inspection on dev (confirm the plugin pipeline appears in wrapper-emitted log lines when the operations consumer opts in).Pending
AC-JDBC-5REQ-PDEV490-005, REQ-PDEV490-006When isWrapperScheme is true, DataSource.kt sets exceptionOverrideClassName = "software.amazon.jdbc.util.HikariCPSQLException" AND the five Aurora-tuning properties (failoverClusterTopologyRefreshRateMs = 2000, failoverReaderConnectTimeoutMs = 5000, failoverWriterReconnectIntervalMs = 2000, loadBalanceReadOnlyTraffic = true, readerInitialConnectionHostSelectorStrategy = leastConnections).Code review + unit test.Pending
AC-JDBC-6REQ-PDEV490-007operations/gradle/libs.versions.toml has the commonModule pin updated to the Phase-2 release tag, and the operations build succeeds against that release.Code review + make -C operations clean build exit code 0.Pending
AC-JDBC-7REQ-PDEV490-008operations/src/main/resources/application.conf has dataSource.jdbcUrl matching the jdbc:aws-wrapper:postgresql://… scheme.Code review.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-RW-1REQ-PDEV490-010During a representative items-page workload exercised against dev, pod-side wrapper logging shows read-only transactions (transaction(readOnly = true)) connecting to an Aurora reader instance.Pod log inspection on Alpha002-dev after Wave 3 deploy.Pending
AC-RW-2REQ-PDEV490-011During the same workload, write-transactions land on the Aurora writer instance.Pod log inspection on Alpha002-dev.Pending
AC-RW-3REQ-PDEV490-012The application-level HikariCP pool size, eviction policy, and caller-facing API surface are unchanged. The existing operations integration-test suite continues to pass without modification.Code review of DataSource.kt’s HikariCP pool block + existing test suite pass.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-IDX-K1REQ-PDEV490-020, REQ-PDEV490-021, REQ-PDEV490-022, REQ-PDEV490-023operations/src/main/resources/resources/kanban/database/migrations/V007__kanban_card_bitemporal_indexes.sql declares the three indexes using CREATE INDEX CONCURRENTLY with no WHERE retired = FALSE predicate. Sidecar .conf sets executeInTransaction=false.Code review of migration + sidecar file.Pending
AC-IDX-K2REQ-PDEV490-020, REQ-PDEV490-021Post-deploy on dev, EXPLAIN ANALYZE on the bitemporal SELECT for kanban_card (cardsForItem-shaped query) shows Index Scan using idx_kanban_card_tenant_item_temporal on kanban_card sq (not Seq Scan, not Bitmap Index Scan on idx_kanban_card_eid).EXPLAIN ANALYZE against Alpha002-dev after Wave 1 migration.Pending
AC-IDX-K3REQ-PDEV490-022\d kanban_card (or equivalent pg_indexes query) on dev shows idx_kanban_card_tenant_id.psql introspection on Alpha002-dev.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-IDX-I1REQ-PDEV490-025, REQ-PDEV490-026operations/src/main/resources/reference/item/database/migrations/V015__item_bitemporal_indexes.sql (or next-available V* number) declares the composite index(es) using CREATE INDEX CONCURRENTLY with no WHERE retired = FALSE predicate. Sidecar .conf sets executeInTransaction=false.Code review.Pending
AC-IDX-I2REQ-PDEV490-025Post-deploy on dev, EXPLAIN ANALYZE on the bitemporal SELECT for item (listWithDetails-shaped query) shows Index Scan using idx_item_tenant_eid_temporal on item sq.EXPLAIN ANALYZE against Alpha002-dev.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-K12-1REQ-PDEV490-030, REQ-PDEV490-031, REQ-PDEV490-032operations/src/main/kotlin/.../ServiceImpl.kt:276-293 matches the “After” shape in specification.md § Task 1a.2: withTotal = false, no flatMap/when block, bare inTransaction { universe.list(…)() }. Both changes appear in a single commit (git log shows one commit touching both the flag and the block).Code review + git-log inspection of the Wave 1 kanban PR.Pending
AC-K12-2REQ-PDEV490-033Integration test cardsForItem on an item with zero cards returns HTTP 200 with Page(records = [], totalCount = null).Integration test in operations/src/test/kotlin/.../ServiceImplTest.kt.Pending
AC-K12-3REQ-PDEV490-030, REQ-PDEV490-031Integration test cardsForItem on an item with multiple cards returns HTTP 200 with Page(records = [...N cards...], totalCount = null).Integration test.Pending
AC-K12-4REQ-PDEV490-NFR-022Post-deploy on dev, pg_stat_statements ranked by call frequency on the kanban DB no longer shows the COUNT statement formerly issued by cardsForItem in the top entries (it was previously near the top).pg_stat_statements snapshot on Alpha002-dev before/after Wave 1.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-TX-1REQ-PDEV490-040common-module/lib/src/main/kotlin/.../AppError.kt declares AppError.Transient sealed branch under AppError.Internal with three subtypes (FailoverSucceeded, TransactionStateUnknown, FailoverFailed).Code review.Pending
AC-TX-2REQ-PDEV490-041common-module/lib/src/main/kotlin/cards/arda/common/lib/lang/errors/AppError.kt:192 Throwable.normalizeToAppError() has new branches that classify each of the three wrapper exception classes (raw and ExposedSQLException-wrapped) to the corresponding AppError.Transient subtype. Unrelated throwables continue to map to the existing AppError.Implementation / AppError.GeneralValidation branches.Code review + unit tests on normalizeToAppError.Pending
AC-TX-3REQ-PDEV490-042, REQ-PDEV490-044Forced-transient integration test (TransientFailureContractTest — Test A): a forced FailoverSuccessSQLException produces HTTP 503 with Retry-After: 2 header and response body matching ErrorResponse(code = 503, …).Integration test on ContainerizedPostgres harness with maxAttempts = 1.Pending
AC-TX-4REQ-PDEV490-043Existing tests for HTTP 500 rendering on AppError.Internal subtypes other than Transient (e.g., IncompatibleState) continue to pass without modification.Existing test suite pass.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-RT-1REQ-PDEV490-050common-module/lib/src/main/kotlin/.../PoolConfig.kt has new fields maxAttempts: Int = 2 and backoffMs: Long = 300.Code review.Pending
AC-RT-2REQ-PDEV490-051, REQ-PDEV490-053Retry-absorbed integration test (TransientFailureContractTest — Test B): a one-shot FailoverSuccessSQLException followed by a healthy connection produces HTTP 200 with the normal happy-path response.Integration test on ContainerizedPostgres harness with default maxAttempts = 2.Pending
AC-RT-3REQ-PDEV490-052Unit test: a non-transient throwable (e.g., IllegalStateException) surfaces immediately without retry.Unit test on the retry wrapper.Pending
AC-RT-4REQ-PDEV490-054operations/src/main/resources/application.conf declares dataSource.pool.maxAttempts = 2 and dataSource.pool.backoffMs = 300.Code review.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-DEC-1REQ-PDEV490-060common-module/lib/src/main/kotlin/.../AbstractScopedUniverse.kt:27 declares the tenantId column as uuid(ScopedMetadata.COLUMN_TENANT_ID) with no chained .index(…).Code review.Pending
AC-DEC-2REQ-PDEV490-061grep -rn '\.index(' common-module/lib/src/main/kotlin/.../universe/ against the post-change tree returns no new declarations (no Exposed-level index declarations added or modified anywhere else under universe/).Code review + grep.Pending
AC-DEC-3REQ-PDEV490-062common-module/lib/src/main/kotlin/.../persistence/rdbms/DbMigration.kt:31 FluentConfiguration chain includes .mixed(true) alongside .group(true). The operations test container (which has all migrations V001..V007 pending) applies the migration tree without raising FlywayMigrateException: Detected both transactional and non-transactional migrations.Code review + green operations build after the operations PR’s commonModule pin bumps to the Wave 2 release.Pending
AC-DEC-4REQ-PDEV490-063common-module/lib/.../persistence/rdbms/DataSource.kt companion has internal val allCreatedPools registering each HikariDataSource returned by newSqlDataSource(). common-module/lib/.../testing/persistence/PoolRegistry.kt provides closeAllPoolsForTests() (test-only helper with explicit “DO NOT CALL FROM PRODUCTION CODE” KDoc). ContainerizedPostgres.stop() invokes PoolRegistry.closeAllPoolsForTests(). Verification: the operations full test suite (make -C operations build) completes without hanging under Gradle’s default parallel test execution on a multi-core developer machine; CI continues to pass.Code review + green local make -C operations build end-to-end without manual --max-workers=1 workaround.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-FO-1REQ-PDEV490-070, REQ-PDEV490-NFR-012During the Wave 5 dev synthetic-failover test, the wrapper’s topology log shows the new writer instance is detected within ~2-5 seconds of the failover trigger.Pod log inspection during Wave 5 test.Pending
AC-FO-2REQ-PDEV490-071, REQ-PDEV490-NFR-010During the Wave 5 test, the steady-load probe shows the HTTP 5xx window lasts ≤ 5 seconds (down from ~30 seconds today).Sentry transaction log + probe-side metrics during Wave 5.Pending
AC-FO-3REQ-PDEV490-072, REQ-PDEV490-NFR-011During the Wave 5 5xx window, HTTP 503 dominates the distribution and HTTP 500 is essentially zero.Sentry transaction log during Wave 5.Pending
AC-FO-4REQ-PDEV490-073The Wave 4 helm chart diff (no chart change beyond common-module consumer wiring) confirms the JVM-level networkaddress.cache.ttl setting is unchanged.Code review of operations helm chart (src/main/helm/values*.yaml).Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-AUD-1REQ-PDEV490-080grep -rnE 'SQLException|ExposedSQLException|PSQLException' src/main/kotlin/ on the operations worktree (Wave 3 PR branch) returns zero hits. Audit recorded in the operations Wave 3 PR description.Grep snapshot included in PR body.Pending
AC-AUD-2REQ-PDEV490-085The tenant_id audit table from analysis.md § Current state / Index coverage is referenced from the operations PR description (or its CHANGELOG entry) as the source of REQ-PDEV490-022’s scope.PR description review.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-DOC-1REQ-PDEV490-090documentation/src/content/docs/current-system/architecture/patterns/persistence/aurora-jdbc-wrapper.md exists with frontmatter and content; rendered preview accessible via make -C documentation preview.Doc-PR review + make pr-checks.Pending
AC-DOC-2REQ-PDEV490-091documentation/src/content/docs/current-system/architecture/patterns/persistence/bitemporal-indexes.md exists with frontmatter and content.Doc-PR review + make pr-checks.Pending
AC-DOC-3REQ-PDEV490-092documentation/src/content/docs/process/craft/database/flyway-authoritative-for-indexes.md exists with frontmatter and content.Doc-PR review + make pr-checks.Pending
AC-DOC-4REQ-PDEV490-093documentation/src/content/docs/current-system/architecture/patterns/api-design.md has a Transient failures and retry contract section documenting the HTTP 503 + Retry-After: 2 surface.Doc-PR review.Pending
AC-DOC-5REQ-PDEV490-094Runbooks present under documentation/src/content/docs/process/operation-notes/: 20260519-aurora-synthetic-failover-test.md (Wave 5 acceptance test), 20260519-aurora-wrapper-troubleshooting.md, and 20260514-aurora-parameter-group-and-operations-bump-rollout.md (rollout runbook, already on this branch).Doc-PR review.Pending
AC-DOC-6REQ-PDEV490-095The Wave 4 documentation PR merges before the Wave 5 dev synthetic-failover test is executed (Wave 3 and Wave 4 can be developed in parallel; the runbook is drafted from the spec and may be refined post-Wave-3-deploy if the deployed shape diverges).Doc-PR review + git-log timing inspection.Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-PERF-1REQ-PDEV490-NFR-001Seven days after PDEV-490 has been promoted to Alpha001-prod, Sentry platform-be transaction-duration p50 for GET /v1/kanban/.../kanban-card/for-item/{item-eid} ≤ 250 ms and p95 ≤ 1,000 ms on a rolling seven-day window. Baseline (2026-05-19): p50 = 1,113 ms, p95 = 2,911 ms.Sentry rolling-window query on platform-be post-rollout.Pending
AC-PERF-2REQ-PDEV490-NFR-002Seven days after PDEV-490 has been promoted to Alpha001-prod and the PDEV-489 front-end consolidation has migrated the items-page consumer off this route (out of scope for PDEV-490), Sentry platform-be transaction-duration p50 for POST /v1/kanban/.../kanban-card/details ≤ 250 ms and p95 ≤ 1,000 ms on a rolling seven-day window. Baseline (2026-05-19): p50 = 289 ms, p95 = 2,035 ms. The conditional dependency on PDEV-489 is explicit; this criterion may be deferred until both projects have shipped.Sentry rolling-window query.Pending (conditional on PDEV-489)
ACREQAcceptance CriterionVerification MethodStatus
AC-SQL-1REQ-PDEV490-NFR-020EXPLAIN ANALYZE on the post-Wave-1 kanban_card bitemporal SELECT (representative cardsForItem-shaped query) shows index scan / index-only scan on the inner correlated subquery. Captured before-and-after in the Wave 1 kanban PR description.EXPLAIN ANALYZE on dev.Pending
AC-SQL-2REQ-PDEV490-NFR-021EXPLAIN ANALYZE on the post-Wave-1 item bitemporal SELECT (representative listWithDetails-shaped query) shows index scan / index-only scan on the inner correlated subquery.EXPLAIN ANALYZE on dev.Pending
AC-SQL-3REQ-PDEV490-NFR-022Covered by AC-K12-4 above (pg_stat_statements confirms the COUNT statement disappears from top-N).(cross-reference)Pending
ACREQAcceptance CriterionVerification MethodStatus
AC-BLD-1REQ-PDEV490-NFR-030make -C common-module clean build exit code 0 on the Wave 2 release PR; all existing tests pass.Build output on Wave 2 PR.Pending
AC-BLD-2REQ-PDEV490-NFR-031make -C operations clean build exit code 0 on each of the three operations PRs (Wave 1 kanban, Wave 1 item, Wave 3 consumer); all existing tests pass.Build output on each PR.Pending
AC-BLD-3REQ-PDEV490-NFR-032Coverage thresholds in the Gradle build scripts continue to be met on both common-module and operations.Coverage report from ./gradlew build.Pending
AC-BLD-4REQ-PDEV490-NFR-033make -C documentation pr-checks exit code 0 on the Wave 4 PR.Documentation build output on Wave 4 PR.Pending
AC-CHG-1REQ-PDEV490-NFR-040, REQ-PDEV490-NFR-041, REQ-PDEV490-NFR-042Each PR carries the CHANGELOG entry specified in specification.md Tasks 1a.4 / 1b.3 (combined into the operations PR), 2.9, 3.5 (combined into the operations PR), and 4.3. common-module’s entry presents the wrapper integration and mixed=true enablement as Added (strictly additive — consumers on jdbc:postgresql: URLs see zero behaviour change). The operations PR entry calls out the 500→503 status-code remap (which only takes effect after the consumer opts in to the wrapper scheme) as Changed.PR description / CHANGELOG review.Pending

Each requirement maps to at least one acceptance criterion.

RequirementACsCovered
REQ-PDEV490-001AC-JDBC-1Yes
REQ-PDEV490-002AC-JDBC-2Yes
REQ-PDEV490-003AC-JDBC-2Yes
REQ-PDEV490-004AC-JDBC-3Yes
REQ-PDEV490-005AC-JDBC-4Yes
REQ-PDEV490-006AC-JDBC-5Yes
REQ-PDEV490-007AC-JDBC-6Yes
REQ-PDEV490-008AC-JDBC-7Yes
REQ-PDEV490-010AC-RW-1Yes
REQ-PDEV490-011AC-RW-2Yes
REQ-PDEV490-012AC-RW-3Yes
REQ-PDEV490-020AC-IDX-K1, AC-IDX-K2Yes
REQ-PDEV490-021AC-IDX-K1, AC-IDX-K2Yes
REQ-PDEV490-022AC-IDX-K1, AC-IDX-K3Yes
REQ-PDEV490-023AC-IDX-K1Yes
REQ-PDEV490-025AC-IDX-I1, AC-IDX-I2Yes
REQ-PDEV490-026AC-IDX-I1Yes
REQ-PDEV490-030AC-K12-1, AC-K12-3Yes
REQ-PDEV490-031AC-K12-1, AC-K12-3Yes
REQ-PDEV490-032AC-K12-1Yes
REQ-PDEV490-033AC-K12-2Yes
REQ-PDEV490-040AC-TX-1Yes
REQ-PDEV490-041AC-TX-2Yes
REQ-PDEV490-042AC-TX-3Yes
REQ-PDEV490-043AC-TX-4Yes
REQ-PDEV490-044AC-TX-3Yes
REQ-PDEV490-050AC-RT-1Yes
REQ-PDEV490-051AC-RT-2Yes
REQ-PDEV490-052AC-RT-3Yes
REQ-PDEV490-053AC-RT-2Yes
REQ-PDEV490-054AC-RT-4Yes
REQ-PDEV490-060AC-DEC-1Yes
REQ-PDEV490-061AC-DEC-2Yes
REQ-PDEV490-062AC-DEC-3Yes
REQ-PDEV490-063AC-DEC-4Yes
REQ-PDEV490-070AC-FO-1Yes
REQ-PDEV490-071AC-FO-2Yes
REQ-PDEV490-072AC-FO-3Yes
REQ-PDEV490-073AC-FO-4Yes
REQ-PDEV490-080AC-AUD-1Yes
REQ-PDEV490-085AC-AUD-2Yes
REQ-PDEV490-090AC-DOC-1Yes
REQ-PDEV490-091AC-DOC-2Yes
REQ-PDEV490-092AC-DOC-3Yes
REQ-PDEV490-093AC-DOC-4Yes
REQ-PDEV490-094AC-DOC-5Yes
REQ-PDEV490-095AC-DOC-6Yes
REQ-PDEV490-NFR-001AC-PERF-1Yes
REQ-PDEV490-NFR-002AC-PERF-2Yes (conditional on PDEV-489)
REQ-PDEV490-NFR-010AC-FO-2Yes
REQ-PDEV490-NFR-011AC-FO-3Yes
REQ-PDEV490-NFR-012AC-FO-1Yes
REQ-PDEV490-NFR-020AC-SQL-1Yes
REQ-PDEV490-NFR-021AC-SQL-2Yes
REQ-PDEV490-NFR-022AC-K12-4, AC-SQL-3Yes
REQ-PDEV490-NFR-030AC-BLD-1Yes
REQ-PDEV490-NFR-031AC-BLD-2Yes
REQ-PDEV490-NFR-032AC-BLD-3Yes
REQ-PDEV490-NFR-033AC-BLD-4Yes
REQ-PDEV490-NFR-040AC-CHG-1Yes
REQ-PDEV490-NFR-041AC-CHG-1Yes
REQ-PDEV490-NFR-042AC-CHG-1Yes

Every functional and non-functional requirement is covered by at least one acceptance criterion. No requirement is uncovered.


CategoryCriteriaCode ReviewUnit / Integration TestEXPLAIN / pg_statSentry / OperationalBuild / CI
JDBC connectivity77001 (pod log)1
Read / write splitting31102 (pod log)0
Bitemporal indexes — kanban_card310200
Bitemporal indexes — item210100
cardsForItem cleanup412100
Transient error contract421001
Retry policy421001
Decorative declaration removal220000
Failover behaviour41003 (Wave 5)0
Operations audit220000
Documentation660000
Performance — post-rollout200020
Performance — SQL plan300300
Build / test / CHANGELOG510004
Total51275787

Verification mix favours code-review-based gates (53%, mostly mechanical checks against the specification), followed by operational measurements (Sentry + pod log inspection during Wave 5 — 16%), EXPLAIN ANALYZE / pg_stat_statements checks (14%), build / CI gates (14%), and unit / integration tests (~10%). Few code-side tests is consistent with a project that is mostly additive (indexes, new error branch, new wiring) on top of an existing well-tested persistence layer.


During implementation, each row’s Status column is updated:

  • Pending — not yet exercised.
  • In Progress — implementation underway; verification artefact not yet produced.
  • Verified — verification artefact present; AC satisfied.
  • Blocked — verification cannot proceed; note inline (e.g., depends on Wave 5).
  • Failed — verification artefact present; AC not satisfied. Implementation must address.

The implementer or sub-agent updates this file as each AC is exercised. A final verification pass (driven by the verify-operations-project skill) reads the matrix and confirms every AC has reached Verified before the project promotes to roadmap/completed/.


  • PDEV-490 goal — project goal and success criteria.
  • analysis.md — entry-state analysis (where each AC’s “before” measurement comes from).
  • requirements.md — EARS requirements (the source of every REQ-PDEV490-* ID).
  • specification.md — phased implementation plan (where each REQ-PDEV490-* is realised, including the tasks each AC validates).

Copyright: (c) Arda Systems 2025-2026, All rights reserved