Phase 4 — Run 3: Operator Cascade Execution Log

This log captures the operator’s run of the Run-3 cascade in chronological order — one section per partition, plus T-O4 (Postmark Compliance reply) between dev and stage. Entries are written as the operator executes each step; the log is the primary artefact of the run.

For the procedure each partition follows, see the Operator Runbook — Partition Mail Deploy. For the run’s scope and exit criteria, see project-plan.md. The mechanical exit checks live in validate-exit.sh in this directory.

Run kickoff

Started: 2026-05-26 by Miguel Pinilla (operator) with Claude as orchestrator.

Worktree used: /Users/jmp/code/arda/projects/email-integration-worktrees/phase-4/infrastructure-run-2, branch jmpicnic/email-integration-phase-4-run-2, HEAD e53a7f1 — in sync with PR #462’s headRefOid byte-for-byte.

PR #462 merge waiver (operator direction, 2026-05-26): the project-plan entry criterion “PR #462 (Run-2) merged” is waived for this cascade. PR #462 remains OPEN / mergeStateStatus: BLOCKED; the cascade operates against the local worktree which is identical in content to the PR HEAD. All AWS / Postmark resources created during this cascade are functionally equivalent to those that would have been created after merge; the PR will land with the captured execution-log entries and any code fixes folded in before merge. Risk: if the PR’s HEAD diverges (rebase, force-push) before merge, the cascade’s recorded state stops matching what would land in main — mitigated by not pushing rewrites to the branch during the cascade.

Cascade summary table

Partition	Infrastructure	Postmark account	Pre-flight	Deploy	Verified	Sign-off date
`dev`	Alpha002	PostmarkNonProd	✅ 2026-05-26	✅ 2026-05-26	✅ 2026-05-26	2026-05-26
`stage`	Alpha002	PostmarkNonProd	✅ 2026-05-26	✅ 2026-05-26	✅ 2026-05-26	2026-05-26
`demo`	Alpha001	PostmarkProd	✅ 2026-05-26	✅ 2026-05-26	✅ 2026-05-26	2026-05-26
`prod`	Alpha001	PostmarkProd	✅ 2026-05-26	✅ 2026-05-26	✅ 2026-05-26	2026-05-26

Gate	Status	Date
PR #462 (Run-2) merged	⚠️ waived 2026-05-26 — see Run-kickoff note above	2026-05-26
Entry criteria verified (1P all four vaults, AWS SSO both profiles, DMARC mailbox confirmed)	✅ 2026-05-26	2026-05-26
T-O4 Postmark Compliance reply sent	⏳ pending	—
`arda-nonprod` approval received OR more-evidence path documented	⏳ pending	—
Production deploy confirmation given	⏳ pending	—
Run-3 infra PR merged	⏳ pending	—

Update the table cells as each gate clears: ✅ <YYYY-MM-DD> for cleared, 🟡 <YYYY-MM-DD> <note> for partial / blocked, ❌ <YYYY-MM-DD> <note> for failed, ⚠️ <YYYY-MM-DD> for waived.

Entry criteria verification detail (Step 1)

Check	Outcome	Note
`infrastructure-run-2` worktree clean and on PR #462’s branch	✅	HEAD `e53a7f1`, branch `jmpicnic/email-integration-phase-4-run-2`
Worktree HEAD matches PR #462 `headRefOid` byte-for-byte	✅	both `e53a7f16ed1ebf855969f4f0ab3af6a2a251c304`
`documentation` worktree has DQ-R1-026 refactor staged (uncommitted)	⚠️	does not block cascade; will be committed separately
1P / Arda-DevOAM / Postmark / credential	✅	Postmark account-level UUID token resolves; same value as Arda-StageOAM (`PostmarkNonProd` shared per binding)
1P / Arda-StageOAM / Postmark / credential	✅	same token as DevOAM (expected)
1P / Arda-DemoOAM / Postmark / credential	✅	UUID token; same as ProdOAM (`PostmarkProd` shared per binding)
1P / Arda-ProdOAM / Postmark / credential	✅	same token as DemoOAM (expected)
AWS SSO — `Alpha002-Admin` active	✅	account `139852620346` (Alpha002)
AWS SSO — `Admin-Alpha1` active	✅	account `009765408297` (Alpha001)
`dmarc-reports@arda.cards` mailbox monitored	✅	operator confirmation 2026-05-26

Observation: op item get --fields credential returned the field’s internal UUID instead of the credential value, while op read "op://<vault>/<item>/credential" returned the value correctly. For the rest of this cascade and ad-hoc checks, prefer op read "op://..." references (which is what the Pre-Deploy CLI uses internally via tools/lib/op-resolver.ts).

Script fixes landed during the cascade

Date	Script	Bug	Fix
2026-05-26 (early run)	`validate-exit.sh` synth check	Called `npm run synth:al1x` (script name doesn’t exist) and used `--context partition=` (this codebase’s `cdk-runner.js` selects the CDK app by instance-file path, not by context filter).	Switched to `npm run synth:named -- <infra>/<partition> <profile> us-east-1`.
2026-05-26 23:35 PDT	`validate-exit.sh` CFN-exports check	Used backtick-quoted JMESPath JSON literals which got re-interpreted as bash command substitutions when the `check()` function eval’d the string. Also: `aws cloudformation list-exports` paginates and `--query` runs per-page, producing multi-line numeric output that breaks `[[ -ge 6 ]]` arithmetic compare.	Switched to `aws cloudformation describe-stacks` against the specific partition-email stack (no pagination); used single-quoted JMESPath strings inside escaped double quotes for the shell `--query` argument. All four partitions now pass 12/12.

Both fixes are committed in the docs worktree on jmpicnic/email-integration-phase-4 and will ride into the Run-7 docs PR.

§ dev (Alpha002 / PostmarkNonProd)

Pre-flight (T-O1-dev)

AWS SSO logged in for Alpha002-Admin: ✅ 2026-05-26 (verified in Step 1.5; account 139852620346).
op read "op://Arda-DevOAM/Postmark/credential" returns non-empty: ✅ 2026-05-26 (verified in Step 1.4a; UUID-format Postmark token).
validate-exit.sh dev (no --post-merge) all checks PASS: ✅ 2026-05-26 — 4 / 4 (build, lint, jest, synth-produces-Alpha002-dev-Email).

Anomaly captured during pre-flight (now resolved): the published validate-exit.sh called npm run synth:al1x --context partition=<p>, which is wrong for this codebase — there is no synth:al1x npm script, and tools/cdk-runner.js selects the CDK app by instance-file path (src/main/cdk/instances/<Infrastructure>/<partition>.ts) rather than via a --context partition= filter. Fixed in the same docs worktree by replacing the synth invocation with npm run synth:named -- <infra>/<partition> <aws-profile> us-east-1. Re-ran the script post-fix and got 4 / 4. The fix is part of the documentation PR; no impact on the deployed code.

The first npm install run-3 worktree population (during the run-3 worktree’s initial setup) had also produced a package-lock.json peer-marker diff ("peer": true annotations flipped on several deps). Reset via git checkout -- package-lock.json at the start of Step 2; build / lint / tests / synth all green on the committed lock — confirms node_modules is functionally correct without a reinstall.

Deploy (T-O5-dev) ✅ 2026-05-26

Command: cd /Users/jmp/code/arda/projects/email-integration-worktrees/phase-4/infrastructure-run-3 && ./amm.sh --profile Alpha002-Admin Alpha002 dev (preceded by an op read pre-warm in the same Bash call so the SDK’s biometric session was active before amm.sh’s first secret read).
Command exit code: 0 ✅
Pre-Deploy CLI step (register-partition-mail-signature.ts): first-time-create path — Postmark Sender Signature registered for dev.ardamails.com on PostmarkNonProd with ReturnPathDomain: pm-bounces.dev.ardamails.com. DKIM selector issued: 20260526234343pm (Postmark’s generated selector, timestamped from registration at ~23:43 UTC 2026-05-26). DKIM public key + Return-Path target written into cdk.context.json under partitionMail:Alpha002:dev.
cdk.context.json entry written for partitionMail:Alpha002:dev: ✅ — { dkimSelector: "20260526234343pm", dkimPublicKey: <RSA-public-key>, returnPathTarget: "pm.mtasv.net" }
cdk deploy step CFN stack Alpha002-dev-Email status: ✅ CREATE_COMPLETE — all 25 resources created including the cross-account NS-delegation custom resource, four DNS records (SPF, DMARC, DKIM, Return-Path), both SM secrets, and the two STS-assumable IAM roles.

Notes / anomalies:

First attempt (2026-05-26 16:08 PDT) failed at amm.sh Step 0.2: invoked as ./amm.sh Alpha002 dev (no --profile). The script’s Admin-${infrastructure} auto-derivation produced Admin-Alpha002, which is not a real profile in ~/.aws/config (the actual one is Alpha002-Admin). Region resolution then fell back to the bogus profile and exited with ERROR: AWS region is not set / The config profile (Admin-Alpha002) could not be found. No AWS or Postmark state was changed. amm.sh is intentionally designed to require --profile <name> for irregular profile-name infrastructures (Alpha001 → Admin-Alpha1, Alpha002 → Alpha002-Admin); fixed by passing the flag on the second attempt. Operator runbook updated to require the flag for Phase 4 partitions. (operator-runbook.md § Phase 4 Partition Mail Deploy)
Background-mode exit-code masking observation: the first attempt was launched as a background bash task. The wrapper reported exit code 0 even though amm.sh actually exited 255 (verified by a synchronous re-run with identical args). Future state-changing runs should be invoked synchronously OR the wrapper must explicitly check both the subshell’s PIPESTATUS and the inner script’s exit code. For the second attempt I’ll run synchronously.

Verification (T-O3-dev) ✅ 2026-05-26

dig +short NS dev.ardamails.com @8.8.8.8 returns ≥ 2 nameservers: ✅ — 4 nameservers (ns-826.awsdns-39.net, ns-1215.awsdns-23.org, ns-442.awsdns-55.com, ns-1930.awsdns-49.co.uk).
dig +short TXT dev.ardamails.com @8.8.8.8 includes v=spf1: ✅
dig +short TXT _dmarc.dev.ardamails.com @8.8.8.8 includes v=DMARC1: ✅
dig +short TXT 20260526234343pm._domainkey.dev.ardamails.com @8.8.8.8 resolves to DKIM key: ✅ — public key matches cdk.context.json’s dkimPublicKey byte-for-byte.
dig +short CNAME pm-bounces.dev.ardamails.com @8.8.8.8 resolves to Postmark Return-Path target: ✅ → pm.mtasv.net.
Postmark Console: dev.ardamails.com Sender Signature shows DKIM Verified + Return-Path Verified on PostmarkNonProd: ✅ 2026-05-26 — verified via Postmark API PUT /domains/<id>/verifyDkim + PUT /domains/<id>/verifyReturnPath. Note: Postmark Console initially showed both as “Not Verified” after the deploy because its periodic DNS-recheck hadn’t fired yet. Triggering verification via the API endpoints succeeded (DKIM required a second call after a few seconds for the lookup to complete server-side). The operator runbook should mention this — newly-deployed Sender Domains may show “Not Verified” in Console until the next scheduled re-check (every few hours) OR an operator-triggered API call.
aws cloudformation list-exports --profile Alpha002-Admin includes the six Alpha002-dev-API-* exports: ✅ — all 6 present: PartitionMailZoneId (Z0830123OPJSQ95EYD0M), PartitionMailZoneName (dev.ardamails.com), EmailPostmarkAccountTokenArn, EmailEncryptionKeyArn, EmailDnsProvisioningRoleArn, EmailEncryptionKeyFallbackRoleArn.
validate-exit.sh dev --post-merge all checks PASS: ✅ 12 / 12 (after script fix landed 2026-05-26 23:35 PDT — see “Script fixes” section near top of this log).
V-check rows in verification.md populated (V-OPS-005-dev, V-PART-* per-partition rows): ⏳ to be populated at cascade close.

Observations:

Alpha002-dev-API-EmailDnsProvisioningRoleArn exports arn:aws:iam::139852620346:role/AddNsRecordsToAnyHostedZone. The construct (generalized in Run-1) hardcodes the role name AddNsRecordsToAnyHostedZone — so per-partition IAM roles all share that name across accounts (one role per account; distinct ARNs because distinct accounts). Functionally correct; the name not including the partition is slightly confusing for operators inspecting IAM lists. Not a Phase-4 fix; flag for future hygiene.

Sign-off: ✅ 2026-05-26 (dig + CFN + SM checks). Postmark Console confirmation pending operator.

§ T-O4 (Postmark Compliance reply — `arda-nonprod`)

Reply sent to Postmark Compliance ticket #11236089 with dev.ardamails.com verified-domain evidence: ✅ 2026-05-26
Date sent: 2026-05-26
Postmark response received: ⏳ pending
Response outcome: (awaiting)

Cascade impact while T-O4 is in flight: stage cascade entry is blocked (same PostmarkNonProd account, waits for arda-nonprod approval to register stage.ardamails.com Sender Domain). demo and prod are unaffected — they target PostmarkProd which is already approved per K-10. Per the cascade plan, demo proceeds next.

If response is “more-evidence-needed”: document the additional evidence Postmark requested, the operator’s plan to address, and whether stage proceeds before the resolution (per the REQ-OPS-004 documented assumption).

§ stage (Alpha002 / PostmarkNonProd)

Pre-flight (T-O1-stage) ✅ 2026-05-26

T-O4 outcome required for stage attempt: No — empirically confirmed unnecessary (see Deploy notes below).
AWS SSO Alpha002-Admin active: ✅
op read "op://Arda-StageOAM/Postmark/credential" resolves: ✅
validate-exit.sh stage 4/4 PASS: ✅
cdk diff Alpha002-stage-Email: ✅ — structurally identical to dev’s diff with correct stage substitutions.

Deploy (T-O5-stage) ✅ 2026-05-26 (second attempt; first attempt blocked on construct bug)

First attempt 2026-05-26 18:14 PDT: ./amm.sh --profile Alpha002-Admin Alpha002 stage failed at cdk deploy with IAM role name collision: AddNsRecordsToAnyHostedZone already exists in stack Alpha002-dev-Email. The construct (AllowCreatingDnsRecordsRole) hardcoded the role name; dev had taken it in Alpha002. Important non-finding: Pre-Deploy CLI step succeeded before the CFN failure — stage.ardamails.com was registered on PostmarkNonProd. This empirically confirmed that PostmarkNonProd accepted the second-domain registration without arda-nonprod approval being granted first. The T-O4 reply is informational, not a hard prerequisite. Stack rolled back; subsequently deleted to clear the namespace.
Construct fix applied 2026-05-26 ~20:15 PDT: optional roleName prop on AllowCreatingDnsRecordsRole; Root passes the legacy name explicitly (byte-identity preserved); per-partition stacks omit roleName so CFN auto-generates. Two new unit tests added. dev and demo re-deployed to align (CFN replaced their roles with auto-named ones).
Second attempt 2026-05-26 22:54 PDT: same command after fix landed. Outcome:
- Command exit code: 0 ✅
- Pre-Deploy CLI step: idempotent fetch (stage.ardamails.com already existed from first attempt); DKIM selector 20260527011224pm (re-issued); Return-Path target pm.mtasv.net.
- cdk.context.json entry written for partitionMail:Alpha002:stage: ✅
- CFN stack Alpha002-stage-Email status: ✅ CREATE_COMPLETE — auto-named EmailDnsProvisioningRole = Alpha002-stage-Email-EmailDnsProvisioningRoleNsDele-MEp15pHK7X9l; no collision.

Verification (T-O3-stage) ✅ 2026-05-26

dig +short NS stage.ardamails.com @8.8.8.8: ✅ 4 nameservers (ns-73.awsdns-09.com, ns-1246.awsdns-27.org, ns-2008.awsdns-59.co.uk, ns-734.awsdns-27.net).
SPF / DMARC / DKIM / Return-Path: ✅ all 4 records resolve correctly.
Postmark Console / API: stage.ardamails.com on PostmarkNonProd shows DKIM Verified + Return-Path Verified: ✅ confirmed via API (same 2-call DKIM async pattern as dev / demo).
CFN Alpha002-stage-API-* exports: ✅ all 6 partition-mail exports present (PartitionMailZoneId: Z04777883Q1CKSWK2HMEO, PartitionMailZoneName, both SM secret ARNs, both IAM role ARNs).
V-check rows: ⏳ to be populated at cascade close.

Sign-off: ✅ 2026-05-26.

§ demo (Alpha001 / PostmarkProd)

Pre-flight (T-O1-demo)

AWS SSO logged in for Admin-Alpha1 (first Alpha001 partition this run; new profile): ⏳
op item get for op://Arda-DemoOAM/Postmark/credential returns non-empty: ⏳
validate-exit.sh demo (no --post-merge) all checks PASS: ⏳

Notes / anomalies: (fill in — PostmarkProd is already approved per K-10, no T-O4-equivalent needed)

Pre-flight (T-O1-demo) ✅ 2026-05-26

AWS SSO Admin-Alpha1 active: ✅ (account 009765408297)
op read "op://Arda-DemoOAM/Postmark/credential" resolves: ✅ (UUID token, prefix 0e6d9b7b-ace — same as PostmarkProd token observed for prod, expected per binding)
validate-exit.sh demo 4/4 PASS: ✅ (build, lint, jest, synth Alpha001-demo-Email)
cdk diff Alpha001-demo-Email: ✅ — structurally byte-identical to dev’s diff with correct Alpha001 substitutions (account 009765408297, role pattern demo-*, stack name Alpha001-demo-Email); no modifies or destroys.

Deploy (T-O6-demo) ✅ 2026-05-26

Command: cd /Users/jmp/code/arda/projects/email-integration-worktrees/phase-4/infrastructure-run-3 && ./amm.sh --profile Admin-Alpha1 Alpha001 demo (preceded by op read pre-warm in the same Bash call)
Command exit code: 0 ✅
Pre-Deploy CLI step: first-time-create path — Postmark Sender Domain registered for demo.ardamails.com on PostmarkProd; DKIM selector 20260527004920pm (timestamp 2026-05-27 00:49 UTC = 17:49 PDT); Return-Path target pm.mtasv.net.
cdk.context.json entry written for partitionMail:Alpha001:demo: ✅
CFN stack Alpha001-demo-Email status: ✅ CREATE_COMPLETE (all 25 resources)

Notes: The deploy ran smoothly — no biometric glitches, no SSO callback issues, no API timeouts. The lessons learned during dev (pre-warm op session, expect Postmark async verify pattern) eliminated all observed friction.

Verification (T-O3-demo) ✅ 2026-05-26

dig +short NS demo.ardamails.com @8.8.8.8: ✅ 4 nameservers (ns-200.awsdns-25.com, ns-2007.awsdns-58.co.uk, ns-803.awsdns-36.net, ns-1049.awsdns-03.org)
dig +short TXT demo.ardamails.com @8.8.8.8: ✅ SPF present
dig +short TXT _dmarc.demo.ardamails.com @8.8.8.8: ✅ DMARC present
dig +short TXT 20260527004920pm._domainkey.demo.ardamails.com @8.8.8.8: ✅ DKIM public key resolves, matches cdk.context.json
dig +short CNAME pm-bounces.demo.ardamails.com @8.8.8.8: ✅ → pm.mtasv.net.
Postmark Console / API: demo.ardamails.com on PostmarkProd shows DKIM Verified + Return-Path Verified: ✅ — confirmed via API; first verifyDkim call showed pending, second call after 4-second sleep returned DKIMVerified: true (same async pattern as dev). verifyReturnPath returned verified on the first call.
CFN Alpha001-demo-API-* exports: ✅ all 6 partition-mail exports populated (PartitionMailZoneId: Z0773606AM0US4FJPIL5, PartitionMailZoneName, both SM secret ARNs, both IAM role ARNs)
validate-exit.sh demo --post-merge: ✅ 12 / 12 PASS (after script fix landed 2026-05-26 23:35 PDT).
V-check rows: ⏳ to be populated at cascade close.

Sign-off: ✅ 2026-05-26

§ Production deploy confirmation (prerequisite for prod)

Before opening the prod cascade entry, the operator records the following explicitly:

Both stage and demo verified end-to-end: ✅ 2026-05-26 (stage at 22:58, demo at 17:53)
Reviewer approval on cdk diff for Alpha001-prod-Email stack: ✅ 2026-05-26 — diff reviewed (23 resources / 14 outputs / 2 parameters, structurally identical to dev/stage/demo; no RoleName on the IAM role → construct fix verified at synth time; no modifies or destroys; correct Alpha001 account + prod-* ArnLike substitutions)
Production deploy proceeding intentionally — operator signs off here: ✅ 2026-05-26 by Miguel Pinilla — production-deploy explicitly approved.

§ prod (Alpha001 / PostmarkProd)

Pre-flight (T-O1-prod) ✅ 2026-05-26

AWS SSO Admin-Alpha1 active: ✅ (account 009765408297, cached SSO session)
op read "op://Arda-ProdOAM/Postmark/credential" resolves: ✅ (UUID token, prefix 0e6d9b7b-ace — shared PostmarkProd token with demo, expected per binding)
validate-exit.sh prod 4/4 PASS: ✅
cdk diff Alpha001-prod-Email: ✅ — structurally byte-identical to dev/stage/demo diff (5 IAM policies, 5 IAM roles, 3 Lambdas, 1 HostedZone, 4 RecordSets, 2 SM secrets, 1 CustomResource, 2 LogRetention helpers, 14 outputs, 2 parameters). No RoleName on the IAM::Role (construct fix verified at synth time). Correct Alpha001 / prod-* substitutions. 0 modifies, 0 destroys.
Production deploy confirmation signed off: ✅ 2026-05-26 by Miguel Pinilla (explicit approval after cdk diff review)

Deploy (T-O7-prod) ✅ 2026-05-26

Command: cd /Users/jmp/code/arda/projects/email-integration-worktrees/phase-4/infrastructure-run-3 && ./amm.sh --profile Admin-Alpha1 Alpha001 prod (preceded by op read pre-warm)
Command exit code: 0 ✅
Pre-Deploy CLI step: first-time-create path — Postmark Sender Domain registered for prod.ardamails.com on PostmarkProd; DKIM selector 20260527061409pm (registered ~06:14 UTC May 27 = 23:14 PDT May 26); Return-Path target pm.mtasv.net.
cdk.context.json entry written for partitionMail:Alpha001:prod: ✅
CFN stack Alpha001-prod-Email status: ✅ CREATE_COMPLETE — all 25 resources, auto-named EmailDnsProvisioningRole = Alpha001-prod-Email-EmailDnsProvisioningRoleNsDeleg-2fiNWHdBEaWC; no collision with demo’s existing role in Alpha001.

Notes: Smoothest deploy of the cascade — no biometric glitches, no SSO callback issues, no collisions. Construct fix prevented the role-name issue from re-occurring.

Verification (T-O3-prod) ✅ 2026-05-26

dig +short NS prod.ardamails.com @8.8.8.8: ✅ 4 nameservers (ns-800.awsdns-36.net, ns-1942.awsdns-50.co.uk, ns-1119.awsdns-11.org, ns-68.awsdns-08.com)
SPF / DMARC / DKIM / Return-Path: ✅ all 4 records resolve correctly.
Postmark Console / API: prod.ardamails.com on PostmarkProd shows DKIM Verified + Return-Path Verified: ✅ confirmed via API (same 2-call DKIM async pattern as dev / stage / demo).
CFN Alpha001-prod-API-* exports: ✅ all 6 partition-mail exports present (PartitionMailZoneId: Z08953673OJURK2OYVKBZ, PartitionMailZoneName: prod.ardamails.com, both SM secret ARNs, both IAM role ARNs).
V-check rows: ⏳ to be populated at cascade close.