Runbook: System Email Server (Provisioning & Rotation)

Author: Miguel Pinilla Last Verified: 2026-07-24 — provisioning and token rotation performed on dev (Alpha002). Applies to: all four active partitions (dev, stage, demo, prod).

This runbook covers the infrastructure lifecycle of the per-partition SystemEmailServer — the Postmark server that backs the System Email capability. It is the operator-facing companion to the design (see §11 Runbook hooks); for how a module consumes the capability, see the Integration guide.

Scope: provision, verify, and rotate the server token. Out of scope: the EmailEncryptionKey (see Encryption-Key Rotation) and the per-tenant email module (see Email Module Runbook).

What gets provisioned

A partition deploy (amm.sh <infrastructure> <partition>) provisions three things for System Email, idempotently:

1. A Postmark SystemEmailServer-{infrastructure}-{partition} server (created via the Account API if absent; reused if present).
2. Its token, custodied in op://Arda-{Env}OAM/SystemEmailServer/credential.
3. A Secrets Manager secret {fqn}-I-EmailSystemServerToken (NoEcho-parameter flow) and its {fqn}-API-EmailSystemServerTokenArn export, delivered to runtime through the δ.1 secret-delivery flow.

Prerequisites

Requirement	Detail
AWS profile	Admin-Alpha1 for demo/prod (Alpha001); Alpha002-Admin for dev/stage (Alpha002).
1Password read	The ambient service-account token (OP_SERVICE_ACCOUNT_TOKEN, sourced from op://Arda-SystemsOAM/IAC-SCRIPTS Service Account Token/credential) resolves every read, including the Postmark account token and the write token below.
1Password write	Non-obvious dependency. The ambient CI token is read-only, so the vault-item write is authorized by a dedicated write token at op://Arda-SystemsOAM/OP-IAC-SCRIPTS-WRITE/credential, resolved by the read token and applied only to the op item create/edit subprocess. That write service account needs create + edit on each Arda-{Env}OAM partition vault. Missing this surfaces as op error (101) You do not have permission to perform this action.

Provisioning a partition

The canonical path is the amm GitHub Actions workflow (workflow_dispatch, input environment = <infrastructure>/<partition>), which runs under CI’s OIDC + service-account identity. The equivalent local invocation is ./amm.sh --profile <aws-profile> <infrastructure> <partition>.

The SystemEmailServer step runs inside the partition-mail Pre-Deploy and reconciles against actual state:

• First run (no server, no vault item): create the server, write the token to the vault, then deploy the CFN secret.
• Steady state (server + vault item): read the token from the vault and deploy — Postmark is not re-consulted for the token (this matters for rotation, below).
• Heal (server exists, vault item missing): re-read ApiTokens[0] from Postmark and re-create the vault item.

Provisioning is idempotent — re-running converges without creating a duplicate server or vault item.

Verifying (read-only)

Before or after any change, use these — none of them mutate anything:

# 1. Offline synth — see the intended CloudFormation (no AWS calls)
npm run synth:named -- <infrastructure>/<partition>

# 2. Dry-run diff against the deployed stack (needs AWS creds)
node tools/cdk-runner.js diff <infrastructure>/<partition> <aws-profile> us-east-1

# 3. Which 1Password items the Pre-Deploy will touch (no secret values)
npx ts-node -r tsconfig-paths/register tools/register-partition-mail-signature.ts --vault-information

# 4. Runtime drift check — asserts server + secret + exports exist
AWS_PROFILE=<aws-profile> npx ts-node -r tsconfig-paths/register tools/runtime-platform-drift.ts <partition>

A healthy partition reports "passed": 18, "failed": 0. Before provisioning, the two SystemEmailServer assertions fail (server + -I-EmailSystemServerToken secret absent) — that is the expected “before” baseline.

Rotating the SystemEmailServer token

Rotate when the token is compromised (e.g. accidental exposure) or on a routine schedule.

Caution

Postmark has no token-rotation API. The Servers API exposes ApiTokens as read-only — there is no regenerate/rotate endpoint. Rotation must start in the Postmark console. Also, because a steady-state deploy reads the token from the vault (not from Postmark), the vault must be updated for the new token to propagate — a plain re-deploy alone will re-custody the old token.

Step 1 — Rotate in the Postmark console

On server SystemEmailServer-{infrastructure}-{partition} → API Tokens tab:

1. Generate New.
2. Delete the old token.

Deleting the old token voids it immediately and leaves the server with exactly one token, so ApiTokens[0] is unambiguously the new one. There is no live sender during a pure infra rotation, so no dual-token window is required; if a consumer is actively sending, keep both tokens until Step 4 completes, then delete the old one.

Step 2 — Re-custody the new token in the vault

Retrieve the new token from the Account API and write it to the vault. Handle the value only in shell variables — never echo it or place it in a command’s argv (that is how it can leak into a log):

ACCT=$(op read "op://Arda-{Env}OAM/Postmark/credential")
NEW=$(curl -s -H "X-Postmark-Account-Token: $ACCT" \
        "https://api.postmarkapp.com/servers/<serverId>" | jq -r '.ApiTokens[0]')
op item edit "SystemEmailServer" --vault "Arda-{Env}OAM" "credential=$NEW"
# verify (compared, not printed):
[ "$(op read "op://Arda-{Env}OAM/SystemEmailServer/credential")" = "$NEW" ] && echo "vault OK"

Find <serverId> from the drift-check output or GET /servers?name=SystemEmailServer-.... (Equivalently: delete the vault item and let the next deploy’s heal path re-read ApiTokens[0] — but only after the old token is deleted, so [0] is the new one.)

Step 3 — Redeploy the partition

Run the amm workflow (or ./amm.sh) for the partition. Steady state reads the now-updated vault token and pushes it into the {fqn}-I-EmailSystemServerToken Secrets Manager secret; ESO then refreshes it into the pods.

Step 4 — Verify

AWS_PROFILE=<aws-profile> npx ts-node -r tsconfig-paths/register tools/runtime-platform-drift.ts <partition>

Expect 18/18. Optionally confirm the SM secret carries the rotated value (compared, not printed):

SM=$(aws secretsmanager get-secret-value --secret-id {fqn}-I-EmailSystemServerToken \
       --query SecretString --output text --profile <aws-profile> | jq -r '.token')
[ "$SM" = "$(op read "op://Arda-{Env}OAM/SystemEmailServer/credential")" ] && echo "SM matches vault"

If a consumer was sending during rotation, restart its pods after the ESO refresh so it picks up the new token, then delete the old Postmark token.

Troubleshooting

Symptom	Cause	Fix
op (101) You do not have permission during the vault write	The write service account lacks create/edit on the partition vault (or the write-token wiring is not resolving OP-IAC-SCRIPTS-WRITE)	Grant the write SA create+edit on Arda-{Env}OAM; confirm op://Arda-SystemsOAM/OP-IAC-SCRIPTS-WRITE/credential resolves. Service accounts are immutable — a permission change means a new SA + re-store the token.
invalid JSON in piped input from op item create	An op invocation reading inherited stdin as a template (fixed: stdin is detached)	Ensure the tooling is at or past the writeOnePasswordItem hardening; re-run.
Pre-Deploy custody-divergence error	Vault item SystemEmailServer exists but no matching Postmark server (server deleted out-of-band)	Verify in the Postmark console; if intentionally deleted, delete the stale vault item and re-run — a fresh server + token is provisioned.
Drift auto-issue: server or secret missing	Deleted out-of-band	Re-run the partition deploy (idempotent) and investigate the deletion.
Sends fail Rejected 401 / InvalidToken	Postmark / vault / SM disagree on the token	Run drift; compare the vault item’s modified time against the last partition deploy; re-custody + redeploy to reconverge.
Redeploy re-custodies the old token after a console rotation	Steady state read the stale vault value	Update the vault first (Step 2), then redeploy — see the caution above.

Related

• System Email — the as-built capability.
• Integration guide — how a module configures, wires, and uses it.
• Design: Invitation Mail Server — full design and decision log.
• Secret Delivery Pattern — the δ.1 flow this token follows.
• Partition Mail Topology — the per-partition secret/export inventory.