Runbook: Security Incident Response

Author: Miguel Pinilla Date: 2026-07-24 Last Verified: 2026-07-24 Environments: dev | stage | demo | prod

This runbook covers incidents that threaten the confidentiality or integrity of Arda’s systems, credentials, and data — exposed secrets, compromised accounts, unauthorized access, supply-chain and configuration exposure. For availability incidents (a service is down or crash-looping), use the Runbook: Incident Response; the two share the same phase skeleton, and a single event can be both.

Preface: Representative Security Incidents

The procedure below is written to cover the incident types in this table. Triage (Phase 1) starts by naming which type — or types — you are facing, because the type drives the containment move and the disclosure obligation.

Type	What it is	Representative example
Credential / secret exposure	A token, key, password, or connection string becomes readable outside its intended custody — a CI log, a committed file, a ticket, a Slack message, an agent transcript.	A Postmark server token printed into a GitHub Actions run log by a subprocess error.
Account / identity compromise	An Arda or third-party identity (Cognito user, GitHub, cloud IAM, 1Password) is accessed by an unauthorized party, or its credentials are phished or stolen.	A developer’s GitHub PAT is used from an unrecognized location to push to a repo.
Unauthorized data access / exfiltration	Tenant data, PII, or business data is read or copied by a party without authorization — a tenant-isolation breach, an exposed export, a query returning another tenant’s rows.	A BFF authorization gap lets one tenant read another tenant’s items.
Dependency / supply-chain compromise	A malicious or compromised package, GitHub Action, or base image enters the build or runtime.	A hijacked npm transitive dependency, or a typosquatted Maven artifact, ships in a release.
Infrastructure / cloud misconfiguration exposure	A resource is unintentionally reachable or over-permissioned — a public bucket, an open security group, a secret projected to the wrong audience.	An S3 bucket policy change makes uploaded objects world-readable.
Vulnerability disclosure	An exploitable flaw is reported (externally or internally) before it is weaponized — a dependency CVE, an auth-bypass in an endpoint.	A CVE lands against a library on the runtime classpath with a known exploit path.
Malicious activity / abuse / DoS	Automated abuse, credential stuffing, or resource exhaustion against an Arda surface.	Credential-stuffing against the sign-in flow, or a scripted flood against a public endpoint.
Insider / accidental misuse	An authorized person mishandles data or access — an over-broad share, a mis-sent notification, deletion of security-relevant evidence.	A partition vault is shared to the wrong group, or a diagnostic log with live data is pasted into a public channel.

Most types are treated the same way structurally; where a type changes the response, the phase notes call it out. Credential/secret exposure is the most common in practice and gets its own detailed procedure below.

Purpose

Take a responder from “a secret, account, or dataset may be exposed or misused” to “the access is cut, the exposure is scoped, the cause is fixed, every live copy of the exposed value is gone, and the right people have been told” — with the evidence preserved to answer what was exposed, for how long, and whether it was used.

When to Apply

Apply this runbook whenever any of the following is true:

• A secret (token, key, password, connection string) appears anywhere outside its custody path — a log, a repository, a ticket, a chat message, a terminal transcript, an artifact.
• An account or credential is suspected compromised, phished, or used by an unauthorized party.
• Data is accessed, returned, or copied across an authorization or tenant-isolation boundary.
• A dependency, Action, or image is discovered to be malicious or compromised.
• A resource is found to be exposed or over-permissioned beyond its intent.
• A vulnerability with a plausible exploit path is disclosed against a running Arda surface.

When in doubt, open the incident record and triage — the cost of a short investigation that finds nothing is far lower than the cost of an unrecorded exposure.

Roles

For most Arda security incidents one or two engineers fulfill all roles. They are listed separately because the responsibilities are distinct and easy to drop under pressure.

Role	Responsibility
Incident Commander	Owns the incident end-to-end. Decides scope, sequencing, and when to declare closure. Maintains the incident record.
Security Lead	Owns the confidentiality/integrity view: containment strategy, exposure assessment, and the disclosure decision. On a one-person incident, the Commander holds this.
Responder	Executes containment and remediation steps (rotation, revocation, isolation, patching).
Communicator / Disclosure owner	Notifies the internal security owner and, when warranted, affected customers, regulators, or upstream vendors. Owns the disclosure timeline.

Response Phases

The state diagram shows the six phases and their transitions, including the Verify-to-Assess loop when verification surfaces a broader exposure than first scoped.

The security response specializes the general incident phases: containment is “cut off unauthorized access” rather than “restore service,” a dedicated Assess Exposure phase scopes blast radius and misuse, and the final phase carries an explicit Disclose obligation.

Phase 1 — Detect & Triage

1. Capture the trigger verbatim — the alert, the log line, the report, the URL. Do not paste a live secret value into the record; record its location and identity (which secret, where it appeared), not the value.
2. Classify the incident type using the Preface table. A single event may be more than one type (an exposed cloud key is both credential exposure and a misconfiguration).
3. Set severity by confidentiality/integrity impact and blast radius: a production credential granting broad access is high; a scoped, already-rotated value with no reachable data is low. Availability impact, if any, is scored per the general incident runbook.
4. Open the incident record (Linear or Markdown) with timestamp, type, severity, and the affected identities/resources.
5. Assign roles and decide the disclosure track early — whether this is plausibly notifiable (PII, customer data, a vendor’s credential) shapes evidence handling from here on.

Phase 2 — Contain (preserve evidence first)

Security containment stops ongoing unauthorized access. The security-specific discipline: capture the evidence you will need before you destroy anything.

1. Preserve evidence first. Before deleting a leaking log, wiping a host, or closing an exposed resource, record what you will need later: what was exposed, where, since when, and who could have seen it. Deleting the artifact is a containment step, not the first step — a prematurely deleted log destroys the exposure-window and misuse evidence you need in Phase 3.
2. Cut access:
   • Credential exposure → revoke or rotate the exposed secret (see the detailed procedure).
   • Account compromise → disable the account / revoke its sessions and tokens; force re-authentication.
   • Unauthorized data access → close the path (revoke the grant, patch the authorization gap, block the caller).
   • Misconfiguration → restore least-privilege on the resource.
   • Supply-chain → pin off the compromised version; halt deploys that would ship it.
3. Record every manual change so it can be reconciled against the declarative source of truth (drift accountability).
4. Containment is not remediation. Once access is cut, transition deliberately to assessment — do not declare closure.

Phase 3 — Assess Exposure

Scope the incident with evidence.

1. What was exposed — the exact secret, dataset, or accounts.
2. For how long — the exposure window: from when the value became reachable to when containment cut it off.
3. Who could access it, and did they — check audit and activity logs for the window (CloudTrail, GitHub audit log, the vendor’s activity/message log, sign-in logs). Distinguish “was reachable” from “was used.”
4. Blast radius — enumerate everything the exposed credential or access could reach. A single leaked token often grants more than the one resource in front of you; a compromised account may hold standing access to several systems. Every reachable secret is now also suspect.

If assessment reveals the exposure is broader than first believed, widen the containment and remediation scope accordingly.

Phase 4 — Eradicate & Remediate

Remove the cause and every live copy.

1. Rotate/revoke the exposed credential and everything it could reach (per blast radius). Confirm the old credential is deleted or disabled, not merely superseded by a new one.
2. Scrub residual copies of the exposed value — CI logs and their retention/backups, downstream log sinks (CloudWatch, Sentry), notification emails/Slack, caches, and terminal or agent transcripts. Rotation voids the credential; scrubbing removes the exposed string.
3. Fix the cause so it cannot recur: the code or config that leaked the secret, the authorization gap, the over-broad permission, the vulnerable dependency version.
4. Prefer the smallest fix that closes the exposure; file larger hardening as follow-ups rather than bundling it into the incident change.

Phase 5 — Recover & Verify

Confirm with positive checks, not absence-of-error.

1. The exposed access is dead — an attempt with the old credential fails, or the vendor/provider confirms revocation.
2. No live residual copies remain — the scrub in Phase 4 is confirmed across each sink you identified.
3. The delivery path is consistent — where a secret is custodied and then delivered (1Password → Secrets Manager → runtime), verify the store now holds the new value and the running system reads it. See the Secret Delivery Pattern.
4. State is healthy — run the relevant drift/health checks for the affected surface (for partition secrets, the runtime-platform-drift check).
5. Reconcile any manual drift incurred during containment.

Phase 6 — Post-Incident & Disclose

1. Write the summary — timeline (UTC), incident type, what was exposed, exposure window, whether it was used, remediation, and residual risk. Never include the exposed value.
2. Disclose per obligation — notify the internal security owner for every incident; notify affected customers or regulators when PII or customer data was exposed; notify the upstream vendor when their credential or product was involved. The Communicator owns this timeline.
3. File follow-ups — detective controls that would have caught it sooner (an alert on the signal that was missed), preventive guardrails (a lint/secret-scan that would have blocked the leak), and any latent issues uncovered during assessment.
4. Blameless review for higher-severity incidents.
5. Update this runbook if anything in the response surprised you.

Credential & Secret Exposure — Detailed Procedure

The most common security incident. The steps below are the concrete instance of Phases 2–6 for an exposed secret, and they draw the line between technical remediation (rotate + fix + scrub) and incident handling (assess + disclose) that is easy to conflate.

1. Preserve, then contain. Record where the secret appeared and since when. Then rotate or revoke it — for a Postmark server token this is the console rotation + vault re-custody + redeploy in the Runbook: System Email Server; for a 1Password service-account token, create a replacement and revoke the old; for a cloud key, deactivate then delete it.
2. Confirm the old value is void. Generating a new secret is not enough — the old one must be deleted or disabled. Verify it, don’t assume it.
3. Re-custody, don’t just rotate. When a component reads its secret from a store on every start (the steady-state read), rotating at the source without updating that store leaves the stale value in place and a later deploy re-custodies it. Update the store of record and confirm the running system reads the new value.
4. Scrub residual copies. Delete the leaking artifact and every mirror: CI run logs (and be aware provider retention/backups may lag), downstream log sinks, notifications, caches, transcripts.
5. Assess exposure. Exposure window, who could read it, whether it was used (vendor activity log, cloud audit trail), and the full set of resources the token could reach — rotate those too.
6. Disclose. Record the incident and notify the security owner; escalate to customers/vendor per obligation.

Caution

Rotation makes the leaked value useless, but it does not close the incident. Scrubbing residual copies, checking for misuse in the exposure window, and disclosure are separate steps that rotation alone does not accomplish.

Verification

Before closing a security incident, confirm all of:

• Unauthorized access is cut — exposed credential rotated/revoked and the old value confirmed dead.
• Blast-radius secrets/accounts reachable via the exposure have been rotated or cleared.
• Every identified residual copy of the exposed value has been scrubbed.
• The cause (code, config, permission, dependency) is fixed so it cannot recur.
• The exposure window was reviewed for actual misuse, with the result recorded.
• Delivery-path and drift/health checks pass for the affected surface.
• The incident is written up and disclosed per obligation; follow-ups are filed.

Common Pitfalls

Pitfall	What it looks like	Consequence
Destroying evidence during containment	Deleting the leaking log before recording what was exposed, since when, and who could see it.	The exposure window and misuse check in Phase 3 become unanswerable.
Rotating without re-custody	Generating a new secret at the source but not updating the store the component reads on start.	The stale value persists; a later deploy re-custodies it and the exposure reopens.
Treating rotation as “done”	Closing the incident once the token is rotated.	Residual copies (backups, sinks, transcripts) survive; misuse in the window goes unchecked; disclosure is skipped.
Under-scoping blast radius	Rotating only the one leaked token and stopping.	Other secrets or systems the credential could reach stay compromised.
Skipping the misuse check	Assuming “rotated, so no harm done” without inspecting activity in the exposure window.	An actual compromise during the window is missed.
Silent handling	Fixing it quietly without notifying the security owner or affected parties.	Disclosure obligations are missed; the same class of incident recurs undetected.

Related Runbooks

• Runbook: Incident Response — the general production-incident procedure; use it when the security incident also causes an outage (shared phase skeleton).
• Runbook: System Email Server — concrete Postmark token-rotation-and-re-custody procedure referenced above.
• Secret Delivery Pattern — how partition secrets are custodied and delivered (the path a rotation must keep consistent).

Templates

• Threat Model — for STRIDE analysis of a feature or component (proactive counterpart to this reactive runbook).
• Security Audit Report — for recording a security review’s findings and remediation.

See Also

• SRE Overview — section landing page.

Change History

Date	Author	Change
2026-07-24	Miguel Pinilla	Initial version — phases, credential-exposure procedure, incident-type preface.