Decision Log: Sentry Configuration as Code
Decisions for the Sentry configuration-as-code capability — the first inhabitant
of the OperationsManagement platform element — and for the alert-rule
migration it executes. Companion to design.md and
goal.md.
Several decisions here overturned assumptions carried in the originating prompt. Where that happened, the evidence is recorded, because the assumption was reasonable and a future reader will otherwise wonder why the design diverges from it.
Decision Table
Section titled “Decision Table”| # | Question | Status | Decision | Round |
|---|---|---|---|---|
| DQ-001 | Where does the code live? | Decided | OperationsManagement platform element, layered per the repo vocabulary | R1 |
| DQ-002 | How does the Application execute? | Decided | Declarative App + our own reconcile engine | R1 |
| DQ-003 | Resource identity and managed set | Decided | Pinned IDs + required managed-name prefix | R1 |
| DQ-004 | Which Sentry API surface? | Decided | Org-scoped /workflows/; legacy /rules/ is a lossy shim | R1 |
| DQ-005 | Are detectors a managed resource? | Decided | Read-only references, resolved symbolically | R1 |
| DQ-006 | Where do fixtures/snapshots live? | Decided | Committed under the repo | R1 |
| DQ-007 | Lookahead for Performance Management | Decided | Design the seam, implement Fault only | R1 |
| DQ-008 | Urgent tier window (30m unavailable) | Decided | 3 in 15m, as a tunable starting value | R2 |
| DQ-009 | Urgent tier scoping | Decided | Two rules, one per Sentry project | R2 |
| DQ-010 | Linear project routing | Decided | KTLO / BE and KTLO / FE separately | R2 |
| DQ-011 | Rule 3190653 environment defect | Decided | Fix in this migration | R2 |
| DQ-012 | Environment naming divergence | Decided | Logical-environment abstraction mapping to physical names | R2 |
| DQ-013 | Uptime detector attachment | Decided | Preserve, flag explicitly in plan output | R2 |
| DQ-014 | Product Viewpoint persona | Decided | Author the Platform Operator persona | R1 |
| DQ-015 | assigneeId on the template rule | Decided | Strip explicitly; assignment stays a Linear concern | R2 |
| DQ-016 | stateId (undeclared in the prompt) | Decided | Reuse Triage | R2 |
| DQ-017 | High → Urgent escalation handling | Decided | Urgent notifies via Slack; only High creates Linear tickets | R3 |
| DQ-018 | Removal mode | Decided | Delete outright | R3 |
| DQ-019 | Splitting cross-project rule 3153115 | Decided | Delete and create two fresh rules | R3 |
| DQ-020 | existing_high_priority_issue trigger | Decided | Drop it | R3 |
| DQ-021 | Slack channel for the Urgent tier | Decided | #1-dev-team (C086HQQ076Y), as a named constant | R4 |
| DQ-022 | How configuration is shaped | Decided | Nested Configuration/Props/Built tree assembled as one SentryAlerts value | R4 |
| DQ-023 | Deploy on merge, and how apply is gated | Decided | Fully automatic on merge; plan also posted on the PR. Drops Constraint 7 for CI | R5 |
| DQ-024 | Where the CI snapshot lives | Decided | Workflow artifact, 90-day retention, uploaded before apply | R5 |
| DQ-025 | When the workflow is enabled | Decided | After the migration lands and plan is verified empty | R5 |
| DQ-026 | Disposition of rule 3190653 | Decided | Delete and recreate, superseding adopt-by-rename | R6 |
| DQ-027 | The absent-environment assumption | Measured | Confirmed by controlled probe; 3190653 was inert | R6 |
| DQ-028 | What exactly is the managed set? | Decided | Prefix ∩ declared. Supersedes part of DQ-003 | R6 |
| DQ-029 | Where identity lives | Decided | A key embedded in the workflow name; pinned ids removed entirely. Supersedes DQ-003 Option A | R6 |
| DQ-030 | Urgent tier destination | Decided | #sre-production (C0BNTF2AJ6N). Supersedes DQ-021 | R6 |
Round 1: Placement and Engine
Section titled “Round 1: Placement and Engine”DQ-001: Where does the code live?
Section titled “DQ-001: Where does the code live?”Context: The originating prompt proposed a standalone layout (src/, bin/,
config/, docs/ at repo root) with zero runtime dependencies and Node native
type stripping. The infrastructure repo has an established four-layer
structure and a documented tools/ vs scripts/ split.
| Option | Description | Trade-offs |
|---|---|---|
| A | Standalone sub-package under tools/sentry/, own package.json | Honors the prompt literally. Precedent exists (tools/create-aws-accounts/). But duplicates postmark-client.ts’s retry/backoff and stands up a second toolchain. |
| B | Integrated across the existing layers as a new platform element | Reuses tools/lib/, the drift family, and the platform/ metadata layer. Requires defining the OperationsManagement element. |
| C | Migration script only, no library | Smallest change; leaves nothing reusable. |
Recommendation: Option B.
Decision: Option B. The capability is the first inhabitant of a fourth
platform element, OperationsManagement, alongside Environments, Root, and the
informal tools/scripts collection. Spelled in full in paths — never oam, which
already denotes infrastructure-supporting constructs under
src/main/cdk/constructs/oam/.
Applied to:
- goal.md § Architecture Placement, § Constraints
infrastructure/knowledge-base/platform-architecture.md(new)infrastructure/CLAUDE.md§ Platform elements
DQ-002: How does the Application execute?
Section titled “DQ-002: How does the Application execute?”Context: In this repo, cdk names IaC machinery generally, and an
Application normally composes Stacks that synthesize CloudFormation. Sentry
workflows have no CloudFormation representation.
| Option | Description | Trade-offs |
|---|---|---|
| A | Declarative App producing a desired-state tree; our reconcile engine applies it | Keeps script → instances → apps → stacks → constructs intact. The engine replaces CloudFormation as executor. Requires building diff/plan/apply. |
| B | CloudFormation custom resources backed by Lambda | Native cdk deploy and CFN drift. But needs Lambda plumbing, puts the Sentry token in the deploy path, and couples the OAM plane to AWS unnecessarily. |
| C | Config module feeding an engine, no App/Stack layers | Least code; abandons the platform framing and leaves future OAM providers no shape to follow. |
Recommendation: Option A.
Decision: Option A. apps/OperationsManagement does not construct a
cdk.App(); it produces a typed desired-state tree that the entry script hands
to the reconcile engine.
Applied to: design.md § 8 Structural Design, § 9.1
DQ-003: Resource identity and managed set
Section titled “DQ-003: Resource identity and managed set”Context: The workflow schema has no free-form metadata field — nowhere to
store a logical key. Worse, name is not unique: workflows 3153115 and
3426957 share the name “Send a notification for high priority issues”. A
plan that proposed deleting everything absent from config would be
catastrophic, since the org contains rules this project does not own.
| Option | Description | Trade-offs |
|---|---|---|
| A | Pin workflow IDs in the Instance file; require a managed-name prefix as a safety assertion | Unambiguous matching plus a guard against a typo pointing at an unmanaged rule. Config carries server-generated IDs. |
| B | Pin IDs only | Simple; no guard against mis-targeting. |
| C | Name-prefix only, no IDs | Config reads cleanly; a UI rename silently orphans a rule and the tool proposes recreating it. |
Recommendation: Option A.
Decision: Option A. The managed set is the intersection of pinned IDs and the name prefix. The engine never proposes deleting a workflow outside that set; unmanaged workflows are reported as untracked, not as drift.
Superseded in part by DQ-028. The intersection as stated here has a destructive gap: a workflow the tool has just created carries the prefix but has no pinned ID yet, so it is claimed as managed and matched by nothing — which is the delete rule. Identity is now pinned ID or natural key. DQ-029 then removed pinned ids altogether, leaving the key as the only path. Option A’s substance survives in that identity is still explicit and declared, and the prefix is still a hard boundary — only its carrier changed.
Applied to: design.md § 8, § 9.3
DQ-004: Which Sentry API surface?
Section titled “DQ-004: Which Sentry API surface?”Context: The prompt specified read-modify-write against issue alert rules
with conditions[] / filters[] / actions[], and cited rule IDs verified from
the Sentry UI. Recon on 2026-08-05 contradicted this.
Evidence:
GET /organizations/arda-systems/workflows/returns all seven rules with exactly the cited IDs.GET /projects/arda-systems/arda-frontend/rules/returns only five entries, with different IDs (16782074,16817922,16817925,16817926, and a synthetic10003433641=10000000000 + 3433641).GET /projects/arda-systems/arda-frontend/rules/3433641/→ 404. The cited IDs do not exist in the legacy namespace.
| Option | Description | Trade-offs |
|---|---|---|
| A | Target /workflows/ exclusively | The only surface where these rules exist. Schema must be derived from live payloads. |
| B | Target the legacy /rules/ shim | Matches the prompt, and matches most published documentation — but the IDs don’t resolve and writes would pass through a lossy translation. |
Recommendation: Option A.
Decision: Option A. The org is on Sentry’s workflow engine. The legacy endpoint is a read shim and is not used.
Applied to: design.md § 2, § 8; the API observations note
DQ-005: Are detectors a managed resource?
Section titled “DQ-005: Are detectors a managed resource?”Context: Workflows attach to projects through detectorIds, not a project
list. Detectors are project-scoped monitors (error, issue_stream,
uptime_domain_failure) carrying live monitoring configuration.
| Option | Description | Trade-offs |
|---|---|---|
| A | Read-only references, named symbolically in config (arda-frontend/issue_stream), resolved to IDs at plan time | Config is legible and survives detector recreation. No risk of the tool mutating monitoring config. |
| B | Full managed resource with create/update/delete | Needed eventually for uptime monitors (a Performance concern); large scope now. |
| C | Raw detector ID strings in config | Trivial to build; opaque, and breaks on recreation. |
Recommendation: Option A.
Decision: Option A.
Applied to: design.md § 8 DetectorResolver
DQ-006: Where do fixtures and snapshots live?
Section titled “DQ-006: Where do fixtures and snapshots live?”Decision: Committed under the repo. They are the type-derivation evidence and the rollback source for a destructive migration, roughly 1 KB each, and contain no secrets — IDs and settings only. A snapshot nobody can locate is not a rollback source.
Applied to: design.md § 12
DQ-007: How far to anticipate Performance Management?
Section titled “DQ-007: How far to anticipate Performance Management?”Decision: Design the seam; implement Fault only. The resource interface, registry, engine, and Stack-layer split accommodate metric alerts and dashboards; only issue alert workflows are built and verified. Extensibility is claimed by structure and demonstrated by review, not by speculative code.
Applied to: design.md § 8, § 12 Out of Scope
DQ-014: Product Viewpoint persona
Section titled “DQ-014: Product Viewpoint persona”Context: The persona catalog contains only business personas. OAM-plane work serves an operator role with no entry.
Decision: Author a Platform Operator persona. All future OAM work needs it, and the design framework’s Product Viewpoint is otherwise unanswerable for this class of work.
Applied to: product/personas/priya-platform-operator.md (new)
Round 2: Live-State Findings
Section titled “Round 2: Live-State Findings”Round 2 exists because recon against the live org surfaced facts that no amount of design reasoning would have produced.
DQ-008: Urgent tier window
Section titled “DQ-008: Urgent tier window”Context: The prompt specified urgentThreshold: { count: 3, windowMinutes: 30 }.
Probing event_frequency_count returned a closed interval enum:
['1m', '5m', '15m', '1h', '1d', '1w', '30d']. 30 minutes is not offered.
| Option | Description | Trade-offs |
|---|---|---|
| A | 3 in 1h | Strictly more sensitive than intended; more Urgent tickets. |
| B | 3 in 15m | Strictly less sensitive; a steady 3-per-30-min failure never escalates. |
| C | Preserve the rate (~6/hour), e.g. 6 in 1h | Keeps intended sensitivity rather than intended numbers. |
Recommendation: Option C.
Decision: Option B — 3 occurrences in 15 minutes, as an explicitly tunable starting value.
This supersedes the rate-preservation method originally chosen. Worth stating plainly rather than burying: 3-in-15m is stricter than the specified 3-in-30m — it implies ~12/hour against the original ~6/hour — so the Urgent tier starts less sensitive than the prompt intended and will only catch sharp bursts. That is deliberate. Starting narrow means the first weeks produce too few Urgent notifications rather than too many, which is the safer error for a tier whose entire value is being rare and unambiguous.
The pair is a named constant in the Instance file precisely so it can be widened once real issue-frequency data exists. Retuning is a one-line config change and a PR, not a redesign.
Applied to: design.md § 8 AlertTier, § 8 ArdaSystems, § 9.2
DQ-009: Urgent tier scoping
Section titled “DQ-009: Urgent tier scoping”Context: See DQ-012 — the two Sentry projects have disjoint environment
names, and a workflow carries a single environment string.
Decision: Two rules, one per Sentry project, each with its own physical environment name and its own detectors. The Construct is parametric on logical environment, so this is one definition instantiated twice.
Applied to: design.md § 8, § 9.2
DQ-010: Linear project routing
Section titled “DQ-010: Linear project routing”Context: projectId is empty on every existing Linear action, so there was
no value to copy. Resolved live: team 4e4f6afc-… is Product Development;
labelId 7f70fbff-… is “Bug”; stateId e6779135-… is “Triage”.
Decision: Route per discipline — KTLO / BE
(b9b219b9-1c61-4602-b7bd-9697cd9f6b95) for platform-be, KTLO / FE
(410e9402-d9cd-4f93-9421-5ba9ac21f464) for arda-frontend.
Consequence: since a workflow emits one action config, per-discipline routing forces one rule per Sentry project for every tier — not just Urgent. This is what makes DQ-019 necessary.
Applied to: design.md § 8, § 9.2
DQ-011 / DQ-012: Environment naming divergence
Section titled “DQ-011 / DQ-012: Environment naming divergence”Context: The projects register disjoint environment names —
arda-frontend: dev, development, local, production, stage;
platform-be: Alpha001-demo, Alpha001-prod, Alpha002-dev, Alpha002-stage.
No name is common to both. Sentry registers environments from incoming events,
so an absent name means no event ever carried it.
Consequence: rule 3190653 carries environment: "production" and attaches to
both projects, so its platform-be attachment can never match. Backend
first-seen Slack alerting from that rule is inert.
Decision (DQ-011): Fix it in this migration rather than deferring — the rules are being restructured anyway, and the gap is a live alerting hole.
Decision (DQ-012): The divergence is an accident of how the projects were first created, not a deliberate scheme. The names map semantically, so the design introduces a logical environment abstraction resolved to physical names per project:
| Logical | arda-frontend | platform-be |
|---|---|---|
prod | production | Alpha001-prod |
stage | stage | Alpha002-stage |
dev | dev | Alpha002-dev |
demo | — | Alpha001-demo |
development and local on arda-frontend are unmapped and treated as
local-development noise.
Applied to: design.md § 8 LogicalEnvironment, § 9.2;
platform/sentry-service.ts
DQ-013: Uptime detector attachment
Section titled “DQ-013: Uptime detector attachment”Context: 3153115 attaches to two uptime_domain_failure detectors on
platform-be in addition to issue streams, so high-priority Linear routing
currently also covers uptime failures.
Decision: Preserve the behavior; make it explicit in plan output so the carry-over is deliberate and visible rather than accidental.
Applied to: design.md § 9.2
DQ-015: assigneeId on the template rule
Section titled “DQ-015: assigneeId on the template rule”Context: Rule 3433641 — designated as the template because it carries a
working labelId and stateId — also carries
assigneeId: a6e34222-…. Assignment is deliberately a Linear-side concern.
Decision: Strip assigneeId explicitly when deriving from the template. A
verbatim copy would silently pre-assign every ticket.
Applied to: design.md § 8 LinearRoutedAlertTier
DQ-016: stateId
Section titled “DQ-016: stateId”Context: Not mentioned in the prompt’s decision block, yet set on both existing Linear actions to the same value.
Decision: Reuse e6779135-… (Triage). Sentry-created tickets landing in
Triage matches Linear’s documented behavior for integration-created issues and
the team’s existing practice.
Applied to: design.md § 8 LinearRoutedAlertTier
Round 3: Migration Semantics
Section titled “Round 3: Migration Semantics”DQ-017: High → Urgent escalation handling
Section titled “DQ-017: High → Urgent escalation handling”Context: An issue may fire the High rule and later cross the Urgent threshold. The originating prompt asserted this would create a second Linear ticket. That assertion was unverified, and Linear’s documentation is silent on de-duplication of repeat firings, so it was probed empirically on 2026-08-06.
Probe method: two throwaway workflows on platform-be scoped to
Alpha002-dev, routing to a throwaway Linear project; seven synthetic events
carrying an explicit fingerprint so they collapsed into exactly one Sentry
issue (7655326314). Rule A triggered on first_seen_event at Linear priority
2; rule B on event_frequency_count at priority 1, two minutes later. Both
workflows deleted afterwards, inventory verified back at seven.
Result — the prompt’s assumption was wrong, and the real behavior is worse:
| Time | Event |
|---|---|
| 00:48:03.421 | Rule A fires — and so do pre-existing 3153115 and 3426957, the same instant |
| 00:48:05.882 | PDEV-1421 created, priority 1, no project (from 3153115) |
| 00:48:07.638 | PDEV-1422 created, priority 2, throwaway project (from rule A) |
| 00:50:02.665 | Rule B fires — priority 1, same Linear project, same Sentry issue |
| 00:50:05.7 | Both tickets’ updatedAt bump. No new ticket. No priority change. |
Two facts follow:
- A temporally separated firing is a silent no-op. Rule B asked for priority 1, fired two minutes after the ticket existed, and PDEV-1422 stayed at priority 2. Escalation left no trace but a timestamp.
- Simultaneous firings race and both create.
3153115and rule A produced two tickets 1.8 seconds apart for one Sentry issue.
So the integration gives duplicates when rules fire together and silence when they fire apart — and High → Urgent is inherently the “apart” case.
Two hypotheses remain indistinguishable from this data: dedup keyed on the Sentry issue with a race window (H1), or keyed on (Sentry issue × Linear project) (H2). Distinguishing them needs a later-firing rule targeting a different Linear project; that probe was not run because the chosen decision makes it moot.
| Option | Description | Trade-offs |
|---|---|---|
| A | Urgent notifies via Slack; only High creates Linear tickets | Escalation becomes visible without depending on ticket creation or priority upgrade. Correct under both H1 and H2. Urgent produces no ticket, so escalation is not tracked as work. |
| B | Probe H2, then route Urgent to a separate Linear project | Preserves two-tier ticketing if H2 holds. Another probe, and two Linear projects per discipline. |
| C | Single Linear tier, escalate by hand in triage | Simplest; escalation depends entirely on someone noticing. |
| D | Accept the blind spot | Rejected — this is precisely the silent-failure mode the project exists to eliminate. |
Decision: Option A. The Urgent tier emits a Slack action rather than a Linear action. Linear ticketing remains a High-tier concern, one ticket per Sentry issue, which is what the integration reliably delivers. Escalation is surfaced as a notification, where duplication is harmless and silence is not possible.
Consequence: the tier Construct is no longer Linear-specific. It becomes
AlertTier, parametric on its action, with linearAction() and slackAction()
builders. linearProjectId is a High-tier concern only.
Probe artifacts — pending cleanup
Section titled “Probe artifacts — pending cleanup”The probe mutated two shared workspaces. The residue is deliberately retained until the implementation PR is reviewed, because it is the only first-hand evidence for a finding that contradicted the project’s stated premise, and contradicted findings are the ones most worth being able to re-check.
| Artifact | State | Removal |
|---|---|---|
[IAC PROBE] DQ-017 Linear project | present | UI delete — no MCP operation exists |
| PDEV-1421 / PDEV-1422 | canceled, priority cleared | delete with the project |
Sentry issue 7655326314 | ignored, 7 synthetic events, tag iac_probe:dq017 | UI delete — the tooling token lacks event:admin |
| Probe workflows 3813713 / 3813714 | deleted, 404-verified | done; inventory back at seven |
A second round of probes on 2026-08-06 (see DQ-027) left:
| Artifact | State | Removal |
|---|---|---|
| PDEV-1433 / PDEV-1434 | canceled | delete alongside the DQ-017 tickets |
Sentry issue 7657516175 | ignored, 1 synthetic event, tag iac_probe | UI delete — same token limitation |
| Probe workflows 3818144–3818159 | deleted, inventory re-verified at seven | done |
Whoever merges the implementation PR should remove the retained items. Both
DQ-017 tickets link to Sentry issue 7655326314; their creation timestamps and
priorities are the evidence recorded in the table above.
Applied to:
Round 4: Consequences of DQ-017
Section titled “Round 4: Consequences of DQ-017”DQ-021: Slack channel for the Urgent tier
Section titled “DQ-021: Slack channel for the Urgent tier”Context: DQ-017 routes the Urgent tier to Slack. The org has exactly one
Slack target configured — #sentry-fe-prod, targetIdentifier C0ALYG3Q55L,
integrationId 377000 — and both the frontend prod rule (3190653) and the
backend fatal-exception rule (3462179) point at it. A channel named
“fe-prod” is already carrying backend alerts.
| Option | Description | Trade-offs |
|---|---|---|
| A | Reuse #sentry-fe-prod for both projects’ Urgent tiers | Zero new setup; compounds an existing naming problem and mixes urgent with routine first-seen noise. |
| B | New dedicated channel, e.g. #sentry-urgent | Urgent is separable and can carry a distinct notification policy. Requires creating the channel and re-pointing the Sentry Slack integration. |
| C | Per-discipline channels mirroring the Linear split | Consistent with KTLO / BE vs KTLO / FE routing; two more channels to create and watch. |
Recommendation: Option B — a single dedicated Urgent channel. Urgent’s value is that it is rare and unambiguous; putting it in the channel that already receives every first-seen event defeats that.
Decision: #1-dev-team — targetIdentifier C086HQQ076Y,
integrationId 377000. An existing team channel rather than a new dedicated
one, so Urgent notifications land where the team already looks.
Declared as a named constant in the Instance file, not inlined in the Construct. Changing the destination must be a one-line config edit reviewable in a PR — the same requirement that applies to the Urgent threshold (DQ-008).
Note on the existing channel: #sentry-fe-prod (C0ALYG3Q55L) remains the
target for the retained Slack rules, including the backend fatal-exception
rule 3462179. A channel named “fe-prod” carrying backend alerts is a
pre-existing naming problem this project does not fix; see Out of Scope in the
design.
Applied to: design.md § 8 AlertTier, § 8 ArdaSystems, § 9.2, § 13
DQ-022: How configuration is shaped
Section titled “DQ-022: How configuration is shaped”Context: Several values in this design are judgement calls that will be retuned once the new alerting runs against real traffic — the Urgent threshold, the Slack destination, the notification frequency, the Linear routing. They must be changeable in one reviewable place. The question is what shape holds them.
| Option | Description | Trade-offs |
|---|---|---|
| A | Flat named consts in the Instance file (URGENT_THRESHOLD, URGENT_SLACK_CHANNEL, …) | Simple and greppable. But adding a third tier or a second provider means adding a parallel family of constants with no structure tying them together, and nothing stops two constants from disagreeing. |
| B | A nested Configuration interface tree, assembled into one SentryAlerts value | Matches the repo’s Configuration / Props / Built discipline. Scales by adding a node, not by adding a constant family. The type system enforces that every tier is complete. |
| C | External config file (YAML/JSON) read at runtime | Editable without touching TypeScript; loses type checking and the compile-time guarantee that every required value is present. |
Recommendation: Option B.
Decision: Option B. Flat constants were rejected because they do not scale: this configuration is inherently a tree — provider → project → tier → action — and flattening it discards exactly the structure that makes adding a Performance tier or a Grafana provider cheap.
Each layer owns its own Configuration interface, per the repo’s IaC
layering rules, and the Instance file holds a single JSON-like const typed to
the App’s Configuration:
- Construct (
AlertTier) —Configuration(what the consumer decides: tier name, trigger, action, environment, frequency),Props(Configuration+ the project and detectors the Stack injects),Built(the producedWorkflowSpec). - Stack (
SentryFaultAlerting) —Configurationnaming one Sentry project, its detectors, and its ordered tiers. - App (
OperationsManagementApp) —Configurationholding aSentryAlertsblock. This is the App’s own contract, not a mechanical union of stack configs: workspace-level Linear identity (team, label, state) is hoisted to the top so it is stated once and projected down, rather than repeated in every tier. - Instance (
ArdaSystems) — values only, no shape definitions.
AlertTier is a non-Construct value object, like the Postmark
thin-wrappers, so it exports validateProps as a free function rather than
a static method — the documented convention for this case.
Scalability property this buys: a Performance-area tier is a new entry in a
Stack’s tiers array; a new Sentry project is a new entry in
sentryAlerts.projects; a Grafana provider is a sibling key beside
sentryAlerts in the App’s Configuration. None of these require touching the
Construct, the engine, or the drivers.
Applied to: design.md § 5 Constraints, § 8 AlertTier, § 8 ArdaSystems, § 12
Round 5: Continuous Deployment
Section titled “Round 5: Continuous Deployment”DQ-023: Deploy on merge, and how apply is gated
Section titled “DQ-023: Deploy on merge, and how apply is gated”Context: OperationsManagement resources should deploy when a PR merges to
main. This collides head-on with the design’s Constraint 7 — “the tool never
applies unattended” — and with the originating brief’s “never call apply
yourself”. The constraint cannot simply be contradicted; it has to be
rewritten, and how depends on the gate.
| Option | Description | Trade-offs |
|---|---|---|
| A | Automatic plan, then apply behind a protected GitHub Environment with required reviewers | Human confirmation survives, but nobody has to remember to run anything. Constraint 7 becomes “never applies without an approval”, which stays true. |
| B | Fully automatic apply on merge | True CD; main and live can never diverge. But a bad config merge reaches production alerting unreviewed, and Constraint 7 is dropped rather than reworded. |
| C | Plan on merge, apply via workflow_dispatch | Closest to today’s design; main and live silently diverge whenever someone forgets to dispatch. |
Recommendation: Option A.
Decision: Option B — fully automatic. Merge to main runs
snapshot → plan → apply → verify with no approval gate. Directed by the
project owner, overriding the recommendation above.
Constraint 7 is dropped for CI, not reworded. It survives only for
operator-initiated runs, where --confirm still applies. In CI there is no
confirmation step: the PR review is the only human checkpoint, and it happens
before merge rather than before apply.
Consequence, stated plainly: a destructive change merged by mistake reaches production alerting immediately. The plan is published but nobody is required to read it, and by the time it exists the change has already been applied.
What compensates, and what it does not cover:
| Mitigation | Covers | Does not cover |
|---|---|---|
Snapshot uploaded before apply | Recovery after a bad apply | Preventing it |
plan published on the PR (see below) | Reviewer sees the changeset while it is still reviewable | A reviewer who does not look |
verify after apply | Detecting an apply that did not achieve desired state | Detecting an undesired desired state |
| Failure opens a GitHub issue | Nobody has to watch the run | Silent success of a wrong change |
| Monthly drift check | Divergence introduced outside the pipeline | Same-day damage |
Design addition made to serve the intent: the workflow gains a
pull_request trigger that runs snapshot → plan and posts the changeset to
the PR, exiting non-zero if changes are pending. Since the PR review is now the
only human checkpoint, the plan must be visible at review time rather than
only after merge. This adds no friction to merging and is the standard shape for
plan-and-apply IaC pipelines.
Trigger scope: directory globs, never file lists, so adding or removing a file never requires editing the workflow.
on: push: branches: [main] paths: - 'src/main/cdk/apps/OperationsManagement/**' - 'src/main/cdk/instances/OperationsManagement/**' - 'src/main/cdk/stacks/operations-management/**' - 'src/main/cdk/platform/constructs/sentry/**' - 'src/main/cdk/platform/sentry-service.ts' - 'tools/lib/sentry/**' - 'tools/lib/desired-state/**' - 'tools/operations-management-*.ts' - '.github/workflows/operations-management-deploy.yml'Two deliberate choices inside that filter:
- Erring broad. The reconcile engine (
tools/lib/desired-state/**) is included even though it holds no desired state. A spurious run is harmless — an unchanged desired state yields an empty plan and the approval is never requested. The expensive error is the opposite one. platform/one-password.tsis excluded despite holding the token reference, because it is shared with Postmark; a Postmark credential edit must not trigger a Sentry deploy.
Consequential refinement: SentryClient moves from
tools/lib/sentry-client.ts to tools/lib/sentry/client.ts, so the single
glob tools/lib/sentry/** covers the whole provider. This diverges from the
flat postmark-client.ts precedent; robustness of the trigger was judged the
higher value given the stated requirement. Reversible if the consistency matters
more.
Applied to: design.md § 5 Constraint 7, § 8
OperationsManagementDeploy, § 9.3, § 11, § 12
DQ-024: Where the CI snapshot lives
Section titled “DQ-024: Where the CI snapshot lives”Context: apply refuses unless a snapshot still describes live state, and
that snapshot is the rollback source. On an ephemeral runner it would vanish with the
job, which would make the refusal a formality rather than a safety property.
| Option | Description | Trade-offs |
|---|---|---|
| A | Upload as a workflow artifact with explicit retention | No new infrastructure; bound to the exact run that made the change; downloadable during an incident. Bounded by retention — a rolling window, not an archive. |
| B | Bot-commit under snapshots/<ISO8601>/ | Permanent and diffable beside the config that produced it; adds bot commits to main and grows the repo indefinitely. |
| C | Push to S3 | Durable and unbounded; needs a bucket, an OIDC role, and lifecycle rules — real infrastructure for a low-volume artifact. |
Recommendation: Option A.
Decision: Option A — workflow artifact, retention set explicitly (90 days).
The snapshot is uploaded before the apply step, so a failed or partial
apply still has a rollback source, and a plan-only run on a PR still leaves one
behind. With apply now automatic
(DQ-023) this artifact is the
primary recovery path rather than a backstop, which is why upload ordering is
part of the decision rather than an implementation detail.
Accepted limitation: retention is a rolling window. Rollback beyond it depends on the committed fixture baseline (DQ-006), which is permanent but only reflects the state at design time. Anything older than the window is reconstructed from the fixtures plus the git history of the Instance file, not from a snapshot.
Applied to: design.md § 9.3, § 11, § 12
DQ-025: When the workflow is enabled
Section titled “DQ-025: When the workflow is enabled”Context: the migration deletes four workflows and creates four. The deploy pipeline’s first run should not be that.
Decision: enable after the migration lands. The migration is run manually and attended, its result verified, and only then is the workflow turned on — so it only ever handles incremental change and the first automated apply is never the destructive one.
Practical consequence: the workflow file may be committed with the migration
PR, but its on.push trigger stays commented or the workflow stays disabled
until the migration is verified. Whoever enables it should confirm plan is
empty against live first — an empty plan immediately post-migration is the
cleanest possible proof the migration and the declared desired state agree.
Applied to: design.md § 9.4, § 11
Superseded content note: the original DQ-017 framing treated duplicate tickets as the risk. The probe showed the opposite — silent escalation loss — and the decision changed accordingly. DQ-008 likewise changed: the rate-preservation method was superseded by an explicit, deliberately narrower starting value.
DQ-018: Removal mode
Section titled “DQ-018: Removal mode”Decision: Delete outright. Rules 3426957 (duplicate) and the retired
arda-frontend Slack rules (3190649, 3190652) are removed, not disabled.
The committed snapshot is the rollback path.
Known limitation: recreated rules receive new IDs. Any external reference to a deleted rule’s ID breaks permanently. This is a property of the API, not a defect in the rollback script.
Applied to: design.md § 9.4, § 11
DQ-019: Splitting cross-project rule 3153115
Section titled “DQ-019: Splitting cross-project rule 3153115”Context: DQ-010 forces one rule per Sentry project. 3153115 is
cross-project and must become two.
| Option | Description | Trade-offs |
|---|---|---|
| A | Repurpose as backend, create new frontend | Preserves the ID and the uptime attachments. Leaves the two rules structurally asymmetric. |
| B | Repurpose as frontend, create new backend | Symmetric to A; discards the uptime attachments from the surviving rule. |
| C | Delete and create two fresh rules | Both rules produced identically from one Construct, no legacy asymmetry. Loses the ID and its history; a brief window mid-apply where high-priority routing is uncovered. |
Decision: Option C. Symmetry is worth more than ID preservation here, because both rules then derive from a single Construct with no special case — which is precisely the property that makes the Performance tier cheap to add later.
Consequence: apply ordering matters. Creates must precede the delete so the uncovered window is as short as possible, and the plan must mark it.
Applied to: design.md § 9.4, § 11
DQ-020: existing_high_priority_issue trigger
Section titled “DQ-020: existing_high_priority_issue trigger”Context: 3153115 triggers on both new_high_priority_issue and
existing_high_priority_issue. The latter lets already-existing issues
re-trigger, churning tickets.
Decision: Drop existing_high_priority_issue. The replacement rules fire
only on newly-escalated issues.
Trade-off accepted: an issue that was already high-priority when the rules are created will never produce a ticket. Given the rules are being recreated (DQ-019), this means the existing high-priority backlog produces no tickets — which is the intended outcome, not a gap.
Applied to: design.md § 8 HighTier, § 9.2
Round 6: Implementation Findings
Section titled “Round 6: Implementation Findings”DQ-026: Disposition of rule 3190653
Section titled “DQ-026: Disposition of rule 3190653”Context: Surfaced during implementation, not during design. Two settled decisions turned out to be jointly unsatisfiable:
- DQ-011 fixes
3190653’s environment defect in this migration, and the migration plan expressed that as adopting the rule and updating itsenvironmentin place. - DQ-019 requires one rule per
Sentry project, and
3190653is cross-project.
A rename-and-update preserves one workflow. The per-project split needs two. One workflow cannot become two, so the in-place path cannot express the outcome DQ-019 requires — the conflict is structural, not a matter of effort.
| Option | Description | Trade-offs |
|---|---|---|
| A | Delete 3190653, let reconcile create both per-project first-seen tiers | Symmetric with 3153115’s treatment under DQ-019; both rules come from one Construct with no special case. Loses the ID and its history. |
| B | Adopt 3190653 as the arda-frontend first-seen tier, create the platform-be one fresh | Preserves the ID for the project whose attachment actually worked. Leaves the pair asymmetric, and the surviving rule carries hand-built structure the Construct did not emit — the exact shape DQ-019 rejected. |
| C | Defer the split; adopt in place now and split later | Two migrations instead of one, with the environment defect fixed but the routing defect still live in between. |
Decision: Option A, confirmed by Miguel on 2026-08-06. This supersedes the adopt-by-rename disposition implied by DQ-011; DQ-011’s substance — that the defect is repaired in this migration rather than deferred — is unchanged.
Consequence: 3190653 joins the delete list and is subject to the same
ordering rule as 3153115 — creates precede deletes, so the window in which
first-seen notification is uncovered is as short as possible. Its ID is lost
permanently, per the known limitation in
DQ-018.
Applied to: design.md § 9.4;
tools/operations-management-migrate.ts RETIRED_WORKFLOWS
DQ-027: The absent-environment assumption is now measured
Section titled “DQ-027: The absent-environment assumption is now measured”Context: DQ-011 rests on an
assumption that was carried, acknowledged and untested through the whole design:
that a workflow whose environment names an environment a project does not have
matches nothing, rather than matching everything. If it were false, retiring
rule 3190653 would be a deliberate behavior change rather than a repair, and
the design and PR would be describing it wrongly.
Method: three workflows on platform-be’s issue_stream detector, all
firing on first_seen_event, differing only in the variable under test — A
scoped to Alpha002-dev (positive control), B to production (which
platform-be does not register), C to Alpha002-dev with the Linear action. One
synthetic error was then injected under Alpha002-dev and lastTriggered polled
on each. Without the control, B not firing would have proved nothing.
Result, 2026-08-06:
| Workflow | Environment | Fired |
|---|---|---|
| A — Slack | Alpha002-dev | yes, 00:19:21.181898Z |
| B — Slack | production | no, lastTriggered stayed null |
| C — Linear | Alpha002-dev | yes, 00:19:21.181943Z |
Decision: the assumption is confirmed and is no longer an assumption.
Rule 3190653’s platform-be attachment was inert, and retiring it repairs a
defect rather than changing behavior.
Delivery was confirmed at the same time, which is a separate thing from a
rule firing: A produced a message in #sentry-fe-prod naming
workflow_id=3818157, and C produced PDEV-1434 in KTLO / BE at High
priority — the routing DQ-010 specifies.
Incidental finding: the probe event also tripped the two live cross-project
rules, 3153115 and 3426957, at the same instant — re-confirming they are
duplicates. 3153115 produced PDEV-1433 with no project and Urgent
priority, which is precisely the untargeted routing DQ-010 replaces. The two
tickets side by side are the clearest statement of what this migration buys.
Applied to: design.md § 13
DQ-028: What exactly is the managed set?
Section titled “DQ-028: What exactly is the managed set?”Context: Three definitions were in circulation, and the documents did not agree with the code.
DQ-003 decided pinned IDs ∩ name
prefix, and explicitly rejected prefix-only as its Option C. The code shipped
prefix only: identify() claims any workflow whose name starts with
arda/, and the planner deletes every claimed workflow that no desired entry
holds an ID for. The guidance-compliance audit found four documents asserting
the intersection while the code did something else.
Then implementation forced the question. The first real apply, on 2026-08-06,
created six workflows and printed their assigned IDs for pinning. verify, run
immediately afterwards, proposed deleting all six — they carried the prefix,
so they were managed; configuration held no IDs for them, so nothing claimed
them; and managed-minus-claimed is the delete rule. The tool’s own desired state
said destroy what you just built.
That is not a corner case. Merge runs apply --confirm unattended
(DQ-023), so between an apply
and the follow-up commit recording IDs, the next merge would have deleted the
live alerting. Only the sequencing convention in
DQ-025 stood in the way, and a
convention is not a mechanism.
| Option | Description | Trade-offs |
|---|---|---|
| A | Restore DQ-003 literally: identify returns null unless the ID is pinned | Matches the recorded decision. But an unpinned prefixed workflow becomes permanently invisible — never updated, never removed, and re-created on every run, accumulating duplicates. Trades a destructive failure for a silent one. |
| B | Keep prefix-only | What shipped. Simple, and orphan cleanup works. Retains the destructive window, which is unacceptable given unattended apply. |
| C | Prefix ∩ declared, where declared means pinned ID or natural key | Closes the window without weakening the prefix boundary. Pinning becomes an optimisation rather than a correctness requirement. Costs a second identity path, and requires names to be unique within the managed set. |
Decision: Option C, confirmed by Miguel on 2026-08-06.
The rule, stated once, for every other document to copy:
A workflow is managed when its name carries the
arda/prefix. A managed workflow is claimed when a declared tier matches it, by the key embedded in its name (DQ-029).planproposes deleting a managed workflow only when nothing claims it. A workflow without the prefix is untracked: reported, never touched.
Consequences
- Pinned IDs remain the primary identity and are still recorded after an apply. Name matching is a fallback, reported in plan output as adopted, so that beginning to manage something is never silent.
- Names must be unique within the managed set. Ambiguity throws rather than guessing — this org contains two live rules sharing one name, which is the observation that motivated ID pinning in DQ-003 in the first place.
arda/becomes a reserved namespace. A hand-made rule namedarda/…and not declared will be proposed for deletion. Recorded in the operator README.- Deleting a tier from configuration still removes the live workflow — orphan cleanup, which Option A would have lost.
Applied to: design.md § 8, § 9.1, § 9.3;
tools/lib/desired-state/plan.ts, tools/lib/desired-state/resource.ts,
tools/lib/sentry/issue-alert.ts, scripts/sentry/README.md
DQ-029: Where identity lives
Section titled “DQ-029: Where identity lives”Context: DQ-028 left two identity paths — a pinned id, falling back to a natural key. That settled the safety question but not the design one: why keep a mechanism whose only remaining job is a case the fallback already handles?
The pinned id also carried a real cost. It cannot be known before the resource exists, so it required a manual transcription step after every create, and the window before that step was — until the natural key existed — actively destructive. It is a hand-maintained state file, and the tooling already writes state files.
The question is where a client-assigned identity can live at all. Sentry’s
workflow schema has no free-form metadata field, and POST does not accept
a caller-supplied id, so the name is the only field this tooling controls that
survives a round trip.
| Option | Description | Trade-offs |
|---|---|---|
| A | Keep pinned ids as primary, key as fallback (DQ-028 as written) | No change. Retains a manual step and two paths that must agree. |
| B | Derive ids from the newest snapshot instead of configuration | Removes the manual step; snapshots already hold the mapping. But identity stops being visible in the PR diff, which is what DQ-003 wanted it for. |
| C | Embed a stable key in the workflow name; remove pinned ids | One identity, visible in configuration and in Sentry, needing no post-hoc step. Costs a suffix on a human-facing string, and names must be unique. |
Decision: Option C, confirmed by Miguel on 2026-08-07.
The name becomes arda/High — platform-be [high-be]: prefix, display name, key.
Readable part first, because this string is what an on-call engineer reads at
the bottom of a Slack alert.
Why the key is separate from tierName. Renaming a tier’s display name is a
legitimate edit and must produce an update; matching on the whole name would
make it a delete-and-recreate. Changing a key means “this is a different
tier”, and the old workflow is correctly deleted. Conflating them would remove
the ability to say which was meant.
Consequences
pinnedIdis gone from the construct, Stack, App and instance file, and with itassertPinnedIdsAreManagedandUnmanagedPinnedIdError, whose only job was guarding a pinned id against pointing outside the managed set.- Keys must be unique across every tier. Asserted by the App at build time, rather than surfacing at plan time as an ambiguity error against the live org.
- The key pattern is constrained to lowercase kebab-case so compose and parse round-trip unambiguously. A malformed suffix parses as no key rather than as a guess.
- This is a narrowing, not a loosening. A workflow is claimed when its key is declared, and by nothing else.
Transition, executed 2026-08-07: the rename applied as six in-place updates
while the pinned ids were still present, because only a pinned id could match a
live workflow whose old name carried no key. The ids were removed immediately
afterwards, and plan then reported no changes with all six matched by key
alone — which is the evidence that identity survived the change.
Applied to: design.md § 8, § 9.1;
platform/sentry-service.ts (composeWorkflowName, parseWorkflowKey),
platform/constructs/sentry/alert-tier.ts, tools/lib/sentry/issue-alert.ts
DQ-030: Urgent tier destination
Section titled “DQ-030: Urgent tier destination”Context: DQ-021 chose
#1-dev-team on the reasoning that escalation should land where people already
look, and that a dedicated channel nobody watches is worse than a busy one.
Decision: #sre-production (C0BNTF2AJ6N), on Miguel’s direction,
2026-08-07. A channel dedicated to production operations is the better home: a
burst clearing this threshold should not compete with conversation for
attention. DQ-021’s concern is answered by the channel having an audience by
construction rather than by being general-purpose.
Precondition, and it is not optional. Sentry’s Slack integration can only
post to channels its app has joined. A workflow pointing at a channel the app
is absent from is accepted at write time and then silently drops every
notification — the same failure shape as rule 3190653. @sentry was confirmed
present in #sre-production before the change was applied.
Delivery itself remains unproven: no event has yet fired into the channel.
Applied to: design.md § 8;
instances/OperationsManagement/arda-systems.ts
Copyright: (c) Arda Systems 2025-2026, All rights reserved
Copyright: © Arda Systems 2025-2026, All rights reserved