Skip to content

Decision Log: Sentry Configuration as Code

Decisions for the Sentry configuration-as-code capability — the first inhabitant of the OperationsManagement platform element — and for the alert-rule migration it executes. Companion to design.md and goal.md.

Several decisions here overturned assumptions carried in the originating prompt. Where that happened, the evidence is recorded, because the assumption was reasonable and a future reader will otherwise wonder why the design diverges from it.

#QuestionStatusDecisionRound
DQ-001Where does the code live?DecidedOperationsManagement platform element, layered per the repo vocabularyR1
DQ-002How does the Application execute?DecidedDeclarative App + our own reconcile engineR1
DQ-003Resource identity and managed setDecidedPinned IDs + required managed-name prefixR1
DQ-004Which Sentry API surface?DecidedOrg-scoped /workflows/; legacy /rules/ is a lossy shimR1
DQ-005Are detectors a managed resource?DecidedRead-only references, resolved symbolicallyR1
DQ-006Where do fixtures/snapshots live?DecidedCommitted under the repoR1
DQ-007Lookahead for Performance ManagementDecidedDesign the seam, implement Fault onlyR1
DQ-008Urgent tier window (30m unavailable)Decided3 in 15m, as a tunable starting valueR2
DQ-009Urgent tier scopingDecidedTwo rules, one per Sentry projectR2
DQ-010Linear project routingDecidedKTLO / BE and KTLO / FE separatelyR2
DQ-011Rule 3190653 environment defectDecidedFix in this migrationR2
DQ-012Environment naming divergenceDecidedLogical-environment abstraction mapping to physical namesR2
DQ-013Uptime detector attachmentDecidedPreserve, flag explicitly in plan outputR2
DQ-014Product Viewpoint personaDecidedAuthor the Platform Operator personaR1
DQ-015assigneeId on the template ruleDecidedStrip explicitly; assignment stays a Linear concernR2
DQ-016stateId (undeclared in the prompt)DecidedReuse TriageR2
DQ-017High → Urgent escalation handlingDecidedUrgent notifies via Slack; only High creates Linear ticketsR3
DQ-018Removal modeDecidedDelete outrightR3
DQ-019Splitting cross-project rule 3153115DecidedDelete and create two fresh rulesR3
DQ-020existing_high_priority_issue triggerDecidedDrop itR3
DQ-021Slack channel for the Urgent tierDecided#1-dev-team (C086HQQ076Y), as a named constantR4
DQ-022How configuration is shapedDecidedNested Configuration/Props/Built tree assembled as one SentryAlerts valueR4
DQ-023Deploy on merge, and how apply is gatedDecidedFully automatic on merge; plan also posted on the PR. Drops Constraint 7 for CIR5
DQ-024Where the CI snapshot livesDecidedWorkflow artifact, 90-day retention, uploaded before applyR5
DQ-025When the workflow is enabledDecidedAfter the migration lands and plan is verified emptyR5
DQ-026Disposition of rule 3190653DecidedDelete and recreate, superseding adopt-by-renameR6
DQ-027The absent-environment assumptionMeasuredConfirmed by controlled probe; 3190653 was inertR6
DQ-028What exactly is the managed set?DecidedPrefix ∩ declared. Supersedes part of DQ-003R6
DQ-029Where identity livesDecidedA key embedded in the workflow name; pinned ids removed entirely. Supersedes DQ-003 Option AR6
DQ-030Urgent tier destinationDecided#sre-production (C0BNTF2AJ6N). Supersedes DQ-021R6

Context: The originating prompt proposed a standalone layout (src/, bin/, config/, docs/ at repo root) with zero runtime dependencies and Node native type stripping. The infrastructure repo has an established four-layer structure and a documented tools/ vs scripts/ split.

OptionDescriptionTrade-offs
AStandalone sub-package under tools/sentry/, own package.jsonHonors the prompt literally. Precedent exists (tools/create-aws-accounts/). But duplicates postmark-client.ts’s retry/backoff and stands up a second toolchain.
BIntegrated across the existing layers as a new platform elementReuses tools/lib/, the drift family, and the platform/ metadata layer. Requires defining the OperationsManagement element.
CMigration script only, no librarySmallest change; leaves nothing reusable.

Recommendation: Option B.

Decision: Option B. The capability is the first inhabitant of a fourth platform element, OperationsManagement, alongside Environments, Root, and the informal tools/scripts collection. Spelled in full in paths — never oam, which already denotes infrastructure-supporting constructs under src/main/cdk/constructs/oam/.

Applied to:

  • goal.md § Architecture Placement, § Constraints
  • infrastructure/knowledge-base/platform-architecture.md (new)
  • infrastructure/CLAUDE.md § Platform elements

Context: In this repo, cdk names IaC machinery generally, and an Application normally composes Stacks that synthesize CloudFormation. Sentry workflows have no CloudFormation representation.

OptionDescriptionTrade-offs
ADeclarative App producing a desired-state tree; our reconcile engine applies itKeeps script → instances → apps → stacks → constructs intact. The engine replaces CloudFormation as executor. Requires building diff/plan/apply.
BCloudFormation custom resources backed by LambdaNative cdk deploy and CFN drift. But needs Lambda plumbing, puts the Sentry token in the deploy path, and couples the OAM plane to AWS unnecessarily.
CConfig module feeding an engine, no App/Stack layersLeast code; abandons the platform framing and leaves future OAM providers no shape to follow.

Recommendation: Option A.

Decision: Option A. apps/OperationsManagement does not construct a cdk.App(); it produces a typed desired-state tree that the entry script hands to the reconcile engine.

Applied to: design.md § 8 Structural Design, § 9.1


Context: The workflow schema has no free-form metadata field — nowhere to store a logical key. Worse, name is not unique: workflows 3153115 and 3426957 share the name “Send a notification for high priority issues”. A plan that proposed deleting everything absent from config would be catastrophic, since the org contains rules this project does not own.

OptionDescriptionTrade-offs
APin workflow IDs in the Instance file; require a managed-name prefix as a safety assertionUnambiguous matching plus a guard against a typo pointing at an unmanaged rule. Config carries server-generated IDs.
BPin IDs onlySimple; no guard against mis-targeting.
CName-prefix only, no IDsConfig reads cleanly; a UI rename silently orphans a rule and the tool proposes recreating it.

Recommendation: Option A.

Decision: Option A. The managed set is the intersection of pinned IDs and the name prefix. The engine never proposes deleting a workflow outside that set; unmanaged workflows are reported as untracked, not as drift.

Superseded in part by DQ-028. The intersection as stated here has a destructive gap: a workflow the tool has just created carries the prefix but has no pinned ID yet, so it is claimed as managed and matched by nothing — which is the delete rule. Identity is now pinned ID or natural key. DQ-029 then removed pinned ids altogether, leaving the key as the only path. Option A’s substance survives in that identity is still explicit and declared, and the prefix is still a hard boundary — only its carrier changed.

Applied to: design.md § 8, § 9.3


Context: The prompt specified read-modify-write against issue alert rules with conditions[] / filters[] / actions[], and cited rule IDs verified from the Sentry UI. Recon on 2026-08-05 contradicted this.

Evidence:

  • GET /organizations/arda-systems/workflows/ returns all seven rules with exactly the cited IDs.
  • GET /projects/arda-systems/arda-frontend/rules/ returns only five entries, with different IDs (16782074, 16817922, 16817925, 16817926, and a synthetic 10003433641 = 10000000000 + 3433641).
  • GET /projects/arda-systems/arda-frontend/rules/3433641/404. The cited IDs do not exist in the legacy namespace.
OptionDescriptionTrade-offs
ATarget /workflows/ exclusivelyThe only surface where these rules exist. Schema must be derived from live payloads.
BTarget the legacy /rules/ shimMatches the prompt, and matches most published documentation — but the IDs don’t resolve and writes would pass through a lossy translation.

Recommendation: Option A.

Decision: Option A. The org is on Sentry’s workflow engine. The legacy endpoint is a read shim and is not used.

Applied to: design.md § 2, § 8; the API observations note


Context: Workflows attach to projects through detectorIds, not a project list. Detectors are project-scoped monitors (error, issue_stream, uptime_domain_failure) carrying live monitoring configuration.

OptionDescriptionTrade-offs
ARead-only references, named symbolically in config (arda-frontend/issue_stream), resolved to IDs at plan timeConfig is legible and survives detector recreation. No risk of the tool mutating monitoring config.
BFull managed resource with create/update/deleteNeeded eventually for uptime monitors (a Performance concern); large scope now.
CRaw detector ID strings in configTrivial to build; opaque, and breaks on recreation.

Recommendation: Option A.

Decision: Option A.

Applied to: design.md § 8 DetectorResolver


DQ-006: Where do fixtures and snapshots live?

Section titled “DQ-006: Where do fixtures and snapshots live?”

Decision: Committed under the repo. They are the type-derivation evidence and the rollback source for a destructive migration, roughly 1 KB each, and contain no secrets — IDs and settings only. A snapshot nobody can locate is not a rollback source.

Applied to: design.md § 12


DQ-007: How far to anticipate Performance Management?

Section titled “DQ-007: How far to anticipate Performance Management?”

Decision: Design the seam; implement Fault only. The resource interface, registry, engine, and Stack-layer split accommodate metric alerts and dashboards; only issue alert workflows are built and verified. Extensibility is claimed by structure and demonstrated by review, not by speculative code.

Applied to: design.md § 8, § 12 Out of Scope


Context: The persona catalog contains only business personas. OAM-plane work serves an operator role with no entry.

Decision: Author a Platform Operator persona. All future OAM work needs it, and the design framework’s Product Viewpoint is otherwise unanswerable for this class of work.

Applied to: product/personas/priya-platform-operator.md (new)


Round 2 exists because recon against the live org surfaced facts that no amount of design reasoning would have produced.

Context: The prompt specified urgentThreshold: { count: 3, windowMinutes: 30 }. Probing event_frequency_count returned a closed interval enum: ['1m', '5m', '15m', '1h', '1d', '1w', '30d']. 30 minutes is not offered.

OptionDescriptionTrade-offs
A3 in 1hStrictly more sensitive than intended; more Urgent tickets.
B3 in 15mStrictly less sensitive; a steady 3-per-30-min failure never escalates.
CPreserve the rate (~6/hour), e.g. 6 in 1hKeeps intended sensitivity rather than intended numbers.

Recommendation: Option C.

Decision: Option B — 3 occurrences in 15 minutes, as an explicitly tunable starting value.

This supersedes the rate-preservation method originally chosen. Worth stating plainly rather than burying: 3-in-15m is stricter than the specified 3-in-30m — it implies ~12/hour against the original ~6/hour — so the Urgent tier starts less sensitive than the prompt intended and will only catch sharp bursts. That is deliberate. Starting narrow means the first weeks produce too few Urgent notifications rather than too many, which is the safer error for a tier whose entire value is being rare and unambiguous.

The pair is a named constant in the Instance file precisely so it can be widened once real issue-frequency data exists. Retuning is a one-line config change and a PR, not a redesign.

Applied to: design.md § 8 AlertTier, § 8 ArdaSystems, § 9.2


Context: See DQ-012 — the two Sentry projects have disjoint environment names, and a workflow carries a single environment string.

Decision: Two rules, one per Sentry project, each with its own physical environment name and its own detectors. The Construct is parametric on logical environment, so this is one definition instantiated twice.

Applied to: design.md § 8, § 9.2


Context: projectId is empty on every existing Linear action, so there was no value to copy. Resolved live: team 4e4f6afc-… is Product Development; labelId 7f70fbff-… is “Bug”; stateId e6779135-… is “Triage”.

Decision: Route per discipline — KTLO / BE (b9b219b9-1c61-4602-b7bd-9697cd9f6b95) for platform-be, KTLO / FE (410e9402-d9cd-4f93-9421-5ba9ac21f464) for arda-frontend.

Consequence: since a workflow emits one action config, per-discipline routing forces one rule per Sentry project for every tier — not just Urgent. This is what makes DQ-019 necessary.

Applied to: design.md § 8, § 9.2


DQ-011 / DQ-012: Environment naming divergence

Section titled “DQ-011 / DQ-012: Environment naming divergence”

Context: The projects register disjoint environment names — arda-frontend: dev, development, local, production, stage; platform-be: Alpha001-demo, Alpha001-prod, Alpha002-dev, Alpha002-stage. No name is common to both. Sentry registers environments from incoming events, so an absent name means no event ever carried it.

Consequence: rule 3190653 carries environment: "production" and attaches to both projects, so its platform-be attachment can never match. Backend first-seen Slack alerting from that rule is inert.

Decision (DQ-011): Fix it in this migration rather than deferring — the rules are being restructured anyway, and the gap is a live alerting hole.

Decision (DQ-012): The divergence is an accident of how the projects were first created, not a deliberate scheme. The names map semantically, so the design introduces a logical environment abstraction resolved to physical names per project:

Logicalarda-frontendplatform-be
prodproductionAlpha001-prod
stagestageAlpha002-stage
devdevAlpha002-dev
demoAlpha001-demo

development and local on arda-frontend are unmapped and treated as local-development noise.

Applied to: design.md § 8 LogicalEnvironment, § 9.2; platform/sentry-service.ts


Context: 3153115 attaches to two uptime_domain_failure detectors on platform-be in addition to issue streams, so high-priority Linear routing currently also covers uptime failures.

Decision: Preserve the behavior; make it explicit in plan output so the carry-over is deliberate and visible rather than accidental.

Applied to: design.md § 9.2


Context: Rule 3433641 — designated as the template because it carries a working labelId and stateId — also carries assigneeId: a6e34222-…. Assignment is deliberately a Linear-side concern.

Decision: Strip assigneeId explicitly when deriving from the template. A verbatim copy would silently pre-assign every ticket.

Applied to: design.md § 8 LinearRoutedAlertTier


Context: Not mentioned in the prompt’s decision block, yet set on both existing Linear actions to the same value.

Decision: Reuse e6779135-… (Triage). Sentry-created tickets landing in Triage matches Linear’s documented behavior for integration-created issues and the team’s existing practice.

Applied to: design.md § 8 LinearRoutedAlertTier


DQ-017: High → Urgent escalation handling

Section titled “DQ-017: High → Urgent escalation handling”

Context: An issue may fire the High rule and later cross the Urgent threshold. The originating prompt asserted this would create a second Linear ticket. That assertion was unverified, and Linear’s documentation is silent on de-duplication of repeat firings, so it was probed empirically on 2026-08-06.

Probe method: two throwaway workflows on platform-be scoped to Alpha002-dev, routing to a throwaway Linear project; seven synthetic events carrying an explicit fingerprint so they collapsed into exactly one Sentry issue (7655326314). Rule A triggered on first_seen_event at Linear priority 2; rule B on event_frequency_count at priority 1, two minutes later. Both workflows deleted afterwards, inventory verified back at seven.

Result — the prompt’s assumption was wrong, and the real behavior is worse:

TimeEvent
00:48:03.421Rule A fires — and so do pre-existing 3153115 and 3426957, the same instant
00:48:05.882PDEV-1421 created, priority 1, no project (from 3153115)
00:48:07.638PDEV-1422 created, priority 2, throwaway project (from rule A)
00:50:02.665Rule B fires — priority 1, same Linear project, same Sentry issue
00:50:05.7Both tickets’ updatedAt bump. No new ticket. No priority change.

Two facts follow:

  1. A temporally separated firing is a silent no-op. Rule B asked for priority 1, fired two minutes after the ticket existed, and PDEV-1422 stayed at priority 2. Escalation left no trace but a timestamp.
  2. Simultaneous firings race and both create. 3153115 and rule A produced two tickets 1.8 seconds apart for one Sentry issue.

So the integration gives duplicates when rules fire together and silence when they fire apart — and High → Urgent is inherently the “apart” case.

Two hypotheses remain indistinguishable from this data: dedup keyed on the Sentry issue with a race window (H1), or keyed on (Sentry issue × Linear project) (H2). Distinguishing them needs a later-firing rule targeting a different Linear project; that probe was not run because the chosen decision makes it moot.

OptionDescriptionTrade-offs
AUrgent notifies via Slack; only High creates Linear ticketsEscalation becomes visible without depending on ticket creation or priority upgrade. Correct under both H1 and H2. Urgent produces no ticket, so escalation is not tracked as work.
BProbe H2, then route Urgent to a separate Linear projectPreserves two-tier ticketing if H2 holds. Another probe, and two Linear projects per discipline.
CSingle Linear tier, escalate by hand in triageSimplest; escalation depends entirely on someone noticing.
DAccept the blind spotRejected — this is precisely the silent-failure mode the project exists to eliminate.

Decision: Option A. The Urgent tier emits a Slack action rather than a Linear action. Linear ticketing remains a High-tier concern, one ticket per Sentry issue, which is what the integration reliably delivers. Escalation is surfaced as a notification, where duplication is harmless and silence is not possible.

Consequence: the tier Construct is no longer Linear-specific. It becomes AlertTier, parametric on its action, with linearAction() and slackAction() builders. linearProjectId is a High-tier concern only.

The probe mutated two shared workspaces. The residue is deliberately retained until the implementation PR is reviewed, because it is the only first-hand evidence for a finding that contradicted the project’s stated premise, and contradicted findings are the ones most worth being able to re-check.

ArtifactStateRemoval
[IAC PROBE] DQ-017 Linear projectpresentUI delete — no MCP operation exists
PDEV-1421 / PDEV-1422canceled, priority cleareddelete with the project
Sentry issue 7655326314ignored, 7 synthetic events, tag iac_probe:dq017UI delete — the tooling token lacks event:admin
Probe workflows 3813713 / 3813714deleted, 404-verifieddone; inventory back at seven

A second round of probes on 2026-08-06 (see DQ-027) left:

ArtifactStateRemoval
PDEV-1433 / PDEV-1434canceleddelete alongside the DQ-017 tickets
Sentry issue 7657516175ignored, 1 synthetic event, tag iac_probeUI delete — same token limitation
Probe workflows 3818144–3818159deleted, inventory re-verified at sevendone

Whoever merges the implementation PR should remove the retained items. Both DQ-017 tickets link to Sentry issue 7655326314; their creation timestamps and priorities are the evidence recorded in the table above.

Applied to:


Context: DQ-017 routes the Urgent tier to Slack. The org has exactly one Slack target configured — #sentry-fe-prod, targetIdentifier C0ALYG3Q55L, integrationId 377000 — and both the frontend prod rule (3190653) and the backend fatal-exception rule (3462179) point at it. A channel named “fe-prod” is already carrying backend alerts.

OptionDescriptionTrade-offs
AReuse #sentry-fe-prod for both projects’ Urgent tiersZero new setup; compounds an existing naming problem and mixes urgent with routine first-seen noise.
BNew dedicated channel, e.g. #sentry-urgentUrgent is separable and can carry a distinct notification policy. Requires creating the channel and re-pointing the Sentry Slack integration.
CPer-discipline channels mirroring the Linear splitConsistent with KTLO / BE vs KTLO / FE routing; two more channels to create and watch.

Recommendation: Option B — a single dedicated Urgent channel. Urgent’s value is that it is rare and unambiguous; putting it in the channel that already receives every first-seen event defeats that.

Decision: #1-dev-teamtargetIdentifier C086HQQ076Y, integrationId 377000. An existing team channel rather than a new dedicated one, so Urgent notifications land where the team already looks.

Declared as a named constant in the Instance file, not inlined in the Construct. Changing the destination must be a one-line config edit reviewable in a PR — the same requirement that applies to the Urgent threshold (DQ-008).

Note on the existing channel: #sentry-fe-prod (C0ALYG3Q55L) remains the target for the retained Slack rules, including the backend fatal-exception rule 3462179. A channel named “fe-prod” carrying backend alerts is a pre-existing naming problem this project does not fix; see Out of Scope in the design.

Applied to: design.md § 8 AlertTier, § 8 ArdaSystems, § 9.2, § 13


Context: Several values in this design are judgement calls that will be retuned once the new alerting runs against real traffic — the Urgent threshold, the Slack destination, the notification frequency, the Linear routing. They must be changeable in one reviewable place. The question is what shape holds them.

OptionDescriptionTrade-offs
AFlat named consts in the Instance file (URGENT_THRESHOLD, URGENT_SLACK_CHANNEL, …)Simple and greppable. But adding a third tier or a second provider means adding a parallel family of constants with no structure tying them together, and nothing stops two constants from disagreeing.
BA nested Configuration interface tree, assembled into one SentryAlerts valueMatches the repo’s Configuration / Props / Built discipline. Scales by adding a node, not by adding a constant family. The type system enforces that every tier is complete.
CExternal config file (YAML/JSON) read at runtimeEditable without touching TypeScript; loses type checking and the compile-time guarantee that every required value is present.

Recommendation: Option B.

Decision: Option B. Flat constants were rejected because they do not scale: this configuration is inherently a tree — provider → project → tier → action — and flattening it discards exactly the structure that makes adding a Performance tier or a Grafana provider cheap.

Each layer owns its own Configuration interface, per the repo’s IaC layering rules, and the Instance file holds a single JSON-like const typed to the App’s Configuration:

  • Construct (AlertTier) — Configuration (what the consumer decides: tier name, trigger, action, environment, frequency), Props (Configuration + the project and detectors the Stack injects), Built (the produced WorkflowSpec).
  • Stack (SentryFaultAlerting) — Configuration naming one Sentry project, its detectors, and its ordered tiers.
  • App (OperationsManagementApp) — Configuration holding a SentryAlerts block. This is the App’s own contract, not a mechanical union of stack configs: workspace-level Linear identity (team, label, state) is hoisted to the top so it is stated once and projected down, rather than repeated in every tier.
  • Instance (ArdaSystems) — values only, no shape definitions.

AlertTier is a non-Construct value object, like the Postmark thin-wrappers, so it exports validateProps as a free function rather than a static method — the documented convention for this case.

Scalability property this buys: a Performance-area tier is a new entry in a Stack’s tiers array; a new Sentry project is a new entry in sentryAlerts.projects; a Grafana provider is a sibling key beside sentryAlerts in the App’s Configuration. None of these require touching the Construct, the engine, or the drivers.

Applied to: design.md § 5 Constraints, § 8 AlertTier, § 8 ArdaSystems, § 12


DQ-023: Deploy on merge, and how apply is gated

Section titled “DQ-023: Deploy on merge, and how apply is gated”

Context: OperationsManagement resources should deploy when a PR merges to main. This collides head-on with the design’s Constraint 7 — “the tool never applies unattended” — and with the originating brief’s “never call apply yourself”. The constraint cannot simply be contradicted; it has to be rewritten, and how depends on the gate.

OptionDescriptionTrade-offs
AAutomatic plan, then apply behind a protected GitHub Environment with required reviewersHuman confirmation survives, but nobody has to remember to run anything. Constraint 7 becomes “never applies without an approval”, which stays true.
BFully automatic apply on mergeTrue CD; main and live can never diverge. But a bad config merge reaches production alerting unreviewed, and Constraint 7 is dropped rather than reworded.
CPlan on merge, apply via workflow_dispatchClosest to today’s design; main and live silently diverge whenever someone forgets to dispatch.

Recommendation: Option A.

Decision: Option B — fully automatic. Merge to main runs snapshotplanapplyverify with no approval gate. Directed by the project owner, overriding the recommendation above.

Constraint 7 is dropped for CI, not reworded. It survives only for operator-initiated runs, where --confirm still applies. In CI there is no confirmation step: the PR review is the only human checkpoint, and it happens before merge rather than before apply.

Consequence, stated plainly: a destructive change merged by mistake reaches production alerting immediately. The plan is published but nobody is required to read it, and by the time it exists the change has already been applied.

What compensates, and what it does not cover:

MitigationCoversDoes not cover
Snapshot uploaded before applyRecovery after a bad applyPreventing it
plan published on the PR (see below)Reviewer sees the changeset while it is still reviewableA reviewer who does not look
verify after applyDetecting an apply that did not achieve desired stateDetecting an undesired desired state
Failure opens a GitHub issueNobody has to watch the runSilent success of a wrong change
Monthly drift checkDivergence introduced outside the pipelineSame-day damage

Design addition made to serve the intent: the workflow gains a pull_request trigger that runs snapshotplan and posts the changeset to the PR, exiting non-zero if changes are pending. Since the PR review is now the only human checkpoint, the plan must be visible at review time rather than only after merge. This adds no friction to merging and is the standard shape for plan-and-apply IaC pipelines.

Trigger scope: directory globs, never file lists, so adding or removing a file never requires editing the workflow.

on:
push:
branches: [main]
paths:
- 'src/main/cdk/apps/OperationsManagement/**'
- 'src/main/cdk/instances/OperationsManagement/**'
- 'src/main/cdk/stacks/operations-management/**'
- 'src/main/cdk/platform/constructs/sentry/**'
- 'src/main/cdk/platform/sentry-service.ts'
- 'tools/lib/sentry/**'
- 'tools/lib/desired-state/**'
- 'tools/operations-management-*.ts'
- '.github/workflows/operations-management-deploy.yml'

Two deliberate choices inside that filter:

  • Erring broad. The reconcile engine (tools/lib/desired-state/**) is included even though it holds no desired state. A spurious run is harmless — an unchanged desired state yields an empty plan and the approval is never requested. The expensive error is the opposite one.
  • platform/one-password.ts is excluded despite holding the token reference, because it is shared with Postmark; a Postmark credential edit must not trigger a Sentry deploy.

Consequential refinement: SentryClient moves from tools/lib/sentry-client.ts to tools/lib/sentry/client.ts, so the single glob tools/lib/sentry/** covers the whole provider. This diverges from the flat postmark-client.ts precedent; robustness of the trigger was judged the higher value given the stated requirement. Reversible if the consistency matters more.

Applied to: design.md § 5 Constraint 7, § 8 OperationsManagementDeploy, § 9.3, § 11, § 12


Context: apply refuses unless a snapshot still describes live state, and that snapshot is the rollback source. On an ephemeral runner it would vanish with the job, which would make the refusal a formality rather than a safety property.

OptionDescriptionTrade-offs
AUpload as a workflow artifact with explicit retentionNo new infrastructure; bound to the exact run that made the change; downloadable during an incident. Bounded by retention — a rolling window, not an archive.
BBot-commit under snapshots/<ISO8601>/Permanent and diffable beside the config that produced it; adds bot commits to main and grows the repo indefinitely.
CPush to S3Durable and unbounded; needs a bucket, an OIDC role, and lifecycle rules — real infrastructure for a low-volume artifact.

Recommendation: Option A.

Decision: Option A — workflow artifact, retention set explicitly (90 days). The snapshot is uploaded before the apply step, so a failed or partial apply still has a rollback source, and a plan-only run on a PR still leaves one behind. With apply now automatic (DQ-023) this artifact is the primary recovery path rather than a backstop, which is why upload ordering is part of the decision rather than an implementation detail.

Accepted limitation: retention is a rolling window. Rollback beyond it depends on the committed fixture baseline (DQ-006), which is permanent but only reflects the state at design time. Anything older than the window is reconstructed from the fixtures plus the git history of the Instance file, not from a snapshot.

Applied to: design.md § 9.3, § 11, § 12


Context: the migration deletes four workflows and creates four. The deploy pipeline’s first run should not be that.

Decision: enable after the migration lands. The migration is run manually and attended, its result verified, and only then is the workflow turned on — so it only ever handles incremental change and the first automated apply is never the destructive one.

Practical consequence: the workflow file may be committed with the migration PR, but its on.push trigger stays commented or the workflow stays disabled until the migration is verified. Whoever enables it should confirm plan is empty against live first — an empty plan immediately post-migration is the cleanest possible proof the migration and the declared desired state agree.

Applied to: design.md § 9.4, § 11


Superseded content note: the original DQ-017 framing treated duplicate tickets as the risk. The probe showed the opposite — silent escalation loss — and the decision changed accordingly. DQ-008 likewise changed: the rate-preservation method was superseded by an explicit, deliberately narrower starting value.


Decision: Delete outright. Rules 3426957 (duplicate) and the retired arda-frontend Slack rules (3190649, 3190652) are removed, not disabled. The committed snapshot is the rollback path.

Known limitation: recreated rules receive new IDs. Any external reference to a deleted rule’s ID breaks permanently. This is a property of the API, not a defect in the rollback script.

Applied to: design.md § 9.4, § 11


DQ-019: Splitting cross-project rule 3153115

Section titled “DQ-019: Splitting cross-project rule 3153115”

Context: DQ-010 forces one rule per Sentry project. 3153115 is cross-project and must become two.

OptionDescriptionTrade-offs
ARepurpose as backend, create new frontendPreserves the ID and the uptime attachments. Leaves the two rules structurally asymmetric.
BRepurpose as frontend, create new backendSymmetric to A; discards the uptime attachments from the surviving rule.
CDelete and create two fresh rulesBoth rules produced identically from one Construct, no legacy asymmetry. Loses the ID and its history; a brief window mid-apply where high-priority routing is uncovered.

Decision: Option C. Symmetry is worth more than ID preservation here, because both rules then derive from a single Construct with no special case — which is precisely the property that makes the Performance tier cheap to add later.

Consequence: apply ordering matters. Creates must precede the delete so the uncovered window is as short as possible, and the plan must mark it.

Applied to: design.md § 9.4, § 11


DQ-020: existing_high_priority_issue trigger

Section titled “DQ-020: existing_high_priority_issue trigger”

Context: 3153115 triggers on both new_high_priority_issue and existing_high_priority_issue. The latter lets already-existing issues re-trigger, churning tickets.

Decision: Drop existing_high_priority_issue. The replacement rules fire only on newly-escalated issues.

Trade-off accepted: an issue that was already high-priority when the rules are created will never produce a ticket. Given the rules are being recreated (DQ-019), this means the existing high-priority backlog produces no tickets — which is the intended outcome, not a gap.

Applied to: design.md § 8 HighTier, § 9.2


Context: Surfaced during implementation, not during design. Two settled decisions turned out to be jointly unsatisfiable:

  • DQ-011 fixes 3190653’s environment defect in this migration, and the migration plan expressed that as adopting the rule and updating its environment in place.
  • DQ-019 requires one rule per Sentry project, and 3190653 is cross-project.

A rename-and-update preserves one workflow. The per-project split needs two. One workflow cannot become two, so the in-place path cannot express the outcome DQ-019 requires — the conflict is structural, not a matter of effort.

OptionDescriptionTrade-offs
ADelete 3190653, let reconcile create both per-project first-seen tiersSymmetric with 3153115’s treatment under DQ-019; both rules come from one Construct with no special case. Loses the ID and its history.
BAdopt 3190653 as the arda-frontend first-seen tier, create the platform-be one freshPreserves the ID for the project whose attachment actually worked. Leaves the pair asymmetric, and the surviving rule carries hand-built structure the Construct did not emit — the exact shape DQ-019 rejected.
CDefer the split; adopt in place now and split laterTwo migrations instead of one, with the environment defect fixed but the routing defect still live in between.

Decision: Option A, confirmed by Miguel on 2026-08-06. This supersedes the adopt-by-rename disposition implied by DQ-011; DQ-011’s substance — that the defect is repaired in this migration rather than deferred — is unchanged.

Consequence: 3190653 joins the delete list and is subject to the same ordering rule as 3153115 — creates precede deletes, so the window in which first-seen notification is uncovered is as short as possible. Its ID is lost permanently, per the known limitation in DQ-018.

Applied to: design.md § 9.4; tools/operations-management-migrate.ts RETIRED_WORKFLOWS


DQ-027: The absent-environment assumption is now measured

Section titled “DQ-027: The absent-environment assumption is now measured”

Context: DQ-011 rests on an assumption that was carried, acknowledged and untested through the whole design: that a workflow whose environment names an environment a project does not have matches nothing, rather than matching everything. If it were false, retiring rule 3190653 would be a deliberate behavior change rather than a repair, and the design and PR would be describing it wrongly.

Method: three workflows on platform-be’s issue_stream detector, all firing on first_seen_event, differing only in the variable under test — A scoped to Alpha002-dev (positive control), B to production (which platform-be does not register), C to Alpha002-dev with the Linear action. One synthetic error was then injected under Alpha002-dev and lastTriggered polled on each. Without the control, B not firing would have proved nothing.

Result, 2026-08-06:

WorkflowEnvironmentFired
A — SlackAlpha002-devyes, 00:19:21.181898Z
B — Slackproductionno, lastTriggered stayed null
C — LinearAlpha002-devyes, 00:19:21.181943Z

Decision: the assumption is confirmed and is no longer an assumption. Rule 3190653’s platform-be attachment was inert, and retiring it repairs a defect rather than changing behavior.

Delivery was confirmed at the same time, which is a separate thing from a rule firing: A produced a message in #sentry-fe-prod naming workflow_id=3818157, and C produced PDEV-1434 in KTLO / BE at High priority — the routing DQ-010 specifies.

Incidental finding: the probe event also tripped the two live cross-project rules, 3153115 and 3426957, at the same instant — re-confirming they are duplicates. 3153115 produced PDEV-1433 with no project and Urgent priority, which is precisely the untargeted routing DQ-010 replaces. The two tickets side by side are the clearest statement of what this migration buys.

Applied to: design.md § 13


Context: Three definitions were in circulation, and the documents did not agree with the code.

DQ-003 decided pinned IDs ∩ name prefix, and explicitly rejected prefix-only as its Option C. The code shipped prefix only: identify() claims any workflow whose name starts with arda/, and the planner deletes every claimed workflow that no desired entry holds an ID for. The guidance-compliance audit found four documents asserting the intersection while the code did something else.

Then implementation forced the question. The first real apply, on 2026-08-06, created six workflows and printed their assigned IDs for pinning. verify, run immediately afterwards, proposed deleting all six — they carried the prefix, so they were managed; configuration held no IDs for them, so nothing claimed them; and managed-minus-claimed is the delete rule. The tool’s own desired state said destroy what you just built.

That is not a corner case. Merge runs apply --confirm unattended (DQ-023), so between an apply and the follow-up commit recording IDs, the next merge would have deleted the live alerting. Only the sequencing convention in DQ-025 stood in the way, and a convention is not a mechanism.

OptionDescriptionTrade-offs
ARestore DQ-003 literally: identify returns null unless the ID is pinnedMatches the recorded decision. But an unpinned prefixed workflow becomes permanently invisible — never updated, never removed, and re-created on every run, accumulating duplicates. Trades a destructive failure for a silent one.
BKeep prefix-onlyWhat shipped. Simple, and orphan cleanup works. Retains the destructive window, which is unacceptable given unattended apply.
CPrefix ∩ declared, where declared means pinned ID or natural keyCloses the window without weakening the prefix boundary. Pinning becomes an optimisation rather than a correctness requirement. Costs a second identity path, and requires names to be unique within the managed set.

Decision: Option C, confirmed by Miguel on 2026-08-06.

The rule, stated once, for every other document to copy:

A workflow is managed when its name carries the arda/ prefix. A managed workflow is claimed when a declared tier matches it, by the key embedded in its name (DQ-029). plan proposes deleting a managed workflow only when nothing claims it. A workflow without the prefix is untracked: reported, never touched.

Consequences

  • Pinned IDs remain the primary identity and are still recorded after an apply. Name matching is a fallback, reported in plan output as adopted, so that beginning to manage something is never silent.
  • Names must be unique within the managed set. Ambiguity throws rather than guessing — this org contains two live rules sharing one name, which is the observation that motivated ID pinning in DQ-003 in the first place.
  • arda/ becomes a reserved namespace. A hand-made rule named arda/… and not declared will be proposed for deletion. Recorded in the operator README.
  • Deleting a tier from configuration still removes the live workflow — orphan cleanup, which Option A would have lost.

Applied to: design.md § 8, § 9.1, § 9.3; tools/lib/desired-state/plan.ts, tools/lib/desired-state/resource.ts, tools/lib/sentry/issue-alert.ts, scripts/sentry/README.md


Context: DQ-028 left two identity paths — a pinned id, falling back to a natural key. That settled the safety question but not the design one: why keep a mechanism whose only remaining job is a case the fallback already handles?

The pinned id also carried a real cost. It cannot be known before the resource exists, so it required a manual transcription step after every create, and the window before that step was — until the natural key existed — actively destructive. It is a hand-maintained state file, and the tooling already writes state files.

The question is where a client-assigned identity can live at all. Sentry’s workflow schema has no free-form metadata field, and POST does not accept a caller-supplied id, so the name is the only field this tooling controls that survives a round trip.

OptionDescriptionTrade-offs
AKeep pinned ids as primary, key as fallback (DQ-028 as written)No change. Retains a manual step and two paths that must agree.
BDerive ids from the newest snapshot instead of configurationRemoves the manual step; snapshots already hold the mapping. But identity stops being visible in the PR diff, which is what DQ-003 wanted it for.
CEmbed a stable key in the workflow name; remove pinned idsOne identity, visible in configuration and in Sentry, needing no post-hoc step. Costs a suffix on a human-facing string, and names must be unique.

Decision: Option C, confirmed by Miguel on 2026-08-07.

The name becomes arda/High — platform-be [high-be]: prefix, display name, key. Readable part first, because this string is what an on-call engineer reads at the bottom of a Slack alert.

Why the key is separate from tierName. Renaming a tier’s display name is a legitimate edit and must produce an update; matching on the whole name would make it a delete-and-recreate. Changing a key means “this is a different tier”, and the old workflow is correctly deleted. Conflating them would remove the ability to say which was meant.

Consequences

  • pinnedId is gone from the construct, Stack, App and instance file, and with it assertPinnedIdsAreManaged and UnmanagedPinnedIdError, whose only job was guarding a pinned id against pointing outside the managed set.
  • Keys must be unique across every tier. Asserted by the App at build time, rather than surfacing at plan time as an ambiguity error against the live org.
  • The key pattern is constrained to lowercase kebab-case so compose and parse round-trip unambiguously. A malformed suffix parses as no key rather than as a guess.
  • This is a narrowing, not a loosening. A workflow is claimed when its key is declared, and by nothing else.

Transition, executed 2026-08-07: the rename applied as six in-place updates while the pinned ids were still present, because only a pinned id could match a live workflow whose old name carried no key. The ids were removed immediately afterwards, and plan then reported no changes with all six matched by key alone — which is the evidence that identity survived the change.

Applied to: design.md § 8, § 9.1; platform/sentry-service.ts (composeWorkflowName, parseWorkflowKey), platform/constructs/sentry/alert-tier.ts, tools/lib/sentry/issue-alert.ts


Context: DQ-021 chose #1-dev-team on the reasoning that escalation should land where people already look, and that a dedicated channel nobody watches is worse than a busy one.

Decision: #sre-production (C0BNTF2AJ6N), on Miguel’s direction, 2026-08-07. A channel dedicated to production operations is the better home: a burst clearing this threshold should not compete with conversation for attention. DQ-021’s concern is answered by the channel having an audience by construction rather than by being general-purpose.

Precondition, and it is not optional. Sentry’s Slack integration can only post to channels its app has joined. A workflow pointing at a channel the app is absent from is accepted at write time and then silently drops every notification — the same failure shape as rule 3190653. @sentry was confirmed present in #sre-production before the change was applied.

Delivery itself remains unproven: no event has yet fired into the channel.

Applied to: design.md § 8; instances/OperationsManagement/arda-systems.ts


Copyright: (c) Arda Systems 2025-2026, All rights reserved