Design: Sentry Configuration as Code
Audience: platform engineers and on-call operators Reading time: ~20 min full; ~4 min for §1–§5
The problem. Arda’s Sentry alerting is configured by hand in the UI: seven rules across two projects, with overlapping scope, one outright duplicate, and two defects nobody could see. One rule was scoped to an environment its backend project does not have, so half of it had never matched anything. Nothing recorded what any of it was for.
The approach. Alerting becomes declared configuration, reconciled by
snapshot → plan → apply → verify. This establishes OperationsManagement, a
fourth platform element for resources that observe and administer the platform
rather than run it; Sentry is its first inhabitant, and the engine that
reconciles it names no vendor.
Three decisions shape everything else:
| The Application produces a desired-state tree, applied by our own engine | Sentry workflows have no CloudFormation representation, so there is nothing to synthesize (DQ-002) |
| Merge applies with no approval step | Which makes the engine’s refusals — never delete the unmanaged, never apply without a valid rollback point — the actual safety mechanism (DQ-023) |
| Identity lives in the workflow name | Sentry assigns ids and offers no metadata field, so the name is the only field we control that survives a round trip (DQ-029) |
One premise was wrong. The brief described the legacy issue-alert-rule API
and cited rule ids verified in the UI. The org had been migrated to Sentry’s
workflow engine without a version bump; those ids do not exist in the legacy
namespace, which still answers 200 with a lossy translation
(DQ-004). Every type here
was derived from payloads captured live, not from documentation — and the
freshness block says when.
Glossary
Section titled “Glossary”| Term | Meaning |
|---|---|
| Workflow | Sentry’s unit of alerting configuration in the new engine: a trigger condition group plus one or more action filter groups. Replaces the legacy “issue alert rule”. |
| Detector | A project-scoped monitor (error, issue_stream, uptime_domain_failure). Workflows attach to projects through detectors, via detectorIds. |
| Data condition group | {logicType, conditions[], actions[]} — the shape of both triggers and each entry in actionFilters. |
| Logical environment | An Arda-level environment name (prod, stage, dev, demo) that resolves to a different physical Sentry environment name per project. |
| Managed set | The workflows this tooling owns: those carrying the required arda/ name prefix. Everything else is untracked and never touched. |
| Claimed | A managed workflow whose embedded tier key a declared tier carries. Only an unclaimed managed workflow is ever proposed for deletion (DQ-028, DQ-029). |
| Tier key | The stable identifier a tier declares and never changes, embedded in the workflow name — arda/High — platform-be [high-be]. The only thing reconcile matches on. |
| Tier | A severity band with its own trigger and its own action. High creates a Linear ticket at priority 2; Urgent posts a Slack notification. They differ in action, not just in priority — see DQ-017. |
1. Position in Arda’s Architecture
Section titled “1. Position in Arda’s Architecture”| Viewpoint | This design’s position |
|---|---|
| Product | Platform Operator — the on-call engineer who receives alerts and maintains alerting config. Persona authored by this project (DQ-014); no operator persona previously existed. |
| Functional | oam → fault. Consumes no Arda services; produces no Arda endpoints. It configures a third-party control plane that observes every component. |
| Artifacts | infrastructure repo only. New modules under platform/, tools/lib/, platform/constructs/, stacks/, apps/, instances/, plus two drivers and a scheduled workflow. No new deployable image or chart. |
| Runtime | Nothing runs in an Arda partition. Execution is operator-initiated or CI-scheduled, against Sentry’s API. The effect is on alerting for arda-frontend and platform-be. |
| OAM | This design is OAM-plane. Its own operational surface is covered in §11. |
| Technology | Infrastructure layer; TypeScript on the repo’s existing ts-node + Jest stack. No new runtime dependency — SentryClient uses native fetch. |
2. Module-Interaction Mechanics
Section titled “2. Module-Interaction Mechanics”| Mechanism | Added by this design | Consumed by this design |
|---|---|---|
| API Endpoints / Services | None. This design exposes no Arda endpoint. | Sentry Workflows API: GET/POST /organizations/{org}/workflows/, GET/PUT/DELETE /organizations/{org}/workflows/{id}/, GET /organizations/{org}/detectors/, GET /projects/{org}/{project}/environments/. |
| References | None. Sentry entities are referenced by opaque ID within OperationsManagement config and never cross into an Arda Universe. | 1Password item reference for the alerts-scoped token. |
| Data Types | WorkflowSpec, DataConditionGroupSpec, ConditionSpec, ActionSpec, LinearActionSettings, LogicalEnvironment — all derived from live payloads, all in-process (no wire contract of ours). | Sentry’s workflow JSON, treated as an external schema pinned by a dated observations note. |
| Bindings | HTTPS + bearer token to https://us.sentry.io. | 1Password (OpResolver), reused unchanged. |
Capability framing: DesiredStateEngine is a Capability, not a Sentry
feature. It is the reconcile mechanism the OperationsManagement plane needs for
any provider whose resources are declared rather than deployed. Grafana or a
future uptime-monitor provider implements the same seam.
3. Overview
Section titled “3. Overview”Arda’s Sentry alerting is configured by hand in the Sentry UI. Seven workflows exist across two projects, and the hand-maintenance shows: one is an outright duplicate, the best-configured Linear routing sits on a rule that is disabled, and one rule is attached to both projects with an environment filter that can only ever match one of them — so backend Slack alerting from it has never fired. There is no record of intended state, no review path for changes, and no way to detect drift.
This design introduces OperationsManagement as a platform element and makes
Sentry its first provider. Desired state is declared through the repo’s existing
IaC vocabulary — an AlertTier Construct parametric on trigger, logical
environment, and action, composed into a SentryFaultAlerting Stack per Sentry
project, assembled by the OperationsManagementApp Application, and pinned
by an ArdaSystems Instance. Because Sentry workflows have no
CloudFormation representation, the Application emits a desired-state tree rather
than synthesizing a template, and a vendor-neutral DesiredStateEngine
reconciles it against live state
(DQ-002). Two
drivers consume it: OperationsManagementConfig for operator-initiated
snapshot/plan/apply/verify, and OperationsManagementDrift for read-only
scheduled conformance — the fourth member of the repo’s existing drift family. A
third consumer, the OperationsManagementDeploy workflow, plans on every PR and
reconciles automatically on merge to main, so main and live cannot diverge
(DQ-023). The
PR plan is deliberate: with no gate at merge, review is the only human
checkpoint, so the changeset has to be visible while it is still reviewable.
In scope: the engine, the Sentry provider, the Fault-area alerting Stack, and a migration that consolidates high-priority routing into Linear, adds an Urgent tier, and retires the frontend Slack rules. Deliberately deferred: the Performance-area resources (metric alerts, dashboards, uptime monitors). The seam accommodates them; this project does not build them (DQ-007).
4. Decision Summary
Section titled “4. Decision Summary”| # | Decision | Chosen Option |
|---|---|---|
| DQ-001 | Where the code lives | OperationsManagement element, layered per repo vocabulary |
| DQ-002 | Execution model | Declarative App + our own reconcile engine |
| DQ-003 | Identity and managed set | Pinned IDs + required name prefix — superseded in part by DQ-028 |
| DQ-004 | API surface | Org-scoped /workflows/ only |
| DQ-005 | Detectors | Read-only, resolved symbolically |
| DQ-006 | Fixtures and snapshots | Committed |
| DQ-007 | Performance lookahead | Seam designed, Fault implemented |
| DQ-008 | Urgent window | 3 in 15m, a tunable starting value (30m unavailable) |
| DQ-009 | Urgent scoping | Two rules, one per project |
| DQ-010 | Linear routing | KTLO / BE and KTLO / FE |
| DQ-011 | Rule 3190653 defect | Fix in this migration |
| DQ-012 | Environment divergence | Logical-environment abstraction |
| DQ-013 | Uptime attachment | Preserve, flag in plan |
| DQ-014 | Product Viewpoint persona | Author the Platform Operator persona |
| DQ-015 | assigneeId | Strip explicitly |
| DQ-016 | stateId | Reuse Triage |
| DQ-017 | High → Urgent escalation | Urgent notifies via Slack; only High tickets |
| DQ-018 | Removal mode | Delete outright |
| DQ-019 | Split 3153115 | Delete, create two fresh |
| DQ-020 | Trigger scope | Drop existing_high_priority_issue |
| DQ-021 | Urgent Slack channel | #1-dev-team — superseded by DQ-030 |
| DQ-022 | Configuration shape | Nested Configuration/Props/Built tree as one SentryAlerts value |
| DQ-023 | Deploy on merge | Fully automatic; plan posted on the PR |
| DQ-024 | CI snapshot | Workflow artifact, 90 days, uploaded before apply |
| DQ-025 | Enablement timing | After the migration lands and plan verifies empty |
| DQ-026 | Disposition of 3190653 | Delete and recreate, superseding adopt-by-rename |
| DQ-027 | Absent-environment assumption | Measured and confirmed; 3190653 was inert |
| DQ-028 | What the managed set is | Prefix ∩ declared; supersedes part of DQ-003 |
| DQ-029 | Where identity lives | A key embedded in the workflow name; no pinned ids |
| DQ-030 | Urgent destination | #sre-production (C0BNTF2AJ6N) |
Full rationale and rejected alternatives in decision-log.md. All decisions are settled; §13 records one stated assumption and one scheduled review.
5. Constraints
Section titled “5. Constraints”- The Sentry schema is external and under-documented. Every type is derived from live payloads or round-trip discovery. Guessing a field name is a stop-and-ask condition. The observations note carries the date each shape was observed.
event_frequency_countintervals are a closed enum:1m, 5m, 15m, 1h, 1d, 1w, 30d. A 30-minute window cannot be expressed.- A workflow carries exactly one
environmentstring and one action config per action. Per-project environment names and per-discipline Linear routing therefore force one workflow per (tier × project). - The workflow schema has no free-form metadata field, and
POSTdoes not accept a caller-supplied id. The name is therefore the only field this tooling controls that survives a round trip, and identity is a key embedded in it (DQ-003, DQ-028, DQ-029). - Deletes are not reversible in place. Recreated workflows receive new IDs.
SENTRY_AUTH_TOKENis already taken in this repo —amm.shresolves a partition-scoped, source-map-upload token under that name. The alerts-scoped token must use a distinct variable to avoid collision.applyis dry-run by default and refuses unless a snapshot still describes live state. For operator-initiated runs it additionally requires--confirm. In CI there is no confirmation step — merge tomainapplies automatically (DQ-023). This is a deliberate reduction from the original constraint (“never applies unattended”), taken with the trade-off understood: the PR review becomes the only human checkpoint, which is why the plan is posted on the PR rather than only after merge.- Configuration follows the repo’s
Configuration/Props/Builtdiscipline. Each layer owns its ownConfigurationinterface; the Instance file holds values only. Thresholds, channels, routing targets, and frequencies are nodes in oneSentryAlertstree, so retuning alerting is a config PR rather than a code change, and adding a tier or a provider is adding a node rather than a parallel family of constants (DQ-022).
6. Quality Attributes
Section titled “6. Quality Attributes”| Attribute | Target | Satisfied by |
|---|---|---|
| Idempotence | Second apply produces an empty plan | IssueAlertResource.normalize stripping all server-managed fields recursively |
| Blast radius | Never mutates an unmanaged workflow | Managed set = the arda/ prefix; deletion needs managed and unclaimed; untracked reported, never touched |
| Auditability | Every mutation reconstructible after the fact | Committed snapshots + JSONL audit log under --audit |
| Recoverability | Any applied change reversible from a snapshot | rollback restores from a named snapshot; new-ID limitation documented |
| Extensibility | New resource type = one file + registry entry | Resource<TSpec, TRemote> seam; engine is vendor-neutral |
7. Stakeholders & Concerns
Section titled “7. Stakeholders & Concerns”| Stakeholder | Primary concern |
|---|---|
| Platform Operator | Alerts fire when they should and route to the right place; changes are reviewable |
| Backend / frontend engineers | Tickets land in their discipline’s KTLO project, already triaged |
| Whoever runs the migration | Nothing is deleted without a recorded snapshot and an explicit confirmation |
8. Structural Design
Section titled “8. Structural Design”The component diagram below shows the four layers. platform holds vendor
constants with no I/O; tools/lib holds the transport and the vendor-neutral
engine; the cdk package declares desired state through the repo’s IaC
vocabulary; and the drivers tie them together. The dependency direction is
strictly downward — instances → apps → stacks → constructs, with drivers
consuming both the Application and the engine.
Key Elements
Section titled “Key Elements”Signatures live in the code. What follows is why each element exists and what it is not allowed to do.
The platform/ layer — vendor facts
Section titled “The platform/ layer — vendor facts”| Element | Exists to |
|---|---|
SentryService | Hold Sentry constants and the API-surface freshness block. Every shape was derived from live payloads, not documentation, because Sentry migrated this org between alerting engines without a version bump (DQ-004). |
LogicalEnvironment | Map an Arda environment to each project’s physical Sentry name, absorbing a divergence that is an accident of project creation order (DQ-011 / DQ-012). |
| Workflow naming | Carry identity in the workflow name, since Sentry assigns ids and has no metadata field (DQ-029). |
The environment mapping returns null — never a blank filter — for a pair that
does not exist, and callers must treat that as this tier does not apply here.
Collapsing the two is precisely the defect that left rule 3190653 inert on the
backend for months, so the type makes the collapse impossible rather than
discouraged.
The tools/lib/ layer — mechanism
Section titled “The tools/lib/ layer — mechanism”| Element | Exists to | Constraint it must respect |
|---|---|---|
SentryClient | Be the single authenticated transport | Honours Retry-After, backs off on 5xx, never retries other 4xx. Mirrors postmark-client.ts so the two age together |
DesiredStateEngine | Reconcile desired against live — diff, plan, apply, snapshot | Names no vendor. A second provider implements Resource and nothing else changes (DQ-002, DQ-007) |
IssueAlertResource | Translate between declared spec and Sentry’s workflow JSON | normalize strips server-managed fields recursively; the nested ids matter as much as the envelope, or the diff never converges and every apply rewrites every rule |
DetectorResolver | Resolve symbolic detector names to the ids a workflow needs | Read-only. A detector is somebody’s monitoring, not ours to rewrite (DQ-005). An unresolvable name is a plan-time error, never a silent empty attachment |
Rendering ids back to symbolic names in diff output was designed and dropped:
it requires teaching the vendor-neutral formatter about Sentry detectors, which
is the coupling the engine exists to prevent. Plan output shows raw ids. A
value-renderer hook on formatPlan would fix it without the import.
AlertTier — the Construct
Section titled “AlertTier — the Construct”Produces one workflow spec for one (tier × project) pair. Three properties carry the design:
- Parametric on its action, not Linear-specific. The
TierActionvariant selects the emitted shape. This is why the Construct is not named for Linear: only the High tier opens tickets, because a second rule firing for an already-ticketed issue creates nothing and upgrades nothing, so escalation routed through Linear is silently lost (DQ-017). - Discriminated unions, not optional fields. A new trigger or action kind is a new variant the compiler forces every consumer to handle, rather than a field that is silently absent.
- Rejects at plan time what would otherwise become a workflow that looks configured and matches nothing: an interval outside Sentry’s closed enum, an environment unmapped for the target project, an empty detector list, a Linear action with no project.
It is deliberately not a constructs.Construct — it emits nothing into a
template — so it exports validateProps as a free function, the documented
convention for non-Construct value objects. assigneeId is emitted explicitly
empty: the template rule this derives from carries a real assignee that must not
propagate (DQ-015).
SentryFaultAlerting, OperationsManagementApp, ArdaSystems
Section titled “SentryFaultAlerting, OperationsManagementApp, ArdaSystems”The Stack is the deployable unit for one project’s Fault-area alerting, and is
where the Fault/Performance split lands — a Performance stack would be a sibling,
deployed independently. It injects project and detectors into each tier, so a
tier definition never names its own project and can be reused across them.
The App composes Stacks into the plane’s complete desired state. It constructs no
cdk.App(): there is nothing to synthesize
(DQ-002). Its
Configuration aggregates and projects rather than unioning its stacks’
shapes — linearWorkspace and managedNamePrefix are hoisted above the
per-project level, so two tiers cannot disagree about which Linear team owns
Arda’s tickets.
The Instance holds const values only. Defining a Configuration in an instance
file is a named anti-pattern in the IaC rules, and a missing value should be a
compile error rather than a runtime guard.
The shape is the extensibility argument (DQ-022). The whole configuration is one JSON-like value, and every axis of growth is an edit to one node of it:
| To add | Edit |
|---|---|
| A Performance tier | Another entry in a tiers array |
| A third Sentry project | Another entry in projects |
| A Grafana provider | A sibling key beside sentryAlerts |
None of them touch the Construct, the engine, or the drivers. This is also why every operationally-tunable value — the Urgent threshold, its Slack destination, the notification frequency — lives here and nowhere else: retuning after real traffic should be a one-line pull request, not a code change.
The drivers
Section titled “The drivers”OperationsManagementConfig is the operator entry point: snapshot, plan, apply,
verify. OperationsManagementDrift is the same plan with no intent to mutate,
which is what stops drift detection and apply from ever disagreeing about what
the world should look like. OperationsManagementMigrate owns the one-time
retirement and nothing else.
OperationsManagementDeploy — the GitHub Actions workflow
Section titled “OperationsManagementDeploy — the GitHub Actions workflow”Reconciles on merge to main, with no approval step
(DQ-023).
That is a deliberate reduction from the original “never applies unattended”
constraint, and the consequences are load-bearing:
- The PR check is the human checkpoint, because merge no longer has one. It posts the plan as a comment rather than blocking, since a pending changeset is the normal state of a PR that edits alerting.
- The snapshot uploads before apply, 90-day retention, so a partial apply still has a rollback source (DQ-024).
- Path filters are directory globs, so adding or removing a file never requires editing the workflow. Deliberately broad: a spurious run yields an empty plan and applies nothing.
- It ships disabled, until the migration has landed and
planverifies empty (DQ-025). The first automated apply must never be the destructive one.
9. Behavioral Design
Section titled “9. Behavioral Design”Behaviors are grouped below. Each cross-links to the Key Element that owns it and maps to entries in Behavior Verification.
9.1 Reconciliation mechanism
Section titled “9.1 Reconciliation mechanism”Owner: DesiredStateEngine, IssueAlertResource.
-
Normalization makes specs and remotes comparable. Server-managed fields are stripped recursively before comparison, so an unchanged workflow yields an empty diff. Verified by round-trip rather than assumed.
-
Membership is the prefix; claiming is the key. A live workflow is managed when its name carries the
arda/prefix. A managed workflow is claimed when a declared tier carries the key embedded in that name. Deletion requires managed and unclaimed (DQ-028).Identity is a key rather than a server-assigned id because no id exists before the resource does. An earlier revision pinned ids in configuration, recorded after each create, and the window before that recording was destructive rather than merely untidy: the workflow carried the prefix, so it was managed, and no declaration held its id, so nothing claimed it — which is the delete rule.
verify, run immediately after the first live apply, proposed deleting all six workflows that apply had just created. The key removes the window by existing before the workflow does (DQ-029).Two consequences worth stating plainly.
arda/is a reserved namespace: a hand-made rule using it, and declaring no matching key, will be proposed for deletion. And keys must be unique within the managed set — a collision is refused rather than guessed, and the App asserts uniqueness at build time so it never reaches the live org. -
Untracked is not drift. Workflows outside the managed set are listed in plan output as untracked and never proposed for deletion.
-
Ordering is not semantic where Sentry does not make it so.
detectorIdsanddata.settingsare order-insensitive and sorted before comparison. Condition order within a group is preserved, becauselogicType: any-shortshort-circuits and order is therefore observable.
9.2 Desired-state production
Section titled “9.2 Desired-state production”Owner: AlertTier, LogicalEnvironment, SentryFaultAlerting.
- A tier resolves its own environment. The Construct takes a logical environment and resolves the physical name for its project. An unmapped pair means the tier is not emitted for that project at all — never emitted with a name that cannot match.
- Linear routing is per discipline.
platform-be→KTLO / BE,arda-frontend→KTLO / FE, both at Triage with the Bug label and no assignee. - The High tier fires on escalation only.
new_high_priority_issuealone;existing_high_priority_issueis dropped (DQ-020). - The Urgent tier fires on occurrence rate and notifies rather than tickets.
event_frequency_countat 3 occurrences in 15 minutes — deliberately narrower than the specified 3-in-30m, since 30m is not an available interval and starting strict produces too few Urgent notifications rather than too many (DQ-008). Its action is Slack, not Linear — a second rule firing for an already-ticketed Sentry issue is a silent no-op that neither creates a ticket nor upgrades priority, proven by probe on 2026-08-06 (DQ-017). Routing escalation to a notification channel makes it visible without depending on behavior the integration does not provide. - Uptime attachment is preserved and surfaced. The backend High tier keeps
the two
uptime_domain_failuredetectors, and plan output states this explicitly so the carry-over is a visible choice (DQ-013).
9.3 Safety gating
Section titled “9.3 Safety gating”Owner: OperationsManagementConfig, DesiredStateEngine.
-
Dry-run is the default.
applywithout--confirmprints the plan and exits without mutating. -
In CI the checkpoint moves earlier, to the PR. The
pull_requestrun posts the plan while the change is still reviewable; thepushrun applies without asking. A reviewer who reads the posted plan is the only thing standing between a merged config error and production alerting — the design does not pretend otherwise. -
No apply without a snapshot that still describes live state. The gate compares the snapshot against live and refuses on any difference, naming each one. Age is not the test: a snapshot taken a second ago is worthless if something changed in that second, and an hour-old one is a perfectly good rollback source if nothing did. The comparison runs on the normalized form, so a rule merely firing between snapshot and apply — which moves
lastTriggered— is not a refusal. In CI the snapshot is taken and uploaded beforeapply, so a partial apply always has a rollback source.An earlier revision of this design specified a session-scoped refusal — “a snapshot from the current run”. That could not work:
snapshotandapplyare separate processes, so a session identifier established at module load could never agree between them, and the gate would have refused every real invocation. Discovered by running the sequence end-to-end against the live org rather than by unit test, which calls the gate inside one process and cannot see it. -
Destructive changes are marked and ordered last. Deletes and detaches carry a
DESTRUCTIVEmarker; a detach that would strip the last detector from a project requires its own confirmation flag. -
planexits non-zero when changes are pending, so CI can gate on it.
9.4 Migration execution
Section titled “9.4 Migration execution”Owner: OperationsManagementConfig.
The migration runs as ordered, individually confirmable phases. Creates precede deletes so the window in which high-priority routing is uncovered — an unavoidable consequence of recreating 3153115 as two rules (DQ-019) — is as short as possible.
The two halves run through different tools, and that is deliberate. The
creates are ordinary reconcile work: OperationsManagementConfig apply performs
them, is snapshot-gated, and runs again on every subsequent deploy. The deletes
cannot go through reconcile at all — the retired rules carry legacy names, so
identify() returns null and the engine classifies them untracked, the
one category it is built never to touch. Making reconcile able to delete them
would mean weakening exactly the guarantee that protects every unmanaged rule in
the org.
So the destructive half lives in OperationsManagementMigrate, which holds the
one-time managed-set exemption, and it refuses to act unless three conditions
hold:
| Precondition | Failure it prevents |
|---|---|
| Every retired id still names what it named at design time | Deleting a rule somebody renamed or recreated in the UI. Ids are opaque; a mismatch is a hard stop, not a warning. |
| All six replacements are live | The ordering rule, enforced rather than trusted. Nothing otherwise stops the delete phase running first and leaving alerting uncovered. |
| A snapshot still describes live state | Deleting with no usable rollback path. |
An id that is already absent reads as already retired rather than an error, so an interrupted migration resumes rather than restarting.
9.5 Credential and decision boundaries
Section titled “9.5 Credential and decision boundaries”The call sequence from wrapper to API is unremarkable and reads plainly in the drivers. Two boundaries it enforces are not obvious from any single file:
- Credentials are resolved in exactly one place and never persisted. The shell wrapper resolves the token for an operator; CI passes a 1Password service-account token and the driver resolves through the SDK. Neither writes it to disk, and snapshots record a token fingerprint rather than the token, so a snapshot can be committed without leaking the credential that produced it.
- Every decision happens in
tools/. The wrapper parses arguments and maps exit codes; it makes no API calls and evaluates no state. This is the repo’stools-versus-scriptssplit, and it is what allows the drift check, the operator, and CI to share one implementation of what the world should look like rather than three that can disagree.
9.6 Robustness
Section titled “9.6 Robustness”- Halt on first error, print the rollback command. Partial application is reported precisely — which changes landed, which did not.
- Rollback recreates from a snapshot. Recreated workflows get new IDs; this is an API property, stated in code and in operator output rather than worked around.
- Round-trip instability is a stop condition. If a workflow does not round-trip cleanly after normalization, the tool reports it rather than papering over it — an unstable field means the schema understanding is wrong.
10. Behavior Verification
Section titled “10. Behavior Verification”Test identifiers follow BV-<group>-<seq>, mirroring §9.
10.1 Reconciliation mechanism
Section titled “10.1 Reconciliation mechanism”| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-1-01 | Normalization strips every server-managed field, including nested ids | Live fixture workflow-3433641.json | Captured workflow fixtures |
| BV-1-02 | Normalized fixture compared against itself yields an empty diff | Same fixture twice | Captured workflow fixtures |
| BV-1-03 | Pinned ID with a non-prefixed name raises a hard error | Fixture renamed without prefix | Captured workflow fixtures |
| BV-1-04 | Unmanaged workflow is reported untracked, never in the delete set | Fixture set including a non-prefixed workflow | Captured workflow fixtures |
| BV-1-05 | A declared tier claims the live workflow carrying its key, rather than deleting it | Prefixed live workflow with a matching key | Captured workflow fixtures |
| BV-1-07 | A tier key round-trips through compose and parse, and survives a change of display name | — | — |
| BV-1-08 | A malformed key suffix parses as no key rather than as a guess | — | — |
| BV-1-06 | Two managed workflows sharing a name raise AmbiguousAdoptionError rather than guessing | Two prefixed fixtures with one name | Captured workflow fixtures |
| BV-1-05 | detectorIds reordering produces no diff; condition reordering does | Two fixtures differing only in order | Captured workflow fixtures |
| BV-1-06 | Diff paths are addressable (actionFilters[0].actions[1].data.settings.priority) | Fixture pair with one changed setting | Captured workflow fixtures |
Integration
Section titled “Integration”| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-1-07 | Live GET → normalize → plan against unchanged desired state yields empty plan | Read-only token | Live read-only harness |
10.2 Desired-state production
Section titled “10.2 Desired-state production”| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-2-01 | Tier resolves physical environment per project | LogicalEnvironment table | — |
| BV-2-02 | Unmapped (project, logical) pair emits no workflow rather than an unmatched filter | demo × arda-frontend | — |
| BV-2-03 | Linear action carries empty assigneeId even when derived from the template | Template fixture with assignee set | Captured workflow fixtures |
| BV-2-04 | Per-discipline projectId routing | Both projects | — |
| BV-2-05 | High tier omits existing_high_priority_issue | — | — |
| BV-2-06 | Urgent tier interval is a member of the closed enum | — | Interval enum guard |
| BV-2-07 | Backend High tier retains uptime detectors | Detector fixture | Detector fixture |
| BV-2-08 | validateProps rejects an out-of-enum interval, an unmapped logical environment, an empty detector list, and a blank Linear projectId | Malformed Props per case | Interval enum guard |
| BV-2-09 | linearWorkspace projected from the App reaches every tier identically | Two-project SentryAlerts value | — |
10.3 Safety gating
Section titled “10.3 Safety gating”| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-3-01 | apply without --confirm mutates nothing | Recording client stub | Recording SentryClient stub |
| BV-3-02 | apply refuses when no snapshot exists, and when the newest no longer matches live | Empty snapshot dir; snapshot diverging from live | Recording SentryClient stub |
| BV-3-03 | plan exits non-zero with pending changes, zero when clean | Two desired states | Recording SentryClient stub |
| BV-3-04 | Last-detector detach requires its own flag | Fixture with one detector | Detector fixture |
| BV-3-05 | Destructive entries ordered after non-destructive | Mixed plan | — |
| BV-3-06 | Path filter matches a new file added under a globbed directory, and does not match platform/one-password.ts | Workflow YAML + candidate paths | Path-filter matcher |
| BV-3-07 | A pull_request-triggered run plans but never applies, and still leaves a snapshot | Stubbed client, PR event | Recording SentryClient stub |
| BV-3-08 | The snapshot artifact is written before any mutating call is issued | Client stub failing on the first write | Recording SentryClient stub |
10.4 Migration execution
Section titled “10.4 Migration execution”Integration
Section titled “Integration”| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-4-01 | Phases run in order; --phase and --from honored | Stubbed engine | Recording SentryClient stub |
| BV-4-02 | Failure mid-apply halts and reports the rollback command | Client stub failing on Nth call | Recording SentryClient stub |
| BV-4-03 | Second run of the full migration produces an empty plan (idempotence) | Post-migration fixture set | Captured workflow fixtures |
| Test ID | Behavior Tested | Required Setup | Test Fixtures |
|---|---|---|---|
| BV-4-04 | Throwaway workflow create → GET → delete round-trips and leaves inventory unchanged | Write-scoped token; disabled, detector-less workflow | Throwaway workflow probe |
Testing Elements
Section titled “Testing Elements”Captured workflow fixtures
Section titled “Captured workflow fixtures”The seven live workflow payloads captured 2026-08-05, committed under the repo (DQ-006). They are simultaneously the type-derivation evidence, the unit-test corpus, and the migration’s rollback baseline.
Detector fixture
Section titled “Detector fixture”The GET /detectors/ payload — eight detectors across the two projects,
including the two uptime_domain_failure monitors whose attachment BV-2-07
asserts.
Recording SentryClient stub
Section titled “Recording SentryClient stub”A SentryClient implementation that records calls and returns scripted
responses without network access. Every safety-gating test asserts on what was
not called, which is the property that matters for a destructive tool.
Interval enum guard
Section titled “Interval enum guard”A compile-time union plus a runtime assertion pinning
1m | 5m | 15m | 1h | 1d | 1w | 30d, dated to the observation. Its purpose is to
fail loudly if someone writes 30m again.
Path-filter matcher
Section titled “Path-filter matcher”Evaluates the deploy workflow’s on.push.paths globs against candidate file
paths using the same matching semantics GitHub applies. Its purpose is to catch
the failure the requirement names directly: a filter so specific that adding a
file silently stops triggering deploys. Asserts both directions — a newly-added
file under a globbed directory does match, and platform/one-password.ts
does not.
Live read-only harness
Section titled “Live read-only harness”Runs plan against the real org with a read-only token. Asserts an empty plan
when config matches live — the conformance check, also the drift driver’s core.
Throwaway workflow probe
Section titled “Throwaway workflow probe”Creates one workflow that is enabled: false with detectorIds: [], so it
cannot fire; captures its payload; deletes it; asserts a 404 and an unchanged
inventory. This is the mechanism by which any future unknown shape gets
discovered rather than guessed.
11. Operations Impact
Section titled “11. Operations Impact”Telemetry / observability
Section titled “Telemetry / observability”- JSONL audit log under
--audit: one record per request/response, with the token fingerprint (a hash) and never the token. DriftReportfromOperationsManagementDrift, in the same shape the three existing drift drivers emit, so it feeds the same failure-issue mechanism.- Snapshot manifests: org, projects, workflow IDs, token fingerprint, timestamp.
Operational controls
Section titled “Operational controls”| Control | Mechanism | Effect when set |
|---|---|---|
| Dry-run | Default; --confirm opts out | No mutation without explicit intent |
| PR plan | pull_request trigger | Posts the changeset while the change is still reviewable; the only human checkpoint |
| Deploy kill switch | Disable the workflow in the Actions UI | Stops automatic apply without reverting code — the emergency stop, since there is no approval gate |
| Deploy trigger | on.push path filter, directory globs | Which changes cause a deploy; broad by design |
| Workflow enablement | Workflow disabled until post-migration | The first automated apply is never the destructive one |
| Snapshot gate | Automatic | apply refuses unless a snapshot still matches live |
| Detach guard | --allow-last-detector-detach | Permits a detach that would leave a project unattached |
| Phase selection | --phase=<name>, --from=<name> | Run or resume a single migration phase |
| Audit log | --audit | Full request/response paper trail |
| Token | Alerts-scoped variable, distinct from SENTRY_AUTH_TOKEN | Absent ⇒ fail fast with a named error |
Cost / capacity
Section titled “Cost / capacity”Negligible. A full plan is a handful of API calls (one workflow list, one detector list, one environment list per project) against endpoints with no per-call cost. The scheduled drift check runs monthly, matching the cadence of the existing drift workflows.
The real cost is Linear ticket volume, and it should fall. Dropping
existing_high_priority_issue removes the churn source, and the Urgent tier
adds no tickets at all — it notifies Slack
(DQ-017). One
ticket per Sentry issue per project is the steady state. Volume should still be
measured after the first week, since per-discipline routing splits the stream in
a way nobody has observed yet.
Runbook hooks
Section titled “Runbook hooks”- Drift check fails. Read the
DriftReport. Divergence means someone edited a managed workflow in the Sentry UI. Runplanto see the changeset; either fold the change into config or re-apply. - Migration halts mid-apply. The driver prints the exact rollback command with the snapshot path. Recreated workflows get new IDs; re-pin them in the Instance file afterward.
- Alerts stop arriving. Check the workflow is
enabled, itsdetectorIdsare non-empty, and itsenvironmentexists on the target project — the last is the failure mode that made rule 3190653 silently inert. - Token expired or descoped.
SentryClientfails fast with a named error. Rotate inArda-SystemsOAM; no code change. - Deploy workflow ran but nothing changed. Expected. The path filter is deliberately broad, so an engine or tooling edit triggers a run whose plan is empty; nothing is applied and the job ends green.
- A bad change was merged and has already applied. There is no approval to
withhold, so recovery is the path: download the snapshot artifact from that
run and roll back, then revert the config commit so the next merge does not
re-apply it. Do both — rolling back without reverting means
mainstill describes the bad state and the next unrelated merge restores it. - Deploy failed part-way. Same halt-and-report contract as a manual apply: the run prints the rollback command and the snapshot artifact is attached to that run. Rollback needs the artifact within its 90-day retention; beyond that, reconstruct from the committed fixtures and the Instance file’s git history (DQ-024).
- Need to stop automatic deploys right now. Disable the workflow in the Actions UI. That is the emergency stop; it does not require a PR.
12. Implementation Artifacts
Section titled “12. Implementation Artifacts”Where the work lands, by layer. Contents are not enumerated — read the code.
| Layer | Path | Holds |
|---|---|---|
| Vendor facts | src/main/cdk/platform/sentry-*.ts | Constants, the API-surface freshness block, workflow wire types, the validating parser, and the naming functions that carry identity |
| Construct | src/main/cdk/platform/constructs/sentry/ | AlertTier — one workflow spec per (tier × project) |
| Stack / App / Instance | src/main/cdk/{stacks/operations-management,apps/OperationsManagement,instances/OperationsManagement}/ | The deployable unit, the composition, and the declared values |
| Engine | tools/lib/desired-state/ | Resource seam, diff, plan, apply, snapshot. Names no vendor |
| Provider | tools/lib/sentry/ | The Sentry Resource, transport, detector resolution, token resolution |
| Drivers | tools/operations-management-*.ts | config (snapshot/plan/apply/verify), drift (read-only), rollback, migrate (the one-time retirement) |
| Operator surface | scripts/sentry/ | Shell wrapper and README. Ergonomy only — no logic |
| CI | .github/workflows/operations-management-{deploy,drift}.yml | Deploy on merge (ships disabled) and monthly drift |
| Fixtures | tools/lib/sentry/__fixtures__/ | Seven workflow payloads captured live on 2026-08-05, plus detectors and projects |
Two placements are deliberate and would otherwise look arbitrary. The workflow
types live under platform/ rather than tools/lib/, because the Construct
layer needs them and src/ may not import tools/. The Sentry client lives
under tools/lib/sentry/ rather than flat, so the deploy trigger can name one
directory glob instead of tracking files
(DQ-023).
Designed and not delivered
Section titled “Designed and not delivered”Recorded because a reader comparing design to code will otherwise wonder.
| Planned | Why it is absent |
|---|---|
tools/lib/sentry/registry.ts — kind → resource | Built, then removed. Nothing consumed it, and a registry with one entry and no reader is speculation rather than extensibility. The seam that matters is Resource itself |
tools/lib/drift/sentry-probe.ts — read helpers | The Resource seam already gives drift and apply one shared read path. A second one is the divergence that rule exists to prevent |
An import command on the config driver | Unnecessary once identity moved into the name: adoption happens on any ordinary plan |
| A Jest suite parsing the deploy workflow YAML (BV-3-06) | Not built. The path filter is unverified by test |
| Symbolic rendering of detector ids in plan output | Requires teaching the vendor-neutral formatter about Sentry. Plan output shows raw ids |
Out of Scope
Section titled “Out of Scope”- Performance-area resources (metric alerts, dashboards, uptime monitors). Revisit when: Performance Management work starts. See DQ-007.
- Managing detectors. Revisit when: uptime monitors need declaring, which is the same trigger as above. See DQ-005.
- Unifying Sentry environment names across projects. The design absorbs the divergence rather than fixing it. Revisit when: someone changes SDK configuration in either codebase. See DQ-012.
- The
#sentry-fe-prodnaming problem. That channel carries backend alerts (rule 3462179) despite its name, and the retained Slack rules keep pointing at it. The Urgent tier uses#sre-productioninstead, so this design neither fixes nor worsens the existing mismatch. Revisit when: Slack routing is restructured, or the retained Slack rules are themselves brought under management. - Reconciling Sentry project topology with the Sentry Integration design
(
arda-operations/arda-accountsvs. the livearda-frontend/platform-be). Revisit when: a third component is onboarded to Sentry. - Wiring
planinto a per-PR CI gate. The command is CI-shaped; only the monthly drift schedule is built. Revisit when: config changes become frequent enough that monthly detection is too slow. - Linear-side automation (triage rules, duplicate merging). Revisit when: simultaneous rule firings produce enough duplicate tickets to be a nuisance — the probe showed the dedup race is real but narrow (DQ-017).
- Two-tier Linear ticketing. Ruled out because a later rule firing neither creates a ticket nor upgrades priority. Revisit when: Linear’s Sentry integration gains documented priority-update semantics, or the H2 hypothesis (dedup keyed on Sentry issue × Linear project) is confirmed — under H2, routing Urgent to its own Linear project would restore ticketed escalation.
13. Open Questions
Section titled “13. Open Questions”All decisions are settled, and what was the design’s one open assumption has since been measured. What remains is a scheduled review, which does not block implementation.
- Settled by measurement: a Sentry
environmentfilter naming an environment absent from a project matches nothing, not everything. This is what makes rule 3190653 inert and therefore a defect rather than a design choice. It was carried through the design as an untested assumption, and was tested on 2026-08-06 with a controlled probe — a workflow scoped to a valid environment fired, an otherwise identical one scoped toproductiononplatform-bedid not (DQ-027). Retiring 3190653 is a repair, and the PR describes it as one. - Review: the Urgent threshold is a starting value, not a conclusion.
{3, "15m"}is deliberately strict, and its destination#sre-production(DQ-030) has not yet received a real alert. Both are nodes in the Instance file’sSentryAlertstree specifically so the first weeks of real traffic can retune them in a one-line PR (DQ-022). Worth revisiting after roughly a week of live operation.
14. Guidance compliance audit
Section titled “14. Guidance compliance audit”A required gate between “implementation complete” and “PR opened for external
review”. It exists because this project twice discovered, mid-build, that
governing guidance had not been read — once for skills, once for the
repository’s knowledge-base/. Both times the code was already written and
already passing, and both times the guidance had something to say about it.
The audit is cheap and mechanical. Run it against the finished branch, record the result in the PR, and treat any gap as work rather than a footnote.
Checklist
Section titled “Checklist”| # | Check | Why it is here |
|---|---|---|
| 1 | Every skill the routing table names for this work was loaded and its doc page read — not just the stub | typescript-coding was a stub over a page that never existed; that was only discovered by trying to read it |
| 2 | knowledge-base/ was listed, every file assessed for relevance, and the relevant ones read in full | Two of nine were read; testing-and-ci.md documented the exact lint rules discovered by trial, and tooling-and-scripts.md held the entry-point convention the drivers need |
| 3 | Conventions encoded in tooling were consulted directly: eslint.config.mjs, tsconfig*.json, jest.config.js | The enforced standard is the config, not the prose; where they disagree the config wins and the prose is stale |
| 4 | Layer rules hold — import/no-restricted-paths, tools/ vs scripts/ split | Silent violations are possible where a zone is not configured |
| 5 | Both typecheck configurations pass, not just the test suite | isolatedModules means a green suite proves nothing about compilation |
| 6 | Guidance found stale or wrong during the work has been corrected or reported | Leaving it costs the next person the same discovery |
| 7 | Documentation contradicted by the code has been updated | A design that no longer describes the code is worse than no design |
| 8 | CHANGELOG entry present and describing intent, not file lists | Repo gate; also the only artifact most readers see |
Recording the result
Section titled “Recording the result”State plainly in the PR which checks passed and what each turned up. “Audit clean” is a claim; “read these four knowledge-base files, two were stale, both corrected” is evidence. A reviewer should not have to take the audit on trust any more than the tests.
References
Section titled “References”- Decision Log — settled decisions and rejected alternatives.
- Goal — the project goal this design serves.
- Sentry Observability — how Arda uses Sentry today.
- Sentry Integration — the project that introduced Sentry and deferred alert rules.
- Platform Operator persona — the Product Viewpoint actor.
- IaC Functional Design — layer responsibilities, the
script → instances → apps → stacks → constructsdirection, and the anti-patterns this design’s configuration shape avoids. - AWS CDK Infrastructure — the
Configuration/Props/Builtpattern and repo-local CDK conventions. infrastructure/knowledge-base/cdk-construct-patterns.md— the mechanical triad rules, including the non-Constructvalue-object exception thatAlertTierfollows.infrastructure/knowledge-base/platform-architecture.md— platform elements and the generalized IaC vocabulary.infrastructure/knowledge-base/tools-vs-scripts-split.md— logic intools/, ergonomy inscripts/.
Copyright: (c) Arda Systems 2025-2026, All rights reserved
Copyright: © Arda Systems 2025-2026, All rights reserved