Overview

Summary

Add Sentry-based error capture and OpenTelemetry performance tracing to Arda’s backend components (operations, accounts) through the shared common-module library. This replaces the homegrown performance monitoring and MDC propagation plugins with standards-based OTel instrumentation, adds health endpoints, and provisions secrets through the existing CloudFormation/external-secrets pipeline with no new infrastructure to deploy.

Motivation

Arda currently has no application-level observability beyond CloudWatch logs. Diagnosing errors requires grepping CloudWatch. There is no distributed tracing, no structured error capture, no release health tracking, and no PII-safe error forwarding. As the platform grows (gRPC inter-service calls, additional components), the lack of instrumentation becomes increasingly costly.

Architecture

Technology Choices

Concern	Technology	Rationale
Error capture	Sentry Kotlin SDK (sentry-kotlin-extensions)	Native Kotlin coroutine support, beforeSend hooks for scrubbing
Performance tracing	OpenTelemetry Java SDK + Ktor instrumentation	Vendor-neutral, first-class gRPC/HTTP/JDBC support, future-proof
Trace export to Sentry	OTel OTLP exporter to Sentry’s OTLP endpoint	Sentry natively ingests OTLP spans; no collector needed
Trace-log correlation	OTel MDC injection (trace_id, span_id in SLF4J MDC)	CloudWatch log lines become searchable by Sentry trace ID
Request ID correlation	X-Request-ID recorded as OTel span attribute arda.request_id	Preserves external API contract; bridges client-facing ID to internal trace

Sentry Project Topology

One Sentry project per component, environment tags for partition separation:

Sentry Project	Component	DSN Secret (AWS SM)
arda-operations	operations	{infrastructure}-SentryDsn-Operations
arda-accounts	accounts-component	{infrastructure}-SentryDsn-Accounts

Environment tag values: alpha001-prod, alpha001-demo, alpha002-dev, alpha002-stage.

Phases

Phase 1: Common Module — SDK Integration

Core library changes in common-module under cards.arda.common.lib.oam.observability:

• Sentry initialization — reads DSN from configuration, graceful degradation when DSN is missing/invalid (application never fails to start due to observability)
• OTel initialization — OTLP HTTP exporter to Sentry when DSN is configured, logging exporter fallback for local development
• Replace homegrown monitoring plugins — remove ServerPerfMonitoringPlugin, ClientPerfMonitoringPlugin, ServerMDCPropagationPlugin, ClientMDCPropagationPlugin; replace with OTel Ktor server/client instrumentation and MDC injection
• X-Request-ID bridge — record as OTel span attribute arda.request_id
• Sentry integration in StatusPages — capture exceptions with request context; classify 5XX as errors, 4XX as breadcrumbs
• Data scrubbing (beforeSend) — filter by error type, scrub PII (tenant IDs, emails, auth headers, DB connection strings), configurable via ComponentBuilder
• Health/readiness endpoints — GET /{component}/oam/health/live and /ready, unauthenticated, excluded from tracing
• OAM configuration enhancement — include release version, Sentry status, OTel status

Phase 2: Helm Chart and Secret Provisioning

• Sentry project setup GitHub Action — idempotent composite action (sentry-project-setup-action) that creates the Sentry project and sets the SENTRY_DSN repo secret automatically
• CloudFormation/CI pipeline — add SentryDsn parameter following the DocumintApiKey pattern through reusable_deployment.yaml → pre.json → pre-install.cfn.yml → AWS Secrets Manager
• Helm chart changes (both components) — ExternalSecret data, secrets.properties mapping, configmap entries, liveness/readiness probes, per-environment sample rates
• Organization-level secrets — SENTRY_AUTH_TOKEN (Sentry API) and SENTRY_SETUP_TOKEN (GitHub PAT for gh secret set)

Phase 3: Component Integration

• Initialize SentryInitializer and OTelInitializer in both operations and accounts Main.kt
• Wrap DataSource with OTel JDBC instrumentation
• Update logback configuration to include trace_id and span_id in log pattern

Phase 4: Verification

• Unit tests for Sentry initialization, beforeSend scrubber, request ID bridge, health endpoints, OAM configuration
• Local integration test (empty DSN, logging exporter, health endpoints)
• Dev environment smoke test (error in Sentry, trace in Sentry, trace_id in CloudWatch, arda.request_id attribute)
• CI recommendation for Sentry release notification

Key Design Decisions

#	Question	Decision
DQ-001	Sentry project topology	One project per component, environment tags for partition separation
DQ-002	Tracing technology	OTel for tracing + Sentry for errors; OTLP export to Sentry (no vendor lock-in)
DQ-003	X-Request-ID correlation	Record as span attribute arda.request_id (decoupled from trace ID)
DQ-004	CloudWatch interaction	Unchanged; OTel MDC adds trace_id/span_id to log lines; homegrown perf logs removed
DQ-005	DSN provisioning	CloudFormation (DocumintApiKey pattern) → AWS SM → external-secrets → secrets.properties
DQ-006	Health endpoints	Added to common-module; unauthenticated, excluded from tracing
DQ-007	Data scrubbing	beforeSend filters 4XX, scrubs PII from 5XX; configurable via ComponentBuilder
DQ-008	Missing DSN behavior	Sentry disabled gracefully; OTel falls back to logging exporter
DQ-009	DSN secret source in CI	Repository-level GitHub Actions secret, one per component repo
DQ-010	OAM package restructuring	oam.observability + oam.configuration packages
DQ-011	Sentry project automation	Idempotent GitHub Action using Sentry API + gh secret set

Scope

In Scope

• Sentry SDK and OTel SDK integration in common-module
• Removal of homegrown monitoring plugins
• X-Request-ID bridge, health endpoints, data scrubbing, OAM enhancement
• Sentry project setup GitHub Action (sentry-project-setup-action)
• CloudFormation, CI pipeline, and Helm chart changes for both components
• Logback trace context injection
• Unit tests and integration verification

Out of Scope

• SQS consumer instrumentation
• gRPC interceptor (deferred until gRPC calls are introduced)
• Sentry alert rule implementation (recommendations provided)
• Source map / ProGuard deobfuscation
• CloudWatch Metrics, X-Ray, or AWS-side observability resources
• New infrastructure deployment (no OTel Collector)

Repositories Affected

• common-module — SDK integration, shared observability code
• operations — initialization, Helm chart, CloudFormation, CI pipeline
• accounts-component — initialization, Helm chart, CloudFormation, CI pipeline
• infrastructure — CloudFormation template changes
• sentry-project-setup-action (new) — idempotent Sentry project creation action

References

• Problem statement: /workspace/projects/ad-hoc/common-module/sentry/description.md
• Full specification: /workspace/projects/ad-hoc/common-module/sentry/specification.md
• Decision log: /workspace/projects/ad-hoc/common-module/sentry/decision-log.md