EKS Cluster

Scope

This page describes the EKS cluster at the Infrastructure level: how it is shaped, what runs on it, how it is shared by the Partitions inside its Infrastructure, and how it is configured, monitored, and secured. For the broader containment model (Root → Infrastructure → Partition → Component → Resource), see Platform Structure.

The audience is two-fold: a high-level overview for newcomers (sections Role in the Platform through Cohabitation across Partitions) and a detailed reference for backend / infrastructure engineers (sections Detailed Structure onwards, plus the OAM section). Read the first half end-to-end; reach into the second half as needed.

Role in the Platform

An EKS cluster is an Infrastructure-level resource — that is, a resource shared across the Partitions hosted by a single Infrastructure, because the cost of dedicating an EKS control plane and its supporting addons per Partition would dominate the platform’s spend without adding isolation that the Partition-level resources (DB cluster, Cognito user pool, API Gateway) do not already provide.

Infrastructure	EKS cluster	Partitions hosted
Alpha001	Alpha001-eks-cluster	demo, prod
Alpha002	Alpha002-eks-cluster	dev, stage
SandboxKyle002	SandboxKyle002-eks-cluster	kyle

The cluster’s name follows the FQN composition described in Platform Structure / Naming: ${Infrastructure}-eks-cluster. The cluster’s Kubernetes version, control-plane logging, addons, and Fargate profiles are uniform across Infrastructures — the only legitimate per-Infrastructure variance is the set of Partition Fargate profiles attached to it (see Cohabitation across Partitions).

The cluster is declared by the EksCluster CDK construct (infrastructure/src/main/cdk/constructs/compute/eks-cluster.ts) and instantiated by the Alpha00*-Compute stack in each Infrastructure’s app. See Stacks and Constructs for the CDK-tier conventions that shape this construct.

Cluster Anatomy

The cluster runs on AWS Fargate exclusively (defaultCapacity: 0, no EC2 node groups). Workloads land on Fargate via per-namespace Fargate profiles that pair a namespace selector with the pod execution role and a private-subnet placement. AWS-managed addons provide cluster-internal services (DNS, metrics, pod identity); helm-installed cluster services provide the load-balancer integration and the secret-delivery bridge. Per-partition resources (ingress controller, Component workloads) cohabit on the same cluster through namespace conventions.

Cohabitation across Partitions

A single cluster hosts multiple Partitions side-by-side. Isolation is namespace-based, with a Fargate profile per Partition mapping every ${purpose}-* namespace to that Partition’s pod execution role.

Element	Cluster-scoped	Per-Partition
EKS cluster, control-plane logs, OIDC provider	yes	—
add-ons Fargate profile (kube-system, aws-load-balancer-controller, external-secrets)	yes	—
Managed addons (CoreDNS, metrics-server)	yes	—
AWS Load Balancer Controller (helm)	yes	—
External Secrets Operator (helm)	yes	—
${purpose}-fargate-profile (selects ${purpose}-* namespaces)	—	yes
${purpose}-ingress-nginx namespace + ingress-nginx helm release	—	yes
${purpose}-<component> namespaces (e.g., prod-operations)	—	yes

Component workloads in different Partitions communicate only through their published API endpoints (per the Cross-Component Access rule); there is no direct in-cluster cross-Partition coupling at the Kubernetes layer.

Namespace naming

The convention is ${purpose}-${component}. Examples in production today:

Namespace	Partition	Owner
kube-system	(cluster)	EKS + addons
aws-load-balancer-controller	(cluster)	AWS LBC helm release
external-secrets	(cluster)	ESO helm release
dev-operations	dev (Alpha002)	operations Component
dev-ingress-nginx	dev (Alpha002)	ingress-nginx helm release
stage-operations	stage (Alpha002)	operations Component
demo-operations	demo (Alpha001)	operations Component
prod-operations	prod (Alpha001)	operations Component
prod-ingress-nginx	prod (Alpha001)	ingress-nginx helm release

This pattern lets the per-Partition Fargate profile use a single namespace selector — ${purpose}-* — to cover every Component a Partition deploys, without the EKS profile having to be amended each time a new Component is introduced.

Detailed Structure

This section is for engineers who need to understand or modify the cluster. Skim or skip if you only need the high-level view.

CDK construct: EksCluster

Source: infrastructure/src/main/cdk/constructs/compute/eks-cluster.ts. Builds, in this order:

1. EksClusterRole — IAM Role assumed by eks.amazonaws.com; carries AmazonEKSClusterPolicy.
2. EksPodExecutionRole — IAM Role assumed by eks-fargate-pods.amazonaws.com; carries AmazonEKSFargatePodExecutionRolePolicy plus an inline policy granting logs:CreateLog{Group,Stream}, logs:PutLogEvents, logs:DescribeLogStreams on *.
3. Two CloudWatch log groups, both with 14-day retention and RemovalPolicy.DESTROY:
   • /aws/eks/${cluster}/eks-logs — control-plane log destination.
   • /aws/eks/${cluster}/fluent-bit-logs — FluentBit-side destination (see Performance).
4. The eks.Cluster itself: K8s 1.33 today (the table in supportedEksVersions also pins 1.32), defaultCapacity: 0, VPC subnets restricted to PRIVATE_WITH_EGRESS, clusterLogging: [API, AUDIT, AUTHENTICATOR, CONTROLLER_MANAGER, SCHEDULER], authenticationMode: API. The control plane uses the KubectlV32Layer lambda layer so CDK-driven kubectl calls run a client within Kubernetes’ ±1 minor skew policy (kubectl 1.32 against a 1.33 control plane).
5. The cluster default security group is augmented with ingress rules on ports 80, 443, and 10254 (the nginx-ingress healthz port) from the VPC CIDR.
6. The OIDC provider (cluster.openIdConnectProvider) is materialised so subsequent CDK code can mint IRSA roles bound to its issuer.
7. ${Infrastructure}-SecretsManagerReadRole — IRSA role for the secret-reader Service Account pattern (see Security).
8. The add-ons Fargate profile (see Fargate profiles), depending on the SecretsManagerReadRole so the role exists before workloads that assume it.
9. Managed addons (see Managed addons), each depending on add-ons so their pods have a profile they can land on.
10. Three EKS::AccessEntry resources, all with the cluster-admin policy arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy:
    • GitHubAccessInfrastructureEntry — for the GitHub Actions role used by infrastructure deploys (amm.sh).
    • GitHubAccessEntry — for the GitHub Actions role used by Component repository deploys.
    • AdminAccessEntry — for the human administrator role.
11. Tagging: a curated subset of construct-created resources receives Environment=${infraId}. Three resources are currently outside the loop (metricsServerAddOn, adminAccessEntry, gitHubAccessEntry) — see Known drift and gaps.

The construct exposes a built interface with cluster, clusterEndpoint, clusterOpenIdConnectUrl, clusterArn, clusterId, namespacePrefix, podExecutionRole, and readSecretRole, which the per-Partition PurposeComputeStack consumes.

Fargate profiles

add-ons (cluster-scoped)

Created by EksCluster itself. Selects three namespaces:

• kube-system — homes CoreDNS, metrics-server, the two drift addons, and any future managed addon (managed addons are scheduled into kube-system by AWS).
• aws-load-balancer-controller — homes the LBC pod set (see AWS Load Balancer Controller).
• external-secrets — homes ESO’s controller pod set (see External Secrets Operator).

All three namespaces share the same pod execution role and the same private-subnet placement (PRIVATE_WITH_EGRESS).

${purpose}-fargate-profile (per-Partition)

Created by the PurposeComputeStack stack (infrastructure/src/main/cdk/stacks/purpose/purpose-compute.ts) via the FargateProfile construct (infrastructure/src/main/cdk/constructs/compute/fargate-profile.ts). One profile per Partition, named ${purpose}-fargate-profile, with a single namespace selector ${purpose}-* and a pod role passed in via the Partition’s app. The profile’s CFN export ${publishingPrefix}-FargateProfileArn is consumed by downstream stacks that need to wait on it before scheduling pods.

This is the mechanism that lets a new Component be deployed into a Partition without any cluster-level change: as long as the Component’s namespace matches ${purpose}-*, it inherits the Partition’s profile.

Managed addons

The cluster currently has four installed managed addons. Two are declared in CDK; two are drift (see Known drift and gaps).

Addon	Source	Version	Notes
coredns	CDK (EksCluster)	Pinned per K8s version (v1.12.2-eksbuild.4 for 1.33)	configurationValues: { "computeType": "Fargate" } — the CoreDNS-specific shortcut that handles Fargate scheduling internally.
metrics-server	CDK (EksCluster)	v0.8.1-eksbuild.6 (1.32 and 1.33)	configurationValues carries a pod-level toleration for the eks.amazonaws.com/compute-type=fargate:NoSchedule taint, because metrics-server has no CoreDNS-style shortcut. Installed by PDEV-491.
eks-pod-identity-agent	Out-of-band (drift)	v1.3.9-eksbuild.5, ACTIVE	Installed manually on 2025-11-12. Provides the pod identity association mechanism that complements IRSA. Should be added to CDK.
amazon-cloudwatch-observability	Out-of-band (drift)	v4.6.0-eksbuild.1, UPDATE_FAILED	Installed manually on 2025-11-12, currently failing reconciliation. Should be added to CDK and the failure investigated (or the addon removed if not needed).

Versions for the CDK-declared addons are paired with the cluster’s K8s version in the supportedEksVersions table at the top of eks-cluster.ts. When the cluster K8s version is bumped, the addon versions must be re-pinned at the same time (verified via aws eks describe-addon-versions --addon-name <name> --kubernetes-version <ver>).

Helm-installed cluster services

These run in cluster-scoped namespaces covered by the add-ons Fargate profile. They are installed by amm.sh Step 1.4 / 1.5 (after CDK has finished and the cluster’s kubeconfig has been refreshed) rather than as EKS managed addons because each one needs configuration that the addon mechanism does not yet expose, or because they predate the addon being available.

AWS Load Balancer Controller

Helm chart aws-load-balancer-controller, version 1.13.4, repo https://aws.github.io/eks-charts, namespace aws-load-balancer-controller. Configured with the cluster’s name and VPC ID and bound to the LBC IAM role exported by the Infrastructure stack as ${Infrastructure}-I-LbcRoleArn. Watches Ingress and Service resources in every namespace; provisions and reconciles AWS Network Load Balancers (NLBs) and their target groups.

External Secrets Operator

Helm chart external-secrets, version 0.19.1, repo https://charts.external-secrets.io, namespace external-secrets. Cluster-scoped CRs (ClusterSecretStore, ClusterPushSecret) are disabled in the helm values; only namespace-scoped SecretStore / ExternalSecret are processed. The webhook runs on port 9443. ESO is the consumer side of the secret-delivery pattern — see Secret Delivery Pattern for the source-to-destination flow.

Per-Partition helm releases

ingress-nginx

Helm chart ingress-nginx, version 4.13.0, repo https://kubernetes.github.io/ingress-nginx, installed by amm.sh once per Partition into the ${purpose}-ingress-nginx namespace. Each Partition owns its own controller deployment, exposed as ClusterIP (the NLB is fronted by the AWS Load Balancer Controller). The IngressClass is ${purpose}-nginx and the controller value is arda.cards/${purpose}-ingress-nginx so that Component Ingress resources can select the right controller by ingressClassName. controller.replicaCount: 2 provides redundancy.

Component releases (e.g. operations)

Each Component repository deploys its workload helm chart into the namespace ${purpose}-${component} via its own deploy.yaml workflow (see Configuration below for the deploy chain and the chart-version contract). The chart typically renders a Deployment, Service, Ingress, HorizontalPodAutoscaler, and one or more ExternalSecret resources. The HPA pulls Resource-type CPU metrics from metrics.k8s.io (served by metrics-server).

Known drift and gaps

The following diverge from “what CDK declares” and are worth tracking as follow-ups:

• amazon-cloudwatch-observability and eks-pod-identity-agent addons are installed but not declared in CDK. Add them to supportedEksVersions and the addon block in EksCluster. Investigate the UPDATE_FAILED state on amazon-cloudwatch-observability while doing so.
• metricsServerAddOn, adminAccessEntry, gitHubAccessEntry are not included in the construct’s tagging loop, so they ship without the Environment=${infraId} tag that every other construct-created resource carries. See PR infrastructure#461 review thread for the discussion that deferred this cleanup.
• The class-level JSDoc at the top of eks-cluster.ts is partially stale — it claims ingress-nginx is in the add-ons profile and mentions ElasticLoadBalancing Enabled / Block Storage enabled / ZonalShiftControl enabled flags that are no longer present in the constructed eks.ClusterProps. Tracked alongside the tagging cleanup above.

Operations, Administration & Maintenance (OAM)

This section adopts a lightweight version of the OAM disciplines from TMN’s FCAPS model (Fault, Configuration, Accounting, Performance, Security). At Arda’s current size, Configuration, Performance, and Security carry most of the load; Fault is folded into Performance (because for us “fault detection” is a monitoring concern, not a separate plane); Accounting is out of scope at the cluster level — billing visibility lives in AWS Cost Explorer and is handled at the Infrastructure / account level rather than per cluster.

Configuration

What configures the cluster

Two layers, in deploy order:

1. CDK + CloudFormation, invoked by amm.sh ${Infrastructure} from the infrastructure repository. This creates / updates the EKS control plane, addons (CDK-declared), Fargate profiles, IAM roles, OIDC provider, and access entries. Versions for the cluster (K8s release), CoreDNS, and metrics-server are pinned in supportedEksVersions (eks-cluster.ts).
2. Helm releases, invoked by amm.sh after the CDK phase (Steps 1.4 / 1.5) for cluster-scoped services, and by each Component’s deploy.yaml workflow for Partition-scoped Component workloads. Versions for the cluster-scoped helm releases (AWS LBC, ESO, ingress-nginx) are hard-coded in amm.sh; versions for Component workloads are operator-supplied per deploy (see below).

amm.sh is described in infrastructure/amm.sh. The operator triggers it; the script is self-defined (every value sourced from the repository) and self-documenting (echoes each step). Calling CDK directly (e.g., npx cdk deploy) is not the supported workflow — it bypasses SSO login, CloudWatch deployment-audit logging, kubeconfig update, the helm installs, and the Amplify BuildSpec drift check.

Two amm.sh invocation shapes are relevant for the cluster:

# Cluster-only change (e.g., an addon version bump, an access-entry addition).
./amm.sh Alpha002          # no partitions argument — Partition loop is skipped.

# Cluster change AND per-Partition reconciliation in the same run.
./amm.sh Alpha001 all      # expands to "demo prod" for Alpha001.

Chart versions and how they tie to a deploy

Every Component workload is delivered to the cluster as a versioned Helm chart published to a registry. The chart version is the only input an operator supplies at deploy time; everything else is derived from that version or from per-Partition values in the chart.

The contract between the upstream build pipeline (Source → CI → ECR) and the cluster (helm-deploy-pipeline-action) is:

• The Component’s container image is built and pushed to ECR with a tag equal to the chart version.
• The Helm chart is packaged with the same version (semver, or <semver>-<user>-<issue> for short-lived PR builds) and pushed to the chart registry (ghcr.io under the Arda-cards organisation by default).
• The chart’s Chart.yaml appVersion is set to the same value, so a single version string addresses the image, the chart, and the deployed Deployment.metadata.labels["app.kubernetes.io/version"].
• Tags are immutable: once a chart version is published, it never changes. Bug fixes ship as a new version (see Build and Deployment / Pipeline Guarantees).

The deploy itself is dispatched per-Partition through the Component repository’s deploy.yaml GitHub Actions workflow (workflow_dispatch with inputs chart_name, chart_version, component_name, purpose). The workflow calls the reusable reusable_deployment.yaml, which in turn invokes Arda-cards/helm-deploy-pipeline-action@v4. The action resolves CloudFormation exports under the ${Infrastructure}-${purpose}-API-* namespace into Helm values, installs / upgrades the release into ${purpose}-${component_name}, and waits for the rollout to complete.

Sentry tie-in

Sentry is configured at two layers and tied to the chart version explicitly:

1. One Sentry organisation, two projects. The org is arda-systems. The backend project is platform-be; the frontend project is arda-frontend. There is no per-Partition project — events from every Partition flow into the same backend project, distinguished by the environment tag (dev, stage, demo, prod).
2. Per-Component config sourced from ESO-projected secrets. The DSN is stored in AWS Secrets Manager as ${Infrastructure}-SentryDsn and projected into the Component’s namespace as a Kubernetes Secret named be-sentry-dsn (key dsn) by an ESO ExternalSecret. A per-Partition scrub salt is stored as ${Infrastructure}-${purpose}-SentryScrubSalt and projected the same way; the salt is consumed by the Sentry SDK’s beforeSend hook to hash PII consistently within a Partition without enabling cross-Partition correlation.
3. Release identity = chart version. The Sentry SDK is initialised at pod startup with release = <chart_version> and environment = <purpose>. Because chart versions are immutable, every Sentry event traces back to exactly one deployable artefact. Release-health sessions are emitted per request (see operations/src/main/kotlin/cards/arda/operations/runtime/Main.kt).

Discovery query for the current Sentry state at any point in time:

# Projects in the org (slugs, names).
sentry-cli projects list --org arda-systems
# Releases for the backend project — confirms which chart versions have been deployed.
sentry-cli releases list --org arda-systems --project platform-be

Performance

The cluster’s performance / fault surface is layered. Each layer is a distinct signal source with its own retention and consumer.

Signal	Source	Destination	Retention	Consumer
Control-plane logs (api, audit, authenticator, controllerManager, scheduler)	EKS managed	CloudWatch /aws/eks/${cluster}/eks-logs	14 days	AWS console, CloudWatch Logs Insights
Pod logs (every namespace)	FluentBit (cluster-side DaemonSet equivalent — see amm.sh Step 1.3)	CloudWatch group /{Infrastructure}/eks-logs, stream prefix from-fluent-bit-	60 days (auto-created at first write)	AWS console, ad-hoc Insights queries
Cluster resource metrics (metrics.k8s.io)	metrics-server managed addon (installed by PDEV-491)	In-cluster (Kubernetes Metrics API)	live (no retention)	kubectl top; HPA controller; future custom autoscalers
Application errors and traces	Sentry Java SDK in each Component	arda-systems org, platform-be project (backend) / arda-frontend (frontend), tagged with environment and release	per Sentry plan (issues are sticky; transactions sampled)	engineering, on-call
End-to-end health probes	Bruno API test suite running as a Kubernetes CronJob	CloudWatch metrics + Slack alerts	(see Service Monitoring)	on-call, see Service Monitoring
API Gateway 5xx, NLB unhealthy targets	AWS native metrics	CloudWatch alarms → Slack	(CloudWatch retention)	P0 / P1 alerts (see Service Monitoring)

HPA reconciliation (post-PDEV-491)

With metrics-server available, the HorizontalPodAutoscaler resources already deployed by each Component’s chart can reconcile replica counts from CPU utilisation. The HPA is set up uniformly today as minReplicas=2, maxReplicas=4 (dev/stage/demo) or 8 (prod), targetCPUUtilization=60%, with conservative scaleUp / scaleDown stabilisation windows. The HPA’s ScalingActive condition surfaces whether the controller is actually computing replica counts — True with reason ValidMetricFound is the healthy steady state.

Known gaps

• amazon-cloudwatch-observability is installed but UPDATE_FAILED; the Container Insights signal it would provide (per-container CPU / memory time series in CloudWatch) is currently not flowing. Fixing this addon is the next observability gap to close.
• There is no in-cluster metric dashboard today (no Grafana, no CloudWatch dashboard JSON checked into IaC). kubectl top is the available view; dashboards are a follow-up.
• Cluster-level cost telemetry (per-namespace pod cost) is not surfaced. AWS Cost Explorer at the EKS-service level is the current substitute.

Security

Network

• Cluster nodes are exclusively on private subnets with NAT egress (PRIVATE_WITH_EGRESS). No pod has a public IP.
• The cluster security group permits ingress on ports 80, 443, and 10254 (the nginx healthz port) from the VPC CIDR only. Cross-AZ traffic between pods uses Fargate-managed ENIs in the same VPC.
• The API server endpoint is publicly accessible (endpointPublicAccess: true, publicAccessCidrs: ["0.0.0.0/0"]) AND privately accessible (endpointPrivateAccess: true). Public access is gated by AWS IAM + Kubernetes RBAC, but exposing the endpoint to the internet is a posture choice worth reviewing — narrowing publicAccessCidrs to the office / VPN / GitHub Actions IPs would shrink the attack surface materially.

Authentication and authorization

• Mode: AuthenticationMode.API — the cluster authenticates exclusively via EKS Access Entries (the modern API-based mechanism); the legacy aws-auth ConfigMap is not used.
• Cluster-admin entries: three principals carry AmazonEKSClusterAdminPolicy:
  • The infrastructure-repo GitHub Actions role (drives amm.sh in CI).
  • The Component-repo GitHub Actions role (drives helm-deploy-pipeline-action).
  • The human administrator role (adminRoleArn passed in per Infrastructure).
• No other access entries exist by default. Components do not get cluster-level RBAC; their pods get pod-level AWS access via IRSA (below).

IRSA and pod identity

The cluster materialises its OIDC provider and creates exactly one IRSA role itself — ${Infrastructure}-SecretsManagerReadRole — bound to system:serviceaccount:*:${infraIdLC}-secrets-reader and system:serviceaccount:*:secrets-reader Service Accounts (a wildcard across all namespaces). The role’s policy grants Secrets Manager read on arn:aws:secretsmanager:${region}:${account}:secret:${infraId}-*, which is the prefix every Infrastructure secret follows.

Other IRSA roles (per-Component, e.g., for S3, SQS access) are created by their respective Component charts or stacks; the cluster’s OIDC issuer is the trust anchor.

The eks-pod-identity-agent addon (currently drift, see Managed addons) provides AWS’s newer Pod Identity Association mechanism alongside IRSA. Today no Component uses Pod Identity; the agent is present but unused.

Secret delivery

The end-to-end secret-delivery pattern is documented in Secret Delivery Pattern. Cluster-side concerns:

• External Secrets Operator (cluster-scoped helm release) watches every namespace for ExternalSecret / SecretStore CRs.
• Each Component namespace defines a SecretStore of kind aws.secretsmanager that assumes ${Infrastructure}-SecretsManagerReadRole via IRSA (the Service Account ${infraIdLC}-secrets-reader is created by the chart).
• ExternalSecret CRs declare which AWS Secrets Manager secrets to project into Kubernetes Secrets in the local namespace. Examples: be-sentry-dsn (Sentry DSN), be-sentry-scrub-salt (per-Partition salt), arda-api-key, hubspot-pat.
• AWS Secrets Manager itself is sourced from 1Password via amm.sh (op read → CFN NoEcho parameter → SM secret). 1Password is the single source of truth.

Container and supply-chain security

Not yet addressed at the IaC layer:

• Container images are pulled from GHCR (Arda-cards organisation) but image signing / verification (Cosign, Notary, or AWS Signer for ECR images) is not enforced.
• No admission-controller policy (OPA Gatekeeper / Kyverno) is installed. Pod Security Standards labels are not applied to namespaces.
• No NetworkPolicy resources are deployed. In-cluster pod-to-pod traffic is unrestricted across namespaces; Partition isolation relies on namespace conventions and IAM rather than network policy.

These are known gaps; closing them is part of the broader platform security roadmap, not scoped to any current ticket.

Ingress traffic — what reaches a Component pod

The trust boundary at request time runs from the public internet through the API Gateway, VPC link, NLB, and finally the per-Partition ingress-nginx, before any Component code sees the request. The sequence below sketches the cluster-side leg.

For the request path before the NLB (DNS resolution, mTLS between API Gateway and the cluster), see Network Routing and mTLS.

See also

• Platform Structure — placement of EKS at the Infrastructure level; naming and FQN composition.
• Runtime Environments — Infrastructures and Partitions in service today.
• Build and Deployment — upstream pipeline that produces the chart versions deployed onto the cluster.
• Network Routing — full request path from client to a Component pod.
• Service Monitoring — Bruno API tests, CloudWatch alarms, alert priorities.
• Secret Delivery Pattern — end-to-end secret delivery feeding the cluster.
• Infrastructure as Code — CDK Constructs / Stacks / Apps conventions that this construct follows.