Drift Warnings — Manual Interventions Outside IaC

Manual AWS CLI modifications applied during implementation that are not captured in CloudFormation templates, CDK constructs, or amm.sh. These represent drift from the infrastructure-as-code state and should be tracked for eventual reconciliation.

Run 1: Infrastructure

1. GITHUB_TOKEN environment variable on demo Amplify app

What: Added GITHUB_TOKEN to the Alpha001-demo Amplify app’s environment variables via AWS CLI.

Why: The amplify.cfn.yaml template does not include GITHUB_TOKEN. The frontend app requires it for @arda-cards/design-system from GitHub Packages. Build failed with npm error 401 Unauthorized.

Command applied:

EXISTING=$(aws --profile Admin-Alpha1 amplify get-app \
  --app-id d2jmcx9om9gokx --region us-east-1 \
  --query "app.environmentVariables" --output json)
MERGED=$(echo "${EXISTING}" | jq --arg token "$(op read 'op://Arda-SystemsOAM/GPR-Read token/credential')" \
  '. + {"GITHUB_TOKEN": $token}')
aws --profile Admin-Alpha1 amplify update-app \
  --app-id d2jmcx9om9gokx --region us-east-1 \
  --environment-variables "${MERGED}"

Affected resource: Amplify App d2jmcx9om9gokx (Alpha001, us-east-1)

Risk: Any CloudFormation stack update to Alpha001-demo-Amplify that triggers an Amplify app resource replacement will lose this env var. Stack updates that only modify other properties (parameters, outputs) are safe — CloudFormation doesn’t touch env vars unless the EnvironmentVariables property itself is changed.

Reconciliation: infrastructure#427 — add GITHUB_TOKEN to the amplify.cfn.yaml template.

Applies to existing apps: The dev, stage, and prod apps also have GITHUB_TOKEN set manually (pre-dating this project). They carry the same drift risk. The ticket covers all apps.

2. AMPLIFY_GITHUB_ACCESSTOKEN PAT regenerated

What: The fine-grained GitHub PAT used by Amplify was regenerated with a new expiration date and updated permissions.

Why: The enterprise policy rejected tokens without an expiration date (lifetime > 366 days). The token also needed “Contents: Read-only” permission added for arda-frontend-app.

Where updated:

• 1Password: Arda-SystemsOAM/Amplify_GitHub_AccessToken — updated with new token value
• GitHub secret: AMPLIFY_GITHUB_ACCESSTOKEN on infrastructure repo — updated via gh secret set

Risk: The token has a finite expiration (within 366 days of 2026-03-19). It will need to be rotated before expiry. Any system reading the old token value from a cache will fail with 401 Bad credentials.

Reconciliation: None needed — this is a secret rotation, not infrastructure drift. Consider adding a calendar reminder for token renewal.

Run 2: Workflows

3. Frontend IAM role OIDC trust policy — environment: subjects — RESOLVED

Resolved by: infrastructure#428 (merged). CDK code matches live state. Will be fully reconciled on next amm.sh CDK deployment after infrastructure#429 merges (both fixes on main).

What: Updated the IAM trust policy on Alpha001-API-GitHubActionFrontEnd and Alpha002-API-GitHubActionFrontEnd directly via AWS CLI to add environment: OIDC subject patterns.

Why: When a GitHub Actions job sets environment: <name>, the OIDC sub claim changes from repo:Org/Repo:ref:refs/heads/<branch> to repo:Org/Repo:environment:<name>. The CDK-deployed trust policy only had ref: patterns, so sts:AssumeRoleWithWebIdentity failed.

Commands applied:

# Alpha001
aws --profile Admin-Alpha1 iam update-assume-role-policy \
  --role-name Alpha001-API-GitHubActionFrontEnd \
  --policy-document file:///tmp/trust-policy.json

# Alpha002
aws --profile Alpha002-Admin iam update-assume-role-policy \
  --role-name Alpha002-API-GitHubActionFrontEnd \
  --policy-document file:///tmp/trust-policy-alpha002.json

The updated policy adds these subjects alongside the existing ref: patterns:

• repo:Arda-cards/arda-frontend-app:environment:dev
• repo:Arda-cards/arda-frontend-app:environment:stage
• repo:Arda-cards/arda-frontend-app:environment:demo
• repo:Arda-cards/arda-frontend-app:environment:prod

Affected resources: IAM roles in both accounts

Reconciliation: infrastructure#428 — CDK code updated in gh-oidc-provider.ts to match the live state. Once merged and deployed via amm.sh, the CDK-managed state will match.

Risk: The next amm.sh CDK deployment to either account will reconcile the trust policy. If PR#428 is not merged first, the CDK deployment will revert the trust policy to the old state (without environment: subjects), breaking the frontend workflow. Merge PR#428 before any amm.sh CDK deployments.

4. ARDA_API_KEY for demo partition — Secrets Manager, Amplify env var, and GitHub org secret

What: Updated the ARDA_API_KEY value in three places:

1. Secrets Manager (Alpha001-demo-ArdaApiKey) — via aws secretsmanager put-secret-value
2. Amplify app env var (d2jmcx9om9gokx) — via aws amplify update-app --environment-variables (merge pattern)
3. GitHub org secret ARDA_API_KEY_DEMO — via gh secret set

Why: The ARDA_API_KEY_DEMO GitHub org secret had a stale value that didn’t match what the demo backend API Gateway expects. The first GH Actions amm.yml run for Alpha001/demo wrote this stale value to Secrets Manager (via partitionSecrets.cfn.yaml), which then propagated to the Amplify app. The backend returned 401 on all API calls, causing a sign-out redirect on every page navigation.

Root cause: The ARDA_API_KEY_DEMO GitHub org secret was out of sync with 1Password (op://Arda-DemoOAM/ARDA-API-KEY/password). The infrastructure/tools/sync-secrets-from-1password.sh script should keep these in sync — verify it covers the DEMO variant.

Risk: The next amm.sh run for Alpha001/demo (via GH Actions or locally) will overwrite Secrets Manager with whatever ARDA_API_KEY it receives. If the GitHub secret or 1Password fallback has a wrong value, the issue recurs. The amplify.cfn.yaml resolves ARDA_API_KEY from Secrets Manager at stack deploy time — but aws cloudformation deploy may report “No changes” because the template definition hasn’t changed, even though the secret value has. A manual amplify update-app or a forced CloudFormation redeploy is needed after any secret rotation.

Reconciliation: The GitHub org secret has been updated. Verify sync-secrets-from-1password.sh includes the DEMO variant. No code changes needed — this is a secret value sync issue.

Run 4: Cutover

5. Auto-build disabled on dev, stage, prod Amplify apps

What: Disabled enableAutoBuild on all three existing Amplify apps via AWS CLI.

Commands applied:

# dev (Alpha002)
aws amplify update-branch --app-id d38w5m1ngjza76 --branch-name dev --no-enable-auto-build --region us-east-1

# stage (Alpha002)
aws amplify update-branch --app-id d1kbrvra79y8sc --branch-name stage --no-enable-auto-build --region us-east-1

# prod (Alpha001)
aws amplify update-branch --app-id duhexavnwh88g --branch-name main --no-enable-auto-build --region us-east-2

Affected resources: Amplify Branch resources on all three existing apps

Risk: If the GitHub Actions pipeline fails and auto-build is not re-enabled, no deployments will occur for the affected partition. See rollback-plan.md for re-enabling auto-build per partition.

Reconciliation: This is the intended final state — auto-build is permanently disabled. The GitHub Actions pipeline owns deployment orchestration. These apps are not managed by CloudFormation, so there is no CDK/CFn drift concern.

7. Frontend IAM role Amplify permissions — added us-east-2 resource ARN — RESOLVED

Resolved by: infrastructure#429. CDK code updated. Will be reconciled on next amm.sh CDK deployment to Alpha001.

What: Updated the inline policy AmplifyDeployment on Alpha001-API-GitHubActionFrontEnd to include arn:aws:amplify:us-east-2:009765408297:apps/* alongside the existing us-east-1 ARN.

Why: The prod Amplify app is in us-east-2 but the CDK stack deploys to us-east-1. The IAM policy only granted Amplify permissions for us-east-1, causing AccessDeniedException on StartJob for prod.

Command applied:

aws --profile Admin-Alpha1 iam put-role-policy \
  --role-name Alpha001-API-GitHubActionFrontEnd \
  --policy-name AmplifyDeployment \
  --policy-document file:///tmp/amplify-policy.json

Reconciliation: infrastructure#429 — CDK code updated. Once merged and deployed via amm.sh, the CDK state will match.

Risk: Same as item 3 — the next amm.sh CDK deployment to Alpha001 will reconcile. If PR#429 is not merged first, CDK will revert the policy to us-east-1 only, breaking prod deployments. Merge PR#429 before any amm.sh CDK deployments to Alpha001.

8. main branch resources on dev, stage, demo Amplify apps

What: Created main branch resources on the dev, stage, and demo Amplify apps. Updated CloudFormation export stacks to point AmplifyBranchName to main instead of the partition-specific branch names (dev, stage, demo).

Why: The deployment workflow passes --commit-id {sha} from main to StartJob. Amplify can only find commits reachable from the connected branch. With the old branch resources (dev, stage, demo), commits on main were unreachable, causing fatal: unable to read tree.

Commands applied:

# Dev: main branch already existed (from PR preview setup)
aws amplify update-branch ... (export stack updated to AmplifyBranchName=main)

# Stage: created main branch + updated export
aws amplify create-branch --app-id d1kbrvra79y8sc --branch-name main --no-enable-auto-build ...

# Demo: deleted CLI-created main branch, redeployed AmplifyBranch CFn stack with Branch=main
aws cloudformation deploy --stack-name Alpha001-demo-AmplifyBranch ... Branch=main

Domain associations updated: Dev and stage domain associations were pointing to the old dev/stage branch resources, meaning the custom domains served stale content while new deploys went to main. Updated via aws amplify update-domain-association to point to main.

Cleanup needed: The old dev, stage branch resources still exist on their respective apps alongside main. Once confirmed stable:

• Domain associations should be moved to the main branch resources
• Old dev/stage branch resources can be deleted
• The demo branch resource on the demo app was replaced by CloudFormation (clean)

Reconciliation: The export stacks (Alpha002-dev-AmplifyExports, Alpha002-stage-AmplifyExports) need to stay in sync. For demo, the Alpha001-demo-AmplifyBranch CloudFormation stack was updated directly (clean). The amm.sh branch name mapping still maps dev→dev, stage→stage — this should be updated to dev→main, stage→main to match.

9. Domain associations updated from old branches to main on dev and stage

What: Updated domain associations on dev and stage Amplify apps to point to the main branch resource instead of the old dev/stage branch resources.

Why: The pipeline deploys to the main branch resource, but the custom domains were serving content from the old dev/stage branch resources (stale builds).

Commands applied:

aws --profile Alpha002-Admin amplify update-domain-association \
  --app-id d38w5m1ngjza76 --domain-name dev.alpha002.app.arda.cards \
  --sub-domain-settings "prefix='',branchName=main" "prefix=www,branchName=main" --region us-east-1

aws --profile Alpha002-Admin amplify update-domain-association \
  --app-id d1kbrvra79y8sc --domain-name stage.alpha002.app.arda.cards \
  --sub-domain-settings "prefix='',branchName=main" "prefix=www,branchName=main" --region us-east-1

Reconciliation: PR #429 needs the amm.sh branch mapping updated (dev→main, stage→main) and the domain association logic in amplifyBranch.cfn.yaml should reference the main branch. Until then, any amm.sh run that redeploys the AmplifyBranch stack could revert the domain association back to the old branch.

10. PR preview disabled on demo main branch

What: Disabled enablePullRequestPreview on demo app’s main branch resource via CLI.

Why: Both dev and demo had PR preview enabled on main. The webhook routed to demo (Alpha001) instead of dev (Alpha002). Disabling on demo allows dev to receive the PR webhook events.

Command applied:

aws --profile Admin-Alpha1 amplify update-branch \
  --app-id d2jmcx9om9gokx --branch-name main \
  --no-enable-pull-request-preview --region us-east-1

Reconciliation: PR #429 changes amplifyBranch.cfn.yaml to set EnablePullRequestPreview: false on the deployment branch. PR preview is handled by the separate conditional AmplifyPRPreviewBranch resource only.

8a. PR preview main branch resource on dev Amplify app — RESOLVED

Resolved by: infrastructure#429. Template updated with EnablePRPreview parameter; amm.sh sets it for dev partition.

What: Created a main branch resource on the dev Amplify app (d38w5m1ngjza76) via AWS CLI to enable PR previews for PRs targeting main.

Why: Amplify PR previews only trigger for PRs targeting the git branch the branch resource is connected to. The dev app’s deployment branch is dev, so PRs targeting main were not triggering previews.

Command applied:

aws --profile Alpha002-Admin amplify create-branch \
  --app-id d38w5m1ngjza76 --branch-name main \
  --no-enable-auto-build --enable-pull-request-preview \
  --framework "Next.js - SSR" --stage PRODUCTION --region us-east-1

Reconciliation: PR #429 adds EnablePRPreview parameter to amplifyBranch.cfn.yaml and sets it in amm.sh for dev. Next amm.sh Alpha002 dev deployment will reconcile.

6. Lightweight CloudFormation export stacks for dev, stage, prod

What: Deployed amplifyExports.cfn.yaml stacks to publish Amplify App ID and Branch Name as CloudFormation exports.

Commands applied:

# All in us-east-1 regardless of Amplify app region
aws cloudformation deploy --stack-name "Alpha002-dev-AmplifyExports" ...
aws cloudformation deploy --stack-name "Alpha002-stage-AmplifyExports" ...
aws cloudformation deploy --stack-name "Alpha001-prod-AmplifyExports" ...

Affected resources: 3 new CloudFormation stacks (us-east-1)

Risk: None — these are read-only export stacks with a WaitConditionHandle placeholder resource. They publish static values. The workflow reads these exports to discover Amplify App IDs at deploy time.

Reconciliation: None needed — these are new IaC-managed resources, not drift. The template is in the infrastructure repo.

Existing Drift (Pre-Project, Noted for Awareness)

These are not caused by this project but are relevant to Run 4 (cutover).

3. Environment variable gaps between existing apps and CloudFormation template

The manually-created dev, stage, and prod apps have environment variables that differ from the amplify.cfn.yaml template:

Variable	dev	stage	prod	CFn template
GITHUB_TOKEN	set	set	set	missing
NEXT_PUBLIC_SENTRY_DSN	set	set	set	missing
TENANT_ID	set	set	set	missing
TENANT_ID_2	set	set	set	missing
CORE_BACKEND_URL	—	—	set	missing
NEXT_PUBLIC_FRONTEND_URL	—	missing	set	present

See Amplify Current Configuration for the full comparison.

Risk for cutover: The existing apps are not managed by CloudFormation. The cutover plan (Run 4) only disables auto-build and deploys lightweight export stacks — it does not modify app environment variables. No drift risk from cutover.

4. Build spec differences between existing apps

Each existing app has a different inline build spec. The dev and prod apps write .env files in preBuild; stage does not. The demo app (created by CloudFormation) uses the build spec from the amplify.yml in the repository root, which differs from all three existing apps.

Risk for cutover: Build specs are not modified during cutover. StartJob uses the existing build spec on each app. No immediate risk, but be aware that the demo app may behave differently from prod due to build spec differences.