Task 2.18: Deploy to Alpha002/dev and Verify

Operator instructions for deploying the image storage infrastructure to the Alpha002/dev environment and running end-to-end verification.

Prerequisites

• Run 1 commit merged into the current infrastructure branch (assets.arda.cards DNS foundation)
• Run 2 code complete (all constructs, stacks, tests passing)
• AWS SSO credentials available for:
  • Root account (Admin-PlatformRoot) — for deploy-root.sh
  • Alpha002 account (Alpha002-Admin) — for amm.sh

Step 1: Deploy Root Account (one-time)

The assets.arda.cards hosted zone must exist in the root account before any infrastructure account can create subdomain zones.

cd /Users/jmp/code/arda/image-upload-infrastructure-worktrees/infrastructure

# Authenticate to root account
aws sso login --profile Admin-PlatformRoot

# Deploy root configuration
./deploy-root.sh

Verify:

aws route53 list-hosted-zones-by-name \
  --dns-name assets.arda.cards \
  --max-items 1 \
  --profile Admin-PlatformRoot

Expected: one zone with Name: assets.arda.cards.

Step 2: Deploy Alpha002 Infrastructure + dev Partition

# Authenticate to Alpha002 account
aws sso login --profile Alpha002-Admin

# Deploy infrastructure (creates assets subdomain zone + ACM cert)
# AND dev partition (creates ImageStorageStack with bucket, CDN, signing keys)
./amm.sh Alpha002 dev

This runs:

1. Infrastructure step: deploys InfrastructureIngress with the new alpha002.assets.arda.cards subdomain zone and ACM cert
2. Partition step: deploys ImageStorageStack (S3 bucket, CloudFront CDN, signing key group)

Wait for ACM certificate: DNS validation may take 1-5 minutes. Check:

aws acm list-certificates \
  --query "CertificateSummaryList[?DomainName=='*.alpha002.assets.arda.cards'].Status" \
  --profile Alpha002-Admin

Expected: ["ISSUED"]

If still PENDING_VALIDATION after 10 minutes, verify NS delegation:

dig NS alpha002.assets.arda.cards

Should return name servers for the subdomain zone. If empty, the root zone NS delegation may not have propagated yet.

Step 3: Collect CloudFormation Export Values

The verification script needs 6 values from CloudFormation exports. Retrieve them:

# Set profile
export AWS_DEFAULT_PROFILE=Alpha002-Admin

# Get all image-related exports for the dev partition
aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'Alpha002-dev')].{Name:Name, Value:Value}" \
  --output table | grep -i image

You need these specific values:

# Bucket name
BUCKET=$(aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'API-ImageAssetBucketName')].Value" \
  --output text | grep Alpha002-dev)

# CDN domain
CDN_DOMAIN=$(aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'API-ImageCdnDomain')].Value" \
  --output text | grep Alpha002-dev)

# Presigning role ARN
PRESIGN_ROLE=$(aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'API-ImagePresignRoleArn')].Value" \
  --output text | grep Alpha002-dev)

# Signing key ID
SIGNING_KEY_ID=$(aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'API-ImageCdnSigningKeyId')].Value" \
  --output text | grep Alpha002-dev)

# Signing key secret ARN
SIGNING_KEY_SECRET=$(aws cloudformation list-exports \
  --query "Exports[?contains(Name, 'API-ImageCdnSigningKeySecretArn')].Value" \
  --output text | grep Alpha002-dev)

# Use any valid tenant UUID for testing
TENANT_ID="1fa48bf2-3ef9-4d08-8858-29e71504a1ed"

Step 4: Run Verification Script

npx ts-node tools/verify-image-cdn.ts \
  --bucket "$BUCKET" \
  --cdn-domain "$CDN_DOMAIN" \
  --presign-role-arn "$PRESIGN_ROLE" \
  --signing-key-id "$SIGNING_KEY_ID" \
  --signing-key-secret-arn "$SIGNING_KEY_SECRET" \
  --tenant-id "$TENANT_ID"

Expected output: All 8 checks pass (PASS), exit code 0.

The script performs:

1. Assumes the presigning role
2. Generates a presigned POST form
3. Uploads a test JPEG to S3
4. Verifies the object via HeadObject
5. Confirms CloudFront returns 403 without cookies
6. Retrieves the signing private key from Secrets Manager
7. Confirms CloudFront returns 200 with valid signed cookies
8. Confirms CloudFront returns 403 with wrong-tenant cookies
9. Cleans up the test object

Step 5: Manual Verification (Optional)

If you want to manually verify CDN access:

# Should return 403
curl -s -o /dev/null -w "%{http_code}" \
  "https://${CDN_DOMAIN}/${TENANT_ID}/images/any-image.jpg"

# Check the CloudFront distribution in AWS Console:
# - Origin: S3 bucket with OAC
# - Behavior: HTTPS-only, GET/HEAD, trusted key groups
# - Custom domain: dev.alpha002.assets.arda.cards (if cert issued)

Error Handling and Rollback

deploy-root.sh failure

Symptoms: CDK bootstrap or deploy fails with IAM/permissions error.

Diagnosis:

# Check if the RootConfiguration stack exists
aws cloudformation describe-stacks \
  --stack-name RootConfiguration \
  --profile Admin-PlatformRoot 2>&1

Rollback: The root stack is idempotent. Fix the error and re-run ./deploy-root.sh. If the stack is in ROLLBACK_COMPLETE:

aws cloudformation delete-stack \
  --stack-name RootConfiguration \
  --profile Admin-PlatformRoot
# Wait for deletion, then re-run deploy-root.sh

Impact of not rolling back: No downstream infrastructure can deploy the assets subdomain zone. All subsequent steps are blocked.

amm.sh infrastructure step failure

Symptoms: InfrastructureIngress stack fails to update.

Diagnosis:

aws cloudformation describe-stack-events \
  --stack-name Alpha002-Ingress \
  --profile Alpha002-Admin \
  --query "StackEvents[?ResourceStatus=='CREATE_FAILED' || ResourceStatus=='UPDATE_FAILED']" \
  --output table

Rollback: CloudFormation automatically rolls back failed updates. The existing infrastructure remains functional. Fix the code, re-run amm.sh. Common causes:

• Root zone doesn’t exist yet → run deploy-root.sh first
• NS delegation Lambda fails → check CloudWatch logs for WriteAssetsDelegationNsRecords
• ACM validation timeout → DNS propagation delay, wait and retry

Impact of not rolling back: Automatic CFN rollback preserves existing infrastructure. Existing io/app/auth domains are unaffected.

amm.sh partition step failure

Symptoms: ImageStorageStack fails to create.

Diagnosis:

aws cloudformation describe-stack-events \
  --stack-name Alpha002-dev-ImageStorage \
  --profile Alpha002-Admin \
  --query "StackEvents[?ResourceStatus=='CREATE_FAILED']" \
  --output table

Rollback: For a new stack (CREATE_FAILED), CloudFormation automatically deletes all created resources. For an update (UPDATE_FAILED), it rolls back to the previous state.

Manual cleanup (only if automatic rollback fails):

# Delete the failed stack
aws cloudformation delete-stack \
  --stack-name Alpha002-dev-ImageStorage \
  --profile Alpha002-Admin
# Wait for DELETE_COMPLETE, then re-run amm.sh

Common causes:

• Signing key Lambda fails → check CloudWatch log group /aws/lambda/Alpha002-dev-ImageStorage-SigningKeyGroup*
• S3 bucket name conflict → bucket name already exists globally (unlikely with fqn prefix)
• CloudFront distribution creation timeout → retry (CF distributions take 5-15 minutes to deploy)

Impact of not rolling back: Other partition stacks (BulkStores, Ingress, DB, etc.) are unaffected. Only the new ImageStorageStack is impacted.

verify-image-cdn.ts failure

Symptoms: Script exits with code 1; one or more checks show FAIL.

Diagnosis by check:

Check	FAIL means	Next step
AssumeRole	Pod role can’t assume presigning role	Check IAM trust policy in the presigning role
Presigned POST	SDK error generating form	Check presigning role has s3:PutObject on the bucket
S3 upload (HTTP != 204)	POST rejected by S3	Check bucket CORS, presigned form fields, bucket policy
HeadObject	Object not found or wrong metadata	Upload failed silently; check S3 bucket directly
403 without cookies	Expected — if this fails (200), trusted key groups are not configured	Check CloudFront distribution behavior in console
200 with cookies	Cookies rejected	Check signing key ID matches CloudFront key group; check cookie policy Resource field
403 wrong tenant	Expected — if this fails (200), tenant isolation is broken	Check cookie policy scopes to /<tenantId>/*

Rollback: The script is read-only (except for the test upload which it cleans up). No rollback needed. Fix the infrastructure issue and re-run the script.

If cleanup fails (test object remains in S3):

aws s3 rm "s3://${BUCKET}/${TENANT_ID}/images/test-*.jpg" \
  --profile Alpha002-Admin

Troubleshooting

Symptom	Cause	Fix
amm.sh fails on ImageStorageStack	Cross-stack import not found	Verify infra step completed first; check alpha002.assets.arda.cards zone exists
ACM cert stuck in PENDING_VALIDATION	NS delegation not propagated	Wait up to 48h; verify dig NS alpha002.assets.arda.cards returns records
verify-image-cdn exits with failures	Various	See “Diagnosis by check” table above
403 even with cookies	Cookie scoping mismatch	Verify the Resource field in the signed policy matches the CDN domain
AssumeRole fails	Pod role ARN mismatch	Verify clientRoleArn in partition.ts matches the EKS pod role
CloudFront returns 502/504	Origin access issue	Verify OAC is configured; check S3 bucket policy allows CloudFront service principal
Lambda custom resource timeout	Key generation takes too long	Check Lambda memory (256MB) and timeout (3min); check Secrets Manager endpoint reachability

Exit Criteria (Task 2.18)

• Root zone deployed (assets.arda.cards)
• Alpha002 infra deployed (subdomain zone + ACM cert ISSUED)
• Alpha002/dev partition deployed (ImageStorageStack)
• 6 CloudFormation exports visible
• verify-image-cdn.ts exits with code 0