Learnings

1. CDK Provider Framework Handler Contract

Lesson: CDK’s custom_resources.Provider wraps the Lambda handler. The handler must return { PhysicalResourceId, Data } — the framework sends the CloudFormation response. If the handler sends the response directly via HTTPS (raw CloudFormation pattern), the Provider sends its own empty response, losing all Data attributes.

Impact: Three failed deployments before identifying the root cause. The error message (Vendor response doesn't contain PublicKeyPem attribute) was misleading — the Lambda logs showed the data was generated correctly.

Recommendation: Document the Provider handler contract in the knowledge base. All future inline Lambdas must follow the return pattern.

2. Cross-Stack OAC Creates Circular Dependencies

Lesson: S3BucketOrigin.withOriginAccessControl() auto-adds a bucket policy referencing the CloudFront distribution ARN. When bucket and distribution are in different stacks, this creates a cyclic CloudFormation reference.

Impact: Required architectural change during implementation — ImageAssetBucket moved from BulkStoresStack to ImageStorageStack (PD-04). This was actually a better design: bucket, CDN, and signing keys form a cohesive unit with shared lifecycle.

Recommendation: Always co-locate S3 buckets with their CloudFront distributions when using OAC.

3. Route53 recordName Is Zone-Relative

Lesson: r53.ARecord’s recordName is appended to the zone name. Passing the full FQDN causes double-qualification.

Impact: Failed deployment with RRSet not permitted in zone.

Recommendation: Use only the subdomain prefix (e.g., locator.id) as recordName, never the full FQDN.

4. Ingress Stack Exports Parent Domain, Not Zone Domain

Lesson: InfrastructureIngress exports domainName as the parent domain (assets.arda.cards) rather than the actual zone name (alpha002.assets.arda.cards). This is consistent with io/app/auth but breaks zone-relative DNS record creation.

Impact: Failed deployment — Route53 tried to create a record in the wrong zone.

Recommendation: When adding new domain families that use zone-relative naming, export the zone’s actual domain name from the Route53HostedZone.built.domainName property.

5. RETAIN Resources Require Manual Cleanup

Lesson: Resources with RemovalPolicy.RETAIN survive stack rollback. Each failed deployment creates orphans (S3 bucket, Secrets Manager secret, CloudFront PublicKey, CloudFront KeyGroup) that block subsequent deployments.

Impact: Required manual cleanup between each retry — deleting key group, public key, secret, bucket, then the stack itself, in the correct order.

Recommendation: Document the cleanup procedure (done in knowledge base). Consider adding a cleanup script for development environments.

6. Planning Documents Need Deployment Validation

Lesson: The specification described the correct architecture, but several implementation details only surfaced during actual deployment: Lambda handler pattern, DNS naming, export values.

Impact: 3 code fixes discovered during deployment (not caught by cdk synth or unit tests). CDK synth validates CloudFormation structure but not runtime behavior of custom resources or DNS resolution.

Recommendation: Include a deployment to a dev environment as an explicit task in Run 2 (which we did — task 2.18). The verification script was essential for catching the remaining issues.

7. Test Infrastructure Should Be Established Early

Lesson: Setting up Jest, path aliases, and understanding CDK assertion limitations took significant time. Several test patterns didn’t work (Match.anyValue() in arrayWith(), hasOutput("*"), snapshot hash instability).

Impact: Multiple test fix iterations during CI.

Recommendation: Establish test infrastructure and patterns in a dedicated task before writing construct code. Document CDK assertion gotchas in the knowledge base (done).