DevOps Terraform AWS Production

Troubleshooting Master Guide

Interview preparation guide for DevOps/Cloud Engineers with troubleshooting methodology.
1. Terraform Apply Succeeded But Resources Not
Created
Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check AWS account, region, state file, CloudTrail, provider configuration. Verify resources exist in
state using 'terraform state list'. Review conditional logic and workspaces. Root causes include
wrong account, wrong region, state drift, or skipped resources.

2. Terraform Plan Shows Unexpected Changes

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check manual console changes, variable changes, provider version updates, module updates, and
drift. Run terraform plan and compare with current infrastructure.

3. Terraform State File Corrupted

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Restore from S3 versioning backup, validate state using terraform state pull, compare backup
versions, and import missing resources if required.

4. Infrastructure Deployment Failed Halfway

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Review apply logs, CloudTrail events, resource dependencies, quotas, and partial resources. Fix
root cause and re-run apply.

5. Terraform Created Duplicate Resources

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check missing state entries, wrong workspace, count/for_each changes, backend issues, and
manual resource creation.

6. Terraform Apply Stuck

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Review AWS API throttling, service quotas, resource dependencies, VPC creation, NAT gateways,
and provider debug logs.

7. State Locking Issue

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check DynamoDB lock table, stale lock records, pipeline failures, and concurrent executions. Use
force-unlock only after validation.

8. EC2 Running But Application Inaccessible

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check application service, SGs, NACLs, route tables, target groups, health checks, and application
logs.

9. SSH Access Failure

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Verify SG port 22, NACLs, route tables, instance health, key pair, disk space, and CPU utilization.

10. CPU Utilization 100%

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Use top, htop, ps, CloudWatch metrics. Investigate memory leaks, traffic spikes, or runaway
processes.

11. Disk Full

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Run df -h and du -sh. Check logs, temp files, Docker images, and extend EBS if necessary.

12. Load Balancer Health Checks Failing

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Validate path, port, application status, SGs, and target group configuration.

13. RDS Connectivity Failure

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Verify endpoint, port, SGs, subnet groups, route tables, DNS resolution, and credentials.

14. EKS Nodes Not Joining

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check IAM node role, bootstrap logs, endpoint access, security groups, and node group
configuration.

15. Pods Cannot Pull Images From ECR

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Verify ECR permissions, image tag, network access, node IAM role, and repository policies.

16. AWS Bill Increased

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Analyze Cost Explorer, Trusted Advisor, tags, recently created resources, NAT Gateway traffic, and
oversized EC2 instances.

17. Production Outage After Terraform Apply

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Identify recent changes, compare plan output, inspect networking and IAM modifications, rollback
safely if needed.

18. Infrastructure Drift

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Run terraform plan, compare actual resources with code, import resources, and remove manual
changes.

19. Route53 DNS Failure

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Check hosted zone, record type, TTL, nameservers, DNS propagation, and health checks.

20. VPC Peering Traffic Not Flowing

Step 1: Check recent changes.
Step 2: Review monitoring and logs.
Step 3: Verify AWS resources.
Step 4: Check IAM permissions.
Step 5: Validate networking.
Step 6: Verify Terraform state/backend.
Step 7: Fix root cause and re-test.
Verify route tables, SGs, NACLs, and ensure CIDR ranges do not overlap.
Common Commands
Terraform: state list, state pull, import, plan, apply, force-unlock.
Linux: top, htop, df -h, free -m, journalctl, netstat.
AWS: CloudWatch, CloudTrail, Cost Explorer, VPC Flow Logs.