0% found this document useful (0 votes)
54 views33 pages

Senior DevOps Engineer Interview Guide

This document is a comprehensive interview guide for Senior DevOps Engineer positions, covering essential topics such as Azure cloud infrastructure, DevOps automation, Kubernetes, CI/CD pipelines, and security. It includes categorized questions ranging from basic to advanced levels, along with detailed answers and strategies for various scenarios. The guide aims to prepare candidates for interviews by testing their knowledge and problem-solving skills in real-world DevOps challenges.

Uploaded by

M.Hammad
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
54 views33 pages

Senior DevOps Engineer Interview Guide

This document is a comprehensive interview guide for Senior DevOps Engineer positions, covering essential topics such as Azure cloud infrastructure, DevOps automation, Kubernetes, CI/CD pipelines, and security. It includes categorized questions ranging from basic to advanced levels, along with detailed answers and strategies for various scenarios. The guide aims to prepare candidates for interviews by testing their knowledge and problem-solving skills in real-world DevOps challenges.

Uploaded by

M.Hammad
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Senior DevOps Engineer

Interview Questions & Answers

5+ Years Experience Level

Organized by Category: Basic to Advanced

Position Overview
This comprehensive interview guide covers Azure cloud infrastructure, DevOps automation, Kubernetes, CI/CD
pipelines, Infrastructure as Code, security, and operational excellence. Questions are organized by category
and difficulty level, progressing from foundational knowledge to complex scenario-based challenges.
Category 1: Azure Cloud Infrastructure & Services

Q1. What are the key differences between Azure App Service, Azure Container Apps, and
Azure Kubernetes Service? When would you choose each?
Difficulty: Basic

Q2. Explain the purpose and configuration of Azure Application Gateway, Front Door, and
WAF. How do they work together?
Difficulty: Basic to Intermediate

Q3. How do you implement high availability for a multi-tier application in Azure?
Difficulty: Intermediate

Q4. Describe your approach to designing a disaster recovery solution in Azure for a
mission-critical application with RPO of 1 hour and RTO of 4 hours.
Difficulty: Intermediate to Advanced

Q5. You have an AKS cluster experiencing intermittent pod failures. Walk me through your
troubleshooting process using Azure Monitor, kubectl, and Azure CLI.
Difficulty: Advanced

Q6. Design a multi-region Azure architecture with automatic failover that handles both
stateless and stateful workloads. Include network topology, data replication, and traffic
management.
Difficulty: Advanced

Category 2: Infrastructure as Code (IaC)

Q7. What are the advantages of using Bicep over ARM templates? Provide a code example
of a basic Azure resource deployment using Bicep.
Difficulty: Basic

Q8. Explain Terraform state management. How do you handle state in a team environment,
and what are the risks of state file corruption?
Difficulty: Intermediate

Q9. How do you structure Terraform code for multiple environments (dev, staging, prod)
while maintaining DRY principles?
Difficulty: Intermediate

Q10. Compare Terraform modules vs. Bicep modules. What are the best practices for
creating reusable, maintainable IaC components?
Difficulty: Intermediate to Advanced

Q11. You need to migrate existing Azure infrastructure (created manually) to IaC. Describe
your strategy, including tools and approaches for state import and validation.
Difficulty: Advanced

Q12. Design an IaC pipeline that supports feature branch deployments, automated testing
(what-if/plan), policy compliance checking, and safe production rollouts with approval
gates.
Difficulty: Advanced

Category 3: CI/CD Pipelines & Automation

Q13. Explain the difference between continuous integration, continuous delivery, and
continuous deployment.
Difficulty: Basic

Q14. What are GitHub Actions workflows? Write a basic workflow that builds a Docker
image and pushes it to Azure Container Registry.
Difficulty: Basic to Intermediate

Q15. How do you implement secret management in CI/CD pipelines across GitHub
Actions, Azure DevOps, and Jenkins?
Difficulty: Intermediate

Q16. Describe strategies for zero-downtime deployments in Kubernetes (blue-green,


canary, rolling updates). How would you implement canary deployments in AKS?
Difficulty: Intermediate to Advanced

Q17. Your Jenkins pipeline is slow (30+ minutes). Analyze potential bottlenecks and
propose optimization strategies including parallelization, caching, and artifact
management.
Difficulty: Advanced

Q18. Design a comprehensive CI/CD strategy for a microservices architecture with 15+
services, including build optimization, test automation, deployment orchestration, and
rollback procedures.
Difficulty: Advanced

Category 4: Containers & Kubernetes

Q19. Explain the difference between a Docker image and a container. What is a Dockerfile
and what are multi-stage builds?
Difficulty: Basic

Q20. What are Kubernetes pods, deployments, services, and ingress? How do they relate
to each other?
Difficulty: Basic to Intermediate
Q21. How do you implement resource limits and requests in Kubernetes? What happens
when a pod exceeds its limits?
Difficulty: Intermediate

Q22. Explain Kubernetes RBAC. How would you set up role-based access for different
teams working on the same AKS cluster?
Difficulty: Intermediate

Q23. A pod is stuck in 'CrashLoopBackOff' state. Walk through your debugging process
including log analysis, describe pod, events, and common causes.
Difficulty: Intermediate to Advanced

Q24. Design a production-grade AKS cluster configuration including node pools, network
policies, pod security policies, monitoring, logging, and disaster recovery considerations.
Difficulty: Advanced

Q25. Explain Helm charts. How do you manage Helm releases across multiple
environments? Discuss Helm vs. GitOps approaches (ArgoCD/Flux).
Difficulty: Advanced

Category 5: Security, Compliance & Governance

Q26. What is Azure RBAC and how does it differ from Kubernetes RBAC? Provide
examples of common role assignments.
Difficulty: Basic to Intermediate

Q27. How do you securely manage secrets in Azure? Compare Key Vault, Managed
Identities, and Azure DevOps Variable Groups.
Difficulty: Intermediate

Q28. Explain network security groups (NSGs) vs. Application Security Groups (ASGs) vs.
Azure Firewall. When would you use each?
Difficulty: Intermediate

Q29. Your organization needs SOC 2, ISO 27001, and PCI DSS compliance. What Azure
services and configurations would you implement to meet these requirements?
Difficulty: Intermediate to Advanced

Q30. Describe your approach to implementing security scanning in CI/CD pipelines


including container image scanning, secrets detection, IaC security analysis, and
dependency vulnerability scanning.
Difficulty: Advanced

Q31. Design a comprehensive security architecture for a multi-tenant AKS environment


that includes network isolation, secret management, pod security standards, audit
logging, and runtime threat detection.
Difficulty: Advanced

Category 6: Monitoring, Logging & Observability

Q32. What is the difference between metrics, logs, and traces? Why are all three important
for observability?
Difficulty: Basic

Q33. Explain Azure Monitor, Log Analytics, and Application Insights. How do they work
together?
Difficulty: Basic to Intermediate

Q34. How would you implement centralized logging for a microservices application
running in AKS?
Difficulty: Intermediate

Q35. Design an alerting strategy for a production system. What metrics would you
monitor, what thresholds would you set, and how would you prevent alert fatigue?
Difficulty: Intermediate to Advanced

Q36. Compare Datadog, New Relic, Prometheus/Grafana, and Azure Monitor. What are the
pros and cons of each for a large-scale Azure deployment?
Difficulty: Advanced

Q37. Your production application is experiencing intermittent latency spikes. Design a


comprehensive investigation approach using distributed tracing, log correlation, metric
analysis, and APM tools.
Difficulty: Advanced

Category 7: Scripting & Automation

Q38. Write a PowerShell script that lists all Azure VMs in a subscription and exports their
name, resource group, location, and power state to a CSV file.
Difficulty: Basic to Intermediate

Q39. Write a Bash script that checks if specific pods are running in a Kubernetes
namespace and restarts them if they're in a failed state.
Difficulty: Intermediate

Q40. Create a Python script that automates the backup of Azure Key Vault secrets to a
secure storage location with versioning.
Difficulty: Intermediate to Advanced

Q41. Develop a comprehensive automation solution that performs nightly health checks of
your infrastructure, generates reports, sends alerts for issues, and creates tickets for
manual intervention items.
Difficulty: Advanced

Category 8: Troubleshooting & Problem Solving

Q42. A developer reports that their application deployment to AKS failed. What
information would you gather first, and what are the most common causes of deployment
failures?
Difficulty: Basic to Intermediate

Q43. Your CI/CD pipeline suddenly started failing with authentication errors to Azure. How
do you troubleshoot this issue?
Difficulty: Intermediate

Q44. Production website is down. Walk through your incident response process from
detection to resolution to post-mortem.
Difficulty: Intermediate to Advanced

Q45. After a deployment, application performance has degraded by 40%. Describe your
systematic approach to identifying the root cause across infrastructure, application, and
configuration layers.
Difficulty: Advanced

Q46. You discover that your AKS cluster is running out of IP addresses in its subnet. This
is causing new pod creation failures. Explain the problem, immediate mitigation, and
long-term solution.
Difficulty: Advanced

Category 9: Architecture & Design

Q47. Explain the concept of Infrastructure as Code and its benefits. What are the potential
risks if not implemented properly?
Difficulty: Basic to Intermediate

Q48. Design a scalable architecture for a web application expecting 10,000 concurrent
users with global distribution requirements.
Difficulty: Intermediate to Advanced

Q49. Your company is acquiring another company with a legacy infrastructure. Design a
migration and integration strategy for consolidating their workloads into your Azure
environment.
Difficulty: Advanced

Q50. Design a complete DevOps platform from scratch for an organization with 50+
developers, including version control strategy, branching model, CI/CD pipelines,
environment management, and operational procedures.
Difficulty: Advanced

Category 10: Leadership & Collaboration

Q51. How do you mentor junior DevOps engineers? What topics do you prioritize in their
learning path?
Difficulty: Intermediate

Q52. Describe a time when you had to convince developers to change their deployment
process. How did you handle resistance?
Difficulty: Intermediate to Advanced

Q53. How do you balance the need for innovation and experimentation with stability and
reliability in production systems?
Difficulty: Advanced

Q54. Your team is struggling with too many production incidents. How would you
implement a culture of operational excellence and continuous improvement?
Difficulty: Advanced
ANSWERS

Category 1: Azure Cloud Infrastructure & Services


Answer 1:
Azure App Service: A fully managed PaaS for hosting web applications, APIs, and mobile backends. Best
for traditional web apps with minimal infrastructure management needs. Supports .NET, Java, Python,
[Link]. Provides built-in scaling, SSL, custom domains. Use when you want zero infrastructure
management and quick deployment.

Azure Container Apps (ACA): Serverless container platform built on Kubernetes. Best for microservices,
event-driven apps, and jobs that need container flexibility without K8s complexity. Features include
auto-scaling to zero, KEDA integration, Dapr support. Use when you want containers without managing
Kubernetes.

Azure Kubernetes Service (AKS): Fully managed Kubernetes service. Best for complex microservices
architectures, when you need full Kubernetes control, custom networking, advanced scaling strategies.
Use when you need complete control over orchestration, have existing K8s expertise, or require specific
Kubernetes features.

Decision criteria: App Service for simple web apps, ACA for containerized microservices with less
operational overhead, AKS for enterprise-grade container orchestration with full control.

Answer 2:
Azure Application Gateway: Regional Layer-7 load balancer with WAF capabilities. Provides SSL
termination, URL-based routing, cookie-based session affinity, health probes. Works within a region/VNet.
Configuration includes: frontend IP, listeners (port 80/443), backend pools (target servers), routing rules,
health probes.

Azure Front Door: Global Layer-7 load balancer and CDN. Provides global routing, intelligent health
probes, URL-based routing across regions, SSL offloading, caching. Works across multiple regions. Key
features: anycast networking, backend health monitoring, custom rules engine.

Web Application Firewall (WAF): Protects web applications from common exploits (OWASP Top 10).
Available on both Application Gateway and Front Door. Includes managed rule sets, custom rules,
geo-filtering, rate limiting, bot protection.

Together: Front Door serves as global entry point with WAF for DDoS/attack protection → routes to
regional Application Gateways → Application Gateways distribute to backend VMs/containers. This
provides defense in depth, global availability, and regional optimization.

Answer 3:
For a multi-tier application (web, app, database):

1. Availability Zones: Deploy resources across 3 availability zones within a region for 99.99% SLA.

2. Load Balancing:
- Azure Front Door for global traffic distribution
- Application Gateway for regional web tier load balancing
- Internal Load Balancer for app tier distribution

3. Compute Redundancy:
- Use Virtual Machine Scale Sets (VMSS) with min 2-3 instances across zones
- Or AKS with multiple node pools across availability zones
- Configure auto-scaling based on CPU/memory/custom metrics

4. Data Tier:
- Azure SQL Database with zone-redundant configuration or Geo-replication
- Or Azure Cosmos DB with multi-region writes
- Regular automated backups with point-in-time restore

5. Storage:
- Use ZRS (Zone-Redundant Storage) or GRS (Geo-Redundant Storage)
- Separate storage accounts for different workloads

6. Monitoring:
- Azure Monitor with availability alerts
- Application Insights for application health
- Health probes on all load balancers

7. Design patterns: Circuit breaker, retry logic, timeout patterns in application code.

Answer 4:
Requirements Analysis: RPO 1 hour = max 1 hour data loss. RTO 4 hours = max 4 hours downtime.

Architecture Components:

1. Primary Region (East US):


- Production environment with all services
- Zone redundant deployments

2. Secondary Region (West US):


- Warm standby with core services
- Reduced capacity (can scale up during DR)

3. Data Replication Strategy:


- Database: Active geo-replication (continuous replication, <1 min RPO possible) or auto-failover groups
- Storage: GRS (Geo-Redundant Storage) or GZRS with last sync time monitoring
- Configuration: Store all IaC in Git, deploy identically to both regions

4. Traffic Management:
- Azure Traffic Manager or Front Door with Priority routing
- Health probe monitors on primary region
- Automatic failover to secondary when primary fails

5. Backup Strategy:
- Hourly transaction log backups (meets RPO)
- Daily full backups retained 30 days
- Point-in-time restore capability

6. DR Procedures:
- Documented runbook for failover process
- Automated failover scripts for RTO compliance
- Regular DR drills (quarterly) with actual failover testing
- Monitoring and alerting for replication lag

7. Failover Process:
- Detection: Automated monitoring detects primary failure
- Decision: On-call engineer reviews and approves failover
- Execution: Automated scripts trigger database failover and traffic redirect (30-60 min)
- Verification: Health checks confirm secondary region operational
- Communication: Status page updated, stakeholders notified

8. Failback:
- Validate primary region fully recovered
- Reverse replicate data from secondary to primary
- Scheduled maintenance window for traffic cutover

Cost Optimization: Use Azure Site Recovery for VM replication to reduce secondary region costs.

Answer 5:
Systematic Troubleshooting Approach:

Step 1 - Initial Assessment:


kubectl get pods -n <namespace> --watch - Observe failure pattern
kubectl describe pod <pod-name> -n <namespace> - Check events, conditions, and state transitions

Step 2 - Log Analysis:


kubectl logs <pod-name> -n <namespace> - Current container logs
kubectl logs <pod-name> -n <namespace> --previous - Previous crashed container logs
kubectl logs <pod-name> -n <namespace> -c <container-name> - Multi-container pod specific logs

Step 3 - Resource Issues:


kubectl top pods -n <namespace> - CPU/Memory usage
kubectl top nodes - Node resource availability
kubectl describe node <node-name> - Check for node pressure (memory, disk, PID)
Check if pods are getting OOMKilled or hitting resource limits

Step 4 - Configuration Validation:


kubectl get pod <pod-name> -n <namespace> -o yaml - Review full pod spec
Verify: image pull secrets, configmaps, secrets mounting, environment variables
kubectl get events -n <namespace> --sort-by='.lastTimestamp' - Recent cluster events

Step 5 - Network Connectivity:


kubectl exec -it <pod-name> -n <namespace> -- sh - Interactive debugging
Test connectivity to dependencies: databases, APIs, storage
Check DNS resolution: nslookup <service-name>

Step 6 - Azure Monitor Integration:


az aks show -n <cluster-name> -g <resource-group> - Cluster health
Azure Portal → AKS cluster → Insights → Containers → Filter failing pods
Check Container Insights for performance metrics and log analytics
Query Log Analytics: ContainerLog | where PodName contains '<pod-name>'

Step 7 - Common Causes Investigation:


- Image Pull Failures: Check image name, registry credentials, network policies
- CrashLoopBackOff: Application startup errors, missing dependencies
- Resource Exhaustion: OOMKilled, insufficient cluster capacity
- Liveness/Readiness Probe Failures: Aggressive timeouts, wrong endpoints
- ConfigMap/Secret Issues: Missing or incorrect configuration
- Node Issues: Node pressure, cordoned nodes, node failures

Step 8 - Correlation Analysis:


Check if failures started after: deployment, scaling event, infrastructure change, time-based patterns
Review recent changes in Git, CI/CD pipeline logs, Azure Activity Log

Step 9 - Resolution:
Based on findings, apply fix: update deployment, adjust resources, fix configuration, restart services
Monitor recovery and document root cause

Step 10 - Prevention:
Implement monitoring/alerting for similar issues, update runbooks, improve health checks

Answer 6:
Architecture Overview: Active-Active multi-region deployment across East US and West Europe.

1. Global Layer (Traffic Management):


- Azure Front Door: Global entry point with WAF enabled
- Routing Policy: Performance-based routing (lowest latency) with health probes
- Health Monitoring: 30-second health probe intervals on both regions
- Automatic Failover: If region fails health checks, traffic redirects within 60 seconds

2. Regional Architecture (Each Region):

Compute - Stateless (Web/API Tier):


- AKS cluster with 3 node pools across availability zones
- Horizontal Pod Autoscaler (HPA) based on CPU/custom metrics
- Cluster Autoscaler for node scaling
- All stateless services deployed identically via GitOps
- No local state - externalize all session data

Compute - Stateful (Processing/Workers):


- Separate node pool with StatefulSets
- Azure Disk storage for persistent volumes
- Active in primary region, standby in secondary

3. Data Layer:

Databases:
- Azure SQL: Auto-failover groups with async replication
- Primary in East US, readable secondary in West Europe
- Automatic failover on primary failure
- Connection string uses failover group endpoint
- Cosmos DB: Multi-region writes enabled
- Strong consistency for critical data
- Session consistency for user data
- Automatic failover with 99.999% SLA

Cache:
- Azure Cache for Redis with geo-replication
- Primary in each region for local reads
- Eventual consistency acceptable for cache

Storage:
- Azure Storage with GZRS (Geo-Zone-Redundant Storage)
- Blob storage for media/files replicated across regions
- Azure Files for shared file systems

Message Queue:
- Azure Service Bus Premium with geo-disaster recovery
- Metadata replication across regions
- Messages replicated to secondary for DR

4. Network Topology:

Hub-Spoke Architecture per Region:


- Hub VNet: Shared services (Azure Firewall, VPN Gateway, monitoring)
- Spoke VNets: Workload isolation (AKS cluster, databases)
- VNet peering within region for hub-spoke connectivity

Cross-Region Connectivity:
- Global VNet peering between regional hubs
- ExpressRoute for private connectivity (optional)
- Azure Private Link for PaaS services

Network Security:
- NSGs on all subnets
- Azure Firewall for egress filtering
- Private endpoints for all PaaS services
- Network policies in AKS for pod-to-pod security

5. Data Replication Strategy:

Synchronous (Strong Consistency):


- Financial transactions
- User authentication data
- Inventory management

Asynchronous (Eventual Consistency):


- User profiles
- Analytics data
- Logs and metrics

6. Failover Scenarios:

Regional Failure (Complete):


1. Front Door detects failed health probes (60 seconds)
2. Traffic automatically redirected to healthy region
3. Database auto-failover triggered (SQL, Cosmos)
4. Stateful workloads scaled up in secondary region
5. Message queues continue processing
6. Total failover time: 2-3 minutes

Partial Failure (Service-level):


- Front Door routes around failed services
- AKS self-healing restarts failed pods
- Circuit breakers prevent cascade failures

Database Failure:
- Auto-failover group promotes secondary
- Connection strings automatically reroute
- Application retry logic handles brief interruption

7. Monitoring & Alerting:


- Azure Monitor for unified observability
- Application Insights for distributed tracing
- Custom alerts for replication lag, health probe failures
- PagerDuty integration for on-call escalation

8. Deployment Strategy:
- GitOps with Flux/ArgoCD
- Blue-green deployments per region
- Staged rollout: Canary → Region 1 → Region 2
- Automated rollback on health check failures

9. Cost Optimization:
- Reserved Instances for baseline compute
- Spot instances for batch workloads
- Auto-scaling to handle traffic variations
- Azure Hybrid Benefit where applicable

10. Testing:
- Monthly DR drills with actual failover
- Chaos engineering with Azure Chaos Studio
- Load testing each region independently
- Quarterly full DR simulation
Category 2: Infrastructure as Code (IaC)
Answer 7:
Advantages of Bicep over ARM Templates:

1. Simpler Syntax: Human-readable syntax vs. verbose JSON


2. Better Tooling: IntelliSense, validation, and VS Code extension
3. Type Safety: Strong typing prevents configuration errors
4. Modularity: Easy module system for reusability
5. No State Management: Azure Resource Manager handles state (unlike Terraform)
6. Day-0 Support: Immediate support for new Azure resources
7. Transparent Compilation: Converts to ARM JSON automatically

Example - Deploy Storage Account with Bicep:

// [Link]
param storageAccountName string = 'mystorageacct${uniqueString(resourceGroup().id)}'
param location string = resourceGroup().location
param sku string = 'Standard_LRS'

@allowed(['Hot', 'Cool'])
param accessTier string = 'Hot'

resource storageAccount '[Link]/storageAccounts@2023-01-01' = {


name: storageAccountName
location: location
sku: {
name: sku
}
kind: 'StorageV2'
properties: {
accessTier: accessTier
minimumTlsVersion: 'TLS1_2'
supportsHttpsTrafficOnly: true
allowBlobPublicAccess: false
}
}

output storageAccountId string = [Link]


output blobEndpoint string = [Link]

Deployment:
az deployment group create \
--resource-group myResourceGroup \
--template-file [Link] \
--parameters sku=Standard_GRS accessTier=Cool

This demonstrates: parameters with defaults, type safety with @allowed decorator, secure defaults
(HTTPS only, no public blob access), and outputs for use in other modules.

Answer 8:
Terraform State Management:

What is Terraform State?


State file ([Link]) maps real-world resources to Terraform configuration. Contains:
- Resource metadata and current configuration
- Resource dependencies
- Provider information
- Sensitive data (passwords, keys) - should be encrypted

Team Environment State Management:

1. Remote State Backend (Azure Storage):


terraform {
backend "azurerm" {
resource_group_name = "terraform-state-rg"
storage_account_name = "tfstatestorage"
container_name = "tfstate"
key = "[Link]"
}
}

2. State Locking:
- Azure Storage provides automatic lease-based locking
- Prevents concurrent modifications
- If locked, other operations wait or fail
- Force unlock only if process crashed: terraform force-unlock <LOCK_ID>

3. State File Security:


- Enable encryption at rest on storage account
- Use Azure RBAC to control access
- Enable soft delete for recovery
- Use managed identity for authentication (avoid keys)
- Never commit state files to Git

4. State File Organization:


- Separate state files per environment: [Link], [Link], [Link]
- Separate by component for large infrastructure: [Link], [Link]
- Use workspaces for simpler scenarios: terraform workspace new dev

Risks of State File Corruption:

1. Data Loss:
- Lose track of managed resources
- Can't properly destroy resources
- Drift between actual and tracked state

2. Inconsistency:
- Terraform doesn't know current resource state
- May try to recreate existing resources
- Potential for duplicate resources or conflicts

3. Security Exposure:
- Corrupted state may leak sensitive data
- Could expose passwords, keys, connection strings
4. Recovery Difficulty:
- Manual reconciliation required
- May need to import existing resources
- Time-consuming and error-prone

Protection Strategies:

1. Enable Versioning: Azure Storage blob versioning for rollback capability


2. Backup State: Automated daily backups to separate storage
3. State File Validation: Run terraform validate before apply
4. CI/CD Integration: Automated state validation in pipelines
5. Access Controls: Principle of least privilege for state access
6. Audit Logs: Enable Azure Storage logs for state file access tracking

Recovery from Corruption:


1. Restore from backup/version history
2. If not recoverable, use terraform import to rebuild state
3. Compare desired config with actual resources in Azure
4. Incrementally import resources: terraform import azurerm_resource_group.example
/subscriptions/.../resourceGroups/myRG

Answer 9:
Multi-Environment Terraform Structure (DRY Principles):

Directory Structure:

terraform/
■■■ modules/
■ ■■■ aks-cluster/
■ ■ ■■■ [Link]
■ ■ ■■■ [Link]
■ ■ ■■■ [Link]
■ ■■■ networking/
■ ■■■ database/
■■■ environments/
■ ■■■ dev/
■ ■ ■■■ [Link]
■ ■ ■■■ [Link]
■ ■ ■■■ [Link]
■ ■■■ staging/
■ ■■■ prod/
■■■ shared/
■ ■■■ [Link] # Common variables
■ ■■■ [Link] # Common locals
■■■ [Link]

1. Reusable Modules (modules/aks-cluster/[Link]):

variable "cluster_name" {}
variable "location" {}
variable "node_count" {}
variable "node_size" {}
variable "environment" {}
resource "azurerm_kubernetes_cluster" "aks" {
name = "${var.cluster_name}-${[Link]}"
location = [Link]
resource_group_name = var.resource_group_name
dns_prefix = var.cluster_name

default_node_pool {
name = "default"
node_count = var.node_count
vm_size = var.node_size
}

identity {
type = "SystemAssigned"
}

tags = merge(var.common_tags, {
Environment = [Link]
})
}

2. Environment-Specific Configuration (environments/dev/[Link]):

terraform {
required_version = ">= 1.0"
backend "azurerm" {
resource_group_name = "terraform-state-rg"
storage_account_name = "tfstate"
container_name = "tfstate"
key = "[Link]"
}
}

locals {
environment = "dev"
common_tags = {
Environment = "dev"
ManagedBy = "Terraform"
CostCenter = "Engineering"
}
}

module "networking" {
source = "../../modules/networking"
environment = [Link]
location = [Link]
common_tags = local.common_tags
}

module "aks_cluster" {
source = "../../modules/aks-cluster"
cluster_name = "myapp"
environment = [Link]
location = [Link]
node_count = 2 # Dev has fewer nodes
node_size = "Standard_D2s_v3" # Dev uses smaller VMs
resource_group_name = [Link].resource_group_name
common_tags = local.common_tags
}

3. Environment Variables (environments/dev/[Link]):

location = "eastus"
subscription_id = "dev-subscription-id"
allowed_ip_ranges = ["[Link]/8"]

4. Production Differences (environments/prod/[Link]):

locals {
environment = "prod"
common_tags = {
Environment = "prod"
ManagedBy = "Terraform"
CostCenter = "Operations"
}
}

module "aks_cluster" {
source = "../../modules/aks-cluster"
cluster_name = "myapp"
environment = [Link]
location = [Link]
node_count = 5 # Production has more nodes
node_size = "Standard_D4s_v3" # Production uses larger VMs
resource_group_name = [Link].resource_group_name
common_tags = local.common_tags
}

5. Shared Configuration (shared/[Link]):

variable "location" {
description = "Azure region"
type = string
}

variable "subscription_id" {
description = "Azure subscription ID"
type = string
sensitive = true
}

Best Practices:
1. Use Modules: All reusable infrastructure in modules/
2. Environment-Specific Values: Only environment configs in environments/
3. Separate State: Different state file per environment
4. Variables: Use .tfvars files for environment-specific values
5. Locals: Compute derived values to avoid duplication
6. Outputs: Chain modules using outputs from one as inputs to another
7. Tags: Common tagging strategy with environment-specific additions
8. Versioning: Pin module versions in production for stability

Deployment Process:
cd environments/dev
terraform init
terraform plan -var-file=[Link]
terraform apply -var-file=[Link]

Answer 10:
Terraform Modules vs. Bicep Modules Comparison:

Terraform Modules:

Strengths:
- Multi-cloud support (Azure, AWS, GCP)
- Mature ecosystem with extensive public registry
- Provider-agnostic patterns
- Rich state management
- Powerful looping and conditionals

Module Example:
# modules/storage/[Link]
variable "name" { type = string }
variable "location" { type = string }
variable "account_tier" { type = string default = "Standard" }

resource "azurerm_storage_account" "storage" {


name = [Link]
location = [Link]
resource_group_name = var.resource_group_name
account_tier = var.account_tier
account_replication_type = "LRS"
}

output "storage_account_id" {
value = azurerm_storage_account.[Link]
}

# Usage
module "storage" {
source = "./modules/storage"
name = "mystorageacct"
location = "eastus"
resource_group_name = azurerm_resource_group.[Link]
}

Bicep Modules:
Strengths:
- Native Azure integration
- Day-0 support for new Azure features
- Simpler syntax
- Better Azure-specific validation
- No state file management

Module Example:
// modules/[Link]
@description('Storage account name')
param name string

@description('Location')
param location string = resourceGroup().location

@allowed(['Standard', 'Premium'])
param accountTier string = 'Standard'

resource storage '[Link]/storageAccounts@2023-01-01' = {


name: name
location: location
sku: {
name: '${accountTier}_LRS'
}
kind: 'StorageV2'
}

output storageAccountId string = [Link]

// Usage in [Link]
module storage 'modules/[Link]' = {
name: 'storageDeployment'
params: {
name: 'mystorageacct'
location: 'eastus'
accountTier: 'Standard'
}
}

Best Practices for Both:

1. Single Responsibility:
- Each module should manage one logical component
- E.g., 'networking' module, 'aks-cluster' module, not 'entire-infrastructure' module

2. Well-Defined Interfaces:
- Clear input parameters with descriptions and types
- Sensible defaults for optional parameters
- Useful outputs for module chaining

3. Versioning:
- Use semantic versioning (1.0.0, 1.1.0, 2.0.0)
- Tag releases in Git
- Document breaking changes
- Pin versions in production

4. Documentation:
- [Link] with usage examples
- Parameter descriptions
- Architecture diagrams for complex modules
- Changelog for version history

5. Testing:
- Unit tests for module logic
- Integration tests in test environments
- Validate plans before applying

6. Security:
- Mark sensitive parameters (passwords, keys)
- Enable security features by default (encryption, HTTPS only)
- Validate inputs with constraints

7. Composition Over Inheritance:


- Build complex infrastructure by composing simple modules
- Avoid overly deep module nesting

8. Resource Naming:
- Parameterize names for flexibility
- Use naming conventions consistently
- Support name prefixes/suffixes

9. Tags/Labels:
- Support common tagging as parameter
- Merge module-specific tags with common tags

10. Error Handling:


- Validate parameters at module boundary
- Provide clear error messages
- Use preconditions (Terraform) or decorators (Bicep)

Decision Criteria:
- Use Terraform: Multi-cloud, existing Terraform infrastructure, need provider ecosystem
- Use Bicep: Azure-only, want native Azure experience, simpler syntax preferred
Answer 11:
Strategy for Migrating Existing Azure Infrastructure to IaC:

Phase 1: Assessment & Planning (Week 1-2)

1. Infrastructure Discovery:
- Document all existing resources using Azure CLI:
az resource list --query "[].{Name:name, Type:type, RG:resourceGroup}" -o table
- Create inventory spreadsheet with: Resource type, Name, Dependencies, Owner, Criticality
- Identify dependencies between resources (network → compute → database)
- Map resources to applications/services

2. Choose IaC Tool:


- Select Terraform or Bicep based on team skills and requirements
- For this answer, I'll use Terraform (same principles apply to Bicep)

3. Prioritization:
- Start with non-production environments
- Begin with stateless resources (networking, storage accounts)
- Leave stateful resources (databases with data) for later
- Prioritize by: Low risk → Medium risk → High risk

Phase 2: Code Generation (Week 2-4)

1. Use Azure Export Tools:

# Export Resource Group to ARM template


az group export --name myResourceGroup --output-file [Link]

# Convert ARM to Terraform using aztfexport


aztfexport resource-group myResourceGroup

2. Generate Initial Terraform Code:


- Install aztfexport: go install [Link]/Azure/aztfexport@latest
- Run export: aztfexport rg myResourceGroup
- Review generated .tf files
- Note: Generated code will need significant cleanup

3. Manual Refinement:
- Remove auto-generated IDs and computed values
- Organize into logical files ([Link], [Link], [Link])
- Extract hardcoded values into variables
- Apply consistent naming conventions
- Add proper resource dependencies

Phase 3: State Import (Week 4-6)

1. Setup Terraform Backend:


terraform {
backend "azurerm" {
resource_group_name = "terraform-state-rg"
storage_account_name = "tfstatemigration"
container_name = "tfstate"
key = "[Link]"
}
}

2. Import Resources:

# Import VNet
terraform import azurerm_virtual_network.main /subscriptions/{subscription-id}/resourceGroups/{rg}/provid
ers/[Link]/virtualNetworks/{vnet-name}

# Import Subnet
terraform import azurerm_subnet.subnet /subscriptions/{subscription-id}/resourceGroups/{rg}/providers/Mic
[Link]/virtualNetworks/{vnet}/subnets/{subnet}

# Import AKS Cluster


terraform import azurerm_kubernetes_cluster.aks /subscriptions/{subscription-id}/resourceGroups/{rg}/prov
iders/[Link]/managedClusters/{aks-name}

3. Automated Import Script:


#!/bin/bash
# [Link]

SUBSCRIPTION_ID="your-sub-id"
RESOURCE_GROUP="myResourceGroup"

# Import all VMs


VMS=$(az vm list -g $RESOURCE_GROUP --query "[].name" -o tsv)
for VM in $VMS; do
echo "Importing VM: $VM"
terraform import "azurerm_linux_virtual_machine.vm[\"$VM\"]" \
"/subscriptions/$SUBSCRIPTION_ID/resourceGroups/$RESOURCE_GROUP/providers/[Link]
e/virtualMachines/$VM"
done

Phase 4: Validation (Week 6-8)

1. Plan Validation:
terraform plan -out=tfplan
# Expected: No changes needed
# Reality: Will likely show some differences

2. Address Drift:
- Review each difference in plan output
- Update Terraform code to match actual resource configuration
- Or update actual resources to match desired state
- Iterate until plan shows no changes

3. Validation Checklist:
- [ ] All resources imported into state
- [ ] terraform plan shows no changes
- [ ] Variables properly parameterized
- [ ] Outputs defined for key resources
- [ ] Code organized and modular
- [ ] Documentation complete

Phase 5: Testing (Week 8-10)

1. Test in Lower Environment First:


- Create identical resources in dev using Terraform
- Verify all features work correctly
- Test destroy and recreate

2. Validate No Unintended Changes:


- Run terraform apply in production (dry run first)
- Monitor for any unexpected modifications
- Have rollback plan ready

3. Test Disaster Recovery:


- Verify backups still work
- Test restoration procedures

Phase 6: Documentation & Handover (Week 10-12)

1. Create Documentation:
- Architecture diagrams
- Terraform structure explanation
- Deployment procedures
- Troubleshooting guide
- Variable reference

2. Training:
- Team walkthrough of Terraform code
- Deployment demonstrations
- Q&A sessions

3. CI/CD Integration:
- Setup automated Terraform plans on PRs
- Create deployment pipelines
- Implement approval workflows

Common Challenges & Solutions:

1. Circular Dependencies:
- Problem: Resources reference each other
- Solution: Use depends_on or break into modules

2. Immutable Resources:
- Problem: Some resources can't be updated in place
- Solution: Plan for recreation during maintenance window

3. Sensitive Data:
- Problem: Secrets in state file
- Solution: Use Key Vault references, encrypt state backend

4. Resource Naming:
- Problem: Existing names don't follow conventions
- Solution: Accept existing names initially, plan migration later
5. Partial State:
- Problem: Some resources imported, others missed
- Solution: Comprehensive resource inventory and validation

Success Metrics:
- 100% of resources managed by IaC
- Zero manual changes in portal
- All deployments via CI/CD
- terraform plan always shows accurate state
- Team comfortable with IaC workflow

Answer 12:
Comprehensive IaC Pipeline Design:

Architecture Overview:
Feature Branch → PR → Automated Validation → Approval → Deployment

Pipeline Stages:

Stage 1: PR Validation (On Pull Request)

# .github/workflows/[Link]
name: Terraform PR Validation

on:
pull_request:
branches: [main]
paths:
- 'terraform/**'

jobs:
validate:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Setup Terraform


uses: hashicorp/setup-terraform@v2
with:
terraform_version: 1.6.0

- name: Terraform Format Check


run: terraform fmt -check -recursive

- name: Terraform Init


run: terraform init -backend=false

- name: Terraform Validate


run: terraform validate

- name: TFLint
uses: terraform-linters/setup-tflint@v3
run: |
tflint --init
tflint --recursive

- name: Checkov Security Scan


uses: bridgecrewio/checkov-action@master
with:
directory: terraform/
framework: terraform
output_format: sarif
soft_fail: true

plan-dev:
runs-on: ubuntu-latest
needs: validate
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Azure Login


uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS }}

- name: Terraform Init


working-directory: terraform/environments/dev
run: terraform init

- name: Terraform Plan


working-directory: terraform/environments/dev
run: |
terraform plan -out=tfplan -input=false
terraform show -json tfplan > [Link]

- name: Policy Check (Azure Policy)


run: |
# Check against Azure Policy definitions
az policy state list --filter "ResourceType eq '[Link]/virtualMachines'"

- name: Cost Estimation


uses: infracost/actions/setup@v2
run: |
infracost breakdown --path=[Link] --format=json --out-file=[Link]
infracost comment github --path=[Link] --behavior=update

- name: Upload Plan


uses: actions/upload-artifact@v3
with:
name: tfplan-dev
path: terraform/environments/dev/tfplan

- name: Comment PR
uses: actions/github-script@v6
with:
script: |
const output = `#### Terraform Plan ■
■ Format and Validation Passed
■ Security Checks Passed
■ Cost Estimate: [View Details]
■ Review the plan artifact before merging`;
[Link]({
issue_number: [Link],
owner: [Link],
repo: [Link],
body: output
})

Stage 2: Feature Branch Deployment (Optional)

# Deploy to ephemeral environment for testing


deploy-feature:
if: contains([Link].pull_request.labels.*.name, 'deploy-preview')
runs-on: ubuntu-latest
steps:
- name: Create Ephemeral Environment
run: |
BRANCH_NAME=$(echo ${{ github.head_ref }} | sed 's/[^a-zA-Z0-9]/-/g')
terraform workspace new $BRANCH_NAME || terraform workspace select $BRANCH_NAME
terraform apply -auto-approve

- name: Comment Preview URL


run: |
URL=$(terraform output -raw preview_url)
echo "Preview environment: $URL" >> $GITHUB_STEP_SUMMARY

Stage 3: Development Deployment (On Merge to Main)

# .github/workflows/[Link]
name: Terraform Deploy

on:
push:
branches: [main]
paths:
- 'terraform/**'

jobs:
deploy-dev:
runs-on: ubuntu-latest
environment: development
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Azure Login


uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS_DEV }}
- name: Terraform Init
working-directory: terraform/environments/dev
run: terraform init

- name: Terraform Apply


working-directory: terraform/environments/dev
run: terraform apply -auto-approve

- name: Run Integration Tests


run: |
./scripts/[Link] dev

- name: Notify Team


if: always()
uses: 8398a7/action-slack@v3
with:
status: ${{ [Link] }}
text: 'Dev deployment completed'

Stage 4: Staging Deployment (With Approval)

deploy-staging:
runs-on: ubuntu-latest
needs: deploy-dev
environment:
name: staging
# GitHub Environment Protection Rules:
# - Required reviewers: DevOps Team
# - Wait timer: 0 minutes
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Azure Login


uses: azure/login@v1
with:
creds: ${{ secrets.AZURE_CREDENTIALS_STAGING }}

- name: Terraform Plan


working-directory: terraform/environments/staging
run: |
terraform init
terraform plan -out=tfplan

- name: Manual Approval Check


run: |
echo "Plan approved via GitHub Environment protection rules"

- name: Terraform Apply


working-directory: terraform/environments/staging
run: terraform apply -auto-approve tfplan

- name: Smoke Tests


run: ./scripts/[Link] staging

Stage 5: Production Deployment (With Multiple Approvals)

deploy-prod:
runs-on: ubuntu-latest
needs: deploy-staging
environment:
name: production
# GitHub Environment Protection Rules:
# - Required reviewers: 2+ from [DevOps Lead, CTO]
# - Wait timer: 30 minutes (change freeze window)
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Create Change Ticket


run: |
# Create ServiceNow/Jira change ticket
./scripts/[Link]

- name: Pre-deployment Backup


run: |
az backup protection backup-now \
--resource-group prod-rg \
--vault-name prod-backup-vault \
--container-name prod-vm-container \
--item-name prod-database

- name: Enable Maintenance Mode


run: |
kubectl annotate deployment/app maintenance="true"

- name: Terraform Apply with State Lock


working-directory: terraform/environments/prod
timeout-minutes: 30
run: |
terraform init
terraform plan -out=tfplan
terraform apply -auto-approve tfplan

- name: Health Check


run: |
./scripts/[Link] prod
if [ $? -ne 0 ]; then
echo "Health check failed, initiating rollback"
exit 1
fi

- name: Disable Maintenance Mode


if: success()
run: |
kubectl annotate deployment/app maintenance-
- name: Rollback on Failure
if: failure()
run: |
cd terraform/environments/prod
# Restore from previous state
terraform state pull > [Link]
terraform state push [Link]
terraform apply -auto-approve

- name: Update Documentation


if: success()
run: |
./scripts/[Link]

- name: Close Change Ticket


if: always()
run: |
./scripts/[Link] ${{ [Link] }}

Additional Pipeline Features:

1. State Backup:
# Run before each apply
- name: Backup Terraform State
run: |
terraform state pull > state-backup-$(date +%Y%m%d-%H%M%S).json
az storage blob upload \
--account-name tfstatebackups \
--container backups \
--file state-backup-*.json

2. Drift Detection:
# Daily scheduled job
on:
schedule:
- cron: '0 8 * * *' # Daily at 8 AM

jobs:
detect-drift:
steps:
- name: Check for Drift
run: |
terraform plan -detailed-exitcode
if [ $? -eq 2 ]; then
echo "■■ Infrastructure drift detected"
terraform show > [Link]
# Send alert to team
fi

3. Compliance Scanning:
- Checkov for Terraform security
- TFSec for additional security checks
- Azure Policy compliance validation
- OWASP dependency check

4. Documentation Generation:
- name: Generate Terraform Docs
uses: terraform-docs/gh-actions@v1
with:
working-dir: terraform/
output-file: [Link]
output-method: inject

Pipeline Governance:

1. Branch Protection: Main branch requires PR + reviews


2. Required Checks: All validation must pass before merge
3. Environment Protection: Tiered approval gates
4. Audit Trail: All actions logged and traceable
5. Secret Management: Secrets in GitHub Secrets/Azure Key Vault
6. Role-Based Access: Different teams have different permissions

Success Criteria:
- Zero manual infrastructure changes
- All changes peer-reviewed
- Automated compliance validation
- Safe rollback capability
- Complete audit trail
Category 3: CI/CD Pipelines & Automation
Answer 13:
Continuous Integration (CI): Practice of automatically building and testing code changes whenever
developers commit to version control. Focus is on catching integration issues early. Example: Every git
push triggers automated build and unit tests.

Continuous Delivery (CD): Extension of CI where code changes are automatically prepared for release to
production. Code is always in a deployable state, but deployment to production requires manual approval.
Example: Code passes all tests and is deployed to staging automatically, but requires manual button click
for production.

Continuous Deployment: Further extension where every change that passes automated testing is
automatically deployed to production without human intervention. Example: Commit → Build → Test →
Deploy to Prod (all automated).

Key Differences:
- CI: Focus on integration and testing
- Continuous Delivery: Automated release pipeline, manual production deployment
- Continuous Deployment: Fully automated including production deployment

Most organizations use Continuous Delivery (not full Continuous Deployment) for production due to
compliance, change management, and risk management requirements.

Answer 14:
GitHub Actions Workflows: YAML-based automation platform for CI/CD built into GitHub. Workflows
consist of: triggers (events that start the workflow), jobs (groups of steps), steps (individual tasks), and
actions (reusable units).

Basic Docker Build and Push Workflow:

name: Build and Push Docker Image

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

env:
REGISTRY: [Link]
IMAGE_NAME: myapp

jobs:
build-and-push:
runs-on: ubuntu-latest
permissions:
contents: read

steps:
- name: Checkout repository
uses: actions/checkout@v3

- name: Log in to Azure Container Registry


uses: azure/docker-login@v1
with:
login-server: ${{ [Link] }}
username: ${{ secrets.ACR_USERNAME }}
password: ${{ secrets.ACR_PASSWORD }}

- name: Extract metadata for Docker


id: meta
uses: docker/metadata-action@v4
with:
images: ${{ [Link] }}/${{ env.IMAGE_NAME }}
tags: |
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=sha,prefix={{branch}}-

- name: Build and push Docker image


uses: docker/build-push-action@v4
with:
context: .
push: true
tags: ${{ [Link] }}
labels: ${{ [Link] }}
cache-from: type=registry,ref=${{ [Link] }}/${{ env.IMAGE_NAME }}:buildcache
cache-to: type=registry,ref=${{ [Link] }}/${{ env.IMAGE_NAME }}:buildcache,mode=max

- name: Output image URL


run: |
echo "Image pushed: ${{ [Link] }}"

Key Features:
- Automatic Docker layer caching for faster builds
- Semantic versioning tags
- Secure credential management via GitHub Secrets
- Matrix builds support for multi-platform images

You might also like