Internal
Nice — I’ll give you a complete, practical, and opinionated software development
lifecycle (SDLC) that covers every phase, with processes, roles, tools,
methodologies, and explicit notes on Security, AI, Cloud, and best practices baked
into each phase. I’ll also include checklists, KPIs, and a sample CI/CD pipeline
blueprint you can copy.
High-level lifecycle (summary)
1. Strategy & Portfolio / Discovery
2. Requirements & Analysis
3. Architecture & Design
4. Implementation / Development
5. Testing & Quality Assurance
6. Release & Deployment
7. Operations, Observability & Maintenance
8. Security & Compliance (continuous)
9. AI/ML model lifecycle (where relevant)
10. End-of-life / Retirement
Each phase below shows: goals, processes, roles, recommended tools,
security/AI/cloud considerations, and best-practice checklists.
1. Strategy & Portfolio / Discovery
Goal: Validate the product idea, align to business outcomes, estimate ROI and risks.
Key processes
• Market research, stakeholder interviews, feasibility studies.
• Opportunity assessment, business case, OKRs alignment, cost-benefit and risk
assessment.
• High-level roadmap and backlog prioritization (value vs risk).
Roles
• Product Owner / Head of Product, CTO, Business Analysts, Solution Architect,
Security Officer, Finance/Legal.
Internal
Tools
• Roadmapping: Aha!, Productboard, Notion.
• Collaboration: Miro, Confluence.
• Analytics: Looker, Metabase.
Security / Cloud / AI considerations
• Security: Early identification of regulatory constraints (GDPR, HIPAA), data
residency, and sensitive data classification. Include security lead in discovery.
• Cloud: Identify preferred cloud providers, expected regions, cost model (capex
vs opex), multi-cloud requirements.
• AI: Decide if product uses AI/ML; identify data sources, labeling needs,
privacy/legal constraints.
Checklist
• Business case + ROI estimate.
• Regulatory and compliance constraints documented.
• High-level risk register (security, privacy, vendor lock-in).
• Data gravity / data residency map.
2. Requirements & Analysis
Goal: Convert strategy into clear, testable, prioritized requirements and acceptance
criteria.
Processes
• User story mapping, Jobs-to-be-done, use cases, non-functional requirements
(NFRs).
• Prioritization frameworks: MoSCoW, RICE, WSJF.
• Acceptance criteria & Definition of Done (DoD).
• Threat modeling kickoff.
Roles
• Product Owner, Business Analyst, UX Designer, Security Engineer, Data Engineer
(if data-heavy), Legal/Compliance.
Tools
Internal
• Backlog & tickets: Jira, Azure Boards, GitHub Issues.
• Documentation: Confluence, Notion.
• Requirements modeling: UML tools, Lucidchart.
• Collaboration: Figma for UI/UX.
Security / Cloud / AI
• Security: Threat modeling (STRIDE, PASTA), data flow diagrams (DFDs), privacy
impact assessment, identify sensitive APIs.
• Cloud: Define deployment constraints, SLA targets, expected scale, storage
classes.
• AI: Data requirements for training, labeling rules, model evaluation metrics, bias/
fairness constraints.
Checklist
• User stories + acceptance criteria.
• NFRs: performance, availability, scalability, maintainability, security, legal.
• Completed threat model with mitigation plan.
• Data classification & retention policy.
3. Architecture & Design
Goal: Define system architecture, components, interfaces, data flows, and technology
stack.
Processes
• Architectural decision records (ADRs).
• High-level and low-level design, component diagrams, API contracts.
• Capacity planning and cost modeling.
• API & event contracts (OpenAPI, AsyncAPI).
Roles
• Solution / Enterprise Architect, Tech Lead, Security Architect, Data Architect,
Cloud Architect, UX Lead.
Tools
Internal
• Modeling & ADRs: Structurizr, C4 models, PlantUML.
• API specs: Swagger/OpenAPI, Postman.
• Cloud architecture: Cloud provider consoles, Architecture diagrams in [Link].
• IaC planning: Terraform, Pulumi.
Security / Cloud / AI
• Security: Secure-by-design principles, secure defaults, encryption in transit & at
rest, key management (KMS), IAM model, WAF/Network segmentation.
• Cloud: Decide containers vs serverless vs VMs; design for disposability and
horizontal scale; define multi-AZ/region architecture; backup and DR strategy.
• AI: Model hosting architecture (online vs batch), feature store, MLOps pipeline
design.
Best practices
• Microservices boundaries aligned to business domain (bounded contexts).
• Standardized API error formats, idempotency, retries.
• Single source of truth for config (secrets management).
• ADRs recorded for all significant decisions.
Checklist
• ADRs, component diagrams, sequence diagrams.
• OpenAPI/AsyncAPI for public interfaces.
• IaC plan and modules sketched.
• Security architecture & key controls listed.
4. Implementation / Development
Goal: Build features in increments with high quality and security, using collaborative
workflows.
Processes & methodologies
• Agile (Scrum / Kanban) with short iterations.
• Git-based workflows: trunk-based development or feature branches with short
lifetime.
Internal
• Pair programming, code reviews, mob programming where useful.
• Shift-left security: SAST in dev, secure coding standards.
• Automated CI for builds, tests, linting, dependency checks.
Roles
• Developers, Tech Lead, QA Engineers, Security Engineer, DevOps/SRE, Data
Scientist (if ML), UX designer.
Tools
• Source control: GitHub, GitLab, Bitbucket.
• CI: GitHub Actions, GitLab CI, Jenkins, CircleCI.
• Code review: Pull Requests with templates.
• Static analysis: SonarQube, Semgrep, ESLint, Pylint.
• Package & dependency scanners: Dependabot, Snyk.
• Secrets scanning: git-secrets, TruffleHog.
• Local dev environment: Docker, Docker Compose, Dev Containers (VS Code).
Security / Cloud / AI
• Security: SAST (static analysis) in pipeline, secrets policies, least-privilege IAM
in dev/test. Use secure SDKs and frameworks. Dependency vulnerability gating.
• Cloud: Local emulators for cloud services (LocalStack). IaC modules and CI
validation (terraform plan in CI).
• AI: Use LLMs carefully for developer productivity: code generation (Copilot) with
human review and security checks; track prompt & model versions for audit.
Best practices
• Strong commit messages and change logs.
• Enforce lint + type checks.
• Build artifacts stored immutably (artifact registry).
• Automate environment provisioning via IaC.
Checklist
• PR template enforces DoD and security checklist.
• Automated tests passing in CI.
Internal
• 0 high/critical OSS vulnerabilities allowed in pipeline.
• Code coverage target (as guidance, not an absolute metric).
5. Testing & Quality Assurance
Goal: Verify correctness, performance, security, and compliance before release.
Testing pyramid
• Unit tests (fast, large coverage).
• Integration tests.
• Contract tests (Pact).
• End-to-end tests (Cypress, Playwright) — limited to key flows.
• Performance, load & chaos testing.
• Security testing: SAST, DAST, dependency scanning, container image scanning,
secrets scanning.
• Penetration testing & red team for high-risk systems.
Processes
• Test automation in CI with gating policies.
• Test data management and test environments parity.
• Test plans, traceability matrix linking tests to requirements.
• Regression automation and scheduled canary tests.
Tools
• Unit: JUnit, pytest, Jest.
• E2E: Cypress, Playwright, Selenium.
• Contract tests: Pact.
• Load: k6, Gatling.
• Security testing: OWASP ZAP (DAST), Burp Suite, Trivy (container), Clair.
• Test management: TestRail, Zephyr.
Security / Cloud / AI
• Security: Add SCA (Software Composition Analysis), DAST, container image
scanning, Infrastructure-as-Code scanning (Checkov, tfsec).
Internal
• Cloud: Run tests in ephemeral cloud test environments (spin up via CI).
• AI: Use automated fairness and bias tests for models; synthetic data generation
for tests when production data restricted.
Best practices
• Fail builds on regressions and high severity vulnerabilities.
• Maintain test environment parity with production (k8s manifests, CI-generated
secrets).
• Test data isolation to avoid PII leakage.
Checklist
• Test coverage reports available.
• Performance baseline and SLA tests passed.
• DAST/SAST gates satisfied.
• Pen test scheduled for major releases.
6. Release & Deployment
Goal: Safely deliver changes to users with minimal risk and fast recovery.
Processes
• CI/CD pipelines with stages: build → test → staging → canary/blue-green →
production.
• Release strategies: feature flags, dark launches, canary releases, blue/green
deployments, rolling updates.
• Release approvals: automated and human gating where needed.
• Change management & communication plan.
Roles
• Release Manager (or DevOps Lead), SRE, Product Owner, Security Reviewer, QA
Lead, Support.
Tools
• CD: Argo CD, FluxCD, Spinnaker, AWS CodeDeploy.
• Feature flags: LaunchDarkly, Unleash, Flagsmith.
Internal
• Container orchestration: Kubernetes, ECS.
• Service mesh (if needed): Istio, Linkerd.
• Artifact registries: Artifactory, GitHub Packages, Nexus.
Security / Cloud / AI
• Security: Validate images against registry policies; runtime protection via
eBPF/WAF; IAM roles for deployment limited.
• Cloud: Automate region failovers, infrastructure drift detection.
• AI: If model changes are deployed, version models and run A/B tests; ensure
model explainability logs.
Best practices
• Immutable infrastructure and artifacts.
• Feature toggles for controlled rollouts.
• Automated rollback on error thresholds.
• Release smoke tests and post-deploy validations.
Sample CI/CD pipeline blueprint
1. Build artifact (container image) → sign artifact.
2. Run unit & integration tests.
3. SAST & SCA scans; fail on policy violations.
4. Push to artifact registry.
5. Deploy to staging (k8s or cloud environment).
6. Run automated E2E + performance smoke.
7. Canary to subset of users; monitor errors/latency/SLOs.
8. Promote to prod via Argo/Flux and flip feature flags.
9. Post-release canary analysis and rollback if thresholds breached.
7. Operations, Observability & Maintenance
Goal: Keep systems healthy, performant, secure, and cost-efficient in production.
Processes
• Incident management: detection → triage → remediation → RCA.
Internal
• Capacity planning & autoscaling / cost optimization.
• Patch management and dependency upgrades.
• Continuous improvement through retros and RCA follow-ups.
Roles
• SRE / Ops Engineers, Incident Commander, On-call rotations, Security Incident
Response Team (SIRT), Application Owners, DBAs.
Tools
• Monitoring: Prometheus, Datadog, New Relic.
• Logging: ELK/EFK (Elasticsearch + Fluentd + Kibana), Loki.
• Tracing: Jaeger, Zipkin, OpenTelemetry.
• Incident mgmt: PagerDuty, Opsgenie.
• Runbooks / playbooks: Confluence + runbook automation.
• Cost mgmt: Cloud provider cost tools, CloudHealth.
Security / Cloud / AI
• Security: Continuous monitoring (SIEM): Splunk, Elastic SIEM; EDR on hosts;
ensure periodic vulnerability scanning and patching; secure logging (PII
redaction).
• Cloud: Use autoscaling, right-size resources, leverage spot/discount pricing
where safe. Tagging for cost attribution.
• AI: Monitor model drift, data drift, prediction quality; retrain pipeline and CI for
models (MLOps).
Best practices
• Define SLOs and SLIs; track error budget and tie to release cadence.
• Automated remediation for common issues.
• Post-incident reviews with blameless culture and action items tracked.
Checklist
• Alerts aligned to SLOs with documented escalation.
• Runbooks for common incidents.
• Regular dependency and patch cycles.
Internal
• Backup & restore test completed periodically.
8. Security & Compliance (Continuous)
Goal: Continuously reduce risk while enabling fast delivery.
Processes
• Shift-left security: SAST, secrets checks, dependency scanning in CI.
• Threat modeling during design and periodic re-evaluation.
• Secure SDLC governance: security gates within pipelines and DoD.
• Vulnerability management, patching, and monitoring.
• Incident response and tabletop exercises.
• Compliance audits and evidence collection.
Roles
• Security Engineers, AppSec, DevSecOps, CISO, Compliance Officer.
Tools
• SAST: SonarQube, Semgrep.
• DAST: OWASP ZAP, Burp Suite.
• SCA: Snyk, Whitesource.
• IaC scanning: Checkov, tfsec.
• Secrets management: HashiCorp Vault, AWS Secrets Manager.
• CI policy enforcement: OPA/Gatekeeper, policy-as-code (Sentinel).
• SIEM & SOAR: Splunk, Elastic SIEM, Palo Alto Cortex.
Best practices
• Security gating: prevent merging if critical vulnerabilities found; use ticketing for
remediation.
• Rotate keys, MFA everywhere, least privilege IAM roles.
• Maintain an asset inventory and data flow maps.
• Automated compliance evidence collection (for SOC2, ISO27001).
Checklist
Internal
• Security testing integrated in every pipeline stage.
• Monthly automated scans and weekly triage.
• Incident response plan + contact list.
• Hardening guides and baseline images.
9. AI / ML Model Lifecycle (MLOps) — if applicable
Goal: Reliable, auditable model development, testing, deployment and monitoring.
Processes
• Data collection & labeling, feature engineering, model training, validation,
deployment, monitoring & drift detection.
• Model versioning, reproducibility, and lineage.
• Bias/fairness testing and explainability.
Roles
• Data Scientists, ML Engineers, Data Engineers, MLOps, Product, Privacy Officer.
Tools
• Data/feature store: Feast.
• Model training: TensorFlow, PyTorch, scikit-learn.
• Experiment tracking: MLflow, Weights & Biases.
• Serving: KFServing, TorchServe, Seldon, AWS SageMaker.
• Pipeline: Kubeflow, Airflow, Prefect.
• Monitoring: Evidently, WhyLogs, Prometheus for metrics.
Security / Cloud
• Secure data pipelines, anonymization/pseudonymization.
• Model access control, audit logs for inference.
• Containerized models with RBAC and network policies.
Best practices
• Keep model training environments reproducible (IaC).
• Shadow deployments and A/B testing before full traffic.
Internal
• Document model cards and data sheets.
Checklist
• Dataset lineage and consent records.
• Model test coverage (accuracy, fairness, robustness).
• Rollback & retrain automation.
10. End-of-life / Retirement
Goal: Safely decommission features/services while retaining compliance and data
where necessary.
Processes
• Sunset plan: migration path for users, data export, legal retention schedule.
• Infrastructure teardown via IaC.
• Post-retirement cost cleanup and documentation update.
Roles
• Product, Engineering, Compliance, Support.
Checklist
• User communication plan executed.
• Data retention or deletion policies enforced.
• IaC destroy run and resource usage verified.
Cross-cutting practices (applies to all phases)
• Observability-first design: instrument early; metrics/traces/logs as part of DoD.
• Automation-first: provision, test, deploy, and remediate automatically.
• Immutable infrastructure & artifacts.
• Policy-as-code: enforce security/compliance automatically.
• Least privilege & Zero Trust: for every service and human actor.
• Documentation-as-code: docs versioned with code (e.g., docs folder,
OpenAPI).
• Cost-awareness: include cost reviews in architecture/design decisions.
Internal
• Accessibility & internationalization: include in requirements & testing.
Example role matrix (RACI-lite)
• Product Owner — R/A for features and business priorities.
• Tech Lead / Architect — A for technical decisions, R for design.
• Developers — R for implementation.
• QA/QA Automation — R for testing.
• SRE/DevOps — R/A for CI/CD, deployments and SLIs.
• Security — C (consult) / A for security gating and approvals.
• Data/ML — R for models/data pipelines.
• Support/Customer Success — C/R for operational feedback and incidents.
KPIs & Metrics to track
• Cycle time (PR creation → merge).
• Mean time to detect (MTTD) and mean time to recover (MTTR).
• Change failure rate.
• SLO / error budget burn rate.
• Test pass rate / flakiness.
• Vulnerability counts and time-to-remediate.
• Cost per feature / cloud spend per service.
• Model drift metrics (for ML).
Sample security checklist (developer PR gating)
1. Lint and unit tests pass.
2. SAST scan: no new critical/ high issues.
3. Dependency scan: no critical vulnerabilities (or plan to remediate).
4. Secrets scanner: no secrets committed.
5. Architecture/ADRs updated if required.
Internal
6. Feature flag added for risky changes.
7. Performance smoke tests (if infra changes).
8. Approvals: product + security if needed.
Example: Minimal technology stack for a modern cloud-native app
• Source & CI: GitHub + GitHub Actions
• Container registry & infra: ECR / GCR / Azure ACR + Terraform
• Orchestration: Kubernetes (EKS/GKE/AKS) or serverless (Lambda/Functions)
• CD: Argo CD or Flux
• Monitoring & Tracing: Prometheus + Grafana + OpenTelemetry + Jaeger
• Logging: EFK or managed (Datadog)
• Secrets: HashiCorp Vault or cloud-managed KMS + Secrets Manager
• SAST/SCA: Semgrep + Snyk
• Feature flags: LaunchDarkly / Unleash
• MLOps (if ML): MLflow + Seldon / KFServing
Practical implementation tips & pitfalls to avoid
• Pitfall: Treating security as an afterthought. Fix: shift-left and automation +
security KPIs.
• Pitfall: Too many E2E tests causing flakiness. Fix: invest in unit/integration &
contract tests; run E2E in stable pipelines.
• Pitfall: Manual CI/CD gates. Fix: automate gating with policy-as-code and robust
test suites.
• Pitfall: No production telemetry. Fix: instrument early and continuously.
• Pitfall: Secrets in code. Fix: secret scanning + vault-based secrets.
Quick templates
Definition of Done (DoD) — example
• Code compiles and passes all unit/integration tests.
Internal
• PR reviewed & approved by at least one peer.
• Security checks (SAST/SCA/secrets) pass or accepted remediation plan exists.
• Feature toggles present for risky features.
• Documentation updated (README, API docs).
• Performance baseline validated for infra changes.
• Deployed to staging and smoke-tested.
Incident response flow (high-level)
1. Alert generated → Incident Commander assigned.
2. Triage & classify severity (P1/P2).
3. Mitigation (rollback/canary/feature flag).
4. Communicate status to stakeholders & customers.
5. Postmortem & action items within 72 hours.
Final notes — AI, Cloud, Security in one sentence each
• AI: Use AI to accelerate development, testing, and ops (code assist, test
generation, anomaly detection), but always require human review, provenance
tracking, and guardrails (bias checks, data privacy).
• Cloud: Design for elasticity, disposability, observability and cost; prefer IaC,
immutable artifacts, and automated pipelines.
• Security: Make security measurable and automated — integrate it into every
pipeline stage, and keep the security team as an enabler, not a blocker.
If you’d like, I can:
• produce a starter repo CI/CD pipeline (YAML) tailored to your stack (e.g.,
GitHub Actions + Kubernetes + ArgoCD),
• create a threat model template (DFD + STRIDE checklist) for a sample
architecture,
• or generate a playbook for SRE runbooks and incident templates.
Which of those would you like next?