AI Workflow

The Machine Learning Lifecycle from Concept to Deployment and Beyond

Learning Objectives
Understand the ML lifecycle and its structured stages

Identify challenges between traditional software and ML lifecycles

Explore the various people and roles involved in ML projects

Examine system-level considerations like resources and infrastructure

Appreciate the iterative nature of ML lifecycles and feedback loops

Purpose of the AI Workflow

• Core Question: What are the diverse elements of AI systems and how do
we combine them?

• Practical AI requires orchestrating multiple components

• Systematic perspective reveals how data flow, model training, and
deployment are intertwined
• Analyzing interconnections offers insights for real-world solutions
 The ML Lifecycle: A High-Level Overview

Systematic, interconnected process transforming raw data into

actionable models
Each stage builds upon the previous, creating an iterative cycle of
refinement
Key Characteristic: Continuous feedback loops enable
improvement and adaptation
Figure on the next slide provides a detailed view of this iterative
process
ML Lifecycle Stages
Distilling the Lifecycle: Overarching
Categories
 • A defining feature is its dynamic, non-
linear nature
• Feedback loops ensure systems maintain
effectiveness
The Iterative • This adaptability is critical for addressing:
Nature of the • - Shifting data distributions
ML Lifecycle • - Operational constraints
• - Evolving user requirements
• Provides a foundation for exploring data
engineering, optimization, and
deployment strategies
 • Machine Learning (ML) Lifecycle: A structured,
iterative process defining the key stages in the
development, deployment, and refinement of ML
systems.

Formal • Encompasses:
Definition of • Problem formulation
• Data collection
the ML • Model training
Lifecycle • Evaluation
• Deployment
• Monitoring
• Emphasizes: Feedback loops and continuous
improvement for robust, scalable systems
 • Traditional software is deterministic (behavior is
explicitly programmed)
• ML systems are probabilistic (behavior is learned
from data)

Traditional vs. • Example:

ML Lifecycles • Traditional: 'if account balance > transaction
amount, then allow transaction'
(Introduction) • ML-Based: Learns to recognize suspicious
patterns from historical transaction data

• This fundamental shift reshapes the entire

development lifecycle
The AI Lifecycle • The lifecycle consists of several interconnected stages

Stages
 • Problem: Diabetic Retinopathy (DR) is a
Introducing leading cause of preventable blindness
Our Case • Goal: Develop an AI system to analyze
retinal images and identify signs of DR
Study: with expert-level accuracy
Google's • Challenge: Move from high accuracy in
research to effective deployment in rural
Diabetic clinics (Thailand, India)
Retinopathy • Why it's a good case study: Spans the
Project entire ML lifecycle, highlighting real-world
complexities beyond model accuracy
Retinal Haemorrhages
 • The foundation of all subsequent phases
• Challenge: Define not just what the
Stage 1: system should do, but how it should learn
to do it
Problem • Must balance technical goals (expert-level
Definition accuracy) with practical, operational
constraints (limited hardware, clinical
workflows)
 • Context: 415M diabetic patients at risk;
limited access to specialists
Problem • Requirements: Prioritize cases for early
intervention while operating in resource-
Definition in limited settings
the DR • Needs: Interpretability for healthcare
providers, compliance with regulations
Project (privacy, healthcare standards)
• Integrated learning capabilities with
operational needs from the outset
 • A multi-step workflow bridging technical, operational,
and user considerations

• Steps:

Problem • 1. Identify core objective and constraints

• 2. Collaborate with stakeholders (e.g., clinicians) for
Definition domain knowledge

Workflow • 3. Outline requirements and anticipate deployment

challenges
• 4. Iterate based on regulatory and feasibility concerns

• Ensures alignment between technical feasibility and

real-world relevance
 • Choices here shape data collection, model
development, deployment, and maintenance
• Feedback Loops: Real-world feedback (e.g.,
from clinicians on usability) can necessitate
Systems revisiting and refining the original problem
Thinking in goals
• Emergent Behavior: A system designed for
Problem one task (DR detection) may reveal new use
Definition cases (detecting macular edema), reshaping
the problem's scope
• Resource Dependencies: Decisions on model
complexity affect infrastructure needs and
deployment feasibility
 • A well-defined problem ensures:
• Data collection focuses on relevant features
• Models are developed with the right
Lifecycle constraints
• Deployment strategies align with operational
Implications realities
of Problem
Definition • Forward-thinking (e.g., considering diverse
imaging conditions) reduces need for costly
redesigns later
• Embeds adaptability and scalability into the
system's DNA
Stage 2: Data Collection
• Data is the foundation of ML systems
• Challenges extend beyond gathering examples to maintaining
quality, diversity, and relevance at scale
• For medical AI like DR, data preparation must meet the highest
standards
Data Collection in the DR Project
• Scale:
• Development Dataset: 128,000 retinal images
• Annotations: Evaluated by a panel of 54 ophthalmologists (3-7
experts per image)
• Validation: 12,000 images for independent benchmarking

• Highlight: The critical role of domain expertise and

interdisciplinary collaboration for high-stakes ML
 • ML Need:
• High-quality, expert-annotated images
• Operational Realities:
Data • Different camera equipment across clinics
Requirements • Varying staff expertise
• Limited network connectivity
& Impact (DR
• Patient privacy regulations
Project)
• Impact: Shaped system architecture (local
storage/preprocessing), annotation pipelines,
and security protocols
 • The journey of a single retinal image:
• 1. Capture on retinal camera
• 2. Local storage & preprocessing at clinic
Data • 3. Central aggregation for model training
Infrastructure
(DR Project) • Challenge: Balance local needs with
centralized aggregation. Required flexible
infrastructure for both real-time
transmission and 'store-and-forward'
systems for clinics with poor connectivity
 • Scaling introduced variability: equipment,
workflows, patient demographics
• Each clinic = an independent data node;
Scale and system needed consistent performance
across all
Distribution
• Increasing data volumes (higher-
in Data resolution images) amplified demands on
Collection storage/processing
• Decisions in data collection ripple through
the entire lifecycle (e.g., affecting
compute needs for training)
 • Integral to the data collection process
• Automated Checks: At point of collection
to flag issues (poor focus, framing)
Data • Validation Systems: Verify image quality,
proper labeling, patient association, and
Validation privacy compliance
• Operates at both local and centralized
levels to ensure data reliability and
pipeline integrity
 • Viewing data collection through a lifecycle
lens reveals interconnections
• Feedback Loops:
• 'Data Quality Issues' from monitoring ?
Systems refinements in data preparation
Thinking: • 'Performance Insights' from deployment ?
guides new data collection to cover gaps
Data • Example (DR): Monitoring revealed
Collection underrepresented demographics, triggering
targeted data collection
• Scaling introduces emergent behaviors (e.g.,
network congestion) requiring holistic
management
ML Lifecycle Dependencies
 • Decisions here affect both initial model
quality and the system's long-term ability
Lifecycle to evolve
Implications • Proactively addressing variability and
quality minimizes need for costly
of Data downstream adjustments
Collection • Embedding lifecycle thinking into data
management ensures robustness and
scalability
 • The core of ML systems, but challenges
extend beyond algorithm selection
Stage 3: • Involves designing architectures,
Model optimizing for efficiency, and balancing
performance with deployability
Development
• In healthcare, every design decision
impacts clinical outcomes
 • Goal: Expert-level accuracy while handling
high-resolution, variable images
• Result: Deep neural network achieved an
Model F-score of 0.95, exceeding median
ophthalmologist score (0.91)
Development
• Techniques: Utilized transfer learning and
in the DR interdisciplinary collaboration with
Project medical experts
• Transfer Learning: Reusing a model
developed for one task as the starting
point for a second task
 • ML Requirements:
• High sensitivity and specificity
• System Constraints:
Model • Run on limited hardware in clinics
Requirements • Produce rapid results for clinical
workflows
& Impact (DR
• Be interpretable for healthcare providers
Project) • Impact: Shaped model design away from
largest models towards efficient,
lightweight architectures using techniques
like pruning and quantization
 • Process:
• 1. Data exploration & feature engineering
(with domain experts)
• 2. Model design & training
• 3. Iterative cycles of adjustment
Development (architectures, hyperparameters, new data)

Workflow
• Needs: Tools for handling large data,
collaboration, experiment tracking, and
version control
• Compute: Required extensive GPU resources
and sophisticated distributed training
systems
 • Scaling data volume and model
complexity demands powerful compute
resources
Scale and • Distributed Training: Essential for reducing
Distribution in training time, but introduces complexity
• Data synchronization
Model
• Gradient aggregation
Development • Fault tolerance
• Challenge: Managing expanding repository
of model versions, datasets, and results
 • Decisions about model architecture ripple
through the entire system:
• Affects preprocessing requirements
Systems • Shapes deployment strategies
Thinking: • Influences clinical workflows
Model • Feedback Loops: Insights from
Development deployment guide future model
adjustments
• Scaling can cause emergent behaviors
(bottlenecks in shared resources)
 • Not an isolated task; influences data prep,
training infra, and deployment
• Iterative nature ensures insights feed back
into earlier stages
Lifecycle
• Key Questions for later chapters:
Implications • Scalable training infrastructures?
of Model • Frameworks for distributed training?
Development • Ensuring reproducibility and version
control?
• Balancing accuracy with operational
constraints?
 • Integrating the validated model into
production systems and workflows
• Requires addressing practical challenges:
Stage 4: • System compatibility
Deployment • Scalability
• Operational constraints
• Predictions must be accurate and
actionable in real-world settings
 • Environment: Rural clinics with limited
compute and intermittent internet
• Strategy: Edge deployment (models run
Deployment locally on clinic hardware)
• Measures: Automated quality checks on
in the DR images
Project • Seamless integration with clinical
workflows
• Bridged technological sophistication with
usability and scalability
 • Requirements:
• Operate on limited hardware
• Fit clinical workflow
Deployment • Ensure rapid/interpretable results
Requirements • Interface with Hospital Information
& Impact Systems (HIS)
• Comply with privacy regulations
• Impact: Drove choice for edge deployment
and model optimization
 • Process:
• 1. Testing in simulated environments
replicating clinic constraints
Deployment • 2. Phased rollout to pilot sites for real-
world testing
Workflow • 3. Development of robust APIs and user-
friendly interfaces
• Focus on seamless interaction between
ML system and clinical ecosystem
 • Scaling introduced complexity: each clinic
had unique infrastructure
• Needed flexible deployment strategies for
Scale and consistent performance
Distribution • Challenge: Maintaining synchronized
in updates across all sites
• Required centralized version control and
Deployment automated update pipelines
• Critical: Training and supporting end-users
(clinicians)
 • Paramount in clinical contexts
• Implemented:
Robustness • Fail-safes for poor-quality data or errors
• Automated image quality checks
and • Extensive stress testing for peak usage
Reliability • Redundancy in critical components
• Rigorous testing of external system
integrations
 • Deeply connected to all other lifecycle
stages
• Feedback Loops: Real-world usage
Systems generates insights for model development
• Emergent Behaviors: Unexpected use
Thinking: cases necessitated system adjustments
Deployment • Resource Dependencies: Balancing
compute efficiency with accuracy
• Boundaries are fluid with model
development and maintenance
 • The point where system becomes active
and generates real-world feedback
• This feedback loops back into earlier
Lifecycle stages
• Key Questions for later chapters:
Implications • Balancing compute constraints with
of performance?
Deployment • Frameworks for synchronized
deployments?
• Designing for seamless integration?
• Ensuring user trust and proficiency?
 • Ongoing processes ensuring continued
effectiveness and reliability
• ML systems must account for:
Stage 5: • Data distribution shifts
Maintenance • Changing usage patterns
• Evolving requirements
• Monitoring provides necessary feedback;
Maintenance implements adaptations
 • Monitoring tracked performance across
diverse clinics
• Feedback Loops:
• Performance Insights ? New data
Maintenance collection
in the DR • Data Quality Issues ? Refine data
preparation
Project
• Model Updates ? Initiate retraining
• Proactive Maintenance: Plans to
incorporate 3D imaging (OCT) for
expanded capabilities
 • Technical Needs: Track model
performance, data quality, system
resources
• Operational Realities: Align with clinical
Monitoring workflows, detect demographic shifts
Requirements • Impact: Required real-time monitoring
& Impact capabilities
• Advanced logging and analytics pipelines
• Secure data handling across multiple
clinics
 • Process:
• 1. Define comprehensive monitoring
framework (KPIs, dashboards, alerts)
• 2. Dynamic maintenance: careful
validation of model updates
Maintenance
• 3. Use A/B testing for evaluation in real-
Workflow world conditions
• 4. Implement rollback mechanisms for
quick issue resolution
• Collaboration across technical and
clinical domains
 • Scaling increased complexity: more
clinics = more operational data
• Need to monitor both global metrics and
Scale and site-specific behaviors
Distribution • Advanced analytics systems process data
in from all clinics
• Challenge: Efficiently capturing insights
Maintenance from expanding scenarios
• Integrating new data into training without
disrupting workflows
 • Moving beyond reactive fixes
• Predictive Maintenance: Using patterns to
identify potential problems
Proactive • Continuous Learning Pipelines: Allowing
system to retrain and adapt
Maintenance • Metrics: Adaptability and resilience
became as important as accuracy
• Ensures system can handle future
challenges proactively
 • Deeply integrated with every other
lifecycle stage
• Feedback Loops: Central - monitoring
Systems insights drive updates
• Emergent Behaviors: Identifying subtle
Thinking: system-wide shifts
Maintenance • Resource Dependencies: Monitoring
competes for compute resources
• Maintenance requires skilled personnel
and potential downtime
 • Not isolated; insights feed back into data
collection and model development
• Ensures system evolves in response to
Lifecycle real-world challenges
Implications • Key Questions for later chapters:
of • Detecting subtle performance
degradations?
Maintenance • Efficient maintenance at scale?
• Continuous learning safely?
• Proactive maintenance tools?
 • Building ML systems is a collaborative,
multidisciplinary endeavor
AI Lifecycle • Diverse expertise is crucial for navigating
complexities of modern AI
Roles: It • Each role supports different phases of the
Takes a Team AI development process
• Success depends on effective
collaboration across all roles
 • Focus: Model creation and algorithm
refinement
• Experiment with architectures and refine
Key Role 1: algorithms
• DR Example: Instrumental in architecting
Data neural networks
Scientists • Balanced accuracy and computational
efficiency
• Core responsibility: Turning data into
predictive insights
 • Focus: Design robust data pipelines
• Ensure data is ingested, transformed, and
Key Role 2: stored effectively
• DR Example: Handled data from various
Data clinics
Engineers • Automated quality checks for
standardized inputs
• Built foundation for reliable model training
 • Focus: Integrate models into production
settings
Key Role 3: • Ensure models are scalable and fit
deployment constraints
Machine • DR Example: Optimized models for
Learning resource-limited clinics
Engineers • Enabled on-the-spot diagnosis
capabilities
• Bridge between research and production
 • Focus: Infuse technical progress with
practical relevance
Key Role 4: • Provide real-world context and validation
• DR Example: Ophthalmologists shaped
Domain problem definition
Experts • Ensured AI tools aligned with clinical
needs
• Critical for healthcare applications
 • Focus: Orchestrate workflow automation
and monitoring
• Manage continuous integration and
Key Role 5: deployment systems
MLOps • DR Example: Crafted centralized
monitoring frameworks
Engineers • Ensured streamlined updates across
deployment sites
• Maintain system reliability at scale
 • Focus: Ensure adherence to ethical
standards and legal requirements
Key Role 6: • Address privacy, fairness, and regulatory
concerns
Ethicists & • DR Example: Safeguarded patient privacy
Compliance amidst healthcare regulations
Officers • Ensured compliance with medical
standards
• Critical for responsible AI deployment
 • Focus: Orchestrate timelines, resources,
and communication
• Maintain project momentum and
Key Role 7: alignment with objectives
Project • DR Example: Harmonized efforts between
tech teams and clinicians
Managers • Coordinated with policy makers and
stakeholders
• Ensure project delivery and success
 • Synergy between roles fuels successful
outcomes
• Data engineers build foundation for data
Role scientists
• ML engineers translate models into
Interplay & production for domain experts
Synergy • MLOps, Ethics, and Management provide
scaffolding for sustainable projects
• Collaboration is non-negotiable for
effective AI systems
 • ML lifecycle is a structured, iterative
process:
• Data ? Model ? Deployment ? Monitoring ?
Summary: Data
• Defined by feedback loops and
The Iterative continuous improvement
AI Lifecycle • Fundamentally different from traditional
software development
• Requires balancing technical goals with
operational constraints
 • Problem Definition is foundational and
must integrate operational needs
• Data Collection is a system-level
Summary: challenge, not just a discrete task
• Model Development involves trade-offs
Key between accuracy and deployability
Takeaways • Deployment must bridge technological
sophistication with real-world usability
• Maintenance is ongoing, ensuring systems
adapt and remain relevant
 • This chapter provided a systems-level
overview of the ML lifecycle
• Subsequent chapters will dive deeper into
implementation:
Looking • Data Engineering
Forward • Model Optimization
• Deployment Strategies (MLOps)
• Monitoring Frameworks
• The lifecycle framework is the guide for all
that follows