Scribd
Upload
2 views2 pages

Notes

The document outlines the technical components and lifecycle artifacts essential for MLOps, including CI/CD automation, workflow orchestration, and continuous monitoring. It emphasizes the importance of collaboration, reproducibility, and versioning in machine learning projects, as well as the evaluation metrics and bias assessment methods for large language models. A typical LLMOps lifecycle is also presented, detailing the steps from defining use cases to deploying and monitoring the system.

Uploaded by

Praful Shrestha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
2 views2 pages

Notes

The document outlines the technical components and lifecycle artifacts essential for MLOps, including CI/CD automation, workflow orchestration, and continuous monitoring. It emphasizes the importance of collaboration, reproducibility, and versioning in machine learning projects, as well as the evaluation metrics and bias assessment methods for large language models. A typical LLMOps lifecycle is also presented, detailing the steps from defining use cases to deploying and monitoring the system.

Uploaded by

Praful Shrestha
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Article 1

1 Technical Components
P1: CI/CD Automation: Automatically building, testing, and deploying your code.

P2: Workflow Orchestration: Using DAGs to manage the order of tasks (e.g., fetch data → train →
evaluate).

P3: Reproducibility: Being able to run the same experiment and get the exact same result.

P4: Versioning: Keeping track of different versions of Data, Model, and Code (not just code!).

P5: Collaboration: Reducing "silos" so data scientists and engineers actually talk to each other.

P6: Continuous ML Training (CT): Automatically retraining the model when new data arrives or
performance drops.

P7: Metadata Tracking: Logging the "diary" of every run (parameters used, training time, accuracy).

P8: Continuous Monitoring: Watching the model in production to see if its predictions are getting
worse (Concept Drift).

P9: Feedback Loops: Ensuring that when the monitor sees a problem, the system automatically triggers
a fix or a retrain.

MLOps aims to facilitate the creation of machine learning products by leveraging these principles:
CI/CD automation, workflow orchestration, reproducibility; versioning of data, model, and code;
collaboration; continuous ML training and evaluation; ML metadata tracking and logging; continuous
monitoring; and feedback loops.

Artifacts
1. Model artifacts
2. Data artifacts: dataset, features and labels
3. Reproducibillity (Software artifacts): codes, configeration, and environments
4. Metadata: hyperparameter, parameters, execution statistics, and quality matrics

ML Lifecycle AMS

Lifecycle Artifacts
1. Data Oriented artifacts
a) Data collection: all the data collected and datasetartifacts
b) Data cleaning: cleaned data artifacts
c) Data labeling: labeled data artifacts
d) Feature engineering: feature engineering artifacts
e) source code, log and environments
2. Model Oriented artifacts
a) Model design: codes, logs and environment
b) Model traing
c) Model evaluation
d) Model optimization: parameter, hyperparameter, performance matrics
3. Operation aritfacts
a) Deployment: code of api, docker
b) monitoring: logs, accuracy,

Evaluation metrics
To assess the performance of LLMs, various metrics are used depending on the specific task.
i. Intrinsic evaluation metrics:
ii. Human evaluation metrics:
iii. Efficiency and resource utilization metrics:
iv. Novel metrics:

Bias
🔥 Super Simple Summary
Pairwise → compare 2 answers
List-wise → compare many answers
Swap order → check bias
Metrics:
PC → consistency after swapping
PF → fairness between positions
RS → consistency over repetitions

Bias is based on
model
task
Answer quality: clear answer doesnot create bias, similar answer create bias
Length of text matters little
Hard case means high bias

A Typical LLMOps Lifecycle


1 Define the use case
2 Collect and prepare knowledge sources
3 Select a model
4 Design prompts
5 Build the application pipeline
6 Evaluate outputs
7 Deploy the system
8 Monitor usage and behavior
9 Iterate and improve

You might also like