DATA ENGINEERING AND ML OPS By:
Dr. Sandhya N
BAD714C Prof. & HoD, AIML,
MODULE 4
Preparing for Production: Runtime Environments, Adaptation from Development to Production Environments, Data Access
Before Validation and Launch to Production, Final Thoughts on Runtime Environments, Model Risk Evaluation, The Purpose
of Model Validation, The Origins of ML Model Risk, Quality Assurance for Machine Learning.
Deploying to Production: CI/CD Pipelines, Building ML Artifacts, The Testing Pipeline, Deployment Strategies, Categories
of Model Deployment, Considerations When Sending Models to Production, Maintenance in Production,
Containerization, Scaling Deployments, Requirements and Challenges.
Textbook 2: Part II Chapter 5, Chapter 6
CHAPTER 5
PREPARING FOR PRODUCTION
This chapter explores the steps required to prepare for production (highlighted in the
context of the entire life cycle in Figure 5-1).
The goal is to illustrate, by extension, the elements that must be considered for robust
MLOps systems.
RUNTIME ENVIRONMENTS
▪The first step in sending a model to production is making sure it’s technically possible.
▪Production environments take a wide variety of forms: custom-built services, data
science platforms, dedicated services like TensorFlow Serving, low-level infrastructure
like Kubernetes clusters, JVMs on embedded systems, etc.
▪Ideally, models running in the development environment would be validated and sent
as is to production; this minimizes the amount of adaptation work and improves the
chances that the model in production will behave as it did in development.
ADAPTATION FROM DEVELOPMENT TO
PRODUCTION ENVIRONMENTS
▪In terms of adaptation work, on one end of the spectrum, the development and production
platforms are from the same vendor or are otherwise interoperable, and the dev model can
run without any modification in production.
▪In this case, the technical steps required to push the model into production are reduced to a
few clicks or commands, and all efforts can be focused on validation.
▪On the other end of the spectrum, there are cases where the model needs to be re-
implemented from scratch—possibly by another team, and possibly in another
programming language.
▪The reality is that handing over a model for another team to reimplement and adapt for
the production environment means that model won’t reach production for months.
Tooling considerations:
The format required to send to production should be considered early, as it may have a large impact on the model itself
and the quantity of work required to productionalize it.
For example, when a model is developed using scikit-learn (Python) and production is a Java-based environment that
expects PMML or ONNX as input, conversion is obviously required. In this case, teams should set up tooling while
developing the model, ideally before the first version of the model is finished or even started.
Performance considerations
Another common reason conversion may be required is for performance.
For example, a Python model will typically have higher latency for scoring a single record than its equivalent converted to
C++. The resulting model will likely be dozens of times faster.
Performance also comes into play when the production model must run on a low power device.
Ex: In the specific case of deep neural networks, for example, trained models can become extremely large with billions or hundreds of
billions of parameters. Running them on small devices is simply impossible, and running them on standard servers can be slow and
expensive.
One solution is to use compression techniques:
• With quantization, the model can be trained using 32-bit floating-point numbers and used for inference at a lower precision so that the
model requires less memory and is faster while accuracy is mostly preserved.
• With pruning, one simply removes weights from the neural network. This is a rather radical approach, but some methods allow for the
preservation of accuracy.
• With distillation, a smaller “student” network is trained to mimic a bigger, more powerful network. Done appropriately, this can lead to
better models. These methods are efficient if the initial model is trained in a way that reduces information loss while performing them, so
these operations are not simply conversions of the trained model post hoc, but rather orient the way the model is trained.
DATA ACCESS BEFORE VALIDATION AND
LAUNCH TO PRODUCTION
Another technical aspect that needs to be addressed before validation and launch to
production is data access.
▪For example, a model evaluating apartment prices may use the average market
price in a zip code area; however, the user or the system requesting the scoring will
probably not provide this average and would most likely provide simply the zip
code, meaning a lookup is necessary to fetch the value of the average.
▪In some cases, data can be frozen and bundled with the model. But when this is not
possible the production environment should access a database and thus have the
appropriate network connectivity, libraries, or drivers required to communicate with
the data storage installed, and authentication credentials stored in some form of
production configuration.
FINAL THOUGHTS ON RUNTIME ENVIRONMENTS
▪Training a model is usually the most impressive computation, requiring a high level of software
sophistication, massive data volumes, and high-end machines with powerful GPUs.
▪But in the whole life cycle of a model, there is a good chance that most of the compute is spent
at inference time.
▪It’s important to remember that not all applications require deep learning, and in fact, not all
applications require machine learning at all.
▪A valuable practice to control complexity in production is to develop complex models only to
provide a baseline for what seems achievable.
▪What goes into production can then be a much simpler model, with the advantages of lowering
the operating risk, increasing computational performance, and lowering power consumption.
▪If the simple model is close enough Preparing for Production to the high complexity baseline,
then it can be a much more desirable solution for production.
MODEL RISK EVALUATION
Before exploring how validation should be done in an ideal MLOps system, it’s
important to consider the purpose of validation.
The indirect, real-world impact a model in production can have is never certain, and
the malfunctioning of a seemingly insignificant cog can have tremendous
consequences in a complex system.
THE PURPOSE OF MODEL VALIDATION
▪As organizations become more and more complex, it is essential to understand that
involuntary malfunctions or malicious attacks are potentially threatening in most uses of
machine learning in the enterprise, not only in financial or safety-related applications.
▪Before putting a model in production teams should ask the uncomfortable questions:
• What if the model acts in the worst imaginable way?
• What if a user manages to extract the training data or the internal logic of the model?
• What are the financial, business, legal, safety, and reputational risks?
For high-risk applications, it is essential that the whole team be fully aware of these risks so
that they can design the validation process appropriately and apply the strictness and
complexity appropriate for the magnitude of the risks
However, machine learning introduces new types of risks and liabilities, and as data science
gets democratized, it involves many new organizations or teams that have no experience with
more traditional model risk management.
THE ORIGINS OF ML MODEL RISK
ML model risk originates essentially from:
• Bugs, errors in designing, training, or evaluating the model (including data prep)
• Bugs in the runtime framework, bugs in the model post-processing/conversion, or hidden incompatibilities
between the model and its runtime
• Low quality of training data
• High difference between production data and training data
• Expected error rates, but with failures that have higher consequences than expected
• Misuse of the model or misinterpretation of its outputs
• Adversarial attacks
• Legal risk originating in particular from copyright infringement or liability for the model output
• Reputational risk due to bias, unethical use of machine learning, etc.
The probability of materialization of the risk and its magnitude can be amplified by:
• Broad use of the model
• A rapidly changing environment
• Complex interactions between model
QUALITY ASSURANCE FOR MACHINE LEARNING
Software engineering has developed a mature set of tools and methodologies for quality assurance (QA),
but the equivalent for data and models is still in its infancy, which makes it challenging to incorporate into
MLOps processes.
Though it’s being covered as a part of this chapter on preparing for production, to be clear, QA for
machine learning does not occur only at the final validation stage; rather, it should accompany all stages of
model development.
In the case where the people in charge of validation are not the ones who developed the model, it is
essential that they have enough training in machine learning and understand the risks so that they can
design appropriate validation or detect breaches in the validation proposed by the development team.
Robust MLOps practices dictate that performing QA before sending to production is not only about
technical validation. It is also the occasion to create documentation and validate the model against
organizational guidelines.
End of Chapter 5
CHAPTER 6 DEPLOYING TO PRODUCTION
Business leaders view the rapid deployment of new systems into production as key to maximizing
business value. But this is only true if deployment can be done smoothly and at low risk.
Figure 6-1 presents this phase in the context of the larger life cycle)
CI/CD PIPELINES
CI/CD is a common acronym for continuous integration and continuous delivery.
The two form a modern philosophy of agile software development and a set of practices and tools to release applications
more often and faster, while also better controlling quality and risk.
CI/CD concepts apply to traditional software engineering, but they apply just as well to machine learning systems and are
a critical part of MLOps strategy. After successfully developing a model, a data scientist should push the code, metadata,
and documentation to a central repository and trigger a CI/CD pipeline.
An example of such pipeline could be:
1. Build the model
a. Build the model artifacts
b. Send the artifacts to long-term storage
c. Run basic checks (smoke tests/sanity checks)
d. Generate fairness and explainability reports
2. Deploy to a test environment
a. Run tests to validate ML performance, computational performance
b. Validate manually
3. Deploy to production environment
a. Deploy the model as canary
b. Fully deploy the model
There are common tools and best practices, but there is no one-size-fits-all CI/CD methodology.
BUILDING ML ARTIFACTS
The goal of a continuous integration pipeline is to avoid unnecessary effort in merging the work from several
contributors as well as to detect bugs or development conflicts as soon as possible. The very first step is using
centralized version control systems.
The most common version control system is Git, an open source software initially developed to manage the source
code for the Linux kernel.
While Git is appropriate for code, it was not designed to store other types of assets common in data science
workflows, such as large binary files or to version the data itself.
What’s in an ML Artifact?
Once the code and data is in a centralized repository, a testable and deployable bundle of the project must be
built. These bundles are usually called artifacts in the context of CI/CD. The following artifacts to be available
during production:
• Code for the model and its preprocessing
• Hyperparameters and configuration
• Training and validation data
• Trained model in its runnable form
• An environment including libraries with specific versions, environment variables, etc.
• Documentation
• Code and data for testing scenarios
THE TESTING PIPELINE
The testing pipeline can validate a wide variety of proper ties of the model contained in the artifact. One
of the important operational aspects of testing is that, in addition to verifying compliance with
requirements, good tests should make it as easy as possible to diagnose the source issue when they fail.
For that purpose, naming the tests is extremely important, and carefully choosing a number of datasets to
validate the model against can be valuable. For example:
▪A test on a fixed (not automatically updated) dataset with simple data as “base data”. If the test reports
show that this test failed, there is a strong possibility that the model is way off, and the cause may be a
programming error or a misuse of the model.
▪A number of datasets that each have one specific oddity (missing values, extreme values, etc.). this could
protect the model from malfunctions in the system querying or from adversarial examples
▪An essential part of model validation is testing on recent production data. This category of tests should be
performed and automatically analyzed when the model is already deployed to production.
The most widespread tool for software engineering continuous integration is Jenkins, a very flexible build
system that allows for the building of CI/CD pipelines regardless of the programming language, testing
framework, etc.
DEPLOYMENT STRATEGIES
To understand the details of a deployment pipeline, it is important to distinguish among concepts often used
inconsistently or interchangeably.
Integration
The process of merging a contribution to a central repository (typically merging a Git feature branch to the main
branch) and performing more or less complex tests.
Delivery
As used in the continuous delivery (CD) part of CI/CD, the process of building a fully packaged and validated
version of the model ready to be deployed to production.
Deployment
The process of running a new model version on a target infrastructure. Fully automated deployment is not always
practical or desirable and is a business decision as much as a technical decision, whereas continuous delivery is a
tool for the development team to improve productivity and quality as well as measure progress more reliably.
Continuous delivery is required for continuous deployment, but it also provides enormous value without.
Release
In principle, release is yet another step, as deploying a model version (even to the production infrastructure) does
not necessarily mean that the production work load is directed to the new version. As we will see, multiple
versions of a model can run at the same time on the production infrastructure.
CATEGORIES OF MODEL DEPLOYMENT
In addition to different deployment strategies, there are two ways to approach model
deployment:
• Batch scoring, where whole datasets are processed using a model, such as in daily scheduled
jobs.
• Real-time scoring, where one or a small number of records are scored, such as when an ad is
displayed on a website and a user session is scored by models to decide what to display.
Deploying many real-time scoring systems is conceptually simpler since the records to be scored
can be dispatched between several machines (e.g., using a load balancer).
Batch scoring can also be parallelized, for example by using a parallel processing run time like
Apache Spark, but also by splitting datasets (which is usually called partitioning or sharding) and
scoring the partitions independently.
CONSIDERATIONS WHEN SENDING MODELS TO PRODUCTION,
▪First consideration is often to avoid downtime, in particular for real-time scoring. The basic idea is
that rather than shutting down the system, upgrading it, and then putting it back online, a new
system can be set up next to the stable one, and when it’s functional, the workload can be
directed to the newly deployed version (and if it remains healthy, the old one is shut down). This
deployment strategy is called blue-green—or sometimes red-black— deployment. There are
many variations and frameworks (like Kubernetes) to handle this natively.
▪Another more advanced solution to mitigate the risk is to have canary releases (also called
canary deployments). The idea is that the stable version of the model is kept in production, but a
certain percentage of the workload is redirected to the new model, and results are monitored.
Overall, canary releases are a powerful tool, but they require somewhat advanced tooling to
manage the deployment, gather the metrics, specify and run computations on them, display the
results, and dispatch and process alerts.
MAINTENANCE IN PRODUCTION
Once a model is released, it must be maintained. At a high level, there are three maintenance
measures:
Resource monitoring
Just as for any application running on a server, collecting IT metrics such as CPU, memory, disk, or
network usage can be useful to detect and troubleshoot issues.
Health check
To check if the model is indeed online and to analyze its latency, it is common to implement a
health check mechanism that simply queries the model at a fixed interval (on the order of one
minute) and logs the results.
ML metrics monitoring
This is about analyzing the accuracy of the model and comparing it to another version or
detecting when it is going stale. Since it may require heavy computation, this is typically lower
frequency, but as always, will depend on the application; it is typically done once a week.
CONTAINERIZATION
In particular, it is necessary to provide an exact description of the environment (including, for
example, all the Python libraries used as well as their versions, the system dependencies that
need to be installed, etc.).
In addition, the target machine will typically run multiple models simultaneously, and two models
may have incompatible dependency versions.
Finally, several models running on the same machine could compete for resources, and one
misbehaving model could hurt the performance of multiple cohosted models.
Containerization technology is increasingly used to tackle these challenges. These tools bundle an
application together with all of its related configuration files, libraries, and dependencies that
are required for it to run across different operating environments.
The most well-known containerization technology is the open source platform Docker.
CONTAINERIZATION
When deploying a model, the framework should solve a number of issues:
• Which Docker host(s) should receive the container?
• When a model is deployed in several copies, how can the workload be balanced?
• What happens if the model becomes unresponsive, for example, if the machine hosting it fails? How can that be
detected and a container reprovisioned?
• How can a model running on multiple machines be upgraded, with assurances that old and new versions are
switched on and off, and that the load balancer is updated with a correct sequence?
Kubernetes, an open source platform that has gained a lot of traction in the past few years and is becoming the
standard for container orchestration, greatly simplifies these issues and many others. It provides a powerful
declarative API to run applications in a group of Docker hosts, called a Kubernetes cluster.
Docker with Kubernetes can provide a powerful infrastructure to host applications, including ML models.
Leveraging these products greatly simplifies the implementation of the deployment strategies—like blue-green
deployments or canary releases
SCALING DEPLOYMENTS
As ML adoption grows, organizations face two types of growth challenges:
• The ability to use a model in production with high-scale data
• The ability to train larger and larger numbers of models
Handling more data for real-time scoring is made much easier by frameworks such as
Kubernetes.
There are essentially two types of strategies to distribute the computation:
• Using a framework that handles distributed computation natively, in particular Spark. Spark
is an open source distributed computation framework. It is useful to understand that Spark and
Kubernetes do not play similar roles and can be combined.
•Another way to distribute batch processing is to partition the data. There are many ways to
achieve this, but the general idea is that scoring is typically a row by-row operation
•In particular, scaling the number of models means that the CI/CD pipeline must be able to
handle large numbers of deployments.
•In some applications, it is possible to rely on fully automated continuous deployment if the risks
are well controlled by automated validation, canary releases, and automated canary analysis.
This is suboptimal from efficiency and governance perspectives. While some models may need
highly specific validation pipelines, most projects can probably use a small number of common
patterns.
A dedicated tool to manage large numbers of pipelines can also be used; for example,
Netflix released Spinnaker, an open source continuous deployment and infrastructure
management platform.
Scalable and Elastic Systems
A computational system is said to be horizontally scalable (or just scalable) if it is possible to
incrementally add more computers to expand its processing power. For example, a Kubernetes
cluster can be expanded to hundreds of machines.
An elastic system allows, in addition to being scalable, easy addition and removal of resources
to match the compute requirements. For example, a Kubernetes cluster in the cloud can have
an auto-scaling capability that automatically adds machines when the cluster usage metrics
are high and removes them when they are low.
REQUIREMENTS AND CHALLENGES
When deploying a model, there are several possible scenarios:
• One model deployed on one server
• One model deployed on multiple servers
• Multiple versions of a model deployed on one server
• Multiple versions of a model deployed on multiple servers
• Multiple versions of multiple models deployed on multiple servers
Requirements more specifically, it should cover all of the following situations:
• The system can access and retrieve scoring logs from multiple servers, either in a real-time
scoring use case or in a batch scoring use case.
• When a model is deployed on multiple servers, the system can handle the map ping and
aggregation of all information per model across servers.
• When different versions of a model are deployed, the system can handle the mapping and
aggregation of all information per version of the model across servers.
Challenges:
▪For large-scale machine learning applications, the number of raw event logs generated can be
an issue if there are no pre-processing steps in place to filter and aggregate data.
▪For real-time scoring use cases, logging streaming data requires setting up a whole new set of
tooling that entails a significant engineering effort to maintain.
END of Module 4