completion = [Link].
create(
model=fine_tuned_model,
messages=[
{"role": "system", "content": "You are a
{"role": "user", "content": "Hello!"},
],
)
print([Link][0].message)
While these examples show the fine-tuning process with
OpenAI, the process will be similar with other providers even if
the implementation details may differ.
WARNING
If you decide to leverage fine-tuning, be mindful that you won’t be able to take
advantage of the latest improvements or optimizations in new LLMs, potentially
making the fine-tuning process a waste of your time and money.
With this final optimization step, you should now feel confident
in building GenAI services that not only meet your security and
quality requirements but also achieve your desired throughput
and latency metrics.
Summary
In this chapter, you learned about several optimization
strategies to improve the throughput and quality of your
services. A few optimizations you added covered various
caching (keyword, semantic, context), prompt engineering,
model quantization, and fine-tuning.
In the next chapter, we will shift focus to the last step in
building AI services: deploying your GenAI solution. This
includes exploring deployment patterns for AI services and
containerization with Docker.
1
See the OpenAI Batch API available in the OpenAI API documentation.
2
Learn more about cache control headers at the MDN website.
3
You may still require a trained embedder model for significant cost savings, as
making frequent API calls to an off-the-shelf embedder model could incur additional
costs, diminishing your overall savings.
4
For better security, you still need to sanitize any LLM-generated code before
forwarding it to downstream systems for execution.
Chapter 11. Testing AI Services
CHAPTER GOALS
In this chapter, you will learn about:
How to plan and structure test suites for comprehensive test
coverage, including unit, integration, end-to-end, and
behavioral tests
The concepts of testing boundaries, code coverage, and
idempotency in designing tests
How to identify and avoid common testing pitfalls to improve
test quality
How to leverage test fixtures and implement
parameterization to run tests with multiple inputs for
checking code robustness
How to efficiently set up and tear down testing environments
using pytest
How to maintain idempotency in testing processes by
correctly handling asynchronous flaky tests
How to use mocking and patching to isolate components
from external dependencies in unit tests
How to test GenAI services that use probabilistic models
using behavioral testing and auto-evaluation techniques
How to leverage several testing metrics for GenAI services
In this chapter, you’ll learn the importance of testing and its
challenges when building GenAI services. You’ll also learn
about key concepts such as test plans, the verification and
validation models, the testing pyramid, and the role of testing
data, environments, and boundaries.
To practice testing, you will use pytest , a popular testing
framework with features such as test fixtures, scopes, markers,
and fixture parameterization. You’ll also learn about the
pytest-mock plug-in for patching functions and using stubs,
mocks, and spy objects to simulate and control external
dependencies during tests.
Since mocking can make tests brittle, we’ll also explore
dependency injection, allowing you to inject mock or stub
dependencies directly into the components being tested,
avoiding runtime code modifications.
We’ll discuss the role of isolation and idempotency in tests,
when to use mocks, and how to test both deterministic and
probabilistic GenAI code. By the end of this chapter, you’ll be
confident in writing comprehensive test suites including unit,
integration, end-to-end, and behavioral tests for your own
services.
Before we dive into writing tests, let’s explore the foundational
concepts of traditional software testing and how to approach
testing GenAI services, which can prove challenging due to the
probabilistic nature of AI models.
The Importance of Testing
In theory, everyone agrees that testing is necessary when
building software. You write tests to give you confidence in the
functionality and performance of your systems, especially
when they interact with one another. But realistically, projects
may skip implementing manual or automated tests due to
various constraints including budget, time, or associated labor
costs related to maintaining tests.
The projects that skip testing, partially or entirely, end up
approaching software problems reactively instead of
proactively. This is when technical debt builds up, which you’ll
then have to pay back in labor and server costs, with interest, to
settle up.
The problem of when to test is challenging to solve. If you’re
just experimenting and hacking a prototype together in fast
iterations, you won’t need to worry about testing as much,
realistically. However, as soon as you have a minimum sellable
product, a system that interfaces with sensitive data and
processes user payments, then you must seriously consider
testing plans.
Earlier in my career, I was building a learning management
system for a client. I wrote a webhook endpoint to interface
with Stripe’s payment systems and my own home-brewed
authentication solution that would only register users on
successful first payment. The system had to charge and process
subscription payments of both new and existing customers and
send confirmation emails while tracking user records,
subscriptions, payments, checkout sessions, and invoices. The
logic of that webhook ended up so convoluted and complex that
it led to a monstrosity that became a 1,000-line function. The
function was checking unordered received events of various
types, with multiple round-trips to the database.
The whole solution had to be scrapped at the end since the
webhook’s behavior was so flaky, returning nonconsistent
responses to the same set of inputs. Users couldn’t register even
after successful payments. This flakiness made it unbearable to
debug that webhook, which forced me to rewrite the payment
system integration from scratch. If I had only slowed down to
plan and modularized the logic and wrote tests early on, I could
have saved myself from so much headache.
When you slow down to plan and test your services, you’re
trading off time and effort in exchange for confidence in your
code.
A few other times you should consider implementing tests are
when:
Multiple contributors add changes over time
Maintainers change external dependencies
You increase the number of components and dependencies in
your services
You suddenly spot too many bugs appearing
There is too much at stake if things go wrong—my
experience fell into this bucket
You should now understand how testing will benefit your
project.
Software Testing
Now that you’re familiar with the challenges and potential
approaches to testing GenAI services, let’s review software
testing concepts to understand their relevance to GenAI use
cases and common pitfalls to avoid.
Types of Tests
There are three common types of tests in software testing,
which, ordered by increasing size and complexity, are as
follows:
Unit tests
Focus on testing individual components or functions in
isolation across a discrete set of inputs and edge cases to
validate functionality at singular component level. Unit
tests are atomic with the smallest scope and often don’t
rely on external systems or dependencies.
Integration tests
Check the interaction between various components or
systems to verify they function together as intended.
Integration tests often capture issues with application
behavior at a subsystem level, validating data flows and
interface contracts, (i.e., specifications) between various
components.
End-to-end (E2E) tests
Verify the functionalities of the application at the highest
system level from start to finish by simulating real usage
scenarios. E2E tests give you the highest levels of
confidence in your application functionality and
performance but are the most challenging tests to design,
develop, and maintain.
TIP
E2E tests and integration tests share similarities that make them hard to distinguish
from one another. If a test is big and sometimes flaky, you may be working on an E2E
test.
Integration tests normally check a subset of systems and interactions, not the whole
system or a long chain of subsystems.
Figure 11-1 demonstrates the scope of each test type. Unit tests
shown on the left focus on isolated components, while
integration tests check pairwise interactions of multiple
components, including with external services. Lastly, E2E tests
cover the entire user journey and data flow within the
application to confirm the intended functionality.