interdependent and order-dependent tests often fail together,
preventing you from getting valuable feedback on failures.
But how is flaky behavior related to asynchronous tests?
As discussed in Chapter 5, asynchronous code is leveraging
Python’s built-in scheduler, an event loop, to switch tasks when
faced with a blocking I/O operation. This task switching in a
testing environment can make asynchronous tests challenging
to implement correctly because async operations may not
complete immediately and can be executed out of order.
Async tests often interface with external dependencies like
databases or filesystems executing I/O blocking operations that
can take a long time to run. This is a major issue for unit tests
that must run very quickly so that you can execute them
frequently.
Unlike synchronous code where operations are executed in a
predictable and linear sequence, async code also introduces
variability in timing, execution order, and fixture state,
reducing the consistency of the outcomes across tests.
Additionally, response times from external dependencies can
fluctuate, leading to side effects that violate the test isolation
principle.
To mitigate the risk of side effects and flaky behavior, you’ll
need to correctly handle async tests by:
Awaiting blocking I/O operations
Avoiding unintentional use of blocking synchronous I/O
operations inside async tests
Using correct timeouts for managing delays
Explicitly controlling the sequence of operations, especially
when running async tests in parallel
Perhaps, the best mitigation is to write synchronous tests by
mocking external dependencies, which will decouple your
functions from I/O blocking dependencies. Using mocks, you
can then run fast and reliable tests without having to wait for
I/O operations to complete in the order you need.
TIP
Async tests can still be useful with real dependencies when locally testing a
replicated production environment.
Next, let’s see how to mock external dependencies in unit tests
so that you can write synchronous tests in replacement of slow
async ones.
Mocking and patching
When writing unit tests, you need to isolate your components
from external dependencies to avoid slow-running tests and
consuming unnecessary resources. For instance, you don’t want
to call your GenAI model every time you run the test suite,
which is going to be frequent, as that’ll be compute-intensive
and possibly expensive.
Instead, you can use test doubles to simulate real dependencies
in your unit tests without having to rely on external
dependencies in your tests. In essence, they pretend to be the
real thing, just like stunt doubles in action movies that pretend
to be the main actors. Isolated unit tests that use test doubles
can verify the component state changes or behavior as it
interacts with external dependencies like an LLM API.
WARNING
Be careful not to replace any component behavior you’re trying to test with test
doubles.
For example, if you have a ChatBot class that uses an LLM API and performs
content filtering on the responses, replace only the LLM API calls with test doubles,
not the content filtering logic. Otherwise, you’ll be testing your own test double.
There are five types of test doubles that you can use in your
unit tests, as shown in Figure 11-9.
Figure 11-9. Test doubles
These include the following:
Fake
A simplified implementation of a dependency for testing
purposes
Dummy
A placeholder used for when an argument needs to be
filled in
Stub
Provides fake data to the system under test that is using it
Spy
Keeps track of dependency usage for later verification
Mock
Checks how the dependency will be used and causes
failure if the expectation isn’t met
Except mocks that verify component behavior, the rest of these
doubles can be used to verify state changes. Mocks have an
entirely different setup and verification logic but work exactly
like the other doubles in making the component being tested
believe that it’s interacting with the real dependencies.
Let’s see each double in action to understand their similarities
and differences.
Fakes
Fake objects are fully functional but simplified versions of the
real dependency, possibly taking shortcuts. An example would
be a database client that uses an in-memory database during
tests instead of an actual database server; or an LLM client that
fetches cached responses from a local testing server instead of
an actual LLM.
Example 11-8 demonstrates what a fake LLM client looks like.
Example 11-8. Fake test double
class FakeLLMClient:
def __init__(self):
[Link] = dict()
def invoke(self, query):
if query in [Link]:
return [Link](query)
response = [Link]("[Link]
if response.status_code != 200:
return "Error fetching result"
result = [Link]().get("response")
[Link][query] = result
return result
def process_query(query, llm_client, token):
response = llm_client.invoke(query, token)
return response
def test_fake_llm_client(query):
llm_client = FakeLLMClient()
query = "some query"
response = process_query(query, llm_client, t
assert response == "some response"
A fully functional and simplified version LLM client that
mimics the behavior of the real one by interacting with a
local testing server.
Return cached responses if repeated prompts are used.
Dummies
Dummies are objects that aren’t used in tests, but you pass
around to satisfy parameter requirements of functions. An
example would be passing a fake authentication token to an API
client to prevent errors, even though the token isn’t used for
authentication during the test.
Example 11-9 shows how dummies can be used as test doubles.
Example 11-9. Dummy test double
class DummyLLMClient:
def invoke(self, query, token):
return "some response"
def process_query(query, llm_client, token):
response = llm_client.invoke(query, token)
return response
def test_dummy_llm_client(query):
llm_client = DummyLLMClient()
query = "some query"
response = process_query(query, llm_client, t
assert response == "some response"
Notice the token is not being used but is required to
satisfy the .invoke(query, token) function signature.
Stubs
Stubs are simplified versions of fakes. They don’t have fully
functional implementations and instead return canned
responses to method calls. As an example, a stub LLM client will
return a predefined fixture string when called without making
any actual model requests.
Example 11-10 shows what a stub looks like. Can you spot the
differences when comparing this example with Example 11-8?
Example 11-10. Stub test double
class StubLLMClient:
def invoke(self, query):
if query == "specific query":
return "specific response"
return "default response"
def process_query(query, llm_client):
response = llm_client.invoke(query)
return response
def test_stub_llm_client():
llm_client = StubLLMClient()
query = "specific query"
response = process_query(query, llm_client)
assert response == "specific response"
Return a canned response on a given condition.
Spies
Spies are like stubs but also record method calls and
interactions. They’re extremely useful when you need to verify
how a complex component interacts with a dependency. For
example, with a spy LLM client, you can verify the number of
times it was invoked by the component under test.
Example 11-11 shows a spy test double in action.
Example 11-11. Spy test double
class SpyLLMClient:
def __init__(self):
self.call_count = 0
[Link] = []
def invoke(self, query):
self.call_count += 1
[Link]((query))
return "some response"
def process_query(query, llm_client):
response = llm_client.invoke(query)
return response
def test_process_query_with_spy():
llm_client = SpyLLMClient()
query = "some query"
process_query(query, llm_client)
assert llm_client.call_count == 1
assert llm_client.calls == [("some query")]
Keep track of function calls and arguments passed in.
Mocks
A mock is a smarter stub. If you know in advance how many
times a dependency is called and how (i.e., you have