Building Generative AI Services

with FastAPI
A Practical Approach to Developing Context Rich
Generative AI Applications

With Early Release ebooks, you get books in their earliest form—the
author’s raw and unedited content as they write—so you can take
advantage of these technologies long before the official release of these
titles.

Ali Parandeh
Building Generative AI Services with FastAPI
by Ali Parandeh
Copyright © 2025 Ali Parandeh. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North,
Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales
promotional use. Online editions are also available for most titles
([Link] For more information, contact our
corporate/institutional sales department: 800-998-9938 or
corporate@[Link].

Acquisitions Editor: Amanda Quinn

Development Editor: Rita Fernando

Production Editor: Clare Laylock

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Kate Dullea

March 2025: First Edition

Revision History for the Early Release
2024-03-01: First Release

See [Link] for release

details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc.
Building Generative AI Services with FastAPI, the cover image, and related
trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the author and do not
represent the publisher’s views. While the publisher and the author have
used good faith efforts to ensure that the information and instructions
contained in this work are accurate, the publisher and the author disclaim all
responsibility for errors or omissions, including without limitation
responsibility for damages resulting from the use of or reliance on this
work. Use of the information and instructions contained in this work is at
your own risk. If any code samples or other technology this work contains
or describes is subject to open source licenses or the intellectual property
rights of others, it is your responsibility to ensure that your use thereof
complies with such licenses and/or rights.
978-1-098-16024-1
[FILL IN]
Brief Table of Contents (Not Yet
Final)

Part I. AI Service Development

Chapter 1: Introduction
Chapter 2: Getting Started with FastAPI
Chapter 3: AI Integration and Model Serving (unavailable)
Chapter 4: Implementing Type Safe AI Services (unavailable)
Part II. Enabling Real-time Capabilities
Chapter 5: Asynchronous Programming in AI Workloads (unavailable)
Chapter 6: Real-Time Communication with Generative Models
(unavailable)
Chapter 7: Integrating Databases to AI services with Prisma ORM
(unavailable)
Part III. Security, Testing and Deployment
Chapter 8: Authentication and Authorization (unavailable)
Chapter 9: Testing AI Services (unavailable)
Chapter 10: Deployment and Containerization (unavailable)
Chapter 11: Deployment and Containerization (unavailable)
Chapter 12: Advanced FastAPI Patterns and Technique (unavailable)
Chapter 13: Conclusion and Future Directions (unavailable)
 Part I. AI Service Development

This part will cover all the necessary steps to set up or migrate to a FastAPI
project that will power our generative backend. We will integrate a variety
of generative models into our FastAPI app and expose endpoints to interact
with our generative models.
Chapter 1. Introduction

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the
author’s raw and unedited content as they write—so you can take
advantage of these technologies long before the official release of these
titles.
This will be the 1st chapter of the final book. Please note that the
GitHub repo will be made active later on.
If you have comments about how we might improve the content and/or
examples in this book, or if you notice missing material within this
chapter, please reach out to the editor at rfernando@[Link].
 CHAPTER GOALS
In this chapter, you will learn about:

The role of generative AI (GenAI) in powering future applications

The barriers to broader adoption of AI services
Methods for increasing the autonomy of GenAI systems
The capstone project you will build as you read and code along
with this book

By the end of this chapter, you should be able to identify the role of
GenAI within the roadmap of your own applications and its associated
challenges.

Why Generative AI Services Will Power

Future Applications
We use computers to automate solutions to everyday problems.
In the past, automating a process required you to manually write code,
which could become tedious for complex issues like spam detection.
Nowadays, you can develop a model by training it with sufficient data that
contains the necessary patterns to understand the nuances of the business
process. Once trained, this model can then replace your manually written
application code.
This gave rise to a wave of AI-powered applications in the market, solving
a range of problems including price optimization, product recommendation
or weather forecasting. As part of this wave, generative models emerged
that differed from other types of AI in their ability to generate novel outputs
rather than just predicting, analyzing or classifying data.
As a software engineer, I believe these models have certain capabilities that
will drive the roadmap of future applications. They can:
 1. Facilitate the creative process
2. Suggest contextually relevant solutions
3. Personalize user experience
4. Minimize delay in resolving customer queries
5. Act as an interface to complex systems
6. Scale and democratize content generation

Let’s look at each capability in more detail.

Facilitating the Creative Process

Mastering skills and acquiring knowledge are cognitively demanding. You
can spend a long time studying and practicing something before you form
your own original ideas for producing novel and creative content like an
essay or a design. The creative process requires a deep understanding of the
purpose behind your creation and a clear awareness of sources for
inspiration and ideas you intend to draw upon. Often when you sit down to
make something new like an original essay, you may find it difficult to start
from a blank piece of paper. You will need to have done extensive research
on a topic to have formed your own opinions and the narrative you want to
write about. This also applies to design. For instance, when designing a user
interface, you may need a few hours of design research by browsing design
websites for ideas on the color palette, layout and composition before you
originate a design. Creating something truly original by looking at a blank
canvas can feel like climbing a wall bare-handed. You will need inspiration
and have to follow a creative process.
Producing original content requires creativity. Creativity involves complex,
non-linear thinking and emotional intelligence that makes it challenging to
replicate or automate with scripting and algorithms. Yet it is now possible to
mimic creativity with generative AI.
It is an extraordinary feat for humans to produce original ideas from
scratch. New ideas and creations are often based on inspirations, connecting
ideas and adaptations of other works.
Generative AI allows us to streamline this method by bridging various ideas
and concepts from an extensive repository of human knowledge. These
models can embed knowledge repositories into “latent” mathematical
spaces and then leverage interpolation techniques to navigate this space to
produce new content which never existed in the model’s training data. With
a prompt, you can then navigate the latent space and request content to be
generated in any shape or form to solve a problem or query.
This paves the path towards a new wave of AI-powered applications that
facilitate the creative process. During the creative process, you may have
the writer’s block, difficulties with imagining and visualizing scenes,
navigating ideas, creating narratives, constructing arguments and
understanding relationships between concepts. You can overcome these
challenges and reduce the cognitive load by interacting with a GenAI
service. You may also stumble upon novel ideas that required understanding
a large body of interconnected knowledge and the comprehension of the
interactions between several concepts.
Another place where GenAI plays a role during the creative process is in
assisting with imagining hard to visualize concepts. Here is an example of a
concept that can be hard to visualize:
Example 1-1.
grayscale 4k, In the distance, two moons overlap in a perfect
eclipse,
their edges shimmering with an impossible spectrum of gray,
casting an ethereal glow over the scene.

This can be quite difficult to imagine unless you have been accustomed to
imagining such concepts. However, with the help of a generative model,
anyone can now visualize and communicate challenging concepts to others.
Providing the scene description in Example 1–1 to a GenAI tool such as
Midjourney, produces an output as shown in Figure 1–1:
 Figure 1-1. Midjourney [Source: [Link]

It is fascinating to see how these GenAI tools can help you visualize and
communicate challenging concepts.
These tools allow you to expand your imagination and nudge your
creativity. When you feel stuck or find it difficult to communicate or
imagine novel ideas, you can turn to these tools for help.
In the future, I can see applications including similar features to assist users
in their creative process. If your applications give several suggestions for
the user to build upon, it can help them get onboarded and build
momentum.

Suggesting Contextually Relevant Solutions

Often you will find yourself facing niche problems that do not have a
common solution. Solutions to these problems are not obvious and require a
lot of research, trial and error, consulting with other experts, and reading.
Developers are familiar with this situation as finding relevant solutions to
programming problems can be tricky and not straightforward.
This is because developers must solve problems with a certain context in
mind. A problem cannot be defined without a thorough description of
“circumstances,” and the “situation” arises in the context.
Context narrows down the potential solutions to a problem. With search
engines, you look for sources of information with a few keywords that may
or may not contain a relevant solution. When developers search for
solutions, they paste error logs into Google and be directed to Q&A
programming websites like StackOverflow. Developers must then hope to
find someone who has encountered the exact same problem in the same
context, and that a solution has been provided. This method of finding
solutions to programming problems is not very efficient. You may not
always find the solution you’re looking for as a developer on these
websites. As a result, there has been a significant decrease in traffic to these
sites since the introduction of ChatGPT and GitHub Copilot.
Developers are now turning to generative AI to solve programming issues.
By providing a prompt that describes the context of a problem, the AI can
generate potential solutions. It’s then up to you to determine if the proposed
solution is appropriate. While this method does require some understanding
of the problem area, it’s often quicker than searching for solutions on online
forums and websites. This is because the AI model can generate solutions
that are not only contextually relevant but also based on its learned
knowledge base that may be sourced from those online forums and
websites. Importantly, this approach isn’t just confined to programming
issues.

Personalizing the User Experience

Customers and users of modern software expect a certain level of
personalization and interactivity when they use modern applications.
Previously providing this feature required having complex onboarding
forms and performing an excessive collection of user data, data analysis,
and user segmentation. You would then adjust the user experience based on
these preference settings. Now, you can provide users access to a large
language model with access to relevant systems such as a product database
to personalize their experience and application interface without having to
know much about them. They can talk to the model and ask it for what they
want. Then, the model can perform actions on the system to adjust to the
user’s preferences or direct them to the relevant resource or pages.
When you interact with a service powered by a generative model, you can
ask for the information you seek or an action to be performed on your
behalf. For example, when browsing a travel planning site, you can describe
your ideal holiday and have a bot prepare an itinerary for you based on the
platform’s access to airlines, accommodation providers and the database of
package holiday deals. Or, if you have already booked a holiday, you can
ask for sightseeing recommendations as it can read the itinerary details from
your account data.
Large Language Models can act as a customer service agent by asking
relevant questions until they map customer preferences and unique needs to
a product catalog for generating personalized recommendations. This is
similar to having a personal virtual assistant that understands your desires
and suggests choices for you to consider. If you don’t like the suggestions,
you can provide some feedback to the assistant to fine-tune the suggestions
to your liking.
In education, these GenAI models can be used to describe or visualize
challenging concepts tailored to each student’s learning preferences and
abilities.
In gaming and Virtual Reality (VR), GenAI can be used to construct
dynamic environments and worlds based on user’s interaction with the
application. For instance, in Role-Playing Games (RPG)s you can produce
narratives and character stories on the fly, based on user’s decisions and
dialog choices in real-time using a baked-in large language model. This
process creates a unique experience for the gamer and users of these
applications.
These examples only scratch the surface of all possible features that can be
integrated into existing applications. This flexibility and agility of
generative models opens up many possibilities for novel applications in the
future.

Minimizing Delay in Resolving Customer Queries

Businesses selling products or services often need a customer service
function. As the business grows in operational complexity and customer
numbers, resolving customer queries in a timely manner can become
expensive and require extensive staff training.
GenAI can streamline this process for the customers and the businesses.
Customers can chat or go on a call with a large language model capable of
accessing databases and relevant sources based on the nature of the query.
For example, you can As customers describe their issues, the model can
address these queries in accordance with business policies and can direct
customers to relevant resources when necessary.
Autonomous bots will be the first point of contact for customers who want
their queries swiftly answered before cases are escalated to human agents.
As a customer, you may also prefer talking to a bot first if it means avoiding
long queues and achieving a quick resolution.

Acting as an Interface to Complex Systems

Many people these days still face problems when interacting with complex
systems such as databases or developer tools. Non-developers may need to
access information or perform tasks without having the necessary skills to
execute commands on these complex systems. LLMs and GenAI models
can act as the interface between these systems and their users.
Users can provide a prompt in natural language, and GenAI models can
write and execute queries on complex systems. For instance, an investment
manager can ask a GenAI bot to aggregate the performance of the portfolio
in the company’s database without having to submit requests for reports to
be produced by specialists.
With the rise of new AI startups, there will be novel AI-powered
applications where users can chat with a bot in natural language to perform
actions. There won’t be without a requirement anymore for clicking around
the application or following any complex workflows.

Scaling and Democratizing Content Generation

People love new content and are always on the lookout for new ideas to
explore. Writers can now research and ideate when writing a blog post with
the help of GenAI tools. By conversing with a model, they can brainstorm
ideas and generate outlines.
The productivity boost is enormous for content generation. You no longer
have to perform low-level cognitive tasks of summarizing research or
rewording sentences yourself. The time it takes to produce a quality blog
post is slashed from days to hours. You can now focus on the outline, flow
and structure of the content then use Gen AI to fill in the gaps, polish and
refine the content. It can really help when you struggle with sequencing the
right words for clarity and brevity.
Many businesses have already started utilizing these tools to explore ideas,
draft documents, proposals, social media and blog posts.
These are several reasons why I believe more developers will be integrating
Gen AI features into their applications in the future. However, the
technology is still in its infancy and there are still many challenges to
overcome before it can be widely adopted.

What Prevents the Adoption of Generative AI

Services
Organizations face several challenges when adopting generative AI
services. There are issues related to the inaccuracy, relevance, quality, and
consistency of the outputs generated by the AI. In addition, there are
concerns about data privacy, cyber-security, and potential abuse and misuse
of the models if used in production. Integrating the AI service with existing
systems, such as internal databases, web interfaces, and external APIs, can
pose a challenge. This integration can be challenging due to compatibility
issues, the need for technical expertise, potential disruption of existing
processes, and similar concerns about data security and privacy.
Companies that want to use the service for customer-facing applications
would want consistency and relevance in the model’s responses and to
ensure outputs are not offensive or inappropriate.
There are limitations to the originality and quality of the content that
generative models can produce. As covered before, these tools effectively
bridge between various ideas and concepts that they have been trained
within certain domains. They also cannot produce totally unseen or novel
ideas. Furthermore, they follow common patterns during generation, which
can be generic, repetitive and boring to use without any fine-tuning.
Some challenges - such as data privacy and security issues - can be solved
with software engineering best practices which you will read more about in
this book.
Solution to other challenges requires prompt engineering and fine-tuning of
models to specific domains for improving the relevance, quality, coherence
and consistency of the outputs. With fine-tuning, you re-train the model on
your own data, and with prompt engineering you effectively write prompts
that “code” or “program” the model to produce outputs in a certain way.
Providing more autonomy to the model is another solution to removing
GenAI adoption blockers. Autonomous models will then be able to take
actions on the user’s behalf on gathering additional information and
performing actions in-line with the user’s intent.
Besides from addressing security and scalability challenges, working on
providing autonomy to the models can deliver a significant impact on the
GenAI’s adoption. Let’s look at this point in more detail.

Making Generative Services Autonomous

When you prompt a model with only a few keywords, you will get a
response that may or may not satisfy your intent.
To explain why, let’s look at a search engine and how it works.
Google has invested a lot of capital into building a search engine that can
infer your intent based on the few keywords you enter into the search bar.
The search results then show you pages that closely match your intent.
The fewer keywords you provide, the harder it is for the search engine to
infer what you exactly want and to show you relevant results. For instance,
if you search for “ties”, the search engine has to make assumptions and
infer that your intent was to shop for ties (clothing) versus learning about
ties. If you instead search for “trinity tie” or “types of tie”, there is an
educational intent in your search query, and Google can show the
appropriate results.
This is also true for generative models. When you provide little to no
context as a prompt to the model, the model has to infer your intent and
produce something highly likely correlated with your intent. A generic
response would satisfy this type of query the most. And, that is why you
would often get a generic output to a generic or low keyword prompts.
The more detailed and contextually rich your queries are, the better and
more relevant the responses from your model will be. Another option is to
fine-tune your model to assume an intent. Otherwise, the model will return
non-relevant or generic responses.
That is why context-rich prompts are crucial to obtaining relevant, specific
and high quality results when working with generative models.
However, you may not always know the full context of your query in
advance. For instance, you cannot know without looking at a customer’s
profile what product description to generate for a sales brochure that would
maximize the chances of a customer making a purchase. In such scenarios,
you will want to give your model a series of tools and abilities that makes it
autonomous. But how do you go about doing that?
Prompting techniques can allow you to translate between natural language
and SQL, so that the model can talk to the database given a database
schema. Potentially using these techniques, the model can look up
information in a database and respond back or perform certain actions such
as updating a record. This is particularly useful when asking challenging
questions that require multiple sub-queries, joins and aggregations to obtain
the answer. In this scenario, the model can do the heavy lifting for you.
However, text-to-SQL techniques are still in development and are
specialized to mostly work with relational databases. They will not yet be
able to work with external systems and APIs. Their probabilistic nature
reduces the accuracy and reliability of their results. Because of this, you
may not want to use them in production scenarios.
To provide context to your model from different sources including
databases and APIs, a reliable solution is to wrap the model with a web
server. The web server then obtains explicit instructions—via an HTTP
request—to query the database and external services for enriching the
prompt context given to the model.
With this approach, you will not need to fine-tune your model or use
separate models to generate code. However, you will have to build a client
for your databases and any API you want to interface with so that your web
server can make the calls instead on behalf of your model.
Writing code to connect with external services may be less efficient than if
your model could generate and execute its own code, but it provides more
control and determinism when retrieving data from these services. In
addition, it safeguards the systems from any malicious code generated by
the model if it is directly connected to services. Additionally, You will also
have the ability to moderate what data is returned to the user or the model
based on their permission levels and system restrictions. Furthermore, you
can still maintain the system if schema of databases and external resources
change constantly. With the text-to-SQL and similar approaches, the model
may face issues interfacing with services that change schemas periodically.
You can auto-generate database clients to match the updated databases
schemas. I will cover auto-generating database clients with Prisma in
chapter 7 of this book. External service providers do often provide fully
typed software development kits (SDKs) that reflect their API schemas
changes.

Why Build Generative AI Services with

FastAPI
Generative AI services require performant web frameworks as backend
engines powering event-driven services and applications. FastAPI can
match the performance of Go or [Link] web frameworks while holding
onto the richness of the Python’s deep learning ecosystem. Non-Pythonic
frameworks lack this direct integration required for interacting with a
generative AI model within one service.
Within the python ecosystem, there are several core web frameworks for
building API services. The most popular options include:
Django
A full-stack framework that comes with batteries included. It
is a mature framework with a large community and a lot of
support.

Flask
A micro web framework that is lightweight and extensible.

FastAPI
A modern web framework built for speed and performance.
It is a full-stack framework that comes with batteries
included.

FastAPI, despite its recent entry into the Python web framework space, has
gained traction and popularity. As of writing, it is the second most popular
Python web framework on GitHub and on the trajectory to become more
popular than Django based on its GitHub stars as shown in Figure 1–1.
 Figure 1-2. GitHub star history of popular Python web frameworks - as of October 2023 [Source:
[Link]

It is also the fastest growing Python web framework in terms of package

downloads.
Flask is leading the number of downloads due to its reputation, community
support and extensibility. However, it is important to note that Flask is a
micro web framework and does not provide the same features as FastAPI.
FastAPI is a full-stack framework that provides a lot of features out of the
box such as data validation, type safety, automatic documentation, and a
built-in web server.
Among the frameworks mentioned, Django falls short in performance
compared to FastAPI and Flask lacks out-of-the-box support for schema
validation. Because of this, developers familiar with Python may be
switching from opinionated and older frameworks like Django to FastAPI. I
assume the exceptional developer experience, development freedom,
excellent performance, and the recent AI model serving support via
lifecycle events may be contributing to this.
This book covers the implementation details of developing generative AI
services that can autonomously perform actions and interact with external
services, all powered by the FastAPI web framework.
To learn the relevant concepts, I will be guiding you through a capstone
project that you can work on as you read through the book. Let’s take a
look.

Overview of the Capstone Project

In this book, I will lead you through building an autonomous generative AI
service using FastAPI as the underlying web framework. The application
you will build is a job application assessment bot capable of assessing
candidates for technical positions. It will fetch information from various
sources, listing sources and performs actions on databases and external
services as needed to reach its own conclusion.
The service will:

Be integrated with multiple models including a large language model

and a Stable Diffusion model
 Generate real-time responses to user queries as text, audio or image
Utilize the Retrieval Augmented Generation (RAG) technique to read
uploaded application documents with a vector database when
responding
Scrape the web and communicate with internal databases, external
systems and APIs to gather sufficient information when responding to
queries
Collaborate with other models to produce outputs in a variety of
format (text, audio, video)
Restrict responses based on user permission and prompt hijacking
attempts
Provide sufficient protections against mis-use and abuse
Produce monitoring action logs and conversation histories for human-
in-loop reviews
As the focus of this book is on building a service, I will be providing you
with the relevant user interface (UI) code with the accompanying code
repository, built on React—a popular frontend UI library.

Summary
In this chapter, we looked at the role of AI in software development and
how it has helped automate complex business problems using data. We then
discussed how GenAI differs from other types of AI models - that it can
generate novel text, audio, images or code from learned patterns in its
training data.
Later on, you will learn why GenAI can drive the roadmap of future
applications due to its capabilities in facilitating the creative process,
eliminating intermediaries, personalization of user experience and
democratization of access to complex systems and content generation.
Further on, you were introduced to several challenges preventing
widespread adoption of GenAI alongside several solutions, including
enabling these models to be autonomous.
Finally, you learned more about the project that you will build as you
follow tutorials in this book.
In the next chapter, we will talk in more detail about a web framework that
you can use in building autonomous GenAI applications.
Chapter 2. Getting Started with
FastAPI

A NOTE FOR EARLY RELEASE READERS

With Early Release ebooks, you get books in their earliest form—the
author’s raw and unedited content as they write—so you can take
advantage of these technologies long before the official release of these
titles.
This will be the 2nd chapter of the final book. Please note that the
GitHub repo will be made active later on.
If you have comments about how we might improve the content and/or
examples in this book, or if you notice missing material within this
chapter, please reach out to the editor at rfernando@[Link].
 CHAPTER GOALS
In this chapter, you will learn about:

What FastAPI is alongside its capabilities and limitations

Why FastAPI is a good choice for building GenAI services
how to get started with creating your own FastAPI project
How to structure FastAPI projects
The onion or layered software design pattern
Comparison of FastAPI to other frameworks
How to migrate to FastAPI from other web frameworks
By the end of this chapter, you should feel comfortable with the
FastAPI web framework, setting up FastAPI projects, migrate your
existing projects to FastAPI and articulate your tech stack decisions for
building GenAI services.

Introduction to FastAPI
FastAPI is an Asynchronous Gateway Interface (ASGI) web framework
that enables you to build lean APIs and backend web servers. Being an
ASGI framework means that it can leverage parallelism to process web
requests. It is fast comparable to modern frameworks but also ships with
auto-documentation, data validation and serialization features.
Effectively, FastAPI is a wrapper over the Starlette framework, built by
encode-the same people who built Django-rest-framework. It is light-
weight and has similar usage signature as Flask during development. To
create a simple web server that has one endpoint in FastAPI, all you have to
write is 10 lines of code as shown in Example 2-1.
Example 2-1. A simple FastAPI web server
# [Link]

from fastapi import FastAPI

import uvicorn

app = FastAPI()

[Link]("/")
def root_controller():
return {"data": "Getting started"}

if __name__ = "__main--":
[Link]("main:app", port=8000)

This then starts a web server for you that is accessible from
[Link]
The app object—that is created from the FastAPI class—converts your
Python function with a decorator into an HTTP endpoint. You can trigger
this endpoint by sending an HTTP request.
Finally, uvicorn takes the app object and starts the web server with your
decorated functions as API routes.
In addition to getting a simple web server out of the box, you also get API
documentation automatically generated for you. The documentation follows
the OpenAPI standard which includes an [Link] specification of
your web service and a Swagger documentation page built from the same
specification file.
You can access the auto-generated docs page by going to the /docs route
of your server via [Link] to see a page
similar to Figure 2–1.
From the Swagger docs page, you can send requests to your API to quickly
test an endpoint. The docs page will also take care of send the correct
request headers, methods and parameters.
What I love about the Swagger docs page is that you can quickly prototype
endpoints via the user interface which can be quicker than writing tests
when iterating over the design of your APIs. However, this doesn’t replace
the traditional testing that can check every endpoint when you make
changes. As your application grows, it is still worth writing tests. Once your
endpoint signatures are further established, you can write pytest tests to
systematically test your web service end-to-end.
In addition to auto-documentation, FastAPI also ships with auto-
serialization and validation of data. In the code example 2-1, we returned a
dictionary in the root controller as tou visit [Link]
There is a bit of magic that must happen for data to show up in your
browser. The data must first be serialised from a Pythonic object such as a
dictionary into a JSON string prior to transfer and then be deserialized back
into a javascript object by your browser once transmission is completed.
This is effectively how applications around the web “talk” to each other.
Serialisation is required when using HTTP for data transmissions as only
text data can be transferred via the HTTP.
When building backend services, you may come across a development
pattern called Inversion of Control. Instead of fully implementing a
function, you can develop a series of utility functions that you inject into
your main function as dependencies.
In FastAPI, these functions are called dependencies and are cached at
runtime during request processing to prevent duplicate computation. You
can inject these dependencies into other functions by passing them as
parameters to Depends() for FastAPI to evaluate and cache your function
outputs.
You can create dependencies for your route controller functions as shown
Example 2-2:
Example 2-2. Using a dependency in FastAPI
# [Link]

from fastapi import FastAPI, Depends

def check_age(age: int) -> bool:

return age >= 18

app = FastAPI()
[Link]("/")
def check_age_controller(is_adult: bool = Depends(check_age)):
return is_adult
...

We define a utility function for checking the age of a person. This function
can then be used as a dependency of the check_age_controller to
inject the results of the age evaluation.
This dependency system is just part of many other features that ship with
FastAPI speed up and ease the process of building backend services.
Now, let’s take a look at how FastAPI compares to other web frameworks.

FastAPI Features and Advantages

Let’s say you want a seamless framework that allows you to effortlessly
create your generative AI services, without any unnecessary struggles. This
framework should include all the essential elements for security,
authentication, and performance, while still granting you the flexibility to
incorporate any additional features and integrations you require.
FastAPI can meet these criteria, as it hosts several features and advantages
out of the box.

Inspired by Flask Routing Pattern

In both Flask and FastAPI Flask, you can create any route by decorating a
function with a specialized decorator. You can then configure the routes to
accept and validate headers, cookies, body, path and query parameters to
support your implementation.

Handling Asynchronous and Synchronous Operations

When building services, your service must be able to handle multiple
requests by multiple users to increase usage efficiency of its worker nodes’
resources as you scale the service. FastAPI uses its own thread-pool to
handle multiple incoming requests asynchronously as long as you define
your async I/O blocking functions correctly.
FastAPI seamlessly handles both synchronous and asynchronous routes
within the same application. If you define an asynchronous function using
async def, FastAPI will run it asynchronously, allowing it to handle
concurrent requests without blocking other operations. This is particularly
useful when dealing with I/O-bound operations, such as querying databases,
exchanging data with a GPU (for making inferences) or making HTTP
requests. On the other hand, if you define a synchronous function using
def, FastAPI will execute it synchronously for faster CPU-bound
operations. However, keep in mind they may block other requests until they
complete their execution.

Built-in Support for Background Tasks

You can build routes capable of handling long-running tasks (e.g., sending
emails) without the need of external libraries (e.g., celery). This feature is
handy when working with generative models that need some time to
generate an output.
Not all tasks can be responded to within the patience tolerance of your
users. You don’t want to keep them waiting while the process is finished.
With background tasks, you can let the client know that your service has
accepted and queued the request to process it in the background.
This feature is mostly useful when working with large models that need
some time (even on a GPU) to process a request. For instance, for text-to-
speech applications that need realistic audio outputs not just voice
synthesis.

Freedom to customize any service layer

There are times when you may want to break away from the limitations of
your current web framework. FastAPI provides a solution to this by
allowing you to define custom classes that inherit base classes of Starlette—
the underlying web framework.
This enables you to implement features according to your preferences
without having to struggle against FastAPI. For example, you can
effortlessly create and use your own global serializer to efficiently handle
datetime objects.

Data validation and serialization

For applications that handle large amounts of data, it’s important that the
data you’re about to process is clean and in a known format.
As the complexity of your service grows, you will want to perform data
validation and serialization. In FastAPI, common data types like lists,
dictionaries, and primitives are automatically serialized. You can return lists
and dictionaries in API routes which by default are auto-serialized.
However, for strict data validation, it is possible to define and pass Pydantic
schemas to your route handlers for validating request/response data.
For instance, you can check if a passed-in query parameter value like age
is not negative. Pydantic validators also allow you to validate more
complex data types at runtime like emails, URLs, UUIDs, and more. This
enables you to catch, handle and protect your services from data issues that
are not captured by static type checkers like mypy.

Automatic documentation
With FastAPI, Swagger UI documentation is auto-generated for you to view
and test any routes you create. You can turn off this feature in production.
However, during development, having an interactive docs page allows for
easier and faster debugging and prototyping of your routes until you build
and maintain your own test suites.

Dependency injection system

Having to manage database connections, making external API calls or
checking user credentials in every route handler is tedious and violates the
Don’t Repeat Yourself (DRY) principles of programming.
FastAPI’s dependency injection system allows you to extract commonly
used logic as dependency functions and inject their outputs inside your
routers and handlers in a way that you still have access to the handler’s
request and response objects. All computed dependency outputs are cached
during the lifetime of a request to be reused. This saves you from having to
make several calls to external APIs for a single or repeated requests.
You can also have dependencies injected into other dependencies creating a
hierarchical dependency graph, which is extremely useful for building
cached authentication and authorization flows or complex decision trees
when implementing your application’s business logic.
Another use-case of this feature is to dynamically construct prompts for AI
models.

Lifecycle events (startup / shutdown)

FastAPI’s startup or shutdown events are excellent for handling
initialization and cleanup of your app. For instance, to load a model into
memory, then unload and cleanup temporary artifacts before exiting the
app.

Custom middleware and CORS support

Middlewares help you intercept communications between the client and
your FastAPI service. It exposes the request and response objects for you so
that you can add global logic around how requests should be processed
before and after being handed off to the route handlers and controllers. It is
a common pattern in backend development to use middlewares to add extra
headers to a response, perform authentication and authorization checks, log
and monitor communications, and much more.

Security and authentication components

As with any other framework, you will need baked-in security and
authentication components to secure your service. These are included
within FastAPI. You can also integrate with third-party authentication
providers via plugins to implement Single Sign On flows in enterprise
environments.

Rich ecosystem of plugins

Plugins are Python packages that hook into FastAPI internals and existing
features. They are similar to any other python package you install and
import into your scripts, so you don’t have to configure anything after
installation. Integrating them means extending the functionalities of your
service without having to deal with order of integrations or compatibility
issues. And, you can also remove them without breaking your app. Some of
the well-known plugins include FastAPI Filters, Auth Users, Rate Limiting
and many more.

Bidirectional web socket, GraphQL and custom response

support
When building services, you will often need to move beyond the standard
REST endpoints. For instance, if you’re building a chat application, you
need real-time communications with the server and the client. This is
achievable with web-sockets. You can also use Streaming Server Sent
(SSE) endpoints to stream LLM outputs to the client and in other cases,
expose GraphQL endpoints. FastAPI uses Python strawberry to construct a
schema for you so that your client does not need to over-fetch data from
your service.

Modern Python and IDE integration with sensible defaults

Since the FastAPI tech stack is built on modern Python, all IDE linters and
formatters can natively check and format your codebase. The defaults are
also sufficient to get you started by importing and instantiating the FastAPI
class. Because everything ties neatly with modern IDE and Python features,
anyone can easily get started building, testing, debugging and deploying
their own FastAPI services.

FastAPI Limitations
Given the aforementioned features and benefits, there are also several
drawbacks and tradeoffs you must consider if you are going to adopt
FastAPI for your project. With AI use-cases in mind, FastAPI falls short in
these areas:
Inefficient model memory management
FastAPI does not provide built-in mechanisms for sharing model memory
between multiple instances or processes of the same container. This means,
when scaling web workers horizontally, you need to load a whole new
model instance into the container’s memory. This creates a memory
bottleneck and increases operational costs of high traffic and heavy GenAI
services.

Restricted to Global Interpreter Lock (GIL)

In Python, multithreading can produce unintuitive and often
counterproductive results because of the Global Interpreter Lock (GIL).
FastAPI leverages multi-threading via its thread-pool to asynchronously
handle incoming web requests. However, even with asynchronous
endpoints, the inference requests will block the main event loop, preventing
all other requests from being processed in the main web serving process.
This is because there is no support by native ML libraries for performing
predictions in an async manner. You may consider using multiple web
workers, but because of the first limitation, you cannot scale your solution
due to memory issues.

Lack of support for micro-batch processing inference requests

Deep learning frameworks provide support for vectorization so that
inferences can be batched together, efficiently computed and parallelized.
Unfortunately, in FastAPI prediction requests cannot be batched together to
make predictions efficient.
When scaling services, a solution is to serve heavy models separately and
use FastAPI to authenticate and manage the incoming and outgoing data.

Cannot split requests between CPU and GPU

FastAPI cannot split web requests between the CPU, (which handles
request transformation and validation) and the GPU, (which is a purpose-
built hardware to run ML inference). This can effectively block your CPU
from processing requests when the inference process isn’t even running
there, but on the GPU instead. As this is a big bottleneck when working
with heavier models, it will require a different approach.

Dependency conflicts
When you are deploying ML models, you will face unique challenges
compared to deploying typical web applications. This is due to your model
runtime’s deep coupling with native libraries and hardware. Each
deployment environment can operate on distinct hardware and may require
you to use specific versions of native libraries and containerization
commands.

Lack of Support for Resource Intensive AI Workloads

Despite its incredible capabilities, FastAPI was developed before the rise of
generative AI. As a result, it remains a general purpose web framework
with recent support for AI serving and ML workflows. However, for certain
use-cases - such as serving resource-intensive and complex billion
parameter models, it may be worth exploring other frameworks like
BentoML.
 BENTOML - FASTAPI-INSPIRED FRAMEWORK FOR
RUNNING RESOURCE-INTENSIVE AI MODELS
BentoML is also built on top of Starlette and is designed with FastAPI
patterns in mind but specifically for machine learning. Its architecture
allows for scaling web requests separately from model inference,
providing flexibility in computing distributions.
It addresses unique ML workflow challenges using its Runners,
dependency management, and model versioning systems. Through its
dependency management system, it can effectively speed up
deployments by declaratively auto-generating docker files for you so
that you don’t have to debug complex docker commands to install and
use CUDA libraries for GPU inference.
Later in the book, I will present a FastAPI architecture for resource
intensive AI workflows that uses BentoML as the underlying AI server.
In this architecture, model-serving tasks will be delegated to BentoML
while FastAPI will manage security, caching and business logic for you.

Comparing FastAPI to Other Web Frameworks

All web frameworks focus on providing you with tools for building REST,
GraphQL, websocket and other types of endpoints.
These frameworks are either opinionated - Django (Python) and Nestjs
(Javascript) while others are not - Flask or FastAPI (Python) and Express
(Javascript) give you the option to architect your service; however, you like.
Frameworks are either opinionated or not. Opinionated frameworks, such as
Django (Python) and Nestjs (Javascript), make decisions for you with
certain assumptions about how you will be providing data to your
components. Effectively, they provide structure while restricting what you
are allowed to do. Opinionated frameworks are typically easier to use. On
the other hand, unopinionated frameworks such as Flask or FastAPI
(Python) and Express (Javascript), are more flexible but can give you too
much freedom—many possibilities to achieve the same results”
Because non-opinionated frameworks such as FastAPI give you so much
freedom in building services, you may feel some decision fatigue when
choosing and integrating every single support package yourself. For
instance, to work with a database, you will need to install and integrate
several packages that work well together-- one to access the database, one
to migrate it and another to act as an Object Relational Mapper (ORM).
While doing that, you may run into compatibility issues with older
packages during integration. This makes working with non-opinionated
frameworks difficult, and often you may decide to use an opinionated
framework such as Django, which ships with a tightly integrated and
excellent ORM system for interacting with databases.
Django is a battery-included framework that markets itself as “The Python
web framework for developers with deadlines”. It ships with a fully
integrated and feature-rich ORM system that takes care of your database
migrations and data access needs when you provide the data models.
Additionally, it provides you with an administration panel, a credentials
based user authentication and authorization system, and several web
security features out of the box, so you don’t have to build these yourself.
It’s also been around for a long time, fostering an active community that has
produced excellent documentation, tutorials, and other resources for the
framework. In Django version 4.2, support for async requests has also been
introduced—​allowing you to build concurrency into your services. Django
expects you to adopt the Model View Controller (MVC) architecture
requiring you to define data models, views—​which become routes serving
templated html files—​and controller layers that include the core data
processing and business logic. This makes Django an excellent choice for
monolith progressive web applications (PWA) that deploy as a single
backend with frontend. However, as businesses move towards building
specialized teams for developing backends and frontends, microservice
architectural patterns are becoming more popular. With microservices, you
want to separate your backend and frontend services, build APIs instead of
PWAs and focus on keeping your services as lean as possible. If using
Django, you can still install the django-rest-framework package to build
these APIs, but you will still end up with a heavy application that slows you
down during development, deployment and scaling services. That’s why
non-opinionated frameworks such as Flask are rising in popularity. As is
evident by their number of monthly downloads.
Flask ships with as little code as practically possible for building web
servers. In comparison to FastAPI, Flask doesn’t come packaged with data
validation, auto-documentation, and a dependency injection system that is
often required for building any backend service that is no longer simple or
requires integration with databases and external services. A new web
framework called quart has tried to tackle this problem, which is a good
contender to FastAPI. However, quart at the time of this writing is new and
in comparison to other frameworks does not have a large community of
users and documented resources if you get stuck on a problem.
Another option to consider is Flask. Flask was released in 2010 and
implements a communication protocol called Web Server Gateway
Interface (WSGI) for web serving, meaning that requests are processed
synchronously in comparison to ASGI, which is asynchronous in nature.
Additionally, Flask isn’t designed for handling large number of
simultaneous connections (like an asynchronous framework would)
However, this doesn’t limit the number of parallel requests the server can
handle on its own. In production, you can employ various strategies (like
worker processes or threads) to handle multiple requests concurrently. Also,
because Flask implements WSGI, it does not support websocket endpoints
which are used for maintaining a persistent, bidirectional communication
channel between a client and a server. This is because WSGI does not
natively support websockets. However, you can install Flask extensions to
integrate websocket support.
On the other hand, ASGI-based frameworks give the ability to queue more
requests at the same time via “asynchronous” endpoints, allowing the
framework to handle more requests at scale. To make this happen, the web
framework can use a thread-pool which can be compared to a pool of
workers undertaking tasks in parallel. Once tasks are finished, request
threads share their results with the main web server thread by joining the
main thread. This web server implementation also protects the web server
from existing if a request thread errors out. The web server gathers
information from the worker thread and sends an error response instead.
Modern web frameworks that implement the ASGI standard are not only
more efficient, but also provide backward compatibility for WSGI in case
it’s needed.
This means that Flask will process each request synchronously whereas
FastAPI uses concurrent event loops. Therefore, FastAPI is going to be
much faster with Input/Output (I/O) heavy tasks—for instance, sending data
to the Graphical Processing Units (GPU) and awaiting a response, this
would be blocking an entire worker process in Flask.
In essence, I would recommend Django and other frameworks if you want
to build PWA monoliths, Flask or quart, for simple APIs and frameworks in
other languages if you have more experience with them.
However, if you’re building a backend service that requires AI model
support, connection to external systems and some level of business logic
complexity, I would definitely recommend considering FastAPI as the web
framework of choice.

Setting Up Your Development Environment

Having learned about using FastAPI for creating GenAI services, you’re
now ready to start your own FastAPI project.
Throughout the rest of this chapter, I will guide you through the installation
process of FastAPI and its essential dependencies, enabling you to set up a
basic web server. Additionally, I’ve included a selection of formatters,
loggers, and linters that you can set up to enhance your development
workflow. Moreover, I’ve described various project structures to help you
effectively manage your project as it expands in complexity and size.
Installing Python, FastAPI and Required Packages
Let’s get started with setting your FastAPI project. I like to normally start
with an empty folder and then create a virtual python environment inside
the folder.
Example 2-3.
$ python3 -m venv .venv

This creates a virtual python environment using the same python version as
your global environment. For this project, ensure you have python 3.11
installed in your system. I often avoid latest versions as some deployment
environments and package dependencies may not yet support the latest
python versions.
Once venv package creates your virtual environment in the .venv folder
you can activate it:
Example 2-4.
$ source .venv/bin/activate

You know it is activated if you run the which command:

Example 2-5.
$ which python
>> ./.venv/bin/python

Once activated, you can now install the core packages needed to run the
FastAPI server.
Example 2-6.
$ pip install fastapi uvicorn

The uvicorn package is the bare-bones web server that FastAPI runs on.
fastapi will also install its dependency packages such as starlette
and pydantic.
Before moving forward, let’s set up your Python linters, formatters, and
type checkers for your project.
Setting Up Tooling with IDEs
Python has several third-party packages that allow you to lint, format and a
type check your codebase prior to shipping it into production.
It is the best practice for professional Python developers to use these tools
to catch bugs during development and before merging their changes to the
repository. In fact, I recommend that you run code checks with these tools
against your pull requests and deployments to prevent bugs from appearing
in your services.
Here is a non-exhaustive list of python packages that I recommend
integrating into any project you start:
Linters
These tools analyze source code to flag programming errors,
stylistic errors and unused code snippets
Autoflake: Removes unused imports and variables from
code to improve readability
Flake8: Checks against python enhancement proposals
(PEPs) and code styles

Formatters
Being able to see what you’ve written
Isort: Sorting imports in python modules
Black: Formatting python code for readability
Ruff: Rust-based linter and formatter that is extremely
fast. You may end up using ruff instead of a combination
of flake8, black and isort altogether.

Loggers
Used in parts of the code that gets complex to debug and
monitor your application
 Loguru: Replacing python’s built-in logger module.

Scanners
If you want confidence that you did not commit insecure
code or passwords by chance
Bandit: Vulnerability scanning of your code for
hardcoded passwords

Type Checkers
To catch those bugs that normal linters don’t catch. Also,
great if you want confidence that changes in your schemas
did not break your application
Mypy: A powerful static type checker that can help catch
a lot of bugs in your code
Pylance: a type checker that ships with Microsoft’s Python
extension for VScode
Pyflake: Another common type checker

IDEs such as VSCode or JetBrains Pycharm provide plugins for running

these tools as you type or save your work. They often require some
configuration, but once done, you will get auto-formatting and linting of the
box. In any case, I recommend having a script that lints, checks and formats
your code before you commit to your codebase or deploy it to production.
These are fundamentals of Python programming and software engineering.
They will become crucial when you start working with AI models that can
produce probabilistic outputs as well as external services and databases that
can change schemas at anytime. Maintaining an AI application that changes
schemas and prompts constantly without the aforementioned tools can
definitively become a headache quite fast.
Creating a Simple FastAPI Web Server
Once FastAPI and its dependencies (uvicorn) are installed, you are now
ready to start your own web server. Create a [Link] in root of your
directory as shown in Example 2-7:
Example 2-7. Starter code for a simple FastAPI web server
# [Link]

from fastapi import FastAPI

import uvicorn

app = FastAPI()

@[Link]("/")
def root_controller():
return {"status": "healthy"}

if __name__ == "__main__":
[Link]("main:app", port=8000, reload=True)

You can now start the server as shown in Example 2-8:

Example 2-8. Run FastAPI web server
$ python [Link]
>> ...
>> Application Startup Complete
>> Listening on [Link]

If you visit [Link] in your browser and should see

{"status": "healthy" message. You now have a fully working
server that auto updates as you update the code. Feel free to change the
message and refresh the browser to see your changes reflected in real-time.
In the rest of the book, I will show you how to wrap a FastAPI server
around a generative model and other services as well as how to implement a
working health endpoint.

Building Larger FastAPI Applications

Often when you’re working on a real-world project, you will end up
building services that will span multiple modules, packages and nested
directories. The decision on how to structure your project is going to be
totally up to you.
This is where most people will struggle and end up with a codebase too
overwhelming to navigate. You will end up frustrated having to understand
the codebase and project structure before you can contribute to it. At some
point, the complexity will grow so much that you will dread touching the
project again.
Some files will end up too large to read with bloated functions, or there will
be too many files scattered all over the place. You may also end up having
millions of import errors or circular dependencies breaking your
application.
Learning to structure larger applications will be even more important when
working with generative AI models. These models often need dependencies
and additional utility functions to support them. Therefore, you will have to
add a layer of additional complexity for your models on top of existing
applications layers.
This often happens with frameworks like FastAPI that give you too much
freedom, no structure or an opinionated way of organizing your codebase.
Django helps a lot in this instance by giving you a structure to follow, such
as models, views, templates, etc.
Over the past few years working with FastAPI and data science
applications, I’ve seen many developers come up with their own cookie
cutter templates for starting FastAPI projects. Some even recommend
following a structure popularized by Netflix Dispatch FastAPI project for
larger API applications that has inspired other templates.
When it comes to building real-world applications, you must do everything
you can to keep the codebase as structured as possible. This is for your own
benefit—to help you and others in your team in understanding the code in
the future.
You know you have a good project structure if you can find any function or
component within your codebase. If you start questioning the purpose of a
directory or spending hours searching for a piece of code, then you may
have a structure become too confusing to get your head around.
In these instances, you can refer to a few common project structures that
have recently become popular in the FastAPI community.

FastAPI Project Structures

To keep your FastAPI projects organized when using generative AI models,
there are a few project structures you can adopt.
1. Flat
2. Nested
3. Modular
4. Onion/Layered
Let’s take a detailed look at each one.

Flat Structure
A flat structure is one that application files remain at the root of your
project with no nested directories. You may group all your files under a
single directory for better organization.
The main idea here is to keep all similar code in modules and place together
near the root of your project. For instance, put all your database models in a
[Link] or your endpoints in [Link].
By far, the most common project structure is flat due to its simplicity and
ease of use.
This structure is often great for building the first version of a service or a
tiny microservices.
Here is what the structure could look like:
Example 2-9. A flat FastAPI project structure
flat-project
├── app
│ ├── [Link]
│ ├── [Link]
│ ├── [Link]
│ ├── [Link]
│ └── [Link]
├── [Link]
├── .env
├── .gitignore

You can see in the structure shown in example 2–5, you only have a few
files that contain the core of your application logic. If you’re building a
microservice with FastAPI, by definition, you will want to follow a
structure like example 3–9 to maintain simplicity.
The simplicity of the flat structure allows you to focus on the development
than structure. There aren’t many files to worry about. You also don’t need
to care much about coupling, decomposition or reuse as there aren’t many
lines of codes to deal with.
On the other hand, the flat structure could fall apart if you start appending
lots of code to the files. At this point, it makes sense to break down the
global python modules into packages of their own using the nested
structure.

Nested Structure
The nested structure groups similar modules into packages—effectively
creating a nested structure and hierarchy of modules. You group all modules
under a package that are similar in nature irrespective of the feature they
support. These are loosely coupled modules that contain similar logic for
different entities in your project. For instance, the models package may
contain users and profiles database models.
The nested structure is recommended for larger projects by the official
FastAPI documentation.
You can see a project with nested structure in Example 2-10:
Example 2-10. Nested FastAPI Project
nested-project
├── app
│ ├── [Link]
│ ├── [Link]
│ └── services
│ │ ├── [Link]
│ │ └── [Link]
│ └── models
│ │ ├── [Link]
│ │ └── [Link]
│ └── routers
│ ├── [Link]
│ └── [Link]
├── [Link]
├── .env
├── .gitignore

As per the official FastAPI documentation, nested structures are also good
for larger projects and more complex applications. Perhaps as you add AI
models and several external services and databases, this is a structure that
you will want to adopt.
The main pitfall with this project structure is the ambiguous coupling of
modules. Changes in one module can cascade into other modules, and it can
become difficult to understand the cascading effect of new changes. Over
time, it can be challenging to maintain and change the code without
performing many updates everywhere else. This is referred to as shotgun
updates. Shotgun updates in the context of software development are when
it’s challenging to maintain and change the code without performing many
updates everywhere else.
If you anticipate having difficulty managing module coupling or expecting
to deal with a large application, I would recommend using a modular
structure.

Modular Structure
The modular structure - popularized by the Netflix Dispatch FastAPI
project - is similar to the nested structure because you can place multiple
modules within a package and sub-packages. However, the core difference
is in how you organize your project.
In the modular structure, modules that are closely related and refer to a
specific domain are grouped together. This approach differs from the
previously mentioned nested structure. An example could be the users
package that contain user schemas, database services, dependencies and
routers.
To understand this difference better, have a look at Example 2-11:
Example 2-11. Modular FastAPI project structure
modular-project
├── app
│ └── modules
│ │ ├── auth
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ └── users
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ │ ├── [Link]
│ │ └── profiles
............
│ ├── [Link] # global configs
│ ├── [Link] # global models
│ ├── [Link] # global exceptions
│ └── [Link]
├── [Link]
├── .env
├── .gitignore

In modular project structure, you gather closely interconnected by enabling

a feature within the same package. This kind of encapsulation eliminates
any uncertainty regarding the couplings in your code, resulting in improved
scalability and maintainability.
If you need to include more features, you can create a new package that
contains all the necessary code. Similarly, if you need to modify or delete
code, you can easily determine where the changes should be made and
anticipate how they will impact other parts of the code. This is possible
because the structure of the codebase is transparent and well-encapsulated,
making it clear where different components are connected.

Progressive Re-Organization of Your FastAPI Project

A modular codebase allows you to add and remove components with ease.
You can also reuse components across different parts of your system to
avoid repetition.
When you first start your project, modularity is not as important. You can
get started with just a single or a few Python files to build your services
easily. However, as soon as you introduce AI models, external services and
complex business logic, you will want to consider modularizing your
codebase.
You can achieve modularity by designing components of your system with
re-usability and disposability in mind. Make sure the design of your
modules and functions allow for usage in different environments and that
you place them at the right place in your project directory. Selecting the best
project structure is a matter of preference. However, you may be asking
yourself, “Which project structure should I adopt for building generative AI
services with FastAPI?”
I found that the best way to structure projects is to progressively reorganize
your project from a flat to a modular structure as your service complexity
grows:

1. Flat: If you are starting with a new project and the complexity of your
system is not yet clear, you can focus on writing all your FastAPI code
in a single file before worrying about the project structure. You then
extract your code into several files under the root directory. This is the
initial structure you will adopt when experimenting on the first version
of your service from scratch.
2. Nested: As the number of files in your codebase and service
complexity grows, you can adopt the nested structure. You can search
 for files based on logical grouping (models, routers, schemas, etc.) and
don’t have to worry too much about logical couplings in your code. As
you make changes, only a handful of files are affected. At this point,
you have an AI microservice.
3. Modular: As you move from a microservice to a full backend service,
you will want to adopt a modular structure. There is now an increasing
number of modules, features and complexity. You start grouping your
code into packages based on areas of concern. Your code is now
handling requests, authentication, external systems, etc. while serving
an AI model.
I suggest restructuring your project as outlined. However, you have the
flexibility to adopt any organizational scheme that makes sense to you and
allows you to recall the location and purpose of your files.

TIP
Remember, if you can’t justify the file organization in your codebase to another
developer, it’s time to reconsider your existing structure.

As you build your GenAI service, you will inevitably end up with a large
codebase and a complex application. Thinking about the structure of your
large FastAPI application is only the first step in building production-grade
services. In the next step, you will learn more about a software design
pattern that helps you manage the complexity of your AI services. This is
called the Onion or Layered application design that also introduces several
new concepts which we will talk about next.

Onion / Layered
If you plan on building a fully featured backend service for generative AI,
you will benefit to know more about the Onion or Layered application
design pattern which can be implemented within the nested and modular
project structures. The purpose of this pattern is to create a separation of
concerns between the different parts of your application to simplify the
process of adding, removing and modifying features.
The Onion design pattern has also influenced web frameworks in other
languages such as [Link]—see: [Link]
The Onion design consists of layers, each with a specific responsibility and
dependency direction. The innermost layer contains the domain models and
business logic, while the outer layers contain route handling (in an API
service) or user interfacing code (when serving HTML templates).
 Figure 2-1. Onion design pattern

The pattern is called “onion” because the layers build upon each other, with
the domain model at the center, surrounded by layers of increasing
abstraction promoting testability, maintainability, and flexibility in
maintaining your AI services. The core of the application (domain model
and business logic) is at the inner layers, and all other layers depend
inwardly on it. This approach helps to manage dependencies, promote
separation of concerns, and facilitate a more testable and maintainable
codebase.
The main idea behind this pattern is the Dependency Inversion Principle-
that high-level modules should not directly depend on implementation of
low-level modules, but declare what they need from low-level modules by
leveraging the FastAPI dependency system. The dependency system can
then inject the output of the low-level modules to avoid coupling between
layers.
To implement this software design, you break down your service as an
onion consisting of layers that go deeper and deeper. Each layer (as you
move from outer to inner layers) introduces components that are
responsible for a set of tasks:
1. API Routers: Responsible for grouping multiple controllers/route
handlers to apply common logic across several controllers.
FastAPI provides the APIRouter class to help you with this.
2. Controllers / Route Handlers: Responsible for handling incoming
requests and returning responses to the client via a logical execution of
services or providers.
Good controller design always uses dependencies to inject required
data or logic required for its execution.
 Figure 2-2. API Routers and Controllers

3. Services / Providers: Responsible for executing business logic and

interfacing with external systems.
Services inherit repositories when implementing a more complex
business logic than simple data retrieval or mutation operations.
Each module of your application can have a service of its own.
Providers sit at the same level as the services.
The only difference is that providers allow you to interact with
external systems other than a database.
Example of a provider is an email server, A payment system, or other
microservice clients that are used alongside services to support
controller business logic.
Here is how they work together within a route controller: The Users
database service, fetches a user’s email, then passes that to the
payment system and email server clients (providers) for processing
payments and sending confirmation emails.
4. Repositories (Data Adapters): Responsible for using an Object
Relation Mapper (ORM) or raw SQL commands to execute queries on
your infrastructure like a database, or a memory store for retrieving or
mutating data.
You may implement an abstract interface in this layer to enforce
consistent design across all your repositories—following the CRUD
(Create, Read, Update, Delete) operations.
 Figure 2-3. Services, Providers and Repositories

5. Schemas / Models: Responsible for enforcing type-safety, structure and

validation logic on your data as it flows throughout your service.
 Figure 2-4. Models and Schemas

You will also have components that span across the layers to support the
whole application:
1. Middlewares: Handle requests and responses before and after they are
passed to the application controllers/route handlers.
 Figure 2-5. Middlewares

2. Dependencies: Include reusable functions you define that can be

injected into controllers to support a business logic.
Dependencies can be cached and depend on other dependencies.
3. Pipes: Data transformer functions that you can use across application
layers.
4. Mappers: Data mappers from one schema into another, often passing
data across layers. For instance, from a route controller to the database
service.
5. Exception Filters: Consistently handle exceptions across the layers
6. Guards: Secure and protect controllers from abuse.
Authentication and authorization logic can be implemented as
dependencies or middleware to act as Guards.
 Figure 2-6. Guards

In the next chapters of this book, I will show you how to implement the
onion / layered application design within a modular project structure to
create a maintainable, testable and scalable FastAPI generative AI service.
Now that you understand the implementation of FastAPI for larger projects,
you may be curious about transitioning your existing applications to
FastAPI to fully leverage its capabilities. Next, we will look at the
migration process from other frameworks to FastAPI.

Migrating to FastAPI
After learning about the FastAPI features, benefits and drawbacks, project
structures and the onion architecture, you may now want to move your
existing projects from other frameworks into FastAPI.
I want to make this migration process as easy as possible for you. So, I have
provided a few guidelines on migrating from Django, Flask, and other
frameworks that you may find useful.
I will not show the full implementation details in code as that would be out
of the scope of this book, and the details could vary from one project to
another. ChatGPT, GitHub Copilot X, and other LLM code interpreters can
assist you in generating code to carry out the migrations you need.

Migrating from Django

Django projects come with an extensive and well-integrated ORM system
which is implemented via the Models, Views, Controllers (MVC)
framework. If you have an existing API using Django-Rest-Framework,
you will also have user-defined serializers to serialize or deserialize your
data.
To migrate a Django project to FastAPI, you can follow these steps:

1. Re-define serializer models: You will need to re-define all your

serializer models using Pydantic. FastAPI can then use them to
perform serialization / deserialization for you.
2. Integrate an external ORM and database migrations system: If your
application interfaces with a database, FastAPI doesn’t ship with an
integrated ORM and DB migrations system like Django, so you will
also need your own ORM.
 You have a few options for choosing an ORM including
SQLAlchemy, Tortoise-ORM, or Prisma. If you select
SQLAlchemy or Tortoise-ORM, firstly re-define your database models
with Pydantic. Then, install alembic as your DB migrations system
and generate a migration file.

PRISMA ORM
I recommend selecting Prisma Python as the underlying ORM (chapter 7) as it
auto-generates a fully typed database client for you by reading your database
schema.

3. Install third-party plugins: Django ships with a full-featured

authentication system that is hard to re-implement yourself. If you
need to migrate the user authentication system, I recommend using a
FastAPI plugin such as FastAPI-Users or FastAPI-Admin to
implement similar logic. Bear in mind that if you choose any of these
FastAPI plugins, you will have to lock into the ORM that the package
uses under the hood. Alternatively, in chapter 8, I will show you how
to implement your own authentication system with Prisma ORM. You
will still be able to switch ORMs if needed.
4. Migrate controller and route handler Logic: Implement FastAPI route
handlers and migrate controller logic from your Django application.
There may be some work to integrate your controllers logic with
existing plugins and your selected ORM.
5. Migrate Views as Templates: If you are using views, you can migrate
these as ninja templates and create route handlers to serve them as
static files.
Migrating a Django project to FastAPI can be challenging if there are a lot
of views and custom business logic to move over. However, by following
the steps mentioned, you can migrate your Django application in a modular
fashion.
Migrating from Flask
Migrating from Flask to FastAPI is more straightforward than migrating
from Django. This is largely because FastAPI is heavily inspired by the
Flask project.
To migrate to FastAPI from Flask, you only need to follow a few simple
steps:

1. Re-define Flask route handlers as FastAPI handlers

2. Re-define Flask serializer and ORM models as Pydantic Models
3. Install third-party plugins and dependencies that replicate existing
features
4. Migrate over all controller logic

Migrating from Other Web Frameworks

You may want to migrate from other web frameworks (including those of
other languages) into FastAPI. Luckily, this should be fairly straightforward
as FastAPI route handlers are implemented almost the same as other web
frameworks.
However, what can make migrations more challenging is the having to
reimplement business logic in Python from other languages? Libraries and
language features differ, so you may need to use an LLM to speed up
translation of your business logic into Python and FastAPI code.
The steps to perform the migration should be mostly the same as if you are
migrating from Flask.

Summary
In this chapter, you learned about the FastAPI framework, its capabilities
and drawbacks compared to other frameworks.
You also learned how to set up your own FastAPI project from scratch,
alongside a set of tools you can use to improve your development
experience.
Then you were introduced to several project structures you can adopt when
building your own FastAPI service. As part of this, you learned more about
the onion / layered software design pattern to help manage project
complexity.
Lastly, we went over guidelines for migrating over from other web
frameworks into FastAPI.
You should now be comfortable starting your own FastAPI projects, and
how to manage the project complexity as it evolves over time.
In the next chapter, you will learn how to implement your own GenAI
features in FastAPI for generating text, image, audio and video. You will
understand the inner workings of each model and the role of FastAPI
lifecycle system in model serving, while leveraging Nvidia GPUs for
inference tasks. Finally, you will be introduced to the FastAPI background
tasks system to offload long-running inference operations.
About the Author
Ali Parandeh is a chartered engineer with the UK engineering council and a
certified developer with Microsoft Azure and Google Cloud with 8 years of
experience in a variety of sectors including tech, transport, agriculture,
finance and retail. He has led engineering teams both at large multinational
consultancies and at tech startups
 from

Your gateway to knowledge and culture. Accessible for everyone.

[Link] [Link] [Link] [Link]

O cial Telegram channel

Z-Access

[Link]
ffi