Master Document

Building a Simple Engine:

Introduction
Introduction
Welcome to the "Building a Simple Engine" tutorial series! This series marks a transition from the
foundational Vulkan concepts covered in the previous chapters to a more structured approach
focused on building a reusable rendering engine.

A New Learning Approach

While the previous tutorial series focused on introducing individual Vulkan concepts step by step,
this series takes a different approach:

This series targets readers who have completed the Vulkan Tutorial and feel comfortable with the
fundamentals. We’ll emphasize architectural concepts and design patterns over exhaustive API
permutations, so you develop an engine mindset rather than a collection of snippets. Expect to do
more independent work: fill in smaller gaps, experiment, and lean on the Vulkan Guide, Samples,
and Specification as primary references. If a topic feels too advanced, revisit the original tutorial
and return when ready.

What to Expect
The "Building a Simple Engine" series is designed as a starting point for your journey into engine
development, not a finishing point. We’ll cover:

1. Engine Architecture - How to structure your code for flexibility, maintainability, and
extensibility.

2. Resource Management - More sophisticated approaches to handling models, textures, and

other assets.

3. Rendering Techniques - Implementation of modern rendering approaches within an engine

framework.

4. Performance Considerations - How to design your engine with performance in mind.

5. Publication Considerations - How to prepare your application for distribution in a

professional environment, including packaging, deployment, and platform-specific
considerations.

Throughout this series, we encourage you to experiment, extend the provided examples, and even
challenge some of our design decisions. The best way to learn engine development is by doing, and
sometimes by making (and learning from) mistakes.

1
Throughout our engine implementation, we’re using vk::raii dynamic rendering and C20 modules.
The vk::raii namespace provides Resource Acquisition Is Initialization (RAII) wrappers for Vulkan
objects, which helps with resource management and makes the code cleaner. Dynamic rendering
simplifies the rendering process by eliminating the need for explicit render passes and
framebuffers. C20 modules improve code organization, compilation times, and encapsulation
compared to traditional header files.

How to Use This Tutorial

Each chapter builds on the last to assemble a small but capable engine. Read a chapter end‑to‑end
first, then implement; pause to internalize the concepts; and don’t hesitate to revisit the original
Vulkan tutorial when you need a refresher. Treat the code as a starting point—experiment and
extend it with your own features.

Let’s begin our journey into engine development with these chapters:

1. Engine Architecture - How to structure your code for flexibility, maintainability, and
extensibility.

2. Camera Transformations - Implementation of camera systems and transformations.

3. Lighting & Materials - Basic lighting models and push constants.

4. GUI - Implementation of a graphical user interface using Dear ImGui.

5. Loading Models - More sophisticated approaches to handling models, textures, and other assets.

6. Subsystems - Implementation of Audio and Physics subsystems with Vulkan compute

capabilities.

7. Tooling - CI/CD, Debugging, Crash minidump, Distribution, and Vulkan extensions for
robustness.

8. Mobile Development - Adapting the engine for Android/iOS, focusing on performance

considerations and mobile-specific Vulkan extensions.

9. Advanced Topics - Short, focused tutorials that extend the Simple Engine with specific features
and optimizations.

Previous: Main Tutorial Conclusion | Next: Engine Architecture

Getting Started with Example Assets

To follow along with the attachments-based Simple Engine examples and scenes, fetch the Bistro
assets locally.

[The Bistro scene - a detailed outdoor café environment demonstrating the engine's rendering
capabilities] | images/[Link]

• Linux/macOS (default target: attachments/simple_engine/Assets/bistro at repository root):

$ cd attachments/simple_engine

2
 $ ./fetch_bistro_assets.sh

• Windows (default target: attachments\simple_engine\Assets\bistro at repository root):

> cd attachments\simple_engine
> fetch_bistro_assets.bat

The scripts use SSH (git@[Link]:gpx1000/[Link]) and fall back to HTTPS if SSH is
unavailable. If Git LFS is installed, large files will be pulled automatically.

Next, take advantage of the install_dependencies_* scripts to ensure you have all necessary
dependencies.

Once assets are available and dependencies are ready, build and run the Simple Engine examples
under attachments/simple_engine. See the later chapters for details on scene loading and
subsystems referenced by the example code. :pp: ++

Engine Architecture: Introduction

Introduction
Welcome to the "Engine Architecture" chapter of our "Building a Simple Game Engine" series! In
this chapter, we’ll explore the fundamental architectural patterns and design principles that form
the backbone of a modern Vulkan rendering engine.

While this series focuses primarily on building a rendering engine with Vulkan, the architectural
concepts we’ll discuss are applicable to both rendering engines and full game engines. We’ll clarify
which patterns are particularly well-suited for rendering-focused systems versus more general
game engine development.

We’ll start by taking a step back and considering the overall structure of our engine. A well-
designed architecture is crucial for creating a flexible, maintainable, and extensible rendering
system.

What You’ll Learn

This chapter will take you through the foundational concepts that underpin effective engine design.
We’ll begin by exploring architectural patterns—the proven design approaches that game and
rendering engines rely on to manage complexity and enable extensibility. Understanding these
patterns helps you choose the right structural approach for different engine subsystems.

From there, we’ll dive into component systems, which provide the flexibility to build modular,
reusable code. You’ll see how component-based architecture allows different parts of your engine
to work together while remaining loosely coupled, making your codebase easier to maintain and
extend.

3
Resource management forms another crucial pillar of engine architecture. We’ll examine strategies
for efficiently handling textures, models, shaders, and other assets, ensuring your engine can scale
from simple scenes to complex, asset-heavy applications without performance bottlenecks.

The rendering pipeline design we’ll cover shows you how to create a flexible system that can
accommodate various rendering techniques and effects. This foundation will serve you well as you
add more advanced rendering features in later chapters.

Finally, we’ll implement event systems that enable clean communication between different engine
components. This event-driven approach reduces tight coupling and makes your engine more
maintainable as it grows in complexity.

Prerequisites
This chapter builds directly on the foundation established in the main Vulkan tutorial series, so
completing that series is essential. The architectural concepts we’ll discuss assume you’re
comfortable with Vulkan’s core rendering concepts and have hands-on experience implementing
them.

Beyond Vulkan knowledge, you’ll benefit from familiarity with object-oriented programming
principles, as modern engine architecture relies heavily on encapsulation, inheritance, and
polymorphism to manage complexity. Experience with common design patterns like Observer,
Factory, and Singleton will help you recognize when and why we apply these patterns in our engine
design.

Modern C++ features play a crucial role in our implementation approach. Smart pointers help us
manage resource lifetimes safely, templates enable flexible, reusable components, and other
C++11/14/17 features allow us to write more expressive and maintainable code. If you’re not
comfortable with these concepts, consider reviewing them before proceeding.

You should also be familiar with the following chapters from the main tutorial:

• Basic Vulkan concepts:

◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

Why Architecture Matters

The difference between a hastily assembled renderer and a well-architected engine becomes
apparent as soon as you need to make changes or add features. Good architecture creates a
foundation that supports your development process rather than fighting against it.

Maintainability emerges from clean separation of concerns—when each component has a clear,
focused responsibility, you can update or fix individual pieces without worrying about cascading
effects throughout the system. This becomes invaluable when debugging graphics issues or

4
implementing new rendering techniques.

Extensibility flows naturally from modular design. When your architecture provides clear
extension points and interfaces, adding new features becomes a matter of implementing new
components rather than rewriting existing systems. This allows your engine to evolve with your
project’s needs.

Reusability multiplies your development effort. Well-encapsulated components can move between
projects or serve different purposes within the same project. A thoughtfully designed material
system, for example, might work equally well for both game objects and UI elements.

Performance opportunities often emerge from architectural decisions made early in development.
Good architecture enables optimizations like multithreading (by avoiding tight coupling between
systems), batching (through predictable interfaces), and caching (via clear data flow patterns).
While premature optimization is dangerous, premature architecture decisions can make later
optimization impossible.

Let’s begin our exploration of engine architecture with an overview of common architectural
patterns used in modern rendering engines.

Previous: Building a Simple Engine Introduction | Next: Architectural Patterns :pp: ++

Engine Architecture: Architectural

Patterns
Architectural Patterns
In this section, we’ll provide a quick overview of common architectural patterns used in modern
rendering and game engines, with a focus on Component-Based Architecture which forms the
foundation of our Vulkan rendering engine.

Before diving into specific patterns, it’s important to clarify that while we’re building a Vulkan-
based rendering engine in this tutorial, many of the architectural patterns we’ll discuss are
commonly used in both rendering engines and full game engines. A rendering engine focuses
primarily on graphics rendering capabilities, while a full game engine typically includes additional
systems like physics, audio, AI, and gameplay logic.

Overview of Common Architectural Patterns

Here’s a brief introduction to the most common architectural patterns used in game and rendering
engines:

Layered Architecture

Layered architecture divides the system into distinct layers, each with a specific responsibility.
Typical layers include platform abstraction, resource management, rendering, scene management,

5
and application layers.

[Layered Architecture Diagram showing different layers of a rendering engine] |

../../../images/layered_architecture_diagram.png

Key Benefits:

• Clear separation of concerns

• Easier to understand and maintain

• Can replace or modify individual layers without affecting others

For detailed information and implementation examples, see the Appendix: Layered Architecture.

Data-Oriented Design

Data-Oriented Design (DOD) focuses on organizing data for efficient processing rather than
organizing code around objects. It emphasizes cache-friendly memory layouts and bulk processing
of data.

[Data-Oriented Design Diagram] | ../../../images/data_oriented_design_diagram.svg

Key Benefits:

• Better cache utilization

• More efficient memory usage

• Easier to parallelize

For detailed information and implementation examples, see the Appendix: Data-Oriented Design.

Service Locator Pattern

The Service Locator pattern provides a global point of access to services without coupling
consumers to concrete implementations.

[Service Locator Pattern Diagram] | ../../../images/service_locator_pattern_diagram.svg

Key Benefits:

• Decouples service consumers from service providers

• Allows for easy service replacement

• Facilitates testing with mock services

For detailed information and implementation examples, see the Appendix: Service Locator Pattern.

Component-Based Architecture
Component-based architecture is widely used in modern game engines and forms the foundation of
our Vulkan rendering engine. It promotes composition over inheritance and allows for more

6
flexible entity design.

[Component-Based Architecture Diagram showing entities, components, and systems] |

../../../images/component_based_architecture_diagram.png

Diagram Legend:

• Boxes: Blue boxes represent Entities, orange boxes represent Components, and
green boxes represent Systems

• Line Types:

NOTE ◦ Dashed lines show ownership/containment (Entities contain Components)

◦ Solid lines show processing relationships (Systems process specific

Components)

• Text: All text elements use dark colors for visibility in both light and dark modes

• Directional Flow: Arrows indicate the direction of relationships between

elements

Key Concepts

1. Entities - Basic containers that represent objects in the game world.

2. Components - Modular pieces of functionality that can be attached to entities.

3. Systems - Process entities with specific components to implement game logic.

Benefits of Component-Based Architecture

• Highly modular and flexible

• Avoids deep inheritance hierarchies

• Enables data-oriented design

• Facilitates parallel processing

Implementation Example

// Component base class

class Component {
public:
virtual ~Component() = default;
virtual void Update(float deltaTime) {}
};

// Specific component types

class TransformComponent : public Component {
private:
glm::vec3 position;
glm::quat rotation;
glm::vec3 scale;

7
 public:
// Methods to manipulate transform
};

class MeshComponent : public Component {

private:
Mesh* mesh;
Material* material;

public:
// Methods to render the mesh
};

// Entity class
class Entity {
private:
std::vector<std::unique_ptr<Component>> components;

public:
template<typename T, typename... Args>
T* AddComponent(Args&&... args) {
static_assert(std::is_base_of<Component, T>::value, "T must derive from
Component");
auto component = std::make_unique<T>(std::forward<Args>(args)...);
T* componentPtr = [Link]();
components.push_back(std::move(component));
return componentPtr;
}

template<typename T>
T* GetComponent() {
for (auto& component : components) {
if (T* result = dynamic_cast<T*>([Link]())) {
return result;
}
}
return nullptr;
}

void Update(float deltaTime) {

for (auto& component : components) {
component->Update(deltaTime);
}
}
};

8
Why We’re Focusing on Component Systems
For our Vulkan rendering engine, we’ve chosen to focus on component-based architecture for
several key reasons:

1. Flexibility for Graphics Features: Component systems allow us to easily add, remove, or swap
rendering features (like different shading models, post-processing effects, or lighting
techniques) without major refactoring.

2. Separation of Rendering Concerns: Components naturally separate different aspects of

rendering (geometry, materials, lighting, cameras) into manageable, reusable pieces.

3. Industry Standard: Most modern rendering engines and graphics frameworks use component-
based approaches because they provide the right balance of flexibility, maintainability, and
performance.

4. Extensibility: As graphics technology evolves rapidly, component systems make it easier to

incorporate new Vulkan features or rendering techniques.

5. Compatibility with Data-Oriented Optimizations: While we’re using a component-based

approach, we can still apply data-oriented design principles within our components for
performance-critical rendering paths.

While other architectural patterns have their merits, component-based architecture provides the
best foundation for a modern, flexible rendering engine. That said, we’ll incorporate aspects of
other patterns where appropriate - using layered architecture for our overall engine structure,
data-oriented design for performance-critical systems, and service locators for cross-cutting
concerns.

Conclusion
We’ve provided a brief overview of common architectural patterns, with a focus on Component-
Based Architecture which we’ll use throughout this tutorial. For more detailed information about
other architectural patterns, including implementation examples and comparative analysis, see the
Appendix: Detailed Architectural Patterns.

In the next section, we’ll dive deeper into component systems and how to implement them
effectively in your engine.

Previous: Introduction | Next: Component Systems :pp: ++

Engine Architecture: Component

Systems
Component Systems
In the previous section, we introduced several architectural patterns and explained why we’re

9
focusing on component-based architecture for our Vulkan rendering engine. As we established,
component systems provide the ideal balance of flexibility, modularity, and performance for
modern rendering engines. Now, let’s dive deeper into how to implement effective component
systems in your rendering engine.

The Problem with Deep Inheritance

Traditional game object systems often rely on deep inheritance hierarchies:

class GameObject { /* ... */ };

class PhysicalObject : public GameObject { /* ... */ };
class Character : public PhysicalObject { /* ... */ };
class Player : public Character { /* ... */ };
class Enemy : public Character { /* ... */ };
class FlyingEnemy : public Enemy { /* ... */ };
// And so on...

This approach has several drawbacks:

1. Rigidity - Adding new combinations of behaviors requires creating new classes.

2. Code Duplication - Similar functionality may be duplicated across different branches of the
hierarchy.

3. Bloated Classes - Base classes tend to accumulate functionality over time.

4. Difficult Refactoring - Changes to base classes can have far-reaching consequences.

Component-Based Design Principles

Component-based design addresses these issues by favoring composition over inheritance:

1. Single Responsibility - Each component should have a single, well-defined responsibility.

2. Encapsulation - Components should encapsulate their internal state and behavior.

3. Loose Coupling - Components should minimize dependencies on other components.

4. Reusability - Components should be designed for reuse across different entity types.

Basic Component System Implementation

Let’s build a more complete component system based on the example from the previous section:

// Forward declarations
class Entity;

// Base component class

class Component {
protected:

10
 Entity* owner = nullptr;

public:
virtual ~Component() = default;

virtual void Initialize() {}

virtual void Update(float deltaTime) {}
virtual void Render() {}

void SetOwner(Entity* entity) { owner = entity; }

Entity* GetOwner() const { return owner; }
};

// Entity class
class Entity {
private:
std::string name;
bool active = true;
std::vector<std::unique_ptr<Component>> components;

public:
explicit Entity(const std::string& entityName) : name(entityName) {}

const std::string& GetName() const { return name; }

bool IsActive() const { return active; }
void SetActive(bool isActive) { active = isActive; }

void Initialize() {
for (auto& component : components) {
component->Initialize();
}
}

void Update(float deltaTime) {

if (!active) return;

for (auto& component : components) {

component->Update(deltaTime);
}
}

void Render() {
if (!active) return;

for (auto& component : components) {

component->Render();
}
}

template<typename T, typename... Args>

T* AddComponent(Args&&... args) {

11
 static_assert(std::is_base_of<Component, T>::value, "T must derive from
Component");

// Create new component

auto component = std::make_unique<T>(std::forward<Args>(args)...);
T* componentPtr = [Link]();
componentPtr->SetOwner(this);
components.push_back(std::move(component));
return componentPtr;
}

template<typename T>
T* GetComponent() {
for (auto& component : components) {
if (T* result = dynamic_cast<T*>([Link]())) {
return result;
}
}
return nullptr;
}

template<typename T>
bool RemoveComponent() {
for (auto it = [Link](); it != [Link](); ++it) {
if (dynamic_cast<T*>(it->get())) {
[Link](it);
return true;
}
}
return false;
}
};

Common Component Types

Let’s implement some common component types that you might use in a rendering engine:

// Transform component
// Handles the position, rotation, and scale of an entity in 3D space
// AffineTransform or "Pose" matrix.
class TransformComponent : public Component {
private:
glm::vec3 position = glm::vec3(0.0f);
glm::quat rotation = glm::quat(1.0f, 0.0f, 0.0f, 0.0f); // Identity quaternion
glm::vec3 scale = glm::vec3(1.0f);

// Cached transformation matrix

mutable glm::mat4 transformMatrix = glm::mat4(1.0f);
mutable bool transformDirty = true;

12
public:
void SetPosition(const glm::vec3& pos) {
position = pos;
transformDirty = true;
}

void SetRotation(const glm::quat& rot) {

rotation = rot;
transformDirty = true;
}

void SetScale(const glm::vec3& s) {

scale = s;
transformDirty = true;
}

const glm::vec3& GetPosition() const { return position; }

const glm::quat& GetRotation() const { return rotation; }
const glm::vec3& GetScale() const { return scale; }

glm::mat4 GetTransformMatrix() const {

if (transformDirty) {
// Calculate transformation matrix
glm::mat4 translationMatrix = glm::translate(glm::mat4(1.0f), position);
glm::mat4 rotationMatrix = glm::mat4_cast(rotation);
glm::mat4 scaleMatrix = glm::scale(glm::mat4(1.0f), scale);

transformMatrix = translationMatrix * rotationMatrix * scaleMatrix;

transformDirty = false;
}
return transformMatrix;
}
};

// Mesh component
// Manages the visual representation of an entity by handling its 3D mesh and material
class MeshComponent : public Component {
private:
Mesh* mesh = nullptr;
Material* material = nullptr;

public:
MeshComponent(Mesh* m, Material* mat) : mesh(m), material(mat) {}

void SetMesh(Mesh* m) { mesh = m; }

void SetMaterial(Material* mat) { material = mat; }

Mesh* GetMesh() const { return mesh; }

Material* GetMaterial() const { return material; }

13
 void Render() override {
if (!mesh || !material) return;

// Get transform component

auto transform = GetOwner()->GetComponent<TransformComponent>();
if (!transform) return;

// Render mesh with material and transform

material->Bind();
material->SetUniform("modelMatrix", transform->GetTransformMatrix());
mesh->Render();
}
};

// Camera component
// Defines a viewpoint for rendering the scene by managing view and projection
matrices
class CameraComponent : public Component {
private:
float fieldOfView = 45.0f;
float aspectRatio = 16.0f / 9.0f;
float nearPlane = 0.1f;
float farPlane = 1000.0f;

glm::mat4 viewMatrix = glm::mat4(1.0f);

glm::mat4 projectionMatrix = glm::mat4(1.0f);
bool projectionDirty = true;

public:
void SetPerspective(float fov, float aspect, float near, float far) {
fieldOfView = fov;
aspectRatio = aspect;
nearPlane = near;
farPlane = far;
projectionDirty = true;
}

glm::mat4 GetViewMatrix() const {

// Get transform component
auto transform = GetOwner()->GetComponent<TransformComponent>();
if (transform) {
// Calculate view matrix from transform
glm::vec3 position = transform->GetPosition();
glm::quat rotation = transform->GetRotation();

// Forward vector (local -Z)

glm::vec3 forward = rotation * glm::vec3(0.0f, 0.0f, -1.0f);
// Up vector (local +Y)
glm::vec3 up = rotation * glm::vec3(0.0f, 1.0f, 0.0f);

return glm::lookAt(position, position + forward, up);

14
 }
return glm::mat4(1.0f);
}

glm::mat4 GetProjectionMatrix() const {

if (projectionDirty) {
projectionMatrix = glm::perspective(
glm::radians(fieldOfView),
aspectRatio,
nearPlane,
farPlane
);
projectionDirty = false;
}
return projectionMatrix;
}
};

Component Communication
Components often need to communicate with each other. There are several approaches to
component communication:

Direct References

The simplest approach is to use direct references:

void MeshComponent::Update(float deltaTime) {

auto transform = GetOwner()->GetComponent<TransformComponent>();
if (transform) {
// Use transform data
}
}

This approach is straightforward but creates tight coupling between components. Tight coupling
makes it challenging or impossible to create unit tests and properly test the engine, so this
approach should be avoided in production code.

Event System

A more flexible approach is to use an event system:

// Event base class

class Event {
public:
virtual ~Event() = default;
};

15
 // Specific event types
class CollisionEvent : public Event {
private:
Entity* entity1;
Entity* entity2;

public:
CollisionEvent(Entity* e1, Entity* e2) : entity1(e1), entity2(e2) {}

Entity* GetEntity1() const { return entity1; }

Entity* GetEntity2() const { return entity2; }
};

// Event listener interface

class EventListener {
public:
virtual ~EventListener() = default;
virtual void OnEvent(const Event& event) = 0;
};

// Event system
class EventSystem {
private:
std::vector<EventListener*> listeners;

public:
void AddListener(EventListener* listener) {
listeners.push_back(listener);
}

void RemoveListener(EventListener* listener) {

auto it = std::find([Link](), [Link](), listener);
if (it != [Link]()) {
[Link](it);
}
}

void DispatchEvent(const Event& event) {

for (auto listener : listeners) {
listener->OnEvent(event);
}
}
};

// Component that listens for events

// Handles physics-related behavior and responds to collision events through the event
system
class PhysicsComponent : public Component, public EventListener {
public:
void Initialize() override {

16
 // Register as event listener
GetEventSystem().AddListener(this);
}

~PhysicsComponent() override {
// Unregister as event listener
GetEventSystem().RemoveListener(this);
}

void OnEvent(const Event& event) override {

if (auto collisionEvent = dynamic_cast<const CollisionEvent*>(&event)) {
// Handle collision event
}
}

private:
EventSystem& GetEventSystem() {
// Get event system from somewhere (e.g., service locator)
static EventSystem eventSystem;
return eventSystem;
}
};

This approach decouples components but adds complexity. Crucially, a decoupled component is a
component that can be tested independently of any other component.

Component Lifecycle Management

Managing the lifecycle of components is crucial for a robust component system:

class Component {
public:
enum class State {
Uninitialized,
Initializing,
Active,
Destroying,
Destroyed
};

private:
State state = State::Uninitialized;
Entity* owner = nullptr;

public:
virtual ~Component() {
if (state != State::Destroyed) {
OnDestroy();
state = State::Destroyed;

17
 }
}

void Initialize() {
if (state == State::Uninitialized) {
state = State::Initializing;
OnInitialize();
state = State::Active;
}
}

void Destroy() {
if (state == State::Active) {
state = State::Destroying;
OnDestroy();
state = State::Destroyed;
}
}

bool IsActive() const { return state == State::Active; }

void SetOwner(Entity* entity) { owner = entity; }

Entity* GetOwner() const { return owner; }

protected:
virtual void OnInitialize() {}
virtual void OnDestroy() {}
virtual void Update(float deltaTime) {}
virtual void Render() {}

friend class Entity; // Allow Entity to call protected methods

};

Optimizing Component Access

The GetComponent<T>() method shown earlier uses dynamic_cast, which can be slow. Here’s an
optimized approach using component type IDs:

// Component type ID system

class ComponentTypeIDSystem {
private:
static size_t nextTypeID;

public:
template<typename T>
static size_t GetTypeID() {
static size_t typeID = nextTypeID++;
return typeID;
}

18
};

size_t ComponentTypeIDSystem::nextTypeID = 0;

// Component base class with type ID

class Component {
public:
virtual ~Component() = default;

template<typename T>
static size_t GetTypeID() {
return ComponentTypeIDSystem::GetTypeID<T>();
}
};

// Entity with optimized component access

class Entity {
private:
std::vector<std::unique_ptr<Component>> components;
std::unordered_map<size_t, Component*> componentMap;

public:
template<typename T, typename... Args>
T* AddComponent(Args&&... args) {
static_assert(std::is_base_of<Component, T>::value, "T must derive from
Component");

size_t typeID = Component::GetTypeID<T>();

// Check if component of this type already exists

auto it = [Link](typeID);
if (it != [Link]()) {
return static_cast<T*>(it->second);
}

// Create new component

auto component = std::make_unique<T>(std::forward<Args>(args)...);
T* componentPtr = [Link]();
componentMap[typeID] = componentPtr;
components.push_back(std::move(component));
return componentPtr;
}

template<typename T>
T* GetComponent() {
size_t typeID = Component::GetTypeID<T>();
auto it = [Link](typeID);
if (it != [Link]()) {
return static_cast<T*>(it->second);
}
return nullptr;

19
 }

template<typename T>
bool RemoveComponent() {
size_t typeID = Component::GetTypeID<T>();
auto it = [Link](typeID);
if (it != [Link]()) {
Component* componentPtr = it->second;
[Link](it);

for (auto compIt = [Link](); compIt != [Link]();

++compIt) {
if (compIt->get() == componentPtr) {
[Link](compIt);
return true;
}
}
}
return false;
}
};

Conclusion
Component systems provide a flexible and modular approach to building game objects in your
engine. By following the principles outlined in this section, you can create a robust component
system that:

1. Promotes code reuse through composition

2. Reduces coupling between different parts of your engine

3. Allows for flexible entity creation without deep inheritance hierarchies

4. Can be optimized for performance

In the next section, we’ll explore resource management systems, which are crucial for efficiently
handling assets in your engine.

Previous: Architectural Patterns | Next: Resource Management :pp: ++

Engine Architecture: Resource

Management
Resource Management
Efficient resource management is a critical aspect of any rendering engine. In this section, we’ll

20
explore strategies for managing various types of resources, such as textures, meshes, shaders, and
materials.

Resource Management Challenges

When designing a resource management system, you’ll need to address several challenges:

1. Loading and Unloading - Resources need to be loaded from disk and unloaded when no longer
needed.

2. Caching - Frequently used resources should be cached to avoid redundant loading.

3. Reference Counting - Track how many objects are using a resource to know when it can be
safely unloaded.

4. Hot Reloading - Allow resources to be updated while the application is running (useful during
development).

5. Streaming - Load resources asynchronously to avoid blocking the main thread. It’s good to
realize that "streaming" here is meant in terms of sending data from one location to another in
chunks. It’s the same type of algorithm that might be familiar in networking or internet
downloading, however, it only differs in the sense that it relates to transferring data between
the system memory and the GPU memory.

6. Memory Management - Efficiently allocate and deallocate memory for resources.

Resource Handles
Instead of directly exposing resource pointers, it’s often better to use resource handles:

// Resource handle
template<typename T>
class ResourceHandle {
private:
std::string resourceId;
ResourceManager* resourceManager;

public:
ResourceHandle() : resourceManager(nullptr) {}

ResourceHandle(const std::string& id, ResourceManager* manager)

: resourceId(id), resourceManager(manager) {}

T* Get() const {
if (!resourceManager) return nullptr;
return resourceManager->GetResource<T>(resourceId);
}

bool IsValid() const {

return resourceManager && resourceManager->HasResource<T>(resourceId);
}

21
 const std::string& GetId() const {
return resourceId;
}

// Convenience operators
T* operator->() const {
return Get();
}

T& operator*() const {

return *Get();
}

operator bool() const {

return IsValid();
}
};

Using handles instead of direct pointers provides several benefits:

1. Indirection - The resource manager can move resources in memory without invalidating
references.

2. Validation - Handles can be checked for validity before use.

3. Automatic Resource Management - The resource manager can track which resources are in
use.

Basic Resource Manager

Let’s implement a basic resource manager that can handle different types of resources. This
implementation involves several key steps that work together to provide efficient resource
management for a rendering engine.

Resource Manager: Base Resource Architecture and

State Management
First, we establish the fundamental infrastructure for resource management, defining how
resources track their identity and loading state within the system.

// Resource base class

class Resource {
private:
std::string resourceId; // Unique identifier for this resource within the
system
bool loaded = false; // Loading state flag for resource lifecycle
management

22
 public:
explicit Resource(const std::string& id) : resourceId(id) {}
virtual ~Resource() = default;

// Core resource identity and state access methods

const std::string& GetId() const { return resourceId; }
bool IsLoaded() const { return loaded; }

// Virtual interface for resource-specific loading and unloading behavior

bool Load() {
loaded = doLoad();
return loaded;
}

void Unload() {
doUnload();
loaded = false;
}

protected:
virtual bool doLoad() = 0;
virtual bool doUnload() = 0;
};

The Resource base class provides the foundational contract that all resource types must fulfill. The
resource ID serves as a unique identifier that allows the resource manager to locate and reference
specific resources without ambiguity. This string-based approach enables human-readable resource
names like "main_character_texture" or "level_1_audio" while maintaining the flexibility to use file
paths or other naming schemes.

The loading state management through the boolean flag provides essential lifecycle tracking. This
simple approach allows systems to quickly determine whether a resource is ready for use without
expensive validation checks. The virtual loading interface enables polymorphic behavior where
different resource types can implement their own specialized loading logic while presenting a
consistent interface to the management system.

Resource Manager: Storage Architecture and Type

Safety
Next, we implement the core storage system that organizes resources by type while maintaining
type safety and efficient access patterns.

// Resource manager
class ResourceManager {
private:
// Two-level storage system: organize by type first, then by unique identifier
// This approach enables type-safe resource access while maintaining efficient
lookup

23
 std::unordered_map<std::type_index,
std::unordered_map<std::string, std::shared_ptr<Resource>>>
resources;

// Two-level reference counting system for automatic resource lifecycle management

// First level maps resource type, second level maps resource IDs to their data
struct ResourceData {
std::shared_ptr<Resource> resource; // The actual resource
int refCount; // Reference count for this resource
};
std::unordered_map<std::type_index,
std::unordered_map<std::string, ResourceData>> refCounts;

The storage architecture uses a sophisticated two-level mapping system that solves several critical
problems in resource management. The outer map keyed by std::type_index ensures complete type
separation, preventing name collisions between different resource types. For example, you could
have both a texture named "stone" and a sound effect named "stone" without conflicts, as they’re
stored in separate type-specific containers.

The inner maps provide O(1) average-case lookup performance for individual resources, which is
crucial when the rendering system needs to access hundreds or thousands of resources per frame.
The use of std::shared_ptr provides automatic memory management and enables safe sharing of
resources between different systems without manual lifetime management.

The reference counting system operates independently of the shared_ptr reference counting to
provide application-level lifecycle control. This separation allows the resource manager to
implement custom policies for resource retention and cleanup that go beyond simple memory
management, such as keeping frequently used resources loaded even when not immediately
referenced.

Resource Manager: Resource Loading and Caching

Logic
Then, we implement the intelligent resource loading system that handles caching, reference
counting, and error recovery for efficient resource management.

public:
template<typename T>
ResourceHandle<T> Load(const std::string& resourceId) {
static_assert(std::is_base_of<Resource, T>::value, "T must derive from
Resource");

// Step 3a: Check existing resource cache to avoid redundant loading

auto& typeResources = resources[std::type_index(typeid(T))];
auto it = [Link](resourceId);

if (it != [Link]()) {
// Resource exists in cache - increment reference count and return handle

24
 refCounts[resourceId]++;
return ResourceHandle<T>(resourceId, this);
}

// Step 3b: Create new resource instance and attempt loading

auto resource = std::make_shared<T>(resourceId);
if (!resource->Load()) {
// Loading failed - return invalid handle rather than corrupting cache
return ResourceHandle<T>();
}

// Step 3c: Cache successful resource and initialize reference tracking

typeResources[resourceId] = resource;
refCounts[resourceId] = 1;

return ResourceHandle<T>(resourceId, this);

}

The loading logic implements a sophisticated caching strategy that balances performance with
memory efficiency. The cache-first approach prevents redundant I/O operations and resource
processing, which can be expensive for large textures, complex meshes, or compiled shaders. This
strategy is particularly important in rendering engines where the same resources may be
referenced by multiple objects or systems.

The template-based design with compile-time type checking ensures type safety while maintaining
the flexibility to work with any resource type that derives from the base Resource class. The static
assertion provides clear error messages during development, preventing runtime type errors that
could be difficult to debug in complex rendering scenarios.

Error handling follows the principle of graceful degradation, where loading failures return invalid
handles rather than throwing exceptions or corrupting the resource cache. This approach allows
rendering systems to continue operating with fallback resources or alternative rendering paths
when specific assets are unavailable or corrupted.

Resource Manager: Resource Access and Validation

Interface
After that, we provide the interface for safely accessing cached resources with proper validation
and type checking throughout the resource lifecycle.

template<typename T>
T* GetResource(const std::string& resourceId) {
// Access type-specific resource container using compile-time type information
auto& typeResources = resources[std::type_index(typeid(T))];
auto it = [Link](resourceId);

if (it != [Link]()) {
// Resource found - perform safe downcast and return typed pointer

25
 return static_cast<T*>(it->[Link]());
}

// Resource not found - return null for safe handling by caller

return nullptr;
}

template<typename T>
bool HasResource(const std::string& resourceId) {
// Efficient existence check without resource access overhead
auto resourceIt = [Link](std::type_index(typeid(T)));
return resourceIt != [Link]();
}

The resource access interface prioritizes safety and performance in equal measure. The template-
based approach ensures that clients always receive correctly typed resource pointers, eliminating
the need for manual casting and reducing the potential for type-related runtime errors. The
static_cast is safe because the type_index-based storage guarantees that only objects of type T are
stored in each type-specific container.

The existence check provides an efficient way to validate resource availability without the
overhead of full resource access. This capability is valuable for conditional rendering logic, where
systems can choose alternative rendering paths based on resource availability without triggering
expensive cache misses or I/O operations.

Resource Manager: Reference Counting and Automatic

Cleanup
Finally, we implement intelligent resource lifecycle management through reference counting and
automatic cleanup to prevent memory leaks and optimize resource utilization.

void Release(const std::string& resourceId) {

// Locate reference count entry for this resource
auto it = [Link](resourceId);
if (it != [Link]()) {
it->second--;

// Check if resource has no remaining references

if (it->second <= 0) {
// Step 5a: Locate and unload the unreferenced resource across all
type containers
for (auto& [type, typeResources] : resources) {
auto resourceIt = [Link](resourceId);
if (resourceIt != [Link]()) {
resourceIt->second->Unload(); // Allow resource to clean
up its data
[Link](resourceIt); // Remove from cache
break;

26
 }
}

// Step 5b: Clean up reference counting entry

[Link](it);
}
}
}

void UnloadAll() {
// Emergency cleanup method for system shutdown or major state changes
for (auto& [type, typeResources] : resources) {
for (auto& [id, resource] : typeResources) {
resource->Unload(); // Ensure all resources clean up properly
}
[Link](); // Clear type-specific containers
}
[Link](); // Reset all reference counts
}
};

The reference counting system provides automatic garbage collection for resources that are no
longer actively used. This approach prevents memory leaks while avoiding the overhead of
constantly monitoring resource usage across the entire application. The decrement-and-check
pattern ensures that resources are unloaded immediately when they become unused, helping to
keep memory usage optimal.

The cleanup process is designed to be thorough and safe, ensuring that resources have the
opportunity to properly release their internal data (GPU memory, file handles, etc.) before being
removed from the cache. This two-phase cleanup approach prevents resource leaks and maintains
system stability even under error conditions.

The global unload functionality provides a safety valve for major state transitions like level changes
or application shutdown, where you want to ensure all resources are properly cleaned up
regardless of their reference counts. This capability is essential for preventing resource leaks that
could accumulate over long application runs.

Implementing Specific Resource Types

Now let’s implement some specific resource types that demonstrate how different asset types can
be integrated into our resource management system. These implementations showcase the
flexibility of the base Resource interface while addressing the unique requirements of different
content types.

Texture Resource Implementation

The Texture resource represents one of the most complex resource types in a rendering engine,
requiring careful management of GPU memory, format conversion, and sampling parameters. Let’s

27
break this implementation into logical phases that demonstrate both the technical challenges and
design solutions.

Texture Resource: Resource Structure and Vulkan

State Management
First, we establish the fundamental data structures required for Vulkan texture management,
including GPU resources and metadata needed for proper texture usage.

// Texture resource
class Texture : public Resource {
private:
// Core Vulkan GPU resources for texture representation
vk::Image image; // GPU image object containing pixel data
vk::DeviceMemory memory; // GPU memory allocation backing the image
vk::DeviceSize offset; // Offset within the memory allocation for this
texture
vk::ImageView imageView; // Shader-accessible view into the image
vk::Sampler sampler; // Sampling configuration (filtering, wrapping,
etc.)

// Texture metadata for validation and debugging

int width = 0; // Image width in pixels
int height = 0; // Image height in pixels
int channels = 0; // Number of color channels (RGB=3, RGBA=4, etc.)

public:
explicit Texture(const std::string& id) : Resource(id) {}

~Texture() override {
Unload(); // Ensure proper cleanup when object is destroyed
}

The Vulkan texture pipeline requires four distinct GPU objects that work together to provide
complete texture functionality. The vk::Image represents the actual pixel data storage on the GPU,
while vk::DeviceMemory provides the backing memory allocation. The separation between image
and memory allows for advanced memory management techniques like suballocation and memory
pooling.

The vk::ImageView serves as the interface between shaders and the image data, defining how
shaders interpret the pixel format, mipmap levels, and array layers. The vk::Sampler encapsulates
filtering and addressing modes that control how the GPU interpolates between pixels and handles
texture coordinates outside the [0,1] range. This separation of concerns allows the same image to be
used with different sampling configurations simultaneously.

28
Texture Resource: Loading Pipeline and Data
Acquisition
Next, we implement the texture loading pipeline that transforms disk-based image files into GPU-
ready resources through careful error handling and format conversion.

bool Load() override {

// Step 2a: Construct file path using resource ID and expected format
std::string filePath = "textures/" + GetId() + ".ktx";

// Step 2b: Load raw image data from disk with format detection
unsigned char* data = LoadImageData(filePath, &width, &height, &channels);
if (!data) {
return false; // Failed to load - return failure without partial
state
}

// Step 2c: Transform raw pixel data into Vulkan GPU resources
CreateVulkanImage(data, width, height, channels);

// Step 2d: Clean up temporary CPU memory to prevent leaks

FreeImageData(data);

return Resource::Load(); // Mark resource as successfully loaded

}

The loading pipeline follows a clear sequence that handles the complex transformation from file-
based data to GPU resources. The file path construction assumes a standard naming convention
that maps resource IDs to physical files, enabling consistent asset organization across the project.
Using the KTX format provides several advantages including GPU-native format storage, mipmap
support, and compression compatibility.

Error handling at each stage prevents partial loading states that could leave the resource in an
inconsistent condition. If image data loading fails, the function returns immediately without
creating GPU resources, ensuring that the Texture object remains in a clean, unloaded state. This
approach prevents resource leaks and makes error recovery more predictable for calling code.

The temporary nature of the CPU-side image data reflects the typical texture loading workflow
where pixel data is needed only long enough to upload to the GPU. Once the GPU resources are
created and populated, the CPU copy can be safely discarded, reducing memory pressure and
preventing unnecessary data duplication.

Texture Resource: GPU Resource Cleanup and Memory

Management
Then, we implement comprehensive resource cleanup that ensures all GPU resources are properly
released when the texture is no longer needed, preventing memory leaks in long-running

29
applications. Please note that if you have vk::raii objects, the destructor of the vk::raii objects will
automatically handle the cleanup of the GPU resources. If, however, you have a vk::Device object,
you must manually destroy the GPU resources to prevent memory leaks. Additionally, you need to
have initialized the defaultDispatcher for the vk::Device object types. In the event that you are
using vk::Device objects, the Unload function below details explicit releasing of the objects.

void Unload() override {

// Only perform cleanup if resource is currently loaded
if (IsLoaded()) {
// Step 3a: Obtain device handle for resource destruction
vk::Device device = GetDevice();

// Step 3b: Destroy GPU objects in reverse creation order

// This ordering prevents use-after-free errors in GPU drivers
[Link](sampler); // Destroy sampling configuration
[Link](imageView); // Destroy shader view
[Link](image); // Destroy image object
[Link](memory); // Release GPU memory allocation

// Step 3c: Update base class state to reflect unloaded status

Resource::Unload();
}
}

// Public interface for accessing Vulkan resources safely

vk::Image GetImage() const { return image; }
vk::ImageView GetImageView() const { return imageView; }
vk::Sampler GetSampler() const { return sampler; }

The cleanup sequence follows Vulkan’s object dependency requirements, where objects must be
destroyed in reverse order of their creation to avoid validation errors and potential driver crashes.
The sampler and image view depend on the image, so they must be destroyed first. The memory
allocation is released last since it backs the image object.

The conditional cleanup check prevents double-destruction errors that could occur if Unload() is
called multiple times. This safety mechanism is particularly important in resource management
systems where multiple code paths might trigger cleanup operations during error handling or
shutdown sequences.

The public getter interface provides controlled access to the internal Vulkan resources without
exposing the implementation details or allowing external code to modify the resource state. This
encapsulation ensures that the Texture object maintains complete control over its GPU resources
throughout their lifetime.

Texture Resource: Helper Methods and

Implementation Details
Finally, we provide the supporting infrastructure methods that handle the platform-specific details

30
of image loading and Vulkan resource creation.

private:
unsigned char* LoadImageData(const std::string& filePath, int* width, int* height,
int* channels) {
// Implementation using stb_image or ktx library
// This method abstracts the details of different image format support
// and provides a consistent interface for pixel data loading
// ...
return nullptr; // Placeholder
}

void FreeImageData(unsigned char* data) {

// Implementation using stb_image or ktx library
// Ensures proper cleanup of image loader specific memory allocations
// Different libraries may require different cleanup approaches
// ...
}

void CreateVulkanImage(unsigned char* data, int width, int height, int channels) {
// Implementation to create Vulkan image, allocate memory, and upload data
// This involves complex Vulkan operations including:
// - Format selection based on channel count and data type
// - Memory allocation with appropriate usage flags
// - Image creation with optimal tiling and layout
// - Data upload via staging buffers for efficiency
// - Image view creation for shader access
// - Sampler creation with appropriate filtering settings
// ...
}

vk::Device GetDevice() {
// Get device from somewhere (e.g., singleton or parameter)
// Production code would use dependency injection or service location
// to provide the Vulkan device handle without tight coupling
// ...
return vk::Device(); // Placeholder
}
};

The helper methods abstract away the platform-specific and library-specific details of texture
loading and GPU resource creation. The LoadImageData method encapsulates support for different
image formats and loading libraries, providing a consistent interface regardless of whether you’re
using STB Image, DevIL, FreeImage, or other image loading solutions.

The CreateVulkanImage method represents one of the most complex operations in texture
management, involving multiple Vulkan API calls with careful attention to format selection,
memory alignment, and performance optimization. Production implementations typically use
staging buffers for efficient data transfer and may include mipmap generation, format conversion,

31
and compression support.

The device access pattern shown here as a placeholder represents a common design challenge in
resource management systems: how to provide access to core engine services without creating tight
coupling. Production systems typically use dependency injection, service locators, or context
objects to provide access to the Vulkan device and other core resources.

Mesh Resource Implementation

The Mesh resource represents the geometric foundation of 3D rendering, managing vertex and
index data that define the shape and structure of 3D objects. This implementation demonstrates
how to efficiently manage GPU buffer resources for geometric data.

Mesh Resource: Geometric Data Structure and Buffer

Management
First, we establish the fundamental data structures required for storing and managing geometric
data on the GPU, including both vertex attributes and index connectivity information.

// Mesh resource
class Mesh : public Resource {
private:
// Vertex data management - stores per-vertex attributes like position, normal, UV
coordinates
vk::Buffer vertexBuffer; // GPU buffer containing vertex attribute
data
vk::DeviceMemory vertexBufferMemory; // GPU memory backing the vertex buffer
vk::DeviceSize vertexBufferOffset; // Offset within the memory allocation for
vertex buffer
uint32_t vertexCount = 0; // Number of vertices in this mesh

// Index data management - defines triangle connectivity using vertex indices

vk::Buffer indexBuffer; // GPU buffer containing triangle index
data
vk::DeviceMemory indexBufferMemory; // GPU memory backing the index buffer
vk::DeviceSize indexBufferOffset; // Offset within the memory allocation for
index buffer
uint32_t indexCount = 0; // Number of indices in this mesh
(typically 3 per triangle)

public:
explicit Mesh(const std::string& id) : Resource(id) {}

~Mesh() override {
Unload(); // Ensure GPU resources are cleaned up
}

The mesh resource architecture separates vertex and index data into distinct GPU buffers,

32
following modern graphics API best practices. Vertex buffers contain per-vertex attributes such as
positions, normals, texture coordinates, and color information, while index buffers define how
vertices connect to form triangles. This separation enables efficient vertex reuse, where a single
vertex can be referenced by multiple triangles, significantly reducing memory usage for typical 3D
models.

The buffer-memory pairing reflects Vulkan’s explicit memory management model, where buffer
objects and their backing memory allocations are managed separately. This approach provides fine-
grained control over memory allocation strategies, enabling techniques like memory pooling,
suballocation, and custom alignment requirements that can significantly impact rendering
performance.

The count tracking serves dual purposes: it provides essential information for rendering calls that
specify how many vertices or indices to process, and it enables validation and debugging by
allowing systems to verify that buffer contents match expected data sizes.

Mesh Resource: Data Loading and Format Processing

Pipeline
Next, we implement the mesh loading pipeline that transforms file-based geometric data into GPU-
ready buffer resources through format parsing and data validation.

bool Load() override {

// Step 2a: Construct file path using standardized naming convention
std::string filePath = "models/" + GetId() + ".gltf";

// Step 2b: Parse geometric data from file format into CPU-accessible
structures
std::vector<Vertex> vertices; // Temporary CPU storage for vertex
attributes
std::vector<uint32_t> indices; // Temporary CPU storage for triangle
indices
if (!LoadMeshData(filePath, vertices, indices)) {
return false; // Failed to parse file - abort loading
}

// Step 2c: Transform CPU data into optimized GPU buffer resources
CreateVertexBuffer(vertices); // Upload vertex attributes to GPU
CreateIndexBuffer(indices); // Upload triangle connectivity to GPU

// Step 2d: Cache metadata for efficient rendering operations

vertexCount = static_cast<uint32_t>([Link]());
indexCount = static_cast<uint32_t>([Link]());

return Resource::Load(); // Mark resource as successfully loaded

}

The loading pipeline follows a structured approach that separates file parsing from GPU resource

33
creation, enabling better error handling and code reusability. The choice of glTF format provides
several advantages including industry-standard mesh representation, embedded material
information, and support for advanced features like skeletal animations and morph targets.

The temporary CPU-side storage approach enables validation and processing of geometric data
before committing to GPU resources. This intermediate step allows for mesh optimization
techniques such as vertex cache optimization, triangle strip generation, or level-of-detail processing
that can significantly improve rendering performance.

The metadata caching strategy stores frequently accessed information locally to avoid expensive
GPU queries during rendering. These counts are essential for draw calls, where the GPU needs to
know exactly how many vertices to process and how many triangles to render, making local storage
much more efficient than querying the GPU buffers repeatedly.

Mesh Resource: GPU Resource Cleanup and Memory

Reclamation
Then, we implement comprehensive cleanup that properly releases all GPU resources and memory
allocations when the mesh is no longer needed, ensuring robust memory management in long-
running applications. As mentioned above, if you have vk::raii objects, the destructor of the vk::raii
objects will automatically handle the cleanup of the GPU resources. If, however, you have a
vk::Device object, you must manually destroy the GPU resources to prevent memory leaks.
Additionally, you need to have initialized the defaultDispatcher for the vk::Device object types. In
the event that you are using vk::Device objects, the Unload function below details explicit releasing
of the objects.

void Unload() override {

// Only proceed with cleanup if resources are currently loaded
if (IsLoaded()) {
// Phase 3a: Obtain device handle for resource destruction
vk::Device device = GetDevice();

// Phase 3b: Destroy buffers and free GPU memory in proper sequence
// Index resources cleaned up first to maintain clear dependency order
[Link](indexBuffer); // Destroy index buffer object
[Link](indexBufferMemory); // Release index buffer memory

// Vertex resources cleaned up second

[Link](vertexBuffer); // Destroy vertex buffer object
[Link](vertexBufferMemory); // Release vertex buffer memory

// Phase 3c: Update base class state to reflect unloaded condition

Resource::Unload();
}
}

// Public interface for safe access to GPU resources and metadata

vk::Buffer GetVertexBuffer() const { return vertexBuffer; }

34
 vk::Buffer GetIndexBuffer() const { return indexBuffer; }
uint32_t GetVertexCount() const { return vertexCount; }
uint32_t GetIndexCount() const { return indexCount; }

The cleanup sequence ensures that GPU resources are properly released without causing validation
errors or driver instability. While Vulkan doesn’t impose strict ordering requirements for buffer
destruction, following a consistent pattern (index resources before vertex resources) makes the
code more predictable and easier to debug when issues arise.

The conditional cleanup check prevents double-destruction scenarios that could occur during error
handling or when multiple systems attempt to clean up resources simultaneously. This safety
mechanism is particularly important in complex rendering systems where resource ownership
might be shared between multiple components.

The public access interface provides controlled access to internal GPU resources while maintaining
encapsulation. These getter methods enable rendering systems to bind the appropriate buffers for
draw operations while preventing external code from accidentally modifying the mesh’s internal
state or triggering premature resource destruction.

Mesh Resource: Helper Methods and Implementation

Support Infrastructure
The final phase provides the supporting methods that handle the complex details of mesh data
parsing, buffer creation, and system integration required for complete mesh resource functionality.

private:
bool LoadMeshData(const std::string& filePath, std::vector<Vertex>& vertices,
std::vector<uint32_t>& indices) {
// Implementation using tinygltf or similar library
// This method handles the complex task of:
// - Opening and validating the mesh file format
// - Parsing vertex attributes (positions, normals, UVs, etc.)
// - Extracting index data that defines triangle connectivity
// - Converting from file format to engine-specific vertex structures
// - Performing validation to ensure data integrity
// ...
return true; // Placeholder
}

void CreateVertexBuffer(const std::vector<Vertex>& vertices) {

// Implementation to create Vulkan buffer, allocate memory, and upload data
// This involves several complex Vulkan operations:
// - Calculating buffer size requirements based on vertex count and structure
// - Creating buffer with appropriate usage flags (vertex buffer usage)
// - Allocating GPU memory with optimal memory type selection
// - Uploading data via staging buffer for efficient transfer
// - Setting up memory barriers to ensure data availability
// ...

35
 }

void CreateIndexBuffer(const std::vector<uint32_t>& indices) {

// Implementation to create Vulkan buffer, allocate memory, and upload data
// Similar to vertex buffer creation but optimized for index data:
// - Buffer creation with index buffer specific usage flags
// - Memory allocation optimized for read-heavy access patterns
// - Efficient data transfer using appropriate staging mechanisms
// - Index format validation (16-bit vs 32-bit indices)
// ...
}

vk::Device GetDevice() {
// Get device from somewhere (e.g., singleton or parameter)
// Production implementations typically use dependency injection
// to avoid tight coupling between resource classes and core engine systems
// ...
return vk::Device(); // Placeholder
}
};

The helper methods encapsulate the most complex aspects of mesh resource management, hiding
implementation details while providing clean interfaces for the core loading and creation logic. The
LoadMeshData method abstracts the intricacies of different mesh file formats and parsing libraries,
enabling the resource system to support multiple formats through a consistent interface.

The buffer creation methods represent some of the most performance-critical code in the mesh
resource system, as inefficient GPU memory management can significantly impact rendering
performance. Production implementations typically use staging buffers for data upload, implement
memory pooling to reduce allocation overhead, and carefully select memory types based on GPU
architecture characteristics.

The device access pattern illustrates a common architectural challenge in resource management
systems: balancing convenience with loose coupling. While direct access to global singletons can
simplify implementation, production systems typically use dependency injection or service locator
patterns to maintain testability and flexibility while providing access to core engine services.

Shader Resource Implementation

The Shader resource represents the programmable stages of the graphics pipeline, managing
compilation, loading, and runtime management of shader programs. This implementation
demonstrates how to handle SPIR-V shader modules while providing clean interfaces for shader
stage management and hot reloading support during development.

// Shader resource
class Shader : public Resource {
private:
vk::ShaderModule shaderModule;
vk::ShaderStageFlagBits stage;

36
public:
Shader(const std::string& id, vk::ShaderStageFlagBits shaderStage)
: Resource(id), stage(shaderStage) {}

~Shader() override {
Unload();
}

bool Load() override {

// Determine file extension based on shader stage
std::string extension;
switch (stage) {
case vk::ShaderStageFlagBits::eVertex: extension = ".vert"; break;
case vk::ShaderStageFlagBits::eFragment: extension = ".frag"; break;
case vk::ShaderStageFlagBits::eCompute: extension = ".comp"; break;
default: return false;
}

// Load shader from file

std::string filePath = "shaders/" + GetId() + extension + ".spv";

// Read shader code

std::vector<char> shaderCode;
if (!ReadFile(filePath, shaderCode)) {
return false;
}

// Create shader module

CreateShaderModule(shaderCode);

return Resource::Load();
}

void Unload() override {

// Destroy Vulkan resources
if (IsLoaded()) {
// Get device from somewhere (e.g., singleton or parameter)
vk::Device device = GetDevice();

[Link](shaderModule);

Resource::Unload();
}
}

// Getters for Vulkan resources

vk::ShaderModule GetShaderModule() const { return shaderModule; }
vk::ShaderStageFlagBits GetStage() const { return stage; }

private:

37
 bool ReadFile(const std::string& filePath, std::vector<char>& buffer) {
// Implementation to read binary file
// ...
return true; // Placeholder
}

void CreateShaderModule(const std::vector<char>& code) {

// Implementation to create Vulkan shader module
// ...
}

vk::Device GetDevice() {
// Get device from somewhere (e.g., singleton or parameter)
// ...
return vk::Device(); // Placeholder
}
};

Using the Resource Manager

Here’s how you might use the resource manager in your application:

// Create resource manager

ResourceManager resourceManager;

// Load resources
auto texture = [Link]<Texture>("brick");
auto mesh = [Link]<Mesh>("cube");
auto vertexShader = [Link]<Shader>("basic",
vk::ShaderStageFlagBits::eVertex);
auto fragmentShader = [Link]<Shader>("basic",
vk::ShaderStageFlagBits::eFragment);

// Use resources
if (texture && mesh && vertexShader && fragmentShader) {
// Create material using shaders
Material material(vertexShader, fragmentShader);

// Set texture in material

[Link]("diffuse", texture);

// Create entity with mesh and material

Entity entity("MyEntity");
auto meshComponent = [Link]<MeshComponent>([Link](), &material);
}

// Resources will be automatically released when handles go out of scope

// or you can explicitly release them

38
 [Link]([Link]());

Advanced Resource Management Techniques

Asynchronous Loading

For large resources, it’s often beneficial to load them asynchronously to avoid blocking the main
thread:

class AsyncResourceManager {
private:
ResourceManager resourceManager;
std::thread workerThread;
std::queue<std::function<void()>> taskQueue;
std::mutex queueMutex;
std::condition_variable condition;
bool running = false;

public:
AsyncResourceManager() {
Start();
}

~AsyncResourceManager() {
Stop();
}

void Start() {
running = true;
workerThread = std::thread([this]() {
WorkerThread();
});
}

void Stop() {
{
std::lock_guard<std::mutex> lock(queueMutex);
running = false;
}
condition.notify_one();
if ([Link]()) {
[Link]();
}
}

template<typename T>
void LoadAsync(const std::string& resourceId,
std::function<void(ResourceHandle<T>)> callback) {
std::lock_guard<std::mutex> lock(queueMutex);

39
 [Link]([this, resourceId, callback]() {
auto handle = [Link]<T>(resourceId);
callback(handle);
});
condition.notify_one();
}

private:
void WorkerThread() {
while (running) {
std::function<void()> task;
{
std::unique_lock<std::mutex> lock(queueMutex);
[Link](lock, [this]() {
return ![Link]() || !running;
});

if (!running && [Link]()) {

return;
}

task = std::move([Link]());
[Link]();
}

task();
}
}
};

// Usage example
AsyncResourceManager asyncResourceManager;

[Link]<Texture>("large_texture", [](ResourceHandle<Texture>
texture) {
// This callback will be called when the texture is loaded
if (texture) {
std::cout << "Texture loaded successfully!" << std::endl;
} else {
std::cout << "Failed to load texture." << std::endl;
}
});

Resource Streaming

For very large resources like high-resolution textures or detailed meshes, you might want to
implement streaming:

1. Level of Detail (LOD) - Load lower-resolution versions first, then progressively load higher-
resolution versions.

40
2. Texture Streaming - Load mipmap levels progressively, starting with the smallest.

3. Mesh Streaming - Load simplified versions of meshes first, then add detail.

Hot Reloading

During development, it’s useful to be able to update resources without restarting the application:

class HotReloadResourceManager : public ResourceManager {

private:
std::unordered_map<std::string, std::filesystem::file_time_type> fileTimestamps;
std::thread watcherThread;
bool running = false;

public:
HotReloadResourceManager() {
StartWatcher();
}

~HotReloadResourceManager() {
StopWatcher();
}

void StartWatcher() {
running = true;
watcherThread = std::thread([this]() {
WatcherThread();
});
}

void StopWatcher() {
running = false;
if ([Link]()) {
[Link]();
}
}

template<typename T>
ResourceHandle<T> Load(const std::string& resourceId) {
auto handle = ResourceManager::Load<T>(resourceId);

// Store file timestamp

std::string filePath = GetFilePath<T>(resourceId);
try {
fileTimestamps[filePath] = std::filesystem::last_write_time(filePath);
} catch (const std::filesystem::filesystem_error& e) {
// File doesn't exist or can't be accessed
}

return handle;
}

41
 private:
template<typename T>
std::string GetFilePath(const std::string& resourceId) {
// Determine file path based on resource type and ID
if constexpr (std::is_same_v<T, Texture>) {
return "textures/" + resourceId + ".ktx";
} else if constexpr (std::is_same_v<T, Mesh>) {
return "models/" + resourceId + ".gltf";
} else if constexpr (std::is_same_v<T, Shader>) {
// Simplified for example
return "shaders/" + resourceId + ".spv";
} else {
return "";
}
}

void WatcherThread() {
while (running) {
// Check for file changes
for (auto& [filePath, timestamp] : fileTimestamps) {
try {
auto currentTimestamp =
std::filesystem::last_write_time(filePath);
if (currentTimestamp != timestamp) {
// File has changed, reload resource
ReloadResource(filePath);
timestamp = currentTimestamp;
}
} catch (const std::filesystem::filesystem_error& e) {
// File doesn't exist or can't be accessed
}
}

// Sleep to avoid high CPU usage

std::this_thread::sleep_for(std::chrono::seconds(1));
}
}

void ReloadResource(const std::string& filePath) {

// Extract resource ID and type from file path
// Reload the resource
// ...
}
};

Conclusion
A well-designed resource management system is crucial for efficiently handling assets in your
rendering engine. By implementing the techniques described in this section, you can create a

42
system that:

1. Efficiently loads and unloads resources

2. Prevents redundant loading through caching

3. Manages memory usage through reference counting

4. Supports asynchronous loading for better performance

5. Enables hot reloading for faster development

In the next section, we’ll explore rendering pipeline design, which will build upon the resource
management system to create a flexible and efficient rendering system.

Previous: Component Systems | Next: Rendering Pipeline :pp: ++

Engine Architecture: Rendering

Pipeline
Rendering Pipeline
A well-designed rendering pipeline is essential for creating a flexible and efficient rendering
engine. In this section, we’ll explore how to structure your rendering pipeline to support various
rendering techniques and effects.

Rendering Pipeline Overview

The following diagram provides a high-level overview of a modern Vulkan rendering pipeline:

[Flowchart showing the stages of a modern Vulkan rendering pipeline] |

../../../images/rendering_pipeline_flowchart.png

Diagram Legend:

• Boxes: Represent the different stages of the rendering pipeline

• Arrows: Show the flow of data and execution between stages

NOTE
• Colors: Different colors indicate different types of operations (processing,
management, execution)

• Supporting Components: Rendergraphs and Synchronization primitives are

shown as connected to the main pipeline flow

The rendering pipeline consists of several key stages:

1. Scene Culling - Determine which objects are visible and need to be rendered.

2. Render Pass Management - Organize rendering into passes with specific purposes.

43
3. Command Generation - Generate commands for the GPU to execute.

4. Execution - Submit commands to the GPU for execution.

5. Post-Processing - Apply effects to the rendered image.

Supporting components like Rendergraphs help manage dependencies between render passes,
while Synchronization primitives ensure correct execution order. Different rendering techniques
(Deferred, Forward+, PBR) can be implemented within this pipeline architecture.

Rendering Pipeline Challenges

When designing a rendering pipeline, you’ll need to address several challenges:

1. Flexibility - Support different rendering techniques and effects.

2. Performance - Efficiently utilize the GPU and minimize state changes.

3. Extensibility - Allow for easy addition of new rendering features.

4. Maintainability - Keep the code organized and easy to understand.

5. Platform Independence - Abstract away platform-specific details.

Rendering Pipeline Architecture

Earlier we outlined the major stages of a modern pipeline. Rather than repeating that list, we’ll now
dive into each stage, focusing on responsibilities, data flow, and practical implementation patterns
that keep the engine flexible and performant.

Scene Culling
Before rendering, we need to determine which objects are visible to the camera. This process is
called culling and can significantly improve performance by reducing the number of objects that
need to be rendered.

class CullingSystem {
private:
Camera* camera;
std::vector<Entity*> visibleEntities;

public:
explicit CullingSystem(Camera* cam) : camera(cam) {}

void SetCamera(Camera* cam) {

camera = cam;
}

void CullScene(const std::vector<Entity*>& allEntities) {

[Link]();

44
 if (!camera) return;

// Get camera frustum

Frustum frustum = camera->GetFrustum();

// Check each entity against the frustum

for (auto entity : allEntities) {
if (!entity->IsActive()) continue;

auto meshComponent = entity->GetComponent<MeshComponent>();

if (!meshComponent) continue;

auto transformComponent = entity->GetComponent<TransformComponent>();

if (!transformComponent) continue;

// Get bounding box of the mesh

BoundingBox boundingBox = meshComponent->GetBoundingBox();

// Transform bounding box by entity transform

[Link](transformComponent->GetTransformMatrix());

// Check if bounding box is visible

if ([Link](boundingBox)) {
visibleEntities.push_back(entity);
}
}
}

const std::vector<Entity*>& GetVisibleEntities() const {

return visibleEntities;
}
};

Render Pass Management

Modern rendering techniques often require multiple passes, each with a specific purpose. A render
pass manager helps organize these passes and their dependencies.

In this tutorial, we use Vulkan’s dynamic rendering feature with vk::raii instead of traditional
render passes. Dynamic rendering simplifies the rendering process by allowing us to begin and end
rendering operations with a single command, without explicitly creating VkRenderPass and
VkFramebuffer objects.

Rendergraphs and Synchronization

A rendergraph is a higher-level abstraction that represents the entire rendering process as a
directed acyclic graph (DAG), where nodes are render passes and edges represent dependencies
between them. This approach offers several advantages over traditional render pass management:

45
What is a Rendergraph?

A rendergraph is a data structure that:

1. Describes Resources: Tracks all resources (textures, buffers) used in rendering.

2. Defines Operations: Specifies what operations (render passes) will be performed.

3. Manages Dependencies: Automatically determines the dependencies between operations.

4. Handles Synchronization: Automatically inserts necessary synchronization primitives.

5. Optimizes Memory: Can perform memory aliasing and other optimizations.

Rendergraph: Data Structure Architecture and

Resource Representation
First, we need to establish the fundamental data structures that represent rendering resources and
passes within the rendergraph system.

// A comprehensive rendergraph implementation for automated dependency management

class Rendergraph {
private:
// Resource description and management structure
// Represents Image resource used during rendering (textures)
struct ImageResource {
std::string name; // Human-readable identifier for
debugging and referencing
vk::Format format; // Pixel format (RGBA8, Depth24Stencil8,
etc.)
vk::Extent2D extent; // Dimensions in pixels for 2D resources
vk::ImageUsageFlags usage; // How this resource will be used (color
attachment, texture, etc.)
vk::ImageLayout initialLayout; // Expected layout when the frame begins
vk::ImageLayout finalLayout; // Required layout when the frame ends

// Actual GPU resources - populated during compilation

vk::raii::Image image = nullptr; // The GPU image object
vk::raii::DeviceMemory memory = nullptr; // Backing memory allocation
vk::raii::ImageView view = nullptr; // Shader-accessible view of the image
};

// Render pass representation within the graph structure

// Each pass represents a distinct rendering operation with defined inputs and
outputs
struct Pass {
std::string name; // Descriptive name for debugging and
profiling
std::vector<std::string> inputs; // Resources this pass reads from
(dependencies)
std::vector<std::string> outputs; // Resources this pass writes to

46
 (products)
std::function<void(vk::raii::CommandBuffer&)> executeFunc; // The actual
rendering code
};

// Core data storage for the rendergraph system

std::unordered_map<std::string, Resource> resources; // All resources referenced
in the graph
std::vector<Pass> passes; // All rendering passes in
definition order
std::vector<size_t> executionOrder; // Computed optimal
execution sequence

// Automatic synchronization management

// These objects ensure correct GPU execution order without manual barriers
std::vector<vk::raii::Semaphore> semaphores; // GPU synchronization
primitives
std::vector<std::pair<size_t, size_t>> semaphoreSignalWaitPairs; // (signaling
pass, waiting pass)

vk::raii::Device& device; // Vulkan device for resource creation

public:
explicit Rendergraph(vk::raii::Device& dev) : device(dev) {}

The data structure architecture reflects the core philosophy of rendergraphs: treating rendering as
a series of transformations on resources rather than imperative GPU commands. The Resource
structure encapsulates everything needed to create and manage GPU resources, while the Pass
structure defines rendering operations in terms of their resource dependencies rather than their
implementation details.

This approach enables powerful optimizations like automatic memory aliasing (where multiple
resources share the same memory if their lifetimes don’t overlap) and optimal resource layout
transitions. The separation between resource description and actual GPU objects allows the
rendergraph to make informed decisions about resource management during the compilation
phase.

Rendergraph: Resource Registration and Pass

Definition Interface
Now for the public interface for building the rendergraph by registering resources and defining
rendering passes with their dependencies.

// Resource registration interface for declaring all resources used during

rendering
// This method establishes resource metadata without creating actual GPU resources
void AddResource(const std::string& name, vk::Format format, vk::Extent2D extent,
vk::ImageUsageFlags usage, vk::ImageLayout initialLayout,

47
 vk::ImageLayout finalLayout) {
Resource resource;
[Link] = name; // Store human-readable identifier
[Link] = format; // Define pixel format and bit depth
[Link] = extent; // Set resource dimensions
[Link] = usage; // Specify intended usage patterns
[Link] = initialLayout; // Define starting layout state
[Link] = finalLayout; // Define required ending state

resources[name] = resource; // Register in the global resource

map
}

// Pass registration interface for defining rendering operations and their

dependencies
// This method establishes the logical structure of rendering without immediate
execution
void AddPass(const std::string& name,
const std::vector<std::string>& inputs,
const std::vector<std::string>& outputs,
std::function<void(vk::raii::CommandBuffer&)> executeFunc) {
Pass pass;
[Link] = name; // Assign descriptive identifier
[Link] = inputs; // List all resources this pass reads
[Link] = outputs; // List all resources this pass
writes
[Link] = executeFunc; // Store the actual rendering
implementation

passes.push_back(pass); // Add to the ordered pass list

}

The registration interface enables declarative rendergraph construction where developers specify
what they want to achieve rather than how to achieve it. This high-level approach allows the
rendergraph to analyze the entire rendering pipeline before making resource allocation and
scheduling decisions.

The deferred execution model (where passes store function objects rather than immediate GPU
commands) enables powerful compile-time optimizations. The rendergraph can reorder passes,
merge compatible operations, and optimize resource usage based on the complete dependency
graph rather than making local decisions for each pass.

Rendergraph: Dependency Analysis and Execution

Ordering
Now we implement the core algorithmic logic that analyzes pass dependencies and computes an
optimal execution order for the rendering pipeline.

48
 // Rendergraph compilation - transforms declarative descriptions into executable
pipeline
// This method performs dependency analysis, resource allocation, and execution
planning
void Compile() {
// Dependency Graph Construction
// Build bidirectional dependency relationships between passes
std::vector<std::vector<size_t>> dependencies([Link]()); // What each
pass depends on
std::vector<std::vector<size_t>> dependents([Link]()); // What depends
on each pass

// Track which pass produces each resource (write-after-write dependencies)

std::unordered_map<std::string, size_t> resourceWriters;

// Dependency Discovery Through Resource Usage Analysis

// Analyze each pass to determine data flow relationships
for (size_t i = 0; i < [Link](); ++i) {
const auto& pass = passes[i];

// Process input dependencies - this pass must wait for producers

for (const auto& input : [Link]) {
auto it = [Link](input);
if (it != [Link]()) {
// Found the pass that produces this input - create dependency
link
dependencies[i].push_back(it->second); // This pass depends
on the producer
dependents[it->second].push_back(i); // Producer has this
as dependent
}
}

// Register output production - subsequent passes may depend on these

for (const auto& output : [Link]) {
resourceWriters[output] = i; // Record this pass as
producer
}
}

// Topological Sort for Optimal Execution Order

// Use depth-first search to compute valid execution sequence while detecting
cycles
std::vector<bool> visited([Link](), false); // Track completed
nodes
std::vector<bool> inStack([Link](), false); // Track current
recursion path

std::function<void(size_t)> visit = [&](size_t node) {

if (inStack[node]) {

49
 // Cycle detection - circular dependency found
throw std::runtime_error("Cycle detected in rendergraph");
}

if (visited[node]) {
return; // Already processed this node and its dependencies
}

inStack[node] = true; // Mark as currently being processed

// Recursively process all dependent passes first (post-order traversal)

for (auto dependent : dependents[node]) {
visit(dependent);
}

inStack[node] = false; // Remove from current path

visited[node] = true; // Mark as completely processed
executionOrder.push_back(node); // Add to execution sequence
};

// Process all unvisited nodes to handle disconnected graph components

for (size_t i = 0; i < [Link](); ++i) {
if (!visited[i]) {
visit(i);
}
}

The dependency analysis represents the mathematical core of the rendergraph system,
transforming an abstract description of rendering operations into a concrete execution plan. The
bidirectional dependency tracking enables efficient graph traversal algorithms and provides the
information needed for automatic synchronization.

The topological sort algorithm ensures that passes execute in dependency order while detecting
impossible circular dependencies that would represent logical errors in the rendering pipeline
design. This compile-time validation catches many common rendering pipeline bugs before they
manifest as runtime GPU synchronization issues.

Rendergraph: Automatic Synchronization and

Resource Allocation
Next create the GPU synchronization objects needed for correct execution ordering and allocate the
actual Vulkan resources for all registered resources.

// Automatic Synchronization Object Creation

// Generate semaphores for all dependencies identified during analysis
for (size_t i = 0; i < [Link](); ++i) {
for (auto dep : dependencies[i]) {
// Create a GPU semaphore for this dependency relationship

50
 // The dependent pass will wait on this semaphore before executing
semaphores.emplace_back([Link]({}));
semaphoreSignalWaitPairs.emplace_back(dep, i); // (producer,
consumer) pair
}
}

// Physical Resource Allocation and Creation

// Transform resource descriptions into actual GPU objects
for (auto& [name, resource] : resources) {
// Configure image creation parameters based on resource description
vk::ImageCreateInfo imageInfo;
[Link](vk::ImageType::e2D) // 2D
texture/render target
.setFormat([Link]) // Pixel
format from description
.setExtent({[Link], [Link], 1})
// Dimensions
.setMipLevels(1) // Single
mip level for simplicity
.setArrayLayers(1) // Single
layer (not array texture)
.setSamples(vk::SampleCountFlagBits::e1) // No
multisampling
.setTiling(vk::ImageTiling::eOptimal) // GPU-
optimal memory layout
.setUsage([Link]) // Usage
flags from registration
.setSharingMode(vk::SharingMode::eExclusive) // Single
queue family access
.setInitialLayout(vk::ImageLayout::eUndefined); // Initial
layout (will be transitioned)

[Link] = [Link](imageInfo); // Create

the GPU image object

// Allocate backing memory for the image

vk::MemoryRequirements memRequirements =
[Link]();

vk::MemoryAllocateInfo allocInfo;
[Link]([Link]) // Required
memory size

.setMemoryTypeIndex(FindMemoryType([Link],

vk::MemoryPropertyFlagBits::eDeviceLocal)); // GPU-local memory

[Link] = [Link](allocInfo); // Allocate

GPU memory
[Link](*[Link], 0); // Bind

51
 memory to image

// Create image view for shader access

vk::ImageViewCreateInfo viewInfo;
[Link](*[Link]) // Reference
the created image
.setViewType(vk::ImageViewType::e2D) // 2D view
type
.setFormat([Link]) // Match
image format
.setSubresourceRange({vk::ImageAspectFlagBits::eColor, 0, 1, 0,
1}); // Full image access

[Link] = [Link](viewInfo); // Create

shader-accessible view
}
}

// Resource access interface for retrieving compiled resources

Resource* GetResource(const std::string& name) {
auto it = [Link](name);
return (it != [Link]()) ? &it->second : nullptr;
}

Rendergraph: Execution Engine and Command

Recording
Finally, implement the execution engine that coordinates pass execution with proper
synchronization and resource transitions.

// Rendergraph execution engine - coordinates pass execution with automatic

synchronization
// This method transforms the compiled rendergraph into actual GPU work
void Execute(vk::raii::CommandBuffer& commandBuffer, vk::Queue queue) {
// Execution state management for dynamic synchronization
std::vector<vk::CommandBuffer> cmdBuffers; // Command buffer storage
std::vector<vk::Semaphore> waitSemaphores; // Synchronization
dependencies for current pass
std::vector<vk::PipelineStageFlags> waitStages; // Pipeline stages to
wait on
std::vector<vk::Semaphore> signalSemaphores; // Semaphores to signal
after current pass

// Ordered Pass Execution with Automatic Dependency Management

// Execute each pass in the computed dependency-safe order
for (auto passIdx : executionOrder) {
const auto& pass = passes[passIdx];

52
 // Synchronization Setup - Collect Dependencies for Current Pass
// Determine what this pass must wait for before executing
[Link]();
[Link]();

for (size_t i = 0; i < [Link](); ++i) {

if (semaphoreSignalWaitPairs[i].second == passIdx) {
// This pass depends on the completion of another pass
waitSemaphores.push_back(*semaphores[i]);
// Wait for dependency completion

waitStages.push_back(vk::PipelineStageFlagBits::eColorAttachmentOutput); // Wait at
output stage
}
}

// Collect semaphores that this pass will signal for dependent passes
[Link]();
for (size_t i = 0; i < [Link](); ++i) {
if (semaphoreSignalWaitPairs[i].first == passIdx) {
// Other passes depend on this pass's completion
signalSemaphores.push_back(*semaphores[i]);
// Signal completion for dependents
}
}

// Command Buffer Preparation and Resource Layout Transitions

// Set up command recording and transition resources to appropriate
layouts
[Link]({});
// Begin command recording

// Transition input resources to shader-readable layouts

for (const auto& input : [Link]) {
auto& resource = resources[input];

vk::ImageMemoryBarrier barrier;
[Link]([Link])
// Current resource layout
.setNewLayout(vk::ImageLayout::eShaderReadOnlyOptimal)
// Target layout for reading
.setSrcQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
// No queue family transfer
.setDstQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
.setImage(*[Link])
// Target image
.setSubresourceRange({vk::ImageAspectFlagBits::eColor, 0, 1, 0,
1}) // Full image range
.setSrcAccessMask(vk::AccessFlagBits::eMemoryWrite)
// Previous write access
.setDstAccessMask(vk::AccessFlagBits::eShaderRead);

53
 // Required read access

// Insert pipeline barrier for safe layout transition

[Link](
vk::PipelineStageFlagBits::eAllCommands,
// Wait for all previous work
vk::PipelineStageFlagBits::eFragmentShader,
// Enable fragment shader access
vk::DependencyFlagBits::eByRegion,
// Region-local dependency
0, nullptr, 0, nullptr, 1, &barrier
// Image barrier only
);
}

// Transition output resources to render target layouts

for (const auto& output : [Link]) {
auto& resource = resources[output];

vk::ImageMemoryBarrier barrier;
[Link]([Link])
// Current layout state
.setNewLayout(vk::ImageLayout::eColorAttachmentOptimal)
// Optimal for color output
.setSrcQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
.setDstQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
.setImage(*[Link])
.setSubresourceRange({vk::ImageAspectFlagBits::eColor, 0, 1, 0,
1})
.setSrcAccessMask(vk::AccessFlagBits::eMemoryRead)
// Previous read access
.setDstAccessMask(vk::AccessFlagBits::eColorAttachmentWrite);
// Required write access

// Insert barrier for safe transition to writable state

[Link](
vk::PipelineStageFlagBits::eAllCommands,
vk::PipelineStageFlagBits::eColorAttachmentOutput,
// Enable color attachment writes
vk::DependencyFlagBits::eByRegion,
0, nullptr, 0, nullptr, 1, &barrier
);
}

// Pass Execution - Execute the Actual Rendering Logic

// Call the user-provided rendering function with prepared command buffer
[Link](commandBuffer);
// Execute pass-specific rendering

// Final Layout Transitions - Prepare Resources for Subsequent Use

// Transition output resources to their final required layouts

54
 for (const auto& output : [Link]) {
auto& resource = resources[output];

vk::ImageMemoryBarrier barrier;
[Link](vk::ImageLayout::eColorAttachmentOptimal)
// Current writable layout
.setNewLayout([Link])
// Required final layout
.setSrcQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
.setDstQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED)
.setImage(*[Link])
.setSubresourceRange({vk::ImageAspectFlagBits::eColor, 0, 1, 0,
1})
.setSrcAccessMask(vk::AccessFlagBits::eColorAttachmentWrite)
// Previous write operations
.setDstAccessMask(vk::AccessFlagBits::eMemoryRead);
// Enable subsequent reads

// Insert final barrier for layout transition

[Link](
vk::PipelineStageFlagBits::eColorAttachmentOutput,
// After color writes complete
vk::PipelineStageFlagBits::eAllCommands,
// Before any subsequent work
vk::DependencyFlagBits::eByRegion,
0, nullptr, 0, nullptr, 1, &barrier
);
}

// Command Submission with Synchronization

// Submit command buffer with appropriate dependency and signaling
semaphores
[Link]();
// Finalize command recording

vk::SubmitInfo submitInfo;

[Link](static_cast<uint32_t>([Link]())) //
Dependencies to wait for
.setPWaitSemaphores([Link]())
// Dependency semaphores
.setPWaitDstStageMask([Link]())
// Pipeline stages to wait at
.setCommandBufferCount(1)
// Single command buffer
.setPCommandBuffers(&*commandBuffer)
// Command buffer to execute

.setSignalSemaphoreCount(static_cast<uint32_t>([Link]())) //
Semaphores to signal
.setPSignalSemaphores([Link]());

55
 // Signal semaphores

[Link](1, &submitInfo, nullptr);

// Submit to GPU queue
}
}

The execution engine represents the culmination of the rendergraph system, where all the analysis
and preparation work pays off in coordinated GPU execution. The automatic synchronization
ensures that passes execute in the correct order without manual barrier management, while the
automatic layout transitions handle the complex state management that Vulkan requires for
optimal performance.

This execution model demonstrates the power of the rendergraph abstraction: complex multi-pass
rendering with dozens of resources and dependencies gets reduced to a simple Execute() call, with
all the synchronization and resource management handled automatically based on the declarative
pass and resource descriptions.

private:
uint32_t FindMemoryType(uint32_t typeFilter, vk::MemoryPropertyFlags properties) {
// Implementation to find suitable memory type
// ...
return 0; // Placeholder
}
};

Vulkan Synchronization

Synchronization in Vulkan is one of the most complicated topics. Vulkan provides several
synchronization primitives to ensure correct execution order and memory visibility:

1. Semaphores: Used for synchronization between queue operations (GPU-GPU synchronization).

2. Fences: Used for synchronization between CPU and GPU.

3. Events: Used for fine-grained synchronization within a command buffer.

4. Barriers: Used to synchronize access to resources and perform layout transitions.

Proper synchronization is crucial in Vulkan because:

1. No Implicit Synchronization: Unlike OpenGL, Vulkan doesn’t provide implicit synchronization

between operations.

2. Parallel Execution: Modern GPUs execute commands in parallel, which can lead to race
conditions without proper synchronization.

3. Memory Visibility: Changes made by one operation may not be visible to another without
proper barriers.

The vulkan tutorial includes a more detailed discussion of synchronization, the proper uses of the

56
primitives described above.

• Synchronization

• Frames In Flight

• Compute Shader

• Multithreading

Pipeline Barriers

Pipeline barriers are one of the most important synchronization primitives in Vulkan. They ensure
that operations before the barrier are complete before operations after the barrier begin, and they
can also perform layout transitions for images. Let’s examine how to implement proper image
layout transitions through a comprehensive breakdown of the process.

Image Layout Transition: Barrier Configuration and

Resource Specification
First, we establish the basic barrier structure and identify which image resource needs to transition
between layouts.

// Comprehensive image layout transition implementation

// This function demonstrates proper synchronization and layout management in Vulkan
void TransitionImageLayout(vk::raii::CommandBuffer& commandBuffer,
vk::Image image,
vk::Format format,
vk::ImageLayout oldLayout,
vk::ImageLayout newLayout) {

// Configure the basic image memory barrier structure

// This barrier will coordinate memory access and layout transitions
vk::ImageMemoryBarrier barrier;
[Link](oldLayout) // Current
image layout state
.setNewLayout(newLayout) // Target
layout after transition
.setSrcQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED) // No queue
family ownership transfer
.setDstQueueFamilyIndex(VK_QUEUE_FAMILY_IGNORED) // Same
queue family throughout
.setImage(image) // Target
image for the transition
.setSubresourceRange({vk::ImageAspectFlagBits::eColor, 0, 1, 0, 1}); //
Full color image range

The image memory barrier serves as the fundamental mechanism for coordinating both memory
access patterns and image layout transitions in Vulkan. Unlike OpenGL where these operations
happen automatically, Vulkan requires explicit specification of when and how image layouts

57
change. The queue family settings using VK_QUEUE_FAMILY_IGNORED indicate that we’re not
transferring ownership between different queue families, which simplifies the synchronization
requirements.

The subresource range specification defines exactly which portions of the image are affected by
this barrier. In this case, we’re transitioning the entire color aspect of the image across all mip
levels and array layers, which is the most common scenario for basic texture operations.

Image Layout Transition: Pipeline Stage and Access

Mask Determination
Next, we analyze the specific layout transition being performed and determine the appropriate
pipeline stages and memory access patterns for optimal synchronization.

// Initialize pipeline stage tracking for synchronization timing

// These stages define when operations must complete and when new operations can
begin
vk::PipelineStageFlags sourceStage; // When previous operations must finish
vk::PipelineStageFlags destinationStage; // When subsequent operations can start

// Configure synchronization for undefined-to-transfer layout transitions

// This pattern is common when preparing images for data uploads
if (oldLayout == vk::ImageLayout::eUndefined &&
newLayout == vk::ImageLayout::eTransferDstOptimal) {

// Configure memory access permissions for upload preparation

[Link](vk::AccessFlagBits::eNone) // No
previous access to synchronize
.setDstAccessMask(vk::AccessFlagBits::eTransferWrite); // Enable
transfer write operations

// Set pipeline stage synchronization points for upload workflow

sourceStage = vk::PipelineStageFlagBits::eTopOfPipe; // No
previous work to wait for
destinationStage = vk::PipelineStageFlagBits::eTransfer; // Transfer
operations can proceed

// Configure synchronization for transfer-to-shader layout transitions

// This pattern prepares uploaded images for shader sampling
} else if (oldLayout == vk::ImageLayout::eTransferDstOptimal &&
newLayout == vk::ImageLayout::eShaderReadOnlyOptimal) {

// Configure memory access transition from writing to reading

[Link](vk::AccessFlagBits::eTransferWrite) // Previous
transfer writes must complete
.setDstAccessMask(vk::AccessFlagBits::eShaderRead); // Enable
shader read access

// Set pipeline stage synchronization for shader usage workflow

58
 sourceStage = vk::PipelineStageFlagBits::eTransfer; // Transfer
operations must complete
destinationStage = vk::PipelineStageFlagBits::eFragmentShader; // Fragment
shaders can access

} else {
// Handle unsupported transition combinations
// Production code would include additional common transition patterns
throw std::invalid_argument("Unsupported layout transition!");
}

The pipeline stage and access mask configuration represents the heart of Vulkan’s explicit
synchronization model. By specifying exactly which operations must complete before the barrier
(source stage) and which operations can begin after the barrier (destination stage), we create
precise control over GPU execution timing without unnecessary stalls.

The access mask patterns define the memory visibility requirements for each transition. The
transition from "no access" to "transfer write" enables efficient image upload without waiting for
non-existent previous operations. The transition from "transfer write" to "shader read" ensures that
uploaded data is fully written and visible before shaders attempt to sample from the texture.

Image Layout Transition: Barrier Execution and GPU

Synchronization
Finally, we submit the configured barrier to the GPU command stream, ensuring that the layout
transition and synchronization occur at the correct point in the rendering pipeline.

// Execute the pipeline barrier with configured synchronization

// This commits the layout transition and memory synchronization to the command
buffer
[Link](
sourceStage, // Wait for
these operations to complete
destinationStage, // Before
allowing these operations to begin
vk::DependencyFlagBits::eByRegion, // Enable
region-local optimizations
0, nullptr, // No
memory barriers needed
0, nullptr, // No
buffer barriers needed
1, &barrier // Apply
our image memory barrier
);
}

The pipeline barrier submission represents the culmination of our synchronization planning,

59
where the configured barrier becomes part of the GPU’s command stream. The ByRegion
dependency flag enables GPU optimizations for cases where different regions of the image can be
processed independently, potentially improving performance on tile-based renderers and other
advanced GPU architectures.

The parameter structure clearly separates different types of barriers (memory, buffer, and image),
allowing the GPU driver to apply the most efficient synchronization strategy for each resource type.
In our case, we only need image barrier synchronization, so the other barrier arrays remain empty,
avoiding unnecessary overhead.

Semaphores and Fences

Semaphores and fences are used for coarser-grained synchronization between different stages of
the rendering pipeline and between CPU and GPU operations. Let’s examine how to properly
coordinate frame rendering using these synchronization primitives through a comprehensive
breakdown of the frame rendering process.

Frame Rendering: CPU-GPU Synchronization and

Frame Pacing
First, we ensure proper coordination between CPU frame preparation and GPU execution,
preventing the CPU from getting too far ahead of the GPU and managing resource contention.

// Comprehensive frame rendering with proper synchronization

// This function demonstrates the complete cycle of frame rendering coordination
void RenderFrame(vk::raii::Device& device, vk::Queue graphicsQueue, vk::Queue
presentQueue) {

// Synchronize with previous frame completion

// Prevent CPU from submitting work faster than GPU can process it
vk::Result result = [Link](1, &*inFlightFence, VK_TRUE, UINT64_MAX);

// Reset fence for this frame's completion tracking

// Prepare the fence to signal when this frame's GPU work completes
[Link](1, &*inFlightFence);

The fence-based synchronization serves as the primary mechanism for CPU-GPU coordination in
frame rendering. By waiting for the previous frame’s fence, we ensure that the GPU has completed
processing the previous frame before beginning work on the current frame. This prevents the CPU
from overwhelming the GPU with work and helps maintain stable frame pacing.

The fence reset operation prepares the synchronization object for the current frame. Fences are
binary signals that transition from unsignaled to signaled state when associated GPU work
completes, so they must be explicitly reset before reuse. The timeout value UINT64_MAX effectively
means "wait indefinitely," which is appropriate for frame synchronization where we must ensure
completion.

60
Frame Rendering: Swapchain Image Acquisition and
Resource Preparation
Next, we acquire the next available swapchain image for rendering, coordinating with the
presentation engine to ensure proper image availability.

// Acquire next available image from the swapchain

// This operation coordinates with the presentation engine and display system
uint32_t imageIndex;
result = [Link](*swapchain, // Target
swapchain for acquisition
UINT64_MAX, // Wait
indefinitely for image availability
*imageAvailableSemaphore, // Semaphore
signaled when image is available
nullptr, // No fence
needed for this operation
&imageIndex); // Receives
index of acquired image

// Record command buffer for this frame's rendering

// Command buffer recording happens here with acquired image as render target
// ... (command recording implementation would go here)

The swapchain image acquisition represents a critical synchronization point between the
rendering system and the presentation engine. The operation may block if no images are currently
available (for example, if all swapchain images are being displayed or processed), making it
essential for frame pacing. The semaphore signaled by this operation will be used later to ensure
that rendering doesn’t begin until the acquired image is truly available for modification.

The indefinite timeout ensures that acquisition will eventually succeed even under heavy load or
when dealing with variable refresh rate displays. The acquired image index determines which
swapchain image becomes the render target for this frame, affecting descriptor set bindings and
render pass configuration in the subsequent command recording phase.

Frame Rendering: GPU Work Submission and Inter-

Queue Synchronization
Next, we submit the recorded rendering commands to the GPU with proper synchronization to
coordinate between image acquisition, rendering, and presentation operations.

// Configure GPU work submission with comprehensive synchronization

// This submission coordinates image availability, rendering, and presentation
readiness
vk::SubmitInfo submitInfo;
vk::PipelineStageFlags waitStages[] =

61
 {vk::PipelineStageFlagBits::eColorAttachmentOutput};

[Link](1) // Wait for

one semaphore before execution
.setPWaitSemaphores(&*imageAvailableSemaphore) // Don't
start until image is available
.setPWaitDstStageMask(waitStages) //
Specifically wait before color output
.setCommandBufferCount(1) // Submit
one command buffer
.setPCommandBuffers(&*commandBuffer) // The
recorded rendering commands
.setSignalSemaphoreCount(1) // Signal
one semaphore when complete
.setPSignalSemaphores(&*renderFinishedSemaphore); // Notify
when rendering is finished

// Submit work to GPU with fence-based completion tracking

// The fence allows CPU to know when this frame's GPU work has completed
[Link](1, &submitInfo, *inFlightFence);

The submission configuration demonstrates Vulkan’s explicit synchronization model for

coordinating multiple GPU operations. The wait semaphore ensures that rendering commands
don’t execute until the swapchain image is actually available for modification. The wait stage mask
specifies exactly which part of the graphics pipeline must wait—in this case, color attachment
output—allowing earlier pipeline stages to proceed if they don’t depend on the swapchain image.

The signal semaphore communicates completion of rendering work to other operations that
depend on the rendered result, such as presentation. The fence provides CPU-visible completion
notification, enabling the frame pacing logic we saw in earlier. This three-way synchronization
(wait semaphore, signal semaphore, and fence) creates a complete coordination system for the
frame rendering pipeline.

Frame Rendering: Presentation Coordination and

Display Integration
Finally, we coordinate with the presentation engine to display the rendered frame, ensuring that
presentation waits for rendering completion and handles the transition from rendering to display.

// Present the rendered image to the display

// This operation transfers the completed frame from rendering to display system
vk::PresentInfoKHR presentInfo;
[Link](1) // Wait for
rendering completion
.setPWaitSemaphores(&*renderFinishedSemaphore) // Don't
present until rendering finishes
.setSwapchainCount(1) // Present
to one swapchain

62
 .setPSwapchains(&*swapchain) // Target
swapchain for presentation
.setPImageIndices(&imageIndex); // Present
the image we rendered to

// Submit presentation request to the presentation engine

result = [Link](&presentInfo);
}

The presentation phase completes the frame rendering cycle by coordinating the transfer from
rendering to display. The wait semaphore ensures that presentation doesn’t begin until all
rendering operations have completed, preventing the display of partially rendered frames. This
synchronization is crucial because presentation and rendering may occur on different GPU queues
with different timing characteristics.

The presentation operation itself is asynchronous—it queues the presentation request and returns
immediately, allowing the CPU to begin preparing the next frame. The presentation engine handles
the actual coordination with the display hardware, including timing synchronization with refresh
rates and managing the transition of the swapchain image from "rendering" to "displaying" to
"available for reuse" states.

How Rendergraphs Help with Synchronization

Rendergraphs simplify synchronization by:

1. Automatic Dependency Tracking: The rendergraph knows which passes depend on which
resources, so it can automatically insert the necessary synchronization primitives.

2. Optimal Barrier Placement: The rendergraph can analyze the entire rendering process and
place barriers only where needed, reducing overhead.

3. Layout Transitions: The rendergraph can automatically handle image layout transitions based
on how resources are used.

4. Resource Aliasing: The rendergraph can reuse memory for resources that aren’t used
simultaneously, reducing memory usage.

Dynamic Rendering and Its Integration with Rendergraphs

Dynamic rendering is a modern Vulkan feature that simplifies the rendering process and works
particularly well with rendergraphs. Before diving into implementation examples, let’s understand
what dynamic rendering is and how it relates to our rendering pipeline architecture.

Benefits of Dynamic Rendering

Dynamic rendering offers several advantages over traditional render passes:

1. Simplified API: No need to create and manage VkRenderPass and VkFramebuffer objects,
reducing code complexity.

2. More Flexible Rendering: Easier to change render targets and attachment formats at runtime.

63
3. Improved Compatibility: Works better with modern rendering techniques that don’t fit well
into the traditional render pass model.

4. Reduced State Management: Fewer objects to track and synchronize.

5. Easier Debugging: Simpler rendering code is easier to debug and maintain.

With dynamic rendering, we specify all rendering states (render targets, load/store operations, etc.)
directly within the vkCmdBeginRendering call, rather than setting it up ahead of time in a
VkRenderPass object. This allows for more dynamic rendering workflows and simplifies the
implementation of techniques like deferred rendering.

Dynamic Rendering in Rendergraphs

When combined with rendergraphs, dynamic rendering becomes even more powerful. The
rendergraph handles the resource dependencies and synchronization, while dynamic rendering
simplifies the actual rendering process. This combination provides both flexibility and
performance.

Example: Implementing a Deferred Renderer with a Rendergraph and Dynamic Rendering

Deferred rendering represents a sophisticated rendering technique that separates geometry

processing from lighting calculations, enabling efficient handling of complex lighting scenarios.
Let’s examine how to implement this technique using rendergraphs and dynamic rendering
through a comprehensive breakdown of the setup process.

Deferred Renderer Setup: G-Buffer Resource

Configuration
First, we establish the G-Buffer (Geometry Buffer) resources that will store intermediate geometry
information for the deferred lighting pass.

// Comprehensive deferred renderer setup demonstrating rendergraph resource management

// This implementation shows how to efficiently organize multi-pass rendering
workflows
void SetupDeferredRenderer(Rendergraph& graph, uint32_t width, uint32_t height) {

// Configure position buffer for world-space vertex positions

// High precision format preserves positional accuracy for lighting calculations
[Link]("GBuffer_Position", vk::Format::eR16G16B16A16Sfloat, {width,
height},
vk::ImageUsageFlagBits::eColorAttachment |
vk::ImageUsageFlagBits::eInputAttachment,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eShaderReadOnlyOptimal);

// Configure normal buffer for surface orientation data

// High precision normals enable accurate lighting and reflection calculations
[Link]("GBuffer_Normal", vk::Format::eR16G16B16A16Sfloat, {width,
height},

64
 vk::ImageUsageFlagBits::eColorAttachment |
vk::ImageUsageFlagBits::eInputAttachment,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eShaderReadOnlyOptimal);

// Configure albedo buffer for surface color information

// Standard 8-bit precision sufficient for color data with gamma encoding
[Link]("GBuffer_Albedo", vk::Format::eR8G8B8A8Unorm, {width, height},
vk::ImageUsageFlagBits::eColorAttachment |
vk::ImageUsageFlagBits::eInputAttachment,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eShaderReadOnlyOptimal);

// Configure depth buffer for occlusion and depth testing

// High precision depth enables accurate depth reconstruction in lighting pass
[Link]("Depth", vk::Format::eD32Sfloat, {width, height},
vk::ImageUsageFlagBits::eDepthStencilAttachment |
vk::ImageUsageFlagBits::eInputAttachment,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eDepthStencilAttachmentOptimal);

// Configure final color buffer for the completed lighting result

// Standard color format with transfer capability for presentation or post-
processing
[Link]("FinalColor", vk::Format::eR8G8B8A8Unorm, {width, height},
vk::ImageUsageFlagBits::eColorAttachment |
vk::ImageUsageFlagBits::eTransferSrc,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eTransferSrcOptimal);

The G-Buffer resource configuration represents the foundation of deferred rendering, where each
buffer stores specific geometric information that will be consumed during lighting calculations. The
format choices reflect a balance between precision requirements and memory efficiency: positions
and normals use 16-bit floating point for accurate lighting calculations, while albedo uses 8-bit
integers for color data where gamma correction naturally reduces precision requirements.

The usage flag combinations enable each resource to serve dual roles: first as render targets during
the geometry pass, then as input textures during the lighting pass. This dual usage pattern is
characteristic of deferred rendering workflows, where the same data moves through multiple
pipeline stages with different access patterns.

Deferred Renderer Setup: Geometry Pass

Configuration and Multiple Render Target Setup
Next, we configure the geometry pass that populates the G-Buffer with geometric information from
the scene’s 3D models.

// Configure geometry pass for G-Buffer population

65
 // This pass renders all geometry and stores intermediate data for lighting
calculations
[Link]("GeometryPass",
{}, // No
input dependencies - first pass in pipeline
{"GBuffer_Position", "GBuffer_Normal", "GBuffer_Albedo", "Depth"},
// Outputs all G-Buffer components
[&](vk::raii::CommandBuffer& cmd) {

// Configure multiple render target attachments for G-Buffer

output
// Each attachment corresponds to a different geometric property
std::array<vk::RenderingAttachmentInfoKHR, 3> colorAttachments;

// Configure position attachment - world space vertex positions

colorAttachments[0].setImageView(/* GBuffer_Position view */)
// Target position buffer

.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal) // Optimal for writes

.setLoadOp(vk::AttachmentLoadOp::eClear)
// Clear to known state
.setStoreOp(vk::AttachmentStoreOp::eStore);
// Preserve for lighting pass

// Configure normal attachment - surface normals in world space

colorAttachments[1].setImageView(/* GBuffer_Normal view */)
// Target normal buffer

.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
// Clear to default normal
.setStoreOp(vk::AttachmentStoreOp::eStore);
// Preserve for lighting

// Configure albedo attachment - surface color and material

properties
colorAttachments[2].setImageView(/* GBuffer_Albedo view */)
// Target albedo buffer

.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
// Clear to default color
.setStoreOp(vk::AttachmentStoreOp::eStore);
// Preserve for lighting

// Configure depth attachment for occlusion culling

vk::RenderingAttachmentInfoKHR depthAttachment;
[Link](/* Depth view */)
// Target depth buffer

.setImageLayout(vk::ImageLayout::eDepthStencilAttachmentOptimal) // Optimal for depth

66
 ops
.setLoadOp(vk::AttachmentLoadOp::eClear)
// Clear to far plane
.setStoreOp(vk::AttachmentStoreOp::eStore)
// Preserve for lighting pass
.setClearValue({1.0f, 0});
// Clear to maximum depth

// Assemble complete rendering configuration

vk::RenderingInfoKHR renderingInfo;
[Link]({{0, 0}, {width, height}})
// Full screen rendering
.setLayerCount(1)
// Single layer rendering
.setColorAttachmentCount([Link]())
// Number of G-Buffer targets
.setPColorAttachments([Link]())
// G-Buffer attachment array
.setPDepthAttachment(&depthAttachment);
// Depth testing configuration

// Execute geometry rendering with dynamic rendering

[Link](renderingInfo);
// Begin G-Buffer population

// Bind geometry pipeline and render all scene objects

// Each draw call populates position, normal, and albedo for
visible fragments
// ... (geometry rendering implementation would go here)

[Link]();
// Complete G-Buffer population
});

The geometry pass configuration demonstrates the power of deferred rendering’s separation of
concerns, where geometric complexity is handled independently of lighting complexity. The
multiple render target setup enables simultaneous output to all G-Buffer components in a single
rendering pass, maximizing GPU efficiency compared to multiple separate passes.

The dynamic rendering approach eliminates the need to pre-configure render pass objects,
providing flexibility to adjust G-Buffer formats or attachment counts based on runtime
requirements. This flexibility is particularly valuable for techniques like adaptive quality settings
or optional G-Buffer components for different material types.

Deferred Renderer Setup: Lighting Pass Configuration

and Screen-Space Processing
Now we should set up the lighting pass that reads from the G-Buffer and performs all lighting
calculations in screen space, producing the final rendered image.

67
 // Configure lighting pass for screen-space illumination calculations
// This pass reads G-Buffer data and computes final lighting for each pixel
[Link]("LightingPass",
{"GBuffer_Position", "GBuffer_Normal", "GBuffer_Albedo", "Depth"},
// Read all G-Buffer components
{"FinalColor"}, // Output
final lit result
[&](vk::raii::CommandBuffer& cmd) {

// Configure single color output for final lighting result

vk::RenderingAttachmentInfoKHR colorAttachment;
[Link](/* FinalColor view */)
// Target final color buffer

.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal) // Optimal for color writes

.setLoadOp(vk::AttachmentLoadOp::eClear)
// Clear to background color
.setStoreOp(vk::AttachmentStoreOp::eStore)
// Preserve final result
.setClearValue({0.0f, 0.0f, 0.0f, 1.0f}); //
Clear to black background

// Configure lighting pass rendering without depth testing

// Depth testing unnecessary since we're processing each pixel
exactly once
vk::RenderingInfoKHR renderingInfo;
[Link]({{0, 0}, {width, height}})
// Full screen processing
.setLayerCount(1)
// Single layer output
.setColorAttachmentCount(1)
// Single color output
.setPColorAttachments(&colorAttachment);
// Final color attachment

// Execute screen-space lighting calculations

[Link](renderingInfo);
// Begin lighting pass

// Bind lighting pipeline and draw full-screen quad

// Fragment shader reads G-Buffer textures and computes lighting
for each pixel
// All scene lights are processed in a single screen-space pass
// ... (lighting calculation implementation would go here)

[Link]();
// Complete lighting calculations
});

// Compile the complete rendergraph for execution

68
 // This analyzes dependencies and generates optimal execution plan
[Link]();
}

The lighting pass represents the core advantage of deferred rendering: decoupling lighting
complexity from geometric complexity. By processing lighting in screen space, the cost becomes
proportional to screen resolution rather than scene complexity, enabling efficient handling of
scenes with many lights or complex lighting models.

The single render target configuration reflects the unified nature of the lighting pass, where all
lighting contributions are accumulated into the final color buffer. This approach enables advanced
lighting techniques like physically-based rendering or global illumination algorithms that would be
prohibitively expensive in forward rendering scenarios with complex geometry.

Best Practices for Rendergraphs and Synchronization

1. Minimize Synchronization: Use the rendergraph to minimize the number of synchronization

points.

2. Batch Similar Operations: Group similar operations together to reduce state changes.

3. Use Appropriate Access Flags: Be specific about which access types you need to synchronize.

4. Avoid Redundant Barriers: Let the rendergraph eliminate redundant barriers.

5. Consider Memory Aliasing: Use the rendergraph’s memory aliasing capabilities to reduce
memory usage.

6. Profile and Optimize: Use GPU profiling tools to identify synchronization bottlenecks.

7. Handle Platform Differences: Different GPUs may have different synchronization

requirements.

// Forward declarations
class RenderPass;
class RenderTarget;

// Render pass manager

class RenderPassManager {
private:
std::unordered_map<std::string, std::unique_ptr<RenderPass>> renderPasses;
std::vector<RenderPass*> sortedPasses;
bool dirty = true;

public:
template<typename T, typename... Args>
T* AddRenderPass(const std::string& name, Args&&... args) {
static_assert(std::is_base_of<RenderPass, T>::value, "T must derive from
RenderPass");

auto it = [Link](name);
if (it != [Link]()) {

69
 return dynamic_cast<T*>(it->[Link]());
}

auto pass = std::make_unique<T>(std::forward<Args>(args)...);

T* passPtr = [Link]();
renderPasses[name] = std::move(pass);
dirty = true;

return passPtr;
}

RenderPass* GetRenderPass(const std::string& name) {

auto it = [Link](name);
if (it != [Link]()) {
return it->[Link]();
}
return nullptr;
}

void RemoveRenderPass(const std::string& name) {

auto it = [Link](name);
if (it != [Link]()) {
[Link](it);
dirty = true;
}
}

void Execute(vk::raii::CommandBuffer& commandBuffer) {

if (dirty) {
SortPasses();
dirty = false;
}

for (auto pass : sortedPasses) {

pass->Execute(commandBuffer);
}
}

private:
void SortPasses() {
// Topologically sort render passes based on dependencies
[Link]();

// Create a copy of render passes for sorting

std::unordered_map<std::string, RenderPass*> passMap;
for (const auto& [name, pass] : renderPasses) {
passMap[name] = [Link]();
}

// Perform topological sort

std::unordered_set<std::string> visited;

70
 std::unordered_set<std::string> visiting;

for (const auto& [name, pass] : passMap) {

if ([Link](name) == [Link]()) {
TopologicalSort(name, passMap, visited, visiting);
}
}
}

void TopologicalSort(const std::string& name,

const std::unordered_map<std::string, RenderPass*>& passMap,
std::unordered_set<std::string>& visited,
std::unordered_set<std::string>& visiting) {
[Link](name);

auto pass = [Link](name);

for (const auto& dep : pass->GetDependencies()) {
if ([Link](dep) == [Link]()) {
if ([Link](dep) != [Link]()) {
// Circular dependency detected
throw std::runtime_error("Circular dependency detected in render
passes");
}
TopologicalSort(dep, passMap, visited, visiting);
}
}

[Link](name);
[Link](name);
sortedPasses.push_back(pass);
}
};

// Base render pass class

class RenderPass {
private:
std::string name;
std::vector<std::string> dependencies;
RenderTarget* target = nullptr;
bool enabled = true;

public:
explicit RenderPass(const std::string& passName) : name(passName) {}
virtual ~RenderPass() = default;

const std::string& GetName() const { return name; }

void AddDependency(const std::string& dependency) {

dependencies.push_back(dependency);
}

71
 const std::vector<std::string>& GetDependencies() const {
return dependencies;
}

void SetRenderTarget(RenderTarget* renderTarget) {

target = renderTarget;
}

RenderTarget* GetRenderTarget() const {

return target;
}

void SetEnabled(bool isEnabled) {

enabled = isEnabled;
}

bool IsEnabled() const {

return enabled;
}

virtual void Execute(vk::raii::CommandBuffer& commandBuffer) {

if (!enabled) return;

BeginPass(commandBuffer);
Render(commandBuffer);
EndPass(commandBuffer);
}

protected:
// With dynamic rendering, BeginPass typically calls vkCmdBeginRendering
// instead of vkCmdBeginRenderPass
virtual void BeginPass(vk::raii::CommandBuffer& commandBuffer) = 0;
virtual void Render(vk::raii::CommandBuffer& commandBuffer) = 0;
// With dynamic rendering, EndPass typically calls vkCmdEndRendering
// instead of vkCmdEndRenderPass
virtual void EndPass(vk::raii::CommandBuffer& commandBuffer) = 0;
};

// Render target class

class RenderTarget {
private:
vk::raii::Image colorImage = nullptr;
vk::raii::DeviceMemory colorMemory = nullptr;
vk::raii::ImageView colorImageView = nullptr;

vk::raii::Image depthImage = nullptr;

vk::raii::DeviceMemory depthMemory = nullptr;
vk::raii::ImageView depthImageView = nullptr;

uint32_t width;
uint32_t height;

72
 public:
RenderTarget(uint32_t w, uint32_t h) : width(w), height(h) {
// Create color and depth images
CreateColorResources();
CreateDepthResources();

// Note: With dynamic rendering, we don't need to create VkRenderPass

// or VkFramebuffer objects. Instead, we just create the images and
// image views that will be used directly with vkCmdBeginRendering.
}

// No need for explicit destructor with RAII objects

vk::ImageView GetColorImageView() const { return *colorImageView; }

vk::ImageView GetDepthImageView() const { return *depthImageView; }

uint32_t GetWidth() const { return width; }

uint32_t GetHeight() const { return height; }

private:
void CreateColorResources() {
// Implementation to create color image, memory, and view
// With dynamic rendering, we just need to create the image and image view
// that will be used with vkCmdBeginRendering
// ...
}

void CreateDepthResources() {
// Implementation to create depth image, memory, and view
// With dynamic rendering, we just need to create the image and image view
// that will be used with vkCmdBeginRendering
// ...
}

vk::raii::Device& GetDevice() {
// Get device from somewhere (e.g., singleton or parameter)
// ...
static vk::raii::Device device = nullptr; // Placeholder
return device;
}
};

Implementing Specific Render Passes

Now let’s implement some specific render passes:

// Geometry pass for deferred rendering

class GeometryPass : public RenderPass {

73
 private:
CullingSystem* cullingSystem;

// G-buffer textures
RenderTarget* gBuffer;

public:
GeometryPass(const std::string& name, CullingSystem* culling)
: RenderPass(name), cullingSystem(culling) {
// Create G-buffer render target
gBuffer = new RenderTarget(1920, 1080); // Example resolution
SetRenderTarget(gBuffer);
}

~GeometryPass() override {
delete gBuffer;
}

protected:
void BeginPass(vk::raii::CommandBuffer& commandBuffer) override {
// Begin rendering with dynamic rendering
vk::RenderingInfoKHR renderingInfo;

// Set up color attachment

vk::RenderingAttachmentInfoKHR colorAttachment;
[Link](gBuffer->GetColorImageView())
.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
.setStoreOp(vk::AttachmentStoreOp::eStore)
.setClearValue(vk::ClearColorValue(std::array<float, 4>{0.0f,
0.0f, 0.0f, 1.0f}));

// Set up depth attachment

vk::RenderingAttachmentInfoKHR depthAttachment;
[Link](gBuffer->GetDepthImageView())

.setImageLayout(vk::ImageLayout::eDepthStencilAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
.setStoreOp(vk::AttachmentStoreOp::eStore)
.setClearValue(vk::ClearDepthStencilValue(1.0f, 0));

// Configure rendering info

[Link](vk::Rect2D({0, 0}, {gBuffer->GetWidth(), gBuffer-
>GetHeight()}))
.setLayerCount(1)
.setColorAttachmentCount(1)
.setPColorAttachments(&colorAttachment)
.setPDepthAttachment(&depthAttachment);

// Begin dynamic rendering

[Link](renderingInfo);

74
 }

void Render(vk::raii::CommandBuffer& commandBuffer) override {

// Get visible entities
const auto& visibleEntities = cullingSystem->GetVisibleEntities();

// Render each entity to G-buffer

for (auto entity : visibleEntities) {
auto meshComponent = entity->GetComponent<MeshComponent>();
auto transformComponent = entity->GetComponent<TransformComponent>();

if (meshComponent && transformComponent) {

// Bind pipeline for G-buffer rendering
// ...

// Set model matrix

// ...

// Draw mesh
// ...
}
}
}

void EndPass(vk::raii::CommandBuffer& commandBuffer) override {

// End dynamic rendering
[Link]();
}
};

// Lighting pass for deferred rendering

class LightingPass : public RenderPass {
private:
GeometryPass* geometryPass;
std::vector<Light*> lights;

public:
LightingPass(const std::string& name, GeometryPass* gPass)
: RenderPass(name), geometryPass(gPass) {
// Add dependency on geometry pass
AddDependency(gPass->GetName());
}

void AddLight(Light* light) {

lights.push_back(light);
}

void RemoveLight(Light* light) {

auto it = std::find([Link](), [Link](), light);
if (it != [Link]()) {
[Link](it);

75
 }
}

protected:
void BeginPass(vk::raii::CommandBuffer& commandBuffer) override {
// Begin rendering with dynamic rendering
vk::RenderingInfoKHR renderingInfo;

// Set up color attachment for the lighting pass

vk::RenderingAttachmentInfoKHR colorAttachment;
[Link](GetRenderTarget()->GetColorImageView())
.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
.setStoreOp(vk::AttachmentStoreOp::eStore)
.setClearValue(vk::ClearColorValue(std::array<float, 4>{0.0f,
0.0f, 0.0f, 1.0f}));

// Configure rendering info

[Link](vk::Rect2D({0, 0}, {GetRenderTarget()->GetWidth(),
GetRenderTarget()->GetHeight()}))
.setLayerCount(1)
.setColorAttachmentCount(1)
.setPColorAttachments(&colorAttachment);

// Begin dynamic rendering

[Link](renderingInfo);
}

void Render(vk::raii::CommandBuffer& commandBuffer) override {

// Bind G-buffer textures from the geometry pass
auto gBuffer = geometryPass->GetRenderTarget();

// Set up descriptor sets for G-buffer textures

// With dynamic rendering, we access the G-buffer textures directly as shader
resources
// rather than as subpass inputs

// Render full-screen quad with lighting shader

// ...

// For each light

for (auto light : lights) {
// Set light properties
// ...

// Draw light volume

// ...
}
}

void EndPass(vk::raii::CommandBuffer& commandBuffer) override {

76
 // End dynamic rendering
[Link]();
}
};

// Post-process effect base class

class PostProcessEffect {
public:
virtual ~PostProcessEffect() = default;
virtual void Apply(vk::raii::CommandBuffer& commandBuffer) = 0;
};

// Post-processing pass
class PostProcessPass : public RenderPass {
private:
LightingPass* lightingPass;
std::vector<PostProcessEffect*> effects;

public:
PostProcessPass(const std::string& name, LightingPass* lPass)
: RenderPass(name), lightingPass(lPass) {
// Add dependency on lighting pass
AddDependency(lPass->GetName());
}

void AddEffect(PostProcessEffect* effect) {

effects.push_back(effect);
}

void RemoveEffect(PostProcessEffect* effect) {

auto it = std::find([Link](), [Link](), effect);
if (it != [Link]()) {
[Link](it);
}
}

protected:
void BeginPass(vk::raii::CommandBuffer& commandBuffer) override {
// Begin rendering with dynamic rendering
vk::RenderingInfoKHR renderingInfo;

// Set up color attachment for the post-processing pass

vk::RenderingAttachmentInfoKHR colorAttachment;
[Link](GetRenderTarget()->GetColorImageView())
.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal)
.setLoadOp(vk::AttachmentLoadOp::eClear)
.setStoreOp(vk::AttachmentStoreOp::eStore)
.setClearValue(vk::ClearColorValue(std::array<float, 4>{0.0f,
0.0f, 0.0f, 1.0f}));

// Configure rendering info

77
 [Link](vk::Rect2D({0, 0}, {GetRenderTarget()->GetWidth(),
GetRenderTarget()->GetHeight()}))
.setLayerCount(1)
.setColorAttachmentCount(1)
.setPColorAttachments(&colorAttachment);

// Begin dynamic rendering

[Link](renderingInfo);
}

void Render(vk::raii::CommandBuffer& commandBuffer) override {

// With dynamic rendering, each effect can set up its own rendering state
// and access input textures directly as shader resources

// Apply each post-process effect

for (auto effect : effects) {
effect->Apply(commandBuffer);
}
}

void EndPass(vk::raii::CommandBuffer& commandBuffer) override {

// End dynamic rendering
[Link]();
}
};

Command Generation and Execution

Once we have our render passes set up, we need to generate and execute commands:

class Renderer {
private:
vk::raii::Device device = nullptr;
vk::Queue graphicsQueue;
vk::raii::CommandPool commandPool = nullptr;

RenderPassManager renderPassManager;
CullingSystem cullingSystem;

// Current frame resources

vk::raii::CommandBuffer commandBuffer = nullptr;
vk::raii::Fence fence = nullptr;
vk::raii::Semaphore imageAvailableSemaphore = nullptr;
vk::raii::Semaphore renderFinishedSemaphore = nullptr;

public:
Renderer(vk::raii::Device& dev, vk::Queue queue) : device(dev),
graphicsQueue(queue) {
// Create command pool

78
 // ...

// Create synchronization objects

// ...

// Set up render passes

SetupRenderPasses();
}

// No need for explicit destructor with RAII objects

void SetCamera(Camera* camera) {

[Link](camera);
}

void Render(const std::vector<Entity*>& entities) {

// Wait for previous frame to finish
[Link](UINT64_MAX);
[Link]();

// Reset command buffer

[Link]();

// Perform culling
[Link](entities);

// Record commands
vk::CommandBufferBeginInfo beginInfo;
[Link](beginInfo);

// Execute render passes

[Link](commandBuffer);

[Link]();

// Submit command buffer

vk::SubmitInfo submitInfo;

// With vk::raii, we need to dereference the command buffer

vk::CommandBuffer rawCommandBuffer = *commandBuffer;
[Link](1);
[Link](&rawCommandBuffer);

// Set up wait and signal semaphores

vk::PipelineStageFlags waitStages[] = {
vk::PipelineStageFlagBits::eColorAttachmentOutput };

// With vk::raii, we need to dereference the semaphores

vk::Semaphore rawImageAvailableSemaphore = *imageAvailableSemaphore;
vk::Semaphore rawRenderFinishedSemaphore = *renderFinishedSemaphore;

79
 [Link](1);
[Link](&rawImageAvailableSemaphore);
[Link](waitStages);
[Link](1);
[Link](&rawRenderFinishedSemaphore);

// With vk::raii, we need to dereference the fence

vk::Fence rawFence = *fence;
[Link](1, &submitInfo, rawFence);
}

private:
void SetupRenderPasses() {
// Create geometry pass
auto geometryPass =
[Link]<GeometryPass>("GeometryPass", &cullingSystem);

// Create lighting pass

auto lightingPass =
[Link]<LightingPass>("LightingPass", geometryPass);

// Create post-process pass

auto postProcessPass =
[Link]<PostProcessPass>("PostProcessPass", lightingPass);

// Add post-process effects

// ...
}
};

Advanced Rendering Techniques

For detailed information about advanced rendering techniques such as Deferred Rendering,
Forward+ Rendering, and Physically Based Rendering (PBR), please refer to the Advanced
Rendering Techniques section in the Appendix. This section includes references to valuable
resources for further reading.

Conclusion
A well-designed rendering pipeline is essential for creating a flexible and efficient rendering
engine. By implementing the techniques described in this section, you can create a system that:

1. Efficiently culls invisible objects

2. Organizes rendering into passes with clear dependencies

3. Supports advanced rendering techniques like deferred rendering and PBR

4. Can be easily extended with new effects and features

In the next section, we’ll explore event systems, which provide a flexible way for different parts of

80
your engine to communicate with each other.

Previous: Resource Management | Next: Event Systems :pp: ++

Engine Architecture: Event Systems

Event Systems
Event systems provide a flexible way for different parts of your engine to communicate with each
other without creating tight coupling. In this section, we’ll explore how to design and implement an
effective event system for your rendering engine.

The Need for Event Systems

Even in the simple engine we’re building, subsystems need to communicate with each other
efficiently. As our engine grows, these communication needs become increasingly important:

1. Physics needs to notify Audio when collisions occur.

2. Input needs to notify Game Logic when buttons are pressed.

3. Game Logic needs to notify Rendering when objects change.

4. Resource Management needs to notify Rendering when assets are loaded.

Without an event system, these interactions would require direct references between subsystems,
creating tight coupling and making the code harder to maintain and extend.

Event System Design Principles

When designing an event system, consider these principles:

1. Decoupling - Minimize dependencies between event producers and consumers.

2. Type Safety - Use the type system to prevent errors.

3. Performance - Efficiently dispatch events, especially for high-frequency events.

4. Flexibility - Support different event delivery patterns (immediate, queued, etc.).

5. Debugging - Make it easy to debug event flow.

Basic Event System Implementation

Let’s implement a basic event system:

Base event type and convenience macro

We start with a minimal base event interface and a helper macro to define strongly typed events
without boilerplate.

81
 // Base event class
class Event {
public:
virtual ~Event() = default;

// Get the type of the event

virtual const char* GetType() const = 0;

// Clone the event (for queued events)

virtual Event* Clone() const = 0;
};

// Macro to help define event types

#define DEFINE_EVENT_TYPE(type) \
static const char* GetStaticType() { return #type; } \
virtual const char* GetType() const override { return GetStaticType(); } \
virtual Event* Clone() const override { return new type(*this); }

This lets us identify and copy events generically while keeping concrete event classes small.

Concrete event types

Keep event payloads focused and lightweight; they should represent facts, not behavior.

// Example event types

class WindowResizeEvent : public Event {
private:
int width;
int height;

public:
WindowResizeEvent(int w, int h) : width(w), height(h) {}

int GetWidth() const { return width; }

int GetHeight() const { return height; }

DEFINE_EVENT_TYPE(WindowResizeEvent)
};

class KeyPressEvent : public Event {

private:
int keyCode;
bool repeat;

public:
KeyPressEvent(int key, bool isRepeat) : keyCode(key), repeat(isRepeat) {}

int GetKeyCode() const { return keyCode; }

bool IsRepeat() const { return repeat; }

82
 DEFINE_EVENT_TYPE(KeyPressEvent)
};

Listener and type-safe dispatcher

Listeners receive events; the dispatcher routes a generic Event to a typed handler when types
match.

// Event listener interface

class EventListener {
public:
virtual ~EventListener() = default;
virtual void OnEvent(const Event& event) = 0;
};

// Event dispatcher
class EventDispatcher {
private:
const Event& event;

public:
explicit EventDispatcher(const Event& e) : event(e) {}

// Dispatch event to handler if types match

template<typename T, typename F>
bool Dispatch(const F& handler) {
if ([Link]() == T::GetStaticType()) {
handler(static_cast<const T&>(event));
return true;
}
return false;
}
};

Event bus (immediate vs. queued)

The bus can deliver immediately (low latency) or queue for later (deterministic ordering across
frames).

// Event bus
class EventBus {
private:
std::vector<EventListener*> listeners;
std::queue<std::unique_ptr<Event>> eventQueue;
std::mutex queueMutex;
bool immediateMode = true;

83
 public:
void SetImmediateMode(bool immediate) {
immediateMode = immediate;
}

void AddListener(EventListener* listener) {

listeners.push_back(listener);
}

void RemoveListener(EventListener* listener) {

auto it = std::find([Link](), [Link](), listener);
if (it != [Link]()) {
[Link](it);
}
}

void PublishEvent(const Event& event) {

if (immediateMode) {
// Dispatch event immediately
for (auto listener : listeners) {
listener->OnEvent(event);
}
} else {
// Queue event for later processing
std::lock_guard<std::mutex> lock(queueMutex);
[Link](std::unique_ptr<Event>([Link]()));
}
}

void ProcessEvents() {
if (immediateMode) return;

std::queue<std::unique_ptr<Event>> currentEvents;

{
std::lock_guard<std::mutex> lock(queueMutex);
std::swap(currentEvents, eventQueue);
}

while (![Link]()) {
auto& event = *[Link]();

for (auto listener : listeners) {

listener->OnEvent(event);
}

[Link]();
}
}
};

84
Using the Event System
Here’s how you might use the event system in your application:

// Component that listens for events

class CameraController : public Component, public EventListener {
private:
CameraComponent* camera;
float moveSpeed = 5.0f;
float rotateSpeed = 0.1f;

bool moveForward = false;

bool moveBackward = false;
bool moveLeft = false;
bool moveRight = false;

public:
void Initialize() override {
camera = GetOwner()->GetComponent<CameraComponent>();

// Register as event listener

GetEventBus().AddListener(this);
}

void Update(float deltaTime) override {

if (!camera) return;

// Handle movement
glm::vec3 movement(0.0f);

if (moveForward) movement.z -= 1.0f;

if (moveBackward) movement.z += 1.0f;
if (moveLeft) movement.x -= 1.0f;
if (moveRight) movement.x += 1.0f;

if (glm::length(movement) > 0.0f) {

movement = glm::normalize(movement) * moveSpeed * deltaTime;

auto transform = GetOwner()->GetComponent<TransformComponent>();

if (transform) {
glm::vec3 position = transform->GetPosition();
position += movement;
transform->SetPosition(position);
}
}
}

void OnEvent(const Event& event) override {

EventDispatcher dispatcher(event);

85
 // Handle key press events
[Link]<KeyPressEvent>([this](const KeyPressEvent& e) {
switch ([Link]()) {
case KEY_W: moveForward = true; break;
case KEY_S: moveBackward = true; break;
case KEY_A: moveLeft = true; break;
case KEY_D: moveRight = true; break;
}
return false;
});

// Handle key release events

[Link]<KeyReleaseEvent>([this](const KeyReleaseEvent& e) {
switch ([Link]()) {
case KEY_W: moveForward = false; break;
case KEY_S: moveBackward = false; break;
case KEY_A: moveLeft = false; break;
case KEY_D: moveRight = false; break;
}
return false;
});

// Handle window resize events

[Link]<WindowResizeEvent>([this](const WindowResizeEvent& e) {
if (camera) {
float aspectRatio = static_cast<float>([Link]()) /
static_cast<float>([Link]());
camera->SetAspectRatio(aspectRatio);
}
return false;
});
}

~CameraController() override {
// Unregister as event listener
GetEventBus().RemoveListener(this);
}

private:
EventBus& GetEventBus() {
// Get event bus from somewhere (e.g., singleton or parameter)
static EventBus eventBus;
return eventBus;
}
};

// Input system that generates events

class InputSystem {
private:
EventBus& eventBus;

86
 // Key states
std::unordered_map<int, bool> keyStates;

public:
explicit InputSystem(EventBus& bus) : eventBus(bus) {}

void Update() {
// Poll input events from the platform
// ...

// Example: Process a key press

ProcessKeyPress(KEY_W, false);
}

void ProcessKeyPress(int keyCode, bool repeat) {

bool& keyState = keyStates[keyCode];

if (!keyState || repeat) {
// Key was not pressed before or this is a repeat
KeyPressEvent event(keyCode, repeat);
[Link](event);
}

keyState = true;
}

void ProcessKeyRelease(int keyCode) {

bool& keyState = keyStates[keyCode];

if (keyState) {
// Key was pressed before
KeyReleaseEvent event(keyCode);
[Link](event);
}

keyState = false;
}
};

Advanced Event System Features

Event Categories

Events can be categorized to allow listeners to filter which types of events they receive:

// Event categories
enum class EventCategory {
None = 0,
Application = 1 << 0,

87
 Input = 1 << 1,
Keyboard = 1 << 2,
Mouse = 1 << 3,
MouseButton = 1 << 4,
Window = 1 << 5
};

// Enhanced event base class

class Event {
public:
virtual ~Event() = default;

virtual const char* GetType() const = 0;

virtual Event* Clone() const = 0;

// Get the categories this event belongs to

virtual int GetCategoryFlags() const = 0;

// Check if event is in category

bool IsInCategory(EventCategory category) const {
return GetCategoryFlags() & static_cast<int>(category);
}
};

// Enhanced macro to define event types with categories

#define DEFINE_EVENT_TYPE_CATEGORY(type, categoryFlags) \
static const char* GetStaticType() { return #type; } \
virtual const char* GetType() const override { return GetStaticType(); } \
virtual Event* Clone() const override { return new type(*this); } \
virtual int GetCategoryFlags() const override { return categoryFlags; }

// Example event with categories

class KeyPressEvent : public Event {
private:
int keyCode;
bool repeat;

public:
KeyPressEvent(int key, bool isRepeat) : keyCode(key), repeat(isRepeat) {}

int GetKeyCode() const { return keyCode; }

bool IsRepeat() const { return repeat; }

DEFINE_EVENT_TYPE_CATEGORY(KeyPressEvent,
static_cast<int>(EventCategory::Input) |
static_cast<int>(EventCategory::Keyboard))
};

88
Event Filtering

Listeners can filter events based on categories:

// Enhanced event bus with filtering

class EventBus {
private:
struct ListenerInfo {
EventListener* listener;
int categoryFilter;
};

std::vector<ListenerInfo> listeners;
std::queue<std::unique_ptr<Event>> eventQueue;
std::mutex queueMutex;
bool immediateMode = true;

public:
void AddListener(EventListener* listener, int categoryFilter = -1) {
listeners.push_back({listener, categoryFilter});
}

void RemoveListener(EventListener* listener) {

auto it = std::find_if([Link](), [Link](),
[listener](const ListenerInfo& info) {
return [Link] == listener;
});
if (it != [Link]()) {
[Link](it);
}
}

void PublishEvent(const Event& event) {

if (immediateMode) {
// Dispatch event immediately
for (const auto& info : listeners) {
if ([Link] == -1 || ([Link]() &
[Link])) {
[Link]->OnEvent(event);
}
}
} else {
// Queue event for later processing
std::lock_guard<std::mutex> lock(queueMutex);
[Link](std::unique_ptr<Event>([Link]()));
}
}

// Rest of the implementation...

};

89
Event Priorities

Some events may need to be processed before others:

// Enhanced event bus with priorities

class EventBus {
private:
struct ListenerInfo {
EventListener* listener;
int categoryFilter;
int priority;
};

std::vector<ListenerInfo> listeners;
// Rest of the implementation...

public:
void AddListener(EventListener* listener, int categoryFilter = -1, int priority =
0) {
listeners.push_back({listener, categoryFilter, priority});

// Sort listeners by priority (higher priority first)

std::sort([Link](), [Link](),
[](const ListenerInfo& a, const ListenerInfo& b) {
return [Link] > [Link];
});
}

// Rest of the implementation...

};

Event Bubbling and Capturing

In hierarchical systems like UI, events can propagate through the hierarchy in two ways:

• Event Bubbling - The event starts at the target element and "bubbles up" through parent
elements in the hierarchy. For example, a click event on a button first triggers on the button,
then on its container, and continues up to the root element.

• Event Capturing - The event starts at the root element and travels down the hierarchy to the
target element (the opposite direction of bubbling).

This approach allows parent elements to intercept and handle events triggered on their children,
while also giving children the ability to stop propagation if needed. For hierarchical systems like UI,
this provides a flexible way to handle events at the appropriate level:

// UI event with bubbling

class UIEvent : public Event {
private:

90
 UIElement* target;
bool bubbles;
bool cancelBubble = false;

public:
UIEvent(UIElement* targetElement, bool bubbling = true)
: target(targetElement), bubbles(bubbling) {}

UIElement* GetTarget() const { return target; }

bool Bubbles() const { return bubbles; }

void StopPropagation() {
cancelBubble = true;
}

bool IsPropagationStopped() const {

return cancelBubble;
}

DEFINE_EVENT_TYPE_CATEGORY(UIEvent, static_cast<int>(EventCategory::UI))
};

// UI system with event bubbling

class UISystem {
public:
void DispatchEvent(UIEvent& event) {
UIElement* target = [Link]();

// Capturing phase (top-down)

std::vector<UIElement*> path;
UIElement* current = target;

while (current) {
path.push_back(current);
current = current->GetParent();
}

// Dispatch to each element in the path (bottom-up)

for (auto it = [Link](); it != [Link](); ++it) {
(*it)->OnEvent(event);

if ([Link]()) {
break;
}
}
}
};

91
Conclusion
A well-designed event system is crucial for creating a flexible and maintainable engine
architecture. By implementing the techniques described in this section, you can create a system
that:

1. Decouples subsystems, making your code more modular and easier to maintain

2. Provides type-safe event handling

3. Supports different event delivery patterns

4. Can be extended with advanced features like filtering, priorities, and bubbling

This concludes our exploration of engine architecture. In this chapter, we’ve covered:

1. Architectural patterns for structuring your engine

2. Component systems for building flexible game objects

3. Resource management for efficiently handling assets

4. Rendering pipeline design for flexible and efficient rendering

5. Event systems for decoupled communication between subsystems

With these foundations in place, you’re well-equipped to build a robust and flexible rendering
engine that can be extended to support a wide range of features and techniques.

Previous: Rendering Pipeline | Next: Conclusion :pp: ++

Engine Architecture: Conclusion

Conclusion
In this chapter, we’ve explored the fundamental architectural patterns and design principles that
form the backbone of a modern rendering engine. Let’s recap what we’ve learned and discuss how
to apply these concepts in your own engine development.

What We’ve Covered

This chapter has taken you through the foundational thinking that separates successful engine
development from ad-hoc rendering code. We began by examining architectural patterns that have
proven effective in production engines—layered architecture provides clear separation of
concerns, component-based systems enable flexible object composition, data-oriented design
unlocks performance potential, and service locators manage dependencies cleanly. Understanding
these patterns helps you choose the right structural approach for different engine subsystems,
balancing flexibility against complexity based on your specific needs.

The component system implementation demonstrated how composition can replace deep
inheritance hierarchies, creating more maintainable and flexible code. This approach allows you to

92
build diverse game objects by combining reusable components rather than creating complex class
hierarchies that become difficult to extend and modify.

Resource management emerged as a critical foundation that affects every other system. Our
implementation showcases how reference counting, caching, and hot reloading work together to
optimize memory usage while improving development workflow. These techniques become
essential as your projects scale beyond simple scenes to complex, asset-heavy applications.

The rendering pipeline structure provides the framework for accommodating different rendering
techniques and effects. By organizing stages for scene culling, render pass management, command
generation, and post-processing, we’ve created a system that can evolve with your rendering needs
without requiring fundamental architectural changes.

Finally, the event system implementation shows how to enable communication between engine
subsystems without creating tight coupling. Features like event filtering, priorities, and bubbling
create a flexible communication layer that scales from simple notifications to complex interaction
patterns.

Applying These Concepts

The transition from understanding architectural patterns to implementing them successfully
requires a disciplined approach that balances ambition with pragmatism. Starting with minimal
implementations provides a solid foundation you can build upon—complex architectures often
hide subtle bugs that become exponentially harder to debug as system complexity increases. Each
additional layer of abstraction should solve a concrete problem you’ve encountered, not anticipate
hypothetical future needs.

Interface design becomes your most powerful tool for managing complexity as your engine grows.
Well-defined interfaces act as contracts between subsystems, allowing you to modify or completely
replace implementations without cascading changes throughout your codebase. This separation of
concerns proves invaluable when optimizing performance, adding features, or adapting to new
requirements.

Performance considerations need to inform architectural decisions from the beginning, though this
differs from premature optimization. Certain structural choices—like data layout patterns, memory
allocation strategies, and inter-system communication mechanisms—create performance ceilings
that become extremely expensive to change later. Understanding these implications helps you
make informed trade-offs during initial design phases.

Successful engine development requires embracing iteration and refactoring as core activities
rather than necessary evils. Your understanding of requirements will evolve as you build and use
your engine, and rigid adherence to initial designs often leads to increasingly awkward
workarounds. Regular refactoring keeps your architecture aligned with actual needs rather than
theoretical ideals.

The balance between flexibility and complexity represents perhaps the most challenging aspect of
engine architecture. Every abstraction layer and configurable system adds cognitive overhead and
potential failure points, but insufficient flexibility leads to brittle, hard-to-extend code. Finding the
right balance depends on understanding your specific project constraints, team size, timeline, and

93
performance requirements.

Next Steps
The architectural foundation we’ve established in this chapter will support everything we build in
subsequent chapters. As we progress through camera systems, lighting, model loading, and
advanced features, each new system will integrate with these core patterns rather than existing as
isolated components.

Active implementation proves far more valuable than passive reading when learning engine
architecture. Build the code examples as you encounter them, but don’t stop there—experiment
with variations to understand how different approaches affect your engine’s behavior. This
experimentation develops the intuitive understanding that separates competent engine developers
from those who merely copy implementations.

The architectural concepts we’ve covered provide a foundation, but production engines require
additional sophistication. The Appendix explores advanced rendering techniques and architectural
patterns that build on these fundamentals, helping you understand how simple patterns scale to
handle complex real-world requirements.

Studying existing open-source engines like Hazel or examining the architectural decisions in
established frameworks like LWJGL provides valuable perspective on how these concepts apply in
practice. Look for patterns we’ve discussed and notice how different engines make different trade-
offs based on their specific goals and constraints.

The graphics programming community offers tremendous value for learning and problem-solving.
Engaging with forums, Discord servers, and GitHub discussions exposes you to diverse approaches
and helps you get feedback on your architectural decisions. Often, discussing your implementation
choices with others reveals assumptions you didn’t realize you were making.

Remember that engine development is an iterative process. Your architecture will evolve as you
gain experience and as your requirements change. The concepts we’ve covered provide a
foundation, but the best architecture for your engine will depend on your specific goals and
constraints.

Final Thoughts
Building a rendering engine is a challenging but rewarding endeavor. By applying the architectural
patterns and design principles we’ve explored in this chapter, you’ll be well-equipped to create a
robust, flexible, and maintainable engine that can grow with your needs.

Good luck with your engine development journey!

Previous: Event Systems | Next: Camera Transformations :pp: ++

94
Camera & Transformations:
Introduction
Introduction
Welcome to the "Camera & Transformations" chapter of our "Building a Simple Engine" series! In
this chapter, we’ll dive into the essential mathematics and techniques needed to implement a 3D
camera system in Vulkan.

Understanding how to manipulate 3D space is fundamental to creating interactive 3D applications.

We’ll explore the mathematical foundations of 3D transformations and implement a flexible
camera system that will allow us to navigate and view our 3D scenes from any perspective.

In this chapter, we’ll focus on:

• Understanding the mathematical foundations of 3D transformations

• Implementing different types of transformation matrices (model, view, projection)

• Creating a flexible camera system with different movement modes

• Handling user input to control the camera

• Integrating the camera system with our Vulkan rendering pipeline

By the end of this chapter, you’ll have a solid understanding of 3D transformations and a reusable
camera system that can be integrated into your Vulkan applications.

Prerequisites
Before starting this chapter, you should have completed the main Vulkan tutorial. You should also
be familiar with:

• Basic Vulkan concepts:

◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

• Basic programming concepts and C++

Previous: Engine Architecture Conclusion | Next: Mathematical Foundations :pp: ++

95
Camera & Transformations:
Mathematical Foundations
Mathematical Foundations for 3D Graphics
Before diving into camera implementation, let’s review the essential mathematical concepts that
form the foundation of 3D graphics programming. Understanding these concepts is crucial for
implementing a robust camera system.

Vectors in 3D Graphics
Vectors are fundamental to 3D graphics as they represent positions, directions, and movements in
space. In our Vulkan application, we’ll primarily work with:

• 3D vectors (x, y, z): Used for positions, directions, and normals

• 4D vectors (x, y, z, w): Used for homogeneous coordinates in transformations

Why Vectors Matter in Graphics

In our camera system, vectors serve several critical purposes:

• The camera’s position is represented as a 3D vector

• The camera’s viewing direction is a 3D vector

• The "up" direction that orients the camera is also a vector

Vector Operations and Their Applications

• Addition and Subtraction: Used for calculating relative positions and movements

◦ Example: newPosition = currentPosition + movementDirection * speed

• Scalar Multiplication: Used for scaling movements and directions

◦ Example: Slowing down camera movement by multiplying velocity by a factor < 1

• Dot Product: Calculates the dot product between two vectors. For normalized vectors, this
equals the cosine of the angle between them.

◦ Applications: Determining if objects are facing the camera, calculating lighting intensity

The Right-Hand Rule

The right-hand rule is a convention used in 3D graphics and mathematics to determine the
orientation of coordinate systems and the direction of cross-products.

• For Cross Products: When calculating A × B:

1. Point your right hand’s index finger in the direction of vector A

96
 2. Point your middle finger in the direction of vector B (perpendicular to A)

3. Your thumb now points in the direction of the resulting cross-product

• For Coordinate Systems: In a right-handed coordinate system:

1. Point your right hand’s index finger along the positive X-axis

2. Point your middle finger along the positive Y-axis

3. Your thumb points along the positive Z-axis

// The cross product direction follows the right-hand rule

glm::vec3 xAxis(1.0f, 0.0f, 0.0f); // Point right (positive X)
glm::vec3 yAxis(0.0f, 1.0f, 0.0f); // Point up (positive Y)

// Cross product gives the Z axis in a right-handed system

glm::vec3 zAxis = glm::cross(xAxis, yAxis); // Points forward (positive Z)
// zAxis will be (0.0f, 0.0f, 1.0f)

// If we reverse the order, we get the opposite direction

glm::vec3 negativeZ = glm::cross(yAxis, xAxis); // Points backward (negative Z)
// negativeZ will be (0.0f, 0.0f, -1.0f)

• Cross Product: Creates a vector perpendicular to two input vectors

◦ Applications: Generating the camera’s "right" vector from "forward" and "up" vectors

◦ The direction follows the right-hand rule (explained above)

• Normalization: Preserves the direction while setting length to 1

◦ Applications: Ensuring consistent movement speed regardless of direction

// Vector operations using GLM

glm::vec3 a(1.0f, 2.0f, 3.0f);
glm::vec3 b(4.0f, 5.0f, 6.0f);

// Addition - combining positions or offsets

glm::vec3 sum = a + b; // (5.0, 7.0, 9.0)

// Dot product - useful for lighting calculations

float dotProduct = glm::dot(a, b); // 32.0
// If vectors are normalized, dot product = cosine of angle between them
float cosAngle = glm::dot(glm::normalize(a), glm::normalize(b)); // ~0.974

// Cross product - creating perpendicular vectors (e.g., camera orientation)

glm::vec3 crossProduct = glm::cross(a, b); // (-3.0, 6.0, -3.0)

// Normalization - ensuring consistent movement speeds

glm::vec3 normalized = glm::normalize(a); // (0.267, 0.535, 0.802)

97
Matrices and Transformations
Matrices are used to represent transformations in 3D space. In Vulkan and other graphics APIs, we
typically use 4×4 matrices to represent transformations in homogeneous coordinates.

Why We Use 4×4 Matrices

Even though we work in 3D space, we use 4×4 matrices because:

1. They allow us to represent translation (movement) along with rotation and scaling

2. They can be combined (multiplied) to create complex transformations

3. They work with homogeneous coordinates (x, y, z, w) which are required for perspective
projection

Common Transformation Matrices

• Translation Matrix: Moves objects in 3D space

◦ In a camera system: Moving the camera position

• Rotation Matrix: Rotates objects around an axis

◦ In a camera system: Changing where the camera is looking

• Scale Matrix: Changes the size of objects

◦ Less commonly used for cameras, but important for objects in the scene

• Model Matrix: Combines transformations to position an object in world space

◦ Positions the objects relative to the world origin

• View Matrix: Transforms world space to camera space

◦ Essentially positions the world relative to the camera

• Projection Matrix: Transforms camera space to clip space

◦ Defines how 3D objects are projected onto the 2D screen

◦ Controls perspective, field of view, and visible range (near/far planes)

// Matrix transformations using GLM

// Translation matrix - moving an object
glm::mat4 translationMatrix = glm::translate(glm::mat4(1.0f), glm::vec3(1.0f, 2.0f,
3.0f));

// Rotation matrix (45 degrees around Y axis) - turning an object

glm::mat4 rotationMatrix = glm::rotate(glm::mat4(1.0f), glm::radians(45.0f),
glm::vec3(0.0f, 1.0f, 0.0f));

// Scale matrix - resizing an object

glm::mat4 scaleMatrix = glm::scale(glm::mat4(1.0f), glm::vec3(2.0f, 2.0f, 2.0f));

// Combining transformations (scale, then rotate, then translate)

98
 // Order matters! The rightmost transformation is applied first
glm::mat4 modelMatrix = translationMatrix * rotationMatrix * scaleMatrix;

Matrix Order Matters

The order of matrix multiplication is crucial because transformations are applied from right to left.
Getting the order wrong can completely change your object’s final position and orientation.

Consider this practical example: if you want to rotate a cube around its own center and then move
it to a new position, you must apply the transformations in the correct order:

// CORRECT: Scale first, then rotate, then translate

// This rotates the cube around its own center, then moves it
glm::mat4 modelMatrix = translationMatrix * rotationMatrix * scaleMatrix;

// WRONG: Translate first, then rotate

// This would move the cube away from origin, then rotate it around the world origin
// The cube would orbit around the world center instead of rotating in place!
glm::mat4 wrongMatrix = rotationMatrix * translationMatrix * scaleMatrix;

For our camera pipeline: projectionMatrix * viewMatrix * modelMatrix * vertex Each

transformation prepares the data for the next stage, and changing this order would break the
rendering pipeline.

Visual Example: Why Matrix Order Matters

The following diagram illustrates the difference between correct and incorrect matrix
multiplication order when transforming a cube:

[Matrix Order Comparison showing correct T×R×S vs incorrect R×T×S transformation sequences] |
../../../images/[Link]
Figure 1. Matrix Transformation Order Comparison

Row-Major vs. Column-Major Representation

When working with matrices in graphics programming, it’s important to understand the difference
between row-major and column-major representations:

• Row-Major: Matrix elements are stored row by row in memory

◦ Used by DirectX, C/C++ multi-dimensional arrays

◦ A matrix is accessed as M[row][column]

• Column-Major: Matrix elements are stored column by column in memory

◦ Used by OpenGL, GLSL, and by default in GLM

◦ A matrix is accessed as M[column][row] (in memory layout terms)

99
 // Row-major vs Column-major representation of a 3x3 matrix
// For a matrix:
// [ a b c ]
// [ d e f ]
// [ g h i ]

// Row-major memory layout:

// [a, b, c, d, e, f, g, h, i]

// Column-major memory layout:

// [a, d, g, b, e, h, c, f, i]

// In GLM, matrices are column-major by default

glm::mat4 matrix = glm::mat4(1.0f); // Identity matrix in column-major format

// When passing matrices to Vulkan shaders, you need to be aware of the layout
// Vulkan expects column-major by default, matching GLM's default

Vulkan and Matrix Layouts

Vulkan works with both row-major and column-major formats, but you need to specify which one
you’re using:

• By default, Vulkan expects matrices in column-major format

• You can specify row-major format in your shaders using the row_major qualifier

• GLM (commonly used with Vulkan) uses column-major by default, but can be configured for
row-major

The practical implications:

• Matrix multiplication order may need to be reversed depending on the layout

• When debugging, matrix elements may appear transposed compared to mathematical notation

• When porting code between different APIs, matrix layouts may need to be transposed

Affine Transformations
Affine transformations are a fundamental concept in computer graphics that preserve parallel lines
(but not necessarily angles or distances). They’re essential for representing most common
operations in 3D graphics.

Properties of Affine Transformations

An affine transformation can be represented as a combination of:

• Linear transformations (rotation, scaling, shearing)

• Translation (movement)

100
In mathematical terms, an affine transformation can be expressed as:

f(x) = Ax + b

where A is a matrix (linear transformation) and b is a vector (translation).

Why Affine Transformations Matter in Graphics

• They preserve collinearity (points on a line remain on a line)

• They preserve ratios of distances along a line

• They can represent all the common transformations we need in 3D graphics

• They can be efficiently composed (combined) through matrix multiplication

Representing Affine Transformations with Homogeneous Coordinates

In 3D graphics, we use 4×4 matrices to represent affine transformations using homogeneous

coordinates:

// A 4×4 matrix representing an affine transformation

// [ R R R Tx ]
// [ R R R Ty ]
// [ R R R Tz ]
// [ 0 0 0 1 ]
// Where R represents rotation/scaling/shearing and T represents translation

// Example of an affine transformation matrix in GLM

glm::mat4 affineTransform = glm::mat4(
glm::vec4(r11, r12, r13, tx), // First row
glm::vec4(r21, r22, r23, ty), // Second row
glm::vec4(r31, r32, r33, tz), // Third row
glm::vec4(0.0f, 0.0f, 0.0f, 1.0f) // Last row is always (0,0,0,1) for affine
transformations
);

Affine Transformations in Practice

In our Vulkan application, almost all transformations we perform are affine: * Moving objects
around the scene (translation) * Rotating objects to face different directions * Scaling objects to
make them larger or smaller * Combining these operations to position and orient objects

Pose Matrices
A pose matrix (also called a transformation matrix or rigid body transformation) is a specific type
of affine transformation that represents both the position and orientation of an object in 3D space.

101
Structure of a Pose Matrix

A pose matrix combines rotation and translation in a single 4×4 matrix:

// A pose matrix has this structure:

// [ R R R Tx ]
// [ R R R Ty ]
// [ R R R Tz ]
// [ 0 0 0 1 ]
// Where the 3×3 R submatrix represents rotation and [Tx,Ty,Tz] represents translation

// Creating a pose matrix in GLM

glm::mat4 poseMatrix = glm::mat4(1.0f); // Start with identity matrix
poseMatrix = glm::translate(poseMatrix, position); // Apply translation
poseMatrix = poseMatrix * rotationMatrix; // Apply rotation

Applications of Pose Matrices

Pose matrices are essential in graphics engines for:

• Object Positioning: Defining where objects are located and how they’re oriented

◦ Example: Placing a character model in the world with the correct position and facing
direction

• Camera Representation: Defining the camera’s position and orientation

◦ Example: The view matrix is the inverse of the camera’s pose matrix

• Hierarchical Transformations: Building complex objects from simpler parts

◦ Example: A character’s hand position depends on the arm position, which depends on the
torso position

• Animation: Interpolating between different poses

◦ Example: Smoothly transitioning a camera from one position/orientation to another

Extracting Information from Pose Matrices

We can extract useful information from pose matrices:

// Extracting position from a pose matrix

glm::vec3 extractPosition(const glm::mat4& poseMatrix) {
return glm::vec3(poseMatrix[3]); // The translation is stored in the last column
}

// Extracting forward direction (assuming standard OpenGL orientation)

glm::vec3 extractForwardDirection(const glm::mat4& poseMatrix) {
return -glm::vec3(poseMatrix[2]); // Negative Z axis (third column)
}

102
 // Extracting up direction
glm::vec3 extractUpDirection(const glm::mat4& poseMatrix) {
return glm::vec3(poseMatrix[1]); // Y axis (second column)
}

Implementing a Look-At Function

A "look-at" function is a fundamental tool in camera systems that creates a view matrix to orient
the camera towards a specific target point. This is one of the most common operations in 3D
graphics and provides an excellent example of how the mathematical concepts we’ve discussed are
applied in practice.

Purpose of the Look-At Function

The look-at function serves several important purposes:

• Orients the camera to face a specific point in 3D space

• Establishes the camera’s local coordinate system (right, up, forward vectors)

• Creates a view matrix that transforms world coordinates into camera space

• Simplifies camera control by focusing on a target rather than managing rotation angles

Mathematical Principles

The look-at function works by constructing an orthonormal basis (three perpendicular unit vectors)
that defines the camera’s orientation:

1. Forward Vector (Z): Points from the camera position to the target position

2. Right Vector (X): Perpendicular to both the forward vector and the world up vector

3. Up Vector (Y): Perpendicular to both the forward and right vectors

These three vectors, along with the camera position, form the view matrix that transforms world
coordinates into camera space.

Step-by-Step Implementation

Let’s implement a custom look-at function to understand how it works:

glm::mat4 createLookAtMatrix(
const glm::vec3& cameraPosition, // Where the camera is
const glm::vec3& targetPosition, // What the camera is looking at
const glm::vec3& worldUpVector // Which way is "up" (usually Y axis)
) {
// Step 1: Calculate the camera's forward direction (Z axis)
// Note: We negate this because in OpenGL/Vulkan, the camera looks down the
negative Z-axis
glm::vec3 forward = glm::normalize(cameraPosition - targetPosition);

103
 // Step 2: Calculate the camera's right direction (X axis)
// Using cross product between world up and forward direction
glm::vec3 right = glm::normalize(glm::cross(worldUpVector, forward));

// Step 3: Calculate the camera's up direction (Y axis)

// Using cross product between forward and right to ensure orthogonality
glm::vec3 up = glm::cross(forward, right);

// Step 4: Construct the rotation part of the view matrix

// Each row contains one of the camera's basis vectors
glm::mat4 rotation = glm::mat4(1.0f);
rotation[0][0] = right.x;
rotation[1][0] = right.y;
rotation[2][0] = right.z;
rotation[0][1] = up.x;
rotation[1][1] = up.y;
rotation[2][1] = up.z;
rotation[0][2] = forward.x;
rotation[1][2] = forward.y;
rotation[2][2] = forward.z;

// Step 5: Construct the translation part of the view matrix

glm::mat4 translation = glm::mat4(1.0f);
translation[3][0] = -cameraPosition.x;
translation[3][1] = -cameraPosition.y;
translation[3][2] = -cameraPosition.z;

// Step 6: Combine rotation and translation

// The translation is applied first, then the rotation
return rotation * translation;
}

Using GLM’s Built-in Look-At Function

In practice, we typically use GLM’s built-in lookAt function, which implements the same algorithm:

// Using GLM's built-in lookAt function

glm::mat4 viewMatrix = glm::lookAt(
glm::vec3(0.0f, 0.0f, 5.0f), // Camera position
glm::vec3(0.0f, 0.0f, 0.0f), // Target position (origin)
glm::vec3(0.0f, 1.0f, 0.0f) // World up vector (Y axis)
);

Practical Applications

The look-at function is used in various scenarios:

• First-Person Camera: Looking in the direction of movement

104
 • Third-Person Camera: Following a character while looking at them

• Orbit Camera: Circling around a point of interest

• Cinematic Camera: Creating smooth camera movements that focus on important objects

• Object Inspection: Allowing users to examine 3D models from different angles

Example: Implementing an Orbit Camera

Here’s how you might use the look-at function to implement an orbit camera that circles around a
target:

// Orbit camera implementation

void updateOrbitCamera(float deltaTime) {
// Update the orbit angle based on time
orbitAngle += orbitSpeed * deltaTime;

// Calculate the camera position on a circle around the target

float radius = 10.0f;
glm::vec3 cameraPosition(
targetPosition.x + radius * cos(orbitAngle),
targetPosition.y + 5.0f, // Slightly above the target
targetPosition.z + radius * sin(orbitAngle)
);

// Create the view matrix using lookAt

viewMatrix = glm::lookAt(
cameraPosition,
targetPosition,
glm::vec3(0.0f, 1.0f, 0.0f)
);
}

Example: Smooth Camera Transitions

The look-at function can also be used to create smooth transitions between different camera
positions and targets:

// Smooth camera transition

void transitionCamera(float t) { // t ranges from 0.0 to 1.0
// Interpolate between start and end positions
glm::vec3 currentPosition = glm::mix(startPosition, endPosition, t);

// Interpolate between start and end targets

glm::vec3 currentTarget = glm::mix(startTarget, endTarget, t);

// Update the view matrix

viewMatrix = glm::lookAt(
currentPosition,

105
 currentTarget,
glm::vec3(0.0f, 1.0f, 0.0f)
);
}

By understanding how the look-at function works, you gain insight into how cameras are oriented
in 3D space and how the view matrix transforms the world from the camera’s perspective.

Raycasting in 3D Graphics
Raycasting is a fundamental technique in 3D graphics that involves projecting rays from a point
into the scene and determining what they intersect with. It’s used for a wide range of applications,
from picking objects in a scene to implementing collision detection and visibility determination.

Ray Representation

A ray in 3D space is defined by an origin point and a direction vector:

struct Ray {
glm::vec3 origin; // Starting point of the ray
glm::vec3 direction; // Normalized direction vector
};

// Creating a ray
Ray createRay(const glm::vec3& origin, const glm::vec3& direction) {
Ray ray;
[Link] = origin;
[Link] = glm::normalize(direction); // Ensure direction is normalized
return ray;
}

Ray-Object Intersection

The core of raycasting is determining if and where a ray intersects with objects in the scene. Let’s
look at some common intersection tests:

Ray-Sphere Intersection

One of the simplest intersection tests is between a ray and a sphere:

struct Sphere {
glm::vec3 center;
float radius;
};

bool rayIntersectsSphere(const Ray& ray, const Sphere& sphere, float& t) {

// Vector from ray origin to sphere center

106
 glm::vec3 oc = [Link] - [Link];

// Quadratic equation coefficients

float a = glm::dot([Link], [Link]); // Always 1 if direction is
normalized
float b = 2.0f * glm::dot(oc, [Link]);
float c = glm::dot(oc, oc) - [Link] * [Link];

// Discriminant
float discriminant = b * b - 4 * a * c;

if (discriminant < 0) {
// No intersection
return false;
}

// Find the nearest intersection point

float sqrtDiscriminant = sqrt(discriminant);
float t0 = (-b - sqrtDiscriminant) / (2 * a);
float t1 = (-b + sqrtDiscriminant) / (2 * a);

// Check if intersection is in front of the ray

if (t0 > 0) {
t = t0;
return true;
}

if (t1 > 0) {
t = t1;
return true;
}

// Both intersections are behind the ray

return false;
}

Ray-Triangle Intersection

Triangle intersection is essential for raycasting against 3D models:

struct Triangle {
glm::vec3 v0, v1, v2; // Vertices
};

bool rayIntersectsTriangle(const Ray& ray, const Triangle& triangle, float& t,

glm::vec2& barycentricCoords) {
// Möller–Trumbore algorithm
glm::vec3 edge1 = triangle.v1 - triangle.v0;
glm::vec3 edge2 = triangle.v2 - triangle.v0;
glm::vec3 h = glm::cross([Link], edge2);

107
 float a = glm::dot(edge1, h);

// Check if ray is parallel to triangle

if (a > -0.00001f && a < 0.00001f) {
return false;
}

float f = 1.0f / a;
glm::vec3 s = [Link] - triangle.v0;
float u = f * glm::dot(s, h);

// Check if intersection is outside triangle

if (u < 0.0f || u > 1.0f) {
return false;
}

glm::vec3 q = glm::cross(s, edge1);

float v = f * glm::dot([Link], q);

// Check if intersection is outside triangle

if (v < 0.0f || u + v > 1.0f) {
return false;
}

// Compute intersection distance

t = f * glm::dot(edge2, q);

// Check if intersection is behind the ray

if (t <= 0.0f) {
return false;
}

// Store barycentric coordinates for interpolation

barycentricCoords = glm::vec2(u, v);
return true;
}

Ray-AABB Intersection

Axis-Aligned Bounding Box (AABB) intersection is useful for broad-phase collision detection:

struct AABB {
glm::vec3 min; // Minimum corner
glm::vec3 max; // Maximum corner
};

bool rayIntersectsAABB(const Ray& ray, const AABB& aabb, float& tMin, float& tMax) {
// Compute intersection with each slab
glm::vec3 invDir = 1.0f / [Link];
glm::vec3 t0 = ([Link] - [Link]) * invDir;

108
 glm::vec3 t1 = ([Link] - [Link]) * invDir;

// Handle negative directions

glm::vec3 tSmaller = glm::min(t0, t1);
glm::vec3 tBigger = glm::max(t0, t1);

// Find entry and exit points

tMin = glm::max(tSmaller.x, glm::max(tSmaller.y, tSmaller.z));
tMax = glm::min(tBigger.x, glm::min(tBigger.y, tBigger.z));

// Check if there's a valid intersection

return tMax >= tMin && tMax > 0;
}

Creating Camera Rays

One of the most common uses of raycasting is to create rays from the camera into the scene, which
is essential for picking objects or implementing ray tracing:

Ray createCameraRay(
const glm::vec2& screenCoord, // Normalized screen coordinates (-1 to 1)
const glm::mat4& viewMatrix, // Camera view matrix
const glm::mat4& projectionMatrix // Camera projection matrix
) {
// Convert to clip space
glm::vec4 clipCoords(screenCoord.x, screenCoord.y, -1.0f, 1.0f);

// Convert to view space

glm::mat4 invProjection = glm::inverse(projectionMatrix);
glm::vec4 viewCoords = invProjection * clipCoords;
viewCoords.z = -1.0f; // Point towards negative Z in view space
viewCoords.w = 0.0f; // Convert to direction vector

// Convert to world space

glm::mat4 invView = glm::inverse(viewMatrix);
glm::vec4 worldCoords = invView * viewCoords;

// Create ray
Ray ray;
[Link] = glm::vec3(invView[3]); // Camera position in world space
[Link] = glm::normalize(glm::vec3(worldCoords));

return ray;
}

Applications of Raycasting in Graphics

Raycasting has numerous applications in 3D graphics and game development:

109
 • Object Picking: Determining which object the user clicked on in a 3D scene

◦ Cast a ray from the camera through the mouse position and find the nearest intersection

• Collision Detection: Checking if objects will collide along a movement path

◦ Cast rays in the direction of movement to detect potential collisions

• Line of Sight: Determining if one object can "see" another

◦ Cast a ray between two objects and check for obstructions

• Terrain Height Sampling: Finding the height of terrain at a specific point

◦ Cast a ray downward from above the terrain

• Physics Simulations: Implementing realistic physics behaviors

◦ Raycasting is fundamental to many physics engines for collision resolution

• AI Navigation: Helping AI characters navigate environments

◦ Raycasting can detect obstacles and determine valid paths

Optimizing Raycasting Performance

For complex scenes with many objects, raycasting can become computationally expensive. Here are
some optimization techniques:

• Spatial Partitioning: Use data structures like octrees, BVHs, or k-d trees to quickly eliminate
objects that can’t possibly intersect with the ray

• Bounding Volume Hierarchies: Test against simple bounding volumes (spheres, AABBs) before
performing more expensive tests against detailed geometry

• Level of Detail: Use simpler collision geometry for distant objects

• Ray Batching: Process multiple rays together to take advantage of SIMD instructions

• Early Termination: Stop testing once you’ve found any intersection (if that’s all you need)

Projection in 3D Graphics
Projection is the process of transforming 3D coordinates in view space to 2D coordinates on the
screen. In computer graphics, we use projection matrices to perform this transformation.

Types of Projection

There are two main types of projection used in 3D graphics:

• Perspective Projection: Objects appear smaller as they get farther away, simulating how we
see the world

• Orthographic Projection: Objects maintain their size regardless of distance, useful for
technical drawings, 2D games, and UI elements

110
Perspective Projection

Perspective projection creates a realistic view where distant objects appear smaller, creating the
illusion of depth:

// Creating a perspective projection matrix

glm::mat4 createPerspectiveMatrix(
float fovY, // Vertical field of view in degrees
float aspectRatio, // Width / height of the viewport
float nearPlane, // Distance to the near clipping plane
float farPlane // Distance to the far clipping plane
) {
return glm::perspective(glm::radians(fovY), aspectRatio, nearPlane, farPlane);
}

The perspective projection matrix performs several transformations:

1. Scales the view frustum based on the field of view and aspect ratio

2. Maps the view volume to a canonical view volume (a cube from -1 to 1 in each dimension)

3. Applies perspective division (dividing by w) to create the perspective effect

The resulting matrix has this structure:

// Structure of a perspective projection matrix

// [ (h/w)*cot(fovY/2) 0 0 0 ]
// [ 0 cot(fovY/2) 0 0 ]
// [ 0 0 -(f+n)/(f-n) -2*f*n/(f-n) ]
// [ 0 0 -1 0 ]
// Where:
// - fovY is the vertical field of view
// - w/h is the aspect ratio
// - n is the near plane distance
// - f is the far plane distance

Orthographic Projection

Orthographic projection maintains the size of objects regardless of their distance from the camera:

// Creating an orthographic projection matrix

glm::mat4 createOrthographicMatrix(
float left, // Left plane coordinate
float right, // Right plane coordinate
float bottom, // Bottom plane coordinate
float top, // Top plane coordinate
float nearPlane, // Near plane distance
float farPlane // Far plane distance
) {

111
 return glm::ortho(left, right, bottom, top, nearPlane, farPlane);
}

The orthographic projection matrix simply scales and translates the view volume to the canonical
view volume without applying any perspective division:

// Structure of an orthographic projection matrix

// [ 2/(r-l) 0 0 -(r+l)/(r-l) ]
// [ 0 2/(t-b) 0 -(t+b)/(t-b) ]
// [ 0 0 -2/(f-n) -(f+n)/(f-n) ]
// [ 0 0 0 1 ]
// Where:
// - l, r are the left and right planes
// - b, t are the bottom and top planes
// - n, f are the near and far planes

The View Frustum

The view frustum is the volume of space visible to the camera. For perspective projection, it’s a
truncated pyramid:

• Near Plane: The closest plane to the camera where rendering begins

• Far Plane: The farthest plane from the camera where rendering ends

• Field of View (FOV): The angle that determines how wide the view is

• Aspect Ratio: The ratio of width to height of the viewport

// Calculating the corners of the view frustum

void calculateFrustumCorners(
float fovY,
float aspectRatio,
float nearPlane,
float farPlane,
glm::vec3 corners[8] // Output array for the 8 corners
) {
float tanHalfFovY = tan(glm::radians(fovY) / 2.0f);

// Near plane dimensions

float nearHeight = 2.0f * nearPlane * tanHalfFovY;
float nearWidth = nearHeight * aspectRatio;

// Far plane dimensions

float farHeight = 2.0f * farPlane * tanHalfFovY;
float farWidth = farHeight * aspectRatio;

// Near plane corners (in view space)

corners[0] = glm::vec3(-nearWidth/2, -nearHeight/2, -nearPlane); // Bottom-left
corners[1] = glm::vec3( nearWidth/2, -nearHeight/2, -nearPlane); // Bottom-right

112
 corners[2] = glm::vec3( nearWidth/2, nearHeight/2, -nearPlane); // Top-right
corners[3] = glm::vec3(-nearWidth/2, nearHeight/2, -nearPlane); // Top-left

// Far plane corners (in view space)

corners[4] = glm::vec3(-farWidth/2, -farHeight/2, -farPlane); // Bottom-left
corners[5] = glm::vec3( farWidth/2, -farHeight/2, -farPlane); // Bottom-right
corners[6] = glm::vec3( farWidth/2, farHeight/2, -farPlane); // Top-right
corners[7] = glm::vec3(-farWidth/2, farHeight/2, -farPlane); // Top-left
}

Projection and Unprojection

Projection converts 3D world coordinates to 2D screen coordinates, while unprojection does the
reverse. The following code examples demonstrate these concepts for educational purposes:

These utility functions are provided to help understand the mathematical concepts
behind projection and unprojection. While they may not be directly used in the
NOTE
basic rendering pipeline, they are valuable for implementing features like object
picking, mouse interaction with 3D objects, and custom rendering techniques.

// Project a 3D point to screen space

glm::vec2 projectPoint(
const glm::vec3& worldPoint,
const glm::mat4& viewMatrix,
const glm::mat4& projectionMatrix,
const glm::vec4& viewport // (x, y, width, height)
) {
// Transform to clip space
glm::vec4 clipSpace = projectionMatrix * viewMatrix * glm::vec4(worldPoint, 1.0f);

// Perspective division
glm::vec3 ndcSpace = glm::vec3(clipSpace) / clipSpace.w;

// Map to viewport
glm::vec2 screenPos;
screenPos.x = (ndcSpace.x + 1.0f) * 0.5f * viewport.z + viewport.x;
screenPos.y = (1.0f - ndcSpace.y) * 0.5f * viewport.w + viewport.y; // Y is
flipped

return screenPos;
}

// Unproject a screen point to a ray in world space

Ray unprojectScreenPoint(
const glm::vec2& screenPoint,
const glm::mat4& viewMatrix,
const glm::mat4& projectionMatrix,
const glm::vec4& viewport // (x, y, width, height)
) {

113
 // Convert to normalized device coordinates
glm::vec3 ndcPos;
ndcPos.x = 2.0f * (screenPoint.x - viewport.x) / viewport.z - 1.0f;
ndcPos.y = 1.0f - 2.0f * (screenPoint.y - viewport.y) / viewport.w; // Y is
flipped
ndcPos.z = -1.0f; // Near plane

// Create ray from camera through this point

return createCameraRay(glm::vec2(ndcPos.x, ndcPos.y), viewMatrix,
projectionMatrix);
}

Applications of Projection in Graphics

Projection matrices are used in various ways in 3D graphics:

• Rendering: Converting 3D scene geometry to 2D screen pixels

• Shadow Mapping: Projecting the scene from a light’s perspective to determine shadows

• Reflection/Refraction: Calculating how light bounces off or passes through surfaces

• Texture Projection: Mapping textures onto surfaces based on a projector’s perspective

• Screen-Space Effects: Implementing post-processing effects like screen-space reflections or

ambient occlusion

Choosing the Right Projection

The choice between perspective and orthographic projection depends on the application:

• Use Perspective Projection for:

◦ First-person or third-person games

◦ Realistic 3D visualizations

◦ Any application where depth perception is important

• Use Orthographic Projection for:

◦ 2D games with 3D elements

◦ Technical drawings and CAD applications

◦ UI elements that shouldn’t be affected by perspective

◦ Isometric or top-down games

Quaternions for Rotations

While rotation matrices work well, quaternions offer advantages for certain rotation operations,
particularly for smooth camera movements and avoiding "gimbal lock" (loss of a degree of freedom
in certain orientations).

114
Why Use Quaternions?

• More compact representation (4 components vs. 9 for a rotation matrix)

• Easier to interpolate smoothly between orientations (important for camera animations)

• Avoids gimbal lock issues that can occur with Euler angles (pitch, yaw, roll)

// Quaternion operations using GLM

// Create a quaternion from Euler angles (in radians)
glm::quat rotation = glm::quat(glm::vec3(
glm::radians(30.0f), // pitch (X) - looking up/down
glm::radians(45.0f), // yaw (Y) - looking left/right
glm::radians(60.0f) // roll (Z) - tilting the camera
));

// Convert quaternion to rotation matrix for use in rendering

glm::mat4 rotationMatrix = glm::mat4_cast(rotation);

// Rotate a vector using a quaternion (e.g., rotating the camera's forward vector)
glm::vec3 original(1.0f, 0.0f, 0.0f);
glm::vec3 rotated = rotation * original;

Coordinate Systems in 3D Graphics

Understanding the different coordinate systems is essential for implementing a camera system. As
data moves through the rendering pipeline, it undergoes several transformations:

• Local Space (Object Space): Coordinates relative to the object’s origin

◦ Where vertices are initially defined relative to their own object

• World Space: Coordinates relative to the world origin

◦ Where objects are positioned relative to each other in the scene

• View Space (Camera Space): Coordinates relative to the camera

◦ The world as seen from the camera’s position and orientation

◦ The camera is at the origin (0,0,0) looking down the negative Z-axis

• Clip Space: Coordinates after projection, in the range [-w, w] for each axis

◦ Determines what’s visible on screen (inside the view frustum)

• Screen Space: Final 2D coordinates for display on the screen

◦ The actual pixel positions where objects appear

Handedness of Coordinate Systems

Graphics APIs and engines use either right-handed or left-handed coordinate systems:

• Right-Handed System (used by OpenGL and Vulkan by convention):

115
 ◦ X-axis points right

◦ Y-axis points up

◦ Z-axis points out of the screen (toward the viewer)

◦ Cross product: Z = X × Y (using the right-hand rule)

• Left-Handed System (used by DirectX):

◦ X-axis points right

◦ Y-axis points up

◦ Z-axis points into the screen (away from the viewer)

◦ Cross product: Z = X × Y (using the left-hand rule)

// In Vulkan, we typically use a right-handed coordinate system

// But we can convert between systems if needed

// Converting a point from left-handed to right-handed system

// (just flip the Z coordinate)
glm::vec3 leftHandedPoint(x, y, z);
glm::vec3 rightHandedPoint(x, y, -z);

// When setting up a camera, the handedness affects the view matrix

// In a right-handed system, the camera typically looks down the negative Z-axis
// This is why we often see -Z as the "forward" direction in camera code

Implications for Camera Systems

The handedness of your coordinate system affects how you set up your camera:

• In a right-handed system (Vulkan convention):

◦ The camera typically looks down the negative Z-axis

◦ The "look" vector is often stored as a negative Z direction

◦ The view matrix is constructed using the right-hand rule for cross products

• When extracting axes from a view matrix:

◦ Right vector: X-axis of the view matrix

◦ Up vector: Y-axis of the view matrix

◦ Forward vector: Negative Z-axis of the view matrix

The Transformation Pipeline

The transformation pipeline typically follows this sequence: Local Space → World Space → View
Space → Clip Space → Screen Space

// A typical vertex transformation in a shader

116
 gl_Position = projectionMatrix * viewMatrix * modelMatrix * vec4(vertexPosition, 1.0);

In the next section, we’ll implement these mathematical concepts to create a flexible camera system
for our Vulkan application.

Further Resources
If you’re finding some of the mathematical concepts challenging or want to deepen your
understanding, here are some helpful resources organized by topic:

General 3D Math Resources

• Books:

◦ "Mathematics for 3D Game Programming and Computer Graphics" by Eric Lengyel -

Comprehensive reference for 3D math

◦ "3D Math Primer for Graphics and Game Development" by Fletcher Dunn and Ian Parberry -
Excellent beginner-friendly introduction

◦ "Essential Mathematics for Games and Interactive Applications" by James M. Van Verth and
Lars M. Bishop - Practical approach with code examples

• Online Courses:

◦ Khan Academy Linear Algebra - Free course covering vector and matrix fundamentals

◦ Mathematics for Machine Learning: Linear Algebra - Covers vectors, matrices, and
transformations

• Interactive Tools:

◦ Quaternion Visualizer - Interactive visualization of quaternion rotations

◦ Interactive 3D Transformations - Experiment with different transformations

Vectors and Vector Operations

• Tutorials:

◦ Scratchapixel: Vectors - Detailed explanation with graphics

◦ 3Blue1Brown: Essence of Linear Algebra - Excellent visual explanations of vectors

• Interactive Tools:

◦ GeoGebra: Vector Operations - Interactive vector addition, subtraction, dot and cross
products

◦ Dot Product Visualization - Interactive visualization of dot products

Matrices and Transformations

• Tutorials:

◦ Scratchapixel: Transformations - Detailed explanation of transformation matrices

117
 ◦ LearnOpenGL: Transformations - Practical guide to transformations in graphics

• Interactive Tools:

◦ ShaderToy: Matrix Transformations - Interactive visualization of matrix transformations

◦ Red Blob Games: Interactive Transformations - Visual explanation of 2D transformations

(concepts extend to 3D)

Quaternions

• Tutorials:

◦ 3Blue1Brown: Quaternions and 3D rotation - Visual explanation of quaternions

◦ Understanding Quaternions - Practical guide with code examples

• Interactive Tools:

◦ Quaternion Visualizer - Interactive visualization of quaternion rotations

◦ ShaderToy: Quaternion Rotation - Interactive quaternion rotation visualization

Coordinate Systems and Handedness

• Tutorials:

◦ LearnOpenGL: Coordinate Systems - Explanation of different coordinate systems in graphics

◦ Scratchapixel: Coordinate Systems - Detailed explanation with graphics

• References:

◦ OpenGL Wiki: Coordinate Transformations - Reference for coordinate transformations

◦ Microsoft Docs: Coordinate Systems - Explanation of left-handed vs. right-handed systems

GLM Library (Used in our examples)

• Documentation:

◦ GLM Manual - Official documentation for the GLM math library

◦ GLM API Documentation - API reference

• Tutorials:

◦ LearnOpenGL: Transformations with GLM - Practical guide to using GLM for

transformations

◦ GLM Tutorial - Tutorial on using GLM for graphics math

Interactive Learning Tools

• Visualizations:

◦ GeoGebra 3D Calculator - Create and manipulate 3D objects and transformations

◦ ShaderToy - Experiment with shaders that use 3D math

• Practice Problems:

118
 ◦ Khan Academy: Vectors and Spaces - Practice problems for vector math

◦ Khan Academy: Matrix Transformations - Practice problems for matrix transformations

These resources should help you gain a deeper understanding of the mathematical concepts used in
3D graphics and camera systems. If you’re struggling with a particular concept, try looking at
multiple resources as different explanations might resonate better with your learning style.

Previous: Introduction | Next: Transformation Matrices :pp: ++

Camera & Transformations:

Transformation Matrices
Transformation Matrices
In this section, we’ll dive deeper into the transformation matrices used in 3D graphics and how
they’re applied in our rendering pipeline.

The Model-View-Projection (MVP) Pipeline

The transformation of vertices from object space to screen space involves a series of matrix
multiplications, commonly known as the MVP pipeline:

// The complete transformation pipeline

glm::mat4 MVP = projectionMatrix * viewMatrix * modelMatrix;

Let’s explore each of these matrices in detail.

Model Matrix
The model matrix transforms vertices from object space to world space. It positions, rotates, and
scales objects in the world.

glm::mat4 createModelMatrix(
const glm::vec3& position,
const glm::vec3& rotation,
const glm::vec3& scale
) {
// Start with identity matrix
glm::mat4 model = glm::mat4(1.0f);

// Apply transformations in order: scale, rotate, translate

model = glm::translate(model, position);

119
 // Apply rotations around each axis
model = glm::rotate(model, glm::radians(rotation.x), glm::vec3(1.0f, 0.0f, 0.0f));
model = glm::rotate(model, glm::radians(rotation.y), glm::vec3(0.0f, 1.0f, 0.0f));
model = glm::rotate(model, glm::radians(rotation.z), glm::vec3(0.0f, 0.0f, 1.0f));

// Apply scaling
model = glm::scale(model, scale);

return model;
}

View Matrix
The view matrix transforms vertices from world space to view space (camera space). It represents
the position and orientation of the camera.

glm::mat4 createViewMatrix(
const glm::vec3& cameraPosition,
const glm::vec3& cameraTarget,
const glm::vec3& upVector
) {
return glm::lookAt(cameraPosition, cameraTarget, upVector);
}

The lookAt function creates a view matrix that positions the camera at cameraPosition, looking at
cameraTarget, with upVector defining the up direction.

Projection Matrix
The projection matrix transforms vertices from view space to clip space. It defines how 3D
coordinates are projected onto the 2D screen.

Perspective Projection

Perspective projection simulates how objects appear smaller as they get farther away, which is how
our eyes naturally perceive the world.

glm::mat4 createPerspectiveMatrix(
float fovY,
float aspectRatio,
float nearPlane,
float farPlane
) {
return glm::perspective(glm::radians(fovY), aspectRatio, nearPlane, farPlane);
}

120
Parameters:

• fovY: Field of view angle in degrees (vertical)

• aspectRatio: Width divided by height of the viewport

• nearPlane: Distance to the near clipping plane

• farPlane: Distance to the far clipping plane

Orthographic Projection

Orthographic projection doesn’t have perspective distortion, making it useful for 2D rendering or
technical drawings.

glm::mat4 createOrthographicMatrix(
float left,
float right,
float bottom,
float top,
float nearPlane,
float farPlane
) {
return glm::ortho(left, right, bottom, top, nearPlane, farPlane);
}

Normal Matrix
When applying non-uniform scaling to objects, normals can become incorrect if transformed with
the model matrix. The normal matrix solves this issue:

glm::mat3 createNormalMatrix(const glm::mat4& modelMatrix) {

// The normal matrix is the transpose of the inverse of the upper-left 3x3 part of
the model matrix
return glm::transpose(glm::inverse(glm::mat3(modelMatrix)));
}

Applying Transformations in Shaders

In Vulkan, we typically pass these matrices to our shaders as uniform variables:

// Vertex shader
#version 450

layout(binding = 0) uniform UniformBufferObject {

mat4 model;
mat4 view;
mat4 proj;

121
 } ubo;

layout(location = 0) in vec3 inPosition;

layout(location = 1) in vec3 inNormal;
layout(location = 2) in vec2 inTexCoord;

layout(location = 0) out vec3 fragNormal;

layout(location = 1) out vec2 fragTexCoord;

void main() {
// Apply MVP transformation
gl_Position = [Link] * [Link] * [Link] * vec4(inPosition, 1.0);

// Transform normal using normal matrix

mat3 normalMatrix = transpose(inverse(mat3([Link])));
fragNormal = normalMatrix * inNormal;

fragTexCoord = inTexCoord;
}

Hierarchical Transformations
For complex objects or scenes with parent-child relationships, we use hierarchical transformations:

// Parent transformation
glm::mat4 parentModel = createModelMatrix(parentPosition, parentRotation,
parentScale);

// Child transformation relative to parent

glm::mat4 localModel = createModelMatrix(childLocalPosition, childLocalRotation,
childLocalScale);

// Combined transformation
glm::mat4 childWorldModel = parentModel * localModel;

In the next section, we’ll implement a camera system that uses these transformation concepts to
navigate our 3D scenes.

Previous: Mathematical Foundations | Next: Camera Implementation :pp: ++

Camera & Transformations: Camera

Implementation

122
Camera Implementation
Now that we understand the mathematical foundations and transformation matrices, let’s
implement a flexible camera system for our Vulkan application. We’ll create a camera class that can
be used to navigate our 3D scenes. This implementation is designed for a general-purpose 3D
application or game engine, and the concepts can be applied to various types of applications, from
first-person games to architectural visualization tools.

Camera Types
There are several types of cameras commonly used in 3D applications:

• First-Person Camera: Simulates viewing the world through the eyes of a character.

• Third-Person Camera: Follows a character from behind or another fixed relative position.

• Orbit Camera: Rotates around a fixed point, useful for object inspection.

• Free Camera: Allows unrestricted movement in all directions.

For our implementation, we’ll focus on a versatile camera that can be configured for different use
cases.

Camera Class Design

Our camera system is built around a Camera class that manages 3D navigation and view
generation. Let’s break down the implementation into logical sections to understand both the
technical details and design decisions behind each component.

Camera Architecture: Core Data Members and Spatial

Representation
First, we establish the fundamental data structures that represent the camera’s position,
orientation, and coordinate system within 3D space.

class Camera {
private:
// Spatial positioning and orientation vectors
// These form the camera's local coordinate system in world space
glm::vec3 position; // Camera's location in world coordinates
glm::vec3 front; // Forward direction (where camera is looking)
glm::vec3 up; // Camera's local up direction (for roll control)
glm::vec3 right; // Camera's local right direction (perpendicular to front
and up)
glm::vec3 worldUp; // Global up vector reference (typically Y-axis)

The spatial representation uses a right-handed coordinate system where the camera maintains its
own local coordinate frame within the world space. The position vector defines where the camera

123
exists, while front, up, and right vectors form an orthonormal basis that defines the camera’s
orientation. This approach provides intuitive control where moving along the front vector moves
the camera forward, right moves sideways, and up moves vertically relative to the camera’s current
orientation.

The worldUp vector serves as a reference point for maintaining proper orientation, typically
pointing along the world’s Y-axis. This reference prevents the camera from becoming disoriented
during complex rotations and ensures that operations like "level the horizon" have a consistent
reference point.

Camera Architecture: Euler Angle Representation and

Control Parameters
Next, we define how rotations are represented and controlled, using Euler angles for intuitive user
input while managing the mathematical complexities internally.

// Rotation representation using Euler angles

// Provides intuitive control while managing gimbal lock and other rotation
complexities
float yaw; // Horizontal rotation around the world up-axis (left-
right looking)
float pitch; // Vertical rotation around the camera's right axis (up-
down looking)

// User interaction and behavior parameters

// These control how the camera responds to input and environmental factors
float movementSpeed; // Units per second for translation movement
float mouseSensitivity; // Multiplier for mouse input to rotation angle conversion
float zoom; // Field of view control for perspective projection

Euler angles provide an intuitive interface for camera rotation that maps naturally to user input
devices. Yaw controls horizontal rotation (looking left-right), while pitch controls vertical rotation
(looking up-down). We deliberately avoid roll for most applications as it can be disorienting for
users, though the system could be extended to support it.

The parameter system allows fine-tuning of camera behavior for different use cases. Movement
speed can be adjusted for different scene scales, mouse sensitivity can accommodate user
preferences and different input devices, and zoom provides dynamic field-of-view control for
gameplay or cinematic effects.

Camera Architecture: Internal Methods and State

Management
Next, we define the internal methods responsible for maintaining mathematical consistency and
updating the camera’s coordinate system when rotations change.

124
 // Internal coordinate system maintenance
// Ensures mathematical consistency when orientation changes occur
void updateCameraVectors();

public:

The updateCameraVectors method serves as the mathematical foundation of the camera system,
recalculating the front, right, and up vectors whenever the Euler angles change. This process
involves trigonometric calculations that convert the intuitive Euler angle representation into the
orthonormal vector basis required for matrix operations and movement calculations.

This approach separates the user-friendly angle interface from the computationally efficient vector
operations, allowing the camera to present simple controls while maintaining the mathematical
rigor required for accurate 3D transformations.

Camera Architecture: Public Interface and

Constructor Design
Next, we establish the public interface that external code uses to create, configure, and interact
with camera instances.

// Constructor with sensible defaults for common use cases

// Provides flexibility while ensuring the camera starts in a predictable state
Camera(
glm::vec3 position = glm::vec3(0.0f, 0.0f, 0.0f), // Start at world origin
glm::vec3 up = glm::vec3(0.0f, 1.0f, 0.0f), // Y-axis as world up
float yaw = -90.0f, // Look along negative Z-
axis (OpenGL convention)
float pitch = 0.0f // Level horizon
);

The constructor design reflects common 3D graphics conventions and practical defaults. The
default position at the origin provides a predictable starting point, while the Y-axis world up aligns
with the standard mathematical coordinate system. The initial yaw of -90° follows OpenGL
conventions where the default view looks down the negative Z-axis, creating a right-handed
coordinate system that feels natural to users.

The parameter defaults eliminate the need for complex initialization in simple use cases while still
allowing full customization when needed for specialized applications.

Camera Architecture: Matrix Generation and

Geometric Transformation Interface
Now we define the core mathematical interface that transforms the camera’s spatial representation
into the matrices required by graphics pipelines.

125
 // Matrix generation for graphics pipeline integration
// These methods bridge between the camera's spatial representation and GPU
requirements
glm::mat4 getViewMatrix() const;
glm::mat4 getProjectionMatrix(float aspectRatio, float nearPlane = 0.1f, float
farPlane = 100.0f) const;

The matrix generation methods serve as the critical bridge between our intuitive camera
representation and the mathematical requirements of 3D graphics pipelines. The view matrix
transforms world coordinates into camera space, effectively positioning the world relative to the
camera’s viewpoint. The projection matrix then transforms camera space into clip space, handling
perspective effects and preparing coordinates for rasterization.

The separation of view and projection matrices follows standard graphics pipeline architecture,
allowing independent control over camera positioning and perspective characteristics. This design
enables techniques like changing field-of-view for zoom effects without recalculating the camera’s
spatial relationships.

Camera Architecture: Input Processing and User

Interaction
Finally, let’s define how the camera responds to various forms of user input, providing the interface
between human interaction and camera movement.

// Input processing methods for different interaction modalities

// Each method handles a specific type of user input with appropriate
transformations
void processKeyboard(CameraMovement direction, float deltaTime); // Keyboard-
based translation
void processMouseMovement(float xOffset, float yOffset, bool constrainPitch =
true); // Mouse-based rotation
void processMouseScroll(float yOffset); // Scroll-
based zoom control

// Property access methods for external systems

// Provide controlled access to internal state without exposing implementation
details
glm::vec3 getPosition() const { return position; }
glm::vec3 getFront() const { return front; }
float getZoom() const { return zoom; }
};

The input processing architecture recognizes that different input modalities serve different
purposes in camera control. Keyboard input typically handles discrete directional movement,
mouse movement provides continuous rotation control, and scroll wheels offer intuitive zoom
adjustment. Each method is designed to handle its specific input type with appropriate

126
mathematical transformations and timing considerations.

The getter methods provide controlled access to internal state, allowing external systems (like
audio systems that need listener position, or culling systems that need view direction) to access
camera properties without exposing the internal implementation details or allowing uncontrolled
modification of the camera’s state.

Camera Movement
We’ll define an enum for camera movement directions:

enum class CameraMovement {

FORWARD,
BACKWARD,
LEFT,
RIGHT,
UP,
DOWN
};

And implement the movement logic:

void Camera::processKeyboard(CameraMovement direction, float deltaTime) {

float velocity = movementSpeed * deltaTime;

switch (direction) {
case CameraMovement::FORWARD:
position += front * velocity;
break;
case CameraMovement::BACKWARD:
position -= front * velocity;
break;
case CameraMovement::LEFT:
position -= right * velocity;
break;
case CameraMovement::RIGHT:
position += right * velocity;
break;
case CameraMovement::UP:
position += up * velocity;
break;
case CameraMovement::DOWN:
position -= up * velocity;
break;
}
}

127
Handling Input Events

The camera class provides methods to process input, but integrating these with your application’s
input system requires careful consideration of different input modalities and their unique
characteristics. Let’s break down the input handling implementation to demonstrate both the
technical integration and the design principles behind effective camera controls.

Input Integration: Keyboard Input Processing and

Movement Translation
First, we handle discrete directional input from keyboards, translating key presses into camera
movement commands with proper frame-rate independence.

// Keyboard input processing for camera translation

// Handles discrete directional commands with frame-rate independent timing
void processInput(GLFWwindow* window, Camera& camera, float deltaTime) {
// WASD movement scheme following standard FPS conventions
// Each key press translates to a specific directional movement relative to camera
orientation
if (glfwGetKey(window, GLFW_KEY_W) == GLFW_PRESS)
[Link](CameraMovement::FORWARD, deltaTime); // Move forward
along camera's front vector
if (glfwGetKey(window, GLFW_KEY_S) == GLFW_PRESS)
[Link](CameraMovement::BACKWARD, deltaTime); // Move
backward opposite to front vector
if (glfwGetKey(window, GLFW_KEY_A) == GLFW_PRESS)
[Link](CameraMovement::LEFT, deltaTime); // Strafe left
along camera's right vector
if (glfwGetKey(window, GLFW_KEY_D) == GLFW_PRESS)
[Link](CameraMovement::RIGHT, deltaTime); // Strafe right
along camera's right vector

// Vertical movement controls for 3D navigation

// Space and Control provide intuitive up/down movement
if (glfwGetKey(window, GLFW_KEY_SPACE) == GLFW_PRESS)
[Link](CameraMovement::UP, deltaTime); // Move up
along camera's up vector
if (glfwGetKey(window, GLFW_KEY_LEFT_CONTROL) == GLFW_PRESS)
[Link](CameraMovement::DOWN, deltaTime); // Move down
opposite to up vector
}

The keyboard input processing follows established conventions from first-person games, where
WASD keys control horizontal movement and Space/Control handle vertical movement. This
mapping feels intuitive to users and provides complete 6-degrees-of-freedom movement control.
The frame-rate independence achieved through deltaTime ensures consistent movement speed
regardless of rendering performance, which is crucial for predictable user experience across
different hardware configurations.

128
Each movement command uses the camera’s local coordinate system rather than world
coordinates. Meaning "forward" always moves in the direction the camera is facing, "right" moves
perpendicular to the view direction, and "up" moves along the camera’s local vertical axis. This
approach provides intuitive controls that respond naturally to camera orientation changes.

Input Integration: Mouse Movement Processing and

Rotation State Management
Now, let’s handle continuous mouse input for camera rotation, managing state persistence and
coordinate system conversions for smooth camera control.

// Mouse movement callback for continuous camera rotation

// Manages state persistence and coordinate transformations for smooth rotation
control
void mouseCallback(GLFWwindow* window, double xpos, double ypos) {
// State persistence for calculating movement deltas
// Static variables maintain state between callback invocations
static bool firstMouse = true; // Flag to handle initial mouse position
static float lastX = 0.0f, lastY = 0.0f; // Previous mouse position for delta
calculation

// Handle initial mouse position to prevent sudden camera jumps

// First callback provides absolute position, not relative movement
if (firstMouse) {
lastX = xpos; // Initialize previous position
lastY = ypos;
firstMouse = false; // Disable special handling for subsequent calls
}

// Calculate mouse movement deltas since last callback

// These deltas represent the amount and direction of mouse movement
float xoffset = xpos - lastX; // Horizontal movement (left-
right)
float yoffset = lastY - ypos; // Vertical movement (inverted:
screen Y increases downward, camera pitch increases upward)

// Update state for next callback iteration

lastX = xpos;
lastY = ypos;

// Convert mouse movement to camera rotation

// Delta values drive continuous camera orientation changes
[Link](xoffset, yoffset);
}

The mouse callback demonstrates the complexities of handling continuous input in event-driven
systems. The static variables maintain state between callback invocations, which is necessary
because mouse movement is reported as absolute positions rather than relative deltas. The first-

129
mouse handling prevents jarring camera jumps when the mouse cursor is first captured.

The Y-axis inversion (lastY - ypos) addresses the coordinate system mismatch between screen
space (where Y increases downward) and camera space (where positive pitch looks upward). This
inversion ensures that moving the mouse upward rotates the camera to look up, matching user
expectations from other 3D applications.

Input Integration: Scroll Input Processing and Zoom

Control
Next, let’s work on the scroll-wheel input to give us zoom control, providing a simple interface for
field-of-view adjustments that feel natural to users.

// Scroll wheel callback for zoom control

// Provides intuitive field-of-view adjustment through scroll wheel interaction
void scrollCallback(GLFWwindow* window, double xoffset, double yoffset) {
// Direct scroll-to-zoom mapping
// Positive yoffset (scroll up) typically zooms in, negative (scroll down) zooms
out
[Link](yoffset);
}

The scroll callback maintains simplicity by directly passing the scroll delta to the camera’s zoom
processing method. This design delegates the mathematical details of zoom control to the camera
class while providing a clean interface for scroll wheel events. The scroll direction convention
(positive for zoom in, negative for zoom out) follows standard user interface patterns.

Input Integration: System Integration and Input Mode

Configuration
Finally, we establish the integration between the input callbacks and the windowing system,
configuring mouse capture and callback registration for complete camera control.

// Input system initialization and callback registration

// Establishes the connection between windowing system and camera control callbacks
void setupInputCallbacks(GLFWwindow* window) {
// Register callback functions with the windowing system
// These establish the event-driven connection between hardware input and camera
control
glfwSetCursorPosCallback(window, mouseCallback); // Connect mouse movement
to camera rotation
glfwSetScrollCallback(window, scrollCallback); // Connect scroll wheel to
camera zoom

// Configure mouse capture mode for first-person camera behavior

// Disabling the cursor provides continuous mouse input without cursor

130
 interference
glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
}

The system integration demonstrates how camera controls integrate with the broader application
architecture. The callback registration creates the event-driven connection between hardware
input and camera responses, while the cursor disabling provides the seamless mouse control
expected in 3D applications.

The GLFW_CURSOR_DISABLED mode captures the mouse cursor, allowing unlimited mouse movement
without the cursor hitting screen boundaries. This configuration is essential for first-person camera
controls where users expect to be able to turn the camera continuously in any direction without
cursor limitations.

The specific implementation of input handling will depend on your windowing

library and application architecture. The example above uses GLFW, but similar
NOTE
principles apply to other libraries like SDL, Qt, or platform-specific APIs. For more
details on input handling with GLFW, refer to the GLFW Input Guide.

Camera Rotation
For camera rotation, we’ll use mouse input to adjust the yaw and pitch angles:

void Camera::processMouseMovement(float xOffset, float yOffset, bool constrainPitch) {

xOffset *= mouseSensitivity;
yOffset *= mouseSensitivity;

yaw += xOffset;
pitch += yOffset;

// Constrain pitch to avoid flipping

if (constrainPitch) {
pitch = std::clamp(pitch, -89.0f, 89.0f);
}

// Update camera vectors based on updated Euler angles

updateCameraVectors();
}

Updating Camera Vectors

After changing the camera’s orientation, we need to recalculate the front, right, and up vectors:

void Camera::updateCameraVectors() {
// Calculate the new front vector
glm::vec3 newFront;

131
 newFront.x = cos(glm::radians(yaw)) * cos(glm::radians(pitch));
newFront.y = sin(glm::radians(pitch));
newFront.z = sin(glm::radians(yaw)) * cos(glm::radians(pitch));
front = glm::normalize(newFront);

// Recalculate the right and up vectors

right = glm::normalize(glm::cross(front, worldUp));
up = glm::normalize(glm::cross(right, front));
}

View Matrix
The view matrix transforms world coordinates into view coordinates (camera space):

glm::mat4 Camera::getViewMatrix() const {

return glm::lookAt(position, position + front, up);
}

Projection Matrix
The projection matrix transforms view coordinates into clip coordinates:

glm::mat4 Camera::getProjectionMatrix(float aspectRatio, float nearPlane, float

farPlane) const {
return glm::perspective(glm::radians(zoom), aspectRatio, nearPlane, farPlane);
}

Advanced Topics: Third-Person Camera

Implementation
In this section, we’ll explore advanced techniques for implementing a third-person camera that
follows a character while avoiding occlusion and maintaining focus on the character.

Third-Person Camera Design

A third-person camera typically needs to:

1. Follow the character at a specified distance

2. Maintain a consistent view of the character

3. Avoid being occluded by objects in the environment

4. Provide smooth transitions during movement and rotation

Let’s extend our camera class to support these features by building a specialized
ThirdPersonCamera that addresses the unique challenges of the character-following camera

132
systems.

Third-Person Camera Architecture: Target Tracking

and Spatial Relationship Management
What good is a camera if we can’t use it to target looking at things? Maybe we also want characters
to look at each other or to have them look at the camera. Let’s start work on this by figuring out
how a 'lookat' system would work and how a camera would track a target.

The getter methods provide controlled access to internal state, allowing external systems (like
audio systems that need listener position, or culling systems that need a view direction) to query
the camera without tightly coupling to the implementation. This keeps the camera easy to maintain
and extend as features are added.

Look-At Basics: Pointing the Camera at Something

Before we automate camera behaviors, let’s build an intuition for “look-at.” The idea is simple:
given we know where the camera is: "the eye," we also know what point it should face: "the target,"
and we also know which way is "up." We want to use that information to construct an orientation
that makes the camera face the target while keeping the horizon stable.

Think of it like lining up a real camera:

• Eye: “Where am I standing?”

• Target: “What am I framing in the center of the viewfinder?”

• Up: “Which direction should the top of the frame point (so the picture isn’t tilted)?”

Thus, when we get to the output of "look at," we will have a view orientation. We usually for
convenience will use an affine matrix, but it is only an orientation. After all, rotating to "look at"
something shouldn’t involve translating to a new position; so the eye will maintain the position
throughout our look-at code.

Key takeaways:

• “Look-at” defines an orientation, not a position. The position comes from the eye; look-at figures
out the directions (forward/right/up) from eye→target and the chosen up.

• The up direction should not be parallel to the eye→target direction. If they’re nearly aligned, the
camera won’t know how to keep the horizon level (it can “roll unpredictably.”)

• You can use look-at for both cameras and objects. Characters can face each other, or you can
point a spotlight or turret at a target with the same concept.

In the next section, we’ll take this one-off “point at a target” idea and turn it into a behavior:
smooth, continuous camera target tracking that follows moving subjects without jitter or sudden
snaps.

133
Implementation for camera target relationship
First, establish the fundamental relationship between the camera and its target, managing the
spatial tracking information that drives all third-person camera behaviors.

class ThirdPersonCamera : public Camera {

private:
// Target entity tracking and spatial relationship data
// These properties define the relationship between camera and the character being
followed
glm::vec3 targetPosition; // Current world position of the target character
glm::vec3 targetForward; // Target's forward direction vector for contextual
camera positioning

The target tracking system forms the foundation of third-person camera behavior by maintaining a
continuous connection between the camera and the character being followed. The targetPosition
provides the spatial anchor that the camera revolves around, while targetForward enables context-
aware camera positioning that can anticipate where the character is moving or looking.

This approach allows the camera to make intelligent positioning decisions based on the character’s
state and orientation, creating more dynamic and responsive camera behavior than simple fixed-
offset following.

Third-Person Camera Architecture: Behavioral

Configuration and Control Parameters
Now let’s work on the parameters that control how the camera behaves in relation to its target,
providing artistic and gameplay control over the camera’s characteristics.

// Camera behavior configuration parameters

// These values control the aesthetic and functional characteristics of camera
following
float followDistance; // Desired distance from target (affects intimacy
and field of view)
float followHeight; // Height offset above target (provides better
scene visibility)
float followSmoothness; // Interpolation factor for smooth camera
transitions (0 = instant, 1 = never)

The behavioral parameters provide artistic control over the camera’s personality and functional
characteristics. Follow distance affects both the visual intimacy with the character and the amount
of surrounding environment visible in the frame. Height offset ensures the camera provides good
visibility of both the character and the surrounding terrain or obstacles.

The smoothness parameter controls the camera’s responsiveness to target movement, allowing
designers to balance between immediate response, (which can feel jerky,) and smooth motion

134
(which can feel laggy). This parameter is crucial for creating camera behavior that feels natural and
responsive to different gameplay situations.

Third-Person Camera Architecture: Collision Detection

and Occlusion Management
Now, we have a camera system that will work in basic situations. However, let’s briefly talk about
the complex problem of environmental occlusion, ensuring the camera maintains visibility of the
target even when obstacles interfere with the desired positioning.

// Occlusion avoidance and collision management

// These parameters control how the camera responds to environmental obstacles
float minDistance; // Minimum allowed distance from target (prevents
camera from getting too close)
float raycastDistance; // Maximum distance for occlusion detection rays

The occlusion management system addresses one of the most challenging aspects of third-person
camera implementation: maintaining visibility when environmental geometry interferes with the
desired camera position. The minimum distance prevents the camera from getting uncomfortably
close to the character during collision situations, while the raycast distance defines how far the
camera looks ahead for potential occlusion issues.

This system enables the camera to proactively respond to environmental constraints, smoothly
adjusting its position to maintain optimal visibility without jarring transitions or sudden position
changes that can be disorienting to players.

Third-Person Camera Architecture: Internal State

Management and Motion Control
To get smooth camera motion, we need to be able to understand the FSM (Finite State Machine)
design of the Camera architecture. We manage the internal computational state required for
intelligent positioning decisions and to help solve smooth camera motion.

// Internal computational state for smooth motion control

// These variables manage the mathematical aspects of camera positioning and
movement
glm::vec3 desiredPosition; // Target position the camera wants to reach
(before collision adjustments)
glm::vec3 smoothDampVelocity; // Velocity state for smooth damping interpolation
algorithms

public:

The internal state management separates the desired camera behavior from the actual camera
position, allowing the system to handle complex scenarios where multiple forces influence camera

135
positioning. The desired position represents where the camera would ideally be placed based on
the follow parameters, while the smooth damp velocity enables sophisticated interpolation
algorithms that create natural, physics-inspired camera motion.

This separation of concerns allows the camera system to handle conflicts between desired
positioning and environmental constraints gracefully, maintaining smooth motion even when the
camera must deviate significantly from its preferred location.

Third-Person Camera Architecture: Public Interface

and Configuration Control
Now, let’s examine the external interface that allows game code to interact with and configure the
third-person camera system in a manner that can avoid tight coupling and can keep the camera as
its' own module.

// Constructor with gameplay-tuned defaults

// Default values chosen for common third-person game scenarios
ThirdPersonCamera(
float followDistance = 5.0f, // Medium distance providing good
character visibility and environment context
float followHeight = 2.0f, // Height above target for clear
sightlines over low obstacles
float followSmoothness = 0.1f, // Moderate smoothing for responsive but
stable camera motion
float minDistance = 1.0f // Minimum distance to prevent
uncomfortable close-ups
);

// Core functionality methods for camera behavior control

void updatePosition(const glm::vec3& targetPos, const glm::vec3& targetFwd, float
deltaTime);
void handleOcclusion(const Scene& scene);
void orbit(float horizontalAngle, float verticalAngle);

// Runtime configuration methods for dynamic camera adjustment

void setFollowDistance(float distance) { followDistance = distance; }
void setFollowHeight(float height) { followHeight = height; }
void setFollowSmoothness(float smoothness) { followSmoothness = smoothness; }
};

The public interface design balances ease of use with powerful functionality, providing sensible
defaults that work well for common third-person scenarios while allowing full customization when
needed. The default values are chosen based on common third-person game requirements: medium
distance for good character visibility, moderate height for environmental awareness, and balanced
smoothing for responsive yet stable motion.

The method organization separates the core update functionality (which typically runs every
frame) from configuration methods (which are called less frequently) and specialized behaviors

136
like orbiting (which might be triggered by specific user input). This design makes it easy to integrate
the camera into different game loop architectures while maintaining a clear separation of concerns.

Character Following Algorithm

The core of a third-person camera is the algorithm that positions the camera relative to the
character. Here’s an implementation of the updatePosition method:

void ThirdPersonCamera::updatePosition(
const glm::vec3& targetPos,
const glm::vec3& targetFwd,
float deltaTime
) {
// Update target properties
targetPosition = targetPos;
targetForward = glm::normalize(targetFwd);

// Calculate the desired camera position

// Position the camera behind and above the character
glm::vec3 offset = -targetForward * followDistance;
offset.y = followHeight;

desiredPosition = targetPosition + offset;

// Smooth camera movement using exponential smoothing

position = glm::mix(position, desiredPosition, 1.0f - pow(followSmoothness,
deltaTime * 60.0f));

// Update the camera to look at the target

front = glm::normalize(targetPosition - position);

// Recalculate right and up vectors

right = glm::normalize(glm::cross(front, worldUp));
up = glm::normalize(glm::cross(right, front));
}

This implementation:

1. Positions the camera behind the character based on the character’s forward direction

2. Adds height to give a better view of the character and surroundings

3. Uses exponential smoothing to create natural camera movement

4. Always keeps the camera focused on the character

Occlusion Avoidance

One of the most challenging aspects of a third-person camera is handling occlusion - when objects
in the environment block the view of the character. Here’s an implementation of occlusion
avoidance:

137
 void ThirdPersonCamera::handleOcclusion(const Scene& scene) {
// Cast a ray from the target to the desired camera position
Ray ray;
[Link] = targetPosition;
[Link] = glm::normalize(desiredPosition - targetPosition);

// Check for intersections with scene objects

RaycastHit hit;
if ([Link](ray, hit, glm::length(desiredPosition - targetPosition))) {
// If there's an intersection, move the camera to the hit point
// minus a small offset to avoid clipping
float offsetDistance = 0.2f;
position = [Link] - ([Link] * offsetDistance);

// Ensure we don't get too close to the target

float currentDistance = glm::length(position - targetPosition);
if (currentDistance < minDistance) {
position = targetPosition + [Link] * minDistance;
}

// Update the camera to look at the target

front = glm::normalize(targetPosition - position);
right = glm::normalize(glm::cross(front, worldUp));
up = glm::normalize(glm::cross(right, front));
}
}

This implementation:

1. Casts a ray from the character to the desired camera position

2. If the ray hits an object, moves the camera to the hit point (with a small offset)

3. Ensures the camera doesn’t get too close to the character

4. Updates the camera orientation to maintain focus on the character

Performance Considerations for Occlusion Avoidance

When implementing occlusion avoidance, be mindful of performance:

• Use simplified collision geometry: For raycasting, use simpler collision shapes than your
rendering geometry

• Limit the frequency of occlusion checks: You may not need to check every frame on slower
devices

• Consider spatial partitioning: Use structures like octrees to accelerate raycasts by quickly
eliminating objects that can’t possibly intersect with the ray

• Optimize for mobile platforms: For performance-constrained devices, consider simplifying

the occlusion algorithm or reducing its precision

138
Implementing Orbit Controls

Many third-person games allow the player to orbit the camera around the character. Here’s how to
implement this functionality:

void ThirdPersonCamera::orbit(float horizontalAngle, float verticalAngle) {

// Update yaw and pitch based on input
yaw += horizontalAngle;
pitch += verticalAngle;

// Constrain pitch to avoid flipping

pitch = std::clamp(pitch, -89.0f, 89.0f);

// Calculate the new camera position based on spherical coordinates

float radius = followDistance;
float yawRad = glm::radians(yaw);
float pitchRad = glm::radians(pitch);

// Convert spherical coordinates to Cartesian

glm::vec3 offset;
offset.x = radius * cos(yawRad) * cos(pitchRad);
offset.y = radius * sin(pitchRad);
offset.z = radius * sin(yawRad) * cos(pitchRad);

// Set the desired position

desiredPosition = targetPosition + offset;

// Update camera vectors

front = glm::normalize(targetPosition - desiredPosition);
right = glm::normalize(glm::cross(front, worldUp));
up = glm::normalize(glm::cross(right, front));
}

This implementation:

1. Updates the camera’s yaw and pitch based on user input

2. Constrains the pitch to prevent the camera from flipping

3. Calculates a new camera position using spherical coordinates

4. Keeps the camera focused on the character

Integrating with Character Movement

To create a complete third-person camera system, we need to integrate it with character movement.
Here’s an example of how to use the third-person camera in a game loop:

void gameLoop(float deltaTime) {

// Update character position and orientation based on input
[Link](deltaTime);

139
 // Update camera position to follow the character
[Link](
[Link](),
[Link](),
deltaTime
);

// Handle camera occlusion

[Link](scene);

// Process camera orbit input (if any)

if (mouseInputDetected) {
[Link](mouseDeltaX, mouseDeltaY);
}

// Get the view and projection matrices for rendering

glm::mat4 viewMatrix = [Link]();
glm::mat4 projMatrix = [Link](aspectRatio);

// Use these matrices for rendering the scene

[Link](scene, viewMatrix, projMatrix);
}

For more advanced camera techniques, refer to the Advanced Camera Techniques
NOTE
section in the Appendix.

In the next section, we’ll integrate our camera system with Vulkan to render 3D scenes.

Previous: Transformation Matrices | Next: Vulkan Integration :pp: ++

Camera & Transformations: Vulkan

Integration
Integrating Camera with Vulkan
Libraries Used in This Tutorial
Before we dive into the integration, let’s briefly introduce the key libraries we’ll be using:

• GLFW (Graphics Library Framework): A lightweight, multi-platform library for creating

windows, contexts, and surfaces, handling input, and events. We use it for window
management and input handling. [[Link]

• GLM (OpenGL Mathematics): A mathematics library for graphics programming that provides
vector and matrix operations similar to GLSL. We use it for all our 3D math operations.

140
 [[Link]

Now that we have a camera system and understand transformation matrices, let’s integrate them
with our Vulkan application. We’ll focus on how to set up uniform buffers for our matrices and
update them each frame based on camera movement.

To keep the integration digestible, think of it in five small steps:

• Define the UBO layout (model/view/proj) and create per-frame buffers

• Create a descriptor set layout and allocate descriptor sets for the UBO

• Write descriptor sets and persistently map the buffers for fast updates

• Update the UBO each frame from the camera (view/proj) and model transform

• Bind the descriptor set and draw using the updated matrices

See Transformation matrices and Camera implementation for a refresher on matrix

NOTE
math.

Uniform Buffer Setup

First, we need to define our uniform buffer structure:

struct UniformBufferObject {
glm::mat4 model;
glm::mat4 view;
glm::mat4 proj;
};

Next, we’ll create the uniform buffer and its descriptor set:

Uniform buffers should be allocated per frame-in-flight (maxConcurrentFrames),

NOTE not per swapchain image. This matches how you submit work and synchronize
frames, avoids unnecessary allocations, and simplifies your logic.

// Use a fixed number of frames-in-flight, not the number of swapchain images

constexpr uint32_t maxConcurrentFrames = 2; // Adjust to your renderer

struct UniformBufferObject {
glm::mat4 model;
glm::mat4 view;
glm::mat4 proj;
};

// Keep the mapped pointer alongside the buffer for clarity and safety
struct UboBuffer {
vk::raii::Buffer buffer{nullptr};
vk::raii::DeviceMemory memory{nullptr};

141
 void* mapped = nullptr;
};

std::array<UboBuffer, maxConcurrentFrames> uniformBuffers;

void createUniformBuffers() {
vk::DeviceSize bufferSize = sizeof(UniformBufferObject);
// Create the buffer
vk::BufferCreateInfo bufferInfo{
.size = bufferSize,
.usage = vk::BufferUsageFlagBits::eUniformBuffer,
.sharingMode = vk::SharingMode::eExclusive
};
for (size_t i = 0; i < maxConcurrentFrames; i++) {
uniformBuffers[i].buffer = vk::raii::Buffer(device, bufferInfo);

// Allocate and bind memory

vk::MemoryRequirements memRequirements =
uniformBuffers[i].[Link]();

vk::MemoryAllocateInfo allocInfo{
.allocationSize = [Link],
.memoryTypeIndex = findMemoryType(
[Link],
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent
)
};

uniformBuffers[i].memory = vk::raii::DeviceMemory(device, allocInfo);

uniformBuffers[i].[Link](*uniformBuffers[i].memory, 0);

// Persistently map the buffer memory

uniformBuffers[i].mapped = uniformBuffers[i].[Link](0, bufferSize);
}
}

Descriptor Set Layout

We need to create a descriptor set layout that describes our uniform buffer:

void createDescriptorSetLayout() {
vk::DescriptorSetLayoutBinding uboLayoutBinding{
.binding = 0,
.descriptorType = vk::DescriptorType::eUniformBuffer,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eVertex,
.pImmutableSamplers = nullptr
};

142
 vk::DescriptorSetLayoutCreateInfo layoutInfo{
.bindingCount = 1,
.pBindings = &uboLayoutBinding
};

descriptorSetLayout = [Link](layoutInfo);
}

Descriptor Sets
Now we’ll create descriptor sets that point to our uniform buffers:

void createDescriptorSets() {
std::array<vk::DescriptorSetLayout, maxConcurrentFrames> layouts{};
[Link](*descriptorSetLayout);

vk::DescriptorSetAllocateInfo allocInfo{
.descriptorPool = *descriptorPool,
.descriptorSetCount = maxConcurrentFrames,
.pSetLayouts = [Link]()
};

descriptorSets = [Link](allocInfo);

vk::DescriptorBufferInfo bufferInfo{
.offset = 0,
.range = sizeof(UniformBufferObject)
};

for (size_t i = 0; i < maxConcurrentFrames; i++) {

[Link] = *uniformBuffers[i].buffer;

vk::WriteDescriptorSet descriptorWrite{
.dstSet = descriptorSets[i],
.dstBinding = 0,
.dstArrayElement = 0,
.descriptorCount = 1,
.descriptorType = vk::DescriptorType::eUniformBuffer,
.pBufferInfo = &bufferInfo
};

[Link](1, &descriptorWrite, 0, nullptr);

}
}

143
Updating Uniform Buffers
In our main loop, we’ll update the uniform buffer with the latest camera data:

void updateUniformBuffer(uint32_t currentFrame) {

static auto startTime = std::chrono::high_resolution_clock::now();
auto currentTime = std::chrono::high_resolution_clock::now();
float time = std::chrono::duration<float,
std::chrono::seconds::period>(currentTime - startTime).count();

UniformBufferObject ubo{};

// Model matrix: rotate the model around the Y axis

[Link] = glm::rotate(glm::mat4(1.0f), time * glm::radians(45.0f),
glm::vec3(0.0f, 1.0f, 0.0f));

// View matrix: get from our camera

[Link] = [Link]();

// Projection matrix: get from our camera

[Link] = [Link]([Link] /
(float)[Link]);

// Vulkan's Y coordinate is inverted compared to OpenGL

[Link][1][1] *= -1;

// Copy the data to the uniform buffer for the current frame-in-flight
memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));
}

Handling Input for Camera Movement

We need to handle user input to control the camera:

void processInput() {
// Calculate delta time
static float lastFrame = 0.0f;
float currentFrame = glfwGetTime();
float deltaTime = currentFrame - lastFrame;
lastFrame = currentFrame;

// Process keyboard input for camera movement

if (glfwGetKey(window, GLFW_KEY_W) == GLFW_PRESS)
[Link](CameraMovement::FORWARD, deltaTime);
if (glfwGetKey(window, GLFW_KEY_S) == GLFW_PRESS)
[Link](CameraMovement::BACKWARD, deltaTime);
if (glfwGetKey(window, GLFW_KEY_A) == GLFW_PRESS)
[Link](CameraMovement::LEFT, deltaTime);

144
 if (glfwGetKey(window, GLFW_KEY_D) == GLFW_PRESS)
[Link](CameraMovement::RIGHT, deltaTime);
if (glfwGetKey(window, GLFW_KEY_SPACE) == GLFW_PRESS)
[Link](CameraMovement::UP, deltaTime);
if (glfwGetKey(window, GLFW_KEY_LEFT_CONTROL) == GLFW_PRESS)
[Link](CameraMovement::DOWN, deltaTime);
}

Mouse Callback for Camera Rotation

We’ll also need to handle mouse movement for camera rotation:

// Global variables for mouse handling

float lastX = 0.0f, lastY = 0.0f;
bool firstMouse = true;

void mouseCallback(GLFWwindow* window, double xpos, double ypos) {

if (firstMouse) {
lastX = xpos;
lastY = ypos;
firstMouse = false;
}

float xoffset = xpos - lastX;

float yoffset = lastY - ypos; // Reversed: y ranges bottom to top

lastX = xpos;
lastY = ypos;

[Link](xoffset, yoffset);
}

void scrollCallback(GLFWwindow* window, double xoffset, double yoffset) {

[Link](yoffset);
}

Setting Up Input Callbacks

In our initialization code, we need to set up the input callbacks:

void initWindow() {
// ... existing GLFW initialization code ...

// Set up input callbacks

glfwSetCursorPosCallback(window, mouseCallback);
glfwSetScrollCallback(window, scrollCallback);

145
 // Capture the cursor for camera control
glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
}

Main Loop Integration

Finally, we integrate everything in our main loop:

void mainLoop() {
while (!glfwWindowShouldClose(window)) {
glfwPollEvents();
processInput();

// Update uniform buffer with latest camera data

updateUniformBuffer(currentFrame);

// Draw frame
drawFrame();
}
}

With these components in place, we now have a fully functional camera system integrated with our
Vulkan application. Users can navigate the 3D scene using keyboard and mouse controls, and the
view will update accordingly.

In the next section, we’ll wrap up with a conclusion and discuss potential improvements to our
camera system.

Previous: Camera Implementation | Next: Conclusion :pp: ++

Camera & Transformations:

Conclusion
Conclusion
In this chapter, we’ve built a comprehensive camera system for our Vulkan application. Let’s
summarize what we’ve learned and discuss potential improvements.

What We’ve Learned

• Mathematical Foundations: We explored the essential mathematical concepts for 3D graphics,
including vectors, matrices, quaternions, and coordinate systems.

• Camera Implementation: We designed a flexible camera class that supports different

146
 movement modes and handles user input for navigation.

• Transformation Matrices: We examined the model, view, and projection matrices that form
the MVP pipeline, and how they transform vertices through different coordinate spaces.

• Vulkan Integration: We integrated our camera system with Vulkan by setting up uniform
buffers, descriptor sets, and input handling.

With these components in place, we now have a solid foundation for creating interactive 3D
applications with Vulkan. Our camera system allows users to navigate and explore 3D scenes from
any perspective.

Potential Improvements
While our camera system is functional, there are several ways it could be enhanced:

• Camera Modes: Implement different camera modes (first-person, third-person, orbit) that can
be switched at runtime.

• Smooth Transitions: Add interpolation between camera positions and orientations for
smoother transitions.

• Collision Detection: Implement collision detection to prevent the camera from passing through
objects or walls.

• Camera Paths: Create a system for defining and following predefined camera paths for
cinematic sequences.

• Camera Effects: Add support for camera effects like depth of field, motion blur, or screen-space
reflections.

• Performance Optimization: Optimize the camera system for performance, especially for
mobile or VR applications.

Next Steps
As you continue building your Vulkan engine, consider how the camera system integrates with
other components:

• Scene Graph: How does the camera fit into your scene graph hierarchy?

• Rendering Pipeline: How can you optimize rendering based on the camera’s position and
orientation?

• User Interface: How will users interact with the camera in your application?

By addressing these questions, you can create a more cohesive and user-friendly 3D application.

Final Thoughts
A well-designed camera system is essential for any 3D application. It serves as the user’s window
into your virtual world and significantly impacts the user experience. By understanding the
mathematical foundations and implementing a flexible camera system, you’ve taken a major step

147
toward creating immersive 3D applications with Vulkan.

Remember that the code provided in this chapter is a starting point. Feel free to modify and extend
it to suit your specific needs and application requirements.

Previous: Vulkan Integration | Next: Lighting & Materials :pp: ++

Camera & Transformations

This chapter covers the implementation of a 3D camera system and the mathematical foundations
of 3D transformations in Vulkan.

Contents
• Introduction

• Mathematical Foundations

• Transformation Matrices

• Camera Implementation

• Vulkan Integration

• Conclusion = Introduction to Lighting & Materials

In this chapter, we’ll explore the fundamentals of lighting and materials in 3D rendering, with a
focus on Physically Based Rendering (PBR). Lighting is a crucial aspect of creating realistic and
visually appealing 3D scenes. Without proper lighting, even the most detailed models can appear
flat and lifeless.

[Rendering the Bistro scene at night with PBR pass]

About PBR References: Throughout this tutorial, you may encounter references to
PBR (Physically Based Rendering) before reaching this chapter. PBR is a modern
NOTE rendering approach that simulates how light interacts with surfaces based on
physical principles. We’ll cover PBR in detail in the sections that follow, so don’t
worry if you’re not familiar with these concepts yet.

This chapter serves as the foundation for understanding how light interacts with different
materials in a physically accurate way. The concepts you’ll learn here will be applied in later
chapters, including the Loading_Models chapter where we’ll use this knowledge to render glTF
models with PBR materials.

Throughout our engine implementation, we’ll be using vk::raii dynamic rendering and C++20
modules. The vk::raii namespace provides Resource Acquisition Is Initialization (RAII) wrappers for
Vulkan objects, which helps with resource management and makes the code cleaner. Dynamic
rendering simplifies the rendering process by eliminating the need for explicit render passes and
framebuffers. C++20 modules improve code organization, compilation times, and encapsulation
compared to traditional header files.

148
Why Lighting Matters
Lighting in computer graphics serves several important purposes:

1. Visual Realism: Proper lighting creates shadows, highlights, and gradients that make 3D objects
appear more realistic.

2. Spatial Understanding: Lighting helps viewers understand the spatial relationships between
objects in a scene.

3. Mood and Atmosphere: Different lighting setups can dramatically change the mood and
atmosphere of a scene.

4. Focus and Attention: Lighting can be used to draw attention to important elements in a scene.

Physically Based Rendering (PBR)

Introduction to PBR
Physically Based Rendering (PBR) represents one of the most significant advancements in real-time
graphics over the past decade. Unlike traditional rendering approaches that used ad-hoc shading
models, PBR aims to simulate how light interacts with surfaces in the real world based on the
principles of physics.

The Evolution of Real-Time Rendering

To appreciate PBR, it helps to understand how real-time rendering has evolved:

1. Fixed-Function Pipeline (1990s): Early 3D hardware used fixed lighting models like Gouraud
or Phong shading with limited material properties.

2. Programmable Shaders (2000s): With the introduction of shader programming, developers

could implement custom lighting models, but these were often inconsistent across different
lighting conditions.

3. Physically Based Rendering (2010s): By basing rendering on physical principles, PBR provides
more realistic results that remain consistent across different environments.

The key advantages of PBR include:

• Realism: Materials look correct under any lighting condition

• Consistency: Artists can create materials that work in all environments

• Intuitiveness: Material parameters have physical meaning, making them easier to understand

• Efficiency: Modern PBR implementations are optimized for real-time performance

Core Principles of PBR

PBR is built on several key principles that distinguish it from earlier rendering approaches:

149
Energy Conservation

In the real world, a surface cannot reflect more light than it receives. This principle of energy
conservation is fundamental to PBR:

• The sum of diffuse and specular reflection must not exceed 1.0

• As surfaces become more metallic, they have less diffuse reflection

• As surfaces become rougher, specular highlights become larger but less intense

Microfacet Theory

PBR uses microfacet theory to model surface roughness. This theory assumes that surfaces are
composed of tiny, perfectly reflective microfacets with varying orientations:

• Smooth surfaces have microfacets that are mostly aligned, creating sharp reflections

• Rough surfaces have randomly oriented microfacets, scattering light and creating blurry
reflections

• The distribution of these microfacets is controlled by the roughness parameter

Fresnel Effect

The Fresnel effect describes how reflectivity changes with viewing angle:

• All surfaces become more reflective at grazing angles (angles where the viewing direction is
nearly parallel to the surface)

• This effect is more noticeable on smooth surfaces

• The base reflectivity at normal incidence (F0, when light hits the surface perpendicularly), is
determined by the material’s index of refraction

• For metals, F0 is colored (based on the metal’s properties)

• For non-metals (dielectrics), F0 is typically around 0.04 (4%)

Metallic-Roughness Workflow

The PBR implementation in glTF and many modern engines uses the metallic-roughness workflow,
which defines materials using these primary parameters:

• Base Color: The albedo or diffuse color of the surface

• Metallic: How "metal-like" the surface is (0.0 = non-metal, 1.0 = metal)

• Roughness: How smooth or rough the surface is (0.0 = mirror-like, 1.0 = rough)

This workflow is intuitive for artists and efficient for real-time rendering.

The BRDF in PBR

The Bidirectional Reflectance Distribution Function (BRDF) is at the heart of PBR. It describes how

150
light is reflected from a surface, taking into account:

• The incoming light direction

• The outgoing view direction

• The surface normal

• The material properties

In PBR, the BRDF is typically split into two components:

• Diffuse BRDF: Handles light that penetrates the surface, scatters, and exits

• Specular BRDF: Handles light that reflects directly from the surface

Diffuse BRDF

The simplest diffuse BRDF is the Lambertian model:

f_diffuse = albedo / π

Where:

• albedo is the base color of the surface

• π is a normalization factor

More advanced models like Disney’s diffuse or Oren-Nayar can be used for increased realism,
especially for rough surfaces.

Specular BRDF

For the specular component, PBR typically uses a microfacet BRDF:

f_specular = D * F * G / (4 * (n·ωo) * (n·ωi))

Where:

• D is the Normal Distribution Function (NDF)

• F is the Fresnel term

• G is the Geometry term

• n is the surface normal

• ωo is the outgoing (view) direction

• ωi is the incoming (light) direction

Popular implementations include:

• D: GGX (Trowbridge-Reitz) distribution

151
 • F: Schlick’s approximation

• G: Smith shadowing-masking function

Materials in Computer Graphics

Materials define how surfaces interact with light. Different materials reflect, absorb, and transmit
light in different ways. Understanding materials is crucial for creating realistic renderings.

Material Properties
In computer graphics, materials are defined by various properties:

• Base Color/Albedo: The color of the surface under diffuse lighting

• Metalness: How metallic the surface is (affects specular reflection and diffuse absorption)

• Roughness/Smoothness: How rough or smooth the surface is (affects specular highlight size
and sharpness)

• Normal Map: Adds surface detail without increasing geometric complexity

• Ambient Occlusion: Approximates how much ambient light a surface point receives

• Emissive: Makes parts of the surface emit light

• Opacity/Transparency: Controls how transparent the material is

• Refraction: Controls how light bends when passing through the material

Common Material Types

Different types of materials have different characteristics:

• Metals: High specular reflection, colored specular, no diffuse reflection

• Dielectrics (Non-metals): Lower specular reflection, white specular, strong diffuse reflection

• Translucent Materials: Allow light to pass through and scatter within (e.g., skin, wax, marble)

• Transparent Materials: Allow light to pass through with minimal scattering (e.g., glass, water)

• Anisotropic Materials: Reflect light differently based on direction (e.g., brushed metal, hair)

Push Constants for Material Properties

In our implementation, we’ll use push constants to efficiently pass material properties to our
shaders.

Push constants are a way to send a small amount of data to shaders without having to create and
manage descriptor sets. They’re perfect for frequently changing data like material properties.

152
What You’ll Learn
By the end of this chapter, you’ll understand:

1. How Physically Based Rendering works

2. How to implement PBR in Slang shaders

3. How to use push constants for material properties

4. How to integrate PBR lighting with Vulkan

Let’s get started by exploring the principles of Physically Based Rendering in more detail.

Previous: Camera Transformations - Conclusion | Next: Lighting Models = Lighting Models

In this section, we’ll explore various lighting models used in computer graphics, with a focus on
understanding the concepts rather than implementation details. We’ll discuss how different
lighting models simulate the interaction of light with surfaces, their advantages and limitations,
and when to use each approach.

In this chapter, we’ll introduce Physically Based Rendering (PBR) and other lighting models. The
concepts we cover here will be applied in later chapters, such as the Loading_Models chapter
where we’ll use glTF, which uses PBR with the metallic-roughness workflow for its material system.
By understanding the theory behind different lighting models, including PBR, we can better
leverage the material properties provided by glTF models and extend our rendering capabilities.

Understanding Light-Surface Interaction

Before diving into specific lighting models, it’s important to understand how light interacts with
surfaces in the real world:

• Reflection: Light bounces off the surface

• Absorption: Light is absorbed by the surface and converted to heat

• Transmission: Light passes through the surface (for transparent materials)

• Scattering: Light is scattered in various directions within the material

The way these interactions occur depends on the material properties and the characteristics of the
light.

Types of Reflection
There are two main types of reflection:

• Diffuse Reflection: Light is scattered in many directions, creating a matte appearance

• Specular Reflection: Light is reflected in a specific direction, creating highlights

Most real-world materials exhibit a combination of diffuse and specular reflection.

153
The Evolution of Lighting Models
Lighting models in computer graphics have evolved significantly over time, each with their own
approach to simulating light-surface interactions:

Early Lighting Models

Flat Shading

The simplest lighting model, where each polygon is assigned a single color based on its normal and
the light direction. This creates a faceted appearance with visible polygon edges.

• Advantages: Very fast to compute

• Disadvantages: Unrealistic appearance, visible polygon edges

• When to use: For very low-power devices or stylized rendering

Gouraud Shading

An improvement over flat shading, where lighting is calculated once per vertex and then
interpolated across the polygon. This per-vertex approach is significantly faster than per-pixel
calculations, but means specular highlights that should appear in the middle of a polygon can be
missed entirely since they’re not present at any vertex.

• Advantages: Smoother appearance than flat shading, still relatively fast

• Disadvantages: Cannot accurately represent specular highlights due to vertex-level calculation

• When to use: For low-power devices where Phong shading is too expensive

Phong Lighting Model

One of the most widely used traditional lighting models, developed by Bui Tuong Phong in 1975.
When used with per-pixel shading (Phong Shading), normals are interpolated across the polygon
and lighting is calculated for every pixel, providing much more accurate specular highlights than
Gouraud’s per-vertex approach. The model calculates lighting using three components:

• Ambient: A constant light level to simulate indirect lighting

• Diffuse: Light scattered in all directions (using Lambert’s cosine law)

• Specular: Shiny highlights (using a power function of the reflection vector and view vector)

Characteristics:

• Advantages: Reasonably realistic for many materials, intuitive parameters, accurate specular
highlights with per-pixel shading

• Disadvantages: Not physically accurate, can look artificial under certain lighting conditions

• When to use: For simple real-time applications where PBR is too expensive

154
For more information on the Phong lighting model, see the Wikipedia article.

Blinn-Phong Model

A modification of the Phong model by Jim Blinn that uses the halfway vector between the light and
view directions instead of the reflection vector, making it more efficient to compute.

• Advantages: Faster than Phong, similar visual results

• Disadvantages: Still not physically accurate

• When to use: As a more efficient alternative to Phong

Learn more about Blinn-Phong in this Wikipedia article or this GPU Gems chapter.

Advanced Lighting Models

Cook-Torrance Model

A more physically-based model developed by Robert Cook and Kenneth Torrance in 1982. It uses
microfacet theory to model surface roughness and includes a more accurate specular term.

• Advantages: More physically accurate than Phong or Blinn-Phong

• Disadvantages: More complex to implement and compute

• When to use: When you need more realistic materials but full PBR is too expensive

For more details, see the original Cook-Torrance paper.

Oren-Nayar Model

An extension of the Lambertian diffuse model that accounts for microfacet roughness in diffuse
reflection, making it more suitable for rough surfaces like cloth, concrete, or sand.

• Advantages: More realistic diffuse reflection for rough surfaces

• Disadvantages: More expensive than Lambertian diffuse

• When to use: For materials where diffuse roughness is important

Learn more in the original Oren-Nayar paper.

Physically Based Rendering (PBR)

PBR represents one of the most significant advancements in real-time graphics over the past
decade. Unlike earlier ad-hoc shading models, PBR aims to simulate how light interacts with
surfaces based on the principles of physics.

The key principles of PBR include:

• Energy Conservation: A surface cannot reflect more light than it receives

• Microfacet Theory: Surfaces are modeled as collections of tiny mirrors with varying

155
 orientations

• Fresnel Effect: Reflectivity changes with viewing angle

• Metallic-Roughness Workflow: Materials are defined by their base color, metalness, and
roughness

Considerations for using PBR:

• Advantages: Realistic results that remain consistent across different lighting conditions,
intuitive parameters for artists

• Disadvantages: More complex and computationally expensive

• When to use: For modern games and applications where realism is important

For comprehensive information on PBR, see the Physically Based Rendering book.

[PBR materials with ray-traced shadows - demonstrating metallic surfaces] |

images/PBR_ray_shadows.png

Lighting Models in glTF

The glTF format uses PBR with the metallic-roughness workflow, which defines materials using
these primary parameters:

• Base Color: The albedo or diffuse color of the surface

• Metallic: How "metal-like" the surface is (0.0 = non-metal, 1.0 = metal)

• Roughness: How smooth or rough the surface is (0.0 = mirror-like, 1.0 = rough)

This workflow is intuitive for artists and efficient for real-time rendering. The glTF specification
provides a standardized way to define PBR materials that can be used across different rendering
engines.

For more information on the glTF PBR implementation, see the glTF 2.0 specification.

Light Types
Different lighting models can work with various types of light sources:

1. Point Lights: Light emanates in all directions from a single point.

2. Directional Lights: Light rays are parallel, as if coming from a very distant source (like the
sun).

3. Spot Lights: Light is emitted in a cone shape from a point.

4. Area Lights: Light is emitted from a surface area.

5. Image-Based Lighting (IBL): Light is derived from an environment map, simulating global
illumination.

Each type of light requires specific calculations for the light direction, attenuation, and other

156
properties.

Advanced Lighting Techniques

Beyond basic lighting models, there are several advanced techniques that can enhance the realism
of your rendering:

Global Illumination
Global Illumination (GI) simulates how light bounces between surfaces, creating indirect lighting
effects. Techniques include:

• Radiosity: Calculates diffuse light transfer between surfaces

• Path Tracing: Traces light paths through the scene

• Photon Mapping: Stores light information in a spatial data structure

For more information, see this GPU Gems chapter on radiosity.

Subsurface Scattering
Subsurface Scattering (SSS) simulates how light penetrates and scatters within translucent
materials like skin, wax, or marble.

For more information, see this GPU Gems chapter on subsurface scattering.

Ambient Occlusion
Ambient Occlusion (AO) approximates how much ambient light a surface point would receive,
darkening corners and crevices.

For more information, see this GPU Gems chapter on ambient occlusion.

Choosing the Right Lighting Model

When deciding which lighting model to use for your application, consider:

1. Hardware Constraints: More complex models require more processing power

2. Visual Requirements: How realistic do your materials need to look?

3. Artist Workflow: Some models are more intuitive for artists to work with

4. Consistency: PBR provides more consistent results across different lighting conditions

For our engine, we’ll leverage the PBR implementation from the glTF format, as it provides a good
balance of realism, performance, and artist-friendly parameters.

157
Further Reading
To deepen your understanding of lighting models, here are some valuable resources:

• Physically Based Rendering: From Theory to Implementation - The definitive book on PBR

• LearnOpenGL PBR Tutorial - An accessible introduction to PBR concepts

• Filament Material System - Google’s real-time PBR rendering engine documentation

• glTF 2.0 Material Specification - Details on how PBR is implemented in glTF

• GPU Gems: Materials - Collection of articles on advanced material rendering

In the next section, we’ll explore how to use push constants to efficiently pass material properties
to our shaders.

Previous: Introduction | Next: Push Constants = Push Constants

In this section, we’ll explore push constants, a powerful feature in Vulkan that allows us to
efficiently pass small amounts of data to shaders without the overhead of descriptor sets.

What Are Push Constants?

Push constants are a way to send a small amount of data directly to shaders. Unlike uniform
buffers, which require descriptor sets and memory allocation, push constants are part of the
command buffer itself. This makes them ideal for small, frequently changing data.

Some key characteristics of push constants: they are tiny (typically up to 128 bytes, device
dependent), fast to update per draw, and require no descriptor sets or allocations because they live
on the command buffer. They can be read by any shader stage you enable in the pipeline.

When to Use Push Constants

Use push constants for tiny, per‑draw parameters that change frequently—exactly the kind of
material knobs (base color, metallic, roughness) we tweak per object. If the data is larger than the
device’s push‑constant limit or doesn’t change often, prefer a uniform buffer instead.

Defining Push Constants in Shaders

In GLSL (or SPIR-V), push constants are defined using a uniform block with the push_constant layout
qualifier:

layout(push_constant) uniform PushConstants {

vec4 baseColorFactor;
float metallicFactor;
float roughnessFactor;
int baseColorTextureSet;
int physicalDescriptorTextureSet;

158
 int normalTextureSet;
int occlusionTextureSet;
int emissiveTextureSet;
float alphaMask;
float alphaMaskCutoff;
} material;

In Slang, which we’re using for our engine, the syntax is slightly different:

struct PushConstants {
float4 baseColorFactor;
float metallicFactor;
float roughnessFactor;
int baseColorTextureSet;
int physicalDescriptorTextureSet;
int normalTextureSet;
int occlusionTextureSet;
int emissiveTextureSet;
float alphaMask;
float alphaMaskCutoff;
};

[[vk::push_constant]] PushConstants material;

Setting Up Push Constants in Vulkan

To use push constants in Vulkan with vk::raii, we need to:

1. Define a push constant range when creating the pipeline layout.

2. Use [Link] to send data to the shader.

Here’s how we define a push constant range:

// Set up push constant range for material properties

vk::PushConstantRange pushConstantRange;
[Link](vk::ShaderStageFlagBits::eFragment) // Which shader
stages can access the push constants
.setOffset(0)
.setSize(sizeof(PushConstantBlock)); // Size of our push constant data

// Create pipeline layout with push constants

vk::PipelineLayoutCreateInfo pipelineLayoutInfo;
[Link](1)
.setPSetLayouts(&*descriptorSetLayout)
.setPushConstantRangeCount(1)
.setPPushConstantRanges(&pushConstantRange);

159
 // Create pipeline layout with vk::raii
vk::raii::PipelineLayout pipelineLayout =
[Link](pipelineLayoutInfo);

And here’s how we send data to the shader:

// Define material properties

PushConstantBlock pushConstants{};
[Link] = {1.0f, 1.0f, 1.0f, 1.0f};
[Link] = 1.0f;
[Link] = 0.5f;
[Link] = 0;
[Link] = 1;
[Link] = 2;
[Link] = 3;
[Link] = 4;
[Link] = 0.0f;
[Link] = 0.5f;

// Push constants to shader using vk::raii

[Link](
*pipelineLayout,
vk::ShaderStageFlagBits::eFragment, // Which shader stages will receive the data
0, // Offset
sizeof(PushConstantBlock), // Size
&pushConstants // Data
);

Push Constants vs. Uniform Buffers

While push constants are efficient for small, frequently changing data, they have limitations. For
larger data sets or data that doesn’t change frequently, uniform buffers are often a better choice.

Here’s a comparison:

Feature Push Constants Uniform Buffers

Size Limited (typically 128 bytes) Much larger

Update Mechanism Direct command in command Memory mapping or staging

buffer buffer

Descriptor Sets Not required Required

Memory Allocation Not required Required

Update Frequency Ideal for frequent updates Better for infrequent updates

Access Speed Fast Slightly slower

For our PBR implementation, we’ll use push constants for material properties and uniform buffers

160
for light information and transformation matrices.

In the next section, we’ll implement a basic lighting shader that uses push constants for material
properties.

Previous: Lighting Models | Next: Lighting Implementation = PBR Lighting Implementation

In this section, we’ll implement a Physically Based Rendering (PBR) shader based on the concepts
we’ve explored in the previous sections. This shader will use the metallic-roughness workflow
that’s compatible with glTF models and push constants for material properties. We’ll examine the
shader implementation and then discuss how to integrate it with our engine.

Implementing the PBR Shader

Let’s create a PBR shader, which we’ll name [Link]. This shader implements the metallic-
roughness workflow that we’ve discussed, making it compatible with glTF models. It uses push
constants for material properties and uniform buffers for transformation matrices and light
information.

We’ll break this shader into three distinct sections to better understand its architecture:

Section 1: Shader Setup Code - CPU-GPU

Communication
This section establishes the communication interface between the CPU application and GPU shader.
It defines the data structures and bindings that allow the CPU to pass information to the GPU
efficiently.

// Combined vertex and fragment shader for PBR rendering

// Input from vertex buffer - Data sent per vertex from CPU
struct VSInput {
float3 Position : POSITION; // 3D position in model space
float3 Normal : NORMAL; // Surface normal for lighting calculations
float2 UV : TEXCOORD0; // Texture coordinates for material sampling
float4 Tangent : TANGENT; // Tangent vector for normal mapping (w component
= handedness)
};

// Output from vertex shader / Input to fragment shader - Interpolated data

struct VSOutput {
float4 Position : SV_POSITION; // Required clip space position for rasterization
float3 WorldPos : POSITION; // World space position for lighting calculations
float3 Normal : NORMAL; // World space normal (interpolated)
float2 UV : TEXCOORD0; // Texture coordinates (interpolated)
float4 Tangent : TANGENT; // World space tangent (interpolated)
};

161
 // Uniform buffer - Global data shared across all vertices/fragments
struct UniformBufferObject {
float4x4 model; // Model-to-world transformation matrix
float4x4 view; // World-to-camera transformation matrix
float4x4 proj; // Camera-to-clip space projection matrix
float4 lightPositions[4]; // Light positions in world space
float4 lightColors[4]; // Light intensities and colors
float4 camPos; // Camera position for view-dependent effects
float exposure; // HDR exposure control
float gamma; // Gamma correction value (typically 2.2)
float prefilteredCubeMipLevels; // IBL prefiltered environment map mip levels
float scaleIBLAmbient; // IBL ambient contribution scale
};

// Push constants - Fast, small data updated frequently per material/object

struct PushConstants {
float4 baseColorFactor; // Base color tint/multiplier
float metallicFactor; // Metallic property multiplier
float roughnessFactor; // Surface roughness multiplier
int baseColorTextureSet; // Texture binding index for base color (-1 =
none)
int physicalDescriptorTextureSet; // Texture binding for metallic/roughness
int normalTextureSet; // Texture binding for normal maps
int occlusionTextureSet; // Texture binding for ambient occlusion
int emissiveTextureSet; // Texture binding for emissive maps
float alphaMask; // Alpha masking enable flag
float alphaMaskCutoff; // Alpha cutoff threshold
};

// Mathematical constants
static const float PI = 3.14159265359;

// Resource bindings - Connect CPU resources to GPU shader registers

[[vk::binding(0, 0)]] ConstantBuffer<UniformBufferObject> ubo;
[[vk::binding(1, 0)]] Texture2D baseColorMap;
[[vk::binding(1, 0)]] SamplerState baseColorSampler;
[[vk::binding(2, 0)]] Texture2D metallicRoughnessMap;
[[vk::binding(2, 0)]] SamplerState metallicRoughnessSampler;
[[vk::binding(3, 0)]] Texture2D normalMap;
[[vk::binding(3, 0)]] SamplerState normalSampler;
[[vk::binding(4, 0)]] Texture2D occlusionMap;
[[vk::binding(4, 0)]] SamplerState occlusionSampler;
[[vk::binding(5, 0)]] Texture2D emissiveMap;
[[vk::binding(5, 0)]] SamplerState emissiveSampler;

[[vk::push_constant]] PushConstants material;

Key Concepts Explained:

The vertex input layout defines how vertex data is structured in GPU memory, with semantic

162
annotations like POSITION and NORMAL telling the GPU how to interpret each data component.
This structured approach allows the graphics pipeline to efficiently process vertex attributes and
pass them through the rendering stages.

When it comes to data management, we use two primary mechanisms: uniform buffers and push
constants. Uniform buffers are larger, read-only memory blocks that efficiently store data shared
across many draw calls, making them perfect for transformation matrices and lighting information
that remain constant across multiple objects. Push constants, on the other hand, are smaller
(typically limited to 128 bytes or less) but much faster for frequently changing per-object data like
material properties, making them ideal for our material system.

The resource binding syntax using [[vk::binding(x, y)]] creates the essential link between CPU
resources and GPU shader registers. The first number represents the binding index, while the
second specifies the descriptor set, allowing us to organize and efficiently access textures, samplers,
and other resources from within our shaders.

Finally, the interpolation system works seamlessly in the background, where data in our VSOutput
structure gets automatically interpolated across triangle surfaces by the GPU’s rasterization
hardware, ensuring smooth transitions of attributes like normals and texture coordinates across
the rendered surface.

Section 2: Helper Functions - PBR Mathematics

This section contains the mathematical foundation of Physically Based Rendering. These functions
implement the Cook-Torrance microfacet BRDF model, which approximates how light interacts
with real-world materials at a microscopic level.

// Normal Distribution Function (D) - GGX/Trowbridge-Reitz Distribution

// Describes the statistical distribution of microfacet orientations
float DistributionGGX(float NdotH, float roughness) {
float a = roughness * roughness; // Remapping for more perceptual linearity
float a2 = a * a;
float NdotH2 = NdotH * NdotH;

float nom = a2; // Numerator: concentration factor

float denom = (NdotH2 * (a2 - 1.0) + 1.0);
denom = PI * denom * denom; // Normalization factor

return nom / denom; // Normalized distribution

}

// Geometry Function (G) - Smith's method with Schlick-GGX approximation

// Models self-shadowing and masking between microfacets
float GeometrySmith(float NdotV, float NdotL, float roughness) {
float r = roughness + 1.0;
float k = (r * r) / 8.0; // Direct lighting remapping

// Geometry obstruction from view direction (masking)

float ggx1 = NdotV / (NdotV * (1.0 - k) + k);

163
 // Geometry obstruction from light direction (shadowing)
float ggx2 = NdotL / (NdotL * (1.0 - k) + k);

return ggx1 * ggx2; // Combined masking-shadowing

}

// Fresnel Reflectance (F) - Schlick's approximation

// Models how reflectance changes with viewing angle
float3 FresnelSchlick(float cosTheta, float3 F0) {
return F0 + (1.0 - F0) * pow(1.0 - cosTheta, 5.0);
}

Mathematical Concepts & References:

The foundation of our PBR implementation rests on microfacet theory, which recognizes that real
surfaces consist of countless microscopic facets with varying orientations. Rather than trying to
model each individual facet, the BRDF statistically represents their collective behavior, allowing us
to achieve realistic lighting without the computational complexity of simulating every surface
detail. This approach was thoroughly explored in Walter et al.'s seminal 2007 paper "Microfacet
Models for Refraction through Rough Surfaces," which you can find at their comprehensive BSDF
documentation.

Our choice of the GGX distribution function, also known as Trowbridge-Reitz, stems from its ability
to produce realistic highlight shapes with longer tails compared to older models like Blinn-Phong.
This distribution function has become the standard in modern real-time rendering because it
closely matches measured material data and provides the natural falloff that we observe in real-
world materials. Eric Heitz’s 2014 work "Understanding the Masking-Shadowing Function in
Microfacet-Based BRDFs" provides deep insights into why this distribution works so well in
practice.

The Smith geometry function plays a crucial role by accounting for the statistical correlation
between masking (when the viewer can’t see a microfacet) and shadowing (when light can’t reach a
microfacet). This might seem like a technical detail, but it prevents energy gain at grazing angles
where naive models become unrealistically bright, ensuring our materials look believable under all
viewing conditions.

The Fresnel effect captures a phenomenon we see every day: materials become more reflective at
grazing angles, like water appearing mirror-like when viewed from the side. Schlick’s
approximation gives us this essential behavior while trading some accuracy for the performance
we need in real-time applications. The F0 parameter represents reflectance at normal incidence (0°
viewing angle), allowing us to control how reflective different materials appear when viewed head-
on.

Finally, energy conservation ensures that the sum of reflected and transmitted light never exceeds
the incident light, maintaining physical plausibility. This principle guides how we balance diffuse
and specular components, ensuring our materials look consistent and believable under varying
lighting conditions.

Further Reading:

164
For deeper exploration of these concepts, "Real-Time Rendering, 4th Edition" Chapter 9 on
Physically Based Shading provides comprehensive coverage of the theory and practice. The online
"PBR Book" by Pharr, Jakob, and Humphreys at [Link] offers an exhaustive
mathematical treatment of physically based rendering. For practical implementation insights, Epic
Games' "Real Shading in Unreal Engine 4" presentation from the 2013 Shading Course demonstrates
how these concepts translate into production-ready code.

Section 3: Vertex and Fragment Shader Main Bodies

This section contains the actual shader entry points that execute for each vertex and fragment
(pixel). The vertex shader transforms geometry, while the fragment shader implements the full PBR
lighting model.

// Vertex shader entry point - Executes once per vertex

[[shader("vertex")]]
VSOutput VSMain(VSInput input)
{
VSOutput output;

// Transform vertex position through the rendering pipeline

// Model -> World -> Camera -> Clip space transformation chain
float4 worldPos = mul([Link], float4([Link], 1.0));
[Link] = mul([Link], mul([Link], worldPos));

// Pass world position for fragment lighting calculations

// Fragment shader needs world space position to calculate light vectors
[Link] = [Link];

// Transform normal from model space to world space

// Use only rotation/scale part of model matrix (upper-left 3x3)
// Normalize to ensure unit length after transformation
[Link] = normalize(mul((float3x3)[Link], [Link]));

// Pass through texture coordinates unchanged

// UV coordinates are typically in [0,1] range and don't need transformation
[Link] = [Link];

// Pass tangent vector for normal mapping

// Will be used in fragment shader to construct tangent-space basis
[Link] = [Link];

return output;
}

// Fragment shader entry point - Executes once per pixel

[[shader("fragment")]]
float4 PSMain(VSOutput input) : SV_TARGET
{
// === MATERIAL PROPERTY SAMPLING ===

165
 // Sample base color texture and apply material color factor
float4 baseColor = [Link](baseColorSampler, [Link]) *
[Link];

// Sample metallic-roughness texture (metallic=B channel, roughness=G channel)

// glTF standard: metallic stored in blue, roughness in green
float2 metallicRoughness = [Link](metallicRoughnessSampler,
[Link]).bg;
float metallic = metallicRoughness.x * [Link];
float roughness = metallicRoughness.y * [Link];

// Sample ambient occlusion (typically stored in red channel)

float ao = [Link](occlusionSampler, [Link]).r;

// Sample emissive texture for self-illuminating materials

float3 emissive = [Link](emissiveSampler, [Link]).rgb;

// === NORMAL CALCULATION ===

// Start with interpolated surface normal
float3 N = normalize([Link]);

// Apply normal mapping if texture is available

if ([Link] >= 0) {
// Sample normal map and convert from [0,1] to [-1,1] range
float3 tangentNormal = [Link](normalSampler, [Link]).xyz * 2.0 -
1.0;

// Construct tangent-space to world-space transformation matrix (TBN)

float3 T = normalize([Link]); // Tangent
float3 B = normalize(cross(N, T)) * [Link].w; // Bitangent (w =
handedness)
float3x3 TBN = float3x3(T, B, N); // Tangent-Bitangent-
Normal matrix

// Transform normal from tangent space to world space

N = normalize(mul(tangentNormal, TBN));
}

// === LIGHTING SETUP ===

// Calculate view direction (fragment to camera)
float3 V = normalize([Link] - [Link]);

// Calculate reflection vector for environment mapping

float3 R = reflect(-V, N);

// === PBR MATERIAL SETUP ===

// Calculate F0 (reflectance at normal incidence)
// Non-metals: low reflectance (~0.04), Metals: colored reflectance from base
color
float3 F0 = float3(0.04, 0.04, 0.04); // Dielectric default
F0 = lerp(F0, [Link], metallic); // Lerp to metallic behavior

166
 // Initialize outgoing radiance accumulator
float3 Lo = float3(0.0, 0.0, 0.0);

// === DIRECT LIGHTING LOOP ===

// Calculate contribution from each light source
for (int i = 0; i < 4; i++) {
float3 lightPos = [Link][i].xyz;
float3 lightColor = [Link][i].rgb;

// Calculate light direction and attenuation

float3 L = normalize(lightPos - [Link]); // Light direction
float distance = length(lightPos - [Link]); // Distance for falloff
float attenuation = 1.0 / (distance * distance); // Inverse square falloff
float3 radiance = lightColor * attenuation; // Attenuated light
color

// Calculate half vector (between view and light directions)

float3 H = normalize(V + L);

// === BRDF EVALUATION ===

// Calculate all necessary dot products for BRDF terms
float NdotL = max(dot(N, L), 0.0); // Lambertian falloff
float NdotV = max(dot(N, V), 0.0); // View angle
float NdotH = max(dot(N, H), 0.0); // Half vector for specular
float HdotV = max(dot(H, V), 0.0); // For Fresnel calculation

// Evaluate Cook-Torrance BRDF components

float D = DistributionGGX(NdotH, roughness); // Normal distribution
float G = GeometrySmith(NdotV, NdotL, roughness); // Geometry function
float3 F = FresnelSchlick(HdotV, F0); // Fresnel reflectance

// Calculate specular BRDF

float3 numerator = D * G * F;
float denominator = 4.0 * NdotV * NdotL + 0.0001; // Prevent division by zero
float3 specular = numerator / denominator;

// === ENERGY CONSERVATION ===

// Fresnel term represents specular reflection ratio
float3 kS = F; // Specular contribution
float3 kD = float3(1.0, 1.0, 1.0) - kS; // Diffuse contribution (energy
conservation)
kD *= 1.0 - metallic; // Metals have no diffuse reflection

// === RADIANCE ACCUMULATION ===

// Combine diffuse (Lambertian) and specular (Cook-Torrance) terms
// Multiply by incident radiance and cosine foreshortening
Lo += (kD * [Link] / PI + specular) * radiance * NdotL;
}

// === AMBIENT AND EMISSIVE ===

167
 // Add simple ambient lighting (should be replaced with IBL in production)
float3 ambient = float3(0.03, 0.03, 0.03) * [Link] * ao;

// Combine all lighting contributions

float3 color = ambient + Lo + emissive;

// === HDR TONE MAPPING AND GAMMA CORRECTION ===

// Apply Reinhard tone mapping to compress HDR values to [0,1] range
color = color / (color + float3(1.0, 1.0, 1.0));

// Apply gamma correction for sRGB display (inverse gamma)

color = pow(color, float3(1.0 / [Link], 1.0 / [Link], 1.0 / [Link]));

// Output final color with original alpha

return float4(color, baseColor.a);
}

Vertex Shader Objectives:

The vertex shader serves as the first stage of our rendering pipeline, with its primary responsibility
being geometric transformation. It converts vertex positions through the standard MVP (Model-
View-Projection) matrix pipeline, systematically transforming coordinates from model space to
world space, then to camera space, and finally to clip space in preparation for rasterization. This
transformation chain ensures that our 3D geometry appears correctly positioned and projected for
the viewer.

Beyond basic transformation, the vertex shader handles crucial attribute processing by
transforming normals from model space to world space and passing through texture coordinates
and tangent vectors that the fragment shader will need. This attribute processing ensures that
lighting calculations in the fragment shader receive properly transformed surface information,
while texture coordinates and tangent vectors maintain their relationships for accurate material
sampling and normal mapping.

The vertex shader also performs essential data preparation by setting up interpolated values that
the fragment shader requires for lighting calculations. These interpolated values, such as world
positions and transformed normals, get automatically interpolated across triangle surfaces by the
GPU’s rasterization hardware, providing smooth transitions that enable realistic per-pixel lighting
in the subsequent fragment stage.

Fragment Shader Objectives:

The fragment shader represents the heart of our PBR implementation, beginning with
comprehensive material sampling that extracts surface properties like color, roughness, and
metallic values from texture maps. This sampling process reads multiple texture channels
according to the glTF standard, combining texture data with material parameters passed through
push constants to determine the final surface characteristics for each pixel.

Normal mapping reconstruction forms another critical objective, where the fragment shader takes
encoded normal information from normal maps and reconstructs detailed surface normals that
simulate fine geometric detail without requiring additional geometry. This process involves

168
sampling the normal map, transforming the values from texture space to world space using the
tangent-bitangent-normal matrix, and applying the resulting detailed normals to lighting
calculations.

The core PBR lighting implementation brings together all these elements using the Cook-Torrance
microfacet model with proper energy conservation. This involves evaluating the distribution,
geometry, and Fresnel terms of the BRDF, carefully balancing diffuse and specular contributions to
ensure physically plausible results across all viewing angles and material types.

Finally, post-processing operations convert the HDR linear lighting results into display-appropriate
sRGB values through tone mapping and gamma correction. This final stage compresses the high
dynamic range values generated by realistic lighting calculations into the limited range that
displays can show, while maintaining visual fidelity and preventing the harsh clipping that would
otherwise occur with bright highlights.

Key Implementation Details:

Our implementation carefully follows established conventions and best practices to ensure
compatibility and visual quality. We adhere to the glTF texture channel convention where metallic
information uses the blue channel and roughness uses the green channel, enabling seamless
integration with standard 3D authoring tools and asset pipelines. This convention ensures that
materials created in external tools will render correctly without requiring texture channel
remapping or custom import procedures.

Energy conservation remains paramount throughout our implementation, with careful attention
paid to ensuring that diffuse plus specular contributions never exceed unity through the kS/kD
relationship. This physical constraint prevents materials from appearing to emit more light than
they receive, maintaining believable appearance across different lighting conditions and viewing
angles while avoiding the artificial brightness that can plague non-physically-based approaches.

Numerical stability considerations appear throughout the implementation, with small epsilon
values added to prevent division by zero in BRDF calculations and careful handling of edge cases
where mathematical operations might produce undefined results. These seemingly minor details
prove crucial for robust rendering that handles extreme material parameters and unusual viewing
angles without producing artifacts or rendering failures.

The HDR pipeline architecture ensures that all lighting calculations occur in linear space,
preserving the full dynamic range of realistic lighting throughout the computation stages and only
applying gamma correction at the final output stage. This approach maintains maximum precision
and accuracy in the lighting calculations while ensuring that the final image appears correct on
standard sRGB displays.

This shader implements the PBR lighting model with the metallic-roughness workflow, but the goal
here is not just to show "what" the code does — it’s to explain "why" each piece exists.

Understanding the "Why" behind the shader

169
Why these BRDF terms (D, G, F)
The Normal Distribution Function (D) serves as the statistical heart of our microfacet model,
determining how many surface microfacets are oriented to reflect light directly toward the viewer.
This function explains why rough surfaces produce broader, dimmer highlights while smooth
surfaces create tight, bright reflections. We chose the GGX distribution because it matches
measured material data remarkably well and produces the natural long tails in highlights that we
observe in real-world materials, avoiding the artificial cutoff that plagued older distribution
functions like Blinn-Phong.

The Geometry function (G) addresses a crucial physical reality: microfacets cast shadows on each
other and can be hidden from view depending on the surface roughness and viewing angle.
Without proper geometric consideration, highlights become unrealistically bright as roughness
increases because we’d be ignoring the natural self-shadowing and masking that occurs on rough
surfaces. Smith’s approach with our roughness-derived k parameter provides an efficient yet
physically plausible solution that maintains energy conservation across all viewing conditions.

Fresnel reflectance (F) captures one of the most fundamental optical phenomena we encounter
daily: surfaces become more reflective at grazing angles, just as you can see your reflection clearly
in water when looking across its surface but hardly at all when looking straight down. Schlick’s
approximation gives us this essential angle-dependent behavior with minimal computational cost,
while the F0 parameter allows us to control how reflective materials appear when viewed head-on,
distinguishing between different material types.

Energy conservation ties these components together by ensuring that the sum of reflected light
never exceeds the incident light, maintaining physical plausibility. When more light reflects
specularly (kS), correspondingly less can reflect diffusely (kD = 1 - kS), creating the natural balance
that keeps materials looking believable across different lighting conditions and viewing angles
while preventing the artificial brightness that can make rendered scenes look unrealistic.

Why the metallic-roughness

The metallic-roughness workflow has become the industry standard primarily due to its adoption
by the glTF specification, which standardizes this approach with metallic information stored in the
blue channel and roughness in the green channel by convention. This standardization creates a
seamless ecosystem where assets created in any glTF-compliant tool will render consistently across
different engines and applications, eliminating the texture channel confusion that plagued earlier
workflows and enabling true asset interoperability.

From an artistic perspective, this workflow proves remarkably intuitive because it presents artists
with just two conceptual dials to control: metalness (distinguishing between non-metals and
metals) and roughness (controlling the surface finish from perfectly smooth to completely rough),
plus the base color. This simplification allows artists to focus on the visual intent rather than
getting lost in complex parameter interactions, while still providing the full range of material
appearances found in the real world.

The workflow also handles F0 behavior correctly by encoding the fundamental difference between
metallic and non-metallic materials. Non-metals typically have low F0 values around 0.02 to 0.08

170
(we use 0.04 as a reasonable default), while metals derive their colored specular reflectance directly
from the base color. Our lerp(F0, baseColor, metallic) operation elegantly encodes this physical
distinction, automatically transitioning from the achromatic reflectance of dielectrics to the colored
reflectance of conductors as the metallic parameter increases.

Why normal, occlusion, and emissive maps

Normal mapping represents one of the most powerful techniques in modern real-time rendering,
allowing us to add high-frequency surface detail without increasing geometric complexity. By
storing surface perturbations as RGB values in a texture, we can simulate fine details like scratches,
rivets, or fabric weaves that would be prohibitively expensive to model with actual geometry. The
magic happens in tangent space, where we reconstruct the perturbed normal vector N from the
tangent-bitangent-normal (TBN) matrix, ensuring that lighting calculations respond to these small-
scale surface features as if they were real geometric details. - Ambient occlusion (AO): Dampens
indirect light in crevices the global model doesn’t capture. We multiply the ambient/IBL term by AO
to avoid overly flat shading. - Emissive: Lets materials glow independent of lighting (e.g., LEDs,
screens) and contributes additively so it’s visible even in darkness.

Why HDR, exposure, and tone mapping

• Realistic light intensities create values far beyond [0,1] (e.g., sunlit surfaces, bright emitters). If
we write those directly to an 8-bit display, they clip at 1.0, crushing detail and producing ugly,
step-like highlights.

• Working in HDR (linear float) preserves detail through the lighting pipeline. Only at the end do
we compress dynamic range using a tone mapper to fit the display.

• In this chapter we use simple Reinhard: color / (color + 1). It’s robust and artifact-free, good as a
baseline. Alternatives you might adopt later:

◦ ACES (RRT/ODT): Filmic with good color preservation across extremes; widely used.

◦ Hable/Uncharted2 (“Filmic”): Nice highlight roll-off, tunable via curve parameters.

◦ Reinhard with exposure: Multiply color by an exposure before compressing to shift middle
gray.

• Exposure parameter ([Link]): Conceptually shifts scene brightness so midtones sit well
under your chosen tone mapper. Even if the snippet shows a fixed operator, you can pre-scale
color by exposure to support dynamic auto-exposure.

• Gamma correction ([Link]): Displays are non-linear (approx 2.2). Lighting must happen in
linear space, then we apply pow(color, 1/gamma) right before writing to the sRGB framebuffer.
Skipping this causes washed-out or too-dark images.

• Pipeline note: Prefer sRGB formats for color attachments when presenting. If writing to an sRGB
swapchain image, do gamma in shader OR use sRGB formats so hardware handles it — not both.
Do exactly one.

171
Practical tuning checklist
• If highlights look “plasticky” everywhere, roughness may be too low or kD not reduced by
metallic; verify kD *= (1 - metallic).

• If everything clips to white, add/adjust exposure and switch to ACES or Filmic tone mapping.

• If colors shift in highlights, check that tone mapping happens in linear space and gamma is
applied only once.

• If normal maps look inverted or seams appear, verify tangent handedness (TBN), normal map
channel order, and normal map space.

• If ambient looks flat, confirm AO is applied to ambient/IBL but not to direct specular.

Extending the Renderer

Now that we have our PBR shader, we need to extend our renderer to support it. We’ll need to:

1. Add a new pipeline for our PBR shader

2. Add support for push constants

3. Update the uniform buffer to include light information

Let’s start by adding a new function to create the PBR pipeline. This process involves several
distinct steps, each serving a specific purpose in configuring the Vulkan graphics pipeline for
physically based rendering.

Shader Module Creation and Stage Setup

First, we load our compiled shader and set up the programmable stages of the graphics pipeline.
Vulkan requires us to explicitly specify which shader stages we’ll use and their entry points.

bool Renderer::createPBRPipeline() {
try {
// Load our compiled PBR shader from disk
// The .spv file contains both vertex and fragment shader code compiled by
slangc
auto shaderCode = readFile("shaders/[Link]");

// Create a shader module - this is Vulkan's container for shader bytecode

// The shader module acts as a wrapper around the SPIR-V bytecode that GPU
drivers understand
vk::raii::ShaderModule shaderModule = createShaderModule(shaderCode);

// Configure the vertex shader stage

// This tells Vulkan which shader stage this module serves and its entry point
function
vk::PipelineShaderStageCreateInfo vertShaderStageInfo;
[Link](vk::ShaderStageFlagBits::eVertex)

172
 .setModule(*shaderModule)
.setPName("VSMain"); // Must match the vertex shader
function name

// Configure the fragment shader stage

// Same module, different entry point - this is how combined shaders work
vk::PipelineShaderStageCreateInfo fragShaderStageInfo;
[Link](vk::ShaderStageFlagBits::eFragment)
.setModule(*shaderModule)
.setPName("PSMain"); // Must match the fragment shader
function name

std::array<vk::PipelineShaderStageCreateInfo, 2> shaderStages =

{vertShaderStageInfo, fragShaderStageInfo};

The entry point names ("VSMain" and "PSMain") must exactly match the function names in our
shader code. This explicit binding system gives us fine-grained control over which functions serve
which pipeline stages, and it’s particularly useful when working with shader libraries that contain
multiple variations of vertex or fragment shaders.

Vertex Input Configuration

The vertex input state defines how vertex data flows from our vertex buffers into the vertex
shader. This configuration must precisely match the vertex format expected by our PBR shader.

// Configure how vertex data is structured and fed to the vertex shader
vk::PipelineVertexInputStateCreateInfo vertexInputInfo;

// Define the vertex buffer binding - describes the overall vertex structure
// This tells Vulkan the total size of each vertex and how vertices are
arranged
vk::VertexInputBindingDescription bindingDescription;
[Link](0) // Binding point 0
.setStride(sizeof(float) * 14) // Total vertex size:
pos(3) + normal(3) + uv(2) + tangent(4) + bitangent(2)
.setInputRate(vk::VertexInputRate::eVertex); // Data advances
per vertex (not per instance)

// Define individual vertex attributes - each corresponds to an input in our

vertex shader
std::array<vk::VertexInputAttributeDescription, 5> attributeDescriptions;

// Position attribute: 3D coordinates in model space

attributeDescriptions[0].setBinding(0) // From binding 0
.setLocation(0) // Shader input
location 0
.setFormat(vk::Format::eR32G32B32Sfloat) // Three 32-
bit floats (RGB)
.setOffset(0); // Start of vertex

173
 data

// Normal attribute: surface normal for lighting calculations

attributeDescriptions[1].setBinding(0)
.setLocation(1) // Shader input
location 1
.setFormat(vk::Format::eR32G32B32Sfloat)
.setOffset(sizeof(float) * 3); // After position

// Texture coordinate attribute: UV mapping coordinates

attributeDescriptions[2].setBinding(0)
.setLocation(2) // Shader input
location 2
.setFormat(vk::Format::eR32G32Sfloat) // Two 32-bit
floats (RG)
.setOffset(sizeof(float) * 6); // After position +
normal

// Tangent attribute: tangent vector for normal mapping (includes handedness

in W)
attributeDescriptions[3].setBinding(0)
.setLocation(3) // Shader input
location 3
.setFormat(vk::Format::eR32G32B32A32Sfloat) // Four
32-bit floats (RGBA)
.setOffset(sizeof(float) * 8); // After position +
normal + UV

// Bitangent attribute: completes the tangent space basis

attributeDescriptions[4].setBinding(0)
.setLocation(4) // Shader input
location 4
.setFormat(vk::Format::eR32G32Sfloat)
.setOffset(sizeof(float) * 12); // After all previous
attributes

// Connect the binding and attribute descriptions to the vertex input state
[Link](1)
.setPVertexBindingDescriptions(&bindingDescription)

.setVertexAttributeDescriptionCount(static_cast<uint32_t>([Link]()
))
.setPVertexAttributeDescriptions([Link]());

The vertex input configuration serves as a contract between our vertex buffer data and the vertex
shader inputs. Each attribute description maps a specific piece of vertex data to a shader input
location, with precise format and offset specifications. This explicit mapping system ensures that
the GPU correctly interprets our vertex data regardless of how it’s packed in memory.

The stride calculation (14 floats) reflects our comprehensive vertex format that supports full PBR

174
rendering: position for geometry, normals for basic lighting, UV coordinates for texture sampling,
and tangent vectors for normal mapping. The tangent vector includes a fourth component (W) that
stores handedness information, which is crucial for correctly reconstructing the bitangent vector in
cases where the tangent space might be flipped.

The offset calculations ensure that each attribute starts at the correct byte position within each
vertex. This precise alignment is for performance, as misaligned vertex data can cause significant
performance penalties on some GPU architectures.

Input Assembly and Primitive Processing

The input assembly stage determines how vertices are grouped into geometric primitives and how
the GPU should interpret the vertex stream.

// Configure input assembly - how vertices become triangles

vk::PipelineInputAssemblyStateCreateInfo inputAssembly;
[Link](vk::PrimitiveTopology::eTriangleList) // Every 3
vertices form a triangle
.setPrimitiveRestartEnable(false); // Don't use
primitive restart indices

Triangle lists represent the most straightforward and commonly used primitive topology for
complex 3D models. In this mode, every group of three consecutive vertices defines a complete
triangle, providing maximum flexibility for representing arbitrary geometry. While other
topologies like triangle strips or fans can be more memory-efficient for certain geometric patterns,
triangle lists avoid the complexity of degenerate triangles and vertex ordering constraints that can
arise with more compact representations.

Primitive restart functionality allows special index values to signal the end of one primitive and the
beginning of another, but this feature adds complexity that’s unnecessary for most PBR rendering
scenarios. By disabling it, we ensure predictable behavior and avoid potential performance
penalties associated with index buffer scanning.

Viewport and Dynamic State Configuration

The viewport state manages the transformation from normalized device coordinates to screen
coordinates, while dynamic state configuration allows certain pipeline parameters to be changed
without recreating the entire pipeline.

// Configure viewport and scissor state

// We'll set actual viewport and scissor rectangles dynamically at render time
vk::PipelineViewportStateCreateInfo viewportState;
[Link](1) // Single viewport (most common case)
.setScissorCount(1); // Single scissor rectangle

// Define which pipeline state can be changed dynamically

// This improves performance by avoiding pipeline recreation for common

175
 changes
std::vector<vk::DynamicState> dynamicStates = {
vk::DynamicState::eViewport, // Viewport can change (window resize,
camera changes)
vk::DynamicState::eScissor // Scissor rectangle can change (UI
clipping, effects)
};

vk::PipelineDynamicStateCreateInfo dynamicState;
[Link](static_cast<uint32_t>([Link]()))
.setPDynamicStates([Link]());

Dynamic state configuration represents a key optimization in modern Vulkan applications. By

marking viewport and scissor as dynamic, we avoid the expensive pipeline recreation that would
otherwise be required for common operations like window resizing or camera adjustments. The
GPU driver can efficiently update these parameters at command recording time rather than
requiring a completely new pipeline state object.

The single viewport approach covers the vast majority of rendering scenarios. Multi-viewport
rendering is primarily used for specialized applications like VR stereo rendering or certain shadow
mapping techniques, but single-viewport rendering provides optimal performance for standard
PBR applications.

Rasterization Configuration
The rasterization stage converts geometric primitives into fragments (potential pixels) and applies
various geometric processing options that affect how triangles are converted to pixels.

// Configure rasterization - how triangles become pixels

vk::PipelineRasterizationStateCreateInfo rasterizer;
[Link](false) // Don't clamp
depth values (standard behavior)
.setRasterizerDiscardEnable(false) // Don't
discard primitives before rasterization
.setPolygonMode(vk::PolygonMode::eFill) // Fill
triangles (not wireframe or points)
.setLineWidth(1.0f) // Line width
(only relevant for wireframe)
.setCullMode(vk::CullModeFlagBits::eBack) // Cull back-
facing triangles
.setFrontFace(vk::FrontFace::eCounterClockwise) // Counter-
clockwise vertices = front-facing
.setDepthBiasEnable(false); // No depth
bias (used for shadow mapping)

The rasterization configuration directly impacts both rendering performance and visual quality.
Back-face culling provides a significant performance boost by eliminating triangles that face away
from the camera, effectively halving the fragment processing workload for typical closed meshes.

176
The counter-clockwise winding order follows the standard convention used by most 3D modeling
tools and asset pipelines.

Fill mode produces solid triangles appropriate for PBR rendering, though wireframe mode can be
useful for debugging geometry or creating special visual effects. The line width setting only affects
wireframe rendering, but some graphics drivers require it to be specified even when using fill
mode.

Depth bias (also known as polygon offset) is commonly used in shadow mapping to prevent self-
shadowing artifacts, but it’s unnecessary for standard forward rendering and can introduce its own
artifacts if used inappropriately.

Multisampling and Anti-Aliasing

The multisampling configuration determines how the GPU handles anti-aliasing to reduce visual
artifacts from geometric edges.

// Configure multisampling - anti-aliasing settings

vk::PipelineMultisampleStateCreateInfo multisampling;
[Link](false) // Disable
per-sample shading
.setRasterizationSamples(vk::SampleCountFlagBits::e1); // No
multisampling (1 sample per pixel)

This configuration disables multisampling anti-aliasing (MSAA) for simplicity and performance.
While MSAA can significantly improve visual quality by reducing aliasing artifacts on geometric
edges, it also substantially increases memory bandwidth requirements and fragment processing
costs. For learning purposes and initial implementations, single-sample rendering provides a good
balance between performance and complexity.

In production applications, you might enable MSAA by increasing the sample count to 4x or 8x,
depending on performance requirements and target hardware capabilities. Per-sample shading,
when enabled, runs the fragment shader once per sample rather than once per pixel, providing the
highest quality anti-aliasing at the cost of proportionally increased fragment processing time.

Phase 7: Depth Testing and Z-Buffer Configuration

The depth and stencil state configuration controls how fragments interact with the depth buffer to
achieve proper depth sorting and occlusion.

// Configure depth and stencil testing

vk::PipelineDepthStencilStateCreateInfo depthStencil;
[Link](true) // Enable
depth testing for proper occlusion
.setDepthWriteEnable(true) // Write depth
values to depth buffer
.setDepthCompareOp(vk::CompareOp::eLess) // Fragment

177
 passes if its depth is less (closer)
.setDepthBoundsTestEnable(false) // Don't use
depth bounds testing
.setStencilTestEnable(false); // Don't use
stencil testing

Depth testing forms the foundation of proper 3D rendering by ensuring that closer objects occlude
more distant ones. The "less than" comparison function works with the standard depth buffer
convention where smaller depth values represent closer fragments. This configuration writes depth
values for each rendered fragment, building up the depth buffer that subsequent draw calls can use
for occlusion testing.

Depth bounds testing and stencil testing are advanced features used for specific rendering
techniques like light volume optimization or complex compositing operations. For standard PBR
rendering, they add unnecessary complexity without providing benefits, so we disable them to
maintain optimal performance.

Phase 8: Color Blending and Transparency

The color blend state determines how new fragments combine with existing color values in the
framebuffer, enabling transparency and various compositing effects.

// Configure color blending - how new pixels combine with existing ones
vk::PipelineColorBlendAttachmentState colorBlendAttachment;
[Link](
vk::ColorComponentFlagBits::eR | vk::ColorComponentFlagBits::eG | //
Write all color channels
vk::ColorComponentFlagBits::eB | vk::ColorComponentFlagBits::eA)
.setBlendEnable(true) //
Enable alpha blending
.setSrcColorBlendFactor(vk::BlendFactor::eSrcAlpha) //
New fragment's alpha
.setDstColorBlendFactor(vk::BlendFactor::eOneMinusSrcAlpha)
// One minus new fragment's alpha
.setColorBlendOp(vk::BlendOp::eAdd) //
Add source and destination
.setSrcAlphaBlendFactor(vk::BlendFactor::eOne) //
Preserve new alpha
.setDstAlphaBlendFactor(vk::BlendFactor::eZero) //
Ignore old alpha
.setAlphaBlendOp(vk::BlendOp::eAdd); //
Add alpha values

vk::PipelineColorBlendStateCreateInfo colorBlending;
[Link](false) //
Don't use logical operations
.setAttachmentCount(1) //
Single color attachment

178
 .setPAttachments(&colorBlendAttachment);

This blend configuration implements standard alpha transparency using the classic "over"
compositing operation. The formula (srcAlpha * newColor) + ((1 - srcAlpha) * oldColor)
produces natural-looking transparency effects where fully opaque fragments (alpha = 1) completely
replace the background, while partially transparent fragments blend proportionally.

The separate alpha blending configuration preserves the alpha channel properly for potential
multi-pass rendering or post-processing effects. By setting source alpha factor to one and
destination alpha factor to zero, we ensure that the final alpha value comes entirely from the new
fragment, which is typically the desired behavior for transparency effects.

Phase 9: Pipeline Layout and Resource Binding

The pipeline layout defines how resources like textures, uniform buffers, and push constants are
organized and accessed by the shaders.

// Configure push constants for fast material property updates

vk::PushConstantRange pushConstantRange;
[Link](vk::ShaderStageFlagBits::eFragment) //
Only fragment shader uses these
.setOffset(0) //
Start at beginning
.setSize(sizeof(PushConstantBlock)); //
Size of our material data

// Create the pipeline layout - defines resource organization

vk::PipelineLayoutCreateInfo pipelineLayoutInfo;
[Link](1) //
Single descriptor set
.setPSetLayouts(&*descriptorSetLayout) //
Our texture/uniform bindings
.setPushConstantRangeCount(1) //
One push constant block
.setPPushConstantRanges(&pushConstantRange);

// Create the pipeline layout object

pbrPipelineLayout = [Link](pipelineLayoutInfo);

The pipeline layout serves as a contract between the application and shaders regarding resource
organization. Push constants provide the fastest path for updating small amounts of data (like
material properties) between draw calls, as they bypass the memory hierarchy and are directly
accessible to shader cores. The 128-byte limit on push constants in most implementations makes
them perfect for per-material data but unsuitable for larger datasets.

The descriptor set layout reference connects our pipeline to the texture and uniform buffer
bindings we established earlier. This separation of concerns allows the same descriptor set layout
to be used across multiple pipelines while maintaining clean resource organization.

179
Phase 10: Final Pipeline Creation and Dynamic
Rendering Setup
The final phase assembles all configuration states into a complete graphics pipeline and sets up
dynamic rendering compatibility for modern Vulkan applications.

// Assemble the complete graphics pipeline

vk::GraphicsPipelineCreateInfo pipelineInfo;
[Link](static_cast<uint32_t>([Link]())) //
Number of shader stages
.setPStages([Link]()) //
Shader stage configurations
.setPVertexInputState(&vertexInputInfo) //
Vertex format
.setPInputAssemblyState(&inputAssembly) //
Primitive topology
.setPViewportState(&viewportState) //
Viewport configuration
.setPRasterizationState(&rasterizer) //
Rasterization settings
.setPMultisampleState(&multisampling) //
Anti-aliasing settings
.setPDepthStencilState(&depthStencil) //
Depth/stencil testing
.setPColorBlendState(&colorBlending) //
Blending configuration
.setPDynamicState(&dynamicState) //
Dynamic state settings
.setLayout(*pbrPipelineLayout) //
Resource layout
.setRenderPass(nullptr) //
Using dynamic rendering
.setSubpass(0) //
Subpass index
.setBasePipelineHandle(nullptr); //
No base pipeline

// Configure for dynamic rendering (modern Vulkan approach)

vk::PipelineRenderingCreateInfo renderingInfo;
[Link](1) //
Single color target
.setPColorAttachmentFormats(&swapChainImageFormat) //
Match swapchain format
.setDepthAttachmentFormat(findDepthFormat()); //
Depth buffer format
[Link](&renderingInfo);

// Create the final graphics pipeline

pbrPipeline = [Link](nullptr, pipelineInfo);

180
 return true;
} catch (const std::exception& e) {
std::cerr << "Error creating PBR pipeline: " << [Link]() << std::endl;
return false;
}
}

The pipeline creation represents the culmination of all our configuration work, where Vulkan
validates the entire pipeline specification and compiles it into an optimized form suitable for GPU
execution. The dynamic rendering configuration replaces the traditional render pass system with a
more flexible approach that allows render targets to be specified at command recording time
rather than pipeline creation time.

This flexibility proves particularly valuable for applications that need to render to different targets
(like shadow maps, reflection textures, or post-processing buffers) using the same pipeline. The
format specifications ensure that the pipeline generates output compatible with our target render
surfaces.

The exception handling provides essential feedback during development, as pipeline creation
failures can result from subtle configuration mismatches or resource compatibility issues that are
difficult to debug without proper error reporting.

This function creates a new pipeline for our PBR shader, including support for push constants. We’ll
also need to update our uniform buffer to include light information:

// Update uniform buffer

void Renderer::updateUniformBuffer(uint32_t currentFrame, Entity* entity,
CameraComponent* camera) {
// Get the transform component from the entity
auto transform = entity->GetComponent<TransformComponent>();
if (!transform) {
std::cerr << "Entity does not have a transform component" << std::endl;
return;
}

// Create the uniform buffer object

UniformBufferObject ubo{};

// Set the model matrix from the entity's transform

[Link] = transform->GetModelMatrix();

// Set the view and projection matrices from the camera

if (camera) {
[Link] = camera->GetViewMatrix();
[Link] = camera->GetProjectionMatrix();
} else {
// Default view and projection matrices if no camera is provided
[Link] = glm::lookAt(glm::vec3(2.0f, 2.0f, 2.0f), glm::vec3(0.0f, 0.0f,

181
 0.0f), glm::vec3(0.0f, 0.0f, 1.0f));
[Link] = glm::perspective(glm::radians(45.0f), [Link] /
(float)[Link], 0.1f, 100.0f);
[Link][1][1] *= -1; // Flip Y coordinate for Vulkan
}

// Set up lights
// Light 1: White light from above
[Link][0] = glm::vec4(0.0f, 5.0f, 5.0f, 1.0f);
[Link][0] = glm::vec4(300.0f, 300.0f, 300.0f, 1.0f);

// Light 2: Blue light from the left

[Link][1] = glm::vec4(-5.0f, 0.0f, 0.0f, 1.0f);
[Link][1] = glm::vec4(0.0f, 0.0f, 300.0f, 1.0f);

// Light 3: Red light from the right

[Link][2] = glm::vec4(5.0f, 0.0f, 0.0f, 1.0f);
[Link][2] = glm::vec4(300.0f, 0.0f, 0.0f, 1.0f);

// Light 4: Green light from behind

[Link][3] = glm::vec4(0.0f, -5.0f, 0.0f, 1.0f);
[Link][3] = glm::vec4(0.0f, 300.0f, 0.0f, 1.0f);

// Set camera position for view-dependent effects

[Link] = glm::vec4(camera ? camera->GetPosition() : glm::vec3(2.0f, 2.0f,
2.0f), 1.0f);

// Set PBR parameters

[Link] = 4.5f;
[Link] = 2.2f;
[Link] = 1.0f;
[Link] = 1.0f;

// Copy the uniform buffer object to the device memory using vk::raii
// With vk::raii, we can use the mapped memory directly
memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));
}

Finally, we need to add support for pushing material properties to the shader:

// Push material properties to shader

void Renderer::pushMaterialProperties(vk::CommandBuffer commandBuffer, const Model*
model, uint32_t materialIndex) {
// Get material from the model
const Material& material = model->materials[materialIndex];

// Define push constants

PushConstantBlock pushConstants{};
[Link] = [Link];
[Link] = [Link];

182
 [Link] = [Link];
[Link] = [Link];
[Link] =
[Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link] == AlphaMode::MASK ? 1.0f : 0.0f;
[Link] = [Link];

// Push constants to shader using vk::raii

[Link](
*pbrPipelineLayout,
vk::ShaderStageFlagBits::eFragment,
0,
sizeof(PushConstantBlock),
&pushConstants
);
}

In the next section, we’ll integrate our lighting implementation with the rest of the Vulkan
rendering pipeline.

Previous: Push Constants | Next: Vulkan Integration = Vulkan Integration

In this section, we’ll integrate our PBR implementation with the rest of the Vulkan rendering
pipeline. We’ll update our renderer class to support advanced lighting techniques that can be used
with glTF models and their PBR materials. The techniques we develop here will be applied in the
Loading_Models chapter when we load and render glTF models.

To keep the flow concrete and avoid repeating earlier theory, use this quick roadmap:

1) Extend the renderer with PBR pipeline objects and a material push-constant block 2) Create the
PBR pipeline (layout, shaders, blending, formats) alongside the main pipeline 3) Record draws: bind
PBR pipeline, bind geometry, and push per-material constants per mesh 4) Clean up via RAII (no
special teardown required)

We won’t re-explain PBR theory or push-constant fundamentals here. See push

NOTE
constants, and Introduction (and PBR Rendering) for PBR concepts.

The PBR pass slots into the graphics pipeline as shown below:

[Rendering pipeline flowchart showing where the PBR pass fits] |

../../../images/rendering_pipeline_flowchart.png

Updating the Renderer Class

First, let’s update our renderer class to include the new members we need for our PBR
implementation:

183
 class Renderer {
public:
// ... existing members ...

// PBR pipeline
vk::raii::PipelineLayout pbrPipelineLayout;
vk::raii::Pipeline pbrPipeline;

// Push constant block for PBR material properties

struct PushConstantBlock {
glm::vec4 baseColorFactor;
float metallicFactor;
float roughnessFactor;
int baseColorTextureSet;
int physicalDescriptorTextureSet;
int normalTextureSet;
int occlusionTextureSet;
int emissiveTextureSet;
float alphaMask;
float alphaMaskCutoff;
};

// ... existing methods ...

// New methods
bool createPBRPipeline();
void pushMaterialProperties(vk::CommandBuffer commandBuffer, const Model* model,
uint32_t materialIndex);
};

We’ve added members for the PBR pipeline and a struct for PBR material properties. We’ve also
added methods for creating the PBR pipeline and pushing material properties to the shader.

Updating the Initialization

Next, we need to update the initialization process to create our PBR pipeline:

bool Renderer::Initialize(const std::string& appName, bool enableValidationLayers) {

// ... existing initialization code ...

// Create graphics pipeline

if (!createGraphicsPipeline()) {
return false;
}

// Create PBR pipeline

if (!createPBRPipeline()) {
std::cerr << "Failed to create PBR pipeline" << std::endl;

184
 return false;
}

// ... rest of initialization code ...

initialized = true;
return true;
}

Updating the Cleanup

We also need to update the cleanup process to destroy our PBR pipeline:

void Renderer::Cleanup() {
// ... existing cleanup code ...

// With vk::raii, pipeline and pipeline layout objects are automatically destroyed
// when they go out of scope, so we don't need explicit destruction calls

// ... rest of cleanup code ...

}

Updating the Rendering Process

Finally, we need to update the rendering process to use our PBR pipeline and push material
properties:

void Renderer::recordCommandBuffer(vk::CommandBuffer commandBuffer, uint32_t

imageIndex) {
// ... existing command buffer recording code ...

// Bind the PBR pipeline

[Link](vk::PipelineBindPoint::eGraphics, *pbrPipeline);

// For each model in the scene

for (const auto& model : models) {
// Bind vertex and index buffers
vk::Buffer vertexBuffers[] = {model->vertexBuffer};
vk::DeviceSize offsets[] = {0};
[Link](0, 1, vertexBuffers, offsets);
[Link](model->indexBuffer, 0, vk::IndexType::eUint32);

// For each mesh in the model

for (const auto& mesh : model->meshes) {
// Push material properties
pushMaterialProperties(commandBuffer, model, [Link]);

185
 // Bind descriptor sets
[Link](
vk::PipelineBindPoint::eGraphics,
*pbrPipelineLayout,
0,
1,
&descriptorSets[imageIndex],
0,
nullptr
);

// Draw
[Link]([Link], 1, [Link], 0, 0);
}
}

// ... rest of command buffer recording code ...

}

PBR Shader Reference

This chapter reuses the exact PBR shader defined in the previous section to avoid duplication and
drift. Please refer to Implementing the PBR Shader for the full [Link] source and detailed
explanations. Here we focus strictly on Vulkan integration: pipeline layout, descriptor bindings,
push constants, and draw submission.

Compiling the Shader

After creating the shader file, we need to compile it using slangc. This is typically done as part of
the build process, but we can also do it manually:

slangc shaders/[Link] -target spirv -profile spirv_1_4 -o shaders/[Link]

Testing the Implementation with glTF

Models
To test our implementation, we can use glTF models, which already have PBR materials defined
that are compatible with our implementation. In the Loading_Models chapter, we’ll learn how to
load these models, but for now, let’s assume we have a way to load them.

Here’s an example of how to set up a test scene with glTF models:

void Renderer::renderTestScene() {

186
 // Set up camera
glm::vec3 cameraPos = glm::vec3(0.0f, 0.0f, 3.0f);
glm::vec3 cameraTarget = glm::vec3(0.0f, 0.0f, 0.0f);
glm::vec3 cameraUp = glm::vec3(0.0f, 1.0f, 0.0f);

// Set up lights
// Light 1: White light from above
glm::vec4 lightPos1 = glm::vec4(0.0f, 5.0f, 5.0f, 1.0f);
glm::vec4 lightColor1 = glm::vec4(300.0f, 300.0f, 300.0f, 1.0f);

// Light 2: Blue light from the left

glm::vec4 lightPos2 = glm::vec4(-5.0f, 0.0f, 0.0f, 1.0f);
glm::vec4 lightColor2 = glm::vec4(0.0f, 0.0f, 300.0f, 1.0f);

// Load glTF models

Model* damagedHelmet =
[Link]("models/DamagedHelmet/[Link]");
Model* flightHelmet =
[Link]("models/FlightHelmet/[Link]");

// The models already have PBR materials defined in the glTF file
// We can render them directly with our PBR pipeline

// Render the models with different transformations

renderModel(damagedHelmet, glm::vec3(-1.0f, 0.0f, 0.0f), glm::vec3(0.5f));
renderModel(flightHelmet, glm::vec3(1.0f, 0.0f, 0.0f), glm::vec3(0.5f));

// We can also experiment with modifying the material properties

// For example, to make the damaged helmet more metallic:
if (damagedHelmet->[Link]() > 0) {
// Store the original value to restore later
float originalMetallic = damagedHelmet->materials[0].metallicFactor;

// Modify the material

damagedHelmet->materials[0].metallicFactor = 1.0f;

// Render with modified material

renderModel(damagedHelmet, glm::vec3(-2.0f, 0.0f, 0.0f), glm::vec3(0.5f));

// Restore original value

damagedHelmet->materials[0].metallicFactor = originalMetallic;
}
}

Conclusion
In this section, we’ve integrated our PBR implementation with the rest of the Vulkan rendering
pipeline. We’ve updated our renderer class to support advanced lighting techniques that can be
used with glTF models and their PBR materials. We’ve created a PBR shader based on the concepts

187
we’ve learned and shown how to test the implementation with glTF models.

This approach provides a solid foundation for rendering physically accurate materials, which we’ll
apply in the Loading_Models chapter when we load and render glTF models. It also gives us the
flexibility to modify and extend the material properties as needed for our specific rendering
requirements.

In the next section, we’ll explore how to add high-quality shadows using Vulkan Ray Query.

Previous: Lighting Implementation | Next: Shadows = Conclusion

In this chapter, we’ve explored the fundamentals of lighting and materials in 3D rendering and
introduced Physically Based Rendering (PBR) using the metallic-roughness workflow. We’ve
covered the theory behind PBR, implemented a shader that can be used with glTF models, and
added high-quality shadows using Vulkan Ray Query. We’ve also learned how to use push constants
to efficiently pass material properties to our shaders.

What We’ve Learned

This chapter has taken you through the essential concepts needed to implement physically-based
rendering in a Vulkan engine. We introduced the metallic‑roughness PBR workflow, mapped glTF
material properties to shader inputs, and used push constants to drive per‑draw material
parameters without descriptor churn. You saw how the BRDF pieces cooperate to conserve energy
and produce plausible lighting, and how to plug the shader into a vk::raii‑based pipeline so models
render correctly end‑to‑end. Finally, we integrated hardware-accelerated ray-traced shadows for
improved realism.

Making it click: a mental model of this PBR

pipeline
At a high level, think of your frame as a linear-light computation that transforms physical inputs
into displayable pixels:

• Inputs in linear space: lights with intensities in physical-ish units, baseColor/metallic/roughness

from material, and normal/AO/emissive maps. The work is done in linear HDR so you don’t lose
headroom.

• BRDF roles: D shapes the highlight (roughness controls lobe width), G enforces masking/self-
shadowing on microfacets, F boosts reflectance at grazing angles and ties reflectivity to material
type via F0. Energy conservation links specular (kS) and diffuse (kD) so total doesn’t exceed
what came in.

• Material knobs as levers:

◦ Roughness: widens/narrows the specular lobe and also reduces peak intensity via G.

◦ Metallic: cross-fades between dielectric behavior (colored diffuse + neutral specular) and
conductor behavior (colored specular, no diffuse).

◦ Base color: is diffuse albedo for dielectrics and colored specular for metals.

188
 • Normal/AO/emissive context: normal maps perturb local orientation to add detail, AO damps
indirect/ambient to avoid flat crevices, emissive adds light-independent glow.

• Output staging: after summing ambient/indirect and direct lighting, compress HDR with a tone
mapper (e.g., Reinhard/ACES) and only then apply gamma to match the display. Do gamma
exactly once (either shader pow or sRGB framebuffer).

A quick reasoning loop when results look off:

1. Confirm spaces and order: linear lighting ➜ tone map ➜ gamma. Check you’re not doing double-
gamma.

2. Probe the BRDF: plastic look on everything? Roughness too low or kD not reduced by metallic.
Dim, muddy highlights? Roughness too high or exposure too low.

3. Validate normals/TBN: inverted green channel or wrong tangent handedness causes odd
shading and seams.

4. Calibrate exposure/tone map: if whites clip harshly, add exposure control and/or switch to
ACES/Hable for smoother roll-off.

5. Use AO and emissive judiciously: AO should affect ambient/IBL, not direct specular; emissive is
additive and independent of lights.

This mental model helps you predict how a change to any input will echo through the pipeline and
appear on screen, which is the core of “understanding,” not just following a list.

Potential Improvements
Our PBR pass is a solid baseline. The most impactful upgrades are image‑based lighting
(environment maps for ambient/indirect) and a few material extensions (e.g., clear coat or
anisotropy). On the performance side, consider clustered forward or a deferred path when light
counts grow. If you build an HDR chain, bloom and a more filmic tone mapper (ACES/Hable) round
out the presentation.

Next Steps
Pick one thread and go deep. For lighting, explore GI/AO/volumetrics as time allows. For materials,
design a data‑driven system that maps glTF (and custom) parameters cleanly to your shaders. For
visuals, prototype post effects (fog, bloom, DoF). For performance, profile first, then optimize the
hot spots—especially on mobile.

Remember that lighting is a complex topic with many approaches and techniques. The
implementation we’ve covered in this chapter is just the beginning. As you continue to develop
your engine, you’ll likely want to refine and expand your lighting system to meet the specific needs
of your projects.

In the next chapter, we’ll explore GUI implementation, which will allow us to create interactive
user interfaces for our applications.

Previous: Shadows | Next: GUI = Shadows: Ray Query Integration

189
Shadows are more than just dark patches on the ground; they are fundamental to how we perceive
3D space. They provide critical visual cues about the position, shape, and scale of objects, as well as
the nature of the light sources illuminating them. In this section, we’ll move from simple "flat"
lighting to a more realistic model by implementing hardware-accelerated shadows using Vulkan
Ray Query.

Understanding Shadows
In the physical world, shadows occur when an opaque object obstructs the path of light from a
source to a surface. To simulate this in computer graphics, we must solve the visibility problem:
for any given point on a surface, is there an unobstructed line of sight to the light source?

The Anatomy of a Shadow

Real-world light sources are rarely infinitesimal points. Because lights have physical size (area
lights), shadows often consist of two distinct regions:

• Umbra: The darkest part of the shadow where the light source is completely occluded.

• Penumbra: The "soft" edge of the shadow where the light source is only partially occluded.

While our initial implementation focuses on "hard" shadows (where a point is either 100% in
shadow or 100% lit), the techniques we use here support advanced soft shadowing by sampling the
light as an area rather than a point.

Shadow Mapping vs. Ray Traced Shadows

For decades, Shadow Mapping has been the industry standard. It involves rendering the scene’s
depth from the light’s perspective into a texture, then comparing distances during the main render
pass. However, shadow mapping comes with significant challenges:

• Resolution & Aliasing: Shadows can look "blocky" if the shadow map resolution is too low.

• Biasing Issues: Finding the right "bias" to prevent shadow acne (self-shadowing) and peter-
panning (shadows detaching from objects) is a constant struggle.

• Memory Overhead: Each light source requires its own depth texture.

Ray Tracing (Ray Query) solves these issues by performing precise geometric intersections.
Instead of checking a low-resolution texture, we ask the GPU: "Does this ray hit any triangle
between point A and point B?" This results in pixel-perfect accuracy and simplifies the handling of
multiple light types (point, spot, directional) without managing dozens of depth maps.

Ray Tracing Fundamentals

To perform ray tracing efficiently, we can’t just loop through every triangle in the scene for every
pixel. Instead, we use Acceleration Structures.

1. Bottom-Level Acceleration Structure (BLAS): This stores the raw geometry (vertices and

190
 indices) for a single mesh. Think of it as a spatial index for a single object.

2. Top-Level Acceleration Structure (TLAS): This contains instances of BLASs. Each instance has
its own transformation matrix, allowing us to place the same mesh multiple times in the world
with minimal memory overhead.

Requirements and Setup

Ray Query requires hardware support and specific Vulkan extensions. In our engine, we ensure
these are enabled during device creation:

• VK_KHR_acceleration_structure

• VK_KHR_ray_query

Building Acceleration Structures

In our engine, the Renderer::buildAccelerationStructures method in renderer_ray_query.cpp
handles the creation. We build one BLAS for each unique mesh and then a TLAS that references
them.

bool Renderer::buildAccelerationStructures(const std::vector<Entity *> &entities)

{
// 1. Create BLAS for each unique mesh
for (auto &mesh : uniqueMeshes) {
buildBlas(mesh); // Compiles mesh data into a GPU-optimized format
}

// 2. Create TLAS by instancing BLASs

std::vector<vk::AccelerationStructureInstanceKHR> instances;
for (auto &entity : entities) {
auto mesh = entity->getComponent<MeshComponent>();
auto transform = entity->getComponent<TransformComponent>();

vk::AccelerationStructureInstanceKHR instance{};
[Link] = toVkTransform(transform->getMatrix());
[Link] = getBufferDeviceAddress(mesh-
>[Link]);
[Link] = 0xFF; // Allows filtering objects during ray tests
instances.push_back(instance);
}
buildTlas(instances);

return true;
}

191
Implementing Ray Query in Shaders
With the TLAS built and bound to a descriptor set, we can perform visibility tests directly in our
PBR fragment shader ([Link]).

The Visibility Test

We implement a helper function traceShadowOccluded. It initializes a RayQuery object, traces a ray,
and checks if it hits any geometry before reaching the light.

[[vk::binding(11, 0)]] RaytracingAccelerationStructure tlas;

static const float RASTER_SHADOW_EPS = 0.002;

bool traceShadowOccluded(float3 origin, float3 direction, float tMin, float tMax)

{
RayDesc ray;
[Link] = origin;
[Link] = direction;
[Link] = tMin;
[Link] = tMax;

RayQuery<RAY_FLAG_NONE> q;
[Link](
tlas,
RAY_FLAG_ACCEPT_FIRST_HIT_AND_END_SEARCH, // Optimization: any hit is enough
to shadow
0xFF,
ray
);

while ([Link]()) {
// [Link]() steps through potential hits.
// For simple opaque shadows, we don't need logic here.
}

return ([Link]() == COMMITTED_TRIANGLE_HIT);

}

Integrating with PBR Lighting

In the main lighting loop, we determine the occlusion status before adding a light’s contribution.

// Inside the fragment shader lighting loop

float3 L = normalize([Link] - [Link]);
float distToLight = length([Link] - [Link]);

192
 // Important: Move the origin slightly along the normal to prevent the ray
// from immediately hitting the surface it started from.
float3 shadowOrigin = [Link] + N * RASTER_SHADOW_EPS;

bool occluded = traceShadowOccluded(shadowOrigin, L, RASTER_SHADOW_EPS, distToLight);

if (!occluded) {
// Add diffuse and specular contributions if not in shadow
directLighting += calculatePBR(L, V, N, ...);
}

From Hard to Soft Shadows

Our engine’s Ray Query implementation in ray_query.slang goes beyond simple hard shadows by
implementing stochastic soft shadows. Instead of treating the light as a single point, we treat it as
an area light with a defined radius.

Area Light Approximation

We simulate an area light by jittering the light position for each shadow ray. Using a stable random
number generator and disk sampling, we pick a random point within the "radius" of the light
source.

// Generate a random sample on a disk to simulate light area

float2 diskSample = rqSampleDisk(rngState);
float3 samplePos = lightPos + (T * diskSample.x + B * diskSample.y) * lightRadius;

float3 L = normalize(samplePos - worldPos);

float distToLight = length(samplePos - worldPos);

// Trace a ray toward the sampled point on the light

bool occluded = traceShadowOccluded(shadowOrigin, L, RASTER_SHADOW_EPS, distToLight);

Averaging Multiple Samples

By tracing multiple rays (shadowSampleCount) toward different points on the area light and averaging
the results, we produce a smooth transition between lit and shadowed regions (the penumbra).

float visibilityAcc = 0.0;

for (int i = 0; i < shadowSampleCount; ++i) {
// ... calculate jittered L ...
visibilityAcc += traceShadowOccluded(...) ? 0.0 : 1.0;
}
float finalVisibility = visibilityAcc / float(shadowSampleCount);

// Use visibility to scale the light's contribution

193
 directLighting += calculatePBR(...) * finalVisibility;

Challenges and Best Practices

1. Self-Shadowing (Acne): Even with ray tracing, floating-point precision can cause a ray to hit its
own starting triangle. Always use a small EPSILON offset or a TMin value.

2. Alpha Masking & Transmissivity: For foliage or glass, a simple binary hit test isn’t enough. Our
engine handles this by:

◦ Manual Alpha Testing: In the while([Link]()) loop, we fetch the material’s texture and
discard hits that are transparent.

◦ Transmissive Bypass: We can flag certain materials (like glass) as non-occluding for
shadow rays so they don’t cast pitch-black shadows.

3. Performance: Ray tracing is expensive. While Ray Query is faster than a full ray tracing
pipeline for simple visibility, it still adds cost. For high-performance scenarios, consider:

◦ Denoising: If you use multiple rays for soft shadows, you’ll need a denoiser to clean up the
grain.

◦ Culling: Don’t trace rays for lights that are too far away or behind the surface.

Summary and Comparison

Ray Query provides a powerful and flexible way to implement shadows in a modern engine. While
it requires hardware support, it offers significant advantages over traditional shadow mapping:

Feature Shadow Mapping Ray Query

Precision Limited by texture resolution Pixel-perfect (geometric intersection)

(aliasing)

Complexity High (biasing, multi-light Low (direct visibility test)

management)

Memory High (depth maps per light) Low (acceleration structures)

Soft Shadows Complex (PCSS, blurring) Native (area light sampling)

Next Steps & Further Reading

Shadows are a deep topic. Now that you understand how to implement basic and soft shadows
using Ray Query, you can explore more advanced areas:

• PCSS (Percentage Closer Soft Shadows): A raster-based technique for variable-penumbra

shadows (where shadows get softer as the distance from the occluder increases).

• Ambient Occlusion (RTAO): Use ray tracing to calculate how much ambient light reaches a
point by tracing rays in a hemisphere around the normal.

• Vulkan Ray Tracing Tutorial: The NVIDIA Vulkan Ray Tracing Tutorial is an excellent resource

194
 for deep-diving into these extensions.

In the next chapter, we’ll look at how to add a Graphical User Interface (GUI) to control these
lighting and shadow parameters in real-time.

Previous: Vulkan Integration | Next: Conclusion :pp: ++

Lighting & Materials: Basic lighting

models and push constants
This chapter covers the implementation of basic lighting models and the use of push constants for
material properties in Vulkan. Throughout our engine implementation, we use vk::raii dynamic
rendering and C++20 modules to create a modern, efficient, and maintainable codebase.

Contents
• Introduction

• Lighting Models

• Push Constants

• Lighting Implementation

• Vulkan Integration

• Shadows

• Conclusion :pp: ++

GUI: Introduction
Introduction
Welcome to the "GUI" chapter of our "Building a Simple Engine" series! After implementing a
camera system in the previous chapter, we’ll now focus on adding a graphical user interface (GUI)
to our Vulkan application. A well-designed GUI is essential for creating interactive applications that
allow users to control settings, display information, and interact with the 3D scene.

In this chapter, we’ll integrate a popular immediate-mode GUI library called Dear ImGui with our
Vulkan engine. Dear ImGui is widely used in the game and graphics industry due to its simplicity,
performance, and flexibility. It allows developers to quickly create debug interfaces, tools, and in-
game menus without the complexity of traditional retained-mode GUI systems.

This chapter will guide you through integrating a professional GUI system into your Vulkan engine.
We’ll start by setting up Dear ImGui with Vulkan, establishing the foundation for all GUI
functionality. The integration requires careful management of Vulkan resources—we’ll create

195
dedicated buffers, textures, and pipelines that work alongside your existing rendering systems
without interference.

User input handling becomes more complex when you need to support both 3D scene navigation
and GUI interaction. We’ll implement a system that can distinguish between input intended for the
3D world and input meant for interface elements, ensuring smooth interaction with both.

Rather than overwhelming you with exhaustive widget examples, we’ll focus on the key integration
concepts that enable GUI functionality. Understanding these principles will let you implement any
interface elements your project needs.

The rendering integration presents interesting challenges—your GUI needs to render on top of your
3D scene without disrupting the existing pipeline. We’ll solve this by carefully managing render
passes and ensuring proper depth testing and blending.

Finally, we’ll implement object picking, which bridges the gap between your GUI and 3D scene. This
feature allows users to click on 3D objects and see their properties in the interface, creating a
cohesive development environment.

By the end of this chapter, you’ll have a functional GUI system that you can use to control your
camera, adjust rendering settings, and interact with your 3D scene. This will serve as a foundation
for more advanced features in later chapters, such as material editors, scene hierarchies, and
debugging tools.

Prerequisites
This chapter builds directly on the Camera & Transformations chapter, as we’ll extend the camera
system we developed there to work seamlessly with GUI interaction. The camera controls need to
be aware of when the user is interacting with interface elements versus navigating the 3D scene.

You’ll also need a solid understanding of several core Vulkan concepts. The rendering pipeline and
command buffer knowledge is crucial because GUI rendering requires careful coordination with
your existing 3D rendering—we’ll be recording GUI draw calls into the same command buffers
while managing different pipeline states.

Buffer and image creation skills are essential since Dear ImGui requires dedicated vertex and index
buffers for its geometry, plus texture resources for fonts and any custom UI textures.
Understanding descriptor sets and layouts becomes important as we’ll need to create descriptors
specifically for GUI rendering that don’t interfere with your 3D scene descriptors.

Pipeline creation knowledge ties everything together, as we’ll build a specialized graphics pipeline
for GUI rendering with different vertex input, shaders, and render state than your 3D pipeline.

A basic understanding of input handling concepts will help you follow along as we implement the
dual-mode input system that can distinguish between 3D navigation and GUI interaction.

You should also be familiar with the following chapters from the main tutorial:

• Basic Vulkan concepts:

196
 ◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

• Texture mapping

Let’s begin by exploring how to implement a professional GUI system with Dear ImGui and Vulkan.

Previous: Lighting & Materials Conclusion | Next: Setting Up Dear ImGui :pp: ++

GUI: Setting Up Dear ImGui

Setting Up Dear ImGui
In this section, we’ll set up Dear ImGui in our Vulkan application. Dear ImGui (also known simply
as ImGui) is a bloat-free graphical user interface library for C++. It outputs optimized vertex buffers
that you can render with your 3D-pipeline-enabled application. It’s particularly well-suited for
integration with graphics APIs like Vulkan.

Adding ImGui to Your Project

First, we need to add ImGui to our project. There are several ways to do this:

1. Git Submodule: Add ImGui as a Git submodule to your project

2. Package Manager: Use a package manager like vcpkg or Conan

3. Manual Integration: Download and include the ImGui source files directly

For this tutorial, we’ll use the manual integration approach for simplicity:

# Clone ImGui repository

git clone [Link] external/imgui

# Copy necessary files to your project

cp external/imgui/imgui.h include/
cp external/imgui/[Link] src/
cp external/imgui/imgui_draw.cpp src/
cp external/imgui/imgui_widgets.cpp src/
cp external/imgui/imgui_tables.cpp src/
cp external/imgui/imgui_demo.cpp src/

Next, update your [Link] to include these files:

# ImGui files

197
 set(IMGUI_SOURCES
src/[Link]
src/imgui_draw.cpp
src/imgui_widgets.cpp
src/imgui_tables.cpp
src/imgui_demo.cpp
)

# Our custom ImGui Vulkan integration

set(IMGUI_VULKAN_SOURCES
src/imgui_vulkan_util.cpp
)

add_executable(VulkanApp
src/[Link]
${IMGUI_SOURCES}
${IMGUI_VULKAN_SOURCES}
)

target_include_directories(VulkanApp PRIVATE include)

Creating an ImGui Integration

Let’s implement the ImGuiVulkanUtil class to handle the integration between ImGui and Vulkan.

The ImGuiVulkanUtil class serves as the bridge between ImGui’s immediate-mode GUI system and
Vulkan’s explicit graphics API. This integration requires careful management of GPU resources,
synchronization, and rendering state to efficiently display user interface elements alongside our 3D
graphics. Let’s break down the class architecture into logical components to understand how each
part contributes to the overall integration.

ImGuiVulkanUtil Architecture: GPU Resource

Management Foundation
First, we establish the core Vulkan resources needed to render ImGui’s dynamically generated UI
geometry on the GPU.

// ImGuiVulkanUtil.h
#pragma once

#include <vulkan/vulkan_raii.hpp>
#include <imgui.h>

class ImGuiVulkanUtil {
private:
// Core GPU rendering resources for UI display
// These objects form the foundation of our ImGui-to-Vulkan rendering pipeline
vk::raii::Sampler sampler{nullptr}; // Texture sampling

198
 configuration for font rendering
Buffer vertexBuffer; // Dynamic vertex buffer
for UI geometry
Buffer indexBuffer; // Dynamic index buffer
for UI triangle connectivity
uint32_t vertexCount = 0; // Current vertex count for
draw commands
uint32_t indexCount = 0; // Current index count for
draw commands
Image fontImage; // GPU texture containing
ImGui font atlas
ImageView fontImageView; // Shader-accessible view
of font texture

The GPU resource foundation reflects ImGui’s dynamic rendering model, where UI geometry is
generated fresh each frame based on the current interface layout. The vertex and index buffers use
host-visible memory to enable efficient CPU updates, while the font texture remains static once
loaded. This hybrid approach balances the need for dynamic UI updates with the performance
benefits of GPU-resident font data.

The buffer sizing strategy must accommodate ImGui’s variable geometry output, which can change
dramatically based on UI complexity. Unlike static 3D models, ImGui generates different amounts of
geometry each frame, requiring our buffers to resize dynamically or be pre-allocated with
sufficient capacity for worst-case scenarios.

ImGuiVulkanUtil Architecture: Vulkan Pipeline

Infrastructure
Next, we set up the Vulkan pipeline objects that define how UI geometry is processed and rendered
by the GPU.

// Vulkan pipeline infrastructure for UI rendering

// These objects define the complete GPU processing pipeline for ImGui elements
vk::raii::PipelineCache pipelineCache{nullptr}; // Pipeline compilation
cache for faster startup
vk::raii::PipelineLayout pipelineLayout{nullptr}; // Resource binding layout
(textures, uniforms)
vk::raii::Pipeline pipeline{nullptr}; // Complete graphics
pipeline for UI rendering
vk::raii::DescriptorPool descriptorPool{nullptr}; // Pool for allocating
descriptor sets
vk::raii::DescriptorSetLayout descriptorSetLayout{nullptr}; // Layout defining
shader resource bindings
vk::raii::DescriptorSet descriptorSet{nullptr}; // Actual resource bindings
for font texture

The pipeline infrastructure creates a specialized graphics pipeline optimized for UI rendering,

199
which differs significantly from typical 3D rendering pipelines. UI rendering typically requires
alpha blending for transparency effects, operates in 2D screen space rather than 3D world space,
and uses simpler shading models focused on texture sampling rather than complex lighting
calculations.

Frames-in-flight safety: If your renderer uses more than one frame in flight and you
do not stall the GPU between frames, you must duplicate the dynamic ImGui buffers
(vertex/index) per frame-in-flight. Using a single shared vertex/index buffer risks
NOTE the CPU overwriting data still in use by the GPU from a previous frame. The simple
single-buffer members shown above are for conceptual clarity; in production, store
vectors of buffers/memories sized to the max frames in flight and update/bind the
buffers for the current frame index.

The descriptor system manages the connection between our CPU-side resources and the GPU
shaders. For UI rendering, this primarily involves binding the font atlas texture to the fragment
shader, though more complex UI systems might include additional textures for icons, backgrounds,
or other visual elements.

ImGuiVulkanUtil Architecture: Device Context and

System Integration
Then, we maintain references to the Vulkan device context and manage integration with the
broader graphics system.

// Vulkan device context and system integration

// These references connect our UI system to the broader Vulkan application
context
vk::raii::Device* device = nullptr; // Primary Vulkan device
for resource creation
vk::raii::PhysicalDevice* physicalDevice = nullptr; // GPU hardware info for
capability queries
vk::raii::Queue* graphicsQueue = nullptr; // Command submission queue
for UI rendering
uint32_t graphicsQueueFamily = 0; // Queue family index for
validation

The device context integration demonstrates the explicit nature of Vulkan’s resource management,
where every operation requires specific device and queue references. Unlike higher-level graphics
APIs that maintain global state, Vulkan requires explicit specification of which GPU device and
command queue should handle each operation.

The queue family index enables validation and optimization by ensuring that UI rendering
operations use compatible queue types. While UI rendering typically uses the same graphics queue
as 3D rendering, some applications might benefit from dedicated queues for different rendering
responsibilities.

200
ImGuiVulkanUtil Architecture: UI State and Rendering
Configuration
After that, we manage UI-specific state including styling, rendering parameters, and dynamic
update tracking.

// UI state management and rendering configuration

// These members control the visual appearance and dynamic behavior of the UI
system
ImGuiStyle vulkanStyle; // Custom visual styling
for Vulkan applications

// Push constants for efficient per-frame parameter updates

// This structure enables fast updates of transformation and styling data
struct PushConstBlock {
glm::vec2 scale; // UI scaling factors for
different screen sizes
glm::vec2 translate; // Translation offset for
UI positioning
} pushConstBlock;

// Dynamic state tracking for performance optimization

bool needsUpdateBuffers = false; // Flag indicating buffer
resize requirements

// Modern Vulkan rendering configuration

vk::PipelineRenderingCreateInfo renderingInfo{}; // Dynamic rendering setup
parameters
vk::Format colorFormat = vk::Format::eB8G8R8A8Unorm; // Target framebuffer
format

The styling and configuration management reflects ImGui’s flexibility in visual presentation while
maintaining compatibility with Vulkan’s explicit rendering model. The push constants provide an
efficient mechanism for updating per-frame parameters like screen resolution changes or UI
scaling factors without requiring descriptor set updates.

The dynamic state tracking optimizes performance by avoiding unnecessary GPU resource updates
when the UI layout remains stable between frames. This optimization becomes particularly
important in applications with complex UIs where buffer updates could otherwise impact frame
rates.

ImGuiVulkanUtil Architecture: Public Interface and

Lifecycle Management
Finally, we define the external interface that applications use to integrate ImGui rendering into
their Vulkan rendering pipeline.

201
 public:
// Lifecycle management for proper resource initialization and cleanup
ImGuiVulkanUtil(vk::raii::Device& device, vk::raii::PhysicalDevice&
physicalDevice,
vk::raii::Queue& graphicsQueue, uint32_t graphicsQueueFamily);
~ImGuiVulkanUtil();

// Core functionality methods for ImGui integration

void init(float width, float height); // Initialize ImGui
context and configure display
void initResources(); // Create all Vulkan
resources for rendering
void setStyle(uint32_t index); // Apply visual styling
themes

// Frame-by-frame rendering operations

bool newFrame(); // Begin new ImGui frame
and generate geometry
void updateBuffers(); // Upload updated
geometry to GPU buffers
void drawFrame(vk::raii::CommandBuffer& commandBuffer); // Record rendering
commands to command buffer

// Input event handling for interactive UI elements

void handleKey(int key, int scancode, int action, int mods); // Process keyboard
input events
bool getWantKeyCapture(); // Query if ImGui wants
keyboard focus
void charPressed(uint32_t key); // Handle character input
for text widgets
};

The public interface design balances ease of integration with performance considerations,
separating one-time setup operations from per-frame rendering tasks. The initialization methods
handle the expensive resource creation that should happen once during application startup, while
the frame-by-frame methods focus on efficient updates and rendering.

The input handling interface enables proper integration with existing input systems, allowing
ImGui to capture relevant events while passing through others to the main application. This
cooperative approach ensures that UI elements can respond to user interaction without interfering
with 3D scene controls or other input handling.

Implementing the ImGuiVulkanUtil Class

Now let’s implement the methods of our ImGuiVulkanUtil class for the Vulkan implementation.

202
Constructor and Destructor

First, let’s implement the constructor and destructor:

ImGuiVulkanUtil::ImGuiVulkanUtil(vk::raii::Device& device, vk::raii::PhysicalDevice&

physicalDevice,
vk::raii::Queue& graphicsQueue, uint32_t
graphicsQueueFamily)
: device(&device), physicalDevice(&physicalDevice),
graphicsQueue(&graphicsQueue), graphicsQueueFamily(graphicsQueueFamily),
// Initialize buffers directly
vertexBuffer(*device, 1,
vk::BufferUsageFlagBits::eVertexBuffer,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent),
indexBuffer(*device, 1,
vk::BufferUsageFlagBits::eIndexBuffer,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent) {

// Set up dynamic rendering info

[Link] = 1;
vk::Format formats[] = { colorFormat };
[Link] = &colorFormat;
}

ImGuiVulkanUtil::~ImGuiVulkanUtil() {
// Wait for device to finish operations before destroying resources
// NOTE: waitIdle() is acceptable in destructors/cleanup code but should NEVER be
used
// in the main rendering loop as it causes severe performance issues. For frame
// synchronization, use fences and semaphores instead.
if (device) {
device->waitIdle();
}

// All resources are automatically cleaned up by their destructors

// No manual cleanup needed

// ImGui context is destroyed separately

}

Initialization

Next, let’s implement the initialization methods:

void ImGuiVulkanUtil::init(float width, float height) {

// Initialize ImGui context
IMGUI_CHECKVERSION();

203
 ImGui::CreateContext();

// Configure ImGui
ImGuiIO& io = ImGui::GetIO();
[Link] |= ImGuiConfigFlags_NavEnableKeyboard; // Enable keyboard controls
[Link] |= ImGuiConfigFlags_DockingEnable; // Enable docking

// Set display size

[Link] = ImVec2(width, height);
[Link] = ImVec2(1.0f, 1.0f);

// Set up style
vulkanStyle = ImGui::GetStyle();
[Link][ImGuiCol_TitleBg] = ImVec4(1.0f, 0.0f, 0.0f, 0.6f);
[Link][ImGuiCol_TitleBgActive] = ImVec4(1.0f, 0.0f, 0.0f, 0.8f);
[Link][ImGuiCol_MenuBarBg] = ImVec4(1.0f, 0.0f, 0.0f, 0.4f);
[Link][ImGuiCol_Header] = ImVec4(1.0f, 0.0f, 0.0f, 0.4f);
[Link][ImGuiCol_CheckMark] = ImVec4(0.0f, 1.0f, 0.0f, 1.0f);

// Apply default style

setStyle(0);
}

void ImGuiVulkanUtil::setStyle(uint32_t index) {

ImGuiStyle& style = ImGui::GetStyle();

switch (index) {
case 0:
// Custom Vulkan style
style = vulkanStyle;
break;
case 1:
// Classic style
ImGui::StyleColorsClassic();
break;
case 2:
// Dark style
ImGui::StyleColorsDark();
break;
case 3:
// Light style
ImGui::StyleColorsLight();
break;
}
}

Resource Initialization

Now let’s implement the method to initialize all Vulkan resources needed for ImGui rendering. This
complex process involves several distinct steps that work together to create the GPU resources

204
required for text and UI rendering.

Resource Initialization: Font Data Extraction and

Memory Calculation
First extract font atlas data from ImGui and calculate the memory requirements for GPU storage.

void ImGuiVulkanUtil::initResources() {
// Extract font atlas data from ImGui's internal font system
// ImGui generates a texture atlas containing all glyphs needed for text rendering
ImGuiIO& io = ImGui::GetIO();
unsigned char* fontData; // Raw pixel data from font atlas
int texWidth, texHeight; // Dimensions of the generated font
atlas
[Link]->GetTexDataAsRGBA32(&fontData, &texWidth, &texHeight);

// Calculate total memory requirements for GPU transfer

// Each pixel contains 4 bytes (RGBA) requiring precise memory allocation
vk::DeviceSize uploadSize = texWidth * texHeight * 4 * sizeof(char);

The font data extraction represents the bridge between ImGui’s CPU-based text rendering system
and Vulkan’s GPU-based texture pipeline. ImGui automatically generates a font atlas that combines
all required character glyphs into a single texture, optimizing GPU memory usage and reducing
draw calls during text rendering. The RGBA32 format provides full color and alpha support for anti-
aliased text rendering.

Resource Initialization: GPU Image Creation and

Memory Allocation
Next, create the GPU image resources that will store the font texture data in video memory.

// Define image dimensions and create extent structure

// Vulkan requires explicit specification of all image dimensions
vk::Extent3D fontExtent{
static_cast<uint32_t>(texWidth), // Image width in pixels
static_cast<uint32_t>(texHeight), // Image height in pixels
1 // Single layer (not a 3D texture or
array)
};

// Create optimized GPU image for font texture storage

// This image will be sampled by shaders during UI rendering
fontImage = Image(*device, fontExtent, vk::Format::eR8G8B8A8Unorm,
vk::ImageUsageFlagBits::eSampled |
vk::ImageUsageFlagBits::eTransferDst,
vk::MemoryPropertyFlagBits::eDeviceLocal);

205
 // Create image view for shader access
// The image view defines how shaders interpret the raw image data
fontImageView = ImageView(*device, [Link](),
vk::Format::eR8G8B8A8Unorm,
vk::ImageAspectFlagBits::eColor);

The GPU image creation step establishes the foundation for efficient text rendering by allocating
device-local memory that provides optimal access speeds for the GPU. The dual usage flags
(eSampled | eTransferDst) enable both data upload operations and shader sampling, while the
RGBA8_UNORM format ensures consistent color representation across different GPU architectures.

Resource Initialization — Staging Buffer Creation and

Data Transfer
Next, we create a temporary staging buffer and transfer the font data from CPU memory to GPU
memory.

// Create staging buffer for efficient CPU-to-GPU data transfer

// Host-visible memory allows direct CPU access for data upload
Buffer stagingBuffer(*device, uploadSize, vk::BufferUsageFlagBits::eTransferSrc,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent);

// Map staging buffer memory and copy font data

// Direct memory mapping provides the fastest path for data transfer
void* data = [Link](); // Map GPU memory to
CPU address space
memcpy(data, fontData, uploadSize); // Copy font atlas data
to GPU memory
[Link](); // Unmap memory to
ensure data consistency

The staging buffer approach represents the most efficient method for transferring large amounts of
data from CPU to GPU memory in Vulkan. Host-visible memory enables direct CPU access while
host-coherent ensures that CPU writes are immediately visible to the GPU without requiring explicit
cache flushes. This intermediate step is necessary because device-local memory (where the final
image resides) is typically not directly accessible by the CPU.

Resource Initialization — Image Layout Transitions

and Data Upload
Then, we manage the image layout transitions required for safe data transfer in Vulkan’s explicit
synchronization model.

206
 // Transition image to optimal layout for data reception
// Vulkan requires explicit layout transitions for optimal performance and
correctness
transitionImageLayout([Link](), vk::Format::eR8G8B8A8Unorm,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eTransferDstOptimal);

// Execute the actual buffer-to-image copy operation

// This transfers font data from staging buffer to the final GPU image
copyBufferToImage([Link](), [Link](),
static_cast<uint32_t>(texWidth),
static_cast<uint32_t>(texHeight));

// Transition image to shader-readable layout for rendering

// Final layout optimization enables efficient sampling during UI rendering
transitionImageLayout([Link](), vk::Format::eR8G8B8A8Unorm,
vk::ImageLayout::eTransferDstOptimal,
vk::ImageLayout::eShaderReadOnlyOptimal);

The layout transition sequence ensures that the GPU memory subsystem can optimize its internal
data arrangements for each operation type. The eTransferDstOptimal layout provides the best
performance for receiving data uploads, while eShaderReadOnlyOptimal enables efficient texture
sampling during rendering. These transitions include automatic memory barriers that synchronize
access between different GPU pipeline stages.

Resource Initialization — Texture Sampling

Configuration and Descriptor Management
Finally, we create the sampling configuration and descriptor resources needed for shader access to
the font texture.

// Configure texture sampling parameters for optimal text rendering

// These settings directly impact text quality and performance
vk::SamplerCreateInfo samplerInfo{};
[Link] = vk::Filter::eLinear; // Smooth scaling
when magnified
[Link] = vk::Filter::eLinear; // Smooth scaling
when minified
[Link] = vk::SamplerMipmapMode::eLinear; // Smooth
transitions between mip levels
[Link] = vk::SamplerAddressMode::eClampToEdge; // Prevent
texture wrapping
[Link] = vk::SamplerAddressMode::eClampToEdge; // Clean edge
handling
[Link] = vk::SamplerAddressMode::eClampToEdge; // 3D
consistency
[Link] = vk::BorderColor::eFloatOpaqueWhite; // White border

207
 for clamped areas

sampler = device->createSampler(samplerInfo); // Create the GPU

sampler object

// Create descriptor pool for shader resource binding

// Descriptors provide the interface between shaders and GPU resources
vk::DescriptorPoolSize poolSize{vk::DescriptorType::eCombinedImageSampler, 1};

vk::DescriptorPoolCreateInfo poolInfo{};
[Link] = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet; //
Allow individual descriptor set freeing
[Link] = 2; //
Maximum number of descriptor sets
[Link] = 1; //
Number of pool size specifications
[Link] = &poolSize; // Pool
size configuration

descriptorPool = device->createDescriptorPool(poolInfo); //
Create descriptor pool

// Create descriptor set layout defining shader resource interface

// This layout must match the binding declarations in the ImGui shaders
vk::DescriptorSetLayoutBinding binding{};
[Link] = vk::DescriptorType::eCombinedImageSampler; //
Combined texture and sampler
[Link] = 1; //
Single texture binding
[Link] = vk::ShaderStageFlagBits::eFragment; // Used
in fragment shader
[Link] = 0; //
Shader binding point 0

vk::DescriptorSetLayoutCreateInfo layoutInfo{};
[Link] = 1; //
Number of bindings in layout
[Link] = &binding; //
Binding configuration array

descriptorSetLayout = device->createDescriptorSetLayout(layoutInfo); //
Create layout object

// Allocate descriptor set from pool using the defined layout

// This creates the actual binding that connects GPU resources to shaders
vk::DescriptorSetAllocateInfo allocInfo{};
[Link] = *descriptorPool; //
Source pool for allocation
[Link] = 1; //
Number of sets to allocate
vk::DescriptorSetLayout layouts[] = {*descriptorSetLayout}; //

208
Layout template array
[Link] = layouts; //
Layout configuration

descriptorSet = std::move(device->allocateDescriptorSets(allocInfo).front()); //
Allocate and store set

// Update descriptor set with actual font texture and sampler resources
// This final step connects the physical GPU resources to the shader binding
points
vk::DescriptorImageInfo imageInfo{};
[Link] = vk::ImageLayout::eShaderReadOnlyOptimal; //
Expected image layout
[Link] = [Link](); // Font
texture view
[Link] = *sampler; //
Texture sampler

vk::WriteDescriptorSet writeSet{};
[Link] = *descriptorSet; //
Target descriptor set
[Link] = 1; //
Number of resources to bind
[Link] = vk::DescriptorType::eCombinedImageSampler; //
Resource type
[Link] = &imageInfo; //
Image resource information
[Link] = 0; //
Binding point in shader

device->updateDescriptorSets(1, &writeSet, 0, nullptr); //

Execute the binding update

// Create pipeline cache

vk::PipelineCacheCreateInfo pipelineCacheInfo{};
pipelineCache = device->createPipelineCache(pipelineCacheInfo);

// Create pipeline layout

vk::PushConstantRange pushConstantRange{};
[Link] = vk::ShaderStageFlagBits::eVertex;
[Link] = 0;
[Link] = sizeof(PushConstBlock);

vk::PipelineLayoutCreateInfo pipelineLayoutInfo{};
[Link] = 1;
vk::DescriptorSetLayout setLayouts[] = {*descriptorSetLayout};
[Link] = setLayouts;
[Link] = 1;
[Link] = &pushConstantRange;

pipelineLayout = device->createPipelineLayout(pipelineLayoutInfo);

209
 // Create the graphics pipeline with dynamic rendering
// ... (shader loading, pipeline state setup, etc.)

// For brevity, we're omitting the full pipeline creation code here
// In a real implementation, you would:
// 1. Load the vertex and fragment shaders
// 2. Set up all the pipeline state (vertex input, input assembly, rasterization,
etc.)
// 3. Include the renderingInfo in the pipeline creation to enable dynamic
rendering
}

Frame Management and Rendering

Finally, let’s implement the methods for frame management and rendering:

bool ImGuiVulkanUtil::newFrame() {
// Start a new ImGui frame
ImGui::NewFrame();

// Create your UI elements here

// For example:
ImGui::Begin("Vulkan ImGui Demo");
ImGui::Text("Hello, Vulkan!");
if (ImGui::Button("Click me!")) {
// Handle button click
}
ImGui::End();

// End the frame

ImGui::EndFrame();

// Render to generate draw data

ImGui::Render();

// Check if buffers need updating

ImDrawData* drawData = ImGui::GetDrawData();
if (drawData && drawData->CmdListsCount > 0) {
if (drawData->TotalVtxCount > vertexCount || drawData->TotalIdxCount >
indexCount) {
needsUpdateBuffers = true;
return true;
}
}

return false;
}

void ImGuiVulkanUtil::updateBuffers() {

210
 ImDrawData* drawData = ImGui::GetDrawData();
if (!drawData || drawData->CmdListsCount == 0) {
return;
}

// Calculate required buffer sizes

vk::DeviceSize vertexBufferSize = drawData->TotalVtxCount * sizeof(ImDrawVert);
vk::DeviceSize indexBufferSize = drawData->TotalIdxCount * sizeof(ImDrawIdx);

// Resize buffers if needed

if (drawData->TotalVtxCount > vertexCount) {
// Recreate vertex buffer with new size
vertexBuffer = Buffer(*device, vertexBufferSize,
vk::BufferUsageFlagBits::eVertexBuffer,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent);
vertexCount = drawData->TotalVtxCount;
}

if (drawData->TotalIdxCount > indexCount) {

// Recreate index buffer with new size
indexBuffer = Buffer(*device, indexBufferSize,
vk::BufferUsageFlagBits::eIndexBuffer,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent);
indexCount = drawData->TotalIdxCount;
}

// Upload data to buffers

ImDrawVert* vtxDst = static_cast<ImDrawVert*>([Link]());
ImDrawIdx* idxDst = static_cast<ImDrawIdx*>([Link]());

for (int n = 0; n < drawData->CmdListsCount; n++) {

const ImDrawList* cmdList = drawData->CmdLists[n];
memcpy(vtxDst, cmdList->[Link], cmdList->[Link] *
sizeof(ImDrawVert));
memcpy(idxDst, cmdList->[Link], cmdList->[Link] *
sizeof(ImDrawIdx));
vtxDst += cmdList->[Link];
idxDst += cmdList->[Link];
}

[Link]();
[Link]();
}

Begin a rendering scope

Before issuing any UI draw commands, we open a dynamic rendering scope that targets the current
framebuffer. This replaces vkCmdBeginRenderPass/EndRenderPass and keeps the UI pass

211
lightweight.

void ImGuiVulkanUtil::drawFrame(vk::raii::CommandBuffer& commandBuffer) {

ImDrawData* drawData = ImGui::GetDrawData();
if (!drawData || drawData->CmdListsCount == 0) {
return;
}

// Begin dynamic rendering

vk::RenderingAttachmentInfo colorAttachment{};
// Note: In a real implementation, you would set imageView, imageLayout,
// loadOp, storeOp, and clearValue based on your swapchain image

vk::RenderingInfo renderingInfo{};
[Link] = vk::Rect2D{{0, 0}, {static_cast<uint32_t>(drawData-
>DisplaySize.x),
static_cast<uint32_t>(drawData-
>DisplaySize.y)}};
[Link] = 1;
[Link] = 1;
[Link] = &colorAttachment;

[Link](renderingInfo);

At this point, commands affect the UI overlay only. Next we bind state that doesn’t change per draw.

Bind pipeline and set viewport

// Bind the pipeline used for ImGui

[Link](vk::PipelineBindPoint::eGraphics, *pipeline);

// Configure viewport for UI pixel coordinates

vk::Viewport viewport{};
[Link] = drawData->DisplaySize.x;
[Link] = drawData->DisplaySize.y;
[Link] = 0.0f;
[Link] = 1.0f;
[Link](0, viewport);

The pipeline has blending and raster states tailored for UI. The viewport maps ImGui’s coordinate
system to the framebuffer.

Push per-frame constants

// Convert from ImGui coordinates into NDC via a simple scale/translate

[Link] = glm::vec2(2.0f / drawData->DisplaySize.x, 2.0f / drawData-
>DisplaySize.y);

212
 [Link] = glm::vec2(-1.0f);
[Link](*pipelineLayout, vk::ShaderStageFlagBits::eVertex,
0, sizeof(PushConstBlock), &pushConstBlock);

This keeps the shader simple and avoids per-vertex work for coordinate transforms.

Bind geometry buffers

// We already filled these buffers this frame

vk::Buffer vertexBuffers[] = { [Link]() };
vk::DeviceSize offsets[] = { 0 };
[Link](0, 1, vertexBuffers, offsets);
[Link]([Link](), 0, vk::IndexType::eUint16);

Iterate command lists, set scissor, draw

int vertexOffset = 0;
int indexOffset = 0;

for (int i = 0; i < drawData->CmdListsCount; i++) {

const ImDrawList* cmdList = drawData->CmdLists[i];

for (int j = 0; j < cmdList->[Link]; j++) {

const ImDrawCmd* pcmd = &cmdList->CmdBuffer[j];

// Clip per draw call

vk::Rect2D scissor{};
[Link].x = std::max(static_cast<int32_t>(pcmd->ClipRect.x), 0);
[Link].y = std::max(static_cast<int32_t>(pcmd->ClipRect.y), 0);
[Link] = static_cast<uint32_t>(pcmd->ClipRect.z - pcmd-
>ClipRect.x);
[Link] = static_cast<uint32_t>(pcmd->ClipRect.w - pcmd-
>ClipRect.y);
[Link](0, scissor);

// Bind font (and any UI) textures for this draw

[Link](vk::PipelineBindPoint::eGraphics,
*pipelineLayout, 0, *descriptorSet, {});

// Issue indexed draw for this UI batch

[Link](pcmd->ElemCount, 1, indexOffset, vertexOffset,
0);
indexOffset += pcmd->ElemCount;
}

vertexOffset += cmdList->[Link];
}

213
Each ImDrawCmd provides a scissor rect that clips widgets efficiently without extra passes.

End the rendering scope

// Close the rendering scope for the UI overlay

[Link]();
}

Input Handling
Let’s implement the input handling methods:

void ImGuiVulkanUtil::handleKey(int key, int scancode, int action, int mods) {

ImGuiIO& io = ImGui::GetIO();

// This example uses GLFW key codes and actions, but you can adapt this
// to work with any windowing library's input system

// Map the platform-specific key action to ImGui's key state

// In GLFW: GLFW_PRESS = 1, GLFW_RELEASE = 0
const int KEY_PRESSED = 1; // Generic key pressed value
const int KEY_RELEASED = 0; // Generic key released value

if (action == KEY_PRESSED)
[Link][key] = true;
if (action == KEY_RELEASED)
[Link][key] = false;

// Update modifier keys

// These key codes are GLFW-specific, but you would use your windowing library's
// equivalent key codes for other libraries
const int KEY_LEFT_CTRL = 341; // GLFW_KEY_LEFT_CONTROL
const int KEY_RIGHT_CTRL = 345; // GLFW_KEY_RIGHT_CONTROL
const int KEY_LEFT_SHIFT = 340; // GLFW_KEY_LEFT_SHIFT
const int KEY_RIGHT_SHIFT = 344; // GLFW_KEY_RIGHT_SHIFT
const int KEY_LEFT_ALT = 342; // GLFW_KEY_LEFT_ALT
const int KEY_RIGHT_ALT = 346; // GLFW_KEY_RIGHT_ALT
const int KEY_LEFT_SUPER = 343; // GLFW_KEY_LEFT_SUPER
const int KEY_RIGHT_SUPER = 347; // GLFW_KEY_RIGHT_SUPER

[Link] = [Link][KEY_LEFT_CTRL] || [Link][KEY_RIGHT_CTRL];

[Link] = [Link][KEY_LEFT_SHIFT] || [Link][KEY_RIGHT_SHIFT];
[Link] = [Link][KEY_LEFT_ALT] || [Link][KEY_RIGHT_ALT];
[Link] = [Link][KEY_LEFT_SUPER] || [Link][KEY_RIGHT_SUPER];
}

bool ImGuiVulkanUtil::getWantKeyCapture() {
return ImGui::GetIO().WantCaptureKeyboard;

214
 }

void ImGuiVulkanUtil::charPressed(uint32_t key) {

ImGuiIO& io = ImGui::GetIO();
[Link](key);
}

Using the ImGuiVulkanUtil Class

Now that we’ve implemented our ImGuiVulkanUtil class, let’s see how to use it in a Vulkan
application:

// In your application class

ImGuiVulkanUtil imGui;

// During initialization
void initImGui() {
// Initialize ImGui directly
imGui = ImGuiVulkanUtil(
device,
physicalDevice,
graphicsQueue,
graphicsQueueFamily
);

[Link]([Link], [Link]);
[Link](); // No renderPass needed with dynamic rendering
}

// In your render loop

void drawFrame() {
// ... existing frame preparation code ...

// Update ImGui
if ([Link]()) {
[Link]();
}

// Begin command buffer recording

// Note: With dynamic rendering, we don't need to begin a render pass
// The ImGui drawFrame method will handle dynamic rendering internally

// Render scene using dynamic rendering

// ...

// Render ImGui (in multi-frame renderers, pass the current frame index to bind
per-frame buffers)
[Link](commandBuffer);

215
 // ... submit command buffer ...
}

// Input handling
// This example shows how to handle input with GLFW, but you can adapt this
// to work with any windowing library's input system

// Example key callback function for GLFW

void keyCallback(GLFWwindow* window, int key, int scancode, int action, int mods) {
// First check if ImGui wants to capture this input
[Link](key, scancode, action, mods);

// If ImGui doesn't want to capture the keyboard, process for your application
if (![Link]()) {
// Process key for your application
}
}

// Example character input callback for GLFW

void charCallback(GLFWwindow* window, unsigned int codepoint) {
[Link](codepoint);
}

// With other windowing libraries, you would implement similar callback functions
// using their equivalent APIs and event systems

// Cleanup
void cleanup() {
// ... existing cleanup code ...

// ImGui will be automatically cleaned up when the application exits

// No manual cleanup needed
}

Testing the Integration

To verify that our ImGui integration is working correctly, we can use the ImGui demo window,
which showcases all of ImGui’s features:

// In your ImGuiVulkanUtil::newFrame method

bool ImGuiVulkanUtil::newFrame() {
ImGui::NewFrame();

// Show the demo window

ImGui::ShowDemoWindow();

ImGui::EndFrame();
ImGui::Render();

216
 // Check if buffers need updating
// ...
}

With this implementation, you have a Vulkan implementation for ImGui that allows you to
customize the rendering process to fit your specific needs.

In the next section, we’ll explore how to handle input for both the GUI and the 3D scene.

Previous: Introduction | Next: Input Handling :pp: ++

GUI: Input Handling

Input Handling
One of the challenges when integrating a GUI into a 3D application is managing input events. We
need to ensure that input events are correctly routed to either the GUI or the 3D scene. For
example, if the user is interacting with a UI element, we don’t want their mouse movements to also
rotate the camera.

In this section, we’ll explore how to handle input for both the GUI and the 3D scene, ensuring a
smooth user experience regardless of the windowing library you choose to use.

A windowing library is a software framework that provides functionality for

creating and managing application windows, handling user input events (keyboard,
mouse, touch), and interfacing with the operating system’s display and input
NOTE systems. Examples include GLFW, SDL, Qt, and SFML. These libraries abstract the
platform-specific details of window management and input handling, allowing
developers to write code that works across different operating systems without
dealing with platform-specific APIs directly.

Creating a Platform-Agnostic Input System

To create an effective input system that works with any windowing library, we need to abstract the
input mechanisms and provide a clean interface. Let’s define a simple input system that can be
adapted to different platforms:

// InputSystem.h
#pragma once

#include <functional>
#include <unordered_map>
#include <vector>
#include <glm/[Link]>

// Input actions that our application can respond to

217
 enum class InputAction {
MOVE_FORWARD,
MOVE_BACKWARD,
MOVE_LEFT,
MOVE_RIGHT,
MOVE_UP,
MOVE_DOWN,
LOOK_UP,
LOOK_DOWN,
LOOK_LEFT,
LOOK_RIGHT,
ZOOM_IN,
ZOOM_OUT,
TOGGLE_UI_MODE,
// Add more actions as needed
};

// Input state that tracks the current state of inputs

struct InputState {
glm::vec2 cursorPosition = {0.0f, 0.0f};
glm::vec2 cursorDelta = {0.0f, 0.0f};
bool mouseButtons[3] = {false, false, false};
float scrollDelta = 0.0f;

// For touch input

struct TouchPoint {
int id;
glm::vec2 position;
glm::vec2 delta;
};
std::vector<TouchPoint> touchPoints;

// Reset delta values after each frame

void resetDeltas() {
cursorDelta = {0.0f, 0.0f};
scrollDelta = 0.0f;
for (auto& touch : touchPoints) {
[Link] = {0.0f, 0.0f};
}
}
};

class InputSystem {
public:
static void Initialize();
static void Shutdown();

// Update input state (called once per frame)

static void Update(float deltaTime);

// Register a callback for an input action

218
 static void RegisterActionCallback(InputAction action, std::function<void(float)>
callback);

// Process a platform-specific input event

static bool ProcessInputEvent(void* event);

// Get the current input state

static const InputState& GetInputState();

// Check if ImGui is capturing input

static bool IsImGuiCapturingKeyboard();
static bool IsImGuiCapturingMouse();

private:
static InputState inputState;
static std::unordered_map<InputAction, std::function<void(float)>>
actionCallbacks;
};

Input Prioritization
The general approach for input handling in applications with both 3D navigation and GUI is:

1. First, check if the GUI is capturing input (e.g., mouse is over a UI element)

2. If the GUI is not capturing input, then process the input for 3D navigation

Let’s implement this approach using our cross-platform input system:

void processInput(float deltaTime) {

// Check if ImGui is capturing keyboard input
bool imguiCapturingKeyboard = InputSystem::IsImGuiCapturingKeyboard();

// Check if ImGui is capturing mouse input

bool imguiCapturingMouse = InputSystem::IsImGuiCapturingMouse();

// Get the current input state

const InputState& inputState = InputSystem::GetInputState();

// Process keyboard input for camera movement if ImGui is not capturing keyboard
if (!imguiCapturingKeyboard) {
// Forward these to the camera system
// This could be done through the action callback system
if (InputSystem::IsActionActive(InputAction::MOVE_FORWARD))
[Link](CameraMovement::FORWARD, deltaTime);
if (InputSystem::IsActionActive(InputAction::MOVE_BACKWARD))
[Link](CameraMovement::BACKWARD, deltaTime);
if (InputSystem::IsActionActive(InputAction::MOVE_LEFT))
[Link](CameraMovement::LEFT, deltaTime);
if (InputSystem::IsActionActive(InputAction::MOVE_RIGHT))

219
 [Link](CameraMovement::RIGHT, deltaTime);
if (InputSystem::IsActionActive(InputAction::MOVE_UP))
[Link](CameraMovement::UP, deltaTime);
if (InputSystem::IsActionActive(InputAction::MOVE_DOWN))
[Link](CameraMovement::DOWN, deltaTime);
}

// Process mouse/touch input for camera rotation if ImGui is not capturing mouse
if (!imguiCapturingMouse) {
if ([Link].x != 0.0f || [Link].y != 0.0f) {
[Link]([Link].x,
-[Link].y);
}

if ([Link] != 0.0f) {
[Link]([Link]);
}
}
}

Implementing Platform Adapters for Input

While our input system design is platform-agnostic, we still need platform-specific adapters to
bridge between our unified interface and each windowing library’s native input events. Here’s an
example implementation using GLFW, a popular windowing library:

Example: GLFW Implementation

// InputSystem_GLFW.cpp

#include "InputSystem.h"
#include <GLFW/glfw3.h>
#include <imgui.h>

// Store the GLFW window pointer

static GLFWwindow* gWindow = nullptr;
static bool mouseCaptureMode = false;

// GLFW callback functions

static void glfwMouseButtonCallback(GLFWwindow* window, int button, int action, int
mods) {
if (button >= 0 && button < 3) {
InputState& state = InputSystem::GetInputState();
[Link][button] = action == GLFW_PRESS;
}
}

static void glfwCursorPosCallback(GLFWwindow* window, double xpos, double ypos) {

InputState& state = InputSystem::GetInputState();

220
 // Calculate delta from last position
glm::vec2 newPos(static_cast<float>(xpos), static_cast<float>(ypos));
[Link] = newPos - [Link];
[Link] = newPos;
}

static void glfwScrollCallback(GLFWwindow* window, double xoffset, double yoffset) {

InputState& state = InputSystem::GetInputState();
[Link] = static_cast<float>(yoffset);
}

static void glfwKeyCallback(GLFWwindow* window, int key, int scancode, int action, int
mods) {
// Map GLFW keys to our input actions
if (action == GLFW_PRESS || action == GLFW_RELEASE) {
bool pressed = (action == GLFW_PRESS);

// Toggle mouse capture mode with Escape key

if (key == GLFW_KEY_ESCAPE && pressed) {
mouseCaptureMode = !mouseCaptureMode;

if (mouseCaptureMode) {
glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
} else {
glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_NORMAL);
}
}

// Map other keys to actions

// ...
}
}

void InputSystem::Initialize(GLFWwindow* window) {

gWindow = window;

// Set up GLFW callbacks

glfwSetMouseButtonCallback(window, glfwMouseButtonCallback);
glfwSetCursorPosCallback(window, glfwCursorPosCallback);
glfwSetScrollCallback(window, glfwScrollCallback);
glfwSetKeyCallback(window, glfwKeyCallback);

// Initially capture the cursor for camera control

mouseCaptureMode = true;
glfwSetInputMode(window, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
}

void InputSystem::Update(float deltaTime) {

// Poll for input events
glfwPollEvents();

221
 // Update key states for continuous actions (like movement)
if (glfwGetKey(gWindow, GLFW_KEY_W) == GLFW_PRESS) {
if (auto it = [Link](InputAction::MOVE_FORWARD); it !=
[Link]()) {
it->second(deltaTime);
}
}

// ... other keys ...

// Reset delta values after processing

[Link]();
}

bool InputSystem::IsImGuiCapturingKeyboard() {
return ImGui::GetIO().WantCaptureKeyboard;
}

bool InputSystem::IsImGuiCapturingMouse() {
return ImGui::GetIO().WantCaptureMouse;
}

Input Modes
For applications that need different input modes (e.g., camera control vs. UI interaction), we can
implement a mode system:

// Define input modes

enum class InputMode {
CAMERA_CONTROL,
UI_INTERACTION,
OBJECT_MANIPULATION
};

// Current input mode

static InputMode currentInputMode = InputMode::CAMERA_CONTROL;

// Set the input mode

void setInputMode(InputMode mode) {
currentInputMode = mode;

// Update platform-specific settings based on the mode

// This example shows how to implement this with GLFW
if (mode == InputMode::CAMERA_CONTROL) {
// In GLFW, we can disable the cursor for camera control
glfwSetInputMode(gWindow, GLFW_CURSOR, GLFW_CURSOR_DISABLED);
} else {
// For UI interaction, we want the cursor to be visible

222
 glfwSetInputMode(gWindow, GLFW_CURSOR, GLFW_CURSOR_NORMAL);
}

// With other windowing libraries, you would use their equivalent APIs
}

// Toggle between camera control and UI interaction modes

void toggleInputMode() {
if (currentInputMode == InputMode::CAMERA_CONTROL) {
setInputMode(InputMode::UI_INTERACTION);
} else {
setInputMode(InputMode::CAMERA_CONTROL);
}
}

Handling GUI-Specific Input

Some GUI interactions might require special handling. For example, you might want to implement
drag-and-drop functionality or custom keyboard shortcuts for UI elements:

void drawGUI() {
// Start a new ImGui frame
ImGui::NewFrame();

// Create a window for camera controls

ImGui::Begin("Camera Controls");

// Add a button to reset camera position

if (ImGui::Button("Reset Camera")) {
[Link](glm::vec3(0.0f, 0.0f, 3.0f));
[Link](-90.0f);
[Link](0.0f);
}

// Add sliders for camera settings

float movementSpeed = [Link]();
if (ImGui::SliderFloat("Movement Speed", &movementSpeed, 1.0f, 10.0f)) {
[Link](movementSpeed);
}

float sensitivity = [Link]();

if (ImGui::SliderFloat("Mouse Sensitivity", &sensitivity, 0.1f, 1.0f)) {
[Link](sensitivity);
}

float zoom = [Link]();

if (ImGui::SliderFloat("Zoom", &zoom, 1.0f, 45.0f)) {
[Link](zoom);
}

223
 ImGui::End();

// Render ImGui
ImGui::Render();
}

Integrating Input Handling with the Main Loop

Finally, let’s integrate our input handling system with the main loop:

void mainLoop() {
// Main application loop
while (isRunning) {
// Calculate delta time
float deltaTime = calculateDeltaTime();

// Update input system

InputSystem::Update(deltaTime);

// Process input for camera and other systems

processInput(deltaTime);

// Draw GUI
drawGUI();

// Update uniform buffer with latest camera data

updateUniformBuffer(currentFrame);

// Draw frame
drawFrame();
}
}

Main Loop Integration

The input system needs to be integrated with your application’s main loop. Here’s an example of
how to do this with GLFW, but similar principles apply to other windowing libraries:

// Example main loop with GLFW

void runMainLoop() {
// Initialize input system with your window
// With GLFW, this would look like:
InputSystem::Initialize(window);

// Main loop - with GLFW, we check if the window should close

// Other libraries would have their own condition

224
 while (!glfwWindowShouldClose(window)) {
float deltaTime = calculateDeltaTime();

// Update input and process events

// This would be platform-specific
InputSystem::Update(deltaTime);

// Rest of the main loop is platform-independent

processInput(deltaTime);
drawGUI();
updateUniformBuffer(currentFrame);
drawFrame();
}
}

Advanced Input Handling Techniques

For more complex applications, you might want to consider these advanced input handling
techniques:

Gesture Recognition

Gesture recognition can enhance the user experience regardless of which windowing library you
use:

// GestureRecognizer.h
#pragma once

#include <glm/[Link]>
#include <vector>
#include <functional>

enum class GestureType {

TAP,
DOUBLE_TAP,
LONG_PRESS,
SWIPE,
PINCH,
ROTATE,
PAN
};

struct GestureEvent {
GestureType type;
glm::vec2 position;
glm::vec2 delta;
float scale; // For pinch
float rotation; // For rotate
int pointerCount;

225
 };

class GestureRecognizer {
public:
static void Initialize();
static void Update(const InputState& inputState, float deltaTime);

// Register callbacks for different gesture types

static void RegisterGestureCallback(GestureType type, std::function<void(const
GestureEvent&)> callback);

private:
static void detectTap(const InputState& inputState);
static void detectSwipe(const InputState& inputState);
static void detectPinch(const InputState& inputState);
static void detectRotate(const InputState& inputState);
static void detectPan(const InputState& inputState);

static std::unordered_map<GestureType, std::function<void(const GestureEvent&)>>

gestureCallbacks;
};

Input Context System

For more complex applications with different input requirements in different states:

// InputContext.h
#pragma once

#include <string>
#include <unordered_map>
#include <functional>
#include <stack>

class InputContext {
public:
// Create a new input context
static void CreateContext(const std::string& name);

// Push a context onto the stack (making it active)

static void PushContext(const std::string& name);

// Pop the top context from the stack

static void PopContext();

// Get the current active context

static std::string GetActiveContext();

// Register an action handler for a specific context

static void RegisterActionHandler(const std::string& contextName, InputAction

226
 action, std::function<void(float)> handler);

// Process an action in the current context

static void ProcessAction(InputAction action, float deltaTime);

private:
static std::unordered_map<std::string, std::unordered_map<InputAction,
std::function<void(float)>>> contextHandlers;
static std::stack<std::string> contextStack;
};

With these advanced input handling techniques, your application can provide a consistent and
intuitive user experience. In the next section, we’ll explore how to create various UI elements to
control your application.

Previous: Setting Up Dear ImGui | Next: UI Elements :pp: ++

GUI: UI Elements and Integration

Concepts
UI Elements and Integration Concepts
Now that we have set up ImGui and implemented input handling, let’s explore the key concepts of
integrating a GUI with your Vulkan application. We’ll focus on the integration aspects rather than
exhaustive ImGui widget examples, as those are well-documented in the ImGui documentation.

GUI Integration Concepts

When integrating a GUI into a 3D application, there are several important concepts to consider:

1. Separation of Concerns: Keep your GUI code separate from your rendering code to maintain
clean architecture.

2. Performance Impact: GUIs can impact performance, especially with complex layouts or
frequent updates.

3. Input Management: Properly handle input to ensure it’s routed to either the GUI or the 3D
scene.

4. Rendering Order: The GUI is typically rendered after the 3D scene, as an overlay.

5. State Management: Use the GUI to modify application state in a controlled manner.

Basic ImGui Usage

ImGui follows an immediate-mode paradigm, where the UI is recreated every frame. Here’s a
simple example:

227
 void drawGUI() {
// Start a new ImGui frame
ImGui::NewFrame();

// Create a window
ImGui::Begin("Settings");

// Add UI elements here

static bool enableFeature = false;
if (ImGui::Checkbox("Enable Feature", &enableFeature)) {
// This code runs when the checkbox value changes
updateFeatureState(enableFeature);
}

static float value = 0.5f;

if (ImGui::SliderFloat("Parameter", &value, 0.0f, 1.0f)) {
// This code runs when the slider value changes
updateParameter(value);
}

ImGui::End();

// Render ImGui
ImGui::Render();
}

For a comprehensive guide to all available ImGui widgets and their options, please refer to the
official ImGui documentation and demo: [Link]
imgui_demo.cpp

GUI Design Considerations for Vulkan Applications

When designing a GUI for your Vulkan application, consider these aspects:

Memory Management

ImGui generates vertex and index buffers that need to be uploaded to the GPU. Ensure these
resources are properly managed:

1. Buffer Sizing: Allocate buffers with sufficient size or implement resizing logic

2. Memory Types: Use host-visible memory for frequent updates

3. Synchronization: Ensure buffer updates are synchronized with rendering

Command Buffer Integration

Integrate ImGui rendering commands with your Vulkan command buffers:

228
 // Record commands for scene rendering
// ...

// Record ImGui rendering commands

[Link](commandBuffer);

// Submit command buffer

// ...

Descriptor Resources

ImGui requires descriptors for its font texture. Ensure your descriptor pool has sufficient capacity:

// Create descriptor pool with enough capacity for ImGui

vk::DescriptorPoolSize poolSizes[] = {
{ vk::DescriptorType::eCombinedImageSampler, 50 },
// Other descriptor types...
};

vk::DescriptorPoolCreateInfo poolInfo{};
[Link] = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet;
[Link] = 50;
[Link] = static_cast<uint32_t>(std::size(poolSizes));
[Link] = poolSizes;

descriptorPool = [Link](poolInfo);

Performance Considerations
When integrating ImGui with Vulkan, consider these performance aspects:

1. Command Buffer Recording: Record ImGui commands efficiently, ideally once per frame

2. Descriptor Management: Minimize descriptor set allocations and updates

3. Buffer Updates: Optimize vertex and index buffer updates

4. Pipeline State: Use a dedicated pipeline for ImGui to minimize state changes

5. Render Pass Integration: Consider whether to use a separate render pass or subpass for the
GUI

Frames-in-Flight: Duplicate Dynamic Buffers Per Frame

If your renderer uses multiple frames in flight (e.g., double/triple buffering) without a device wait-
idle between frames, ImGui’s dynamic vertex and index buffers must not be shared across frames.
Otherwise, the CPU can overwrite data that the GPU from a previous frame is still reading.

• Allocate one vertex buffer and one index buffer per frame-in-flight.

229
 • Update/bind the buffers for the current frame index only.

• Size each buffer to the frame’s ImDrawData TotalVtxCount/TotalIdxCount, growing as needed.

Example sketch:

class ImGuiSystem {
// ...
std::vector<vk::raii::Buffer> vertexBuffers;
std::vector<vk::raii::DeviceMemory> vertexMemories;
std::vector<vk::raii::Buffer> indexBuffers;
std::vector<vk::raii::DeviceMemory> indexMemories;
std::vector<uint32_t> vertexCounts;
std::vector<uint32_t> indexCounts;

bool Initialize(Renderer* renderer, uint32_t w, uint32_t h) {

// ... create pipelines, font, descriptors ...
const uint32_t frames = renderer->GetMaxFramesInFlight();
[Link](frames);
[Link](frames);
[Link](frames);
[Link](frames);
[Link](frames, 0);
[Link](frames, 0);
return true;
}

void Render(vk::raii::CommandBuffer& cmd, uint32_t frameIndex) {

ImGui::Render();
updateBuffers(frameIndex);
// bind per-frame buffers
std::array vb = {*vertexBuffers[frameIndex]};
std::array<vk::DeviceSize,1> offs{};
[Link](0, vb, offs);
[Link](*indexBuffers[frameIndex], 0, vk::IndexType::eUint16);
// draw lists...
}

void updateBuffers(uint32_t frameIndex) {

ImDrawData* dd = ImGui::GetDrawData();
if (!dd || dd->CmdListsCount == 0) return;
vk::DeviceSize vbytes = dd->TotalVtxCount * sizeof(ImDrawVert);
vk::DeviceSize ibytes = dd->TotalIdxCount * sizeof(ImDrawIdx);
// grow-per-frame if needed, then map/copy for this frame only
// ...
}
};

When integrating with your main renderer, pass the current frame index to the ImGui render call:

230
 // inside your frame loop after scene rendering
imguiSystem->Render(commandBuffers[currentFrame], currentFrame);

Organizing Your GUI Code

For maintainable GUI code, consider these organizational patterns:

1. Component-Based Approach: Split your GUI into logical components

2. State Management: Use a centralized state store that the GUI can modify

3. Event System: Implement an event system for GUI-triggered actions

4. Lazy Updates: Only update Vulkan resources when GUI settings actually change

// Component-based approach example

class VulkanGUI {
private:
// GUI state
struct {
bool showRenderSettings = true;
bool showPerformance = true;
bool showSceneControls = true;
} state;

// Components
void drawRenderSettingsPanel();
void drawPerformancePanel();
void drawSceneControlsPanel();

public:
void draw() {
// Start a new ImGui frame
ImGui::NewFrame();

// Draw components based on state

if ([Link]) drawRenderSettingsPanel();
if ([Link]) drawPerformancePanel();
if ([Link]) drawSceneControlsPanel();

// Main menu for toggling panels

if (ImGui::BeginMainMenuBar()) {
if (ImGui::BeginMenu("View")) {
ImGui::MenuItem("Render Settings", nullptr,
&[Link]);
ImGui::MenuItem("Performance", nullptr, &[Link]);
ImGui::MenuItem("Scene Controls", nullptr, &[Link]);
ImGui::EndMenu();
}
ImGui::EndMainMenuBar();

231
 }

// Render ImGui
ImGui::Render();
}
};

Displaying Textures in ImGui

A common requirement in GUI systems is displaying textures, such as rendered scenes, material
previews, or icons. ImGui provides the ability to display textures through its ImGui::Image and
ImGui::ImageButton functions. To use these with Vulkan, you need to properly set up descriptor sets
for your textures.

Setting Up Texture Descriptors

To display a Vulkan texture in ImGui, you need to:

1. Create a descriptor set layout for the texture

2. Allocate a descriptor set

3. Update the descriptor set with your texture’s image view and sampler

4. Pass the descriptor set handle to ImGui

Create the descriptor set layout

This layout declares a single combined image sampler the shader can sample from when ImGui
draws the quad.

// Create a descriptor set layout for textures

vk::DescriptorSetLayoutBinding binding{};
[Link] = vk::DescriptorType::eCombinedImageSampler;
[Link] = 1;
[Link] = vk::ShaderStageFlagBits::eFragment;
[Link] = 0;

vk::DescriptorSetLayoutCreateInfo layoutInfo{};
[Link] = 1;
[Link] = &binding;

vk::raii::DescriptorSetLayout textureSetLayout =
[Link](layoutInfo);

Allocate a descriptor set

Allocate one set per texture you want to show in ImGui.

232
 // Allocate a descriptor set for each texture
vk::DescriptorSetAllocateInfo allocInfo{};
[Link] = *descriptorPool;
[Link] = 1;
vk::DescriptorSetLayout layouts[] = {*textureSetLayout};
[Link] = layouts;

vk::raii::DescriptorSet textureDescriptorSet =
std::move([Link](allocInfo).front());

Update the descriptor set

Point the descriptor at your image view and sampler in shader‑read layout.

// Update the descriptor set with your texture

vk::DescriptorImageInfo imageInfo{};
[Link] = vk::ImageLayout::eShaderReadOnlyOptimal;
[Link] = [Link]();
[Link] = *textureSampler;

vk::WriteDescriptorSet writeSet{};
[Link] = *textureDescriptorSet;
[Link] = 1;
[Link] = vk::DescriptorType::eCombinedImageSampler;
[Link] = &imageInfo;
[Link] = 0;

[Link](1, &writeSet, 0, nullptr);

Use it in ImGui

Once you have set up the descriptor set, you can use it with ImGui’s image functions:

// Store the descriptor set as ImTextureID (which is just a void*)

ImTextureID textureId = (ImTextureID)(VkDescriptorSet)*textureDescriptorSet;

// Display the texture in ImGui

ImGui::Begin("Texture Viewer");

// Display as a simple image

ImGui::Image(textureId, ImVec2(width, height));

// Or as an image button
if (ImGui::ImageButton(textureId, ImVec2(width, height))) {
// Handle button click
}

// You can also apply tinting and modify UV coordinates

233
 ImGui::Image(textureId, ImVec2(width, height),
ImVec2(0, 0), ImVec2(1, 1), // UV coordinates (0,0) to (1,1) for the
full texture
ImVec4(1, 1, 1, 1), // Tint color (white = no tint)
ImVec4(1, 1, 1, 0.5)); // Border color

ImGui::End();

Complete Example: Texture Manager for ImGui

Here’s a more complete example of a texture manager class that handles multiple textures for
ImGui:

class ImGuiTextureManager {
private:
vk::raii::Device* device = nullptr;
vk::raii::DescriptorPool* descriptorPool = nullptr;
vk::raii::DescriptorSetLayout descriptorSetLayout{nullptr};

struct TextureInfo {
vk::raii::DescriptorSet descriptorSet{nullptr};
uint32_t width;
uint32_t height;
};

std::unordered_map<std::string, TextureInfo> textures;

public:
ImGuiTextureManager(vk::raii::Device& device, vk::raii::DescriptorPool&
descriptorPool)
: device(&device), descriptorPool(&descriptorPool) {

// Create descriptor set layout for textures

vk::DescriptorSetLayoutBinding binding{};
[Link] = vk::DescriptorType::eCombinedImageSampler;
[Link] = 1;
[Link] = vk::ShaderStageFlagBits::eFragment;
[Link] = 0;

vk::DescriptorSetLayoutCreateInfo layoutInfo{};
[Link] = 1;
[Link] = &binding;

descriptorSetLayout = [Link](layoutInfo);
}

// Register a texture for use with ImGui

ImTextureID registerTexture(const std::string& name, vk::ImageView imageView,
vk::Sampler sampler, uint32_t width, uint32_t height) {

234
 // Allocate descriptor set
vk::DescriptorSetAllocateInfo allocInfo{};
[Link] = **descriptorPool;
[Link] = 1;
vk::DescriptorSetLayout layouts[] = {*descriptorSetLayout};
[Link] = layouts;

vk::raii::DescriptorSet descriptorSet = std::move(device-

>allocateDescriptorSets(allocInfo).front());

// Update descriptor set

vk::DescriptorImageInfo imageInfo{};
[Link] = vk::ImageLayout::eShaderReadOnlyOptimal;
[Link] = imageView;
[Link] = sampler;

vk::WriteDescriptorSet writeSet{};
[Link] = *descriptorSet;
[Link] = 1;
[Link] = vk::DescriptorType::eCombinedImageSampler;
[Link] = &imageInfo;
[Link] = 0;

device->updateDescriptorSets(1, &writeSet, 0, nullptr);

// Store texture info

textures[name] = {std::move(descriptorSet), width, height};

// Return the descriptor set as ImTextureID

return (ImTextureID)(VkDescriptorSet)*textures[name].descriptorSet;
}

// Get a previously registered texture

ImTextureID getTexture(const std::string& name) {
if ([Link](name) == [Link]()) {
throw std::runtime_error("Texture not found: " + name);
}

return (ImTextureID)(VkDescriptorSet)*textures[name].descriptorSet;
}

// Get texture dimensions

ImVec2 getTextureDimensions(const std::string& name) {
if ([Link](name) == [Link]()) {
throw std::runtime_error("Texture not found: " + name);
}

return ImVec2(static_cast<float>(textures[name].width),
static_cast<float>(textures[name].height));
}

235
 };

Usage Example

Here’s how you might use the texture manager in your application:

// During initialization
ImGuiTextureManager textureManager(device, descriptorPool);

// Register textures (e.g., after loading a model or rendering to a texture)

ImTextureID albedoTexId = [Link](
"albedo",
albedoImageView,
textureSampler,
albedoWidth,
albedoHeight
);

ImTextureID normalMapId = [Link](

"normalMap",
normalMapImageView,
textureSampler,
normalMapWidth,
normalMapHeight
);

// In your GUI rendering code

void drawMaterialEditor() {
ImGui::Begin("Material Editor");

// Display textures
ImGui::Text("Albedo Texture:");
ImGui::Image([Link]("albedo"),
ImVec2(200, 200));

ImGui::Text("Normal Map:");
ImGui::Image([Link]("normalMap"),
ImVec2(200, 200));

// Material properties
static float roughness = 0.5f;
if (ImGui::SliderFloat("Roughness", &roughness, 0.0f, 1.0f)) {
updateMaterialProperty("roughness", roughness);
}

static float metallic = 0.0f;

if (ImGui::SliderFloat("Metallic", &metallic, 0.0f, 1.0f)) {
updateMaterialProperty("metallic", metallic);
}

236
 ImGui::End();
}

Performance Considerations

When working with textures in ImGui, keep these performance considerations in mind:

1. Descriptor Management: Create descriptor sets for textures only when needed and reuse them

2. Texture Size: Consider using smaller preview versions of textures for the UI

3. Mipmap Selection: For large textures, ensure proper mipmap selection to avoid aliasing

4. Texture Updates: If a texture changes frequently, use a staging buffer for updates

5. Texture Atlas: For many small textures (like icons), consider using a texture atlas

By properly managing textures in your ImGui integration, you can create rich interfaces that
display rendered content, material previews, and other visual elements directly in your GUI.

Object Picking: Interacting with the 3D Scene

An important aspect of GUI integration is handling object picking - selecting 3D objects with the
mouse. This requires coordination between ImGui and your 3D scene:

void handleMouseInput(float mouseX, float mouseY) {

// First, check if ImGui is using this input
ImGuiIO& io = ImGui::GetIO();
if ([Link]) {
// ImGui is using the mouse, don't use it for 3D picking
return;
}

// ImGui isn't using the mouse, so we can use it for 3D picking

pickObject(mouseX, mouseY);
}

void pickObject(float mouseX, float mouseY) {

// Convert screen coordinates to normalized device coordinates
float ndcX = (2.0f * mouseX) / windowWidth - 1.0f;
float ndcY = 1.0f - (2.0f * mouseY) / windowHeight;

// Create a ray from the camera through the mouse position

glm::vec4 clipCoords(ndcX, ndcY, -1.0f, 1.0f);
glm::vec4 eyeCoords = glm::inverse(projectionMatrix) * clipCoords;
eyeCoords = glm::vec4(eyeCoords.x, eyeCoords.y, -1.0f, 0.0f);

glm::vec3 rayDirection = glm::normalize(glm::vec3(

glm::inverse(viewMatrix) * eyeCoords
));

237
 glm::vec3 rayOrigin = [Link]();

// Test for intersections with scene objects

float closestHit = std::numeric_limits<float>::max();
int hitObjectId = -1;

for (size_t i = 0; i < [Link](); i++) {

float hitDistance;
if (rayIntersectsObject(rayOrigin, rayDirection, sceneObjects[i],
hitDistance)) {
if (hitDistance < closestHit) {
closestHit = hitDistance;
hitObjectId = static_cast<int>(i);
}
}
}

// If we hit an object, select it

if (hitObjectId >= 0) {
selectObject(hitObjectId);
}
}

Implementing Ray-Object Intersection

For object picking to work, you need to implement ray-object intersection tests. Here’s a simple
example for sphere intersection:

bool rayIntersectsSphere(
const glm::vec3& rayOrigin,
const glm::vec3& rayDirection,
const glm::vec3& sphereCenter,
float sphereRadius,
float& outDistance
) {
glm::vec3 oc = rayOrigin - sphereCenter;
float a = glm::dot(rayDirection, rayDirection);
float b = 2.0f * glm::dot(oc, rayDirection);
float c = glm::dot(oc, oc) - sphereRadius * sphereRadius;
float discriminant = b * b - 4 * a * c;

if (discriminant < 0) {
return false; // No intersection
}

// Calculate the closest intersection point

float t = (-b - sqrt(discriminant)) / (2.0f * a);
if (t < 0) {
// Try the other intersection point

238
 t = (-b + sqrt(discriminant)) / (2.0f * a);
if (t < 0) {
return false; // Both intersection points are behind the ray
}
}

outDistance = t;
return true;
}

Visualizing Selected Objects

Once an object is selected, you can visualize the selection:

void drawScene(vk::raii::CommandBuffer& commandBuffer) {

// Draw all objects
for (size_t i = 0; i < [Link](); i++) {
// If this object is selected, use a different pipeline
if (static_cast<int>(i) == selectedObjectId) {
[Link](vk::PipelineBindPoint::eGraphics,
*highlightPipeline);
} else {
[Link](vk::PipelineBindPoint::eGraphics,
*standardPipeline);
}

// Draw the object

drawObject(commandBuffer, sceneObjects[i]);
}
}

Integrating Picking with ImGui

You can also display information about the selected object in the GUI:

void drawObjectPropertiesPanel() {
if (selectedObjectId < 0) {
return; // No object selected
}

ImGui::Begin("Object Properties");

SceneObject& obj = sceneObjects[selectedObjectId];

// Display object properties

ImGui::Text("Object ID: %d", selectedObjectId);
ImGui::Text("Name: %s", [Link].c_str());

239
 // Edit object properties
glm::vec3 position = [Link];
if (ImGui::DragFloat3("Position", &position[0], 0.1f)) {
[Link] = position;
updateObjectTransform(selectedObjectId);
}

glm::vec3 rotation = [Link];

if (ImGui::DragFloat3("Rotation", &rotation[0], 1.0f, -180.0f, 180.0f)) {
[Link] = rotation;
updateObjectTransform(selectedObjectId);
}

glm::vec3 scale = [Link];

if (ImGui::DragFloat3("Scale", &scale[0], 0.1f, 0.1f, 10.0f)) {
[Link] = scale;
updateObjectTransform(selectedObjectId);
}

ImGui::End();
}

Object picking creates a powerful interaction model where users can select and manipulate 3D
objects directly, while using the GUI to fine-tune properties. This combination of direct
manipulation and precise control provides an intuitive user experience.

Balancing GUI and 3D Interaction

When designing your application, consider how to balance GUI-based controls with direct 3D
interaction:

1. Use GUI for:

◦ Precise numerical inputs

◦ Complex settings with many options

◦ Hierarchical data visualization

◦ Application-wide controls

2. Use 3D Interaction for:

◦ Object placement and movement

◦ Camera navigation

◦ Direct manipulation of scene elements

◦ Intuitive spatial operations

3. Hybrid Approaches:

◦ Gizmos for 3D transformation with precise control

◦ Context menus that appear near selected objects

240
 ◦ Property panels that update based on selection

By thoughtfully integrating ImGui with your Vulkan application and implementing object picking,
you can create a powerful and intuitive user interface that combines the strengths of both 2D GUI
controls and direct 3D interaction.

In the next section, we’ll explore more details about integrating the GUI rendering with the Vulkan
rendering pipeline.

Previous: Input Handling | Next: Vulkan Integration :pp: ++

GUI: Vulkan Integration

Vulkan Integration
In this section, we’ll explore how to properly integrate ImGui rendering with the Vulkan rendering
pipeline. While we’ve already covered the basic setup in the "Setting Up Dear ImGui" section, here
we’ll dive deeper into the technical details of how ImGui works with Vulkan and how to optimize
the integration.

Understanding the Rendering Flow

Before we dive into the implementation details, let’s understand how ImGui rendering fits into the
Vulkan rendering pipeline:

1. Prepare Frame: Begin a new frame in ImGui and create UI elements

2. Generate Draw Data: ImGui generates vertex and index buffers for the UI

3. Record Commands: Record Vulkan commands to render the ImGui draw data

4. Submit Commands: Submit the commands to the Vulkan queue

5. Present: Present the rendered frame to the screen

This flow needs to be integrated with your existing Vulkan rendering pipeline, which typically
involves:

1. Acquiring the next swap chain image

2. Recording command buffers for scene rendering

3. Submitting command buffers

4. Presenting the rendered image

Dynamic Rendering Configuration

ImGui can be integrated with Vulkan’s dynamic rendering feature, which simplifies the rendering
process by eliminating the need for explicit render passes and framebuffers:

241
 // When initializing ImGui, we set up our custom Vulkan renderer with dynamic
rendering
ImGuiVulkanRenderer renderer;
// ... configure the renderer ...
[Link](*device, *physicalDevice);

// Set up dynamic rendering info

vk::PipelineRenderingCreateInfo renderingInfo{};
[Link] = 1;
vk::Format formats[] = { vk::Format::eB8G8R8A8Unorm };
[Link] = formats;
[Link](renderingInfo);

Dynamic rendering simplifies the integration by removing the dependency on render passes and
framebuffers, making the code more flexible and easier to maintain.

Command Buffer Integration

There are two main approaches to integrating ImGui commands with your Vulkan command
buffers:

1. Single Command Buffer: Record both scene and ImGui rendering commands in the same
command buffer

2. Multiple Command Buffers: Use separate command buffers for scene and ImGui rendering

Let’s look at both approaches:

Single Command Buffer Approach

This is the simplest approach and works well for most applications. With dynamic rendering, the
code becomes even cleaner:

Command Buffer Initialization

The frame rendering process begins with command buffer preparation, where we set up the
recording state and prepare for GPU command submission.

void drawFrame() {
// ... existing frame preparation code ...

// Initialize command buffer recording

// This tells Vulkan we're about to record a sequence of GPU commands
vk::CommandBufferBeginInfo beginInfo{};
[Link](beginInfo);

Command buffer recording represents the heart of Vulkan’s explicit GPU control model. Unlike

242
older APIs where rendering commands are immediately submitted to the GPU, Vulkan allows us to
build up a complete sequence of operations before submission. This approach enables powerful
optimizations like command reordering, parallel command buffer construction, and efficient GPU
scheduling.

The 'begin' operation transitions the command buffer from an initial state into a recording state,
where subsequent API calls will be captured as GPU instructions rather than executed immediately.
This explicit state management gives us precise control over when and how GPU work is submitted,
enabling the fine-grained performance control that makes Vulkan so powerful for demanding
applications.

Dynamic Rendering Attachment Setup

Dynamic rendering requires us to explicitly describe our render targets and their properties,
replacing the traditional render pass system with a more flexible approach.

// Configure color attachment for the main render target

// This describes how the GPU should handle the color output
vk::RenderingAttachmentInfo colorAttachment{};
[Link] = *swapChainImageViews[imageIndex]; // Target
swapchain image
[Link] = vk::ImageLayout::eColorAttachmentOptimal; //
Optimal layout for color output
[Link] = vk::AttachmentLoadOp::eClear; // Clear the
image before rendering
[Link] = vk::AttachmentStoreOp::eStore; // Preserve
results after rendering
[Link] = std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f};
// Clear to black

// Configure depth attachment for 3D depth testing

// This enables proper occlusion and depth sorting for 3D objects
vk::RenderingAttachmentInfo depthAttachment{};
[Link] = *depthImageView; // Depth
buffer image
[Link] = vk::ImageLayout::eDepthStencilAttachmentOptimal; //
Optimal for depth operations
[Link] = vk::AttachmentLoadOp::eClear; // Clear depth
buffer to far plane
[Link] = vk::AttachmentStoreOp::eDontCare; // Don't
preserve depth after rendering
[Link] = vk::ClearDepthStencilValue{1.0f, 0}; //
Clear to maximum depth

The attachment configuration system provides explicit control over how the GPU handles our
render targets throughout the rendering process. By specifying load and store operations, we can
optimize memory bandwidth by only preserving data that needs to carry forward to subsequent
passes. The clear operations ensure we start with a known state, preventing visual artifacts from

243
previous frame data.

Image layout transitions happen automatically based on our specifications, with the GPU driver
handling the necessary memory barriers and cache flushes to ensure data coherency. The optimal
layouts we specify here tell the driver to arrange the image data in whatever format provides the
best performance for the intended usage, rather than forcing a specific memory organization.

Dynamic Rendering Pass Setup

With our attachments configured, we now assemble them into a complete rendering pass that
describes the full rendering operation to the GPU.

// Assemble the complete rendering operation description

// This ties together all our attachments and rendering parameters
vk::RenderingInfo renderingInfo{};
[Link] = vk::Rect2D{{0, 0}, swapChainExtent}; // Render to
entire swapchain area
[Link] = 1; // Single
layer (not array rendering)
[Link] = 1; // One color
output
[Link] = &colorAttachment; // Our
configured color attachment
[Link] = &depthAttachment; // Our
configured depth attachment

// Begin the dynamic rendering pass

// This establishes the rendering context for subsequent draw commands
[Link](renderingInfo);

Dynamic rendering represents a significant evolution from traditional Vulkan render passes,
providing greater flexibility while maintaining the performance benefits of explicit GPU control.
Instead of pre-defining render pass objects at initialization time, we can specify render targets and
their properties at command recording time, enabling more dynamic and flexible rendering
architectures.

The render area specification allows for partial-screen rendering, which can provide significant
performance benefits when only portions of the screen need updating. For full-screen rendering
like our case, we specify the entire swapchain extent to ensure complete coverage.

3D Scene Rendering
The main scene rendering phase handles all 3D geometry, lighting, and material rendering within
the established rendering context.

// Execute 3D scene rendering

// All your existing 3D geometry, lighting, and material rendering happens here

244
 // ... your existing scene rendering code ...

// Complete the 3D rendering pass

// This finalizes all 3D rendering operations and prepares for UI overlay
[Link]();

The scene rendering phase operates within the rendering context we established, with the GPU
automatically handling depth testing, color blending, and other rasterization operations according
to our pipeline configurations. All draw commands issued between beginRendering and
endRendering will target our configured attachments with the specified clear and store behaviors.

The explicit endRendering call ensures that all scene rendering operations are properly completed
and that render targets are transitioned to appropriate states for subsequent operations. This
explicit control allows the GPU driver to perform optimal scheduling and memory management for
the rendering workload.

UI Overlay Integration
The final rendering phase integrates ImGui UI elements as an overlay on top of the 3D scene,
requiring careful coordination between the two rendering systems.

// Render ImGui UI overlay on top of the 3D scene

// The custom renderer handles ImGui's own dynamic rendering setup internally
// This includes vertex buffer uploads, pipeline binding, and draw command
generation
[Link](ImGui::GetDrawData(), commandBuffer);

// Finalize command buffer recording

// This transitions the command buffer to executable state for GPU submission
[Link]();

// Submit command buffer

// ... your existing submission code ...
}

Multiple Command Buffers Approach

This approach gives you more flexibility and can be useful for more complex rendering pipelines.
With dynamic rendering, it becomes even more straightforward:

Multi-Buffer: Scene Command Buffer Recording

The multiple command buffer approach begins by isolating 3D scene rendering into its own
dedicated command buffer, providing greater flexibility for complex rendering pipelines.

void drawFrame() {
// ... existing frame preparation code ...

245
 // Initialize scene-specific command buffer recording
// This dedicated buffer will contain only 3D geometry and lighting operations
vk::CommandBufferBeginInfo beginInfo{};
[Link](beginInfo);

Separating scene rendering into its own command buffer provides several architectural
advantages. First, it enables parallel command buffer recording where different threads can
simultaneously build scene and UI command sequences, improving CPU utilization on multi-core
systems. Second, it allows for independent optimization of each rendering phase, where scene
rendering can use different GPU queues or submission timing than UI rendering.

This separation also facilitates advanced rendering techniques like multi-frame latency
optimization, where scene rendering can be decoupled from UI updates to maintain consistent
frame timing even when one system experiences performance variations.

Multi-Buffer: Scene Attachment Configuration

The scene rendering setup mirrors the single-buffer approach but with explicit ownership of the
attachment configuration within the scene command buffer.

// Configure scene rendering attachments with explicit ownership

// These configurations belong specifically to the scene rendering pass
vk::RenderingAttachmentInfo colorAttachment{};
[Link] = *swapChainImageViews[imageIndex]; // Target
swapchain image
[Link] = vk::ImageLayout::eColorAttachmentOptimal; //
Optimal for color rendering
[Link] = vk::AttachmentLoadOp::eClear; // Clear for
fresh scene start
[Link] = vk::AttachmentStoreOp::eStore; // Preserve
for UI overlay
[Link] = std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f};
// Clear to black

// Configure depth attachment for 3D scene depth testing

// UI rendering won't need depth testing, so this is scene-specific
vk::RenderingAttachmentInfo depthAttachment{};
[Link] = *depthImageView; // Scene
depth buffer
[Link] = vk::ImageLayout::eDepthStencilAttachmentOptimal; //
Optimal for depth ops
[Link] = vk::AttachmentLoadOp::eClear; // Clear
depth for new frame
[Link] = vk::AttachmentStoreOp::eDontCare; // UI doesn't
need depth data
[Link] = vk::ClearDepthStencilValue{1.0f, 0}; //
Clear to far plane

246
The attachment configuration for scene rendering emphasizes the separation of concerns between
3D and UI rendering. The store operation for the color attachment ensures that scene rendering
results are preserved for the subsequent UI overlay, while the depth attachment uses "don’t care"
storage since UI elements typically render without depth testing.

This explicit configuration makes the rendering dependencies clear and helps optimize memory
bandwidth by only preserving the data that subsequent passes actually need.

Multi-Buffer: Scene Rendering Execution

The scene rendering execution occurs within its dedicated command buffer, providing isolated
control over 3D rendering operations.

// Assemble scene rendering configuration

// This defines the complete 3D rendering context
vk::RenderingInfo renderingInfo{};
[Link] = vk::Rect2D{{0, 0}, swapChainExtent}; // Full
screen rendering
[Link] = 1; // Single
rendering layer
[Link] = 1; // One color
output
[Link] = &colorAttachment; // Scene
color configuration
[Link] = &depthAttachment; // Scene
depth configuration

// Execute complete 3D scene rendering pass

[Link](renderingInfo);
// All 3D geometry, lighting, materials, and effects render here
// ... your existing scene rendering code ...
[Link]();

// Finalize scene command buffer for submission

[Link]();

The scene rendering execution benefits from having its own isolated command buffer context,
where all GPU state changes and draw calls are contained within a clearly defined scope. This
isolation makes debugging easier, as scene-specific rendering issues can be analyzed independently
of UI rendering complexity.

Command buffer finalization with end() transitions the buffer to an executable state, ready for GPU
submission, while maintaining clear boundaries between different rendering responsibilities.

Multi-Buffer: UI Command Buffer Setup

The UI rendering phase begins with its own command buffer recording, configured specifically for
overlay rendering requirements.

247
 // Initialize UI-specific command buffer recording
// This dedicated buffer handles only UI overlay operations
[Link](beginInfo);

// Configure UI attachment to preserve scene rendering results

// This is the key difference from scene rendering - we load existing content
[Link] = vk::AttachmentLoadOp::eLoad; // Preserve
scene rendering

// Ensure proper ordering/visibility between scene and UI when using multiple

command buffers.
// If you submit scene and UI command buffers separately, synchronize them either
by:
// - Submitting both on the same queue with a semaphore (scene signals, UI waits
with stage = COLOR_ATTACHMENT_OUTPUT), or
// - Recording a pipeline barrier in the UI command buffer before beginRendering()
to make scene color writes visible.
// Example barrier inserted in the UI command buffer:
{
vk::ImageMemoryBarrier2 barrier{
.srcStageMask = vk::PipelineStageFlagBits2::eColorAttachmentOutput,
.srcAccessMask = vk::AccessFlagBits2::eColorAttachmentWrite,
.dstStageMask = vk::PipelineStageFlagBits2::eColorAttachmentOutput,
.dstAccessMask = vk::AccessFlagBits2::eColorAttachmentRead |
vk::AccessFlagBits2::eColorAttachmentWrite,
.oldLayout = vk::ImageLayout::eColorAttachmentOptimal,
.newLayout = vk::ImageLayout::eColorAttachmentOptimal,
.image = *swapChainImages[imageIndex],
.subresourceRange = { vk::ImageAspectFlagBits::eColor, 0, 1, 0, 1 }
};
vk::DependencyInfo depInfo{ .imageMemoryBarrierCount = 1,
.pImageMemoryBarriers = &barrier };
imguiCommandBuffer.pipelineBarrier2(depInfo);
}

// UI rendering typically doesn't need depth testing

// Remove depth attachment to optimize UI rendering performance
[Link] = nullptr;

The UI command buffer setup demonstrates the power of the multi-buffer approach through its
different attachment configuration. By changing the load operation to eLoad, we preserve the scene
rendering results as the foundation for UI overlay rendering. This approach is more explicit and
controllable than relying on automatic render pass dependencies.

Removing the depth attachment for UI rendering eliminates unnecessary depth testing overhead,
since UI elements typically render in screen space without complex occlusion relationships. This
optimization can provide measurable performance improvements, especially on mobile GPUs
where bandwidth is at a premium.

248
Multi-Buffer: UI Rendering and Submission
Coordination
The final phase handles UI rendering execution and coordinates the submission of both command
buffers in the correct order.

// Execute UI overlay rendering

// The custom renderer handles ImGui's dynamic rendering internally
[Link](ImGui::GetDrawData(), imguiCommandBuffer);

// Finalize UI command buffer

[Link]();

// Coordinate submission of both command buffers in dependency order

// Scene must complete before UI to ensure proper overlay rendering
std::array<vk::CommandBuffer, 2> submitCommandBuffers = {
*sceneCommandBuffer, // Execute scene rendering first
*imguiCommandBuffer // Then execute UI overlay
};

// Configure batch submission for optimal GPU utilization

vk::SubmitInfo submitInfo{};
[Link] =
static_cast<uint32_t>([Link]());
[Link] = [Link]();

// Submit both command buffers as a cohesive frame

// ... rest of your submission code ...
}

Handling Multiple Viewports

ImGui supports multiple viewports, which allows UI windows to be detached from the main
window. To support this feature, we need to handle additional steps:

// In your main loop, after rendering ImGui

if (ImGui::GetIO().ConfigFlags & ImGuiConfigFlags_ViewportsEnable) {
ImGui::UpdatePlatformWindows();
ImGui::RenderPlatformWindowsDefault();
}

This will render any detached ImGui windows. Note that this feature requires additional platform-
specific code and may not be necessary for all applications.

249
Handling Window Resize
When the window is resized, you need to recreate the swap chain and update ImGui:

void recreateSwapChain() {
// ... existing swap chain recreation code ...

// Update ImGui display size

ImGuiIO& io = ImGui::GetIO();
[Link] = ImVec2(static_cast<float>([Link]),
static_cast<float>([Link]));
}

Performance Considerations
Here are some tips to optimize ImGui rendering performance in Vulkan:

1. Minimize State Changes: Try to render all ImGui elements in a single pass to minimize state
changes.

2. Use Appropriate Descriptor Pool Sizes: Allocate enough descriptors for ImGui to avoid
running out of descriptors.

3. Consider Secondary Command Buffers: For complex UIs, consider using secondary command
buffers to record ImGui commands in parallel.

4. Optimize UI Updates: Only update UI elements that change, and consider using ImGui’s Begin()
function with the ImGuiWindowFlags_NoDecoration flag for static UI elements.

5. Use ImGui’s Memory Allocators: ImGui allows you to provide custom memory allocators,
which can be useful for controlling memory usage.

Complete Integration Example

Let’s put everything together in a complete example that integrates ImGui with a Vulkan
application:

class VulkanApplication {
private:
// ... existing Vulkan members ...

// ImGui-specific members
vk::raii::DescriptorPool imguiPool = nullptr;
bool showDemoWindow = true;
bool showMetricsWindow = false;

public:
void initVulkan() {
// ... existing Vulkan initialization ...

250
 // Initialize ImGui
createImGuiDescriptorPool();
initImGui();
}

void createImGuiDescriptorPool() {
// ImGui typically needs a handful of descriptors (font texture + user UI
textures).
// Adjust these values to your app's needs (e.g., expected number of UI
textures, buffers).
// As a starting point:
vk::DescriptorPoolSize poolSizes[] =
{
{ vk::DescriptorType::eSampler, 8 },
{ vk::DescriptorType::eCombinedImageSampler, 128 }, // font + user-
provided textures
{ vk::DescriptorType::eSampledImage, 128 },
{ vk::DescriptorType::eStorageImage, 8 },
{ vk::DescriptorType::eUniformTexelBuffer, 8 },
{ vk::DescriptorType::eStorageTexelBuffer, 8 },
{ vk::DescriptorType::eUniformBuffer, 32 },
{ vk::DescriptorType::eStorageBuffer, 32 },
{ vk::DescriptorType::eUniformBufferDynamic, 16 },
{ vk::DescriptorType::eStorageBufferDynamic, 16 },
{ vk::DescriptorType::eInputAttachment, 8 }
};

// A conservative maxSets equals the sum of descriptor counts.

uint32_t maxSets = 0;
for (const auto& ps : poolSizes) maxSets += [Link];

vk::DescriptorPoolCreateInfo poolInfo{
.flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet,
.maxSets = maxSets,
.poolSizeCount = static_cast<uint32_t>(std::size(poolSizes)),
.pPoolSizes = poolSizes
};

imguiPool = vk::raii::DescriptorPool(device, poolInfo);

}

void initImGui() {
// Initialize ImGui context
IMGUI_CHECKVERSION();
ImGui::CreateContext();
ImGuiIO& io = ImGui::GetIO();
[Link] |= ImGuiConfigFlags_NavEnableKeyboard;
[Link] |= ImGuiConfigFlags_DockingEnable;

// Set up ImGui style

251
 ImGui::StyleColorsDark();

// Initialize our custom backend

int width = static_cast<int>([Link]);
int height = static_cast<int>([Link]);
ImGuiPlatform::Init(width, height);

// Initialize our custom ImGui Vulkan renderer with dynamic rendering

ImGuiVulkanRenderer renderer;
[Link](
*instance,
*physicalDevice,
*device,
graphicsFamily,
*graphicsQueue,
*imguiPool,
static_cast<uint32_t>([Link]()),
vk::SampleCountFlagBits::e1
);

// Set up dynamic rendering info

vk::PipelineRenderingCreateInfo renderingInfo{};
[Link] = 1;
vk::Format formats[] = { swapChainImageFormat };
[Link] = formats;
[Link](renderingInfo);

// Upload ImGui fonts

vk::raii::CommandBuffer commandBuffer = beginSingleTimeCommands();
[Link](commandBuffer);
endSingleTimeCommands(commandBuffer);
}

void drawFrame() {
// ... existing frame preparation code ...

// Start the ImGui frame

ImGui::NewFrame();

// Create ImGui UI
createImGuiUI();

// Render ImGui
ImGui::Render();

// ... existing command buffer recording code ...

// Begin dynamic rendering for scene

vk::RenderingAttachmentInfo colorAttachment{};
[Link] = *swapChainImageViews[imageIndex];
[Link] = vk::ImageLayout::eColorAttachmentOptimal;

252
 [Link] = vk::AttachmentLoadOp::eClear;
[Link] = vk::AttachmentStoreOp::eStore;
[Link] = std::array<float, 4>{0.0f, 0.0f, 0.0f,
1.0f};

vk::RenderingAttachmentInfo depthAttachment{};
[Link] = *depthImageView;
[Link] = vk::ImageLayout::eDepthStencilAttachmentOptimal;
[Link] = vk::AttachmentLoadOp::eClear;
[Link] = vk::AttachmentStoreOp::eDontCare;
[Link] = vk::ClearDepthStencilValue{1.0f, 0};

vk::RenderingInfo renderingInfo{};
[Link] = vk::Rect2D{{0, 0}, swapChainExtent};
[Link] = 1;
[Link] = 1;
[Link] = &colorAttachment;
[Link] = &depthAttachment;

[Link](renderingInfo);

// Render 3D scene
// ... your existing scene rendering code ...

[Link]();

// Render ImGui using our custom renderer

// ImGui will handle its own dynamic rendering internally
[Link](ImGui::GetDrawData(), commandBuffer);

// ... existing command buffer submission code ...

}

void createImGuiUI() {
// Menu bar
if (ImGui::BeginMainMenuBar()) {
if (ImGui::BeginMenu("File")) {
if (ImGui::MenuItem("Exit", "Alt+F4")) {
// Generic way to request application exit
requestApplicationExit();
}
ImGui::EndMenu();
}

if (ImGui::BeginMenu("View")) {
ImGui::MenuItem("Demo Window", nullptr, &showDemoWindow);
ImGui::MenuItem("Metrics", nullptr, &showMetricsWindow);
ImGui::EndMenu();
}

ImGui::EndMainMenuBar();

253
 }

// Demo window
if (showDemoWindow) {
ImGui::ShowDemoWindow(&showDemoWindow);
}

// Metrics window
if (showMetricsWindow) {
ImGui::ShowMetricsWindow(&showMetricsWindow);
}

// Custom windows
ImGui::Begin("Settings");

static float color[3] = { 0.5f, 0.5f, 0.5f };

if (ImGui::ColorEdit3("Clear Color", color)) {
// Update clear color
clearColor = { color[0], color[1], color[2], 1.0f };
}

static int selectedModel = 0;

const char* models[] = { "Cube", "Sphere", "Teapot", "Custom Model" };
if (ImGui::Combo("Model", &selectedModel, models, IM_ARRAYSIZE(models))) {
// Change model
loadModel(models[selectedModel]);
}

ImGui::End();
}

void cleanup() {
// ... existing cleanup code ...

// Cleanup ImGui
[Link]();
ImGuiPlatform::Shutdown(); // Our custom platform backend
ImGui::DestroyContext();
}
};

Advanced Topics
Custom Shaders for ImGui

ImGui uses its own shaders for rendering, but you can customize them if needed:

// Create custom shader modules

vk::raii::ShaderModule customVertShaderModule =

254
 createShaderModule("custom_imgui_vert.spv");
vk::raii::ShaderModule customFragShaderModule =
createShaderModule("custom_imgui_frag.spv");

// Initialize our custom renderer with custom shaders and dynamic rendering
ImGuiVulkanRenderer renderer;
[Link](
*instance,
*physicalDevice,
*device,
queueFamily,
*queue,
*descriptorPool,
minImageCount,
imageCount,
vk::SampleCountFlagBits::e1
);

// Set up dynamic rendering info

vk::PipelineRenderingCreateInfo renderingInfo{};
[Link] = 1;
vk::Format formats[] = { swapChainImageFormat };
[Link] = formats;
[Link](renderingInfo);

// Set custom shaders

[Link](
customVertShaderModule,
customFragShaderModule
);

Rendering ImGui to a Texture

You can render ImGui to a texture instead of directly to the screen, which can be useful for creating
in-game UI elements:

// Create a texture to render ImGui to

vk::raii::Image imguiTargetImage = createImage(
width, height,
vk::Format::eR8G8B8A8Unorm,
vk::ImageTiling::eOptimal,
vk::ImageUsageFlagBits::eColorAttachment | vk::ImageUsageFlagBits::eSampled
);

// Create image view

vk::raii::ImageView imguiTargetImageView = createImageView(
imguiTargetImage,
vk::Format::eR8G8B8A8Unorm,
vk::ImageAspectFlagBits::eColor

255
 );

// Render ImGui to the texture using dynamic rendering

vk::RenderingAttachmentInfo colorAttachment{};
[Link] = *imguiTargetImageView;
[Link] = vk::ImageLayout::eColorAttachmentOptimal;
[Link] = vk::AttachmentLoadOp::eClear;
[Link] = vk::AttachmentStoreOp::eStore;
[Link] = std::array<float, 4>{0.0f, 0.0f, 0.0f, 0.0f};

vk::RenderingInfo renderingInfo{};
[Link] = vk::Rect2D{{0, 0}, {width, height}};
[Link] = 1;
[Link] = 1;
[Link] = &colorAttachment;

[Link](renderingInfo);
[Link](ImGui::GetDrawData(), commandBuffer);
[Link]();

// Later, use the texture in your 3D scene

// ...

Handling High DPI Displays

For high DPI displays, you need to handle scaling correctly across different platforms:

// Cross-platform display scaling

void updateDisplayScale(int width, int height, float scaleX, float scaleY) {
ImGuiIO& io = ImGui::GetIO();
[Link] = ImVec2(static_cast<float>(width), static_cast<float>(height));
[Link] = ImVec2(scaleX, scaleY);

// Update our platform backend

ImGuiPlatform::SetDisplaySize(width, height);
}

// Platform-specific implementations
// Here's an example using GLFW, but you can implement similar functions
// for any windowing library you choose to use

void updateDisplayScaleWithGLFW(GLFWwindow* window) {

// Get the framebuffer size (which may differ from window size on high DPI
displays)
int width, height;
glfwGetFramebufferSize(window, &width, &height);

// Get the content scale (DPI scaling factor)

float xscale, yscale;
glfwGetWindowContentScale(window, &xscale, &yscale);

256
 // Update ImGui with the correct display size and scale
updateDisplayScale(width, height, xscale, yscale);
}

// With other windowing libraries, you would use their equivalent APIs
// to get the framebuffer size and DPI scaling factor

ImGui Utility Class

To encapsulate all the ImGui functionality in a way that works across different platforms, let’s
create a utility class similar to the one mentioned in the Vulkan-Samples repository:

// ImGuiUtil.h
#pragma once

import vulkan_hpp;
#include <imgui.h>
#include <functional>
#include <memory>

class ImGuiUtil {
public:
// Initialize ImGui with Vulkan using dynamic rendering
static void Init(
vk::raii::Instance& instance,
vk::raii::PhysicalDevice& physicalDevice,
vk::raii::Device& device,
uint32_t queueFamily,
vk::raii::Queue& queue,
uint32_t minImageCount,
uint32_t imageCount,
vk::Format swapChainImageFormat,
vk::SampleCountFlagBits msaaSamples = vk::SampleCountFlagBits::e1
);

// Shutdown ImGui
static void Shutdown();

// Start a new frame

static void NewFrame();

// Render ImGui draw data to a command buffer

static void Render(vk::raii::CommandBuffer& commandBuffer);

// Update display size

static void UpdateDisplaySize(int width, int height, float scaleX = 1.0f, float
scaleY = 1.0f);

257
 // Process platform-specific input event
static bool ProcessInputEvent(void* event);

// Set input callback

static void SetInputCallback(std::function<void(ImGuiIO&)> callback);

private:
// Create descriptor pool for ImGui
static void createDescriptorPool();

// Upload fonts
static void uploadFonts();

// Begin single-time commands

static vk::raii::CommandBuffer beginSingleTimeCommands();

// End single-time commands

static void endSingleTimeCommands(vk::raii::CommandBuffer& commandBuffer);

// Vulkan objects - using inline static initialization (C++17)

inline static vk::raii::Instance* instance = nullptr;
inline static vk::raii::PhysicalDevice* physicalDevice = nullptr;
inline static vk::raii::Device* device = nullptr;
inline static uint32_t queueFamily = 0;
inline static vk::raii::Queue* queue = nullptr;
inline static vk::raii::DescriptorPool descriptorPool = nullptr;
inline static vk::raii::CommandPool commandPool = nullptr;
inline static vk::PipelineRenderingCreateInfo renderingInfo{};

// Input callback
inline static std::function<void(ImGuiIO&)> inputCallback = nullptr;

// Initialization state
inline static bool initialized = false;
};

// [Link]
#include "ImGuiUtil.h"

void ImGuiUtil::Init(
vk::raii::Instance& instance,
vk::raii::PhysicalDevice& physicalDevice,
vk::raii::Device& device,
uint32_t queueFamily,
vk::raii::Queue& queue,
uint32_t minImageCount,
uint32_t imageCount,
vk::Format swapChainImageFormat,
vk::SampleCountFlagBits msaaSamples
) {
ImGuiUtil::instance = &instance;

258
ImGuiUtil::physicalDevice = &physicalDevice;
ImGuiUtil::device = &device;
ImGuiUtil::queueFamily = queueFamily;
ImGuiUtil::queue = &queue;

// Set up dynamic rendering info

[Link] = 1;
vk::Format formats[] = { swapChainImageFormat };
[Link] = formats;

// Create command pool for font upload

vk::CommandPoolCreateInfo poolInfo{
.flags = vk::CommandPoolCreateFlagBits::eTransient,
.queueFamilyIndex = queueFamily
};
commandPool = vk::raii::CommandPool(device, poolInfo);

// Create descriptor pool

createDescriptorPool();

// Initialize ImGui context

IMGUI_CHECKVERSION();
ImGui::CreateContext();
ImGuiIO& io = ImGui::GetIO();
[Link] |= ImGuiConfigFlags_NavEnableKeyboard;
[Link] |= ImGuiConfigFlags_DockingEnable;

// Set up ImGui style

ImGui::StyleColorsDark();

// Initialize our custom Vulkan renderer with dynamic rendering

renderer = ImGuiVulkanRenderer();
[Link](
*instance,
*physicalDevice,
*device,
queueFamily,
*queue,
*descriptorPool,
minImageCount,
imageCount,
msaaSamples
);

// Set dynamic rendering info

[Link](renderingInfo);

// Upload fonts
uploadFonts();

initialized = true;

259
 }

void ImGuiUtil::Shutdown() {
if (!initialized) return;

// Wait for device to finish operations

device->waitIdle();

// Cleanup ImGui
[Link]();
ImGui::DestroyContext();

// Cleanup Vulkan resources

commandPool = nullptr;
descriptorPool = nullptr;

// Reset pointers
instance = nullptr;
physicalDevice = nullptr;
device = nullptr;
queue = nullptr;

initialized = false;
}

void ImGuiUtil::NewFrame() {
if (!initialized) return;

// Update ImGui IO with platform-specific input

ImGuiIO& io = ImGui::GetIO();

// Call input callback if registered

if (inputCallback) {
inputCallback(io);
}

ImGui::NewFrame();
}

void ImGuiUtil::Render(vk::raii::CommandBuffer& commandBuffer) {

if (!initialized) return;

ImGui::Render();
[Link](ImGui::GetDrawData(), commandBuffer);
}

void ImGuiUtil::UpdateDisplaySize(int width, int height, float scaleX, float scaleY) {

if (!initialized) return;

ImGuiIO& io = ImGui::GetIO();
[Link] = ImVec2(static_cast<float>(width), static_cast<float>(height));

260
 [Link] = ImVec2(scaleX, scaleY);
}

bool ImGuiUtil::ProcessInputEvent(void* event) {

// Platform-specific event processing would go here
// This is a placeholder for the actual implementation
return false;
}

void ImGuiUtil::SetInputCallback(std::function<void(ImGuiIO&)> callback) {

inputCallback = callback;
}

void ImGuiUtil::createDescriptorPool() {
// Tune these to match your expected number of UI textures and buffers.
vk::DescriptorPoolSize poolSizes[] =
{
{ vk::DescriptorType::eSampler, 8 },
{ vk::DescriptorType::eCombinedImageSampler, 128 },
{ vk::DescriptorType::eSampledImage, 128 },
{ vk::DescriptorType::eStorageImage, 8 },
{ vk::DescriptorType::eUniformTexelBuffer, 8 },
{ vk::DescriptorType::eStorageTexelBuffer, 8 },
{ vk::DescriptorType::eUniformBuffer, 32 },
{ vk::DescriptorType::eStorageBuffer, 32 },
{ vk::DescriptorType::eUniformBufferDynamic, 16 },
{ vk::DescriptorType::eStorageBufferDynamic, 16 },
{ vk::DescriptorType::eInputAttachment, 8 }
};

uint32_t maxSets = 0;
for (const auto& ps : poolSizes) maxSets += [Link];

vk::DescriptorPoolCreateInfo poolInfo{
.flags = vk::DescriptorPoolCreateFlagBits::eFreeDescriptorSet,
.maxSets = maxSets,
.poolSizeCount = static_cast<uint32_t>(std::size(poolSizes)),
.pPoolSizes = poolSizes
};

descriptorPool = vk::raii::DescriptorPool(*device, poolInfo);

}

void ImGuiUtil::uploadFonts() {
vk::raii::CommandBuffer commandBuffer = beginSingleTimeCommands();
[Link](commandBuffer);
endSingleTimeCommands(commandBuffer);
}

vk::raii::CommandBuffer ImGuiUtil::beginSingleTimeCommands() {
vk::CommandBufferAllocateInfo allocInfo{

261
 .commandPool = *commandPool,
.level = vk::CommandBufferLevel::ePrimary,
.commandBufferCount = 1
};

vk::raii::CommandBuffer commandBuffer = vk::raii::CommandBuffers(*device,

allocInfo).front();

vk::CommandBufferBeginInfo beginInfo{
.flags = vk::CommandBufferUsageFlagBits::eOneTimeSubmit
};

[Link](beginInfo);

return commandBuffer;
}

void ImGuiUtil::endSingleTimeCommands(vk::raii::CommandBuffer& commandBuffer) {

[Link]();

vk::SubmitInfo submitInfo{
.commandBufferCount = 1,
.pCommandBuffers = &*commandBuffer
};

queue->submit(submitInfo);
queue->waitIdle();
}

Conclusion
In this section, we’ve explored how to integrate ImGui with Vulkan, including command buffer
integration, render pass configuration, and performance considerations. By creating a flexible
implementation, we’ve ensured that our GUI system works well with any windowing system you
choose.

The key improvements we’ve made include:

1. Creating a platform-agnostic integration approach

2. Implementing a flexible input system that works with various windowing libraries

3. Developing a versatile ImGui utility class

4. Designing a window-system-independent integration

With this knowledge, you can create a robust GUI system for your Vulkan application that provides
a smooth user experience regardless of which windowing system you use.

In the next section, we’ll wrap up with a conclusion and discuss potential improvements to our GUI
system.

262
Previous: UI Elements | Next: Conclusion :pp: ++

GUI: Conclusion
Conclusion
In this chapter, we’ve built a comprehensive GUI system for our Vulkan application using Dear
ImGui. Let’s summarize what we’ve learned and discuss potential improvements.

What We’ve Learned

• Flexible ImGui Setup: We explored how to integrate Dear ImGui with Vulkan in a way that
works across different platforms, including desktop and mobile. We created an implementation
that doesn’t rely on specific windowing systems like GLFW.

• Versatile Input Handling: We implemented a robust input handling system that correctly
routes input events to either the GUI or the 3D scene, ensuring a smooth user experience on any
device.

• UI Elements: We learned how to create various UI elements, from basic components like
buttons and sliders to more complex elements like tables and plots, and how to organize them
into a cohesive interface that works well on both desktop and mobile platforms.

• Vulkan Integration: We dove deep into the technical details of integrating ImGui with the
Vulkan rendering pipeline, including command buffer integration, render pass configuration,
and performance considerations.

With these components in place, we now have a solid foundation for creating interactive
applications with Vulkan that can run on multiple platforms. Our GUI system allows users to
control settings, display information, and interact with the 3D scene through an intuitive interface,
whether they’re using a desktop computer, a mobile phone, or a tablet.

Potential Improvements
While our GUI system is functional, there are several ways it could be enhanced:

• Targeted Optimizations: Implement specific optimizations for better performance on each

target platform.

• Touch-Friendly UI: Enhance the UI elements to be more touch-friendly for mobile platforms,
with larger hit areas and gesture support.

• Adaptive Layouts: Create layouts that automatically adapt to different screen sizes and
orientations, from desktop monitors to mobile phones.

• Custom Styling: Create a custom theme that matches your application’s visual style, rather than
using the default ImGui style.

• Localization: Add support for multiple languages by implementing a localization system for UI
text.

263
 • Accessibility: Improve accessibility by adding features like keyboard navigation, screen reader
support, and high-contrast modes.

• Persistent Settings: Implement a system to save and load UI settings between application
sessions.

• Advanced Layout: Use ImGui’s docking features to create more complex UI layouts, such as
dockable panels.

• Custom Widgets: Develop custom widgets for specific needs in your application, such as a color
wheel, a curve editor, or a node graph editor.

• Performance Optimization: Profile and optimize the GUI rendering to minimize its impact on
overall application performance, especially on mobile devices with limited resources.

• Battery Efficiency: For mobile platforms, optimize the GUI rendering to minimize battery
usage.

Integration with Other Systems

As you continue building your Vulkan engine, consider how the GUI system integrates with other
components:

• Scene Graph: How can the GUI be used to visualize and edit the scene graph hierarchy across
different platforms?

• Material System: Can you create a material editor using the GUI to adjust material properties
in real-time, with interfaces that work well on both desktop and mobile?

• Animation System: How might the GUI be used to control and visualize animations, with
controls that are appropriate for each platform?

• Physics System: Could the GUI provide tools for setting up and debugging physics simulations,
with different interaction models for desktop and mobile?

• Device-Specific Features: How can you leverage specific features (like haptic feedback on
mobile) while maintaining a consistent core experience?

By addressing these questions, you can create a more cohesive and powerful engine that leverages
the GUI for both development and runtime functionality across multiple platforms.

Cross-Platform Considerations
When developing a GUI system that works across platforms, keep these considerations in mind:

• Input Methods: Different platforms have different primary input methods (mouse/keyboard vs.
touch).

• Screen Sizes: Interfaces need to work on screens ranging from small phones to large monitors.

• Performance Constraints: Mobile devices typically have less processing power and memory
than desktops.

• Battery Life: On mobile devices, efficient rendering is crucial for battery life.

• Platform Conventions: Users expect applications to follow platform-specific UI conventions.

264
 • Testing: Cross-platform applications require testing on all target platforms.

Alternative GUI Libraries for Vulkan

While we’ve focused on Dear ImGui in this chapter, there are several other GUI libraries that work
well with Vulkan. Understanding the options can help you choose the right tool for your specific
needs:

• Nuklear: A minimalist immediate-mode GUI library with a small footprint. It’s designed to be
embedded directly into applications and supports Vulkan among other rendering backends.
Nuklear is used in smaller indie games and tools due to its simplicity and low overhead.

• Qt: A comprehensive UI framework that added Vulkan support in Qt 5.10. Qt provides a more
traditional retained-mode GUI approach with a rich set of widgets and tools. It’s used in
applications like the Autodesk Maya viewport and various CAD software.

• CEGUI: The Crazy Eddie’s GUI system is a free library providing windowing and widgets for
games and simulation applications. It has Vulkan renderer support and is used in some indie
game engines.

• Ultralight: A lightweight, high-performance HTML renderer designed for game and application
UIs. It can be integrated with Vulkan and is used by developers who want to leverage web
technologies for their interfaces.

• Noesis GUI: A commercial UI middleware that supports XAML and can render through Vulkan.
It’s used in games like Dauntless and provides a designer-friendly workflow.

When choosing a GUI library for your Vulkan application, consider factors like:

• Development paradigm (immediate-mode vs. retained-mode)

• Performance requirements

• Designer-friendliness

• Learning curve

• Licensing and cost

• Platform support

• Integration complexity

Dear ImGui, which we’ve used in this chapter, strikes a good balance for many developers due to its
simplicity, performance, and ease of integration with Vulkan.

Final Thoughts
A well-designed GUI is essential for creating user-friendly applications that can reach a wide
audience. It serves as the primary way users interact with your application and can significantly
impact the user experience. By understanding how to integrate Dear ImGui with Vulkan and
implementing a robust input handling system that works with basic inputs for mouse and
keyboard, you’ve taken a major step toward creating professional-quality applications.

265
Remember that the code provided in this chapter is a starting point. Feel free to modify and extend
it to suit your specific needs and application requirements. The flexibility of our approach allows
for a wide range of customization and extension while maintaining compatibility with multiple
platforms.

In the next chapter, we’ll explore how to load and render 3D models, which will allow us to create
more complex and visually interesting scenes.

Previous: Vulkan Integration | Next: Loading Models :pp: ++

GUI: Introduce working with a GUI

and handling input
Unresolved directive in 04_GUI_index.adoc - include::01_introduction.adoc[]

Unresolved directive in 04_GUI_index.adoc - include::02_imgui_setup.adoc[]

Unresolved directive in 04_GUI_index.adoc - include::03_input_handling.adoc[]

Unresolved directive in 04_GUI_index.adoc - include::04_ui_elements.adoc[]

Unresolved directive in 04_GUI_index.adoc - include::05_vulkan_integration.adoc[]

Unresolved directive in 04_GUI_index.adoc - include::06_conclusion.adoc[] :pp: ++

Loading Models: Introduction

Introduction
Welcome to the "Loading Models" chapter of our "Building a Simple Engine" series! After exploring
engine architecture and camera systems in the previous chapters, we’re now ready to focus on
handling 3D assets within our engine framework.

In this chapter, we’ll set up a robust model loading system that can handle modern 3D assets.
Building upon the engine architecture we’ve established and the camera system we’ve
implemented, we’ll now add the ability to load and render complex 3D models. In the chapter on
glTF and KTX2 from the main tutorial, we learned about migrating from OBJ to glTF format and the
basics of loading glTF models. Now, we’ll integrate that knowledge into our engine structure to
create a more complete implementation.

This chapter will transform your understanding of 3D asset handling from simple model loading to
sophisticated engine-level systems. We’ll begin by building a scene graph, which provides the
hierarchical organization that complex 3D scenes require. This foundation enables you to group
objects logically, apply transformations at different levels, and efficiently manage scene complexity.

Animation support forms a crucial part of modern 3D applications. We’ll implement a system that

266
can handle glTF’s skeletal animations, giving life to your 3D models through smooth character
movement, object animations, and complex multi-part systems.

The PBR material system we’ll create bridges the gap between the lighting concepts from previous
chapters and real-world asset integration. You’ll see how to map glTF material properties to your
shaders seamlessly, creating a workflow that artists can understand and use effectively.

Rendering multiple objects with different transformations presents both technical and
organizational challenges. We’ll solve these through careful engine architecture that can batch
similar objects efficiently while maintaining the flexibility to handle unique materials and
transformations per object.

Throughout this implementation, we’ll structure our code with engine-level thinking rather than
tutorial-style solutions. This approach will serve you well as your projects grow in complexity and
scope, providing a solid foundation for creating complex scenes with animated models.

Prerequisites
This chapter builds on the foundation established in the main Vulkan tutorial, particularly Chapter
16 (Multiple Objects), as we’ll extend those concepts to handle more complex scene organization
and asset management. The multiple objects chapter introduced the basic concepts of rendering
different geometry, which we’ll now scale up to handle complete 3D models with materials and
animations.

You’ll need solid familiarity with core Vulkan concepts that form the backbone of our model
loading system. Command buffers become more complex when handling multiple models with
different materials, as we’ll need to manage descriptor sets and push constants efficiently.
Understanding graphics pipelines is crucial since different materials might require different
pipeline configurations.

Experience with vertex and index buffers translates directly to model loading, where glTF files
contain vertex data in specific formats that we’ll need to parse and upload to GPU buffers. Uniform
buffers knowledge becomes essential as we’ll use them for transformation matrices, lighting
information, and material properties.

Texture mapping skills are particularly important since glTF models often include multiple textures
per material (diffuse, normal, metallic-roughness, etc.), and we’ll need to load and bind these
textures efficiently.

Finally, basic 3D math understanding (matrices, vectors, quaternions) is crucial for handling model
transformations, animations, and scene hierarchies. If you need a refresher, see the Camera
Transformations chapter for detailed coverage of these mathematical concepts.

Previous: GUI | Next: Setting Up the Project :pp: ++

267
Loading Models: Asset Pipeline
Concepts
1. Understanding Asset Pipelines
After exploring engine architecture and camera systems, it’s important to understand how 3D
assets are managed in rendering engines. A well-designed asset pipeline is crucial for efficiently
handling models, textures, and other resources in any production environment.

1.1. Asset Organization Concepts

When designing an asset organization system, consider these key principles:

1. Categorization - Group similar assets together

2. Hierarchy - Use a nested structure to manage complexity

3. Discoverability - Make assets easy to find and reference

4. Scalability - Design for growth as your project expands

Here’s an example of how assets might be organized in a final product, demonstrating all four
principles:

assets/
├── models/ // 3D model files (Categorization)
│ ├── characters/ // Character models (Hierarchy)
│ │ ├── player/ // Player character models (Hierarchy)
│ │ └── npc/ // Non-player character models (Hierarchy)
│ ├── environments/ // Environment models
│ │ ├── indoor/ // Indoor environment models
│ │ └── outdoor/ // Outdoor environment models
│ └── props/ // Prop models
├── textures/ // Texture files (Categorization)
│ ├── common/ // Shared textures (Discoverability)
│ └── high_resolution/ // High-res textures for close-up views
(Scalability)
├── shaders/ // Shader files
│ ├── core/ // Essential shaders (Discoverability)
│ ├── effects/ // Special effect shaders
│ └── mobile/ // Mobile-optimized shaders (Scalability)
└── config/ // Configuration files
└── quality_presets/ // Different quality settings (Scalability)

This example demonstrates all four principles:

• Categorization: Assets are grouped by type (models, textures, shaders, config)

268
 • Hierarchy: Assets are organized in a nested structure (e.g., models > characters > player)

• Discoverability: Common assets are placed in dedicated folders (e.g., common textures, core
shaders) making them easy to find

• Scalability: The structure accommodates different quality levels and platform-specific assets
(e.g., high-resolution textures, mobile shaders, quality presets)

The specific organization should be tailored to your project’s needs, but the underlying principles
remain consistent across different engines.

1.2. Asset Pipeline Concepts

A professional asset pipeline typically involves several stages, regardless of the specific engine
implementation:

1. Creation - Artists create models in 3D modeling software

2. Export - Models are exported to interchange formats suitable for game engines

3. Validation - Models are checked for issues (e.g., incorrect scale, missing textures)

4. Optimization - Models are optimized for runtime performance

5. Conversion - Development assets are converted to production-ready formats

6. Integration - Assets are imported into the engine

7. Runtime Loading - The engine loads assets efficiently during execution

When designing an asset pipeline, consider these important factors:

1.2.1. File Format Selection

Different file formats offer different trade-offs:

1. Interchange Formats (e.g., glTF, FBX, Collada)

◦ Pros: Widely supported by modeling tools, preserve most data

◦ Cons: May contain unnecessary data, not optimized for runtime

2. Runtime Formats (e.g., glb, engine-specific binary formats)

◦ Pros: Optimized for loading speed and memory usage

◦ Cons: May not be editable outside the engine

1.2.2. Texture Compression

Texture compression is crucial for performance:

1. Development Formats (e.g., PNG, JPEG)

◦ Pros: Lossless or high quality, widely supported by editing tools

◦ Cons: Large file sizes, not optimized for GPU

2. Runtime Formats (e.g., ktx, compressed GPU formats)

269
 ◦ Pros: Smaller file sizes, directly usable by GPU

◦ Cons: May have quality loss, platform-specific considerations

1.2.3. Asset Bundling

Consider how assets are packaged:

1. Separate Files

◦ Pros: Easier to update individual assets, simpler version control

◦ Cons: More file operations, potential for missing dependencies

2. Bundled Assets

◦ Pros: Fewer file operations, guaranteed dependencies

◦ Cons: Larger atomic updates, more complex version control

1.3. Artist-Engine Collaboration Concepts

Successful integration of art assets into a rendering engine requires clear communication and
established workflows between artists and programmers. Here are key concepts to consider:

1.3.1. Technical Specifications

Regardless of the specific engine, you’ll need to define:

1. Coordinate System - Different applications use different coordinate systems (e.g., Y-up vs. Z-up)

2. Scale - Establish a consistent scale (e.g., 1 unit = 1 meter or 1 unit = 1 foot)

3. Origin Placement - Define where the origin point should be for different asset types

4. Level of Detail - Specify polygon count ranges for different asset types and usage scenarios

1.3.2. Workflow Documentation

Create documentation that addresses:

1. Naming Conventions - Consistent naming helps with organization and automation

2. Material Standards - Define how materials should be structured (e.g., PBR parameters)

3. Export Settings - Document the correct export settings for your chosen interchange formats

4. Quality Checklists - Provide criteria for validating assets before submission

1.3.3. Technical Art Bridge

Consider establishing a technical art role that:

1. Creates tools to streamline the art-to-engine pipeline

2. Validates assets before they enter the engine

270
3. Provides feedback to artists on technical requirements

4. Helps troubleshoot issues when assets don’t appear correctly in-engine

1.4. Development to Production Concepts

The transition from artist-friendly development assets to optimized production assets involves
several important concepts:

1.4.1. Development vs. Production Assets

Understanding the different needs at each stage:

1. Development Assets

◦ Prioritize editability and iteration speed

◦ Use formats that are widely supported by content creation tools

◦ May be larger and less optimized for runtime performance

◦ Focus on preserving maximum quality and information

2. Production Assets

◦ Prioritize runtime performance and memory efficiency

◦ Use formats optimized for the target platform(s)

◦ Apply appropriate compression and optimization techniques

◦ Balance quality against performance requirements

1.4.2. Asset Validation

Implement validation at key points in the pipeline:

1. Pre-Submission Validation

◦ Check for adherence to technical specifications

◦ Verify that all required textures and materials are present

◦ Ensure proper scale, orientation, and origin placement

2. Pre-Conversion Validation

◦ Verify that assets can be successfully processed by conversion tools

◦ Check for issues that might cause problems during conversion

3. Post-Conversion Validation

◦ Verify that converted assets maintain visual fidelity

◦ Check for performance issues or memory consumption problems

◦ Ensure compatibility with target platforms

271
1.4.3. Automation Considerations

As projects grow, automation becomes increasingly important:

1. Batch Processing

◦ Develop scripts or tools to process multiple assets at once

◦ Implement automated validation checks

2. Continuous Integration

◦ Consider integrating asset processing into your CI/CD pipeline

◦ Automatically validate and convert assets when they’re committed

3. Versioning

◦ Track changes to assets and their processed versions

◦ Implement dependency tracking to rebuild only what’s necessary

1.5. Implementation Considerations

When implementing a model loading system in any rendering engine, several key considerations
should guide your approach:

1.5.1. Abstraction Layers

Design your model loading system with appropriate abstraction layers:

1. File Format Layer

◦ Handles parsing specific file formats (e.g., glTF, FBX)

◦ Isolates format-specific code to make supporting multiple formats easier

◦ Converts from file format structures to your engine’s internal structures

2. Resource Management Layer

◦ Manages memory and GPU resources

◦ Handles caching and reference counting

◦ Provides a consistent interface regardless of the underlying file format

3. Scene Graph Layer

◦ Organizes models in a hierarchical structure

◦ Manages transformations and parent-child relationships

◦ Facilitates operations like culling and scene traversal

1.5.2. Performance Considerations

Balance flexibility with performance:

1. Asynchronous Loading

272
 ◦ Consider loading models in background threads to avoid blocking the main thread

◦ Implement a system for handling partially loaded models

2. Memory Management

◦ Develop strategies for handling large models

◦ Consider level-of-detail (LOD) systems for complex scenes

◦ Implement streaming for very large environments

3. Batching and Instancing

◦ Group similar models for efficient rendering

◦ Use instancing for repeated elements

1.5.3. Extensibility

Design for future expansion:

1. Material System

◦ Create a flexible material system that can represent various shading models

◦ Support both simple and complex materials

2. Animation System

◦ Design for different animation types (skeletal, morph targets, etc.)

◦ Consider how animations will interact with physics and gameplay systems

3. Custom Data

◦ Allow for engine-specific metadata to be associated with models

◦ Support custom properties for gameplay or rendering purposes

Understanding these concepts provides a solid foundation for designing and implementing model
loading systems in any rendering engine. By carefully considering abstraction, performance, and
extensibility from the beginning, you can create a robust system that will scale with your project’s
needs and adapt to changing requirements.

2. Our Project Implementation

Now that we’ve explored the general concepts of asset pipelines, let’s discuss how our specific
project will implement these concepts.

2.1. File Formats and Directory Structure

For our engine, we’ll use the following file formats and directory structure:

1. Model Format: We’ll use glTF 2.0 binary format (.glb) with embedded KTX2 textures. This
format offers several advantages:

◦ Compact binary representation for efficient storage and loading

273
 ◦ Ability to embed textures, reducing file operations

◦ Support for animations, skinning, and PBR materials

◦ Industry standard with wide tool support

2. Texture Format: We’ll use KTX2 with Basis Universal compression for textures, which
provides:

◦ Significant size reduction compared to PNG/JPEG

◦ GPU-ready formats that can be directly uploaded

◦ Cross-platform compatibility through transcoding

◦ Support for mipmaps and various compression formats

3. Directory Structure:

assets/
├── models/ // 3D model files
│ ├── characters/ // Character models
│ │ └── [Link] // Example character model
│ ├── environments/ // Environment models
│ │ └── [Link] // Example environment model
│ └── props/ // Prop models
│ └── [Link] // Example prop model
└── shaders/ // Shader files
└── [Link] // PBR shader

2.2. Tools and Libraries

We’ll use the following tools and libraries to implement our asset pipeline:

1. Model Loading: We’ll use the tinygltf library to parse glTF files. This library provides:

◦ Comprehensive support for the glTF 2.0 specification

◦ Efficient parsing of binary glTF files

◦ Access to all glTF components (meshes, materials, animations, etc.)

2. Texture Loading: We’ll use the KTX-Software library to load KTX2 textures, which offers:

◦ Support for loading and transcoding Basis Universal compressed textures

◦ Efficient mipmap handling

◦ Integration with Vulkan texture formats

3. Asset Conversion: For converting development assets to production assets, we’ll use:

◦ KTX-Tools for texture conversion (PNG/JPEG to KTX2)

◦ glTF-Transform for model processing and optimization

◦ Custom scripts for automating the conversion process

274
2.3. Integration with Engine Architecture
Our model loading system will integrate with the engine architecture from previous chapters:

1. Resource Management: We’ll leverage the resource management system from the Engine
Architecture chapter to:

◦ Cache loaded models and textures

◦ Implement reference counting for efficient memory management

◦ Support asynchronous loading of models

2. Component System: We’ll create the following components:

◦ ModelComponent: Manages model rendering and animation

◦ MaterialComponent: Handles material properties and textures

◦ These components will work with the TransformComponent from the Camera
Transformations chapter

3. Rendering Pipeline: Our model loading system will integrate with the rendering pipeline by:

◦ Providing mesh data for the geometry pass

◦ Supporting PBR materials for the lighting pass

◦ Enabling instanced rendering for repeated models

2.4. Artist Workflow

Our workflow for artists will be:

1. Development Phase:

◦ Artists create models in tools like Blender or Maya

◦ Export to standard glTF (.gltf) with separate PNG/JPEG textures

◦ Test with glTF viewers to ensure correct appearance

2. Technical Requirements:

◦ Right-handed coordinate system with Y-up

◦ 1 unit = 1 meter scale

◦ PBR materials using the metallic-roughness workflow

◦ Textures with power-of-two dimensions

3. Conversion Process:

◦ Validate models against technical requirements

◦ Convert textures to KTX2 with Basis Universal compression

◦ Embed textures into glb files

◦ Optimize models (remove unused vertices, compress meshes, etc.)

4. Integration:

275
 ◦ Place converted assets in the appropriate directories

◦ Register assets in the resource management system

◦ Create entities with appropriate components

2.5. Runtime Loading

At runtime, our engine will:

1. Load Models:

◦ Parse glb files using tinygltf

◦ Extract mesh data, materials, and animations

◦ Create Vulkan buffers for vertices and indices

2. Process Materials:

◦ Load embedded KTX2 textures

◦ Create Vulkan image views and samplers

◦ Set up descriptor sets for PBR rendering

3. Handle Animations:

◦ Parse animation data from glTF

◦ Implement skeletal animation system

◦ Support animation blending and transitions

4. Render Models:

◦ Use the scene graph to organize models hierarchically

◦ Apply transformations from the transform component

◦ Render with appropriate materials and shaders

By implementing these specific approaches, our engine will have a robust and efficient asset
pipeline that aligns with the general concepts discussed earlier in this chapter.

Previous: Introduction | Next: Implementing the Model Loading System :pp: ++

Loading Models: Implementing the

Model Loading System
1. Implementing the Model Loading System

276
1.1. Building on glTF Knowledge
As we learned in the glTF and KTX2 Migration chapter, glTF is a modern 3D format that supports a
wide range of features including PBR materials, animations, and scene hierarchies. In this chapter,
we’ll leverage these capabilities to build a more robust engine.

While the previous chapter covered the basics of loading glTF models, here we’ll focus on
organizing the loaded data into a proper scene graph and implementing animation support. This
approach will allow us to create more complex and dynamic scenes.

In this chapter, we’ll not only implement the technical aspects of model loading but also discuss the
architectural decisions behind our design and how developers can effectively use this system in
their applications. Understanding these concepts is crucial for building a maintainable and
extensible engine.

1.2. Setting Up Our Engine’s Model System

We’ll start with the same tinygltf library setup as in the previous chapter:

// Include tinygltf for model loading

#include <tiny_gltf.h>

However, instead of just loading the model data directly into vertex and index buffers, we’ll create
a more structured approach with proper data classes to represent our scene.

1.3. Defining Data Structures

To handle the rich data provided by glTF, we need to define several data structures:

// Vertex structure with position, normal, color, and texture coordinates

struct Vertex {
glm::vec3 pos;
glm::vec3 normal;
glm::vec3 color;
glm::vec2 texCoord;

// Binding and attribute descriptions for Vulkan

static vk::VertexInputBindingDescription getBindingDescription() {
return { 0, sizeof(Vertex), vk::VertexInputRate::eVertex };
}

static std::array<vk::VertexInputAttributeDescription, 4>

getAttributeDescriptions() {
return {
vk::VertexInputAttributeDescription( 0, 0, vk::Format::eR32G32B32Sfloat,
offsetof(Vertex, pos) ),
vk::VertexInputAttributeDescription( 1, 0, vk::Format::eR32G32B32Sfloat,

277
 offsetof(Vertex, normal) ),
vk::VertexInputAttributeDescription( 2, 0, vk::Format::eR32G32B32Sfloat,
offsetof(Vertex, color) ),
vk::VertexInputAttributeDescription( 3, 0, vk::Format::eR32G32Sfloat,
offsetof(Vertex, texCoord) )
};
}

// Equality operator and hash function for vertex deduplication

bool operator==(const Vertex& other) const {
return pos == [Link] && normal == [Link] && color == [Link] &&
texCoord == [Link];
}
};

// Structure for PBR material properties

struct Material {
glm::vec4 baseColorFactor = glm::vec4(1.0f);
float metallicFactor = 1.0f;
float roughnessFactor = 1.0f;
glm::vec3 emissiveFactor = glm::vec3(0.0f);

int baseColorTextureIndex = -1;

int metallicRoughnessTextureIndex = -1;
int normalTextureIndex = -1;
int occlusionTextureIndex = -1;
int emissiveTextureIndex = -1;
};

// Structure for a mesh with vertices, indices, and material

struct Mesh {
std::vector<Vertex> vertices;
std::vector<uint32_t> indices;
int materialIndex = -1;
};

1.4. Why We Need a Scene Graph

A scene graph is a tree-like data structure that organizes the spatial representation of a graphical
scene. While it might seem tempting to use a simple collection or map to store 3D objects, scene
graphs offer several critical advantages:

1.4.1. Benefits of Using a Scene Graph

• Hierarchical Transformations: Scene graphs allow child objects to inherit transformations

from their parents. When you move, rotate, or scale a parent node, all its children are
automatically transformed relative to the parent. This is essential for complex models like
characters where moving the torso should also move the attached limbs.

• Spatial Organization: Scene graphs organize objects based on their spatial relationships,

278
 making it easier to perform operations like culling, collision detection, and level-of-detail
management.

• Animation Support: Hierarchical structures are crucial for skeletal animations, where
movements propagate through a chain of bones.

• Scene Management: Scene graphs facilitate operations like saving/loading scenes, instancing
(reusing the same model in different locations), and dynamic scene modifications.

1.4.2. Scene Graphs vs. Simple Collections

Unlike a simple map or array of objects, a scene graph:

• Maintains parent-child relationships between objects

• Automatically propagates transformations down the hierarchy

• Provides a natural structure for traversal algorithms (rendering, picking, collision)

• Supports local-to-global coordinate transformations

For example, with a flat collection of objects, if you wanted to move a character and all its
equipment, you’d need to update each piece individually. With a scene graph, you simply move the
character node, and all attached equipment moves automatically.

1.4.3. Scene Graphs vs. Spatial Partitioning Systems (Game Maps)

It’s important to distinguish between scene graphs and spatial partitioning systems (often referred
to as "game maps" in engine development):

• Scene Graphs focus on hierarchical relationships and transformations between objects.

• Spatial Partitioning Systems focus on efficiently organizing objects in space for collision
detection, visibility determination, and physics calculations.

While scene graphs organize objects based on logical relationships (like a character and its
equipment), spatial partitioning systems organize objects based on their physical location in the
game world.

[Link]. Common Spatial Partitioning Systems

Several spatial partitioning techniques are used in game development:

• Octrees: Divide 3D space into eight equal octants recursively. Used for large open worlds where
objects are distributed unevenly. Octrees adapt to object density, with more subdivisions in
crowded areas.

• Binary Space Partitioning (BSP): Recursively divides space using planes. Particularly efficient
for indoor environments and was popularized by early first-person shooters like Doom and
Quake.

• Quadtrees: The 2D equivalent of octrees, dividing space into four quadrants recursively.
Commonly used for 2D games or for terrain in 3D games.

• Axis-Aligned Bounding Boxes (AABB) Trees: Organize objects based on their bounding boxes,

279
 creating a hierarchy that allows for efficient collision checks.

• Portal Systems: Divide the world into "rooms" connected by "portals." This approach is
particularly effective for indoor environments with distinct areas.

• Spatial Hashing: Maps 3D positions to a hash table, allowing for constant-time lookups of
nearby objects. Useful for particle systems and other scenarios with many similar-sized objects.

• Bounding Volume Hierarchies (BVH): Create a tree of nested bounding volumes, allowing for
efficient ray casting and collision detection.

[Link]. Spatial Partitioning in Popular Engines

Different game engines use different spatial partitioning systems, often combining multiple
approaches:

• Unreal Engine: Uses a combination of octrees for the overall world and BSP for detailed indoor
environments. Also uses a custom system called "Unreal Visibility Determination" that combines
portals and potentially visible sets.

• Unity: Implements a quadtree/octree hybrid system for its physics and rendering. For
navigation, it uses a navigation mesh system.

• CryEngine/CRYENGINE: Uses octrees for outdoor environments and portal systems for indoor
areas.

• Godot: Employs BVH trees for its physics engine and octrees for rendering.

• Source Engine (Valve): Famous for its Binary Space Partitioning (BSP) combined with a portal
system called "Potentially Visible Set" (PVS).

• id Tech (id Software): Early versions (Doom, Quake) pioneered BSP usage. Later versions use
combinations of BSP, octrees, and portal systems.

• Frostbite (EA): Uses a hierarchical grid system combined with octrees for its large-scale
destructible environments.

In practice, many modern engines use hybrid approaches, selecting the appropriate partitioning
system based on the specific needs of different parts of the game world.

1.5. Architectural Decisions

When designing our model system, we made several key architectural decisions:

• Node-Based Structure: We use a node-based approach where each node can have a mesh,
transformation, and children. This provides flexibility for complex scene hierarchies.

• Separation of Concerns: We separate geometric data (vertices, indices) from material

properties and transformations, allowing for more efficient memory use and easier updates.

• Animation-Ready: Our design includes dedicated structures for animations, supporting

keyframe interpolation and different animation channels (translation, rotation, scale).

• Memory Management: We use a centralized ownership model where the Model class owns all
nodes, simplifying cleanup and preventing memory leaks.

280
 • Efficient Traversal: We maintain both a hierarchical structure (nodes) and a flat list
(linearNodes) to support different traversal patterns efficiently.

1.6. How Developers Would Use the Model System

Here’s how a developer would typically use this model system in their application:

1.6.1. Loading and Initializing Models

// Create and load a model

Model* characterModel = new Model();
loadFromFile(characterModel, "[Link]");

// Find specific nodes in the model

Node* headNode = characterModel->findNode("Head");
Node* weaponAttachPoint = characterModel->findNode("RightHand");

// Attach additional objects to the model

Model* weaponModel = new Model();
loadFromFile(weaponModel, "[Link]");
weaponAttachPoint->children.push_back(weaponModel->nodes[0]);

1.6.2. Updating and Animating Models

// Play an animation
float deltaTime = 0.016f; // 16ms or ~60 FPS NB: Keep this relative to frame
instead of a constant in actual code as some systems are faster resulting in
faster animation on a constant that isn't tied to the frame time.
characterModel->updateAnimation(0, deltaTime); // Play the first animation

// Manually transform nodes

headNode->rotation = glm::rotate(headNode->rotation, glm::radians(15.0f), glm::vec3(0,
1, 0)); // Look to the side

1.6.3. Rendering Models

void renderModel(Model* model, VkCommandBuffer commandBuffer) {

// Traverse all nodes in the model
for (auto& node : model->linearNodes) {
if (node->[Link]() > 0) {
// Get the global transformation matrix
glm::mat4 nodeMatrix = node->getGlobalMatrix();

// Update uniform buffer with the node's transformation

updateUniformBuffer(nodeMatrix);

281
 // Bind the appropriate material
if (node->[Link] >= 0) {
bindMaterial(model->materials[node->[Link]]);
}

// Draw the mesh

vkCmdDrawIndexed(commandBuffer,
static_cast<uint32_t>(node->[Link]()),
1, 0, 0, 0);
}
}
}

1.7. Back to our tutorial

Now that you’ve seen how the model system API is used from a hypothetical developer’s
perspective, it’s time to implement this functionality. In the following sections, we’ll guide you
through implementing the scene graph, animation system, and model class that will power the
engine.

1.8. Implementing a Scene Graph

Now let’s look at the implementation of our scene graph structure:

// Structure for a node in the scene graph

struct Node {
Node* parent = nullptr;
std::vector<Node*> children;
Mesh mesh;
glm::mat4 matrix = glm::mat4(1.0f);

// For animation
glm::vec3 translation = glm::vec3(0.0f);
glm::quat rotation = glm::quat(1.0f, 0.0f, 0.0f, 0.0f);
glm::vec3 scale = glm::vec3(1.0f);

glm::mat4 getLocalMatrix() {
return glm::translate(glm::mat4(1.0f), translation) *
glm::toMat4(rotation) *
glm::scale(glm::mat4(1.0f), scale) *
matrix;
}

glm::mat4 getGlobalMatrix() {
glm::mat4 m = getLocalMatrix();
Node* p = parent;
while (p) {
m = p->getLocalMatrix() * m;

282
 p = p->parent;
}
return m;
}
};

1.9. Animation Structures

To support animations, we need additional structures:

// Structure for animation keyframes

struct AnimationChannel {
enum PathType { TRANSLATION, ROTATION, SCALE };
PathType path;
Node* node = nullptr;
uint32_t samplerIndex;
};

// Structure for animation interpolation

struct AnimationSampler {
enum InterpolationType { LINEAR, STEP, CUBICSPLINE };
InterpolationType interpolation;
std::vector<float> inputs; // Key frame timestamps
std::vector<glm::vec4> outputsVec4; // Key frame values (for rotations)
std::vector<glm::vec3> outputsVec3; // Key frame values (for translations and
scales)
};

// Structure for animation

struct Animation {
std::string name;
std::vector<AnimationSampler> samplers;
std::vector<AnimationChannel> channels;
float start = std::numeric_limits<float>::max();
float end = std::numeric_limits<float>::min();
float currentTime = 0.0f;
};

1.10. The Model Class

Now we can define a Model class that brings everything together:

// Structure for a model with nodes, meshes, materials, textures, and animations
struct Model {
std::vector<Node*> nodes;
std::vector<Node*> linearNodes;
std::vector<Material> materials;

283
 std::vector<Animation> animations;

~Model() {
for (auto node : linearNodes) {
delete node;
}
}

Node* findNode(const std::string& name) {

auto nodeIt = std::ranges::find_if(linearNodes, [&name](auto const& node) {
return node->name == name;
});
return (nodeIt != [Link]()) ? *nodeIt : nullptr;
}

void updateAnimation(uint32_t index, float deltaTime) {

assert(![Link]() && index < [Link]());

Animation& animation = animations[index];

[Link] += deltaTime;
if ([Link] > [Link]) {
[Link] = [Link];
}

for (auto& channel : [Link]) {

AnimationSampler& sampler = [Link][[Link]];

// Find the current key frame using binary search

auto keyFrameIt = std::ranges::lower_bound([Link],
[Link]);
if (keyFrameIt != [Link]() && keyFrameIt !=
[Link]()) {
size_t i = std::distance([Link](), keyFrameIt) - 1;
float t = ([Link] - [Link][i]) /
([Link][i + 1] - [Link][i]);

switch ([Link]) {
case AnimationChannel::TRANSLATION: {
glm::vec3 start = sampler.outputsVec3[i];
glm::vec3 end = sampler.outputsVec3[i + 1];
[Link]->translation = glm::mix(start, end, t);
break;
}
case AnimationChannel::ROTATION: {
glm::quat start = glm::quat(sampler.outputsVec4[i].w,
sampler.outputsVec4[i].x, sampler.outputsVec4[i].y, sampler.outputsVec4[i].z);
glm::quat end = glm::quat(sampler.outputsVec4[i + 1].w,
sampler.outputsVec4[i + 1].x, sampler.outputsVec4[i + 1].y, sampler.outputsVec4[i +
1].z);

284
 [Link]->rotation = glm::slerp(start, end, t);
break;
}
case AnimationChannel::SCALE: {
glm::vec3 start = sampler.outputsVec3[i];
glm::vec3 end = sampler.outputsVec3[i + 1];
[Link]->scale = glm::mix(start, end, t);
break;
}
}
break;
}
}
}
}
};

1.11. Next Steps: Loading glTF Files

Now that we’ve designed our model system’s architecture and implemented the core data
structures, the next step is to actually load 3D models from glTF files. In the next chapter, we’ll
explore how to parse glTF files using the tinygltf library and populate our scene graph with the
loaded data. We’ll learn how to extract meshes, materials, textures, and animations from glTF files
and convert them into our engine’s internal representation.

Previous: Setting Up the Project | Next: Loading a glTF Model :pp: ++

Loading Models: Understanding

glTF
1. Understanding glTF
1.1. What is glTF?
glTF (GL Transmission Format) is a standard 3D file format developed by the Khronos Group (the
same organization behind OpenGL and Vulkan). It’s often called the "JPEG of 3D" because it aims to
be a universal, efficient format for 3D content.

The main purpose of glTF is to bridge the gap between 3D content creation tools (like Blender,
Maya, 3ds Max) and real-time rendering applications like games and visualization tools. Before
glTF, developers often had to create custom exporters or use intermediate formats that weren’t
optimized for real-time rendering.

Key advantages of glTF include:

285
 • Efficiency: Optimized for loading speed and rendering performance with minimal processing

• Completeness: Contains geometry, materials, textures, animations, and scene hierarchy in a

single format

• PBR Support: Built-in support for modern physically-based rendering materials

• Standardization: Widely adopted across the industry, reducing the need for custom exporters

• Extensibility: Supports extensions for vendor-specific features while maintaining compatibility

1.2. glTF File Structure and Data Organization

A glTF file contains several key components organized in a structured way:

• Scenes and Nodes: The hierarchical structure that organizes objects in a scene graph

• Meshes: The 3D geometry data (vertices, indices, attributes like normals and UVs)

• Materials: Surface properties using a physically-based rendering (PBR) model

• Textures and Images: Visual data for materials, with support for various texture types

• Animations: Keyframe data for animating nodes (position, rotation, scale)

• Skins: Data for skeletal animations (joint hierarchies and vertex weights)

• Cameras: Perspective or orthographic camera definitions

1.2.1. The Buffer System: Efficient Binary Data Storage

One of glTF’s most powerful features is its three-level buffer system:

1. Buffers: Raw binary data blocks (like files on disk)

2. BufferViews: Views into buffers with specific offset and length

3. Accessors: Descriptions of how to interpret data in a bufferView (type, component type, count,
etc.)

This system allows different attributes (positions, normals, UVs) to share the same underlying
buffer, reducing memory usage and file size. For example:

• A single buffer might contain all vertex data

• One bufferView points to the position data within that buffer

• Another bufferView points to the normal data

• Accessors describe how to interpret each bufferView (e.g., as vec3 floats)

1.3. Using the tinygltf Library for Efficient Parsing

Rather than writing a glTF parser from scratch (which would be a significant undertaking), we’ll
use the tinygltf library:

• It’s a lightweight, header-only C++ library that’s easy to integrate

286
 • It handles both .gltf and .glb formats transparently

• It manages the complex task of parsing JSON and binary data

• It provides a clean API for accessing all glTF components

• It handles the details of the buffer system, including base64-encoded data

Using tinygltf allows us to focus on the higher-level task of converting the parsed data into our
engine’s structures rather than dealing with the low-level details of parsing JSON and binary data.

1.4. Implementing a Robust glTF Loader

When implementing a production-ready glTF loader, several considerations come into play:

• Error Handling: Robust handling of malformed files and graceful failure

• Format Detection: Supporting both .gltf and .glb formats

• Memory Management: Efficient allocation and handling of large data

• Extension Support: Handling optional glTF extensions

Let’s look at how we implement the initial file loading:

void loadModel(const std::string& modelPath) {

// Create a tinygltf loader
tinygltf::Model gltfModel;
tinygltf::TinyGLTF loader;
std::string err, warn;

// Detect file extension to determine which loader to use

bool ret = false;
std::string extension = [Link](modelPath.find_last_of(".") + 1);
std::transform([Link](), [Link](), [Link](), ::tolower);

if (extension == "glb") {
ret = [Link](&gltfModel, &err, &warn, modelPath);
} else if (extension == "gltf") {
ret = [Link](&gltfModel, &err, &warn, modelPath);
} else {
err = "Unsupported file extension: " + extension + ". Expected .gltf or .glb";
}

// Handle errors and warnings

if (![Link]()) {
std::cout << "glTF warning: " << warn << std::endl;
}
if (![Link]()) {
std::cout << "glTF error: " << err << std::endl;
}
if (!ret) {
throw std::runtime_error("Failed to load glTF model");

287
 }

// Clear existing model data

model = Model();

// Process the loaded data (covered in the following sections)

}

Supporting both .gltf and .glb formats gives artists flexibility in their workflow.

glTF comes in two formats, each with its own advantages:

• .gltf: A JSON-based format with external binary and image files

◦ Human-readable and easier to debug

◦ Allows for easier asset management (textures as separate files)

◦ Better for development workflows

• .glb: A binary format that combines everything in a single file

◦ More compact and efficient for distribution

◦ Reduces the number of file operations during loading

◦ Better for deployment and distribution

1.5. Understanding Physically Based Rendering (PBR)

Materials
This section provides a brief overview of PBR materials as they relate to glTF
loading. For a more comprehensive explanation of PBR concepts and lighting
 models, please refer to the Physically Based Rendering section in the Lighting
Materials chapter.

Materials define how surfaces look when rendered. Modern games and engines use Physically
Based Rendering (PBR), which simulates how light interacts with real-world materials based on
physical principles.

1.5.1. The Evolution of Material Systems

Material systems in 3D graphics have evolved significantly:

1. Basic Materials (1990s): Simple diffuse colors with optional specular highlights

2. Multi-Texture Materials (2000s): Multiple texture maps combined for different effects

3. Shader-Based Materials (Late 2000s): Custom shader programs for advanced effects

4. Physically Based Rendering (2010s): Materials based on physical properties of real-world

surfaces

PBR represents the current state of the art in real-time graphics. It provides more realistic results

288
across different lighting conditions and ensures consistent appearance regardless of the
environment.

1.5.2. Key PBR Material Properties

The PBR model in glTF is based on the "metallic-roughness" workflow, which uses these key
properties:

• Base Color: The albedo or diffuse color of the surface (RGB or texture)

• Metalness: How metal-like the surface is (0.0 = non-metal, 1.0 = metal)

◦ Metals have no diffuse reflection but high specular reflection

◦ Non-metals (dielectrics) have diffuse reflection and minimal specular reflection

• Roughness: How smooth or rough the surface is (0.0 = mirror-like, 1.0 = rough)

◦ Controls the microsurface detail that causes light scattering

◦ Affects the sharpness of reflections and specular highlights

• Normal Map: Adds surface detail without extra geometry

◦ Perturbs surface normals to create the illusion of additional detail

◦ More efficient than adding actual geometry

• Occlusion Map: Approximates self-shadowing within surface crevices

◦ Darkens areas that would receive less ambient light

◦ Enhances the perception of depth and detail

• Emissive: Makes the surface emit light (RGB or texture)

◦ Used for glowing objects like screens, lights, or neon signs

◦ Not affected by scene lighting

These properties can be specified as constant values or as texture maps for spatial variation across
the surface. We’ll go into details about PBR in the next few chapters.

[Bookstand with complex PBR materials - demonstrating wood] | images/[Link]

1.5.3. Texture Formats and Compression

In our engine, we use KTX2 with Basis Universal compression for textures. This approach offers
several advantages:

• Reduced File Size: Basis Universal compression significantly reduces texture sizes while
maintaining visual quality

• GPU-Ready Formats: KTX2 textures can be directly transcoded to platform-specific GPU

formats

• Cross-Platform Compatibility: Basis Universal textures work across different platforms and
graphics APIs

• Mipmap Support: KTX2 includes support for mipmaps, improving rendering quality and

289
 performance

[Link]. Embedded Textures in glTF/glb

The glTF format supports two ways to include textures:

1. External References: The .gltf file references external image files

2. Embedded Data: Images are embedded directly in the .glb file as binary data

For our engine, we use the .glb format with embedded KTX2 textures. This approach:

• Reduces the number of file operations during loading

• Ensures all textures are always available with the model

• Simplifies asset management and distribution

The glTF specification supports embedded textures through the bufferView property of image
objects. When using KTX2 textures, the mimeType is set to "image/ktx2" to indicate the format.

The texture loading process involves several complex steps that bridge the gap between glTF’s
abstract texture references and Vulkan’s low-level GPU resources.

1.6. Texture Loading: glTF Texture Iteration and

Metadata Extraction
First, we iterate through the glTF model’s texture definitions and extracting the fundamental
information needed to locate and identify each texture resource.

// First, load all textures from the model

std::vector<Texture> textures;
for (size_t i = 0; i < [Link](); i++) {
const auto& texture = [Link][i];
const auto& image = [Link][[Link]];

Texture tex;
[Link] = [Link]() ? "texture_" + std::to_string(i) : [Link];

The glTF texture system uses an indirection approach where textures reference images, and images
contain the actual pixel data or references to it. This separation allows multiple textures to share
the same image data but with different sampling parameters (like different filtering or wrapping
modes). Our iteration process builds a comprehensive inventory of all texture resources that
materials will eventually reference.

The naming strategy provides essential debugging and asset management capabilities. When artists
create textures in their 3D applications, meaningful names help developers identify which textures
serve which purposes during development. The fallback naming scheme ensures every texture has
a unique identifier even when artists haven’t provided descriptive names.

290
1.7. Texture Loading: Format Detection and Buffer
Access
Next, we need to figure out whether textures are embedded in the glTF file and identify their
format, setting up the foundation for appropriate loading strategies.

// Check if the image is embedded as KTX2

if ([Link] == "image/ktx2" && [Link] >= 0) {
// Get the buffer view that contains the KTX2 data
const auto& bufferView = [Link][[Link]];
const auto& buffer = [Link][[Link]];

// Extract the KTX2 data from the buffer

const uint8_t* ktx2Data = [Link]() + [Link];
size_t ktx2Size = [Link];

The MIME type detection ensures we’re working with KTX2 format specifically, which provides
several advantages over traditional image formats like PNG or JPEG. KTX2 is designed specifically
for GPU textures and supports advanced features like basis universal compression, multiple
mipmap levels, and direct GPU format compatibility. The bufferView check confirms that the image
data is embedded within the glTF file rather than referenced externally.

The buffer access pattern demonstrates glTF’s sophisticated data organization system. Rather than
copying data unnecessarily, we obtain direct pointers to the KTX2 data within the loaded glTF
buffer. This approach minimizes memory usage and avoids expensive copy operations, which is
particularly important when dealing with large texture datasets that can easily consume hundreds
of megabytes.

1.8. Texture Loading: KTX2 Parsing and Validation

Now we need to load the KTX2 texture data using the specialized KTX-Software library and perform
initial validation to ensure the texture data is usable.

// Load the KTX2 texture using KTX-Software library

ktxTexture2* ktxTexture = nullptr;
KTX_error_code result = ktxTexture2_CreateFromMemory(
ktx2Data, ktx2Size,
KTX_TEXTURE_CREATE_LOAD_IMAGE_DATA_BIT,
&ktxTexture
);

if (result != KTX_SUCCESS) {
std::cerr << "Failed to load KTX2 texture: " << ktxErrorString(result) <<
std::endl;
continue;
}

291
The KTX-Software library provides robust parsing of the complex KTX2 format, handling details
like multiple mipmap levels, various pixel formats, and metadata that would be extremely complex
to implement correctly from scratch. The KTX_TEXTURE_CREATE_LOAD_IMAGE_DATA_BIT flag instructs the
library to immediately load the actual pixel data into memory, preparing it for subsequent
processing steps.

Error handling at this stage is crucial because texture files can become corrupted during asset
pipeline processing or file transfer. By continuing with the next texture when one fails to load, we
ensure that a single problematic texture doesn’t prevent the entire model from loading. This
graceful degradation approach is essential for robust production systems where content issues
shouldn’t crash the application.

1.9. Texture Loading: Basis Universal Transcoding

Next, we handle the transcoding process that converts Basis Universal compressed textures into
GPU-native formats for optimal runtime performance.

// If the texture uses Basis Universal compression, transcode it to a GPU-

friendly format
if (ktxTexture->isCompressed && ktxTexture2_NeedsTranscoding(ktxTexture)) {
// Choose the appropriate format based on GPU capabilities
ktx_transcode_fmt_e transcodeFmt = KTX_TTF_BC7_RGBA;

// For devices that don't support BC7, use alternatives

// if (!deviceSupportsBC7) {
// transcodeFmt = KTX_TTF_ASTC_4x4_RGBA;
// }
// if (!deviceSupportsASTC) {
// transcodeFmt = KTX_TTF_ETC2_RGBA;
// }

// Transcode the texture

result = ktxTexture2_TranscodeBasis(ktxTexture, transcodeFmt, 0);
if (result != KTX_SUCCESS) {
std::cerr << "Failed to transcode KTX2 texture: " <<
ktxErrorString(result) << std::endl;
ktxTexture2_Destroy(ktxTexture);
continue;
}
}

Basis Universal represents a revolutionary approach to texture compression that solves a

fundamental problem in cross-platform development: different GPUs support different texture
compression formats. Traditional approaches required storing multiple texture versions for
different platforms, dramatically increasing storage requirements. Basis Universal stores textures
in an intermediate format that can be quickly transcoded to any GPU-native format at load time.

The format selection logic (shown in commented form) demonstrates how production systems

292
handle GPU capability differences. Desktop GPUs typically support BC7 compression which
provides excellent quality, while mobile GPUs often use ASTC or ETC2 formats. The transcoding
process happens at runtime based on the actual capabilities of the target GPU, ensuring optimal
performance and quality on every platform.

The transcoding operation itself is computationally intensive but happens only once during asset
loading. The resulting GPU-native format provides significantly better performance during
rendering compared to uncompressed textures, making the upfront transcoding cost worthwhile.
Failed transcoding attempts trigger cleanup of partially processed resources, preventing memory
leaks in error conditions.

1.10. Texture Loading: Vulkan Resource Creation and

GPU Upload
Finally, create the Vulkan resources needed for GPU rendering and uploads the processed texture
data to video memory.

// Create Vulkan image, memory, and view

vk::Format format =
static_cast<vk::Format>(ktxTexture2_GetVkFormat(ktxTexture));
vk::Extent3D extent{
static_cast<uint32_t>(ktxTexture->baseWidth),
static_cast<uint32_t>(ktxTexture->baseHeight),
static_cast<uint32_t>(ktxTexture->baseDepth)
};
uint32_t mipLevels = ktxTexture->numLevels;

// Create the Vulkan image

vk::ImageCreateInfo imageCreateInfo{
.imageType = vk::ImageType::e2D,
.format = format,
.extent = extent,
.mipLevels = mipLevels,
.arrayLayers = 1,
.samples = vk::SampleCountFlagBits::e1,
.tiling = vk::ImageTiling::eOptimal,
.usage = vk::ImageUsageFlagBits::eSampled |
vk::ImageUsageFlagBits::eTransferDst,
.sharingMode = vk::SharingMode::eExclusive,
.initialLayout = vk::ImageLayout::eUndefined
};

// Create the image, allocate memory, and bind them

// ... (code omitted for brevity)

// Upload the texture data to the image

ktxTexture2_VkUploadEx(ktxTexture, &ktxVulkanTexture, &vkDevice, &vkQueue,
&ktxVulkanDeviceMemory, &ktxVulkanImage,
&ktxVulkanImageView, &ktxVulkanImageLayout,

293
 &ktxVulkanImageMemory);

// Store the Vulkan resources in our texture object

[Link] = ktxVulkanImage;
[Link] = ktxVulkanImageView;
[Link] = ktxVulkanImageMemory;

// Clean up KTX resources

ktxTexture2_Destroy(ktxTexture);
} else {
// Handle other image formats or external references
// ... (code omitted for brevity)
}

// Create a sampler for the texture

VkSamplerCreateInfo samplerInfo = {};
// ... (code omitted for brevity)

textures.push_back(tex);
}

// Now load materials and associate them with textures

for (const auto& material : [Link]) {
Material mat;

// Base color
if ([Link]() == 4) {
[Link].r = [Link][0];
[Link].g = [Link][1];
[Link].b = [Link][2];
[Link].a = [Link][3];
}

// Metallic and roughness factors

[Link] = [Link];
[Link] = [Link];

// Associate textures with the material

if ([Link] >= 0) {
const auto& texture =
[Link][[Link]];
[Link] = &textures[[Link]];
}

if ([Link] >= 0) {
const auto& texture =
[Link][[Link]];
[Link] = &textures[[Link]];
}

if ([Link] >= 0) {

294
 const auto& texture = [Link][[Link]];
[Link] = &textures[[Link]];
}

if ([Link] >= 0) {
const auto& texture = [Link][[Link]];
[Link] = &textures[[Link]];
}

if ([Link] >= 0) {
const auto& texture = [Link][[Link]];
[Link] = &textures[[Link]];
}

[Link].push_back(mat);
}

Now, let’s talk about how this all fits together.

1.11. Understanding Scene Graphs and Hierarchical

Transformations
A scene graph is a hierarchical tree-like data structure that organizes the spatial representation of a
3D scene. It’s a fundamental concept in computer graphics and game engines, serving as the
backbone for organizing complex scenes.

1.11.1. Why Scene Graphs Matter

Scene graphs offer several critical advantages over flat collections of objects:

• Hierarchical Transformations: Children inherit transformations from their parents, making it

natural to model complex relationships

• Spatial Organization: Objects are organized based on their logical relationships, making scene
management easier

• Animation Support: Hierarchical structures are crucial for skeletal animations and complex
movement patterns

• Efficient Traversal: Enables optimized rendering, culling, and picking operations

• Instancing Support: The same object can appear multiple times with different transformations

Consider these practical examples:

1. Character with Equipment: When a character moves, all attached equipment (weapons,
armor) should move with it. With a scene graph, you move the character node, and all child
nodes automatically inherit the transformation.

2. Vehicle with Moving Parts: A vehicle might have wheels that rotate independently while the
whole vehicle moves. A scene graph makes this hierarchy of movements natural to express.

295
3. Articulated Animations: Characters with skeletons need joints that move relative to their
parent joints. A scene graph directly models this parent-child relationship.

1.11.2. Transformations in Scene Graphs

One of the most powerful aspects of scene graphs is how they handle transformations:

• Each node has a local transformation relative to its parent

• The global transformation is calculated by combining the node’s local transformation with its
parent’s global transformation

• This allows for intuitive modeling of complex hierarchical movements

The transformation pipeline typically works like this:

1. Each node stores its local transformation (translation, rotation, scale)

2. When rendering, we calculate the global transformation by multiplying with parent

transformations

3. This global transformation is used to position the object in world space

Here’s how we build a scene graph from glTF data:

// First pass: create all nodes

for (size_t i = 0; i < [Link](); i++) {
const auto& node = [Link][i];
[Link][i] = new Node();
[Link][i]->index = static_cast<uint32_t>(i);
[Link][i]->name = [Link];

// Get transformation data

if ([Link]() == 3) {
[Link][i]->translation = glm::vec3(
[Link][0], [Link][1], [Link][2]
);
}
// ... handle rotation and scale
}

// Second pass: establish parent-child relationships

for (size_t i = 0; i < [Link](); i++) {
const auto& node = [Link][i];
for (int childIdx : [Link]) {
[Link][childIdx]->parent = [Link][i];
[Link][i]->children.push_back([Link][childIdx]);
}
}

We use a two-pass approach to ensure all nodes exist before we try to link them together.

296
1.12. Understanding 3D Geometry and Mesh Data
3D models are represented as meshes - collections of vertices, edges, and faces that define the shape
of an object. Understanding how this data is structured is crucial for efficient rendering.

1.12.1. The Building Blocks of 3D Models

The fundamental components of 3D geometry are:

• Vertices: Points in 3D space that define the shape

• Indices: References to vertices that define how they connect to form triangles

• Attributes: Additional data associated with vertices:

◦ Positions: 3D coordinates (x, y, z)

◦ Normals: Direction vectors perpendicular to the surface (for lighting calculations)

◦ Texture Coordinates (UVs): 2D coordinates for mapping textures onto the surface

◦ Tangents and Bitangents: Vectors used for normal mapping

◦ Colors: Per-vertex color data

◦ Skinning Weights and Indices: For skeletal animations

Modern 3D graphics use triangle meshes because:

• Triangles are always planar (three points define a plane)

• Triangles are the simplest polygon that can represent any surface

• Graphics hardware is optimized for triangle processing

1.12.2. Mesh Organization in glTF

glTF organizes mesh data in a way that’s efficient for both storage and rendering:

• Meshes: Collections of primitives that form a logical object

• Primitives: Individual parts of a mesh, each with its own material

• Attributes: Vertex data like positions, normals, and texture coordinates

• Indices: References to vertices that define triangles

This organization allows for:

• Efficient memory use through data sharing

• Material variation within a single mesh

• Optimized rendering through batching

Here’s how we extract mesh data:

// Load meshes

297
 for (size_t i = 0; i < [Link](); i++) {
const auto& node = [Link][i];
if ([Link] >= 0) {
const auto& mesh = [Link][[Link]];

// Process each primitive

for (const auto& primitive : [Link]) {
Mesh newMesh;

// Set material
if ([Link] >= 0) {
[Link] = [Link];
}

// Extract vertex positions, normals, and texture coordinates

// ... (code omitted for brevity)

// Extract indices that define triangles

// ... (code omitted for brevity)

// Assign the mesh to the node

[Link][i]->mesh = newMesh;
}
}
}

1.13. Understanding Animation Systems

Animation is what transforms static 3D models into living, breathing entities in our virtual worlds.
A robust animation system is essential for creating engaging and dynamic 3D applications.

1.13.1. Animation Techniques in 3D Graphics

Several animation techniques are commonly used in 3D graphics:

• Keyframe Animation: Defining specific poses at specific times, with interpolation between
them

• Skeletal Animation: Using a hierarchy of bones to deform a mesh

• Morph Target Animation: Interpolating between predefined mesh shapes

• Procedural Animation: Generating animation through algorithms and physics

• Particle Systems: Animating many small elements with simple rules

Modern games typically use a combination of these techniques, with skeletal animation forming the
backbone of character movement.

298
1.13.2. Core Animation Concepts

Several key concepts are fundamental to understanding animation systems:

• Keyframes: Specific points in time where animation values are explicitly defined

• Interpolation: Calculating values between keyframes to create smooth motion

• Channels: Targeting specific properties (like position or rotation) for animation

• Blending: Combining multiple animations with different weights

• Retargeting: Applying animations created for one model to another

1.13.3. The glTF Animation System

glTF uses a flexible animation system that can represent various animation techniques:

• Animations: Collections of channels and samplers

• Channels: Links between samplers and node properties (translation, rotation, scale)

• Samplers: Keyframe data with timestamps, values, and interpolation methods

• Targets: The properties being animated (translation, rotation, scale, or weights for morph
targets)

glTF supports three interpolation methods:

• LINEAR: Smooth transitions with constant velocity

• STEP: Sudden changes with no interpolation

• CUBICSPLINE: Smooth curves with control points for acceleration and deceleration

This system allows for complex animations that can target specific parts of a model independently,
enabling actions like walking, facial expressions, and complex interactions.

Here’s how we load animation data:

// Load animations
for (const auto& anim : [Link]) {
Animation animation;
[Link] = [Link];

// Load keyframe data

for (const auto& sampler : [Link]) {
AnimationSampler animSampler{};

// Set interpolation type (LINEAR, STEP, or CUBICSPLINE)

// ... (code omitted for brevity)

// Extract keyframe times and values

// ... (code omitted for brevity)

[Link].push_back(animSampler);

299
 }

// Connect samplers to node properties

for (const auto& channel : [Link]) {
AnimationChannel animChannel{};

// Set target node and property (translation, rotation, or scale)

// ... (code omitted for brevity)

[Link].push_back(animChannel);
}

[Link].push_back(animation);
}

1.14. Integration with the Rendering Pipeline

Now that we’ve loaded our model data, let’s discuss how it integrates with the rest of our rendering
pipeline.

1.14.1. From Asset Loading to Rendering

The journey from a glTF file to pixels on the screen involves several stages:

1. Asset Loading: The glTF loader populates our Model, Node, Mesh, and Material structures

2. Scene Management: The engine maintains a collection of loaded models in the scene

3. Update Loop: Each frame, animations are updated based on elapsed time

4. Culling: The engine determines which objects are potentially visible

5. Rendering: The scene graph is traversed, and each visible mesh is rendered with its material

This pipeline allows for efficient rendering of complex scenes with animated models.

1.14.2. Rendering Optimizations

Several optimizations can improve the performance of model rendering:

• Batching: Group similar objects to reduce draw calls

• Instancing: Render multiple instances of the same mesh with different transforms

• Level of Detail (LOD): Use simpler versions of models at greater distances

• Frustum Culling: Skip rendering objects outside the camera’s view

• Occlusion Culling: Skip rendering objects hidden behind other objects

1.14.3. Memory Management Considerations

When loading models, especially large ones, memory management becomes crucial:

300
 • Vertex Data: Store in GPU buffers for efficient rendering

• Indices: Use 16-bit indices when possible to save memory

• Textures: Use KTX2 with Basis Universal compression to significantly reduce memory usage

• Instancing: Reuse the same model data for multiple instances with different transforms

[Link]. Efficient Texture Memory Management with KTX2 and Basis Universal

Textures often consume the majority of GPU memory in 3D applications. KTX2 with Basis Universal
compression provides several memory optimization benefits:

• Supercompression: Basis Universal can reduce texture size by 4-10x compared to

uncompressed formats

• GPU-Native Formats: Textures are transcoded to formats that GPUs can directly sample from,
avoiding runtime decompression

• Mipmaps: KTX2 supports mipmaps, which not only improve visual quality but also reduce
memory usage for distant objects

• Format Selection: The transcoder can choose the optimal format based on the target GPU’s
capabilities:

◦ BC7 for desktop GPUs (NVIDIA, AMD, Intel)

◦ ASTC for mobile GPUs (ARM, Qualcomm)

◦ ETC2 for older mobile GPUs

[Link]. Integration with Vulkan Rendering Pipeline

To efficiently integrate KTX2 textures with Vulkan:

1. Descriptor Sets: Create descriptor sets that bind texture image views and samplers to shader
binding points

2. Pipeline Layout: Define a pipeline layout that includes these descriptor sets

3. Shader Access: In shaders, access textures using the appropriate binding points

Here’s a simplified example of setting up descriptor sets for PBR textures:

// Create descriptor set layout for PBR textures

std::array<vk::DescriptorSetLayoutBinding, 5> bindings{
// Base color texture
vk::DescriptorSetLayoutBinding{
.binding = 0,
.descriptorType = vk::DescriptorType::eCombinedImageSampler,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eFragment
},
// Metallic-roughness texture
vk::DescriptorSetLayoutBinding{
.binding = 1,

301
 .descriptorType = vk::DescriptorType::eCombinedImageSampler,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eFragment
},
// Normal map
vk::DescriptorSetLayoutBinding{
.binding = 2,
.descriptorType = vk::DescriptorType::eCombinedImageSampler,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eFragment
},
// Occlusion map
vk::DescriptorSetLayoutBinding{
.binding = 3,
.descriptorType = vk::DescriptorType::eCombinedImageSampler,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eFragment
},
// Emissive map
vk::DescriptorSetLayoutBinding{
.binding = 4,
.descriptorType = vk::DescriptorType::eCombinedImageSampler,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eFragment
}
};

vk::DescriptorSetLayoutCreateInfo layoutInfo{
.bindingCount = static_cast<uint32_t>([Link]()),
.pBindings = [Link]()
};

vk::raii::DescriptorSetLayout descriptorSetLayout(device, layoutInfo);

// For each material, create a descriptor set and update it with the material's
textures
for (const auto& material : [Link]) {
// Allocate descriptor set from the descriptor pool
vk::DescriptorSetAllocateInfo allocInfo{
.descriptorPool = descriptorPool,
.descriptorSetCount = 1,
.pSetLayouts = &*descriptorSetLayout
};

vk::raii::DescriptorSet descriptorSet = std::move(vk::raii::DescriptorSets(device,

allocInfo).front());

// Update descriptor set with texture image views and samplers

std::vector<vk::WriteDescriptorSet> descriptorWrites;

if ([Link]) {

302
 vk::DescriptorImageInfo imageInfo{
.sampler = [Link]->sampler,
.imageView = [Link]->imageView,
.imageLayout = vk::ImageLayout::eShaderReadOnlyOptimal
};

vk::WriteDescriptorSet write{
.dstSet = *descriptorSet,
.dstBinding = 0,
.dstArrayElement = 0,
.descriptorCount = 1,
.descriptorType = vk::DescriptorType::eCombinedImageSampler,
.pImageInfo = &imageInfo
};

descriptorWrites.push_back(write);
}

// Similar writes for other textures

// ...

[Link](descriptorWrites, {});

// Store the descriptor set with the material for later use during rendering
[Link] = *descriptorSet;
}

[Link]. Best Practices for Texture Memory Management

To optimize texture memory usage:

1. Texture Atlasing: Combine multiple small textures into a single larger texture to reduce state
changes

2. Mipmap Management: Generate and use mipmaps for all textures to improve performance
and quality

3. Texture Streaming: For very large scenes, implement texture streaming to load higher
resolution textures only when needed

4. Memory Budgeting: Implement a texture budget system that can reduce texture quality when
memory is constrained

5. Format Selection: Choose the appropriate format based on the texture content:

◦ BC7/ASTC for color textures with alpha

◦ BC1/ETC1 for color textures without alpha

◦ BC5/ETC2 for normal maps

◦ BC4/EAC for single-channel textures (roughness, metallic, etc.)

303
1.15. Summary and Next Steps
In this chapter, we’ve explored the process of loading 3D models from glTF files and organizing
them into a scene graph. We’ve covered:

• The structure and advantages of the glTF format

• How to use the tinygltf library for efficient parsing

• The physically-based material system used in modern rendering

• How scene graphs organize objects in a hierarchical structure

• The representation of 3D geometry in meshes

• Animation systems for bringing models to life

• Integration with the rendering pipeline

Our glTF loader creates a complete scene graph with:

• Nodes organized in a hierarchy

• Meshes attached to nodes

• Materials defining surface properties

• Animations that can change node properties over time

This structure allows us to:

• Render complex 3D scenes

• Animate characters and objects

• Apply transformations that propagate through the hierarchy

• Optimize rendering for performance

In the next chapter, we’ll explore how to render these models using physically-based rendering
techniques, bringing our loaded assets to life with realistic lighting and materials.

Previous: Implementing the Model Loading System | Next: Implementing PBR Rendering :pp: ++

Loading Models: Implementing PBR

for glTF Models
1. Applying PBR to glTF Models
1.1. Building on PBR Knowledge
In the Lighting & Materials chapter, we explored the fundamentals of Physically Based Rendering
(PBR), including its core principles, the BRDF, and material properties. Now, we’ll apply that

304
knowledge to implement a PBR pipeline for the glTF models we’ve loaded.

As we learned in the glTF and KTX2 Migration chapter, glTF uses PBR with the metallic-roughness
workflow for its material system. This aligns perfectly with the PBR concepts we’ve already
covered, making it straightforward to render our glTF models with physically accurate lighting.

1.2. Leveraging glTF’s PBR Materials

The glTF format already includes all the material properties we need for PBR:

• Base Color: Defined by the baseColorFactor and baseColorTexture

• Metallic and Roughness: Defined by metallicFactor, roughnessFactor, and

metallicRoughnessTexture

• Normal Maps: For surface detail without additional geometry

• Occlusion Maps: For approximating ambient occlusion

• Emissive Maps: For self-illuminating parts of the material

By using these properties directly, we can ensure our rendering matches the artist’s intent and
produces physically accurate results.

1.3. Implementing PBR in Our Engine

Now that we understand the theory behind PBR, let’s implement it in our engine. We’ll build on the
material data we loaded from glTF files in the previous chapter.

1.3.1. Uniform Buffer for PBR

We need to extend our uniform buffer to include PBR parameters:

// Structure for uniform buffer object

struct UniformBufferObject {
alignas(16) glm::mat4 model;
alignas(16) glm::mat4 view;
alignas(16) glm::mat4 proj;

// PBR parameters
alignas(16) glm::vec4 lightPositions[4]; // Position and radius
alignas(16) glm::vec4 lightColors[4]; // RGB color and intensity
alignas(16) glm::vec4 camPos; // Camera position for view-dependent
effects
alignas(4) float exposure = 4.5f; // Exposure for HDR rendering
alignas(4) float gamma = 2.2f; // Gamma correction value
alignas(4) float prefilteredCubeMipLevels = 1.0f; // For image-based lighting
alignas(4) float scaleIBLAmbient = 1.0f; // Scale factor for ambient lighting
};

305
This uniform buffer includes:

1. Standard Transformation Matrices: Model, view, and projection matrices for vertex
transformation

2. Light Information: Positions and colors of up to four light sources

3. Camera Position: Needed for view-dependent effects like Fresnel

4. Rendering Parameters: Exposure, gamma, and other values for post-processing

5. Image-Based Lighting Parameters: For environment reflections (we’ll cover this in a later
chapter)

1.3.2. Push Constants for Materials

We introduced push constants earlier in push constants; here we focus on how the
 same mechanism carries glTF metallic‑roughness material knobs efficiently per
draw.

We’ll use push constants to pass material properties to the shader:

// Structure for push constants

struct PushConstantBlock {
glm::vec4 baseColorFactor; // RGB base color and alpha
float metallicFactor; // How metallic the surface is
float roughnessFactor; // How rough the surface is
int baseColorTextureSet; // Texture coordinate set for base color
int physicalDescriptorTextureSet; // Texture coordinate set for metallic-
roughness
int normalTextureSet; // Texture coordinate set for normal map
int occlusionTextureSet; // Texture coordinate set for occlusion
int emissiveTextureSet; // Texture coordinate set for emission
float alphaMask; // Whether to use alpha masking
float alphaMaskCutoff; // Alpha threshold for masking
};

Push constants are ideal for material properties because:

• They can be updated quickly between draw calls

• They don’t require descriptor sets

• They’re perfect for per-object data like material properties

1.3.3. Setting Up the Descriptor Sets

To implement PBR, we need to set up descriptor sets for our textures and uniform buffer:

// Create descriptor set layout

void createDescriptorSetLayout() {
// Binding for uniform buffer

306
 vk::DescriptorSetLayoutBinding uboBinding{
.binding = 0,
.descriptorType = vk::DescriptorType::eUniformBuffer,
.descriptorCount = 1,
.stageFlags = vk::ShaderStageFlagBits::eVertex |
vk::ShaderStageFlagBits::eFragment
};

// Bindings for textures

std::array<vk::DescriptorSetLayoutBinding, 5> textureBindings{};

// Base color texture

textureBindings[0].binding = 1;
textureBindings[0].descriptorType = vk::DescriptorType::eCombinedImageSampler;
textureBindings[0].descriptorCount = 1;
textureBindings[0].stageFlags = vk::ShaderStageFlagBits::eFragment;

// Metallic-roughness texture
textureBindings[1].binding = 2;
textureBindings[1].descriptorType = vk::DescriptorType::eCombinedImageSampler;
textureBindings[1].descriptorCount = 1;
textureBindings[1].stageFlags = vk::ShaderStageFlagBits::eFragment;

// Normal map
textureBindings[2].binding = 3;
textureBindings[2].descriptorType = vk::DescriptorType::eCombinedImageSampler;
textureBindings[2].descriptorCount = 1;
textureBindings[2].stageFlags = vk::ShaderStageFlagBits::eFragment;

// Occlusion map
textureBindings[3].binding = 4;
textureBindings[3].descriptorType = vk::DescriptorType::eCombinedImageSampler;
textureBindings[3].descriptorCount = 1;
textureBindings[3].stageFlags = vk::ShaderStageFlagBits::eFragment;

// Emissive map
textureBindings[4].binding = 5;
textureBindings[4].descriptorType = vk::DescriptorType::eCombinedImageSampler;
textureBindings[4].descriptorCount = 1;
textureBindings[4].stageFlags = vk::ShaderStageFlagBits::eFragment;

// Combine all bindings

std::array<vk::DescriptorSetLayoutBinding, 6> bindings = {
uboBinding,
textureBindings[0],
textureBindings[1],
textureBindings[2],
textureBindings[3],
textureBindings[4]
};

307
 // Create the descriptor set layout
vk::DescriptorSetLayoutCreateInfo layoutInfo{
.bindingCount = static_cast<uint32_t>([Link]()),
.pBindings = [Link]()
};

descriptorSetLayout = vk::raii::DescriptorSetLayout(device, layoutInfo);

}

1.3.4. Setting Up the Pipeline

Our PBR pipeline needs to be configured for the specific requirements of physically-based
rendering:

void createPipeline() {
// ... (standard pipeline setup code)

// Enable alpha blending

vk::PipelineColorBlendAttachmentState colorBlendAttachment{
.blendEnable = vk::True,
.srcColorBlendFactor = vk::BlendFactor::eSrcAlpha,
.dstColorBlendFactor = vk::BlendFactor::eOneMinusSrcAlpha,
.colorBlendOp = vk::BlendOp::eAdd,
.srcAlphaBlendFactor = vk::BlendFactor::eOne,
.dstAlphaBlendFactor = vk::BlendFactor::eZero,
.alphaBlendOp = vk::BlendOp::eAdd,
.colorWriteMask =
vk::ColorComponentFlagBits::eR |
vk::ColorComponentFlagBits::eG |
vk::ColorComponentFlagBits::eB |
vk::ColorComponentFlagBits::eA
};

// Set up push constants for material properties

vk::PushConstantRange pushConstantRange{
.stageFlags = vk::ShaderStageFlagBits::eFragment,
.offset = 0,
.size = sizeof(PushConstantBlock)
};

// Create the pipeline layout

vk::PipelineLayoutCreateInfo pipelineLayoutInfo{
.setLayoutCount = 1,
.pSetLayouts = &descriptorSetLayout,
.pushConstantRangeCount = 1,
.pPushConstantRanges = &pushConstantRange
};

pipelineLayout = vk::raii::PipelineLayout(device, pipelineLayoutInfo);

308
 // ... (rest of pipeline creation)
}

1.4. PBR Shader Implementation

The heart of our PBR implementation is in the fragment shader. Here’s a simplified version of a PBR
fragment shader written in Slang:

// Input from vertex shader

struct VSOutput {
float3 WorldPos : POSITION; // Automatically assigned to location 0
float3 Normal : NORMAL; // Automatically assigned to location 1
float2 UV : TEXCOORD0; // Automatically assigned to location 2
float4 Tangent : TANGENT; // Automatically assigned to location 3
};

// Uniform buffer
struct UniformBufferObject {
float4x4 model;
float4x4 view;
float4x4 proj;
float4 lightPositions[4];
float4 lightColors[4];
float4 camPos;
float exposure;
float gamma;
float prefilteredCubeMipLevels;
float scaleIBLAmbient;
};

// Push constants for material properties

struct PushConstants {
float4 baseColorFactor;
float metallicFactor;
float roughnessFactor;
int baseColorTextureSet;
int physicalDescriptorTextureSet;
int normalTextureSet;
int occlusionTextureSet;
int emissiveTextureSet;
float alphaMask;
float alphaMaskCutoff;
};

// Constants
static const float PI = 3.14159265359;

// Bindings

309
 ConstantBuffer<UniformBufferObject> ubo;
Texture2D baseColorMap;
SamplerState baseColorSampler;
Texture2D metallicRoughnessMap;
SamplerState metallicRoughnessSampler;
Texture2D normalMap;
SamplerState normalSampler;
Texture2D occlusionMap;
SamplerState occlusionSampler;
Texture2D emissiveMap;
SamplerState emissiveSampler;

[[vk::push_constant]] PushConstants material;

// PBR functions
float DistributionGGX(float NdotH, float roughness) {
float a = roughness * roughness;
float a2 = a * a;
float NdotH2 = NdotH * NdotH;

float nom = a2;

float denom = (NdotH2 * (a2 - 1.0) + 1.0);
denom = PI * denom * denom;

return nom / denom;

}

float GeometrySmith(float NdotV, float NdotL, float roughness) {

float r = roughness + 1.0;
float k = (r * r) / 8.0;

float ggx1 = NdotV / (NdotV * (1.0 - k) + k);

float ggx2 = NdotL / (NdotL * (1.0 - k) + k);

return ggx1 * ggx2;

}

float3 FresnelSchlick(float cosTheta, float3 F0) {

return F0 + (1.0 - F0) * pow(1.0 - cosTheta, 5.0);
}

// Main fragment shader function

float4 main(VSOutput input) : SV_TARGET
{
// Sample material textures
float4 baseColor = [Link](baseColorSampler, [Link]) *
[Link];
float2 metallicRoughness = [Link](metallicRoughnessSampler,
[Link]).bg;
float metallic = metallicRoughness.x * [Link];
float roughness = metallicRoughness.y * [Link];

310
 float ao = [Link](occlusionSampler, [Link]).r; //
link:[Link] occlusion]
float3 emissive = [Link](emissiveSampler, [Link]).rgb; //
link:[Link] lighting] (self-illumination)

// Calculate normal in link:[Link]

Mapping[tangent space]
float3 N = normalize([Link]);
if ([Link] >= 0) {
// Apply link:[Link]
mapping]
float3 tangentNormal = [Link](normalSampler, [Link]).xyz * 2.0 -
1.0;
float3 T = normalize([Link]);
float3 B = normalize(cross(N, T)) * [Link].w;
float3x3 TBN = float3x3(T, B, N);
N = normalize(mul(tangentNormal, TBN));
}

// Calculate view and reflection vectors

float3 V = normalize([Link] - [Link]);
float3 R = reflect(-V, N);

// Calculate F0 (base reflectivity)

float3 F0 = float3(0.04, 0.04, 0.04);
F0 = lerp(F0, [Link], metallic);

// Initialize lighting
float3 Lo = float3(0.0, 0.0, 0.0);

// Calculate lighting for each light

for (int i = 0; i < 4; i++) {
float3 lightPos = [Link][i].xyz;
float3 lightColor = [Link][i].rgb;

// Calculate light direction and distance

float3 L = normalize(lightPos - [Link]);
float distance = length(lightPos - [Link]);
float attenuation = 1.0 / (distance * distance);
float3 radiance = lightColor * attenuation;

// Calculate half vector (the normalized vector halfway between view and light
direction)
// Used in
link:[Link]
and PBR models
float3 H = normalize(V + L);

// Calculate BRDF terms

float NdotL = max(dot(N, L), 0.0);
float NdotV = max(dot(N, V), 0.0);

311
 float NdotH = max(dot(N, H), 0.0);
float HdotV = max(dot(H, V), 0.0);

// Specular BRDF
float D = DistributionGGX(NdotH, roughness);
float G = GeometrySmith(NdotV, NdotL, roughness);
float3 F = FresnelSchlick(HdotV, F0);

float3 numerator = D * G * F;
float denominator = 4.0 * NdotV * NdotL + 0.0001;
float3 specular = numerator / denominator;

// link:[Link] conservation]
float3 kS = F;
float3 kD = float3(1.0, 1.0, 1.0) - kS;
kD *= 1.0 - metallic;

// Add to outgoing radiance

Lo += (kD * [Link] / PI + specular) * radiance * NdotL;
}

// Add ambient and emissive

float3 ambient = float3(0.03, 0.03, 0.03) * [Link] * ao;
float3 color = ambient + Lo + emissive;

// link:[Link]
link:[Link] and
link:[Link] correction]
color = color / (color + float3(1.0, 1.0, 1.0));
color = pow(color, float3(1.0 / [Link], 1.0 / [Link], 1.0 / [Link]));

return float4(color, baseColor.a);

}

This shader implements the core PBR lighting model, including:

• Sampling material textures

• Calculating normal mapping

• Computing the specular BRDF with D, F, and G terms

• Applying energy conservation

• Handling multiple light sources

• Tone mapping and gamma correction

1.4.1. Lighting Setup for PBR

PBR requires careful setup of light sources to achieve realistic results. Here’s how we can set up
lights in our application:

312
 void setupLights() {
// Set up four lights with different positions and colors
std::array<glm::vec4, 4> lightPositions = {
glm::vec4(-10.0f, 10.0f, 10.0f, 1.0f),
glm::vec4(10.0f, 10.0f, 10.0f, 1.0f),
glm::vec4(-10.0f, -10.0f, 10.0f, 1.0f),
glm::vec4(10.0f, -10.0f, 10.0f, 1.0f)
};

std::array<glm::vec4, 4> lightColors = {

glm::vec4(300.0f, 300.0f, 300.0f, 1.0f), // White
glm::vec4(300.0f, 300.0f, 0.0f, 1.0f), // Yellow
glm::vec4(0.0f, 0.0f, 300.0f, 1.0f), // Blue
glm::vec4(300.0f, 0.0f, 0.0f, 1.0f) // Red
};

// Update uniform buffer with light data

for (size_t i = 0; i < maxConcurrentFrames; i++) {
UniformBufferObject ubo{};
// ... (set up transformation matrices)

// Set light positions and colors

for (int j = 0; j < 4; j++) {
[Link][j] = lightPositions[j];
[Link][j] = lightColors[j];
}

// Set camera position for view-dependent effects

[Link] = glm::vec4([Link](), 1.0f);

// Set other PBR parameters

[Link] = 4.5f;
[Link] = 2.2f;

// Copy to uniform buffer (per frame-in-flight)

memcpy(uniformBuffers[i].mapped, &ubo, sizeof(ubo));
}
}

1.4.2. Camera Integration for PBR

PBR relies on view-dependent effects like the Fresnel effect, so we need to integrate our camera
system:

void updateUniformBuffer(uint32_t currentFrame) {

UniformBufferObject ubo{};

// Update transformation matrices

[Link] = glm::mat4(1.0f); // Or get from the model's node

313
 [Link] = [Link]();
[Link] = [Link]([Link] /
(float)[Link]);

// Vulkan's Y coordinate is inverted compared to OpenGL

[Link][1][1] *= -1;

// Update camera position for PBR calculations

[Link] = glm::vec4([Link](), 1.0f);

// ... (update other PBR parameters)

// Copy to uniform buffer (per frame-in-flight)

memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));
}

1.5. Rendering with PBR

Finally, let’s put it all together to render our models with PBR:

void drawModel(vk::raii::CommandBuffer& commandBuffer, Model* model) {

// Bind descriptor set with uniform buffer and textures
[Link](
vk::PipelineBindPoint::eGraphics,
pipelineLayout,
0,
1,
&descriptorSets[currentFrame],
0,
nullptr
);

// Traverse the model's scene graph

for (auto& node : model->linearNodes) {
if (node->[Link]() > 0) {
// Get the global transformation matrix
glm::mat4 nodeMatrix = node->getGlobalMatrix();

// Update model matrix in uniform buffer

// (In a real implementation, we'd use a separate UBO for each model)

// Set up push constants for material properties

if (node->[Link] >= 0) {
Material& mat = model->materials[node->[Link]];

PushConstantBlock pushConstants{};
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];

314
 [Link] = [Link];
[Link] =
[Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];

[Link](
pipelineLayout,
vk::ShaderStageFlagBits::eFragment,
0,
sizeof(PushConstantBlock),
&pushConstants
);
}

// Bind vertex and index buffers

vk::Buffer vertexBuffers[] = {*node->[Link]};
vk::DeviceSize offsets[] = {0};
[Link](0, 1, vertexBuffers, offsets);
[Link](*node->[Link], 0,
vk::IndexType::eUint32);

// Draw the mesh

[Link](
static_cast<uint32_t>(node->[Link]()),
1,
0,
0,
0
);
}
}
}

1.6. Advanced PBR Techniques

While we’ve covered the basics of PBR implementation, there are several advanced techniques that
can enhance the realism of your rendering:

1.6.1. Image-Based Lighting (IBL)

IBL uses environment maps to simulate global illumination: * Diffuse IBL: Uses irradiance maps
for ambient lighting * Specular IBL: Uses pre-filtered environment maps and BRDF integration
maps for reflections

1.6.2. Subsurface Scattering

For materials like skin, wax, or marble where light penetrates the surface:

315
 • Simulates how light scatters within translucent materials

• Can be approximated with techniques like subsurface scattering profiles

1.6.3. Clear Coat

For materials with a thin, glossy layer on top:

• Automotive paint, varnished wood, etc.

• Implemented as an additional specular lobe

1.6.4. Anisotropy

For materials with directional reflections:

• Brushed metal, hair, fabric, etc.

• Requires additional material parameters and modified BRDFs

1.7. Conclusion and Next Steps

In this chapter, we’ve applied the PBR knowledge from the Lighting & Materials chapter to
implement a PBR pipeline for our glTF models. We’ve learned:

• How to leverage the material properties from glTF for PBR rendering

• How to set up uniform buffers and push constants for PBR parameters

• How to implement a PBR shader that works with glTF materials

• How to integrate our camera system with PBR for view-dependent effects

• How to render glTF models with physically accurate lighting

This implementation allows us to render the glTF models we loaded in the previous chapter with
physically accurate materials, resulting in more realistic and consistent rendering across different
lighting conditions.

In the next chapter, we’ll explore how to render multiple objects with different transformations,
which will allow us to create more complex scenes with our PBR-enabled engine.

If you want to dive deeper into lighting and materials, refer back to the Lighting & Materials
chapter, where we explored the theory behind PBR in detail.

Previous: Loading a glTF Model | Next: Rendering Multiple Objects :pp: ++

Loading Models: Managing Multiple

Objects

316
1. Managing Multiple Objects in a 3D Scene
1.1. Introduction to Multi-Object Rendering
In previous chapters, we’ve focused on loading and rendering a single 3D model. However, real-
world applications rarely display just one object. Games, simulations, and visualizations typically
contain many objects that interact within a shared environment. This chapter explores how to
efficiently manage and render multiple objects in a 3D scene.

The ability to render multiple objects is fundamental to creating rich, interactive environments. It
involves not just duplicating models, but also managing their unique properties, spatial
relationships, and rendering states. As we’ll see, this introduces both challenges and opportunities
for optimization.

1.2. Approaches to Managing Multiple Objects

There are several strategies for handling multiple objects in a 3D engine, each with different trade-
offs:

1.2.1. Object Instances vs. Multiple Models

When creating a scene with multiple similar objects (like trees in a forest or buildings in a city), we
have two main approaches:

• Multiple Model Instances: Load the model once but render it multiple times with different
transformations

◦ Advantages: Memory efficient, single asset to manage

◦ Use cases: Repeated elements like trees, rocks, furniture

• Unique Models: Load separate models for each unique object

◦ Advantages: Greater variety, independent modifications

◦ Use cases: Main characters, unique structures, varied elements

For our engine, we’ll implement the instancing approach, which is more memory-efficient and
suitable for many common scenarios.

1.2.2. Scene Organization Strategies

Beyond simply having multiple objects, we need to organize them effectively:

• Flat Collection: Store all objects in a simple list or array

◦ Advantages: Simplicity, easy iteration

◦ Disadvantages: No spatial relationships, inefficient for large scenes

• Spatial Partitioning: Organize objects by their location in 3D space

◦ Advantages: Efficient culling and queries, better performance for large scenes

317
 ◦ Examples: Octrees, BSP trees, grid systems

• Scene Graph: Organize objects in a hierarchical tree structure

◦ Advantages: Parent-child relationships, hierarchical transformations

◦ Use cases: Articulated models, complex object relationships

Our implementation will use a simple collection for this example, but in a more advanced engine,
you would typically combine this with spatial partitioning and scene graph techniques.

1.3. Performance Considerations

Rendering multiple objects efficiently requires careful attention to performance:

1.3.1. Draw Call Optimization

Each object typically requires at least one draw call, which can become a bottleneck:

• Batching: Combining similar objects into a single draw call

• Instanced Rendering: Using hardware instancing to draw multiple copies of the same mesh

• Level of Detail (LOD): Using simpler models for distant objects

1.3.2. Culling Techniques

Not all objects need to be rendered every frame:

• Frustum Culling: Skip rendering objects outside the camera’s view

• Occlusion Culling: Skip rendering objects hidden behind other objects

• Distance Culling: Skip rendering objects too far from the camera

1.3.3. Memory Management

With multiple objects, memory usage becomes more critical:

• Shared Resources: Reuse meshes, textures, and materials across objects

• Asset Streaming: Load and unload assets based on proximity to the camera

• Instance Data: Store only transformation and material variations per instance

1.4. Implementing Object Instances

Now let’s implement a system for managing multiple object instances. We’ll start with a simple
structure to store instance data:

// Object instances - using the same structure as in our model system

struct ObjectInstance {
glm::vec3 position; // Position in world space
glm::vec3 rotation; // Rotation in Euler angles (degrees)

318
 glm::vec3 scale; // Scale factors for each axis
};

// Collection of object instances

std::vector<ObjectInstance> objectInstances;

This structure stores the position, rotation, and scale for each instance, along with a method to
compute the model matrix. The model matrix transforms the object from its local space to world
space, combining all three transformations.

Next, we’ll set up several instances with different transformations:

void setupObjectInstances() {
// Create multiple instances of the model with different positions
const int MAX_OBJECTS = 10; // Define how many objects we want
[Link](MAX_OBJECTS);

// Instance 1 - Center
objectInstances[0].position = glm::vec3(0.0f, 0.0f, 0.0f);
objectInstances[0].rotation = glm::vec3(0.0f, 0.0f, 0.0f);
objectInstances[0].scale = glm::vec3(1.0f);

// Instance 2 - Left
objectInstances[1].position = glm::vec3(-2.0f, 0.0f, -1.0f);
objectInstances[1].rotation = glm::vec3(0.0f, 45.0f, 0.0f);
objectInstances[1].scale = glm::vec3(0.8f);

// Instance 3 - Right
objectInstances[2].position = glm::vec3(2.0f, 0.0f, -1.0f);
objectInstances[2].rotation = glm::vec3(0.0f, -45.0f, 0.0f);
objectInstances[2].scale = glm::vec3(0.8f);

// Instance 4 - Back Left

objectInstances[3].position = glm::vec3(-1.5f, 0.0f, -3.0f);
objectInstances[3].rotation = glm::vec3(0.0f, 30.0f, 0.0f);
objectInstances[3].scale = glm::vec3(0.7f);

// Instance 5 - Back Right

objectInstances[4].position = glm::vec3(1.5f, 0.0f, -3.0f);
objectInstances[4].rotation = glm::vec3(0.0f, -30.0f, 0.0f);
objectInstances[4].scale = glm::vec3(0.7f);

// Instance 6 - Front Left

objectInstances[5].position = glm::vec3(-1.5f, 0.0f, 1.5f);
objectInstances[5].rotation = glm::vec3(0.0f, -30.0f, 0.0f);
objectInstances[5].scale = glm::vec3(0.6f);

// Instance 7 - Front Right

objectInstances[6].position = glm::vec3(1.5f, 0.0f, 1.5f);
objectInstances[6].rotation = glm::vec3(0.0f, 30.0f, 0.0f);

319
 objectInstances[6].scale = glm::vec3(0.6f);

// Instance 8 - Above
objectInstances[7].position = glm::vec3(0.0f, 2.0f, -2.0f);
objectInstances[7].rotation = glm::vec3(45.0f, 0.0f, 0.0f);
objectInstances[7].scale = glm::vec3(0.5f);

// Instance 9 - Below
objectInstances[8].position = glm::vec3(0.0f, -1.0f, -2.0f);
objectInstances[8].rotation = glm::vec3(-30.0f, 0.0f, 0.0f);
objectInstances[8].scale = glm::vec3(0.5f);

// Instance 10 - Far Back

objectInstances[9].position = glm::vec3(0.0f, 0.5f, -5.0f);
objectInstances[9].rotation = glm::vec3(0.0f, 180.0f, 0.0f);
objectInstances[9].scale = glm::vec3(1.2f);
}

This function creates ten instances of our model, each with a unique position, rotation, and scale.
This allows us to create a more interesting scene with varied object placements.

1.5. Rendering Multiple Objects

Now that we have our object instances set up, we need to render them. Here’s how we can modify
our rendering loop to handle multiple objects:

void drawFrame() {
// ... (standard Vulkan frame setup)

// Begin command buffer recording

[Link]({});

// Transition image layout for rendering

transition_image_layout(
imageIndex,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eColorAttachmentOptimal,
{},
vk::AccessFlagBits2::eColorAttachmentWrite,
vk::PipelineStageFlagBits2::eTopOfPipe,
vk::PipelineStageFlagBits2::eColorAttachmentOutput
);

// Set up rendering attachments

vk::ClearValue clearColor = vk::ClearColorValue(0.0f, 0.0f, 0.0f, 1.0f);
vk::ClearValue clearDepth = vk::ClearDepthStencilValue(1.0f, 0);

vk::RenderingAttachmentInfo colorAttachmentInfo = {
.imageView = swapChainImageViews[imageIndex],

320
 .imageLayout = vk::ImageLayout::eColorAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,
.clearValue = clearColor
};

vk::RenderingAttachmentInfo depthAttachmentInfo = {
.imageView = depthImageView,
.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,
.clearValue = clearDepth
};

vk::RenderingInfo renderingInfo = {
.renderArea = { .offset = { 0, 0 }, .extent = swapChainExtent },
.layerCount = 1,
.colorAttachmentCount = 1,
.pColorAttachments = &colorAttachmentInfo,
.pDepthAttachment = &depthAttachmentInfo
};

// Begin dynamic rendering

[Link](renderingInfo);

// Bind pipeline
[Link](vk::PipelineBindPoint::eGraphics, graphicsPipeline);

// Set viewport and scissor

[Link](0, vk::Viewport(0.0f, 0.0f,
static_cast<float>([Link]), static_cast<float>([Link]),
0.0f, 1.0f));
[Link](0, vk::Rect2D(vk::Offset2D(0, 0), swapChainExtent));

// Bind descriptor set with uniform buffer and textures

[Link](
vk::PipelineBindPoint::eGraphics,
pipelineLayout,
0,
1,
&descriptorSets[currentFrame],
0,
nullptr
);

// Update view and projection in uniform buffer

UniformBufferObject ubo{};
[Link] = [Link]();
[Link] = [Link]([Link] /
(float)[Link]);
[Link][1][1] *= -1; // Vulkan's Y coordinate is inverted

321
 // Copy to uniform buffer (per frame-in-flight)
memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));

// Render each object instance

for (size_t i = 0; i < [Link](); i++) {
const auto& instance = objectInstances[i];

// Create model matrix for this instance

glm::mat4 modelMatrix = glm::mat4(1.0f);
modelMatrix = glm::translate(modelMatrix, [Link]);
modelMatrix = glm::rotate(modelMatrix, glm::radians([Link].x),
glm::vec3(1.0f, 0.0f, 0.0f));
modelMatrix = glm::rotate(modelMatrix, glm::radians([Link].y),
glm::vec3(0.0f, 1.0f, 0.0f));
modelMatrix = glm::rotate(modelMatrix, glm::radians([Link].z),
glm::vec3(0.0f, 0.0f, 1.0f));
modelMatrix = glm::scale(modelMatrix, [Link]);

// Render all nodes in the model

renderNode(commandBuffer, [Link], modelMatrix);
}

// End dynamic rendering

[Link]();

// Transition image layout for presentation

transition_image_layout(
imageIndex,
vk::ImageLayout::eColorAttachmentOptimal,
vk::ImageLayout::ePresentSrcKHR,
vk::AccessFlagBits2::eColorAttachmentWrite,
{},
vk::PipelineStageFlagBits2::eColorAttachmentOutput,
vk::PipelineStageFlagBits2::eBottomOfPipe
);

// End command buffer recording

[Link]();

// ... (submit command buffer and present)

}

// Helper function to recursively render all nodes in the model

void renderNode(const vk::raii::CommandBuffer& commandBuffer, const
std::vector<Node*>& nodes, const glm::mat4& parentMatrix) {
for (const auto node : nodes) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// If this node has a mesh, render it

322
 if (!node->[Link]() && !node->[Link]() &&
node->vertexBufferIndex >= 0 && node->indexBufferIndex >= 0) {

// Set up push constants for material properties

PushConstantBlock pushConstants{};

if (node->[Link] >= 0 && node->[Link] <

static_cast<int>([Link]())) {
const auto& material = [Link][node->[Link]];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link] >=
0 ? 1 : -1;
[Link] =
[Link] >= 0 ? 2 : -1;
[Link] = [Link] >= 0 ? 3
: -1;
[Link] = [Link] >=
0 ? 4 : -1;
[Link] = [Link] >= 0
? 5 : -1;
} else {
// Default material properties
[Link] = glm::vec4(1.0f);
[Link] = 1.0f;
[Link] = 1.0f;
[Link] = 1;
[Link] = -1;
[Link] = -1;
[Link] = -1;
[Link] = -1;
}

// Update model matrix in push constants

[Link](pipelineLayout,
vk::ShaderStageFlagBits::eFragment, 0, sizeof(PushConstantBlock), &pushConstants);

// Bind vertex and index buffers

[Link](0, *vertexBuffers[node-
>vertexBufferIndex], {0});
[Link](*indexBuffers[node->indexBufferIndex], 0,
vk::IndexType::eUint32);

// Draw the mesh

[Link](static_cast<uint32_t>(node-
>[Link]()), 1, 0, 0, 0);
}

// Recursively render children

if (!node->[Link]()) {

323
 renderNode(commandBuffer, node->children, nodeMatrix);
}
}
}

This rendering approach leverages our model system to efficiently render multiple instances of a
model:

1. It uses the scene graph structure to handle complex models with multiple parts

2. It properly handles parent-child relationships and hierarchical transformations

3. It applies material properties to each mesh using push constants

4. It supports animations through the node transformation system

While this approach is more sophisticated than a simple flat list of objects, it does have some
limitations:

1. It still requires a separate draw call for each mesh in each instance, which can be inefficient for
large numbers of objects

2. It doesn’t implement any culling or batching optimizations

3. For very large scenes, additional spatial partitioning would be beneficial

1.6. Advanced Techniques: Hardware Instancing

For more efficient rendering of many similar objects, we can use hardware instancing. This allows
us to draw multiple instances of the same model with a single draw call:

// Instance data for GPU instancing

struct InstanceData {
glm::mat4 model; // Model matrix for this instance
};

// Create buffers to hold instance data for each node with a mesh
std::vector<vk::raii::Buffer> instanceBuffers;
std::vector<vk::raii::DeviceMemory> instanceBufferMemories;
std::vector<void*> instanceBuffersMapped;

void setupInstanceBuffers() {
// Create an instance buffer for each node with a mesh
for (auto node : [Link]) {
if (node->[Link]() || node->[Link]()) {
continue;
}

// Calculate buffer size

vk::DeviceSize bufferSize = sizeof(InstanceData) * [Link]();

// Create the buffer

324
 vk::raii::Buffer instanceBuffer = nullptr;
vk::raii::DeviceMemory instanceBufferMemory = nullptr;
createBuffer(
bufferSize,
vk::BufferUsageFlagBits::eVertexBuffer,
vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent,
instanceBuffer,
instanceBufferMemory
);

// Map the buffer memory

void* instanceBufferMapped = [Link](instanceBufferMemory, 0,
bufferSize, {});

// Store buffer and memory

instanceBuffers.push_back(std::move(instanceBuffer));
instanceBufferMemories.push_back(std::move(instanceBufferMemory));
instanceBuffersMapped.push_back(instanceBufferMapped);

// Set the instance buffer index for this node

node->instanceBufferIndex = static_cast<int>([Link]() - 1);
}

// Update all instance buffers

updateInstanceBuffers();
}

void updateInstanceBuffers() {
// For each node with an instance buffer
for (auto node : [Link]) {
if (node->instanceBufferIndex < 0) {
continue;
}

// Prepare instance data for this node

std::vector<InstanceData> instanceData([Link]());
for (size_t i = 0; i < [Link](); i++) {
// Create model matrix for this instance
glm::mat4 modelMatrix = glm::mat4(1.0f);
modelMatrix = glm::translate(modelMatrix, objectInstances[i].position);
modelMatrix = glm::rotate(modelMatrix,
glm::radians(objectInstances[i].rotation.x), glm::vec3(1.0f, 0.0f, 0.0f));
modelMatrix = glm::rotate(modelMatrix,
glm::radians(objectInstances[i].rotation.y), glm::vec3(0.0f, 1.0f, 0.0f));
modelMatrix = glm::rotate(modelMatrix,
glm::radians(objectInstances[i].rotation.z), glm::vec3(0.0f, 0.0f, 1.0f));
modelMatrix = glm::scale(modelMatrix, objectInstances[i].scale);

// Combine with node's local matrix

instanceData[i].model = modelMatrix * node->getLocalMatrix();

325
 }

// Copy to instance buffer

memcpy(instanceBuffersMapped[node->instanceBufferIndex], [Link](),
sizeof(InstanceData) * [Link]());
}
}

// Modify vertex input state to include instance data

vk::PipelineVertexInputStateCreateInfo vertexInputInfo{};
// ... (standard vertex input setup)

// Add instance data bindings and attributes

vk::VertexInputBindingDescription instanceBindingDescription{};
[Link] = 1; // Use binding point 1 for instance data
[Link] = sizeof(InstanceData);
[Link] = vk::VertexInputRate::eInstance; // Advance per
instance

// Four attributes for the 4x4 matrix (one per row)

std::array<vk::VertexInputAttributeDescription, 4> instanceAttributeDescriptions{};
for (uint32_t i = 0; i < 4; i++) {
instanceAttributeDescriptions[i].binding = 1;
instanceAttributeDescriptions[i].location = 4 + i; // Start after vertex
attributes
instanceAttributeDescriptions[i].format = vk::Format::eR32G32B32A32Sfloat;
instanceAttributeDescriptions[i].offset = sizeof(float) * 4 * i;
}

// Combine vertex and instance bindings/attributes

std::array<vk::VertexInputBindingDescription, 2> bindingDescriptions = {
vertexBindingDescription,
instanceBindingDescription
};

std::vector<vk::VertexInputAttributeDescription> attributeDescriptions;
// Add vertex attributes
for (const auto& attr : vertexAttributeDescriptions) {
attributeDescriptions.push_back(attr);
}
// Add instance attributes
for (const auto& attr : instanceAttributeDescriptions) {
attributeDescriptions.push_back(attr);
}

// Update vertex input info

[Link] =
static_cast<uint32_t>([Link]());
[Link] = [Link]();
[Link] =
static_cast<uint32_t>([Link]());

326
 [Link] = [Link]();

With hardware instancing set up, we can modify our rendering loop to draw all instances in a
single call:

The same five steps apply here; the difference is in step 4 where we bind the instance buffer and
draw N instances:

• Begin and describe attachments

• Begin rendering, bind pipeline, set viewport/scissor

• Update camera UBO (view/projection)

• Bind per‑mesh vertex + index buffers and a per‑mesh instance buffer, then draw instanced

• End rendering and present

void drawFrame() {
// ... (standard Vulkan frame setup)

// Begin command buffer recording

[Link]({});

// Transition image layout for rendering

transition_image_layout(
imageIndex,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eColorAttachmentOptimal,
{},
vk::AccessFlagBits2::eColorAttachmentWrite,
vk::PipelineStageFlagBits2::eTopOfPipe,
vk::PipelineStageFlagBits2::eColorAttachmentOutput
);

// Set up rendering attachments

vk::ClearValue clearColor = vk::ClearColorValue(0.0f, 0.0f, 0.0f, 1.0f);
vk::ClearValue clearDepth = vk::ClearDepthStencilValue(1.0f, 0);

vk::RenderingAttachmentInfo colorAttachmentInfo = {
.imageView = swapChainImageViews[imageIndex],
.imageLayout = vk::ImageLayout::eColorAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,
.clearValue = clearColor
};

vk::RenderingAttachmentInfo depthAttachmentInfo = {
.imageView = depthImageView,
.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,

327
 .clearValue = clearDepth
};

vk::RenderingInfo renderingInfo = {
.renderArea = { .offset = { 0, 0 }, .extent = swapChainExtent },
.layerCount = 1,
.colorAttachmentCount = 1,
.pColorAttachments = &colorAttachmentInfo,
.pDepthAttachment = &depthAttachmentInfo
};

// Begin dynamic rendering

[Link](renderingInfo);

// Bind pipeline
[Link](vk::PipelineBindPoint::eGraphics, graphicsPipeline);

// Set viewport and scissor

[Link](0, vk::Viewport(0.0f, 0.0f,
static_cast<float>([Link]), static_cast<float>([Link]),
0.0f, 1.0f));
[Link](0, vk::Rect2D(vk::Offset2D(0, 0), swapChainExtent));

// Update view and projection in uniform buffer

UniformBufferObject ubo{};
[Link] = [Link]();
[Link] = [Link]([Link] /
(float)[Link]);
[Link][1][1] *= -1; // Vulkan's Y coordinate is inverted

// Copy to uniform buffer (per frame-in-flight)

memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));

// Bind descriptor set

[Link](
vk::PipelineBindPoint::eGraphics,
pipelineLayout,
0,
1,
&descriptorSets[currentFrame],
0,
nullptr
);

// Render all nodes in the model with instancing

renderNodeInstanced(commandBuffer, [Link]);

// End dynamic rendering

[Link]();

// Transition image layout for presentation

328
 transition_image_layout(
imageIndex,
vk::ImageLayout::eColorAttachmentOptimal,
vk::ImageLayout::ePresentSrcKHR,
vk::AccessFlagBits2::eColorAttachmentWrite,
{},
vk::PipelineStageFlagBits2::eColorAttachmentOutput,
vk::PipelineStageFlagBits2::eBottomOfPipe
);

// End command buffer recording

[Link]();

// ... (submit command buffer and present)

}

// Helper function to recursively render all nodes in the model with instancing
void renderNodeInstanced(const vk::raii::CommandBuffer& commandBuffer, const
std::vector<Node*>& nodes) {
for (const auto node : nodes) {
// If this node has a mesh and an instance buffer, render it
if (!node->[Link]() && !node->[Link]() &&
node->vertexBufferIndex >= 0 && node->indexBufferIndex >= 0 &&
node->instanceBufferIndex >= 0) {

// Set up push constants for material properties

PushConstantBlock pushConstants{};

if (node->[Link] >= 0 && node->[Link] <

static_cast<int>([Link]())) {
const auto& material = [Link][node->[Link]];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link] >=
0 ? 1 : -1;
[Link] =
[Link] >= 0 ? 2 : -1;
[Link] = [Link] >= 0 ? 3
: -1;
[Link] = [Link] >=
0 ? 4 : -1;
[Link] = [Link] >= 0
? 5 : -1;
} else {
// Default material properties
[Link] = glm::vec4(1.0f);
[Link] = 1.0f;
[Link] = 1.0f;
[Link] = 1;
[Link] = -1;

329
 [Link] = -1;
[Link] = -1;
[Link] = -1;
}

// Update push constants

[Link](pipelineLayout,
vk::ShaderStageFlagBits::eFragment, 0, sizeof(PushConstantBlock), &pushConstants);

// Bind vertex and instance buffers

vk::Buffer vertexBuffers[] = {*vertexBuffers[node->vertexBufferIndex],
*instanceBuffers[node->instanceBufferIndex]};
vk::DeviceSize offsets[] = {0, 0};
[Link](0, 2, vertexBuffers, offsets);
[Link](*indexBuffers[node->indexBufferIndex], 0,
vk::IndexType::eUint32);

// Draw all instances of this mesh in a single call

[Link](
static_cast<uint32_t>(node->[Link]()),
static_cast<uint32_t>([Link]()), // Instance count
0, 0, 0
);
}

// Recursively render children

if (!node->[Link]()) {
renderNodeInstanced(commandBuffer, node->children);
}
}
}

This approach is much more efficient for rendering large numbers of similar objects, as it reduces
the number of draw calls and uniform buffer updates.

1.7. Vertex Shader Modifications for Instancing

To support hardware instancing, we need to modify our vertex shader to use the instance data:

#version 450

// Vertex attributes
layout(location = 0) in vec3 inPosition;
layout(location = 1) in vec3 inNormal;
layout(location = 2) in vec3 inColor;
layout(location = 3) in vec2 inTexCoord;

// Instance attributes (model matrix, one row per attribute)

layout(location = 4) in vec4 instanceModelRow0;

330
layout(location = 5) in vec4 instanceModelRow1;
layout(location = 6) in vec4 instanceModelRow2;
layout(location = 7) in vec4 instanceModelRow3;

// Uniform buffer for view and projection matrices

layout(binding = 0) uniform UniformBufferObject {
mat4 model;
mat4 view;
mat4 proj;

// PBR parameters (not used in this shader but included for compatibility)
vec4 lightPositions[4];
vec4 lightColors[4];
vec4 camPos;
float exposure;
float gamma;
float prefilteredCubeMipLevels;
float scaleIBLAmbient;
} ubo;

// Output to fragment shader

layout(location = 0) out vec3 fragPosition;
layout(location = 1) out vec3 fragNormal;
layout(location = 2) out vec2 fragTexCoord;
layout(location = 3) out vec3 fragColor;

void main() {
// Reconstruct model matrix from instance attributes
mat4 instanceModel = mat4(
instanceModelRow0,
instanceModelRow1,
instanceModelRow2,
instanceModelRow3
);

// Calculate world position

vec4 worldPos = instanceModel * vec4(inPosition, 1.0);

// Output position in clip space

gl_Position = [Link] * [Link] * worldPos;

// Pass data to fragment shader

fragPosition = [Link];
fragNormal = mat3(instanceModel) * inNormal; // This is simplified; should use
normal matrix
fragTexCoord = inTexCoord;
fragColor = inColor;
}

331
1.8. Beyond Basic Instancing: Material Variations
So far, we’ve focused on positioning multiple instances of the same model with the same material.
In a real application, you might want to vary the materials as well:

// Create materials with variations for each instance

void createMaterialVariations() {
// Resize the materials vector to hold one material per instance
[Link]([Link]());

for (size_t i = 0; i < [Link](); i++) {

// Get reference to this instance's material
Material& material = [Link][i];

// Vary materials based on position or other factors

float distanceFromCenter = glm::length(objectInstances[i].position);
float angle = atan2(objectInstances[i].position.z,
objectInstances[i].position.x);

// Vary color based on angle

float hue = (angle + glm::pi<float>()) / (2.0f * glm::pi<float>());
glm::vec3 color = hsvToRgb(glm::vec3(hue, 0.7f, 0.9f));
[Link] = glm::vec4(color, 1.0f);

// Vary metallic/roughness based on distance

[Link] = glm::clamp(distanceFromCenter / 5.0f, 0.0f, 1.0f);
[Link] = glm::clamp(1.0f - distanceFromCenter / 5.0f, 0.1f,
0.9f);

// Vary emissive strength for some objects

[Link] = (i % 3 == 0) ? glm::vec3(1.0f) : glm::vec3(0.0f);
// Every third object glows
}

// Update material indices for all nodes

for (auto node : [Link]) {
// For demonstration, we'll assign materials based on node index
// In a real application, you might use more sophisticated logic
if (!node->[Link]()) {
size_t materialIndex = node->index % [Link]();
node->[Link] = static_cast<int>(materialIndex);
}
}
}

// Helper function to convert HSV to RGB

glm::vec3 hsvToRgb(glm::vec3 hsv) {
float h = hsv.x;
float s = hsv.y;
float v = hsv.z;

332
 float r, g, b;

int i = floor(h * 6);

float f = h * 6 - i;
float p = v * (1 - s);
float q = v * (1 - f * s);
float t = v * (1 - (1 - f) * s);

switch (i % 6) {
case 0: r = v, g = t, b = p; break;
case 1: r = q, g = v, b = p; break;
case 2: r = p, g = v, b = t; break;
case 3: r = p, g = q, b = v; break;
case 4: r = t, g = p, b = v; break;
case 5: r = v, g = p, b = q; break;
}

return glm::vec3(r, g, b);

}

// To use these material variations, call createMaterialVariations() after loading the

model
// The renderNode() and renderNodeInstanced() methods will automatically use the
assigned materials

This approach allows for much more visual variety in your scene, even when using the same base
model for all instances.

1.9. Conclusion and Next Steps

In this chapter, we’ve explored how to manage and render multiple objects in a 3D scene. We’ve
covered:

• Different approaches to organizing multiple objects

• Performance considerations for multi-object rendering

• Basic implementation of object instances

• Advanced techniques like hardware instancing

• Material variations for visual diversity

These techniques form the foundation for creating complex, visually rich 3D scenes. In the next
chapter, we’ll build upon this foundation to implement a complete scene rendering system that
integrates all the components we’ve developed so far.

Previous: Understanding Physically Based Rendering | Next: Rendering the Scene :pp: ++

333
Loading Models: Rendering the
Scene
1. Rendering the Scene
1.1. Introduction to Scene Rendering
Scene rendering is the process of transforming a 3D scene description into a 2D image that can be
displayed on screen. In our engine, this involves traversing the scene graph, applying
transformations, setting material properties, and issuing draw commands to the GPU.

The scene rendering process is a critical part of the rendering pipeline, as it’s where all the
components we’ve built so far come together:

• The model system provides the scene graph structure and mesh data

• The material system defines the appearance of objects

• The camera system determines the viewpoint

• The lighting system illuminates the scene

In this chapter, we’ll explore how these components work together to render a complete scene.

1.2. Scene Graph Traversal

A scene graph is a hierarchical tree structure that organizes objects in a scene. Each node in the
tree can have a transformation (position, rotation, scale) and may contain a mesh to render. Nodes
can also have child nodes, which inherit their parent’s transformation.

To render a scene graph, we need to traverse it in a depth-first manner, calculating the global
transformation matrix for each node and rendering any meshes we encounter:

void renderScene(const vk::raii::CommandBuffer& commandBuffer, Model& model, const

glm::mat4& viewMatrix, const glm::mat4& projectionMatrix) {
// Start traversal from the root nodes with an identity matrix
glm::mat4 rootMatrix = glm::mat4(1.0f);
renderNode(commandBuffer, [Link], rootMatrix);
}

The renderNode function is the heart of our scene rendering system. It recursively traverses the
scene graph, calculating the global transformation matrix for each node and rendering any meshes
it contains:

334
1.3. Node traversal and transform calculation
The rendering process begins with systematic traversal of the scene graph, where each node’s
transformation is calculated by combining its local transformation with its parent’s accumulated
transformation matrix.

void renderNode(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const glm::mat4& parentMatrix) {
for (const auto node : nodes) {
// Calculate the cumulative transformation from root to current node
// This combines the parent's world transformation with this node's local
transformation
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

The transformation calculation represents the core of hierarchical scene graph rendering. Each
node’s getLocalMatrix() returns its transformation relative to its parent, which we then combine
with the accumulated parent transformation using matrix multiplication. This mathematical
operation effectively "chains" transformations down the hierarchy, ensuring that moving a parent
node automatically moves all its children in world space.

The order of multiplication is critical here: parentMatrix * nodeLocalMatrix ensures that the node’s
local transformation occurs first (in the node’s local coordinate space), followed by the parent’s
transformation that places it in world space. This ordering preserves the hierarchical relationship
where children are positioned relative to their parents.

1.4. Mesh validation and rendering preparation

Before rendering, we must validate that the node contains valid mesh data and has been properly
uploaded to GPU buffers, ensuring robust rendering that handles incomplete or invalid scene graph
nodes.

// Validate that this node has complete, renderable mesh data

// All conditions must be met for safe GPU rendering
if (!node->[Link]() && !node->[Link]() &&
node->vertexBufferIndex >= 0 && node->indexBufferIndex >= 0) {

This validation step prevents rendering errors that could occur from incomplete scene graph nodes.
Not every node in a scene graph necessarily contains renderable geometry - some nodes exist
purely for organization or as transformation anchors for child objects. By checking for non-empty
vertex and index arrays plus valid buffer indices, we ensure that we only attempt to render nodes
that have been properly prepared with GPU resources.

The buffer index checks (>= 0) confirm that the mesh data has been successfully uploaded to GPU
buffers and assigned valid indices (0 or greater) in our buffer management system. Note that 0 is a
valid buffer index - negative values (typically -1) indicate uninitialized or failed buffer allocations.

335
1.5. Material property configuration
This material setup step translates high-level material descriptions into GPU-ready push constants
that control the appearance and lighting properties of the rendered geometry.

// Initialize push constants structure for material data transfer

PushConstantBlock pushConstants{};

// Configure material properties if a valid material is assigned

if (node->[Link] >= 0 && node->[Link] <
static_cast<int>([Link]())) {
const auto& material = [Link][node->[Link]];

// Set PBR material factors that control surface appearance

[Link] = [Link]; //
Surface color tint
[Link] = [Link]; //
Metallic vs. dielectric
[Link] = [Link]; //
Surface roughness

// Configure texture binding indices (-1 indicates no texture)

[Link] = [Link] >=
0 ? 1 : -1;
[Link] =
[Link] >= 0 ? 2 : -1;
[Link] = [Link] >= 0 ? 3
: -1;
[Link] = [Link] >=
0 ? 4 : -1;
[Link] = [Link] >= 0
? 5 : -1;
} else {
// Apply sensible default material properties for unassigned materials
[Link] = glm::vec4(1.0f); //
White base color
[Link] = 1.0f; //
Fully metallic (safe default)
[Link] = 1.0f; //
Fully rough (safe default)
[Link] = -1; // No
texture for default material
[Link] = -1; // No
metallic/roughness texture
[Link] = -1; // No
normal map
[Link] = -1; // No
ambient occlusion
[Link] = -1; // No
emissive texture

336
 }

The material configuration system bridges the gap between artist-authored materials and GPU
shader parameters. Push constants provide the fastest path for updating per-object material data,
as they bypass the GPU’s memory hierarchy and are directly accessible to shader cores. This makes
them ideal for material properties that change frequently between draw calls.

The texture index mapping system (-1 for unused, positive integers for active bindings) allows
shaders to conditionally sample textures based on availability. This approach provides flexibility
where some materials might have normal maps while others don’t, without requiring different
shader variants or complex branching logic.

The default material properties are chosen conservatively to prevent rendering artifacts when
materials are missing or improperly configured. Metallic and roughness values of 1.0 tend to
produce visually acceptable results across different lighting conditions, though they may not
represent the intended material appearance.

1.6. GPU resource binding and draw command

execution
The final rendering phase binds GPU resources and issues the actual draw command that
transforms the scene graph node into rendered pixels on the screen.

// Upload material properties to GPU via push constants

// This provides fast, per-draw-call material parameter updates
[Link](*pipelineLayout,
vk::ShaderStageFlagBits::eFragment,
0, sizeof(PushConstantBlock), &pushConstants);

// Bind geometry data buffers for GPU access

// Vertex buffer contains position, normal, texture coordinate data
[Link](0, *vertexBuffers[node-
>vertexBufferIndex], {0});
// Index buffer defines triangle connectivity and enables vertex reuse
[Link](*indexBuffers[node->indexBufferIndex], 0,
vk::IndexType::eUint32);

// Execute the draw command to render this mesh

// GPU processes indices to generate triangles and runs vertex/fragment
shaders
[Link](static_cast<uint32_t>(node-
>[Link]()), 1, 0, 0, 0);
}

The resource binding sequence follows Vulkan’s explicit binding model where each resource type
must be bound before use. Vertex buffers provide the per-vertex attribute data (positions, normals,
texture coordinates), while index buffers define how vertices connect to form triangles. This

337
indexed rendering approach reduces memory usage by allowing vertex reuse across multiple
triangles.

The drawIndexed command triggers GPU execution of the entire graphics pipeline for this mesh. The
GPU processes each index to fetch vertex data, runs the vertex shader to transform geometry,
rasterizes triangles to generate fragments, and executes the fragment shader to determine final
pixel colors. All the material properties we configured via push constants become available to the
fragment shader during this process.

1.7. Hierarchical recursion

Finally, ensure complete scene graph traversal by recursively processing child nodes with the
accumulated transformation matrix, maintaining the hierarchical structure throughout the
rendering process.

// Recursively process child nodes with accumulated transformation

// This maintains the hierarchical transformation chain down the scene graph
if (!node->[Link]()) {
renderNode(commandBuffer, node->children, nodeMatrix);
}
}
}

This traversal approach ensures that:

1. Each node’s transformation is correctly combined with its parent’s transformation

2. Child nodes are rendered with the correct global transformation

3. The scene graph hierarchy is preserved during rendering

1.8. Understanding the Rendering Process

Let’s break down the rendering process in more detail:

1.8.1. Transformation Calculation

The first step in rendering a node is calculating its global transformation matrix:

// Calculate global matrix for this node

glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

This combines the node’s local transformation (position, rotation, scale) with its parent’s global
transformation. The result is a matrix that transforms from the node’s local space to world space.

The getLocalMatrix method (defined in the Node class) combines the node’s translation, rotation, and
scale properties:

338
 glm::mat4 getLocalMatrix() {
return glm::translate(glm::mat4(1.0f), translation) *
glm::toMat4(rotation) *
glm::scale(glm::mat4(1.0f), scale) *
matrix;
}

1.8.2. Material Setup

We covered PBR material theory and shader details earlier in PBR Rendering, so
we won’t restate that here. This section focuses on the wiring: how material
 properties are packed into push constants and consumed by the draw call in this
chapter’s context.

If the node has a mesh, we need to set up its material properties before rendering:

// Set up push constants for material properties

PushConstantBlock pushConstants{};

if (node->[Link] >= 0 && node->[Link] <

static_cast<int>([Link]())) {
const auto& material = [Link][node->[Link]];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link];
[Link] = [Link] >= 0 ? 1 : -1;
[Link] =
[Link] >= 0 ? 2 : -1;
[Link] = [Link] >= 0 ? 3 : -1;
[Link] = [Link] >= 0 ? 4 : -1;
[Link] = [Link] >= 0 ? 5 : -1;
} else {
// Default material properties
[Link] = glm::vec4(1.0f);
[Link] = 1.0f;
[Link] = 1.0f;
[Link] = -1;
[Link] = -1;
[Link] = -1;
[Link] = -1;
[Link] = -1;
}

// Update push constants

[Link](*pipelineLayout, vk::ShaderStageFlagBits::eFragment, 0,
sizeof(PushConstantBlock), &pushConstants);

339
This code:

1. Retrieves the material associated with the mesh

2. Sets up push constants with the material properties

3. Passes these properties to the fragment shader using push constants

The material properties include:

• Base color factor (albedo)

• Metallic factor

• Roughness factor

• Texture set indices for various material maps (base color, metallic-roughness, normal,
occlusion, emissive)

1.8.3. Mesh Rendering

Once the transformation and material are set up, we can render the mesh:

// Bind vertex and index buffers

[Link](0, *vertexBuffers[node->vertexBufferIndex], {0});
[Link](*indexBuffers[node->indexBufferIndex], 0,
vk::IndexType::eUint32);

// Draw the mesh

[Link](static_cast<uint32_t>(node->[Link]()), 1, 0, 0,
0);

This code:

1. Binds the vertex buffer containing the mesh’s vertices

2. Binds the index buffer containing the mesh’s indices

3. Issues a draw command to render the mesh

1.8.4. Recursive Traversal

After rendering the current node, we recursively traverse its children:

// Recursively render children

if (!node->[Link]()) {
renderNode(commandBuffer, node->children, nodeMatrix);
}

This ensures that all nodes in the scene graph are visited and rendered with the correct
transformations.

340
1.9. Integrating Scene Rendering in the Main Loop
To use our scene rendering system in the main rendering loop, we need to set up the necessary
Vulkan state and call the renderScene function. To keep this digestible, think of the frame as five
steps:

1) Begin and describe attachments (dynamic rendering inputs) 2) Begin rendering, bind pipeline,
set viewport/scissor 3) Update camera UBO (view/projection) 4) Traverse scene graph and issue per-
mesh draws 5) End rendering and present

void drawFrame() {
// ... (standard Vulkan frame setup)

// Begin command buffer recording

[Link]({});

// Transition image layout for rendering

transition_image_layout(
imageIndex,
vk::ImageLayout::eUndefined,
vk::ImageLayout::eColorAttachmentOptimal,
{},
vk::AccessFlagBits2::eColorAttachmentWrite,
vk::PipelineStageFlagBits2::eTopOfPipe,
vk::PipelineStageFlagBits2::eColorAttachmentOutput
);

// Set up rendering attachments

vk::ClearValue clearColor = vk::ClearColorValue(0.0f, 0.0f, 0.0f, 1.0f);
vk::ClearValue clearDepth = vk::ClearDepthStencilValue(1.0f, 0);

vk::RenderingAttachmentInfo colorAttachmentInfo = {
.imageView = swapChainImageViews[imageIndex],
.imageLayout = vk::ImageLayout::eColorAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,
.clearValue = clearColor
};

vk::RenderingAttachmentInfo depthAttachmentInfo = {
.imageView = depthImageView,
.imageLayout = vk::ImageLayout::eDepthStencilAttachmentOptimal,
.loadOp = vk::AttachmentLoadOp::eClear,
.storeOp = vk::AttachmentStoreOp::eStore,
.clearValue = clearDepth
};

vk::RenderingInfo renderingInfo = {
.renderArea = { .offset = { 0, 0 }, .extent = swapChainExtent },
.layerCount = 1,

341
 .colorAttachmentCount = 1,
.pColorAttachments = &colorAttachmentInfo,
.pDepthAttachment = &depthAttachmentInfo
};

// Begin dynamic rendering

[Link](renderingInfo);

// Bind pipeline
[Link](vk::PipelineBindPoint::eGraphics, graphicsPipeline);

// Set viewport and scissor

[Link](0, vk::Viewport(0.0f, 0.0f,
static_cast<float>([Link]), static_cast<float>([Link]),
0.0f, 1.0f));
[Link](0, vk::Rect2D(vk::Offset2D(0, 0), swapChainExtent));

// Bind descriptor set with uniform buffer and textures

[Link](
vk::PipelineBindPoint::eGraphics,
pipelineLayout,
0,
1,
&descriptorSets[currentFrame],
0,
nullptr
);

// Update view and projection in uniform buffer

UniformBufferObject ubo{};
[Link] = [Link]();
[Link] = [Link]([Link] /
(float)[Link]);
[Link][1][1] *= -1; // Vulkan's Y coordinate is inverted

// Copy to uniform buffer (per frame-in-flight)

memcpy(uniformBuffers[currentFrame].mapped, &ubo, sizeof(ubo));

// Render the scene

renderScene(commandBuffer, model, [Link], [Link]);

// End dynamic rendering

[Link]();

// Transition image layout for presentation

transition_image_layout(
imageIndex,
vk::ImageLayout::eColorAttachmentOptimal,
vk::ImageLayout::ePresentSrcKHR,
vk::AccessFlagBits2::eColorAttachmentWrite,
{},

342
 vk::PipelineStageFlagBits2::eColorAttachmentOutput,
vk::PipelineStageFlagBits2::eBottomOfPipe
);

// End command buffer recording

[Link]();

// ... (submit command buffer and present)

}

This code:

1. Sets up the Vulkan rendering state (command buffer, image transitions, rendering attachments)

2. Binds the graphics pipeline and descriptor sets

3. Updates the view and projection matrices in the uniform buffer

4. Calls renderScene to render the entire scene

5. Finalizes the rendering and presents the result

1.10. Performance Considerations

Rendering a complex scene can be performance-intensive. Here are some techniques to optimize
scene rendering:

1.10.1. Frustum Culling

Frustum culling is the process of skipping the rendering of objects that are outside the camera’s
view frustum. This can significantly improve performance by reducing the number of draw calls:

bool isNodeVisible(const Node* node, const glm::mat4& viewProjection) {

// Calculate the node's bounding sphere in world space
glm::vec3 center = glm::vec3(node->getGlobalMatrix() * glm::vec4(node-
>[Link], 1.0f));
float radius = node->[Link] * glm::length(glm::vec3(node-
>getGlobalMatrix()[0])); // Scale radius by the largest scale factor

// Check if the bounding sphere is visible in the view frustum

for (int i = 0; i < 6; i++) {
// Extract frustum planes from the view-projection matrix
glm::vec4 plane = getFrustumPlane(viewProjection, i);

// Calculate the signed distance from the sphere center to the plane
float distance = glm::dot(glm::vec4(center, 1.0f), plane);

// If the sphere is completely behind the plane, it's not visible

if (distance < -radius) {
return false;
}

343
 }

return true;
}

void renderNodeWithCulling(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const glm::mat4& parentMatrix, const glm::mat4&
viewProjection) {
for (const auto node : nodes) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// Check if the node is visible

if (isNodeVisible(node, viewProjection)) {
// Render the node (same as before)
// ...

// Recursively render children

if (!node->[Link]()) {
renderNodeWithCulling(commandBuffer, node->children, nodeMatrix,
viewProjection);
}
}
}
}

1.10.2. Level of Detail (LOD)

Level of Detail (LOD) involves using simpler versions of models for objects that are far from the
camera:

void renderNodeWithLOD(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const glm::mat4& parentMatrix, const glm::vec3&
cameraPosition) {
for (const auto node : nodes) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// Calculate distance to camera

glm::vec3 nodePosition = glm::vec3(nodeMatrix[3]);
float distanceToCamera = glm::distance(nodePosition, cameraPosition);

// Select LOD level based on distance

int lodLevel = 0;
if (distanceToCamera > 50.0f) {
lodLevel = 2; // Low detail
} else if (distanceToCamera > 20.0f) {
lodLevel = 1; // Medium detail
}

344
 // Render the node with the selected LOD level
// ...

// Recursively render children

if (!node->[Link]()) {
renderNodeWithLOD(commandBuffer, node->children, nodeMatrix,
cameraPosition);
}
}
}

1.10.3. Occlusion Culling

Occlusion culling involves skipping the rendering of objects that are hidden behind other objects:

void renderNodeWithOcclusion(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const glm::mat4& parentMatrix) {
// Sort nodes by distance to camera (front to back)
std::vector<std::pair<Node*, float>> sortedNodes;
for (const auto node : nodes) {
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();
glm::vec3 nodePosition = glm::vec3(nodeMatrix[3]);
float distanceToCamera = glm::length(nodePosition - cameraPosition);
sortedNodes.push_back({node, distanceToCamera});
}
std::sort([Link](), [Link](), [](const auto& a, const auto& b)
{
return [Link] < [Link];
});

// Render nodes from front to back

for (const auto& [node, distance] : sortedNodes) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// Begin occlusion query

vk::QueryPool occlusionQueryPool = createOcclusionQueryPool();
[Link](occlusionQueryPool, 0, {});

// Render the node's bounding box with depth write but no color write
renderBoundingBox(commandBuffer, node, nodeMatrix);

// End occlusion query

[Link](occlusionQueryPool, 0);

// Check if the node is visible

uint64_t occlusionResult = getOcclusionQueryResult(occlusionQueryPool);
if (occlusionResult > 0) {
// Node is visible, render it

345
 // ...

// Recursively render children

if (!node->[Link]()) {
renderNodeWithOcclusion(commandBuffer, node->children, nodeMatrix);
}
}
}
}

1.10.4. Instanced Rendering

For scenes with many identical objects, instanced rendering can significantly improve
performance:

void renderInstanced(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const std::vector<glm::mat4>& instanceMatrices) {
for (const auto node : nodes) {
// If this node has a mesh, render it with instancing
if (!node->[Link]() && !node->[Link]() &&
node->vertexBufferIndex >= 0 && node->indexBufferIndex >= 0) {

// Set up material properties (same as before)

// ...

// Bind vertex and index buffers

[Link](0, *vertexBuffers[node-
>vertexBufferIndex], {0});
[Link](*indexBuffers[node->indexBufferIndex], 0,
vk::IndexType::eUint32);

// Create and bind instance buffer

vk::raii::Buffer instanceBuffer = createInstanceBuffer(instanceMatrices);
[Link](1, *instanceBuffer, {0});

// Draw the mesh with instancing

[Link](
static_cast<uint32_t>(node->[Link]()),
static_cast<uint32_t>([Link]()),
0, 0, 0
);
}

// Recursively render children

if (!node->[Link]()) {
renderInstanced(commandBuffer, node->children, instanceMatrices);
}
}
}

346
1.11. Advanced Scene Rendering Techniques
Beyond basic scene rendering, there are several advanced techniques that can enhance the visual
quality and performance of your renderer:

1.11.1. Hierarchical Culling

Hierarchical culling involves using the scene graph structure to accelerate culling operations:

bool isNodeAndChildrenVisible(const Node* node, const glm::mat4& viewProjection, const

glm::mat4& parentMatrix) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// Check if the node's bounding volume is visible

if (!isNodeVisible(node, viewProjection, nodeMatrix)) {
// If the node is not visible, none of its children are visible either
return false;
}

// Node is visible, check if it has a mesh to render

bool hasVisibleContent = !node->[Link]() && !node-
>[Link]();

// Recursively check children

for (const auto child : node->children) {
hasVisibleContent |= isNodeAndChildrenVisible(child, viewProjection,
nodeMatrix);
}

return hasVisibleContent;
}

void renderNodeHierarchical(const vk::raii::CommandBuffer& commandBuffer, const

std::vector<Node*>& nodes, const glm::mat4& parentMatrix, const glm::mat4&
viewProjection) {
for (const auto node : nodes) {
// Calculate global matrix for this node
glm::mat4 nodeMatrix = parentMatrix * node->getLocalMatrix();

// Check if the node and its children are visible

if (isNodeAndChildrenVisible(node, viewProjection, glm::mat4(1.0f))) {
// Render the node if it has a mesh
if (!node->[Link]() && !node->[Link]() &&
node->vertexBufferIndex >= 0 && node->indexBufferIndex >= 0) {
// Render the node (same as before)
// ...
}

// Recursively render children

347
 if (!node->[Link]()) {
renderNodeHierarchical(commandBuffer, node->children, nodeMatrix,
viewProjection);
}
}
}
}

The hierarchical culling algorithm presented here works correctly but is relatively
naïve. In its current form, it’s unlikely to provide significant performance
improvements over simpler culling approaches. For real performance gains in
production environments, more efficient and complex handling would be needed,
such as:
 • Using bounding volume hierarchies (BVH) with efficient spatial data structures

• Implementing GPU-based culling with compute shaders

• Employing temporal coherence to cache visibility results across frames

• Using occlusion queries to skip entire subtrees hidden behind other geometry

1.11.2. Deferred Rendering

Deferred rendering separates the geometry and lighting passes, which can improve performance
for scenes with many lights:

void renderSceneDeferred(const vk::raii::CommandBuffer& commandBuffer, Model& model) {

// Geometry pass: render scene to G-buffer
beginGeometryPass(commandBuffer);
renderNode(commandBuffer, [Link], glm::mat4(1.0f));
endGeometryPass(commandBuffer);

// Lighting pass: apply lighting to G-buffer

beginLightingPass(commandBuffer);
for (const auto& light : lights) {
renderLight(commandBuffer, light);
}
endLightingPass(commandBuffer);
}

1.11.3. Clustered Rendering

Clustered rendering divides the view frustum into 3D cells to efficiently handle many lights:

void setupLightClusters() {
// Divide the view frustum into a 3D grid of clusters
const int clusterCountX = 16;
const int clusterCountY = 9;

348
 const int clusterCountZ = 24;

// Assign lights to clusters based on their position and radius

for (const auto& light : lights) {
for (int x = 0; x < clusterCountX; x++) {
for (int y = 0; y < clusterCountY; y++) {
for (int z = 0; z < clusterCountZ; z++) {
if (lightAffectsCluster(light, x, y, z)) {
lightClusters[x][y][z].push_back([Link]);
}
}
}
}
}

// Upload light cluster data to GPU

updateLightClusterBuffer();
}

void renderSceneClustered(const vk::raii::CommandBuffer& commandBuffer, Model& model)

{
// Bind light cluster buffer
[Link](
vk::PipelineBindPoint::eGraphics,
pipelineLayout,
1,
1,
&lightClusterDescriptorSet,
0,
nullptr
);

// Render scene normally

renderNode(commandBuffer, [Link], glm::mat4(1.0f));
}

1.12. Conclusion
In this chapter, we’ve explored the process of rendering a scene using a scene graph. We’ve seen
how to traverse the scene graph, calculate transformations, apply materials, and render meshes.
We’ve also discussed various optimization techniques to improve performance.

The scene rendering system we’ve built is flexible and extensible, allowing for the rendering of
complex scenes with multiple objects, materials, and lighting conditions. In the next chapter, we’ll
build on this foundation to implement animations, bringing our scenes to life.

Previous: Rendering Multiple Objects | Next: Updating Animations :pp: ++

349
Loading Models: Updating
Animations
1. Understanding and Implementing
Animations
1.1. Introduction to 3D Animations
Animation is a crucial aspect of modern 3D applications, bringing static models to life with
movement and interactivity. In our engine, we’ve implemented a robust animation system that
supports skeletal animations from glTF files.

Animations in 3D graphics typically involve:

• Keyframes: Specific points in time where the state of an object is explicitly defined

• Interpolation: The process of calculating intermediate states between keyframes

• Channels: Different properties that can be animated (position, rotation, scale)

• Bones/Joints: A hierarchical structure that defines how parts of a model move together

glTF provides a standardized way to store and transfer animations, which our engine can load and
play back.

1.2. Animation Data Structures

As we saw in the Model System chapter, our engine uses several structures to represent animations:

// Structure for animation keyframes

struct AnimationChannel {
enum PathType { TRANSLATION, ROTATION, SCALE };
PathType path;
Node* node = nullptr;
uint32_t samplerIndex;
};

// Structure for animation interpolation

struct AnimationSampler {
enum InterpolationType { LINEAR, STEP, CUBICSPLINE };
InterpolationType interpolation;
std::vector<float> inputs; // Key frame timestamps
std::vector<glm::vec4> outputsVec4; // Key frame values (for rotations)
std::vector<glm::vec3> outputsVec3; // Key frame values (for translations and
scales)
};

350
 // Structure for animation
struct Animation {
std::string name;
std::vector<AnimationSampler> samplers;
std::vector<AnimationChannel> channels;
float start = std::numeric_limits<float>::max();
float end = std::numeric_limits<float>::min();
float currentTime = 0.0f;
};

These structures work together to define how animations are stored and processed:

• Animation: Contains multiple channels and samplers, representing a complete animation

sequence

• AnimationChannel: Links a node in the scene graph to a specific animation property

(translation, rotation, or scale)

• AnimationSampler: Defines how to interpolate between keyframes for a specific channel

1.3. How Animation Playback Works

The animation update process is the heart of our animation system, responsible for translating
time-based animation data into actual transformations applied to scene graph nodes.

1.4. Animation Update: Validation and Time

Management
Before we start anything, we should validate that we have valid animation data and manage the
progression of animation time, including looping behavior for cyclical animations.

void Model::updateAnimation(uint32_t index, float deltaTime) {

// Validate animation data and index bounds
if ([Link]() || index >= [Link]()) {
return;
}

// Update animation timing with automatic looping

Animation& animation = animations[index];
[Link] += deltaTime;
while ([Link] >= [Link]) {
[Link] -= ([Link] - [Link]);
}

Animation validation is critical for robust systems because not all models contain animations, and
external code might request non-existent animation indices. By performing this check early, we
avoid crashes and undefined behavior when working with static models or invalid animation

351
requests. This defensive programming approach is essential in production game engines where
content from various sources might have inconsistent animation data.

Time management forms the foundation of animation playback, where the deltaTime parameter
represents the elapsed time since the last update. This frame-rate independent approach ensures
animations play at consistent speeds regardless of rendering performance. The automatic looping
mechanism seamlessly restarts animations when they reach their end time, creating continuous
motion that’s essential for idle animations, walking cycles, and other repetitive movements.

1.5. Animation Update: Channel Iteration and Sampler

Access
Now we iterate through all animation channels, establishing the connection between abstract
animation data and the specific nodes in our scene graph that will receive transformation updates.

// Process each animation channel to update corresponding scene nodes

for (auto& channel : [Link]) {
assert([Link] < [Link]());
AnimationSampler& sampler = [Link][[Link]];

The channel iteration represents the heart of our animation-to-scene-graph mapping system. Each
channel defines a specific transformation type (position, rotation, or scale) for a particular node in
the scene hierarchy. This one-to-many relationship allows complex animations where multiple
properties of multiple nodes can be animated simultaneously, enabling sophisticated character
animations with dozens of moving parts.

The sampler access pattern demonstrates the separation of concerns in our animation architecture.
Samplers contain the actual keyframe data and interpolation logic, while channels define what gets
animated. This design allows multiple channels to share the same sampler data, reducing memory
usage when the same animation curve applies to different nodes or when different transformation
components follow identical patterns.

1.6. Animation Update: Keyframe Location and

Interpolation Factor Calculation
Next, locate the appropriate keyframes that surround the current animation time and calculate the
precise interpolation factor needed for smooth transitions between discrete animation samples.

// Find the current keyframe pair that brackets the animation time using
binary search
auto nextKeyFrameIt = std::lower_bound([Link](),
[Link](), [Link]);
if (nextKeyFrameIt != [Link]() && nextKeyFrameIt !=
[Link]()) {
size_t i = std::distance([Link](), nextKeyFrameIt) - 1;
// Calculate normalized interpolation factor between keyframes

352
 float t = ([Link] - [Link][i]) / ([Link][i
+ 1] - [Link][i]);

The keyframe search algorithm uses std::lower_bound to perform a binary search, finding the pair
of keyframes that bracket the current animation time with O(log n) complexity. This efficient
approach is ideal for animation data with many keyframes, providing optimal performance
compared to linear scanning. The binary search returns an iterator to the first keyframe whose
time is greater than or equal to the current animation time, allowing us to determine the bracketing
pair by looking at the previous keyframe.

The interpolation factor calculation creates a normalized value between 0.0 and 1.0 that represents
exactly where the current time falls between two keyframes. When t=0.0, we’re at the first
keyframe; when t=1.0, we’re at the second keyframe; values in between create smooth transitions.
This mathematical foundation enables all the interpolation techniques that follow, whether for
linear position changes or complex quaternion rotations.

1.7. Animation Update: Property-Specific Interpolation

and Node Updates
Finally, apply the appropriate mathematical interpolation technique based on the transformation
type, updating the actual scene graph nodes with the computed animation values.

// Apply transformation based on the specific animation channel type

switch ([Link]) {
case AnimationChannel::TRANSLATION: {
// Linear interpolation for position changes
glm::vec3 start = sampler.outputsVec3[i];
glm::vec3 end = sampler.outputsVec3[i + 1];
[Link]->translation = glm::mix(start, end, t);
break;
}
case AnimationChannel::ROTATION: {
// Spherical linear interpolation for smooth rotation
transitions
glm::quat start = glm::quat(sampler.outputsVec4[i].w,
sampler.outputsVec4[i].x, sampler.outputsVec4[i].y, sampler.outputsVec4[i].z);
glm::quat end = glm::quat(sampler.outputsVec4[i + 1].w,
sampler.outputsVec4[i + 1].x, sampler.outputsVec4[i + 1].y, sampler.outputsVec4[i +
1].z);
[Link]->rotation = glm::slerp(start, end, t);
break;
}
case AnimationChannel::SCALE: {
// Linear interpolation for scaling transformations
glm::vec3 start = sampler.outputsVec3[i];
glm::vec3 end = sampler.outputsVec3[i + 1];
[Link]->scale = glm::mix(start, end, t);
break;

353
 }
}
break;
}
}
}
}

This method:

1. Updates the animation’s current time based on the delta time

2. Loops the animation if it reaches the end

3. For each channel in the animation:

1. Finds the current keyframe based on the current time

2. Calculates the interpolation factor between the current and next keyframe

3. Interpolates between keyframe values based on the channel type (translation, rotation, or
scale)

4. Updates the corresponding node’s transformation

1.8. Integrating Animation Updates in the Main Loop

To animate our models, we need to update the animation state each frame:

void mainLoop() {
while (!glfwWindowShouldClose(window)) {
glfwPollEvents();

// Update animation time

static auto lastTime = std::chrono::high_resolution_clock::now();
auto currentTime = std::chrono::high_resolution_clock::now();
float deltaTime = std::chrono::duration<float,
std::chrono::seconds::period>(currentTime - lastTime).count();
lastTime = currentTime;

// Update model animations

animationTime += deltaTime;
if (![Link]()) {
[Link](0, deltaTime);
}

drawFrame();
}

[Link]();
}

354
This code:

1. Calculates the time elapsed since the last frame (deltaTime)

2. Updates a global animation time counter (useful for custom animations)

3. Calls updateAnimation on the model if it has animations

4. Renders the frame with the updated animation state

1.9. Advanced Animation Techniques

While our basic animation system handles most common use cases, there are several advanced
techniques you might want to implement:

1.9.1. Animation Blending

Animation blending is a technique that combines multiple animations to create smooth transitions
or entirely new animations. This is essential for creating realistic character movement and
responsive gameplay.

[Link]. Understanding Animation Blending

At its core, animation blending works by interpolating between the transformations (position,
rotation, scale) of corresponding bones or nodes in different animations. The key concepts include:

• Blend Factor: A value between 0.0 and 1.0 that determines how much of each animation
contributes to the final result

• Blend Space: A multidimensional space where animations are positioned based on parameters
(like speed, direction)

• Blend Trees: Hierarchical structures that organize multiple blends into complex animation
systems

[Link]. Types of Animation Blending

There are several common types of animation blending:

• Linear Blending: Simple interpolation between two animations (e.g., transitioning from walk to
run)

• Additive Blending: One animation is added on top of another (e.g., adding a "wounded" limp to
any movement animation)

• Partial Blending: Blending that affects only certain parts of the skeleton (e.g., aiming a weapon
while walking)

• Parametric Blending: Blending multiple animations based on continuous parameters (e.g.,

direction + speed)

[Link]. Implementing Basic Animation Blending

Here’s how to implement a simple linear blend between two animations:

355
 void blendAnimations(uint32_t fromIndex, uint32_t toIndex, float blendFactor) {
// Store original node transformations
std::vector<glm::vec3> originalTranslations;
std::vector<glm::quat> originalRotations;
std::vector<glm::vec3> originalScales;

for (auto node : [Link]) {

originalTranslations.push_back(node->translation);
originalRotations.push_back(node->rotation);
originalScales.push_back(node->scale);
}

// Apply first animation fully

[Link](fromIndex, 0.0f);

// Store intermediate transformations

std::vector<glm::vec3> fromTranslations;
std::vector<glm::quat> fromRotations;
std::vector<glm::vec3> fromScales;

for (auto node : [Link]) {

fromTranslations.push_back(node->translation);
fromRotations.push_back(node->rotation);
fromScales.push_back(node->scale);
}

// Restore original transformations

for (size_t i = 0; i < [Link](); i++) {
[Link][i]->translation = originalTranslations[i];
[Link][i]->rotation = originalRotations[i];
[Link][i]->scale = originalScales[i];
}

// Apply second animation fully

[Link](toIndex, 0.0f);

// Blend between the two animations

for (size_t i = 0; i < [Link](); i++) {
[Link][i]->translation = glm::mix(fromTranslations[i],
[Link][i]->translation, blendFactor);
[Link][i]->rotation = glm::slerp(fromRotations[i],
[Link][i]->rotation, blendFactor);
[Link][i]->scale = glm::mix(fromScales[i], [Link][i]-
>scale, blendFactor);
}
}

This implementation:

1. Captures the original state of all nodes

356
2. Applies the first animation and stores its transformations

3. Restores the original state

4. Applies the second animation

5. Blends between the two animations using linear interpolation for positions and scales, and
spherical interpolation for rotations

[Link]. Advanced Blending Techniques

For more complex scenarios, we can implement more sophisticated blending:

// Multi-way blending with weights

void blendMultipleAnimations(const std::vector<uint32_t>& animationIndices,
const std::vector<float>& weights) {
if ([Link]() || [Link]() ||
[Link]() != [Link]()) {
return;
}

// Normalize weights using std::accumulate for cleaner code

float totalWeight = std::accumulate([Link](), [Link](), 0.0f);

std::vector<std::vector<glm::vec3>> allTranslations;
std::vector<std::vector<glm::quat>> allRotations;
std::vector<std::vector<glm::vec3>> allScales;

// Store original transformations

std::vector<glm::vec3> originalTranslations;
std::vector<glm::quat> originalRotations;
std::vector<glm::vec3> originalScales;

// Reserve space to avoid reallocations

[Link]([Link]());
[Link]([Link]());
[Link]([Link]());

for (auto node : [Link]) {

originalTranslations.push_back(node->translation);
originalRotations.push_back(node->rotation);
originalScales.push_back(node->scale);
}

// Collect transformations from all animations

for (uint32_t animIndex : animationIndices) {
// Reset to original state
for (size_t i = 0; i < [Link](); i++) {
[Link][i]->translation = originalTranslations[i];
[Link][i]->rotation = originalRotations[i];
[Link][i]->scale = originalScales[i];
}

357
 // Apply this animation
[Link](animIndex, 0.0f);

// Store transformations
std::vector<glm::vec3> translations;
std::vector<glm::quat> rotations;
std::vector<glm::vec3> scales;

for (auto node : [Link]) {

translations.push_back(node->translation);
rotations.push_back(node->rotation);
scales.push_back(node->scale);
}

allTranslations.push_back(std::move(translations));
allRotations.push_back(std::move(rotations));
allScales.push_back(std::move(scales));
}

// Reset to original state

for (size_t i = 0; i < [Link](); i++) {
[Link][i]->translation = originalTranslations[i];
[Link][i]->rotation = originalRotations[i];
[Link][i]->scale = originalScales[i];
}

// Apply weighted blend

for (size_t nodeIdx = 0; nodeIdx < [Link](); nodeIdx++) {
glm::vec3 blendedTranslation(0.0f);
glm::quat blendedRotation(0.0f, 0.0f, 0.0f, 0.0f);
glm::vec3 blendedScale(0.0f);

// First pass for translations and scales

for (size_t animIdx = 0; animIdx < [Link](); animIdx++) {
float normalizedWeight = weights[animIdx] / totalWeight;
blendedTranslation += allTranslations[animIdx][nodeIdx] *
normalizedWeight;
blendedScale += allScales[animIdx][nodeIdx] * normalizedWeight;
}

// Special handling for quaternions (rotations)

// We use nlerp (normalized lerp) for multiple quaternions
for (size_t animIdx = 0; animIdx < [Link](); animIdx++) {
float normalizedWeight = weights[animIdx] / totalWeight;
if (animIdx == 0) {
blendedRotation = allRotations[animIdx][nodeIdx] * normalizedWeight;
} else {
// Ensure we're interpolating along the shortest path
if (glm::dot(blendedRotation, allRotations[animIdx][nodeIdx]) < 0) {
blendedRotation += -allRotations[animIdx][nodeIdx] *

358
 normalizedWeight;
} else {
blendedRotation += allRotations[animIdx][nodeIdx] *
normalizedWeight;
}
}
}

// Normalize the resulting quaternion

blendedRotation = glm::normalize(blendedRotation);

// Apply the blended transformations

[Link][nodeIdx]->translation = blendedTranslation;
[Link][nodeIdx]->rotation = blendedRotation;
[Link][nodeIdx]->scale = blendedScale;
}
}

This more advanced implementation allows for blending between any number of animations with
different weights, which is essential for complex animation systems like locomotion or facial
expressions.

[Link]. Blend Spaces

For character movement, blend spaces are particularly useful. A blend space is a 2D or 3D space
where animations are positioned based on parameters like speed and direction:

// Simple 2D blend space for locomotion (direction + speed)

struct BlendSpaceAnimation {
uint32_t animationIndex;
float directionAngle; // In degrees, 0 = forward, 180 = backward
float speed; // In units/second
};

void updateLocomotionBlendSpace(float currentDirection, float currentSpeed) {

// Define our blend space animations
std::vector<BlendSpaceAnimation> blendSpace = {
{0, 0.0f, 0.0f}, // Idle
{1, 0.0f, 1.0f}, // Walk Forward
{2, 0.0f, 3.0f}, // Run Forward
{3, 90.0f, 1.0f}, // Walk Right
{4, 90.0f, 3.0f}, // Run Right
{5, 180.0f, 1.0f}, // Walk Backward
{6, 180.0f, 3.0f}, // Run Backward
{7, 270.0f, 1.0f}, // Walk Left
{8, 270.0f, 3.0f} // Run Left
};

// Find the closest animations and their weights

std::vector<uint32_t> animIndices;

359
 std::vector<float> weights;

// Normalize direction to 0-360 range

currentDirection = fmod(currentDirection + 360.0f, 360.0f);

// Find the 3 closest animations in the blend space

// This is a simplified approach - a real implementation would use triangulation
for (const auto& anim : blendSpace) {
float distDir = std::min(std::abs(currentDirection - [Link]),
360.0f - std::abs(currentDirection -
[Link]));
float distSpeed = std::abs(currentSpeed - [Link]);

// Calculate distance in blend space (weighted combination of direction and

speed)
float distance = std::sqrt(distDir * distDir * 0.01f + distSpeed * distSpeed);

// Use inverse distance weighting

if (distance < 0.001f) {
// If we're very close to an exact animation, just use that one
animIndices = {[Link]};
weights = {1.0f};
break;
}

float weight = 1.0f / (distance + 0.1f); // Add small epsilon to avoid

division by zero
animIndices.push_back([Link]);
weights.push_back(weight);

// Limit to 3 closest animations for performance

// Note: This works because we're inside the loop - we remove one element each
time
// the size exceeds 3, maintaining a maximum of 3 elements as we iterate
if ([Link]() > 3) {
// Find the smallest weight
auto minIt = std::min_element([Link](), [Link]());
size_t minIdx = std::distance([Link](), minIt);

// Remove the animation with the smallest weight

[Link]([Link]() + minIdx);
[Link]([Link]() + minIdx);
}
}

// Blend the selected animations

blendMultipleAnimations(animIndices, weights);
}

This blend space implementation allows for smooth transitions between different movement

360
animations based on the character’s current direction and speed.

While animation blending gives us powerful tools to combine pre-created animations, sometimes
we need to adapt animations to dynamic environments in real-time. For example, how do we make
a character’s hand precisely grab an object, or ensure feet properly plant on uneven terrain? This is
where our next technique comes in.

1.9.2. Inverse Kinematics (IK)

Inverse Kinematics complements our animation system by allowing procedural adjustments to

character poses. While the animation playback we implemented earlier uses Forward Kinematics
(calculating positions from rotations), IK works in reverse - determining the joint rotations needed
to achieve a specific end position.

[Link]. Forward vs. Inverse Kinematics

To understand IK, it helps to contrast it with Forward Kinematics:

• Forward Kinematics (FK): Given joint angles, calculate the position of the end effector

◦ Straightforward to compute

◦ Predictable and stable

◦ Used in most animation playback

• Inverse Kinematics (IK): Given a desired end effector position, calculate the joint angles

◦ More complex to compute

◦ May have multiple solutions or no solution

◦ Essential for adaptive animations and interactions

[Link]. Common IK Applications

Just as we use animation blending to create smooth transitions between predefined animations, we
use IK to adapt those animations to dynamic environments. IK enhances our animation system in
several key ways:

• Foot Placement: Remember how our animations update node transformations? With IK, we
can adjust those transformations to ensure feet properly contact uneven terrain, preventing the
"floating feet" problem common in games

• Hand Placement: Similar to our blend space example where we interpolate between different
animations, IK lets us precisely position a character’s hands to grab objects at any position

• Aiming: We can use IK to orient a character’s upper body toward a target while the lower body
follows a different animation

• Procedural Animation: IK allows us to generate new animations on-the-fly based on

environmental constraints

• Ragdoll Physics: When transitioning from animated to physics-driven movement (like when a
character falls), IK helps create realistic physical responses

361
[Link]. IK Algorithms

Just as we have different interpolation methods for animation keyframes (LINEAR, STEP,
CUBICSPLINE in our AnimationSampler), we have different algorithms for solving IK problems:

• Analytical Methods: For simple cases like two-bone chains (arm or leg), we can use closed-form
mathematical solutions - similar to how we directly interpolate between two keyframes

• Cyclic Coordinate Descent (CCD): An iterative approach that adjusts one joint at a time,
working backward from the end effector

• FABRIK (Forward And Backward Reaching Inverse Kinematics): Works by iteratively

adjusting the entire chain, often converging faster than CCD

• Jacobian Inverse: Uses matrix operations to find optimal joint adjustments for complex chains

[Link]. Implementing Two-Bone IK

The simplest and most common IK scenario involves a two-bone chain (like an arm or leg). Here’s
an implementation of the analytical two-bone IK solution:

// Two-bone IK solver
bool solveTwoBoneIK(
Node* rootNode, // The root joint (e.g., shoulder or hip)
Node* midNode, // The middle joint (e.g., elbow or knee)
Node* endNode, // The end effector (e.g., hand or foot)
const glm::vec3& targetPosition, // Target world position
const glm::vec3& hingeAxis, // Axis of rotation for the middle joint
float preferredAngle = 0.0f // Preferred angle for resolving ambiguity
) {
// Get the original global positions
glm::mat4 rootGlobal = rootNode->getGlobalMatrix();
glm::mat4 midGlobal = midNode->getGlobalMatrix();
glm::mat4 endGlobal = endNode->getGlobalMatrix();

glm::vec3 rootPos = glm::vec3(rootGlobal[3]);

glm::vec3 midPos = glm::vec3(midGlobal[3]);
glm::vec3 endPos = glm::vec3(endGlobal[3]);

// Calculate bone lengths

float bone1Length = glm::length(midPos - rootPos);
float bone2Length = glm::length(endPos - midPos);
float totalLength = bone1Length + bone2Length;

// Calculate the distance to the target

float targetDistance = glm::length(targetPosition - rootPos);

// Check if the target is reachable

if (targetDistance > totalLength) {
// Target is too far - stretch as far as possible
glm::vec3 direction = glm::normalize(targetPosition - rootPos);

362
 // Set mid node position
glm::vec3 newMidPos = rootPos + direction * bone1Length;

// Convert to local space and update node

glm::mat4 rootInv = glm::inverse(rootGlobal);
glm::vec3 localMidPos = glm::vec3(rootInv * glm::vec4(newMidPos, 1.0f));
midNode->translation = localMidPos;

// Update mid global matrix after changes

midGlobal = midNode->getGlobalMatrix();

// Set end node position

glm::vec3 newEndPos = newMidPos + direction * bone2Length;

// Convert to local space and update node

glm::mat4 midInv = glm::inverse(midGlobal);
glm::vec3 localEndPos = glm::vec3(midInv * glm::vec4(newEndPos, 1.0f));
endNode->translation = localEndPos;

return false; // Target not fully reached

}

// Target is reachable - apply cosine law to find the angles

float a = bone1Length;
float b = targetDistance;
float c = bone2Length;

// Calculate the angle between the first bone and the target direction
float cosAngle1 = (b*b + a*a - c*c) / (2*b*a);
cosAngle1 = glm::clamp(cosAngle1, -1.0f, 1.0f); // Avoid numerical errors
float angle1 = acos(cosAngle1);

// Calculate the direction to the target

glm::vec3 targetDir = glm::normalize(targetPosition - rootPos);

// Create a rotation that aligns the x-axis with the target direction
glm::vec3 xAxis(1.0f, 0.0f, 0.0f);
glm::vec3 rotAxis = glm::cross(xAxis, targetDir);

if (glm::length(rotAxis) < 0.001f) {

// Target is along the x-axis, use the up vector
rotAxis = glm::vec3(0.0f, 1.0f, 0.0f);
} else {
rotAxis = glm::normalize(rotAxis);
}

float rotAngle = acos(glm::dot(xAxis, targetDir));

glm::quat targetRot = glm::angleAxis(rotAngle, rotAxis);

// Create a rotation around the target direction by the preferred angle

glm::quat prefRot = glm::angleAxis(preferredAngle, targetDir);

363
 // Combine rotations
glm::quat finalRot = prefRot * targetRot * glm::angleAxis(angle1, hingeAxis);

// Apply the rotation to the root node

rootNode->rotation = finalRot;

// Update the mid node's global matrix after root changes

midGlobal = midNode->getGlobalMatrix();
midPos = glm::vec3(midGlobal[3]);

// Calculate the angle for the middle joint

float cosAngle2 = (a*a + c*c - b*b) / (2*a*c);
cosAngle2 = glm::clamp(cosAngle2, -1.0f, 1.0f); // Avoid numerical errors
float angle2 = acos(cosAngle2);

// The middle joint bends in the opposite direction (PI - angle2)

glm::quat midRot = glm::angleAxis(glm::pi<float>() - angle2, hingeAxis);
midNode->rotation = midRot;

return true; // Target reached

}

This implementation:

1. Calculates the positions and lengths of the bones

2. Checks if the target is reachable

3. Uses the law of cosines to calculate the necessary angles

4. Applies rotations to the joints to reach the target position

[Link]. Implementing CCD (Cyclic Coordinate Descent)

For chains with more than two bones, CCD is a popular iterative approach:

// CCD IK solver
void solveCCDIK(
std::vector<Node*> chain, // Joint chain from root to end effector
const glm::vec3& targetPosition, // Target world position
int maxIterations = 10, // Maximum iterations
float threshold = 0.01f // Distance threshold for success
) {
if ([Link]() < 2) return;

// Get the end effector

Node* endEffector = [Link]();

for (int iteration = 0; iteration < maxIterations; iteration++) {

// Get current end effector position
glm::vec3 endPos = glm::vec3(endEffector->getGlobalMatrix()[3]);

364
// Check if we're close enough to the target
if (glm::distance(endPos, targetPosition) < threshold) {
return; // Success
}

// Work backwards from the second-to-last joint to the root

for (int i = [Link]() - 2; i >= 0; i--) {
Node* currentJoint = chain[i];

// Get joint position in world space

glm::mat4 jointGlobal = currentJoint->getGlobalMatrix();
glm::vec3 jointPos = glm::vec3(jointGlobal[3]);

// Get updated end effector position

endPos = glm::vec3(endEffector->getGlobalMatrix()[3]);

// Calculate vectors from joint to end effector and target

glm::vec3 toEnd = glm::normalize(endPos - jointPos);
glm::vec3 toTarget = glm::normalize(targetPosition - jointPos);

// Calculate rotation to align the vectors

float cosAngle = glm::dot(toEnd, toTarget);
cosAngle = glm::clamp(cosAngle, -1.0f, 1.0f);

float angle = acos(cosAngle);

// If the angle is small enough, skip this joint

if (angle < 0.01f) continue;

// Calculate rotation axis

glm::vec3 rotAxis = glm::cross(toEnd, toTarget);

// Handle the case where vectors are parallel

if (glm::length(rotAxis) < 0.001f) {
// Find an arbitrary perpendicular axis
glm::vec3 tempAxis(0.0f, 1.0f, 0.0f);
if (abs(glm::dot(toEnd, tempAxis)) > 0.9f) {
tempAxis = glm::vec3(1.0f, 0.0f, 0.0f);
}
rotAxis = glm::cross(toEnd, tempAxis);
}

rotAxis = glm::normalize(rotAxis);

// Create rotation quaternion

glm::quat rotation = glm::angleAxis(angle, rotAxis);

// Apply rotation to the joint

currentJoint->rotation = rotation * currentJoint->rotation;

365
 // Check if we're close enough after this adjustment
endPos = glm::vec3(endEffector->getGlobalMatrix()[3]);
if (glm::distance(endPos, targetPosition) < threshold) {
return; // Success
}
}
}
}

This CCD implementation:

1. Iteratively processes each joint from the end effector toward the root

2. For each joint, calculates the rotation needed to bring the end effector closer to the target

3. Applies the rotation and continues to the next joint

4. Repeats until the target is reached or the maximum iterations are exhausted

[Link]. Implementing FABRIK (Forward And Backward Reaching IK)

FABRIK is another popular IK algorithm that often converges faster than CCD:

// FABRIK IK solver
void solveFABRIK(
std::vector<Node*> chain, // Joint chain from root to end effector
const glm::vec3& targetPosition, // Target world position
bool constrainRoot = true, // Whether to keep the root fixed
int maxIterations = 10, // Maximum iterations
float threshold = 0.01f // Distance threshold for success
) {
if ([Link]() < 2) return;

// Store original positions and bone lengths

std::vector<glm::vec3> positions;
std::vector<float> lengths;
glm::vec3 rootOriginalPos;

// Initialize positions and calculate lengths

for (size_t i = 0; i < [Link](); i++) {
glm::vec3 pos = glm::vec3(chain[i]->getGlobalMatrix()[3]);
positions.push_back(pos);

if (i > 0) {
lengths.push_back(glm::distance(positions[i], positions[i-1]));
}
}

rootOriginalPos = positions[0];

// Check if the target is reachable

float totalLength = 0.0f;

366
for (float length : lengths) {
totalLength += length;
}

glm::vec3 rootToTarget = targetPosition - positions[0];

float targetDistance = glm::length(rootToTarget);

if (targetDistance > totalLength) {

// Target is unreachable - stretch the chain
glm::vec3 direction = glm::normalize(rootToTarget);

// Set all joints along the line to the target

positions[0] = constrainRoot ? rootOriginalPos : positions[0];

for (size_t i = 1; i < [Link](); i++) {

positions[i] = positions[i-1] + direction * lengths[i-1];
}
} else {
// Target is reachable - apply FABRIK
for (int iteration = 0; iteration < maxIterations; iteration++) {
// Check if we're already close enough
if (glm::distance([Link](), targetPosition) < threshold) {
break;
}

// BACKWARD PASS: Set the end effector to the target and work backwards
[Link]() = targetPosition;

for (int i = [Link]() - 2; i >= 0; i--) {

// Get the direction from this joint to the next
glm::vec3 direction = glm::normalize(positions[i] - positions[i+1]);

// Set the position of this joint

positions[i] = positions[i+1] + direction * lengths[i];
}

// FORWARD PASS: Fix the root and work forwards

if (constrainRoot) {
positions[0] = rootOriginalPos;
}

for (size_t i = 0; i < [Link]() - 1; i++) {

// Get the direction from this joint to the next
glm::vec3 direction = glm::normalize(positions[i+1] - positions[i]);

// Set the position of the next joint

positions[i+1] = positions[i] + direction * lengths[i];
}

// Check if we're close enough after this iteration

if (glm::distance([Link](), targetPosition) < threshold) {

367
 break;
}
}
}

// Apply the new positions to the joints by calculating rotations

for (size_t i = 0; i < [Link]() - 1; i++) {
Node* currentJoint = chain[i];

// Calculate the original direction in local space

glm::mat4 parentGlobal = i > 0 ? chain[i-1]->getGlobalMatrix() :
glm::mat4(1.0f);
glm::mat4 localToGlobal = currentJoint->getGlobalMatrix() *
glm::inverse(parentGlobal);
glm::vec3 originalDir = glm::normalize(glm::vec3(localToGlobal *
glm::vec4(1.0f, 0.0f, 0.0f, 0.0f)));

// Calculate the new direction

glm::vec3 newDir = glm::normalize(positions[i+1] - positions[i]);

// Calculate the rotation to align the directions

float cosAngle = glm::dot(originalDir, newDir);
cosAngle = glm::clamp(cosAngle, -1.0f, 1.0f);

float angle = acos(cosAngle);

// If the angle is small, skip this joint

if (angle < 0.01f) continue;

// Calculate rotation axis

glm::vec3 rotAxis = glm::cross(originalDir, newDir);

// Handle the case where vectors are parallel

if (glm::length(rotAxis) < 0.001f) {
// Find an arbitrary perpendicular axis
glm::vec3 tempAxis(0.0f, 1.0f, 0.0f);
if (abs(glm::dot(originalDir, tempAxis)) > 0.9f) {
tempAxis = glm::vec3(1.0f, 0.0f, 0.0f);
}
rotAxis = glm::cross(originalDir, tempAxis);
}

rotAxis = glm::normalize(rotAxis);

// Create rotation quaternion

glm::quat rotation = glm::angleAxis(angle, rotAxis);

// Apply rotation to the joint

currentJoint->rotation = rotation * currentJoint->rotation;
}

368
 }

The FABRIK algorithm:

1. Works by alternating between forward and backward passes along the joint chain

2. In the backward pass, it positions joints working from the end effector toward the root

3. In the forward pass, it repositions joints from the root toward the end effector

4. This process quickly converges to a solution that satisfies the constraints

[Link]. IK Constraints

In practice, IK systems need constraints to produce realistic results:

// Apply joint constraints to a node

void applyJointConstraints(Node* node,
const glm::vec3& minAngles,
const glm::vec3& maxAngles) {
// Convert quaternion to Euler angles
glm::vec3 eulerAngles = glm::degrees(glm::eulerAngles(node->rotation));

// Apply constraints
eulerAngles.x = glm::clamp(eulerAngles.x, minAngles.x, maxAngles.x);
eulerAngles.y = glm::clamp(eulerAngles.y, minAngles.y, maxAngles.y);
eulerAngles.z = glm::clamp(eulerAngles.z, minAngles.z, maxAngles.z);

// Convert back to quaternion

glm::quat constrainedRotation = glm::quat(glm::radians(eulerAngles));

// Apply the constrained rotation

node->rotation = constrainedRotation;
}

[Link]. Integrating IK with Animation

Now that we’ve implemented several IK algorithms, let’s see how they integrate with our animation
system. Remember that our animation system updates node transformations based on keyframes,
but sometimes we need to override or adjust these transformations based on runtime conditions.
Here’s how we can blend IK adjustments with our existing animation playback:

// Apply IK on top of an animation

void applyIKToAnimation(Model* model, uint32_t animationIndex, float deltaTime,
Node* endEffector, const glm::vec3& targetPosition,
float ikWeight = 1.0f) {
// First, update the animation normally
model->updateAnimation(animationIndex, deltaTime);

// If IK weight is zero, we're done

369
 if (ikWeight <= 0.0f) return;

// Build the joint chain from end effector to root

std::vector<Node*> chain;
Node* current = endEffector;

// Add up to 3 joints to the chain (e.g., hand, elbow, shoulder)

while (current && [Link]() < 3) {
chain.push_back(current);
current = current->parent;
}

// Reverse the chain to go from root to end effector

std::reverse([Link](), [Link]());

// Store original rotations

std::vector<glm::quat> originalRotations;
for (Node* node : chain) {
originalRotations.push_back(node->rotation);
}

// Apply IK
solveTwoBoneIK(chain[0], chain[1], chain[2], targetPosition,
glm::vec3(0.0f, 0.0f, 1.0f));

// Blend between original and IK rotations based on weight

if (ikWeight < 1.0f) {
for (size_t i = 0; i < [Link](); i++) {
chain[i]->rotation = glm::slerp(originalRotations[i],
chain[i]->rotation,
ikWeight);
}
}
}

[Link]. Use Cases and Limitations

IK is powerful but comes with considerations:

• Performance: Iterative IK algorithms can be computationally expensive

• Stability: IK can produce jittery results without proper damping and constraints

• Realism: Without constraints, IK can produce physically impossible poses

• Integration: Blending IK with existing animations requires careful tuning

Despite these challenges, IK is essential for:

• Environmental Adaptation: Making characters interact with varying terrain and objects

• Procedural Animation: Generating animations that respond to dynamic conditions

370
 • Interactive Gameplay: Allowing precise control over character limbs for gameplay mechanics

1.9.3. Animation State Machines

So far, we’ve explored how to play individual animations, blend between animations, and adjust
animations with IK. But in a real game, characters often have dozens of animations that need to be
triggered based on player input and game state. How do we organize and manage all these
animations and their transitions? This is where animation state machines come in.

For complex characters, a state machine can manage transitions between animations:

enum class AnimationState {

IDLE,
WALKING,
RUNNING,
JUMPING
};

class CharacterAnimator {
private:
Model* model;
AnimationState currentState = AnimationState::IDLE;
float blendTime = 0.3f;
float currentBlend = 0.0f;

struct StateAnimation {
uint32_t animationIndex;
float speed;
bool loop;
};

std::unordered_map<AnimationState, StateAnimation> stateMap;

public:
CharacterAnimator(Model* model) : model(model) {
// Map states to animations
stateMap[AnimationState::IDLE] = {0, 1.0f, true};
stateMap[AnimationState::WALKING] = {1, 1.0f, true};
stateMap[AnimationState::RUNNING] = {2, 1.0f, true};
stateMap[AnimationState::JUMPING] = {3, 1.0f, false};
}

void setState(AnimationState newState) {

if (newState != currentState) {
// Start blending to new animation
currentBlend = 0.0f;
currentState = newState;
}
}

371
 void update(float deltaTime) {
// Handle blending if needed
if (currentBlend < blendTime) {
currentBlend += deltaTime;
float t = currentBlend / blendTime;
// Implement blending logic here
} else {
// Just update current animation
StateAnimation& anim = stateMap[currentState];
model->updateAnimation([Link], deltaTime * [Link]);
}
}
};

1.9.4. Procedural Animations

You can also create animations procedurally:

void applyProceduralAnimation(float time) {

// Find the head node
Node* headNode = nullptr;
for (auto node : [Link]) {
if (node->name == "Head") {
headNode = node;
break;
}
}

if (headNode) {
// Apply a simple bobbing motion
float bobAmount = sin(time * 2.0f) * 0.05f;
headNode->translation.y += bobAmount;

// Apply a simple looking around motion

float lookAmount = sin(time * 0.5f) * 0.2f;
glm::quat lookRotation = glm::angleAxis(lookAmount, glm::vec3(0.0f, 1.0f,
0.0f));
headNode->rotation = lookRotation * headNode->rotation;
}
}

1.10. Performance Considerations

Animations can be computationally expensive, especially with complex models. Here are some
optimization techniques:

• Level of Detail (LOD): Use simpler animations for distant objects

• Animation Culling: Don’t update animations for objects outside the view frustum

372
 • Keyframe Reduction: Reduce the number of keyframes in animations that don’t need high
precision

• Parallel Processing: Update animations in parallel using multiple threads

1.11. Conclusion
Our animation system provides a solid foundation for bringing 3D models to life. By leveraging the
glTF format and our scene graph structure, we can efficiently load, play, and blend animations to
create dynamic and engaging scenes.

In the next chapter, we’ll wrap up our exploration of the model loading system and discuss future
enhancements.

Previous: Rendering the Scene | Next: Conclusion :pp: ++

Loading Models: Conclusion

1. Conclusion
In this chapter, we’ve completed our simple engine by integrating model loading capabilities with
the architecture and camera systems developed in the previous chapters. Building upon our
knowledge of glTF from the main tutorial, we’ve implemented:

1. A hierarchical scene graph for organizing 3D objects

2. Support for glTF animations

3. A PBR material system that leverages glTF’s material properties

4. Multi-object rendering with individual transformations

This approach demonstrates how the concepts learned throughout this tutorial series can be
structured into a more reusable and extensible engine architecture. By combining the engine
architecture principles, camera transformation systems, and now model loading capabilities, we’ve
created a foundation that you can build upon for your own projects.

As you continue to develop your engine, consider exploring these advanced topics:

1. A more sophisticated material system

2. Advanced lighting techniques

3. Post-processing effects

4. Physics integration

5. Audio systems

The code for this chapter can be found in the simple_engine/20_loading_models.cpp file.

C++ code

373
Previous: Updating Animations | Next: Subsystems | Back to Building a Simple Engine :pp: ++

Loading Models: Integrating a glTF

loader with animation and PBR
1. Chapter Overview
Welcome to the third chapter of the "Building a Simple Engine" series! After exploring engine
architecture and camera systems in the previous chapters, we’ll now focus on creating a robust
model loading system that can handle modern 3D assets using the glTF format. We’ll implement a
scene graph, animation system, and PBR rendering to complete our engine foundation.

This chapter is divided into several sections to make it easier to follow:

1. Introduction - An overview of what we’ll be building and prerequisites

2. Setting Up the Project - How to structure our engine project

3. Implementing the Model Loading System - Creating the core data structures

4. Loading a glTF Model - Parsing and processing glTF files

5. Implementing PBR Rendering - Setting up physically-based rendering

6. Rendering Multiple Objects - Managing multiple model instances

7. Rendering the Scene - Drawing the scene graph

8. Updating Animations - Animating models

9. Conclusion - Summary and future directions

Each section builds upon the previous ones, so it’s recommended to follow them in order. This
chapter also builds upon the concepts introduced in the Engine Architecture and Camera
Transformations chapters. By completing this chapter, you’ll have a comprehensive foundation for
a Vulkan-based 3D engine with a well-structured architecture, camera system, and model loading
capabilities.

Back to Building a Simple Engine :pp: ++

Subsystems: Introduction
1. Introduction to Engine Subsystems
In previous chapters, we’ve built the foundation of our simple engine, implementing core
components like the rendering pipeline, camera systems, and model loading. Now, we’re ready to
expand our engine’s capabilities by adding two critical subsystems: Audio and Physics.

374
These subsystems are essential for creating immersive and interactive experiences in modern
games and simulations. While they may seem separate from the graphics pipeline we’ve been
focusing on, modern engines can leverage Vulkan’s computational power to enhance both audio
processing and physics simulations.

1.1. What We’ll Cover

This chapter will take you through implementing two crucial engine subsystems that bring games
and simulations to life. We’ll begin with an audio subsystem, starting from the fundamentals of
playing sounds and music, then advancing to sophisticated techniques like Head-Related Transfer
Function (HRTF) processing for convincing 3D spatial audio. The progression shows how Vulkan
compute shaders can transform basic audio playback into immersive soundscapes that respond to
your 3D world.

Our physics subsystem follows a similar path, beginning with essential collision detection and
response mechanisms that make objects interact believably. As we develop these foundations, we’ll
demonstrate how Vulkan’s parallel processing capabilities can accelerate physics calculations
dramatically, enabling simulations with large numbers of interacting objects that would overwhelm
traditional CPU-based approaches.

Throughout this chapter, we’ll continue our modern C++ approach from previous chapters.

1.2. Why Vulkan for Audio and Physics?

The decision to use Vulkan for audio processing and physics simulations might seem
unconventional at first, but it represents a forward-thinking approach to engine development that
leverages modern hardware capabilities.

Modern GPUs provide massive parallel processing power through thousands of cores designed for
simultaneous computation. Through Vulkan’s compute shaders, we can harness this computational
muscle for tasks far beyond graphics rendering. Audio processing benefits tremendously from
parallel operations—imagine processing dozens of simultaneous sound sources with real-time
spatial effects, or running complex physics simulations with thousands of interacting objects.

Vulkan’s unified memory model creates opportunities for efficiency that traditional separated
approaches cannot match. When graphics, audio, and physics processing share memory spaces, we
eliminate the costly data transfers that would otherwise shuttle information between CPU and GPU
repeatedly. This shared memory architecture enables sophisticated interactions—physics
simulations can directly influence particle systems, audio processing can respond to visual effects,
and all systems can work together seamlessly.

Cross-platform consistency becomes increasingly valuable as projects target multiple devices. By

implementing these subsystems through Vulkan, we maintain identical behavior across Windows,
Linux, mobile platforms, and emerging devices. This consistency reduces debugging time and
ensures that audio and physics behavior remains predictable regardless of deployment target.

The performance benefits extend beyond raw computational power. Offloading intensive
calculations to the GPU frees CPU resources for game logic, scripting, AI processing, and other tasks

375
that require sequential processing or complex branching. This separation allows each processor
type to focus on tasks it handles most efficiently.

Additionally, the intention here is to offer a perspective of using Vulkan for more than just Graphics
in your application. Our goal with this tutorial isn’t to provide you a production quality game
engine. It’s to provide you with the tools necessary to tackle any Vulkan application development
and to think critically about how your applications can benefit from the GPU.

1.3. Practical considerations: Don’t offload everything

to the GPU
While Vulkan compute can deliver impressive speedups, it’s not always advisable to offload every
subsystem to the GPU—especially on mobile:

• Mobile power and thermals: Many phones and tablets use [Link] CPU clusters and mobile
GPUs that are power/thermal constrained. Sustained heavy GPU compute can quickly lead to
thermal throttling, causing frame rate drops and inconsistent latency.

• Scheduling and latency: GPUs excel at throughput, but certain tasks (tight control loops, small
pointer-heavy updates) can prefer CPU execution due to launch overheads and scheduling
latency.

• Determinism and debugging: For gameplay-critical physics, determinism and step-by-step

debugging on the CPU can be advantageous. Consider keeping broad-phase or whole-physics on
the CPU on mobile, or use a hybrid approach (e.g., CPU broad-phase + GPU narrow-phase).

• Memory bandwidth: On integrated architectures, GPU/CPU share memory bandwidth.

Aggressively moving everything to GPU can contend with graphics and hurt frame time and
battery life.

Audio-specific guidance:

• Prefer platform audio APIs and any available dedicated audio hardware/DSP when feasible (for
mixing, resampling, effects). This path often provides lower latency, better power
characteristics, and more predictable behavior across devices.

• HRTF support in dedicated hardware is not widespread in the wild. Many consumer devices
rely on software HRTF in the OS or application. Evaluate your needs: a software HRTF pipeline
may be perfectly adequate; reserving GPU compute strictly for spatial audio is rarely necessary
unless you have many sources or complex effects.

Practical recommendations:

• Profile first: Establish CPU and GPU baselines before moving work.

• Favor hybrid designs: Offload the clearly parallel, heavy kernels (e.g., batched constraint solves,
FFT/IR convolution) while keeping control/coordination on CPU.

• Plan for mobility: Provide runtime toggles to switch between CPU/GPU paths based on device
class, thermal state, and power mode.

376
1.4. Prerequisites
This chapter builds extensively on the engine architecture and Vulkan foundations established in
previous chapters. The modular design patterns we’ve implemented become crucial when adding
subsystems that need to integrate cleanly with existing rendering, camera, and resource
management systems.

Experience with Vulkan compute shaders is essential, as we’ll leverage compute capabilities to
accelerate both audio processing and physics calculations. If you haven’t worked through the
compute shader sections in the main tutorial, review them before proceeding—the parallel
processing concepts and GPU memory management techniques translate directly to our subsystem
implementations.

A basic understanding of audio and physics concepts in game development will help you appreciate
the design decisions we make throughout the implementation. While we’ll explain the
fundamentals as we build each system, familiarity with concepts like sound attenuation, collision
detection, and rigid body dynamics will deepen your understanding of how these subsystems serve
the broader goals of interactive applications.

You should also be familiar with the following chapters from the main tutorial:

• Basic Vulkan concepts:

◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

• Compute shaders

Let’s begin by exploring how to implement a basic audio subsystem and then enhance it with
Vulkan’s computational capabilities.

Previous: Loading Models Conclusion | Next: Audio Basics :pp: ++

Subsystems: Audio Basics

1. Audio System Fundamentals
Before we dive into how Vulkan can enhance audio processing, let’s establish a foundation by
implementing a basic audio system for our engine. This will give us a framework that we can later
extend with Vulkan compute capabilities.

1.1. Audio System Architecture

A typical game audio system consists of several key components:

377
 • Audio Engine: The core component that manages audio playback, mixing, and effects
processing.

• Sound Resources: Audio files loaded into memory and prepared for playback.

• Audio Channels: Logical paths for audio to flow through, often grouped by type (e.g., music,
sound effects, dialogue).

• Spatial Audio: System for positioning sounds in 3D space relative to the listener.

• Effects Processing: Application of effects like reverb, echo, or equalization to audio streams.

Let’s implement a simple audio system that covers these basics, using a modern C++ approach
consistent with our engine’s design.

1.2. Basic Audio System Implementation

We’ll start by defining the core classes for our audio system:

// Audio.h
#pragma once

#include <string>
#include <unordered_map>
#include <memory>
#include <vector>
#include <glm/[Link]>

namespace Engine {
namespace Audio {

class AudioClip {
public:
AudioClip(const std::string& filename);
~AudioClip();

// Get raw audio data

const float* GetData() const { return m_Data.data(); }
size_t GetSampleCount() const { return m_Data.size(); }
int GetChannelCount() const { return m_ChannelCount; }
int GetSampleRate() const { return m_SampleRate; }

private:
std::vector<float> m_Data;
int m_ChannelCount;
int m_SampleRate;
};

class AudioSource {
public:
AudioSource();
~AudioSource();

378
 void SetClip(std::shared_ptr<AudioClip> clip) { m_Clip = clip; }
void SetPosition(const glm::vec3& position) { m_Position = position; }
void SetVolume(float volume) { m_Volume = volume; }
void SetLooping(bool looping) { m_Looping = looping; }

void Play();
void Stop();
void Pause();

bool IsPlaying() const { return m_IsPlaying; }

const glm::vec3& GetPosition() const { return m_Position; }

float GetVolume() const { return m_Volume; }

private:
std::shared_ptr<AudioClip> m_Clip;
glm::vec3 m_Position = glm::vec3(0.0f);
float m_Volume = 1.0f;
bool m_Looping = false;
bool m_IsPlaying = false;

// Implementation-specific playback state

size_t m_CurrentSample = 0;
};

class AudioListener {
public:
void SetPosition(const glm::vec3& position) { m_Position = position; }
void SetOrientation(const glm::vec3& forward, const glm::vec3& up) {
m_Forward = forward;
m_Up = up;
}

const glm::vec3& GetPosition() const { return m_Position; }

const glm::vec3& GetForward() const { return m_Forward; }
const glm::vec3& GetUp() const { return m_Up; }

private:
glm::vec3 m_Position = glm::vec3(0.0f);
glm::vec3 m_Forward = glm::vec3(0.0f, 0.0f, -1.0f);
glm::vec3 m_Up = glm::vec3(0.0f, 1.0f, 0.0f);
};

class AudioSystem {
public:
AudioSystem();
~AudioSystem();

void Initialize();
void Shutdown();

379
 // Update audio system (call once per frame)
void Update(float deltaTime);

// Resource management
std::shared_ptr<AudioClip> LoadClip(const std::string& name, const std::string&
filename);
std::shared_ptr<AudioClip> GetClip(const std::string& name);

// Source management
std::shared_ptr<AudioSource> CreateSource();
void DestroySource(std::shared_ptr<AudioSource> source);

// Listener (typically attached to camera)

AudioListener& GetListener() { return m_Listener; }

private:
std::unordered_map<std::string, std::shared_ptr<AudioClip>> m_Clips;
std::vector<std::shared_ptr<AudioSource>> m_Sources;
AudioListener m_Listener;

// Implementation-specific audio backend state

void* m_AudioBackend = nullptr;
};

} // namespace Audio
} // namespace Engine

This basic structure provides a foundation for loading and playing audio files with spatial
positioning. In a real implementation, you would integrate with an audio library like OpenAL,
FMOD, or Wwise to handle the low-level audio playback.

1.3. Integrating with the Engine

To integrate our audio system with the rest of our engine, we’ll add it to our engine’s main class:

// Engine.h
#include "Audio.h"

namespace Engine {

class Engine {
public:
// ... existing engine code ...

Audio::AudioSystem& GetAudioSystem() { return m_AudioSystem; }

private:
// ... existing engine members ...

380
 Audio::AudioSystem m_AudioSystem;
};

} // namespace Engine

And we’ll initialize it during engine startup:

// [Link]
void Engine::Initialize() {
// ... existing initialization code ...

m_AudioSystem.Initialize();
}

void Engine::Shutdown() {
m_AudioSystem.Shutdown();

// ... existing shutdown code ...

}

1.4. Basic Usage Example

Here’s how you might use this audio system in a game:

// Game code
void Game::LoadResources() {
// Load audio clips
auto explosionSound = m_Engine.GetAudioSystem().LoadClip("explosion",
"sounds/[Link]");
auto backgroundMusic = m_Engine.GetAudioSystem().LoadClip("music",
"sounds/[Link]");

// Create and configure audio sources

m_MusicSource = m_Engine.GetAudioSystem().CreateSource();
m_MusicSource->SetClip(backgroundMusic);
m_MusicSource->SetLooping(true);
m_MusicSource->SetVolume(0.5f);
m_MusicSource->Play();
}

void Game::OnExplosion(const glm::vec3& position) {

// Create a temporary source for the explosion sound
auto source = m_Engine.GetAudioSystem().CreateSource();
source->SetClip(m_Engine.GetAudioSystem().GetClip("explosion"));
source->SetPosition(position);
source->Play();

381
 // In a real implementation, you'd need to manage the lifetime of this source
}

void Game::Update(float deltaTime) {

// Update listener position and orientation based on camera
auto& listener = m_Engine.GetAudioSystem().GetListener();
[Link](m_Camera.GetPosition());
[Link](m_Camera.GetForward(), m_Camera.GetUp());

// Update audio system

m_Engine.GetAudioSystem().Update(deltaTime);
}

1.5. Limitations of Basic Audio Systems

While this basic audio system provides the essential functionality for playing sounds in a game, it
has several limitations:

1. Limited Spatial Audio: Basic distance-based attenuation doesn’t accurately model how sound
propagates in 3D space.

2. CPU-Intensive Processing: Effects and 3D audio calculations can consume significant CPU
resources.

3. Limited Scalability: Processing hundreds or thousands of sound sources can become a

performance bottleneck.

In the next section, we’ll explore how Vulkan compute shaders can address these limitations by
offloading audio processing to the GPU, particularly for implementing more realistic spatial audio
through Head-Related Transfer Functions (HRTF).

Previous: Introduction | Next: Vulkan for Audio Processing :pp: ++

Subsystems: Vulkan for Audio

Processing
1. Enhancing Audio with Vulkan
In the previous section, we implemented a basic audio system for our engine. Now, we’ll explore
how Vulkan’s compute capabilities can enhance audio processing, particularly for implementing
realistic 3D spatial audio using Head-Related Transfer Functions (HRTF).

1.1. Understanding HRTF

Head-Related Transfer Functions (HRTF) are a set of acoustic filters that model how sound is altered
by the head, outer ear, and torso before reaching the eardrums. These filters vary based on the

382
direction of the sound source relative to the listener.

HRTF processing allows us to create convincing 3D audio by applying the appropriate filters to
sound sources based on their position. This creates a more immersive experience than simple
stereo panning and distance attenuation.

The challenge with HRTF processing is that it’s computationally expensive:

1. Each sound source requires a unique set of filters based on its position

2. These filters must be applied to the audio stream in real-time

3. The process involves complex convolutions (multiplying audio samples with filter coefficients)

This is where Vulkan compute shaders can help by offloading these calculations to the GPU.

1.2. Why Use Vulkan for Audio Processing?

Traditional audio processing is done on the CPU, but there are several advantages to using Vulkan
compute shaders for certain audio tasks:

1. Parallelism: Audio processing, especially HRTF convolution, can be highly parallelized, making
it well-suited for GPU computation.

2. Reduced CPU Load: Offloading audio processing to the GPU frees up CPU resources for game
logic, AI, and other tasks.

3. Scalability: GPU-based processing can more easily scale to handle hundreds or thousands of
simultaneous sound sources.

4. Unified Memory: With Vulkan, we can share memory between graphics and audio processing,
reducing data transfer overhead.

1.3. Implementing HRTF Processing with Vulkan

Let’s extend our audio system to include HRTF processing using Vulkan compute shaders.

First, we’ll add HRTF-related structures to our audio system:

// Audio.h (additions)
#include <vulkan/vulkan_raii.hpp>
#include <array>

namespace Engine {
namespace Audio {

// HRTF data for a specific direction

struct HRTFData {
std::array<float, 256> leftEarImpulseResponse;
std::array<float, 256> rightEarImpulseResponse;
};

383
 // HRTF database containing filters for different directions
class HRTFDatabase {
public:
HRTFDatabase(const std::string& filename);

// Get HRTF data for a specific direction

const HRTFData& GetHRTFData(float azimuth, float elevation) const;

private:
// In a real implementation, this would be a more sophisticated data structure
std::vector<HRTFData> m_Data;
// Mapping from direction to data index
// ...
};

// Extended AudioSystem with Vulkan-based HRTF processing

class AudioSystem {
public:
// ... existing methods ...

// Enable/disable HRTF processing

void SetHRTFEnabled(bool enabled) { m_HRTFEnabled = enabled; }
bool IsHRTFEnabled() const { return m_HRTFEnabled; }

// Set the HRTF database to use

void SetHRTFDatabase(std::shared_ptr<HRTFDatabase> database) { m_HRTFDatabase =
database; }

private:
// ... existing members ...

// HRTF processing
bool m_HRTFEnabled = false;
std::shared_ptr<HRTFDatabase> m_HRTFDatabase;

// Vulkan resources for HRTF processing

struct VulkanResources {
vk::raii::ShaderModule computeShaderModule = nullptr;
vk::raii::DescriptorSetLayout descriptorSetLayout = nullptr;
vk::raii::PipelineLayout pipelineLayout = nullptr;
vk::raii::Pipeline computePipeline = nullptr;
vk::raii::DescriptorPool descriptorPool = nullptr;

// Buffers for audio data

vk::raii::Buffer inputBuffer = nullptr;
vk::raii::DeviceMemory inputBufferMemory = nullptr;
vk::raii::Buffer outputBuffer = nullptr;
vk::raii::DeviceMemory outputBufferMemory = nullptr;
vk::raii::Buffer hrtfBuffer = nullptr;
vk::raii::DeviceMemory hrtfBufferMemory = nullptr;

384
 // Descriptor sets
std::vector<vk::raii::DescriptorSet> descriptorSets;

// Command buffer for compute operations

vk::raii::CommandPool commandPool = nullptr;
vk::raii::CommandBuffer commandBuffer = nullptr;
};

VulkanResources m_VulkanResources;

// Initialize Vulkan resources for HRTF processing

void InitializeVulkanResources();
void CleanupVulkanResources();

// Process audio with HRTF using Vulkan

void ProcessAudioWithVulkan(float* inputBuffer, float* outputBuffer, size_t
frameCount);
};

} // namespace Audio
} // namespace Engine

Now, let’s implement the Vulkan-based HRTF processing:

// [Link] (implementation)

void AudioSystem::InitializeVulkanResources() {
// Get Vulkan device from the engine
auto& device = m_Engine.GetVulkanDevice();

// Create compute shader module

auto shaderCode = LoadShaderFile("shaders/hrtf_processing.[Link]");
vk::ShaderModuleCreateInfo shaderModuleCreateInfo({}, [Link]() *
sizeof(uint32_t),
reinterpret_cast<const
uint32_t*>([Link]()));
m_VulkanResources.computeShaderModule = vk::raii::ShaderModule(device,
shaderModuleCreateInfo);

// Create descriptor set layout

std::array<vk::DescriptorSetLayoutBinding, 3> bindings = {
// Input audio buffer
vk::DescriptorSetLayoutBinding(0, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute),
// Output audio buffer
vk::DescriptorSetLayoutBinding(1, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute),
// HRTF data buffer
vk::DescriptorSetLayoutBinding(2, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute)

385
 };

vk::DescriptorSetLayoutCreateInfo descriptorSetLayoutCreateInfo({}, bindings);

m_VulkanResources.descriptorSetLayout = vk::raii::DescriptorSetLayout(device,
descriptorSetLayoutCreateInfo);

// Create pipeline layout

vk::PipelineLayoutCreateInfo pipelineLayoutCreateInfo({},
*m_VulkanResources.descriptorSetLayout);
m_VulkanResources.pipelineLayout = vk::raii::PipelineLayout(device,
pipelineLayoutCreateInfo);

// Create compute pipeline

vk::PipelineShaderStageCreateInfo shaderStageCreateInfo({},
vk::ShaderStageFlagBits::eCompute,

*m_VulkanResources.computeShaderModule, "main");
vk::ComputePipelineCreateInfo computePipelineCreateInfo({}, shaderStageCreateInfo,

*m_VulkanResources.pipelineLayout);
m_VulkanResources.computePipeline = vk::raii::Pipeline(device, nullptr,
computePipelineCreateInfo);

// Create descriptor pool

std::array<vk::DescriptorPoolSize, 1> poolSizes = {
vk::DescriptorPoolSize(vk::DescriptorType::eStorageBuffer, 3)
};
vk::DescriptorPoolCreateInfo descriptorPoolCreateInfo({}, 1, poolSizes);
m_VulkanResources.descriptorPool = vk::raii::DescriptorPool(device,
descriptorPoolCreateInfo);

// Allocate descriptor sets

vk::DescriptorSetAllocateInfo
descriptorSetAllocateInfo(*m_VulkanResources.descriptorPool,
1,
&*m_VulkanResources.descriptorSetLayout);
m_VulkanResources.descriptorSets = vk::raii::DescriptorSets(device,
descriptorSetAllocateInfo);

// Create buffers for audio data

// In a real implementation, you would size these appropriately and handle
multiple frames
CreateBuffer(device, sizeof(float) * 1024,
vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.inputBuffer, m_VulkanResources.inputBufferMemory);
CreateBuffer(device, sizeof(float) * 2048,
vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.outputBuffer, m_VulkanResources.outputBufferMemory);
CreateBuffer(device, sizeof(float) * 512, vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.hrtfBuffer, m_VulkanResources.hrtfBufferMemory);

386
 // Update descriptor sets
std::array<vk::DescriptorBufferInfo, 3> bufferInfos = {
vk::DescriptorBufferInfo(*m_VulkanResources.inputBuffer, 0, VK_WHOLE_SIZE),
vk::DescriptorBufferInfo(*m_VulkanResources.outputBuffer, 0, VK_WHOLE_SIZE),
vk::DescriptorBufferInfo(*m_VulkanResources.hrtfBuffer, 0, VK_WHOLE_SIZE)
};

std::array<vk::WriteDescriptorSet, 3> descriptorWrites = {

vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 0, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[0]),
vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 1, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[1]),
vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 2, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[2])
};

[Link](descriptorWrites, {});

// Create command pool and command buffer

vk::CommandPoolCreateInfo commandPoolCreateInfo({},
m_Engine.GetVulkanQueueFamilyIndex());
m_VulkanResources.commandPool = vk::raii::CommandPool(device,
commandPoolCreateInfo);

vk::CommandBufferAllocateInfo
commandBufferAllocateInfo(*m_VulkanResources.commandPool,

vk::CommandBufferLevel::ePrimary, 1);
auto commandBuffers = vk::raii::CommandBuffers(device, commandBufferAllocateInfo);
m_VulkanResources.commandBuffer = std::move(commandBuffers[0]);
}

void AudioSystem::ProcessAudioWithVulkan(float* inputBuffer, float* outputBuffer,

size_t frameCount) {
if (!m_HRTFEnabled || !m_HRTFDatabase) {
// If HRTF is disabled, just copy input to output (or do simple stereo
panning)
memcpy(outputBuffer, inputBuffer, frameCount * sizeof(float));
return;
}

auto& device = m_Engine.GetVulkanDevice();

auto& queue = m_Engine.GetVulkanComputeQueue();

// Copy input audio data to the input buffer

void* data;
vkMapMemory(device, *m_VulkanResources.inputBufferMemory, 0, frameCount *
sizeof(float), 0, &data);

387
 memcpy(data, inputBuffer, frameCount * sizeof(float));
vkUnmapMemory(device, *m_VulkanResources.inputBufferMemory);

// Update HRTF data based on source positions

// In a real implementation, you would update this for each sound source
// For simplicity, we're just using a single HRTF filter here
const auto& hrtfData = m_HRTFDatabase->GetHRTFData(0.0f, 0.0f);
vkMapMemory(device, *m_VulkanResources.hrtfBufferMemory, 0, sizeof(HRTFData), 0,
&data);
memcpy(data, &hrtfData, sizeof(HRTFData));
vkUnmapMemory(device, *m_VulkanResources.hrtfBufferMemory);

// Record command buffer

vk::CommandBufferBeginInfo
beginInfo(vk::CommandBufferUsageFlagBits::eOneTimeSubmit);
m_VulkanResources.[Link](beginInfo);

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,
*m_VulkanResources.computePipeline);

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,

*m_VulkanResources.pipelineLayout, 0,

*m_VulkanResources.descriptorSets[0], {});

// Dispatch compute shader

// The workgroup size should match what's defined in the shader
m_VulkanResources.[Link](frameCount / 64 + 1, 1, 1);

m_VulkanResources.[Link]();

// Submit command buffer

vk::SubmitInfo submitInfo({}, {}, *m_VulkanResources.commandBuffer);
[Link](submitInfo, nullptr);
[Link]();

// Copy output audio data from the output buffer

vkMapMemory(device, *m_VulkanResources.outputBufferMemory, 0, frameCount * 2 *
sizeof(float), 0, &data);
memcpy(outputBuffer, data, frameCount * 2 * sizeof(float));
vkUnmapMemory(device, *m_VulkanResources.outputBufferMemory);
}

void AudioSystem::Update(float deltaTime) {

// Process all active audio sources
for (auto& source : m_Sources) {
if (source->IsPlaying()) {
// Get audio data from the source
auto clip = source->GetClip();
if (!clip) continue;

388
 // Calculate spatial position relative to listener
glm::vec3 relativePosition = source->GetPosition() -
m_Listener.GetPosition();

// Rotate relative position based on listener orientation

glm::mat3 listenerOrientation(
glm::cross(m_Listener.GetForward(), m_Listener.GetUp()),
m_Listener.GetUp(),
-m_Listener.GetForward()
);
relativePosition = listenerOrientation * relativePosition;

// Calculate azimuth and elevation

float distance = glm::length(relativePosition);
float azimuth = atan2(relativePosition.x, relativePosition.z);
float elevation = atan2(relativePosition.y, sqrt(relativePosition.x *
relativePosition.x + relativePosition.z * relativePosition.z));

// Get audio data from the clip

const float* audioData = clip->GetData() + source->GetCurrentSample();
size_t remainingSamples = clip->GetSampleCount() - source-
>GetCurrentSample();
size_t framesToProcess = std::min(remainingSamples, size_t(1024));

// Process audio with HRTF using Vulkan

float processedAudio[2048]; // Stereo output (2 channels)
ProcessAudioWithVulkan(const_cast<float*>(audioData), processedAudio,
framesToProcess);

// Send processed audio to the audio backend

// ...

// Update source state

source->IncrementSample(framesToProcess);
}
}
}

1.4. HRTF Compute Shader

Here’s the compute shader that performs the HRTF convolution:

// hrtf_processing.comp
#version 450

layout(local_size_x = 64, local_size_y = 1, local_size_z = 1) in;

// Input mono audio buffer

389
 layout(std430, binding = 0) buffer InputBuffer {
float samples[];
} inputBuffer;

// Output stereo audio buffer

layout(std430, binding = 1) buffer OutputBuffer {
float leftSamples[];
float rightSamples[];
} outputBuffer;

// HRTF data
layout(std430, binding = 2) buffer HRTFBuffer {
float leftImpulseResponse[256];
float rightImpulseResponse[256];
} hrtfBuffer;

void main() {
uint gID = gl_GlobalInvocationID.x;

// Check if this invocation is within the audio buffer

if (gID >= [Link]()) {
return;
}

// Perform convolution with HRTF impulse responses

float leftSample = 0.0;
float rightSample = 0.0;

for (int i = 0; i < 256; i++) {

int sampleIndex = int(gID) - i;
if (sampleIndex >= 0 && sampleIndex < [Link]()) {
leftSample += [Link][sampleIndex] *
[Link][i];
rightSample += [Link][sampleIndex] *
[Link][i];
}
}

// Write to output buffer

[Link][gID] = leftSample;
[Link][gID] = rightSample;
}

1.5. Performance Considerations

When implementing HRTF processing with Vulkan, consider these performance optimizations:

1. Batch Processing: Process multiple audio frames in a single dispatch to amortize the overhead
of command submission.

390
2. Memory Transfers: Minimize transfers between CPU and GPU memory by processing larger
chunks of audio at once.

3. Multiple Sources: Process multiple sound sources in a single shader invocation to maximize
GPU utilization.

4. Dynamic HRTF Selection: Only update HRTF filters when sound source positions change
significantly.

5. Workgroup Size: Tune the workgroup size based on your target hardware for optimal
performance.

1.6. Integration with the Audio System

To integrate the Vulkan-based HRTF processing into our audio system, we need to modify the
AudioSystem::Initialize method:

void AudioSystem::Initialize() {
// Initialize audio backend
// ...

// Initialize Vulkan resources for HRTF processing

if (m_Engine.IsVulkanInitialized()) {
InitializeVulkanResources();
}

// Load default HRTF database

m_HRTFDatabase = std::make_shared<HRTFDatabase>("data/hrtf/[Link]");
m_HRTFEnabled = true;
}

void AudioSystem::Shutdown() {
// Cleanup Vulkan resources
if (m_Engine.IsVulkanInitialized()) {
CleanupVulkanResources();
}

// Shutdown audio backend

// ...
}

1.7. Advantages of Vulkan-Based HRTF

See the core benefits listed in Why Use Vulkan for Audio Processing? for a summary of why
compute shaders are a good fit. In the context of HRTF specifically, two practical advantages are
worth highlighting:

1. Quality: You can afford higher-order HRTF filters without significant performance impact,
improving spatial realism.

391
2. Advanced Effects: The GPU’s compute power enables more sophisticated effects (e.g., room
acoustics simulation) alongside HRTF.

1.8. Limitations and Considerations

While Vulkan-based audio processing offers many advantages, there are some limitations to
consider:

1. Latency: GPU processing introduces additional latency, which may be problematic for real-time
audio.

2. Complexity: Implementing and debugging GPU-based audio processing is more complex than
CPU-based solutions.

3. Platform Support: Not all platforms support Vulkan, so you may need fallback CPU
implementations.

4. Power Consumption: GPU processing may increase power consumption, which is a

consideration for mobile devices.

1.9. Real-World Applications

Several modern game engines and audio middleware solutions are beginning to leverage GPU
acceleration for audio processing:

1. Steam Audio: Valve’s audio SDK supports GPU acceleration for its spatial audio processing.

2. Wwise: Audiokinetic’s Wwise can offload certain DSP effects to the GPU.

3. Custom Solutions: AAA game studios often implement custom GPU-accelerated audio
processing for their titles.

By implementing Vulkan-based HRTF processing in our engine, we’re following industry best
practices for high-performance audio in modern games.

In the next section, we’ll shift our focus to the physics subsystem and explore how Vulkan compute
shaders can accelerate physics simulations.

Previous: Audio Basics | Next: Physics Basics :pp: ++

Subsystems: Physics Basics

1. Physics System Fundamentals
Before we explore how Vulkan can accelerate physics simulations, let’s establish a foundation by
implementing a basic physics system for our engine. This will give us a framework that we can later
enhance with Vulkan compute capabilities.

392
1.1. Physics System Architecture
A typical game physics system consists of several key components:

• Rigid Body Dynamics: Simulation of solid objects with mass, velocity, and rotational properties.

• Collision Detection: Determining when objects intersect or contact each other.

• Collision Response: Calculating how objects should react when they collide.

• Constraints: Limiting the movement of objects based on joints, hinges, or other connections.

• Continuous Collision Detection: Handling fast-moving objects that might pass through others
between frames.

• Spatial Partitioning: Optimizing collision detection by dividing the world into regions.

Let’s implement a simple physics system that covers these basics, using a modern C++ approach
consistent with our engine’s design.

1.2. Basic Physics System Implementation

We’ll start by defining the core classes for our physics system:

// Physics.h
#pragma once

#include <vector>
#include <memory>
#include <unordered_map>
#include <string>
#include <glm/[Link]>
#include <glm/gtc/[Link]>

namespace Engine {
namespace Physics {

enum class ColliderType {

Box,
Sphere,
Capsule,
Mesh
};

class Collider {
public:
virtual ~Collider() = default;
virtual ColliderType GetType() const = 0;

void SetOffset(const glm::vec3& offset) { m_Offset = offset; }

const glm::vec3& GetOffset() const { return m_Offset; }

393
 protected:
glm::vec3 m_Offset = glm::vec3(0.0f);
};

class BoxCollider : public Collider {

public:
BoxCollider(const glm::vec3& halfExtents) : m_HalfExtents(halfExtents) {}

ColliderType GetType() const override { return ColliderType::Box; }

const glm::vec3& GetHalfExtents() const { return m_HalfExtents; }

void SetHalfExtents(const glm::vec3& halfExtents) { m_HalfExtents = halfExtents; }

private:
glm::vec3 m_HalfExtents;
};

class SphereCollider : public Collider {

public:
SphereCollider(float radius) : m_Radius(radius) {}

ColliderType GetType() const override { return ColliderType::Sphere; }

float GetRadius() const { return m_Radius; }

void SetRadius(float radius) { m_Radius = radius; }

private:
float m_Radius;
};

class RigidBody {
public:
RigidBody();
~RigidBody();

// Kinematic state
void SetPosition(const glm::vec3& position) { m_Position = position; }
void SetRotation(const glm::quat& rotation) { m_Rotation = rotation; }
void SetLinearVelocity(const glm::vec3& velocity) { m_LinearVelocity = velocity; }
void SetAngularVelocity(const glm::vec3& velocity) { m_AngularVelocity = velocity;
}

const glm::vec3& GetPosition() const { return m_Position; }

const glm::quat& GetRotation() const { return m_Rotation; }
const glm::vec3& GetLinearVelocity() const { return m_LinearVelocity; }
const glm::vec3& GetAngularVelocity() const { return m_AngularVelocity; }

// Physical properties
void SetMass(float mass);
float GetMass() const { return m_Mass; }
float GetInverseMass() const { return m_InverseMass; }

394
 void SetRestitution(float restitution) { m_Restitution = restitution; }
float GetRestitution() const { return m_Restitution; }

void SetFriction(float friction) { m_Friction = friction; }

float GetFriction() const { return m_Friction; }

// Collider management
void SetCollider(std::shared_ptr<Collider> collider) { m_Collider = collider; }
std::shared_ptr<Collider> GetCollider() const { return m_Collider; }

// Forces and impulses

void ApplyForce(const glm::vec3& force);
void ApplyImpulse(const glm::vec3& impulse);
void ApplyTorque(const glm::vec3& torque);
void ApplyTorqueImpulse(const glm::vec3& torqueImpulse);

// Simulation flags
void SetKinematic(bool kinematic) { m_IsKinematic = kinematic; }
bool IsKinematic() const { return m_IsKinematic; }

void SetGravityEnabled(bool enabled) { m_UseGravity = enabled; }

bool IsGravityEnabled() const { return m_UseGravity; }

private:
// Kinematic state
glm::vec3 m_Position = glm::vec3(0.0f);
glm::quat m_Rotation = glm::quat(1.0f, 0.0f, 0.0f, 0.0f);
glm::vec3 m_LinearVelocity = glm::vec3(0.0f);
glm::vec3 m_AngularVelocity = glm::vec3(0.0f);

// Forces
glm::vec3 m_AccumulatedForce = glm::vec3(0.0f);
glm::vec3 m_AccumulatedTorque = glm::vec3(0.0f);

// Physical properties
float m_Mass = 1.0f;
float m_InverseMass = 1.0f;
glm::mat3 m_InertiaTensor = glm::mat3(1.0f);
glm::mat3 m_InverseInertiaTensor = glm::mat3(1.0f);
float m_Restitution = 0.5f;
float m_Friction = 0.5f;

// Collision
std::shared_ptr<Collider> m_Collider;

// Flags
bool m_IsKinematic = false;
bool m_UseGravity = true;

// Update inertia tensor based on mass and collider

395
 void UpdateInertiaTensor();

friend class PhysicsSystem;

};

struct CollisionInfo {
std::shared_ptr<RigidBody> bodyA;
std::shared_ptr<RigidBody> bodyB;
glm::vec3 contactPoint;
glm::vec3 normal;
float penetrationDepth;
};

class PhysicsSystem {
public:
PhysicsSystem();
~PhysicsSystem();

void Initialize();
void Shutdown();

// Update physics simulation

void Update(float deltaTime);

// RigidBody management
std::shared_ptr<RigidBody> CreateRigidBody();
void DestroyRigidBody(std::shared_ptr<RigidBody> body);

// World settings
void SetGravity(const glm::vec3& gravity) { m_Gravity = gravity; }
const glm::vec3& GetGravity() const { return m_Gravity; }

// Collision detection
bool Raycast(const glm::vec3& origin, const glm::vec3& direction, float
maxDistance, RaycastHit& hit);

private:
std::vector<std::shared_ptr<RigidBody>> m_RigidBodies;
glm::vec3 m_Gravity = glm::vec3(0.0f, -9.81f, 0.0f);

// Simulation steps
void IntegrateForces(RigidBody& body, float deltaTime);
void IntegrateVelocities(RigidBody& body, float deltaTime);

// Collision detection and response

void DetectCollisions(std::vector<CollisionInfo>& collisions);
void ResolveCollisions(std::vector<CollisionInfo>& collisions);

// Helper functions for collision detection

bool CheckCollision(const RigidBody& bodyA, const RigidBody& bodyB, CollisionInfo&
info);

396
 bool SphereVsSphere(const RigidBody& bodyA, const RigidBody& bodyB, CollisionInfo&
info);
bool BoxVsBox(const RigidBody& bodyA, const RigidBody& bodyB, CollisionInfo&
info);
bool SphereVsBox(const RigidBody& bodyA, const RigidBody& bodyB, CollisionInfo&
info);
};

struct RaycastHit {
std::shared_ptr<RigidBody> body;
glm::vec3 point;
glm::vec3 normal;
float distance;
};

} // namespace Physics
} // namespace Engine

This basic structure provides a foundation for simulating rigid body physics with collision detection
and response. In a real implementation, you would likely use a physics library like Bullet, PhysX, or
Havok for more advanced features and optimizations.

1.3. Integrating with the Engine

To integrate our physics system with the rest of our engine, we’ll add it to our engine’s main class:

// Engine.h
#include "Physics.h"

namespace Engine {

class Engine {
public:
// ... existing engine code ...

Physics::PhysicsSystem& GetPhysicsSystem() { return m_PhysicsSystem; }

private:
// ... existing engine members ...

Physics::PhysicsSystem m_PhysicsSystem;
};

} // namespace Engine

And we’ll initialize it during engine startup:

// [Link]

397
 void Engine::Initialize() {
// ... existing initialization code ...

m_PhysicsSystem.Initialize();
}

void Engine::Shutdown() {
m_PhysicsSystem.Shutdown();

// ... existing shutdown code ...

}

1.4. Basic Implementation of Physics Simulation

To keep the update loop easy to follow, think of a fixed‑timestep frame as six steps:

1) Accumulate forces (e.g., gravity, user forces) 2) Integrate forces (update velocities with damping)
3) Detect collisions (broad/narrow checks per pair) 4) Resolve collisions (impulses + positional
correction) 5) Integrate velocities (update positions and orientations) 6) Clear forces (prepare for
next step)

Let’s implement the core physics simulation functions:

// [Link]
#include "Physics.h"

namespace Engine {
namespace Physics {

void PhysicsSystem::Update(float deltaTime) {

// Fixed timestep for stability
const float fixedTimeStep = 1.0f / 60.0f;

// Accumulate forces (e.g., gravity)

for (auto& body : m_RigidBodies) {
if (!body->IsKinematic() && body->IsGravityEnabled()) {
body->m_AccumulatedForce += m_Gravity * body->m_Mass;
}
}

// Integrate forces
for (auto& body : m_RigidBodies) {
if (!body->IsKinematic()) {
IntegrateForces(*body, fixedTimeStep);
}
}

// Detect and resolve collisions

std::vector<CollisionInfo> collisions;

398
 DetectCollisions(collisions);
ResolveCollisions(collisions);

// Integrate velocities
for (auto& body : m_RigidBodies) {
if (!body->IsKinematic()) {
IntegrateVelocities(*body, fixedTimeStep);
}
}

// Clear accumulated forces

for (auto& body : m_RigidBodies) {
body->m_AccumulatedForce = glm::vec3(0.0f);
body->m_AccumulatedTorque = glm::vec3(0.0f);
}
}

void PhysicsSystem::IntegrateForces(RigidBody& body, float deltaTime) {

// Update linear velocity
body.m_LinearVelocity += (body.m_AccumulatedForce * body.m_InverseMass) *
deltaTime;

// Update angular velocity

body.m_AngularVelocity += glm::vec3(body.m_InverseInertiaTensor *
glm::vec4(body.m_AccumulatedTorque, 0.0f)) * deltaTime;

// Apply damping
const float linearDamping = 0.01f;
const float angularDamping = 0.01f;
body.m_LinearVelocity *= (1.0f - linearDamping);
body.m_AngularVelocity *= (1.0f - angularDamping);
}

void PhysicsSystem::IntegrateVelocities(RigidBody& body, float deltaTime) {

// Update position
body.m_Position += body.m_LinearVelocity * deltaTime;

// Update rotation
glm::quat angularVelocityQuat(0.0f, body.m_AngularVelocity.x,
body.m_AngularVelocity.y, body.m_AngularVelocity.z);
body.m_Rotation += (angularVelocityQuat * body.m_Rotation) * 0.5f * deltaTime;
body.m_Rotation = glm::normalize(body.m_Rotation);
}

void PhysicsSystem::DetectCollisions(std::vector<CollisionInfo>& collisions) {

// Simple O(n²) collision detection
for (size_t i = 0; i < m_RigidBodies.size(); i++) {
for (size_t j = i + 1; j < m_RigidBodies.size(); j++) {
auto& bodyA = m_RigidBodies[i];
auto& bodyB = m_RigidBodies[j];

399
 // Skip if both bodies are kinematic
if (bodyA->IsKinematic() && bodyB->IsKinematic()) {
continue;
}

// Skip if either body doesn't have a collider

if (!bodyA->GetCollider() || !bodyB->GetCollider()) {
continue;
}

CollisionInfo info;
if (CheckCollision(*bodyA, *bodyB, info)) {
[Link] = bodyA;
[Link] = bodyB;
collisions.push_back(info);
}
}
}
}

void PhysicsSystem::ResolveCollisions(std::vector<CollisionInfo>& collisions) {

for (auto& collision : collisions) {
auto bodyA = [Link];
auto bodyB = [Link];

// Calculate relative velocity

glm::vec3 relativeVelocity = bodyB->m_LinearVelocity - bodyA-
>m_LinearVelocity;

// Calculate impulse magnitude

float velocityAlongNormal = glm::dot(relativeVelocity, [Link]);

// Don't resolve if velocities are separating

if (velocityAlongNormal > 0) {
continue;
}

// Calculate restitution (bounciness)

float restitution = std::min(bodyA->m_Restitution, bodyB->m_Restitution);

// Calculate impulse scalar

float j = -(1.0f + restitution) * velocityAlongNormal;
j /= bodyA->m_InverseMass + bodyB->m_InverseMass;

// Apply impulse
glm::vec3 impulse = [Link] * j;

if (!bodyA->IsKinematic()) {
bodyA->m_LinearVelocity -= impulse * bodyA->m_InverseMass;
}

400
 if (!bodyB->IsKinematic()) {
bodyB->m_LinearVelocity += impulse * bodyB->m_InverseMass;
}

// Resolve penetration (position correction)

const float percent = 0.2f; // usually 20% to 80%
const float slop = 0.01f; // small penetration allowed
glm::vec3 correction = std::max([Link] - slop, 0.0f) *
percent * [Link] / (bodyA->m_InverseMass + bodyB->m_InverseMass);

if (!bodyA->IsKinematic()) {
bodyA->m_Position -= correction * bodyA->m_InverseMass;
}

if (!bodyB->IsKinematic()) {
bodyB->m_Position += correction * bodyB->m_InverseMass;
}
}
}

bool PhysicsSystem::CheckCollision(const RigidBody& bodyA, const RigidBody& bodyB,

CollisionInfo& info) {
auto colliderA = [Link]();
auto colliderB = [Link]();

if (colliderA->GetType() == ColliderType::Sphere && colliderB->GetType() ==

ColliderType::Sphere) {
return SphereVsSphere(bodyA, bodyB, info);
}
else if (colliderA->GetType() == ColliderType::Box && colliderB->GetType() ==
ColliderType::Box) {
return BoxVsBox(bodyA, bodyB, info);
}
else if (colliderA->GetType() == ColliderType::Sphere && colliderB->GetType() ==
ColliderType::Box) {
return SphereVsBox(bodyA, bodyB, info);
}
else if (colliderA->GetType() == ColliderType::Box && colliderB->GetType() ==
ColliderType::Sphere) {
bool result = SphereVsBox(bodyB, bodyA, info);
if (result) {
// Flip normal direction
[Link] = -[Link];
}
return result;
}

// Unsupported collision types

return false;
}

401
 bool PhysicsSystem::SphereVsSphere(const RigidBody& bodyA, const RigidBody& bodyB,
CollisionInfo& info) {
auto sphereA = std::static_pointer_cast<SphereCollider>([Link]());
auto sphereB = std::static_pointer_cast<SphereCollider>([Link]());

glm::vec3 posA = [Link]() + sphereA->GetOffset();

glm::vec3 posB = [Link]() + sphereB->GetOffset();

float radiusA = sphereA->GetRadius();

float radiusB = sphereB->GetRadius();

glm::vec3 direction = posB - posA;

float distance = glm::length(direction);
float minDistance = radiusA + radiusB;

if (distance >= minDistance) {

return false;
}

// Normalize direction
direction = distance > 0.0001f ? direction / distance : glm::vec3(0, 1, 0);

[Link] = posA + direction * radiusA;

[Link] = direction;
[Link] = minDistance - distance;

return true;
}

// Implementation of BoxVsBox and SphereVsBox collision detection would go here

// These are more complex and would require additional helper functions

} // namespace Physics
} // namespace Engine

1.5. Basic Usage Example

Here’s how you might use this physics system in a game:

// Game code
void Game::Initialize() {
// Create a ground plane
auto ground = m_Engine.GetPhysicsSystem().CreateRigidBody();
ground->SetPosition(glm::vec3(0.0f, -1.0f, 0.0f));
ground->SetKinematic(true); // Static object
auto groundCollider = std::make_shared<Physics::BoxCollider>(glm::vec3(50.0f,
1.0f, 50.0f));
ground->SetCollider(groundCollider);

402
 // Create a dynamic box
auto box = m_Engine.GetPhysicsSystem().CreateRigidBody();
box->SetPosition(glm::vec3(0.0f, 5.0f, 0.0f));
box->SetMass(1.0f);
auto boxCollider = std::make_shared<Physics::BoxCollider>(glm::vec3(0.5f, 0.5f,
0.5f));
box->SetCollider(boxCollider);

// Create a dynamic sphere

auto sphere = m_Engine.GetPhysicsSystem().CreateRigidBody();
sphere->SetPosition(glm::vec3(1.0f, 10.0f, 0.0f));
sphere->SetMass(2.0f);
auto sphereCollider = std::make_shared<Physics::SphereCollider>(0.7f);
sphere->SetCollider(sphereCollider);

// Store references to our objects

m_PhysicsObjects.push_back(ground);
m_PhysicsObjects.push_back(box);
m_PhysicsObjects.push_back(sphere);
}

void Game::Update(float deltaTime) {

// Update physics
m_Engine.GetPhysicsSystem().Update(deltaTime);

// Update visual representations of physics objects

for (auto& physicsObject : m_PhysicsObjects) {
auto visualObject = m_PhysicsToVisualMap[physicsObject];
if (visualObject) {
visualObject->SetPosition(physicsObject->GetPosition());
visualObject->SetRotation(physicsObject->GetRotation());
}
}
}

void Game::OnExplosion(const glm::vec3& position, float force) {

// Apply radial impulse to nearby objects
for (auto& physicsObject : m_PhysicsObjects) {
if (!physicsObject->IsKinematic()) {
glm::vec3 direction = physicsObject->GetPosition() - position;
float distance = glm::length(direction);

if (distance < 10.0f) {

direction = glm::normalize(direction);
float impulseMagnitude = force * (1.0f - distance / 10.0f);
physicsObject->ApplyImpulse(direction * impulseMagnitude);
}
}
}
}

403
1.6. Limitations of Basic Physics Systems
While this basic physics system provides the essential functionality for simulating rigid bodies in a
game, it has several limitations:

1. Performance: The O(n²) collision detection becomes a bottleneck with many objects.

2. Limited Collision Shapes: We’ve only implemented basic shapes like boxes and spheres.

3. Stability Issues: Simple integrators and collision resolution can lead to instability.

4. No Continuous Collision Detection: Fast-moving objects might tunnel through thin obstacles.

5. Limited Constraints: We haven’t implemented joints, springs, or other constraints.

6. CPU-Bound Processing: All calculations are performed on the CPU, limiting scalability.

In the next section, we’ll explore how Vulkan compute shaders can address these limitations by
offloading physics calculations to the GPU, particularly for large-scale simulations with many
objects.

Previous: Vulkan for Audio Processing | Next: Vulkan for Physics Simulation :pp: ++

Subsystems: Vulkan for Physics

Simulation
1. Enhancing Physics with Vulkan
In the previous section, we implemented a basic physics system for our engine. Now, we’ll explore
how Vulkan’s compute capabilities can enhance physics simulations, particularly for large-scale
scenarios with many interacting objects.

1.1. Why Use Vulkan for Physics?

Traditional physics simulations are performed on the CPU, but there are several compelling reasons
to leverage Vulkan compute shaders for physics calculations:

1. Parallelism: Physics calculations for multiple objects can be performed in parallel, making
them well-suited for GPU computation.

2. Scalability: GPU-based physics can handle thousands or even millions of objects with relatively
little performance degradation.

3. Reduced CPU Load: Offloading physics to the GPU frees up CPU resources for game logic, AI,
and other tasks.

4. Unified Memory: With Vulkan, we can share memory between physics and graphics, reducing
data transfer overhead.

5. Specialized Hardware: Modern GPUs often include hardware features specifically designed to

404
 accelerate physics-like calculations.

1.2. Common GPU Physics Applications

While not all physics calculations are suitable for GPU acceleration, several common physics tasks
can benefit significantly:

1. Particle Systems: Simulating thousands of particles for effects like smoke, fire, or fluid.

2. Cloth Simulation: Calculating the behavior of cloth, hair, or other deformable objects.

3. Soft Body Physics: Simulating objects that can bend, stretch, or compress.

4. Broad-Phase Collision Detection: Quickly identifying potential collision pairs among many
objects.

5. Rigid Body Dynamics: Simulating the movement of large numbers of rigid bodies.

Let’s focus on implementing GPU-accelerated rigid body dynamics and collision detection using
Vulkan compute shaders.

1.3. GPU-Accelerated Rigid Body Physics

To implement GPU-accelerated physics, we’ll need to:

1. Store physics data in GPU-accessible buffers

2. Create compute shaders to perform physics calculations

3. Integrate the GPU physics with our existing CPU-based system

Let’s extend our physics system to include Vulkan-accelerated components. We’ll approach it in
four steps:

1) Step 1: Data layout (GPUPhysicsData/GPUCollisionData structures) 2) Step 2: GPU resource setup

(descriptor set layout, pipelines, storage buffers, descriptor sets) 3) Step 3: Simulation dispatch
(integrate → broad‑phase → narrow‑phase → resolve with pipeline barriers) 4) Step 4:
Synchronization and readback (update GPU buffers, submit, read back state, integrate in Update)

We avoid repeating Vulkan compute fundamentals here; focus stays on

physics‑specific wiring. Use earlier chapters (Resource Management, Rendering
 Pipeline) or the Vulkan Guide ([Link] if you need a
refresher on descriptors, buffers, or pipeline creation.

// Physics.h (additions)
#include <vulkan/vulkan_raii.hpp>

namespace Engine {
namespace Physics {

// Structure for GPU physics data

struct GPUPhysicsData {

405
 glm::vec4 position; // xyz = position, w = inverse mass
glm::vec4 rotation; // quaternion
glm::vec4 linearVelocity; // xyz = velocity, w = restitution
glm::vec4 angularVelocity; // xyz = angular velocity, w = friction
glm::vec4 force; // xyz = force, w = is kinematic (0 or 1)
glm::vec4 torque; // xyz = torque, w = use gravity (0 or 1)
glm::vec4 colliderData; // type-specific data (e.g., radius for spheres)
glm::vec4 colliderData2; // additional collider data (e.g., box half extents)
};

// Structure for GPU collision data

struct GPUCollisionData {
uint32_t bodyA;
uint32_t bodyB;
glm::vec4 contactNormal; // xyz = normal, w = penetration depth
glm::vec4 contactPoint; // xyz = contact point, w = unused
};

// Extended PhysicsSystem with Vulkan acceleration

class PhysicsSystem {
public:
// ... existing methods ...

// Enable/disable GPU acceleration

void SetGPUAccelerationEnabled(bool enabled) { m_GPUAccelerationEnabled = enabled;
}
bool IsGPUAccelerationEnabled() const { return m_GPUAccelerationEnabled; }

// Set the maximum number of objects that can be simulated on the GPU
void SetMaxGPUObjects(uint32_t maxObjects);

private:
// ... existing members ...

// GPU acceleration
bool m_GPUAccelerationEnabled = false;
uint32_t m_MaxGPUObjects = 1024;
uint32_t m_MaxGPUCollisions = 4096;

// Vulkan resources for physics simulation

struct VulkanResources {
// Shader modules
vk::raii::ShaderModule integrateShaderModule = nullptr;
vk::raii::ShaderModule broadPhaseShaderModule = nullptr;
vk::raii::ShaderModule narrowPhaseShaderModule = nullptr;
vk::raii::ShaderModule resolveShaderModule = nullptr;

// Pipeline layouts and compute pipelines

vk::raii::DescriptorSetLayout descriptorSetLayout = nullptr;
vk::raii::PipelineLayout pipelineLayout = nullptr;
vk::raii::Pipeline integratePipeline = nullptr;

406
 vk::raii::Pipeline broadPhasePipeline = nullptr;
vk::raii::Pipeline narrowPhasePipeline = nullptr;
vk::raii::Pipeline resolvePipeline = nullptr;

// Descriptor pool and sets

vk::raii::DescriptorPool descriptorPool = nullptr;
std::vector<vk::raii::DescriptorSet> descriptorSets;

// Buffers for physics data

vk::raii::Buffer physicsBuffer = nullptr;
vk::raii::DeviceMemory physicsBufferMemory = nullptr;
vk::raii::Buffer collisionBuffer = nullptr;
vk::raii::DeviceMemory collisionBufferMemory = nullptr;
vk::raii::Buffer pairBuffer = nullptr;
vk::raii::DeviceMemory pairBufferMemory = nullptr;
vk::raii::Buffer counterBuffer = nullptr;
vk::raii::DeviceMemory counterBufferMemory = nullptr;

// Command buffer for compute operations

vk::raii::CommandPool commandPool = nullptr;
vk::raii::CommandBuffer commandBuffer = nullptr;
};

VulkanResources m_VulkanResources;

// Initialize Vulkan resources for physics simulation

void InitializeVulkanResources();
void CleanupVulkanResources();

// Update physics data on the GPU

void UpdateGPUPhysicsData();

// Read back physics data from the GPU

void ReadbackGPUPhysicsData();

// Perform GPU-accelerated physics simulation

void SimulatePhysicsOnGPU(float deltaTime);
};

} // namespace Physics
} // namespace Engine

Now, let’s implement the Vulkan-based physics simulation:

// [Link] (implementation)

void PhysicsSystem::InitializeVulkanResources() {
// Get Vulkan device from the engine
auto& device = m_Engine.GetVulkanDevice();

407
 // Create compute shader modules
auto integrateShaderCode = LoadShaderFile("shaders/physics_integrate.[Link]");
vk::ShaderModuleCreateInfo integrateShaderModuleCreateInfo({},
[Link]() * sizeof(uint32_t),
reinterpret_cast<const
uint32_t*>([Link]()));
m_VulkanResources.integrateShaderModule = vk::raii::ShaderModule(device,
integrateShaderModuleCreateInfo);

auto broadPhaseShaderCode =
LoadShaderFile("shaders/physics_broad_phase.[Link]");
vk::ShaderModuleCreateInfo broadPhaseShaderModuleCreateInfo({},
[Link]() * sizeof(uint32_t),
reinterpret_cast<const
uint32_t*>([Link]()));
m_VulkanResources.broadPhaseShaderModule = vk::raii::ShaderModule(device,
broadPhaseShaderModuleCreateInfo);

auto narrowPhaseShaderCode =
LoadShaderFile("shaders/physics_narrow_phase.[Link]");
vk::ShaderModuleCreateInfo narrowPhaseShaderModuleCreateInfo({},
[Link]() * sizeof(uint32_t),
reinterpret_cast<const
uint32_t*>([Link]()));
m_VulkanResources.narrowPhaseShaderModule = vk::raii::ShaderModule(device,
narrowPhaseShaderModuleCreateInfo);

auto resolveShaderCode = LoadShaderFile("shaders/physics_resolve.[Link]");

vk::ShaderModuleCreateInfo resolveShaderModuleCreateInfo({},
[Link]() * sizeof(uint32_t),
reinterpret_cast<const
uint32_t*>([Link]()));
m_VulkanResources.resolveShaderModule = vk::raii::ShaderModule(device,
resolveShaderModuleCreateInfo);

// Create descriptor set layout

std::array<vk::DescriptorSetLayoutBinding, 4> bindings = {
// Physics data buffer
vk::DescriptorSetLayoutBinding(0, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute),
// Collision data buffer
vk::DescriptorSetLayoutBinding(1, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute),
// Pair buffer (for broad phase)
vk::DescriptorSetLayoutBinding(2, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute),
// Counter buffer
vk::DescriptorSetLayoutBinding(3, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute)
};

408
 vk::DescriptorSetLayoutCreateInfo descriptorSetLayoutCreateInfo({}, bindings);
m_VulkanResources.descriptorSetLayout = vk::raii::DescriptorSetLayout(device,
descriptorSetLayoutCreateInfo);

// Create pipeline layout

vk::PipelineLayoutCreateInfo pipelineLayoutCreateInfo({},
*m_VulkanResources.descriptorSetLayout);
m_VulkanResources.pipelineLayout = vk::raii::PipelineLayout(device,
pipelineLayoutCreateInfo);

// Create compute pipelines

vk::PipelineShaderStageCreateInfo integrateShaderStageCreateInfo({},
vk::ShaderStageFlagBits::eCompute,

*m_VulkanResources.integrateShaderModule, "main");
vk::ComputePipelineCreateInfo integrateComputePipelineCreateInfo({},
integrateShaderStageCreateInfo,

*m_VulkanResources.pipelineLayout);
m_VulkanResources.integratePipeline = vk::raii::Pipeline(device, nullptr,
integrateComputePipelineCreateInfo);

vk::PipelineShaderStageCreateInfo broadPhaseShaderStageCreateInfo({},
vk::ShaderStageFlagBits::eCompute,

*m_VulkanResources.broadPhaseShaderModule, "main");
vk::ComputePipelineCreateInfo broadPhaseComputePipelineCreateInfo({},
broadPhaseShaderStageCreateInfo,

*m_VulkanResources.pipelineLayout);
m_VulkanResources.broadPhasePipeline = vk::raii::Pipeline(device, nullptr,
broadPhaseComputePipelineCreateInfo);

vk::PipelineShaderStageCreateInfo narrowPhaseShaderStageCreateInfo({},
vk::ShaderStageFlagBits::eCompute,

*m_VulkanResources.narrowPhaseShaderModule, "main");
vk::ComputePipelineCreateInfo narrowPhaseComputePipelineCreateInfo({},
narrowPhaseShaderStageCreateInfo,

*m_VulkanResources.pipelineLayout);
m_VulkanResources.narrowPhasePipeline = vk::raii::Pipeline(device, nullptr,
narrowPhaseComputePipelineCreateInfo);

vk::PipelineShaderStageCreateInfo resolveShaderStageCreateInfo({},
vk::ShaderStageFlagBits::eCompute,

*m_VulkanResources.resolveShaderModule, "main");
vk::ComputePipelineCreateInfo resolveComputePipelineCreateInfo({},
resolveShaderStageCreateInfo,

409
 *m_VulkanResources.pipelineLayout);
m_VulkanResources.resolvePipeline = vk::raii::Pipeline(device, nullptr,
resolveComputePipelineCreateInfo);

// Create descriptor pool

std::array<vk::DescriptorPoolSize, 1> poolSizes = {
vk::DescriptorPoolSize(vk::DescriptorType::eStorageBuffer, 4)
};
vk::DescriptorPoolCreateInfo descriptorPoolCreateInfo({}, 1, poolSizes);
m_VulkanResources.descriptorPool = vk::raii::DescriptorPool(device,
descriptorPoolCreateInfo);

// Allocate descriptor sets

vk::DescriptorSetAllocateInfo
descriptorSetAllocateInfo(*m_VulkanResources.descriptorPool,
1,
&*m_VulkanResources.descriptorSetLayout);
m_VulkanResources.descriptorSets = vk::raii::DescriptorSets(device,
descriptorSetAllocateInfo);

// Create buffers for physics data

CreateBuffer(device, sizeof(GPUPhysicsData) * m_MaxGPUObjects,
vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.physicsBuffer,
m_VulkanResources.physicsBufferMemory);

CreateBuffer(device, sizeof(GPUCollisionData) * m_MaxGPUCollisions,

vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.collisionBuffer,
m_VulkanResources.collisionBufferMemory);

CreateBuffer(device, sizeof(uint32_t) * 2 * m_MaxGPUCollisions,

vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.pairBuffer, m_VulkanResources.pairBufferMemory);

CreateBuffer(device, sizeof(uint32_t) * 2,
vk::BufferUsageFlagBits::eStorageBuffer,
m_VulkanResources.counterBuffer,
m_VulkanResources.counterBufferMemory);

// Update descriptor sets

std::array<vk::DescriptorBufferInfo, 4> bufferInfos = {
vk::DescriptorBufferInfo(*m_VulkanResources.physicsBuffer, 0, VK_WHOLE_SIZE),
vk::DescriptorBufferInfo(*m_VulkanResources.collisionBuffer, 0,
VK_WHOLE_SIZE),
vk::DescriptorBufferInfo(*m_VulkanResources.pairBuffer, 0, VK_WHOLE_SIZE),
vk::DescriptorBufferInfo(*m_VulkanResources.counterBuffer, 0, VK_WHOLE_SIZE)
};

std::array<vk::WriteDescriptorSet, 4> descriptorWrites = {

vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 0, 0, 1,

410
 vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[0]),
vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 1, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[1]),
vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 2, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[2]),
vk::WriteDescriptorSet(*m_VulkanResources.descriptorSets[0], 3, 0, 1,
vk::DescriptorType::eStorageBuffer, nullptr,
&bufferInfos[3])
};

[Link](descriptorWrites, {});

// Create command pool and command buffer

vk::CommandPoolCreateInfo commandPoolCreateInfo({},
m_Engine.GetVulkanQueueFamilyIndex());
m_VulkanResources.commandPool = vk::raii::CommandPool(device,
commandPoolCreateInfo);

vk::CommandBufferAllocateInfo
commandBufferAllocateInfo(*m_VulkanResources.commandPool,

vk::CommandBufferLevel::ePrimary, 1);
auto commandBuffers = vk::raii::CommandBuffers(device, commandBufferAllocateInfo);
m_VulkanResources.commandBuffer = std::move(commandBuffers[0]);

// Initialize counter buffer

uint32_t initialCounters[2] = { 0, 0 }; // [0] = pair count, [1] = collision count
void* data;
vkMapMemory(device, *m_VulkanResources.counterBufferMemory, 0,
sizeof(initialCounters), 0, &data);
memcpy(data, initialCounters, sizeof(initialCounters));
vkUnmapMemory(device, *m_VulkanResources.counterBufferMemory);
}

void PhysicsSystem::UpdateGPUPhysicsData() {
auto& device = m_Engine.GetVulkanDevice();

// Map the physics buffer

void* data;
vkMapMemory(device, *m_VulkanResources.physicsBufferMemory, 0,
sizeof(GPUPhysicsData) * m_RigidBodies.size(), 0, &data);

// Copy physics data to the buffer

GPUPhysicsData* gpuData = static_cast<GPUPhysicsData*>(data);
for (size_t i = 0; i < m_RigidBodies.size(); i++) {
auto& body = m_RigidBodies[i];

gpuData[i].position = glm::vec4(body->GetPosition(), body->GetInverseMass());

411
 gpuData[i].rotation = glm::vec4(body->GetRotation().x, body->GetRotation().y,
body->GetRotation().z, body->GetRotation().w);
gpuData[i].linearVelocity = glm::vec4(body->GetLinearVelocity(), body-
>GetRestitution());
gpuData[i].angularVelocity = glm::vec4(body->GetAngularVelocity(), body-
>GetFriction());
gpuData[i].force = glm::vec4(body->m_AccumulatedForce, body->IsKinematic() ?
1.0f : 0.0f);
gpuData[i].torque = glm::vec4(body->m_AccumulatedTorque, body-
>IsGravityEnabled() ? 1.0f : 0.0f);

// Set collider data based on collider type

auto collider = body->GetCollider();
if (collider) {
switch (collider->GetType()) {
case ColliderType::Sphere: {
auto sphereCollider =
std::static_pointer_cast<SphereCollider>(collider);
gpuData[i].colliderData = glm::vec4(sphereCollider->GetRadius(),
0.0f, 0.0f,

static_cast<float>(ColliderType::Sphere));
gpuData[i].colliderData2 = glm::vec4(collider->GetOffset(), 0.0f);
break;
}
case ColliderType::Box: {
auto boxCollider =
std::static_pointer_cast<BoxCollider>(collider);
gpuData[i].colliderData = glm::vec4(boxCollider->GetHalfExtents(),

static_cast<float>(ColliderType::Box));
gpuData[i].colliderData2 = glm::vec4(collider->GetOffset(), 0.0f);
break;
}
default:
// Unsupported collider type
gpuData[i].colliderData = glm::vec4(0.0f, 0.0f, 0.0f, -1.0f);
gpuData[i].colliderData2 = glm::vec4(0.0f);
break;
}
} else {
// No collider
gpuData[i].colliderData = glm::vec4(0.0f, 0.0f, 0.0f, -1.0f);
gpuData[i].colliderData2 = glm::vec4(0.0f);
}
}

vkUnmapMemory(device, *m_VulkanResources.physicsBufferMemory);

// Reset counters
uint32_t initialCounters[2] = { 0, 0 }; // [0] = pair count, [1] = collision count

412
 vkMapMemory(device, *m_VulkanResources.counterBufferMemory, 0,
sizeof(initialCounters), 0, &data);
memcpy(data, initialCounters, sizeof(initialCounters));
vkUnmapMemory(device, *m_VulkanResources.counterBufferMemory);
}

void PhysicsSystem::ReadbackGPUPhysicsData() {
auto& device = m_Engine.GetVulkanDevice();

// Map the physics buffer

void* data;
vkMapMemory(device, *m_VulkanResources.physicsBufferMemory, 0,
sizeof(GPUPhysicsData) * m_RigidBodies.size(), 0, &data);

// Copy physics data from the buffer

GPUPhysicsData* gpuData = static_cast<GPUPhysicsData*>(data);
for (size_t i = 0; i < m_RigidBodies.size(); i++) {
auto& body = m_RigidBodies[i];

// Skip kinematic bodies

if (body->IsKinematic()) {
continue;
}

body->SetPosition(glm::vec3(gpuData[i].position));
body->SetRotation(glm::quat(gpuData[i].rotation.w, gpuData[i].rotation.x,
gpuData[i].rotation.y, gpuData[i].rotation.z));
body->SetLinearVelocity(glm::vec3(gpuData[i].linearVelocity));
body->SetAngularVelocity(glm::vec3(gpuData[i].angularVelocity));
}

vkUnmapMemory(device, *m_VulkanResources.physicsBufferMemory);
}

void PhysicsSystem::SimulatePhysicsOnGPU(float deltaTime) {

auto& device = m_Engine.GetVulkanDevice();
auto& queue = m_Engine.GetVulkanComputeQueue();

// Update physics data on the GPU

UpdateGPUPhysicsData();

// Record command buffer

vk::CommandBufferBeginInfo
beginInfo(vk::CommandBufferUsageFlagBits::eOneTimeSubmit);
m_VulkanResources.[Link](beginInfo);

// Bind descriptor set

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,

*m_VulkanResources.pipelineLayout, 0,

413
 *m_VulkanResources.descriptorSets[0], {});

// Push constants for simulation parameters

struct {
float deltaTime;
float gravity[3];
uint32_t numBodies;
} pushConstants;

[Link] = deltaTime;
[Link][0] = m_Gravity.x;
[Link][1] = m_Gravity.y;
[Link][2] = m_Gravity.z;
[Link] = static_cast<uint32_t>(m_RigidBodies.size());

m_VulkanResources.[Link](*m_VulkanResources.pipelineLayout,
vk::ShaderStageFlagBits::eCompute, 0,
sizeof(pushConstants),
&pushConstants);

// Step 1: Integrate forces and velocities

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,
*m_VulkanResources.integratePipeline);
m_VulkanResources.[Link](([Link] + 63) / 64, 1,
1);

// Memory barrier to ensure integration is complete before collision detection

vk::MemoryBarrier memoryBarrier(vk::AccessFlagBits::eShaderWrite,
vk::AccessFlagBits::eShaderRead);

m_VulkanResources.[Link](vk::PipelineStageFlagBits::eComputeSha
der,

vk::PipelineStageFlagBits::eComputeShader,
{}, memoryBarrier, {}, {});

// Step 2: Broad-phase collision detection

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,
*m_VulkanResources.broadPhasePipeline);
// Each thread checks one pair of objects
uint32_t numPairs = ([Link] * ([Link] - 1)) / 2;
m_VulkanResources.[Link]((numPairs + 63) / 64, 1, 1);

// Memory barrier to ensure broad phase is complete before narrow phase

m_VulkanResources.[Link](vk::PipelineStageFlagBits::eComputeSha
der,

vk::PipelineStageFlagBits::eComputeShader,
{}, memoryBarrier, {}, {});

414
 // Step 3: Narrow-phase collision detection
m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,

*m_VulkanResources.narrowPhasePipeline);
// We don't know how many pairs were generated, so we use a conservative estimate
m_VulkanResources.[Link]((m_MaxGPUCollisions + 63) / 64, 1, 1);

// Memory barrier to ensure narrow phase is complete before resolution

m_VulkanResources.[Link](vk::PipelineStageFlagBits::eComputeSha
der,

vk::PipelineStageFlagBits::eComputeShader,
{}, memoryBarrier, {}, {});

// Step 4: Collision resolution

m_VulkanResources.[Link](vk::PipelineBindPoint::eCompute,
*m_VulkanResources.resolvePipeline);
// We don't know how many collisions were detected, so we use a conservative
estimate
m_VulkanResources.[Link]((m_MaxGPUCollisions + 63) / 64, 1, 1);

m_VulkanResources.[Link]();

// Submit command buffer

vk::SubmitInfo submitInfo({}, {}, *m_VulkanResources.commandBuffer);
[Link](submitInfo, nullptr);
[Link]();

// Read back physics data from the GPU

ReadbackGPUPhysicsData();
}

void PhysicsSystem::Update(float deltaTime) {

if (m_GPUAccelerationEnabled && m_RigidBodies.size() <= m_MaxGPUObjects) {
// Use GPU-accelerated physics
SimulatePhysicsOnGPU(deltaTime);
} else {
// Fall back to CPU physics
// ... existing CPU physics code ...
}
}

1.4. Physics Compute Shaders

Now, let’s implement the compute shaders for our GPU-accelerated physics system:

// physics_integrate.comp

415
 #version 450

layout(local_size_x = 64, local_size_y = 1, local_size_z = 1) in;

// Push constants
layout(push_constant) uniform PushConstants {
float deltaTime;
vec3 gravity;
uint numBodies;
} pushConstants;

// Physics data
struct PhysicsData {
vec4 position; // xyz = position, w = inverse mass
vec4 rotation; // quaternion
vec4 linearVelocity; // xyz = velocity, w = restitution
vec4 angularVelocity; // xyz = angular velocity, w = friction
vec4 force; // xyz = force, w = is kinematic (0 or 1)
vec4 torque; // xyz = torque, w = use gravity (0 or 1)
vec4 colliderData; // type-specific data (e.g., radius for spheres)
vec4 colliderData2; // additional collider data (e.g., box half extents)
};

layout(std430, binding = 0) buffer PhysicsBuffer {

PhysicsData bodies[];
} physicsBuffer;

// Quaternion multiplication
vec4 quatMul(vec4 q1, vec4 q2) {
return vec4(
q1.w * q2.x + q1.x * q2.w + q1.y * q2.z - q1.z * q2.y,
q1.w * q2.y - q1.x * q2.z + q1.y * q2.w + q1.z * q2.x,
q1.w * q2.z + q1.x * q2.y - q1.y * q2.x + q1.z * q2.w,
q1.w * q2.w - q1.x * q2.x - q1.y * q2.y - q1.z * q2.z
);
}

// Quaternion normalization
vec4 quatNormalize(vec4 q) {
float len = length(q);
if (len > 0.0001) {
return q / len;
}
return vec4(0, 0, 0, 1);
}

void main() {
uint gID = gl_GlobalInvocationID.x;

// Check if this invocation is within the number of bodies

if (gID >= [Link]) {

416
 return;
}

// Get physics data for this body

PhysicsData body = [Link][gID];

// Skip kinematic bodies

if ([Link].w > 0.5) {
return;
}

// Apply gravity if enabled

if ([Link].w > 0.5) {
[Link] += [Link] / [Link].w;
}

// Integrate forces
[Link] += [Link] * [Link].w *
[Link];
[Link] += [Link] * [Link]; //
Simplified, should use inertia tensor

// Apply damping
const float linearDamping = 0.01;
const float angularDamping = 0.01;
[Link] *= (1.0 - linearDamping);
[Link] *= (1.0 - angularDamping);

// Integrate velocities
[Link] += [Link] * [Link];

// Update rotation
vec4 angularVelocityQuat = vec4([Link] * 0.5, 0.0);
vec4 rotationDelta = quatMul(angularVelocityQuat, [Link]);
[Link] = quatNormalize([Link] + rotationDelta *
[Link]);

// Write updated data back to buffer

[Link][gID] = body;
}

// physics_broad_phase.comp
#version 450

layout(local_size_x = 64, local_size_y = 1, local_size_z = 1) in;

// Push constants
layout(push_constant) uniform PushConstants {
float deltaTime;
vec3 gravity;

417
 uint numBodies;
} pushConstants;

// Physics data
struct PhysicsData {
vec4 position; // xyz = position, w = inverse mass
vec4 rotation; // quaternion
vec4 linearVelocity; // xyz = velocity, w = restitution
vec4 angularVelocity; // xyz = angular velocity, w = friction
vec4 force; // xyz = force, w = is kinematic (0 or 1)
vec4 torque; // xyz = torque, w = use gravity (0 or 1)
vec4 colliderData; // type-specific data (e.g., radius for spheres)
vec4 colliderData2; // additional collider data (e.g., box half extents)
};

layout(std430, binding = 0) buffer PhysicsBuffer {

PhysicsData bodies[];
} physicsBuffer;

// Pair buffer for potential collisions

layout(std430, binding = 2) buffer PairBuffer {
uvec2 pairs[];
} pairBuffer;

// Counter buffer
layout(std430, binding = 3) buffer CounterBuffer {
uint pairCount;
uint collisionCount;
} counterBuffer;

// Compute AABB for a body

void computeAABB(PhysicsData body, out vec3 min, out vec3 max) {
// Default to a small AABB
min = [Link] - vec3(0.1);
max = [Link] + vec3(0.1);

// Check collider type

int colliderType = int([Link].w);

if (colliderType == 0) { // Sphere
float radius = [Link].x;
vec3 center = [Link] + [Link];
min = center - vec3(radius);
max = center + vec3(radius);
}
else if (colliderType == 1) { // Box
vec3 halfExtents = [Link];
vec3 center = [Link] + [Link];
// This is simplified - should account for rotation
min = center - halfExtents;
max = center + halfExtents;

418
 }
}

bool aabbOverlap(vec3 minA, vec3 maxA, vec3 minB, vec3 maxB) {

return all(lessThan(minA, maxB)) && all(lessThan(minB, maxA));
}

void main() {
uint gID = gl_GlobalInvocationID.x;

// Calculate which pair of bodies this thread should check

uint numBodies = [Link];
uint numPairs = (numBodies * (numBodies - 1)) / 2;

if (gID >= numPairs) {

return;
}

// Convert linear index to pair indices (i, j) where i < j

uint i = 0;
uint j = 0;

// This is a mathematical formula to convert a linear index to a pair of indices

uint row = uint(floor(sqrt(float(2 * gID + 0.25)) - 0.5));
i = row;
j = gID - (row * (row + 1)) / 2;

// Ensure j > i
j += i + 1;

// Get physics data for both bodies

PhysicsData bodyA = [Link][i];
PhysicsData bodyB = [Link][j];

// Skip if both bodies are kinematic

if ([Link].w > 0.5 && [Link].w > 0.5) {
return;
}

// Skip if either body doesn't have a collider

if ([Link].w < 0 || [Link].w < 0) {
return;
}

// Compute AABBs
vec3 minA, maxA, minB, maxB;
computeAABB(bodyA, minA, maxA);
computeAABB(bodyB, minB, maxB);

// Check for AABB overlap

if (aabbOverlap(minA, maxA, minB, maxB)) {

419
 // Add to potential collision pairs
uint pairIndex = atomicAdd([Link], 1);
[Link][pairIndex] = uvec2(i, j);
}
}

The narrow-phase and resolve shaders would follow a similar pattern, implementing the detailed
collision detection and resolution algorithms.

1.5. Performance Considerations

When implementing GPU-accelerated physics with Vulkan, consider these performance
optimizations:

1. Batch Processing: Process multiple physics steps in a single dispatch to amortize the overhead
of command submission.

2. Memory Transfers: Minimize transfers between CPU and GPU memory by keeping physics data
on the GPU when possible.

3. Spatial Partitioning: Implement grid or tree-based spatial partitioning to reduce the number of
potential collision pairs.

4. Workgroup Size: Tune the workgroup size based on your target hardware for optimal
performance.

5. Memory Layout: Organize physics data for optimal cache coherency on the GPU.

1.6. Integration with the Engine

To integrate the GPU-accelerated physics into our engine, we need to modify the
PhysicsSystem::Initialize method:

void PhysicsSystem::Initialize() {
// Initialize basic physics system
// ...

// Initialize Vulkan resources for GPU-accelerated physics

if (m_Engine.IsVulkanInitialized()) {
InitializeVulkanResources();
m_GPUAccelerationEnabled = true;
}
}

void PhysicsSystem::Shutdown() {
// Cleanup Vulkan resources
if (m_Engine.IsVulkanInitialized()) {
CleanupVulkanResources();
}

420
 // Shutdown basic physics system
// ...
}

1.7. Advantages of Vulkan-Based Physics

By implementing physics simulation with Vulkan compute shaders, we gain several advantages:

1. Scalability: The GPU can simulate thousands or even millions of objects in parallel.

2. Performance: GPU-accelerated physics can be orders of magnitude faster than CPU-based

solutions for large-scale simulations.

3. CPU Offloading: Physics processing no longer competes with game logic for CPU resources.

4. Advanced Simulations: The GPU’s computational power enables more complex physics
simulations like fluid dynamics or cloth.

1.8. Limitations and Considerations

While Vulkan-based physics offers many advantages, there are some limitations to consider:

1. Complexity: Implementing and debugging GPU-based physics is more complex than CPU-based
solutions.

2. Precision: GPUs typically use single-precision floating-point, which may lead to numerical
stability issues in some simulations.

3. Platform Support: Not all platforms support Vulkan, so you may need fallback CPU
implementations.

4. Synchronization: Keeping CPU and GPU physics data in sync can be challenging and may
introduce latency.

1.9. Real-World Applications

Several modern game engines and physics middleware solutions leverage GPU acceleration for
physics simulations:

1. NVIDIA PhysX: Supports GPU acceleration for certain physics calculations.

2. Bullet Physics: Has experimental GPU acceleration using compute shaders.

3. Flex: NVIDIA’s particle-based physics solver designed specifically for GPU acceleration.

4. Custom Solutions: AAA game studios often implement custom GPU-accelerated physics for
their titles.

By implementing Vulkan-based physics in our engine, we’re following industry best practices for
high-performance physics in modern games.

421
1.10. Conclusion
In this chapter, we’ve explored how Vulkan compute shaders can be used to accelerate both audio
and physics processing in a game engine. By leveraging the GPU’s massive parallel processing
capabilities, we can create more immersive and dynamic game worlds with realistic audio and
physics simulations.

The techniques we’ve covered demonstrate the versatility of Vulkan beyond traditional graphics
rendering. As you continue to develop your engine, consider other areas where GPU acceleration
might provide benefits, such as AI pathfinding, procedural generation, or particle systems.

Previous: Physics Basics | Next: Conclusion :pp: ++

Subsystems: Conclusion
1. Conclusion
In this chapter, we’ve explored how to implement and enhance two critical engine
subsystems—Audio and Physics—using Vulkan’s compute capabilities. Let’s summarize what we’ve
learned and discuss potential future directions.

1.1. What We’ve Learned

1.1.1. Audio Subsystems

We started by implementing a basic audio system that provides the foundation for sound playback
in our engine. This system includes:

• Audio resource management for loading and playing sound files

• Spatial audio positioning based on listener and source positions

• A flexible architecture that can be integrated with various audio backends

We then enhanced this basic system with Vulkan compute shaders to implement Head-Related
Transfer Function (HRTF) processing for more realistic 3D audio. This approach demonstrated:

• How to offload computationally intensive audio processing to the GPU

• Techniques for implementing real-time convolution using compute shaders

• Methods for sharing data efficiently between CPU and GPU audio processing

1.1.2. Physics Subsystems

Similarly, we implemented a basic physics system that provides rigid body dynamics and collision
detection. This system includes:

• Rigid body simulation with forces, impulses, and collisions

422
 • Various collider types for different geometric shapes

• Integration with the rest of our engine for visual representation of physics objects

We then enhanced this system with Vulkan compute shaders to accelerate physics calculations,
demonstrating:

• Techniques for parallel physics simulation on the GPU

• Multi-stage physics processing (integration, broad phase, narrow phase, resolution)

• Methods for handling large numbers of physics objects efficiently

1.1.3. Vulkan Integration

Throughout both subsystems, we leveraged Vulkan’s compute capabilities. We demonstrated:

• Creating and managing compute pipelines for non-graphical tasks

• Efficient memory sharing between CPU and GPU

• Synchronization techniques for ensuring correct execution order

• Performance optimization strategies for compute shader workloads

1.2. Potential Improvements

While our implementations provide a solid foundation, there are several areas where they could be
enhanced:

1.2.1. Audio Improvements

• Advanced HRTF Models: Implement more sophisticated HRTF models that account for
individual differences in head and ear shapes.

• Environmental Effects: Add reverb, occlusion, and other environmental effects based on scene
geometry.

• Streaming Audio: Implement streaming for large audio files to reduce memory usage.

• Compression: Add support for compressed audio formats to reduce memory and bandwidth
requirements.

• Voice Communication: Integrate real-time voice processing for multiplayer games.

1.2.2. Physics Improvements

• Advanced Collision Shapes: Add support for more complex collision shapes like convex hulls
and trimeshes.

• Constraints and Joints: Implement various types of constraints and joints for more complex
mechanical systems.

• Continuous Collision Detection: Add support for detecting collisions between fast-moving
objects.

423
 • Soft Body Physics: Extend the system to support deformable objects like cloth, ropes, and soft
bodies.

• Fluid Simulation: Implement fluid dynamics for realistic water, smoke, and fire effects.

1.2.3. General Improvements

• Driver and Platform Coverage: Test the subsystems across a representative set of Vulkan-
capable platforms and drivers (e.g., Windows/Linux, major IHVs, Android, and macOS via
MoltenVK). Non-Vulkan fallbacks are out of scope for this tutorial.

• Profiling and Optimization: Add detailed profiling to identify and address performance
bottlenecks.

• Memory Management: Use allocator suballocation strategies (e.g., Vulkan Memory Allocator or
custom pools), batch buffer/image allocations, and group resources by usage to reduce
fragmentation and improve cache locality.

• Multi-Threading: Further optimize CPU-side processing with multi-threading where

appropriate.

1.3. Integration with Other Engine Systems

As you continue developing your engine, consider how these subsystems interact with other
components:

• Rendering System: Visualize physics debug information, audio sources, and listener positions.

• Animation System: Synchronize animations with audio events and physics interactions.

• Scripting System: Provide high-level interfaces for controlling audio and physics from game
scripts.

• Networking: Implement efficient synchronization of audio and physics state across networked
clients.

1.4. Real-World Considerations

When using these subsystems in production applications, keep these considerations in mind:

• Performance Profiling: Regularly profile your audio and physics systems to ensure they’re not
becoming bottlenecks.

• Memory Usage: Monitor memory usage, especially for large numbers of audio sources or
physics objects.

• Platform Differences: Test on various hardware configurations to ensure consistent behavior.

• Power Consumption: Be mindful of power usage, especially on mobile devices where GPU
compute can drain batteries quickly.

424
1.5. Final Thoughts
Audio and physics are essential components that contribute significantly to the immersion and
interactivity of modern games. By leveraging Vulkan’s compute capabilities, we can create more
sophisticated and performant implementations of these subsystems, enabling richer and more
dynamic game experiences.

The techniques we’ve explored in this chapter demonstrate the versatility of Vulkan beyond
traditional graphics rendering. As you continue to develop your engine, consider other areas where
GPU acceleration might provide benefits, such as AI pathfinding, procedural generation, or particle
systems.

Remember that the implementations provided here are starting points. Real-world engines often
require customization and optimization based on the specific needs of your games and target
platforms. Don’t hesitate to experiment and extend these systems to meet your unique
requirements.

1.6. Code Examples

The complete code for this chapter can be found in the following files:

• simple_engine/30_audio_subsystem.cpp: Implementation of the audio subsystem with Vulkan

HRTF processing

• simple_engine/31_physics_subsystem.cpp: Implementation of the physics subsystem with Vulkan

acceleration

Audio Subsystem C++ code Physics Subsystem C++ code

Previous: Vulkan for Physics Simulation | Next: Tooling | Back to Building a Simple Engine :pp: ++

Subsystems
This chapter covers the implementation of critical engine subsystems - Audio and Physics - with a
focus on leveraging Vulkan’s compute capabilities for enhanced performance.

• Introduction

• Audio Basics

• Vulkan for Audio Processing

• Physics Basics

• Vulkan for Physics Simulation

• Conclusion

Previous: Loading Models Conclusion | Back to Building a Simple Engine :pp: ++

425
Tooling: Introduction
1. Introduction to Engine Tooling
In previous chapters, we’ve built the foundation of our simple engine, implementing core
components like the rendering pipeline, camera systems, model loading, and essential subsystems
like audio and physics. Now, we’re ready to explore the tooling ecosystem that supports the
development, debugging, and distribution of a professional Vulkan application.

Effective tooling is critical for maintaining productivity, ensuring quality, and delivering a robust
final product. While these tools may seem separate from the engine itself, they are integral to the
development process and can significantly impact the quality and maintainability of your code.

1.1. What We’ll Cover

This chapter will equip you with the professional tooling ecosystem that transforms a working
Vulkan application into a maintainable, debuggable, and deployable product. We’ll begin by
implementing a continuous integration and continuous deployment pipeline specifically designed
for Vulkan’s unique requirements. This foundation ensures that your application builds
consistently across platforms while catching integration issues before they reach users.

Debugging Vulkan applications presents unique challenges that traditional debugging approaches
can’t address effectively. We’ll master both Vulkan’s built-in debugging extensions like
VK_KHR_debug_utils and external tools like RenderDoc, creating a comprehensive debugging
workflow that can diagnose everything from validation layer warnings to complex rendering
pipeline issues.

Robust crash handling becomes crucial as your application moves toward production deployment.
We’ll implement systems that can gracefully handle unexpected failures, generate detailed
minidumps for post-mortem analysis, and provide users with meaningful recovery options rather
than abrupt terminations.

Finally, we’ll explore Vulkan extensions designed specifically for application robustness, such as
VK_EXT_robustness2, which help your application handle edge cases and undefined behavior
gracefully. These extensions transform potential crashes into recoverable situations, improving the
overall user experience.

1.2. Prerequisites
This chapter assumes solid understanding of the Vulkan fundamentals and engine architecture
we’ve built throughout the previous chapters. The tooling we’ll implement needs to integrate with
your existing systems—CI/CD pipelines must understand your project structure, debugging tools
must work with your rendering pipeline, and crash handling must respect your engine’s resource
management patterns.

Experience with modern C concepts becomes particularly important here, as professional tooling

426
often leverages advanced language features for reliability and maintainability. C17 and C++20
features like structured bindings, concepts, and coroutines appear frequently in production tooling
code, and understanding these patterns will help you implement robust solutions.

A basic familiarity with software development workflows and tools will provide context for the
systems we’ll build. While we’ll explain the specific implementations, understanding why
continuous integration matters, how debugging fits into development cycles, and why crash
reporting improves user experience will help you appreciate the architectural decisions we make
throughout this chapter.

You should also be familiar with the following chapters from the main tutorial:

• Basic Vulkan concepts:

◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

Let’s begin by exploring how to set up a CI/CD pipeline for Vulkan projects.

Previous: Subsystems Conclusion | Next: CI/CD for Vulkan Projects :pp: ++

Tooling: CI/CD for Vulkan Projects

1. Continuous Integration and Deployment
for Vulkan
Continuous Integration (CI) and Continuous Deployment (CD) are essential practices in modern
software development. They help ensure code quality, catch issues early, and streamline the release
process. For Vulkan applications, which often need to run on multiple platforms with different GPU
architectures, a robust CI/CD pipeline is particularly valuable.

1.1. Setting Up a CI/CD Pipeline

Let’s explore how to set up a CI/CD pipeline specifically tailored for Vulkan projects. We’ll use
GitHub Actions as our example platform, but the concepts apply to other CI/CD systems like GitLab
CI, Jenkins, or Azure DevOps.

1.1.1. Basic Pipeline Structure

A typical CI/CD pipeline for a Vulkan project might include these stages:

1. Build: Compile the application on multiple platforms (Windows, Linux, macOS)

2. Test: Run unit tests and integration tests

427
3. Package: Create distributable packages for each platform

4. Deploy: Deploy to a staging environment or release to users

Here’s a basic GitHub Actions workflow file for a Vulkan project:

name: Vulkan CI/CD

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
build:
runs-on: ${{ [Link] }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
build_type: [Debug, Release]

steps:
- uses: actions/checkout@v3
with:
submodules: recursive

- name: Install Vulkan SDK

uses: humbletim/install-vulkan-sdk@v1.1.1
with:
version: latest
cache: true

- name: Configure CMake

run: cmake -B ${{[Link]}}/build
-DCMAKE_BUILD_TYPE=${{matrix.build_type}}

- name: Build
run: cmake --build ${{[Link]}}/build --config ${{matrix.build_type}}

- name: Test
working-directory: ${{[Link]}}/build
run: ctest -C ${{matrix.build_type}}

- name: Package
if: matrix.build_type == 'Release'
run: |
# Platform-specific packaging commands
if [ "${{ [Link] }}" == "ubuntu-latest" ]; then
# Linux packaging (e.g., .deb or .AppImage)
echo "Packaging for Linux"

428
 elif [ "${{ [Link] }}" == "windows-latest" ]; then
# Windows packaging (e.g., .exe installer)
echo "Packaging for Windows"
elif [ "${{ [Link] }}" == "macos-latest" ]; then
# macOS packaging (e.g., .app bundle or .dmg)
echo "Packaging for macOS"
fi

1.1.2. Vulkan-Specific Considerations

When setting up CI/CD for Vulkan projects, consider these specific challenges:

[Link]. Vulkan SDK Installation

Ensure your CI environment has the Vulkan SDK installed. Many CI platforms don’t include it by
default. In the example above, we used a GitHub Action to install the SDK.

[Link]. GPU Availability in CI Environments

Most CI environments don’t have GPUs available, which can make testing Vulkan applications
challenging. Consider these approaches:

• Use software rendering (e.g., SwiftShader) for basic tests

• Implement a headless testing mode that doesn’t require a display

• Use cloud-based GPU instances for more comprehensive testing

[Link]. Platform-Specific Vulkan Loaders

Different platforms handle Vulkan loading differently. Ensure your build system correctly handles
these differences:

• Windows: [Link] is typically loaded at runtime

• Linux: [Link].1 is loaded at runtime

• macOS: MoltenVK provides Vulkan support via Metal

[Link]. Shader Compilation

Shader compilation can be a complex part of the build process. Consider these approaches:

• Pre-compile shaders during the build phase

• Include shader compilation in your CI pipeline to catch GLSL/SPIR-V errors early

• Use a shader management system that handles cross-platform differences

1.2. Automating Testing for Vulkan Applications

Testing Vulkan applications presents unique challenges. Here are some approaches to consider:

429
1.2.1. Unit Testing Vulkan Code

import std;
import vulkan_raii;

// A testable function using vk::raii

bool create_pipeline(vk::raii::Device& device,
vk::raii::RenderPass& render_pass,
vk::raii::PipelineLayout& layout,
vk::raii::Pipeline& out_pipeline) {
try {
// Pipeline creation code using RAII
return true;
} catch (vk::SystemError& err) {
std::cerr << "Failed to create pipeline: " << [Link]() << std::endl;
return false;
}
}

// In a test file
TEST_CASE("Pipeline creation") {
// Setup test environment with mock or real Vulkan objects
vk::raii::Context context;
auto instance = create_test_instance(context);
auto device = create_test_device(instance);
auto render_pass = create_test_render_pass(device);
auto layout = create_test_pipeline_layout(device);

vk::raii::Pipeline pipeline{nullptr};
REQUIRE(create_pipeline(device, render_pass, layout, pipeline));
REQUIRE(pipeline);
}

1.2.2. Integration Testing

For integration testing, consider creating a headless rendering mode that can run in CI
environments:

import std;
import vulkan_raii;

class HeadlessRenderer {
public:
HeadlessRenderer() {
// Initialize Vulkan without surface
init_vulkan();
}

bool render_frame() {

430
 // Render to an image without presenting
try {
// Rendering code
return true;
} catch (vk::SystemError& err) {
std::cerr << "Render failed: " << [Link]() << std::endl;
return false;
}
}

// Compare rendered image with reference

bool verify_output(const std::string& reference_image) {
// Image comparison code
return true;
}

private:
void init_vulkan() {
// Vulkan initialization code
}

vk::raii::Context context;
vk::raii::Instance instance{nullptr};
vk::raii::PhysicalDevice physical_device{nullptr};
vk::raii::Device device{nullptr};
// Other Vulkan objects
};

// In a test file
TEST_CASE("Render output matches reference") {
HeadlessRenderer renderer;
REQUIRE(renderer.render_frame());
REQUIRE(renderer.verify_output("reference_image.png"));
}

1.3. Distribution Considerations

Once your application passes all tests, the final stage is packaging and distribution. Here are some
considerations:

1.3.1. Packaging Vulkan Applications

• Include the appropriate Vulkan loader for each platform

• Package shader files or pre-compiled SPIR-V

• Consider using platform-specific packaging tools:

◦ Windows: NSIS, WiX, or MSIX

◦ Linux: AppImage, Flatpak, or .deb/.rpm packages

431
 ◦ macOS: DMG or App Store packages

1.3.2. Handling Vulkan Dependencies

Ensure your package includes or correctly handles all dependencies:

• Vulkan loader (or instructions to install it)

• Any required Vulkan extensions

• GPU driver requirements

1.3.3. Versioning and Updates

Implement a versioning system that includes:

• Application version

• Minimum required Vulkan version

• Required extensions and their versions

1.4. Conclusion
A well-designed CI/CD pipeline is essential for maintaining quality and productivity when
developing Vulkan applications. By automating building, testing, and packaging, you can focus
more on developing features and less on manual processes.

In the next section, we’ll explore debugging tools for Vulkan applications, including the powerful
VK_KHR_debug_utils extension and external tools like RenderDoc.

Previous: Introduction | Next: Debugging with VK_KHR_debug_utils and RenderDoc :pp: ++

Tooling: Debugging with

VK_KHR_debug_utils and RenderDoc
1. Debugging Vulkan Applications
Debugging graphics applications can be challenging due to their complex, parallel nature and the
fact that much of the processing happens on the GPU. Vulkan, with its explicit design, provides
powerful debugging tools that can help identify and fix issues in your application. In this section,
we’ll explore two key approaches to debugging Vulkan applications:

1. Using the VK_KHR_debug_utils extension for in-application debugging

2. Using external tools like RenderDoc for frame capture and analysis

432
1.1. Using VK_KHR_debug_utils
The VK_KHR_debug_utils extension provides a comprehensive set of tools for debugging Vulkan
applications. It allows you to:

• Label objects with meaningful names

• Mark the beginning and end of command buffer regions

• Insert debug markers

• Set up debug messengers to receive validation layer messages

Let’s explore how to use these features with C++20 modules and vk::raii.

1.1.1. Setting Up Debug Messaging

First, let’s set up a debug messenger to receive validation layer messages:

import std;
import vulkan_raii;

// Debug callback function

VKAPI_ATTR VkBool32 VKAPI_CALL debug_callback(
VkDebugUtilsMessageSeverityFlagBitsEXT message_severity,
VkDebugUtilsMessageTypeFlagsEXT message_type,
const VkDebugUtilsMessengerCallbackDataEXT* callback_data,
void* user_data) {

// Convert severity to string

std::string severity;
if (message_severity & VK_DEBUG_UTILS_MESSAGE_SEVERITY_VERBOSE_BIT_EXT) {
severity = "VERBOSE";
} else if (message_severity & VK_DEBUG_UTILS_MESSAGE_SEVERITY_INFO_BIT_EXT) {
severity = "INFO";
} else if (message_severity & VK_DEBUG_UTILS_MESSAGE_SEVERITY_WARNING_BIT_EXT) {
severity = "WARNING";
} else if (message_severity & VK_DEBUG_UTILS_MESSAGE_SEVERITY_ERROR_BIT_EXT) {
severity = "ERROR";
}

// Convert type to string

std::string type;
if (message_type & VK_DEBUG_UTILS_MESSAGE_TYPE_GENERAL_BIT_EXT) {
type = "GENERAL";
} else if (message_type & VK_DEBUG_UTILS_MESSAGE_TYPE_VALIDATION_BIT_EXT) {
type = "VALIDATION";
} else if (message_type & VK_DEBUG_UTILS_MESSAGE_TYPE_PERFORMANCE_BIT_EXT) {
type = "PERFORMANCE";
}

// Log the message

433
 std::cerr << "[" << severity << ": " << type << "] "
<< callback_data->pMessage << std::endl;

// Return false to indicate the Vulkan call should not be aborted

return VK_FALSE;
}

// Create a debug messenger using vk::raii

vk::raii::DebugUtilsMessengerEXT create_debug_messenger(vk::raii::Instance& instance)
{
vk::DebugUtilsMessengerCreateInfoEXT create_info{};
create_info.setMessageSeverity(
vk::DebugUtilsMessageSeverityFlagBitsEXT::eVerbose |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eInfo |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eWarning |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eError
);
create_info.setMessageType(
vk::DebugUtilsMessageTypeFlagBitsEXT::eGeneral |
vk::DebugUtilsMessageTypeFlagBitsEXT::eValidation |
vk::DebugUtilsMessageTypeFlagBitsEXT::ePerformance
);
create_info.setPfnUserCallback(debug_callback);

return vk::raii::DebugUtilsMessengerEXT(instance, create_info);

}

1.1.2. Object Naming

One of the most useful features of VK_KHR_debug_utils is the ability to give meaningful names to
Vulkan objects. This makes debugging much easier, as you can identify objects in validation layer
messages and tools like RenderDoc:

// Helper function to set a name on any Vulkan handle

template<typename T>
void set_object_name(vk::raii::Device& device, T handle, const std::string& name) {
vk::DebugUtilsObjectNameInfoEXT name_info{};
name_info.setObjectType(get_object_type<T>());
name_info.setObjectHandle(reinterpret_cast<uint64_t>(static_cast<T>(handle)));
name_info.setPObjectName(name.c_str());

[Link](name_info);
}

// Example usage
void name_vulkan_objects(vk::raii::Device& device) {
// Name the device itself
set_object_name(device, *device, "Main Device");

434
 // Name a buffer
vk::BufferCreateInfo buffer_info{};
// ... set buffer creation parameters
vk::raii::Buffer buffer(device, buffer_info);
set_object_name(device, *buffer, "Vertex Buffer");

// Name a pipeline
vk::raii::Pipeline pipeline = create_graphics_pipeline(device);
set_object_name(device, *pipeline, "Main Render Pipeline");
}

1.1.3. Command Buffer Labeling

You can also label regions of command buffer execution, which helps identify where issues occur
during rendering:

void record_command_buffer(vk::raii::CommandBuffer& cmd_buffer) {

cmd_buffer.begin({vk::CommandBufferUsageFlagBits::eOneTimeSubmit});

// Begin a labeled region

vk::DebugUtilsLabelEXT label_info{};
label_info.setPLabelName("Shadow Pass");
label_info.setColor(std::array<float, 4>{0.0f, 0.0f, 0.0f, 1.0f}); // Black for
shadow pass
cmd_buffer.beginDebugUtilsLabelEXT(label_info);

// Record shadow pass commands

// ...

// End the labeled region

cmd_buffer.endDebugUtilsLabelEXT();

// Begin another labeled region

label_info.setPLabelName("Main Render Pass");
label_info.setColor(std::array<float, 4>{0.0f, 1.0f, 0.0f, 1.0f}); // Green for
main pass
cmd_buffer.beginDebugUtilsLabelEXT(label_info);

// Record main render pass commands

// ...

// Insert a marker within this region

cmd_buffer.insertDebugUtilsLabelEXT({
"Drawing Opaque Objects",
std::array<float, 4>{1.0f, 1.0f, 1.0f, 1.0f}
});

// More rendering commands

// ...

435
 // End the labeled region
cmd_buffer.endDebugUtilsLabelEXT();

cmd_buffer.end();
}

1.1.4. Queue Labeling

Similarly, you can label operations submitted to a queue:

void submit_work(vk::raii::Queue& queue, vk::raii::CommandBuffer& cmd_buffer) {

// Begin a labeled region for the queue submission
vk::DebugUtilsLabelEXT label_info{};
label_info.setPLabelName("Frame Rendering");
label_info.setColor(std::array<float, 4>{0.0f, 0.5f, 1.0f, 1.0f}); // Blue for
frame
[Link](label_info);

// Submit the command buffer

vk::SubmitInfo submit_info{};
submit_info.setCommandBufferCount(1);
submit_info.setPCommandBuffers(&(*cmd_buffer));
[Link](submit_info, nullptr);

// End the labeled region

[Link]();
}

1.2. Using RenderDoc

RenderDoc is a graphics frame debugger and capture/analysis tool (not a compiler). It allows you to
capture frames from your application and analyze them in detail. It’s particularly useful for Vulkan
applications due to its comprehensive support for the API.

1.2.1. Integrating RenderDoc with Your Application

You can integrate RenderDoc directly into your application using its in-application API:

import std;
import vulkan_raii;

#include <renderdoc_app.h>

// Load the RenderDoc API

RENDERDOC_API_1_4_1* renderdoc_api = nullptr;

436
 bool load_renderdoc_api() {
#if defined(_WIN32)
HMODULE renderdoc_module = LoadLibraryA("[Link]");
#else
void* renderdoc_module = dlopen("[Link]", RTLD_NOW | RTLD_NOLOAD);
#endif

if (!renderdoc_module) {
std::cerr << "RenderDoc not loaded in this application" << std::endl;
return false;
}

#if defined(_WIN32)
pRENDERDOC_GetAPI get_api = (pRENDERDOC_GetAPI)GetProcAddress(renderdoc_module,
"RENDERDOC_GetAPI");
#else
pRENDERDOC_GetAPI get_api = (pRENDERDOC_GetAPI)dlsym(renderdoc_module,
"RENDERDOC_GetAPI");
#endif

if (!get_api) {
std::cerr << "Failed to get RenderDoc API function" << std::endl;
return false;
}

int ret = get_api(eRENDERDOC_API_Version_1_4_1, (void**)&renderdoc_api);

if (ret != 1) {
std::cerr << "Failed to initialize RenderDoc API" << std::endl;
return false;
}

std::cout << "RenderDoc API initialized successfully" << std::endl;

return true;
}

// Trigger a capture
void capture_frame() {
if (renderdoc_api) {
renderdoc_api->TriggerCapture();
}
}

1.2.2. Analyzing Captures

Once you’ve captured a frame, you can analyze it in the RenderDoc application. Here are some key
features to look for:

1. Pipeline State: Examine the full graphics pipeline state for each draw call

2. Resource Inspection: View the contents of buffers, textures, and other resources

437
3. Shader Debugging: Step through shader execution for specific pixels

4. Timing Information: Analyze performance of different parts of your frame

1.2.3. Best Practices for RenderDoc

To get the most out of RenderDoc:

1. Use Object Names: As discussed earlier, naming your Vulkan objects makes them much easier
to identify in RenderDoc (you’ll see them in the Resource Inspector and Pipeline State views).

2. Use Command Buffer Labels: These appear in RenderDoc’s Event Browser and help you
navigate to the relevant draw/dispatch quickly.

3. Capture the Problem Frame: Trigger a capture exactly when the issue occurs (via hotkey or
the in-application API) to minimize unrelated events and noise.

4. Minimize to a Repro: Create a minimal reproducible scene or toggle features off to isolate the
problem. If you reduce resolution, make sure it doesn’t alter ordering/timing in a way that hides
the bug.

1.3. Combining VK_KHR_debug_utils and RenderDoc

The real power comes from combining these approaches:

1. Use VK_KHR_debug_utils to add rich debugging information to your application

2. Use RenderDoc to capture and analyze frames with this information

3. Use validation layers to catch API usage errors

Here’s an example of setting up a debugging environment that combines these approaches:

import std;
import vulkan_raii;

class DebugManager {
public:
DebugManager() {
// Try to load RenderDoc API
load_renderdoc_api();
}

void setup_instance_debugging(vk::raii::Context& context, vk::InstanceCreateInfo&

create_info) {
// Add validation layers
std::vector<const char*> validation_layers = {"VK_LAYER_KHRONOS_validation"};
create_info.setPEnabledLayerNames(validation_layers);

// Add debug utils extension

std::vector<const char*> extensions = {VK_EXT_DEBUG_UTILS_EXTENSION_NAME};
// Add any existing extensions
if (create_info.enabledExtensionCount > 0) {

438
 for (uint32_t i = 0; i < create_info.enabledExtensionCount; i++) {
extensions.push_back(create_info.ppEnabledExtensionNames[i]);
}
}
create_info.setPEnabledExtensionNames(extensions);

// Store debug messenger create info for instance creation

debug_create_info.setMessageSeverity(
vk::DebugUtilsMessageSeverityFlagBitsEXT::eVerbose |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eInfo |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eWarning |
vk::DebugUtilsMessageSeverityFlagBitsEXT::eError
);
debug_create_info.setMessageType(
vk::DebugUtilsMessageTypeFlagBitsEXT::eGeneral |
vk::DebugUtilsMessageTypeFlagBitsEXT::eValidation |
vk::DebugUtilsMessageTypeFlagBitsEXT::ePerformance
);
debug_create_info.setPfnUserCallback(debug_callback);

// Add to pNext chain

debug_create_info.pNext = create_info.pNext;
create_info.pNext = &debug_create_info;
}

void setup_debug_messenger(vk::raii::Instance& instance) {

debug_messenger = vk::raii::DebugUtilsMessengerEXT(instance,
debug_create_info);
}

template<typename T>
void set_name(vk::raii::Device& device, T handle, const std::string& name) {
try {
vk::DebugUtilsObjectNameInfoEXT name_info{};
name_info.setObjectType(get_object_type<T>());

name_info.setObjectHandle(reinterpret_cast<uint64_t>(static_cast<T>(handle)));
name_info.setPObjectName(name.c_str());

[Link](name_info);
} catch (vk::SystemError& err) {
std::cerr << "Failed to set object name: " << [Link]() << std::endl;
}
}

void capture_next_frame() {
if (renderdoc_api) {
renderdoc_api->TriggerCapture();
}
}

439
 private:
vk::DebugUtilsMessengerCreateInfoEXT debug_create_info{};
vk::raii::DebugUtilsMessengerEXT debug_messenger{nullptr};
RENDERDOC_API_1_4_1* renderdoc_api = nullptr;
};

1.4. Conclusion
Effective debugging is essential for developing complex Vulkan applications. By combining the
power of VK_KHR_debug_utils for in-application debugging and RenderDoc for frame capture and
analysis, you can quickly identify and fix issues in your rendering pipeline.

In the next section, we’ll explore crash handling and minidumps, which are crucial for diagnosing
issues that occur in production environments.

Previous: CI/CD for Vulkan Projects | Next: Crash Handling and Minidumps :pp: ++

Tooling: Crash Handling and GPU

Crash Dumps
1. Crash Handling in Vulkan Applications
Even with thorough testing and debugging, crashes can still occur in production environments.
When they do, having robust crash handling mechanisms can help you diagnose and fix issues
quickly. This chapter focuses on practical GPU crash diagnostics (e.g., NVIDIA Nsight Aftermath,
AMD Radeon GPU Detective) and clarifies the role and limitations of OS process minidumps, which
usually lack GPU state and are rarely sufficient to root-cause graphics/device-lost issues on their
own.

1.1. Understanding Crashes in Vulkan Applications

Vulkan applications can crash for various reasons:

1. API Usage Errors: Incorrect use of the Vulkan API that validation layers would catch in debug
builds

2. Driver Bugs: Issues in the GPU driver that may only manifest with specific hardware or
workloads

3. Resource Management Issues: Memory leaks, double frees, or accessing destroyed resources

4. Shader Errors: Runtime errors in shaders that cause the GPU to hang

5. System-Level Issues: Out of memory conditions, system instability, etc.

Let’s explore how to handle these crashes and gather diagnostic information.

440
1.2. Implementing Basic Crash Handling
First, let’s implement a basic crash handler that can catch unhandled exceptions and segmentation
faults:

import std;
import vulkan_raii;

// Global state for crash handling

namespace crash_handler {
std::string app_name;
std::string crash_log_path;
bool initialized = false;

// Log basic system information

void log_system_info(std::ofstream& log) {
log << "Application: " << app_name << std::endl;
log << "Timestamp: " << std::chrono::system_clock::now() << std::endl;

// Log OS information
#if defined(_WIN32)
log << "OS: Windows" << std::endl;
#elif defined(__linux__)
log << "OS: Linux" << std::endl;
#elif defined(__APPLE__)
log << "OS: macOS" << std::endl;
#else
log << "OS: Unknown" << std::endl;
#endif

// Log CPU information

log << "CPU Cores: " << std::thread::hardware_concurrency() << std::endl;

// Log memory information

#if defined(_WIN32)
MEMORYSTATUSEX mem_info;
mem_info.dwLength = sizeof(MEMORYSTATUSEX);
GlobalMemoryStatusEx(&mem_info);
log << "Total Physical Memory: " << mem_info.ullTotalPhys / (1024 * 1024) << "
MB" << std::endl;
log << "Available Memory: " << mem_info.ullAvailPhys / (1024 * 1024) << " MB"
<< std::endl;
#elif defined(__linux__)
// Linux-specific memory info code
#elif defined(__APPLE__)
// macOS-specific memory info code
#endif
}

// Log Vulkan-specific information

441
 void log_vulkan_info(std::ofstream& log, vk::raii::PhysicalDevice* physical_device
= nullptr) {
if (physical_device) {
auto properties = physical_device->getProperties();
log << "GPU: " << [Link] << std::endl;
log << "Driver Version: " << [Link] << std::endl;
log << "Vulkan API Version: "
<< VK_VERSION_MAJOR([Link]) << "."
<< VK_VERSION_MINOR([Link]) << "."
<< VK_VERSION_PATCH([Link]) << std::endl;
} else {
log << "No Vulkan physical device information available" << std::endl;
}
}

// Handler for unhandled exceptions

void handle_exception(const std::exception& e, vk::raii::PhysicalDevice*
physical_device = nullptr) {
try {
std::ofstream log(crash_log_path, std::ios::app);
log << "==== Crash Report ====" << std::endl;
log_system_info(log);
log_vulkan_info(log, physical_device);

log << "Exception: " << [Link]() << std::endl;

log << "==== End of Crash Report ====" << std::endl << std::endl;

[Link]();
} catch (...) {
// Last resort if we can't even write to the log
std::cerr << "Failed to write crash log" << std::endl;
}
}

// Signal handler for segfaults, etc.

void signal_handler(int signal) {
try {
std::ofstream log(crash_log_path, std::ios::app);
log << "==== Crash Report ====" << std::endl;
log_system_info(log);

log << "Signal: " << signal << " (";

switch (signal) {
case SIGSEGV: log << "SIGSEGV - Segmentation fault"; break;
case SIGILL: log << "SIGILL - Illegal instruction"; break;
case SIGFPE: log << "SIGFPE - Floating point exception"; break;
case SIGABRT: log << "SIGABRT - Abort"; break;
default: log << "Unknown signal"; break;
}
log << ")" << std::endl;

442
 log << "==== End of Crash Report ====" << std::endl << std::endl;

[Link]();
} catch (...) {
// Last resort if we can't even write to the log
std::cerr << "Failed to write crash log" << std::endl;
}

// Re-raise the signal for the default handler

signal(signal, SIG_DFL);
raise(signal);
}

// Initialize the crash handler

void initialize(const std::string& application_name, const std::string& log_path)
{
if (initialized) return;

app_name = application_name;
crash_log_path = log_path;

// Set up signal handlers

signal(SIGSEGV, signal_handler);
signal(SIGILL, signal_handler);
signal(SIGFPE, signal_handler);
signal(SIGABRT, signal_handler);

initialized = true;
}
}

// Example usage in main application

int main() {
try {
// Initialize crash handler
crash_handler::initialize("MyVulkanApp", "crash_log.txt");

// Initialize Vulkan
vk::raii::Context context;
auto instance = create_instance(context);
auto physical_device = select_physical_device(instance);
auto device = create_device(physical_device);

// Main application loop

while (true) {
try {
// Render frame
render_frame(device);
} catch (const vk::SystemError& e) {
// Handle Vulkan errors that we can recover from
std::cerr << "Vulkan error: " << [Link]() << std::endl;

443
 }
}
} catch (const std::exception& e) {
// Handle unrecoverable exceptions
crash_handler::handle_exception(e);
return 1;
}

return 0;
}

1.3. GPU Crash Diagnostics (Vulkan)

While OS process minidumps capture CPU-side state, GPU crashes (device lost, TDRs, hangs) require
GPU-specific crash dumps to be actionable. In practice, you’ll want to integrate vendor tooling that
can record GPU execution state around the fault.

1.3.1. NVIDIA: Nsight Aftermath (Vulkan)

Overview:

• Collects GPU crash dumps with information about the last executed draw/dispatch, bound
pipeline/shaders, markers, and resource identifiers.

• Works alongside your Vulkan app; you analyze dumps with NVIDIA tools to pinpoint the failing
work and shader.

Practical steps:

1. Enable object names and labels

◦ Use VK_EXT_debug_utils to name pipelines, shaders, images, buffers, and to insert command
buffer labels for major passes and draw/dispatch groups. These names surface in crash
reports and greatly aid triage.

2. Add frame/work markers

◦ Insert meaningful labels before/after critical rendering phases. If available on your target,
also use vendor checkpoint/marker extensions (e.g., VK_NV_device_diagnostic_checkpoints)
to provide fine-grained breadcrumbs.

3. Build shaders with unique IDs and optional debug info

◦ Ensure each pipeline/shader can be correlated (e.g., include a stable hash/UUID in your
pipeline cache and application logs). Keep the mapping from IDs to source for analysis.

4. Initialize and enable GPU crash dumps

◦ Integrate the Nsight Aftermath Vulkan SDK per NVIDIA’s documentation. Register a callback
to receive crash dump data, write it to disk, and include your marker string table for
symbolication.

5. Handle device loss

444
 ◦ On VK_ERROR_DEVICE_LOST (or Windows TDR), flush any in-memory marker logs, persist
the crash dump, and then terminate cleanly. Attempting to continue rendering is undefined.

References: NVIDIA Nsight Aftermath SDK and documentation.

1.3.2. AMD: Radeon GPU Detective (RGD)

• AMD provides tools to capture and analyze GPU crash information on RDNA hardware. Similar
principles apply: enable object names, label command buffers, and preserve pipeline/shader
identifiers so RGD can point back to your content.

• See AMD Radeon GPU Detective and related documentation for Vulkan integration and analysis
workflows.

1.3.3. Vendor-agnostic groundwork that helps all tools

• Name everything via VK_EXT_debug_utils.

• Insert command buffer labels at meaningful boundaries (frame, pass, material batch, etc.).

• Persist build/version, driver, Vulkan API/UUID, and pipeline cache UUID in your logs and crash
artifacts.

• Implement robust device lost handling: stop submitting, free/teardown safely, write artifacts,
exit.

1.4. Generating Minidumps

Use OS process minidumps to capture CPU-side call stacks, threads, and memory snapshots at the
time of a crash. For graphics issues and device loss, they rarely contain the GPU execution state you
need—treat minidumps as a complement to GPU crash dumps, not a replacement.

Below is a brief outline for generating minidumps with platform APIs (useful for correlating CPU
context with a GPU crash):

import std;
import vulkan_raii;

namespace crash_handler {
std::string app_name;
std::string dump_path;
bool initialized = false;

#if defined(_WIN32)
// Windows implementation using Windows Error Reporting (WER)
LONG WINAPI windows_exception_handler(EXCEPTION_POINTERS* exception_pointers) {
// Create a unique filename for the minidump
std::string filename = dump_path + "\\" + app_name + "_" +

std::to_string(std::chrono::system_clock::now().time_since_epoch().count()) + ".dmp";

445
 // Create the minidump file
HANDLE file = CreateFileA(
filename.c_str(),
GENERIC_WRITE,
0,
nullptr,
CREATE_ALWAYS,
FILE_ATTRIBUTE_NORMAL,
nullptr
);

if (file != INVALID_HANDLE_VALUE) {
// Initialize minidump info
MINIDUMP_EXCEPTION_INFORMATION exception_info;
exception_info.ThreadId = GetCurrentThreadId();
exception_info.ExceptionPointers = exception_pointers;
exception_info.ClientPointers = FALSE;

// Write the minidump

MiniDumpWriteDump(
GetCurrentProcess(),
GetCurrentProcessId(),
file,
MiniDumpWithFullMemory, // Dump type
&exception_info,
nullptr,
nullptr
);

CloseHandle(file);

std::cerr << "Minidump written to: " << filename << std::endl;
} else {
std::cerr << "Failed to create minidump file" << std::endl;
}

// Continue with normal exception handling

return EXCEPTION_CONTINUE_SEARCH;
}

void initialize(const std::string& application_name, const std::string&

minidump_path) {
if (initialized) return;

app_name = application_name;
dump_path = minidump_path;

// Create the dump directory if it doesn't exist

CreateDirectoryA(dump_path.c_str(), nullptr);

// Set up the exception handler

446
 SetUnhandledExceptionFilter(windows_exception_handler);

initialized = true;
}

#elif defined(__linux__)
// Linux implementation using Google Breakpad
// Note: This requires linking against the Google Breakpad library

#include "client/linux/handler/exception_handler.h"

// Callback for when a minidump is generated

static bool minidump_callback(const google_breakpad::MinidumpDescriptor&
descriptor,
void* context, bool succeeded) {
std::cerr << "Minidump generated: " << [Link]() << std::endl;
return succeeded;
}

google_breakpad::ExceptionHandler* exception_handler = nullptr;

void initialize(const std::string& application_name, const std::string&

minidump_path) {
if (initialized) return;

app_name = application_name;
dump_path = minidump_path;

// Create the dump directory if it doesn't exist

std::filesystem::create_directories(dump_path);

// Set up the exception handler

google_breakpad::MinidumpDescriptor descriptor(dump_path);
exception_handler = new google_breakpad::ExceptionHandler(
descriptor,
nullptr,
minidump_callback,
nullptr,
true,
-1
);

initialized = true;
}

#elif defined(__APPLE__)
// macOS implementation using Google Breakpad
// Similar to Linux implementation
#endif
}

447
1.5. Analyzing Minidumps
Minidumps are best used to understand CPU-side state around a crash (e.g., which thread faulted,
call stacks leading to vkQueueSubmit/vkQueuePresent, allocator misuse) and to correlate with a
GPU crash dump from vendor tools. Here’s a brief workflow on different platforms:

1.5.1. Windows

On Windows, you can use Visual Studio or WinDbg to analyze minidumps:

1. Visual Studio:

◦ Open Visual Studio

◦ Go to File > Open > File and select the .dmp file

◦ Visual Studio will load the minidump and show the call stack at the time of the crash

2. WinDbg:

◦ Open WinDbg

◦ Open the minidump file

◦ Use commands like .ecxr to examine the exception context record

◦ Use k to view the call stack

1.5.2. Linux and macOS

On Linux and macOS, you can use tools like GDB or LLDB to analyze minidumps generated by
Google Breakpad:

1. Using minidump_stackwalk (part of Google Breakpad): ` minidump_stackwalk

minidump_file.dmp /path/to/symbols > [Link] `

2. Using GDB: ` gdb /path/to/executable (gdb) core-file /path/to/minidump (gdb) bt `

1.6. Vulkan-Specific Crash Information

For Vulkan applications, it’s helpful to include additional information in your crash reports:

void log_vulkan_detailed_info(std::ofstream& log, vk::raii::PhysicalDevice&

physical_device,
vk::raii::Device& device) {
// Log physical device properties
auto properties = physical_device.getProperties();
log << "GPU: " << [Link] << std::endl;
log << "Driver Version: " << [Link] << std::endl;
log << "Vulkan API Version: "
<< VK_VERSION_MAJOR([Link]) << "."
<< VK_VERSION_MINOR([Link]) << "."
<< VK_VERSION_PATCH([Link]) << std::endl;

448
 // Log memory usage
auto memory_properties = physical_device.getMemoryProperties();
log << "Memory Heaps:" << std::endl;
for (uint32_t i = 0; i < memory_properties.memoryHeapCount; i++) {
log << " Heap " << i << ": "
<< (memory_properties.memoryHeaps[i].size / (1024 * 1024)) << " MB";
if (memory_properties.memoryHeaps[i].flags &
vk::MemoryHeapFlagBits::eDeviceLocal) {
log << " (Device Local)";
}
log << std::endl;
}

// Log enabled extensions

auto extensions = [Link]();
log << "Enabled Extensions:" << std::endl;
for (const auto& ext : extensions) {
log << " " << [Link] << " (version " << [Link] << ")" <<
std::endl;
}

// Log current pipeline cache state

// This can be useful for diagnosing shader-related crashes
try {
auto pipeline_cache_data = [Link]();
log << "Pipeline Cache Size: " << pipeline_cache_data.size() << " bytes" <<
std::endl;
} catch (const vk::SystemError& e) {
log << "Failed to get pipeline cache data: " << [Link]() << std::endl;
}
}

1.7. Integrating with Telemetry Systems

For production applications, you might want to automatically upload crash reports to a telemetry
system for analysis:

import std;
import vulkan_raii;
#include <curl/curl.h>

namespace crash_handler {
// ... existing code ...

std::string telemetry_url;
bool telemetry_enabled = false;

// Upload a minidump to the telemetry server

bool upload_minidump(const std::string& minidump_path) {

449
 if (!telemetry_enabled || telemetry_url.empty()) {
return false;
}

CURL* curl = curl_easy_init();

if (!curl) {
std::cerr << "Failed to initialize curl" << std::endl;
return false;
}

// Set up the form data

curl_mime* form = curl_mime_init(curl);

// Add the minidump file

curl_mimepart* field = curl_mime_addpart(form);
curl_mime_name(field, "minidump");
curl_mime_filedata(field, minidump_path.c_str());

// Add application information

field = curl_mime_addpart(form);
curl_mime_name(field, "product");
curl_mime_data(field, app_name.c_str(), CURL_ZERO_TERMINATED);

// Add version information

field = curl_mime_addpart(form);
curl_mime_name(field, "version");
curl_mime_data(field, "1.0.0", CURL_ZERO_TERMINATED); // Replace with your
version

// Set up the request

curl_easy_setopt(curl, CURLOPT_URL, telemetry_url.c_str());
curl_easy_setopt(curl, CURLOPT_MIMEPOST, form);

// Perform the request

CURLcode res = curl_easy_perform(curl);

// Clean up
curl_mime_free(form);
curl_easy_cleanup(curl);

if (res != CURLE_OK) {
std::cerr << "Failed to upload minidump: " << curl_easy_strerror(res) <<
std::endl;
return false;
}

return true;
}

// Enable telemetry
void enable_telemetry(const std::string& url) {

450
 telemetry_url = url;
telemetry_enabled = true;

// Initialize curl
curl_global_init(CURL_GLOBAL_ALL);
}

// Disable telemetry
void disable_telemetry() {
telemetry_enabled = false;

// Clean up curl
curl_global_cleanup();
}
}

1.8. Best Practices for Crash Handling (Vulkan/GPU-

focused)
To make crash data actionable for graphics issues, prefer these concrete steps:

1. Name and label aggressively

◦ Use VK_EXT_debug_utils to name all objects and insert command buffer labels at
pass/material boundaries and before large draw/dispatch batches. Persist a small in-
memory ring buffer of recent labels for inclusion in crash artifacts.

2. Prepare for device loss

◦ Implement a central handler for VK_ERROR_DEVICE_LOST. Stop submitting work, flush

logs/markers, request vendor GPU crash dump data, and exit. Avoid attempting recovery in
the same process unless you have a robust reinitialization path.

3. Capture GPU crash dumps on supported hardware

◦ Integrate NVIDIA Nsight Aftermath and/or AMD RGD depending on your target audience.
Ship with crash dumps enabled in development/beta builds; provide a toggle for users.

4. Make builds symbol-friendly

◦ Keep a mapping from pipeline/shader hashes to source/IR/SPIR-V and build IDs. Enable
shader debug info where feasible for diagnosis builds.

5. Record environment info

◦ Log driver version, Vulkan version, GPU name/PCI ID, pipeline cache UUID, app
build/version, and relevant feature toggles. Include this alongside minidumps and GPU
crash dumps.

6. Reproduce deterministically

◦ Provide a way to disable background variability (e.g., async streaming) and to replay a
captured sequence of commands/scenes to reproduce the crash locally.

7. Respect privacy and distribution concerns

451
 ◦ Clearly document what crash data is collected (minidumps, GPU crash dumps, logs) and
require opt‑in for uploads. Strip user-identifiable data.

1.9. Conclusion
Robust crash handling is essential for maintaining a high-quality Vulkan application. Combine
vendor GPU crash dumps (Aftermath, RGD, etc.) with CPU-side minidumps and thorough logging to
quickly diagnose and fix issues in production. Treat minidumps as complementary context; the
actionable details for graphics faults typically come from GPU crash dump tooling.

In the next section, we’ll explore Vulkan extensions for robustness, which can reduce undefined
behavior and help prevent crashes in the first place.

Previous: Debugging with VK_KHR_debug_utils and RenderDoc | Next: Vulkan Extensions for
Robustness :pp: ++

Tooling: Vulkan Extensions for

Robustness
1. Vulkan Extensions for Robustness
Vulkan’s explicit design gives developers fine-grained control over the graphics pipeline, but this
control comes with responsibility. Undefined behavior can occur when applications make mistakes
like accessing out-of-bounds memory or using uninitialized resources. In this section, we’ll explore
Vulkan extensions that can help make your application more robust against such issues, with a
particular focus on VK_EXT_robustness2.

1.1. Understanding Undefined Behavior in Vulkan

Before diving into robustness extensions, let’s understand what kinds of undefined behavior can
occur in Vulkan applications:

1. Out-of-bounds Access: Accessing memory outside the bounds of a buffer or image

2. Use-after-free: Using a resource after it has been destroyed

3. Uninitialized Memory: Reading from memory that hasn’t been initialized

4. Invalid Descriptors: Using descriptors that point to invalid or incompatible resources

5. Shader Execution Errors: Division by zero, infinite loops, etc.

In standard Vulkan, these errors can lead to unpredictable behavior, including:

• Application crashes

• GPU hangs requiring a system restart

• Corrupted rendering

452
 • Security vulnerabilities

• Inconsistent behavior across different hardware

Robustness extensions aim to provide more predictable behavior in these scenarios, often at a
small performance cost.

1.2. VK_EXT_robustness2 Extension

The VK_EXT_robustness2 extension is an improved version of the original VK_EXT_robustness
extension. It provides more comprehensive protection against undefined behavior, particularly for
out-of-bounds accesses.

1.2.1. Key Features

VK_EXT_robustness2 offers several important features:

1. Robust Buffer Access: Out-of-bounds reads from buffers return zero values instead of causing
undefined behavior

2. Robust Image Access: Out-of-bounds reads from images return zero or transparent black

3. Null Descriptor Handling: Reads from null descriptors return zero values

4. Robust Buffer Access 2: An improved version that also handles out-of-bounds writes by
discarding them

1.2.2. Enabling VK_EXT_robustness2

Let’s see how to enable and use this extension.

bool check_robustness2_support(vk::raii::PhysicalDevice& physical_device) {

// Check if the extension is supported
auto available_extensions = physical_device.enumerateDeviceExtensionProperties();

for (const auto& extension : available_extensions) {

if (strcmp([Link], VK_EXT_ROBUSTNESS_2_EXTENSION_NAME) == 0)
{
return true;
}
}

return false;
}

void enable_robustness2(vk::DeviceCreateInfo& device_create_info,

std::vector<const char*>& enabled_extensions) {
// Add the extension to the list of enabled extensions
enabled_extensions.push_back(VK_EXT_ROBUSTNESS_2_EXTENSION_NAME);
device_create_info.setPEnabledExtensionNames(enabled_extensions);

453
 // Set up the robustness2 features
vk::PhysicalDeviceRobustness2FeaturesEXT robustness2_features{};
robustness2_features.setRobustBufferAccess2(VK_TRUE);
robustness2_features.setRobustImageAccess2(VK_TRUE);
robustness2_features.setNullDescriptor(VK_TRUE);

// Add to the pNext chain

robustness2_features.pNext = device_create_info.pNext;
device_create_info.pNext = &robustness2_features;
}

vk::raii::Device create_robust_device(vk::raii::PhysicalDevice& physical_device,

vk::raii::Instance& instance) {
// Check for support
if (!check_robustness2_support(physical_device)) {
std::cerr << "VK_EXT_robustness2 is not supported on this device" <<
std::endl;
// Fall back to less robust behavior or abort
}

// Set up device creation

std::vector<const char*> enabled_extensions;
// Add your other required extensions here

vk::DeviceCreateInfo create_info{};
// Set up your queues, features, etc.

// Enable robustness2
enable_robustness2(create_info, enabled_extensions);

// Create the device

return vk::raii::Device(physical_device, create_info);
}

1.2.3. Using Robust Access in Practice

Once you’ve enabled the extension, robust buffer and image access will be applied automatically.
However, you should be aware of some considerations:

1. Performance Impact: Robust access can have a performance cost, as the GPU needs to perform
bounds checking

2. Not a Substitute for Correctness: While robustness extensions make your application more
resilient, they don’t fix the underlying bugs

3. Debug vs. Release: Consider enabling robustness in debug builds for development and testing,
but evaluate the performance impact for release builds

Here’s an example of how robust buffer access can prevent crashes:

// Without robust buffer access, this could crash or produce undefined results

454
 void potentially_dangerous_operation(vk::raii::CommandBuffer& cmd_buffer,
vk::raii::Buffer& buffer,
vk::raii::DescriptorSet& descriptor_set,
uint32_t dynamic_offset,
uint32_t buffer_size) {
// If dynamic_offset is too large, this would normally cause undefined behavior
// With robust buffer access, out-of-bounds reads will return zero
cmd_buffer.bindDescriptorSets(
vk::PipelineBindPoint::eCompute,
pipeline_layout,
0,
1,
&(*descriptor_set),
1,
&dynamic_offset
);

// Dispatch compute work that might read out of bounds

cmd_buffer.dispatch(buffer_size / 64 + 1, 1, 1); // Potentially too many
workgroups
}

1.3. Other Robustness Extensions

While VK_EXT_robustness2 is the focus of this section, there are other extensions that can help
improve application robustness:

1.3.1. VK_KHR_buffer_device_address

This extension allows you to use physical device addresses for buffers, which can be useful for
advanced techniques. It includes robustness features for handling invalid addresses (when
combined with robust access features like VK_EXT_robustness2 or core robustBufferAccess):

void enable_buffer_device_address(vk::DeviceCreateInfo& device_create_info,

std::vector<const char*>& enabled_extensions) {
enabled_extensions.push_back(VK_KHR_BUFFER_DEVICE_ADDRESS_EXTENSION_NAME);
device_create_info.setPEnabledExtensionNames(enabled_extensions);

// Enable Buffer Device Address features

vk::PhysicalDeviceBufferDeviceAddressFeatures buffer_device_address_features{};
buffer_device_address_features.setBufferDeviceAddress(VK_TRUE);
buffer_device_address_features.setBufferDeviceAddressCaptureReplay(VK_TRUE);

// Optionally chain robustness features to ensure invalid addresses read as zero

and writes are discarded
// (If you've already enabled VK_EXT_robustness2 elsewhere, this is not required
here.)
vk::PhysicalDeviceRobustness2FeaturesEXT robustness2_features{};
robustness2_features.setRobustBufferAccess2(VK_TRUE);

455
 robustness2_features.setRobustImageAccess2(VK_TRUE);
robustness2_features.setNullDescriptor(VK_TRUE);

// Chain features: robustness2 -> BDA -> existing pNext

robustness2_features.pNext = &buffer_device_address_features;
buffer_device_address_features.pNext = device_create_info.pNext;
device_create_info.pNext = &robustness2_features;
}

1.3.2. VK_EXT_descriptor_indexing

This extension allows for more flexible descriptor indexing, including robustness-related
capabilities such as tolerating out-of-bounds indices (reads become zero when robust access is
enabled), partially bound descriptor sets, and update-after-bind. To actually make use of these
behaviors you need to enable both device features and descriptor set layout binding flags:

void enable_descriptor_indexing(vk::DeviceCreateInfo& device_create_info,

std::vector<const char*>& enabled_extensions) {
enabled_extensions.push_back(VK_EXT_DESCRIPTOR_INDEXING_EXTENSION_NAME);
device_create_info.setPEnabledExtensionNames(enabled_extensions);

vk::PhysicalDeviceDescriptorIndexingFeatures indexing_features{};
// Shader indexing capabilities (commonly needed alongside robustness)
indexing_features.setShaderSampledImageArrayNonUniformIndexing(VK_TRUE);
indexing_features.setShaderStorageBufferArrayNonUniformIndexing(VK_TRUE);

// Robustness-enabling behaviors
indexing_features.setRuntimeDescriptorArray(VK_TRUE);
indexing_features.setDescriptorBindingPartiallyBound(VK_TRUE);
indexing_features.setDescriptorBindingSampledImageUpdateAfterBind(VK_TRUE);
indexing_features.setDescriptorBindingStorageBufferUpdateAfterBind(VK_TRUE);
indexing_features.setDescriptorBindingUpdateUnusedWhilePending(VK_TRUE);

// Add to the pNext chain (can be chained together with VK_EXT_robustness2)

indexing_features.pNext = device_create_info.pNext;
device_create_info.pNext = &indexing_features;
}

For descriptor arrays, you must also specify binding flags at layout creation time:

// Example: descriptor set layout with a runtime-sized array that can be partially
bound
vk::DescriptorSetLayoutBinding binding{};
[Link] = 0;
[Link] = vk::DescriptorType::eCombinedImageSampler;
[Link] = 128; // example array size; for true runtime arrays also
enable variable descriptor counts
[Link] = vk::ShaderStageFlagBits::eFragment;

456
 vk::DescriptorBindingFlags binding_flags =
vk::DescriptorBindingFlagBits::ePartiallyBound |
vk::DescriptorBindingFlagBits::eUpdateAfterBind;

vk::DescriptorSetLayoutBindingFlagsCreateInfo flags_ci{};
flags_ci.setBindingCount(1);
flags_ci.setPBindingFlags(&binding_flags);

vk::DescriptorSetLayoutCreateInfo dsl_ci{};
dsl_ci.setPBindings(&binding);
dsl_ci.setBindingCount(1);
// Required when using update-after-bind flags
// (some descriptor types require pool and layout flags to match update-after-bind
usage)
dsl_ci.flags |= vk::DescriptorSetLayoutCreateFlagBits::eUpdateAfterBindPool;

dsl_ci.pNext = &flags_ci;

vk::raii::DescriptorSetLayout set_layout{device, dsl_ci};

If you need truly variable-length descriptor arrays at runtime, also enable variable descriptor
counts and use the corresponding allocate info:

// Enable the device feature

// indexing_features.setDescriptorBindingVariableDescriptorCount(VK_TRUE); // do this
where features are enabled

uint32_t max_descriptors_for_set0 = 1024; // requested at allocation time

vk::DescriptorSetVariableDescriptorCountAllocateInfo variable_counts_info{};
variable_counts_info.setDescriptorSetCount(1);
variable_counts_info.setPDescriptorCounts(&max_descriptors_for_set0);

vk::DescriptorSetAllocateInfo alloc_info{};
alloc_info.setDescriptorPool(descriptor_pool);
alloc_info.setDescriptorSetCount(1);
alloc_info.setPSetLayouts(&*set_layout);
alloc_info.pNext = &variable_counts_info;

auto descriptor_sets = vk::raii::DescriptorSets{device, alloc_info};

Note: With VK_EXT_robustness2’s nullDescriptor = VK_TRUE and descriptor indexing’s partially-

bound behavior, unbound array elements will read as zero rather than invoking undefined
behavior.

457
1.4. Combining Robustness Extensions with Debugging
Tools
For maximum effectiveness, combine robustness extensions with the debugging tools we discussed
in previous sections:

class RobustVulkanApplication {
public:
RobustVulkanApplication() {
initialize_vulkan();
}

void run() {
// Main application loop
while (!should_close()) {
try {
update();
render();
} catch (const vk::SystemError& e) {
// Handle recoverable Vulkan errors
std::cerr << "Vulkan error: " << [Link]() << std::endl;
// Attempt recovery
if (!recover_from_error()) {
break;
}
}
}

cleanup();
}

private:
void initialize_vulkan() {
// Create instance with validation layers in debug builds
#ifdef _DEBUG
enable_validation_layers = true;
#else
enable_validation_layers = false;
#endif

instance = create_instance();

// Set up debug messenger if validation is enabled

if (enable_validation_layers) {
debug_messenger = create_debug_messenger(instance);
}

// Select physical device

physical_device = select_physical_device(instance);

458
 // Check for robustness support
has_robustness2 = check_robustness2_support(physical_device);

// Create logical device with robustness if available

device = create_device(physical_device);

// Initialize other Vulkan resources

// ...
}

vk::raii::Device create_device(vk::raii::PhysicalDevice& physical_device) {

std::vector<const char*> extensions;
// Add required extensions

vk::DeviceCreateInfo create_info{};
// Set up queues, etc.

// Enable robustness if available

if (has_robustness2) {
enable_robustness2(create_info, extensions);
}

// Enable other robustness-related extensions

enable_buffer_device_address(create_info, extensions);
enable_descriptor_indexing(create_info, extensions);

return vk::raii::Device(physical_device, create_info);

}

bool recover_from_error() {
// Attempt to recover from errors
// This might involve recreating swapchain, command buffers, etc.
try {
// Reset command buffers
// Recreate swapchain if needed
// ...
return true;
} catch (const std::exception& e) {
std::cerr << "Failed to recover: " << [Link]() << std::endl;
return false;
}
}

// Vulkan objects
vk::raii::Context context;
vk::raii::Instance instance{nullptr};
vk::raii::DebugUtilsMessengerEXT debug_messenger{nullptr};
vk::raii::PhysicalDevice physical_device{nullptr};
vk::raii::Device device{nullptr};

459
 // Flags
bool enable_validation_layers = false;
bool has_robustness2 = false;
};

1.5. Best Practices for Using Robustness Extensions

To make the most of robustness extensions:

1. Check for Support: Always check if the extension is supported before trying to use it

2. Fallback Behavior: Implement fallback behavior for devices that don’t support the extensions

3. Performance Testing: Measure the performance impact of enabling robustness features

4. Combine with Validation: Use validation layers during development to catch issues early

5. Don’t Rely on Robustness: Fix the underlying issues rather than relying on robustness
extensions to mask them

6. Document Usage: Clearly document which extensions your application requires and why

1.6. Conclusion
Vulkan robustness extensions, particularly VK_EXT_robustness2, provide valuable tools for making
your application more resilient to undefined behavior. By combining these extensions with proper
error handling, validation layers, and debugging tools, you can create a more stable and reliable
Vulkan application.

In the next and final section, we’ll summarize what we’ve learned about tooling for Vulkan
applications and discuss how to apply these techniques in your own projects.

Previous: Crash Handling and Minidumps | Next: Packaging and Distribution :pp: ++

Tooling: Packaging and Distribution

1. Packaging and Distributing Vulkan
Applications
After developing and testing your Vulkan application, the final step is to package and distribute it to
users. This process involves preparing your application for different platforms, handling
dependencies, and creating installers or packages that provide a smooth installation experience. In
this section, we’ll explore the key considerations and techniques for packaging and distributing
Vulkan applications.

460
1.1. Platform-Specific Packaging Considerations
Each platform has its own packaging formats and distribution mechanisms. Let’s explore the
considerations for the major platforms:

1.1.1. Windows Packaging

On Windows, common packaging formats include:

• Executable Installers: Created with tools like NSIS (Nullsoft Scriptable Install System), Inno
Setup, or WiX Toolset

• MSIX Packages: Modern Windows app packages that support clean installation and
uninstallation

• Portable Applications: Self-contained applications that don’t require installation

Here’s an example of creating a basic NSIS installer script for a Vulkan application:

; Basic NSIS installer script for a Vulkan application

!include "[Link]"

Name "My Vulkan Application"

OutFile "MyVulkanApp_Installer.exe"
InstallDir "$PROGRAMFILES\MyVulkanApp"

!insertmacro MUI_PAGE_WELCOME
!insertmacro MUI_PAGE_DIRECTORY
!insertmacro MUI_PAGE_INSTFILES
!insertmacro MUI_PAGE_FINISH

!insertmacro MUI_UNPAGE_CONFIRM
!insertmacro MUI_UNPAGE_INSTFILES

!insertmacro MUI_LANGUAGE "English"

Section "Install"
SetOutPath "$INSTDIR"

; Application files
File "[Link]"
File "*.dll"
File /r "shaders"
File /r "assets"

; Vulkan Runtime
File "[Link]"

; Create uninstaller
WriteUninstaller "$INSTDIR\[Link]"

461
 ; Create shortcuts
CreateDirectory "$SMPROGRAMS\MyVulkanApp"
CreateShortcut "$SMPROGRAMS\MyVulkanApp\[Link]" "$INSTDIR\[Link]"
CreateShortcut "$SMPROGRAMS\MyVulkanApp\[Link]" "$INSTDIR\[Link]"
SectionEnd

Section "Uninstall"
; Remove application files
Delete "$INSTDIR\[Link]"
Delete "$INSTDIR\*.dll"
RMDir /r "$INSTDIR\shaders"
RMDir /r "$INSTDIR\assets"

; Remove uninstaller
Delete "$INSTDIR\[Link]"

; Remove shortcuts
Delete "$SMPROGRAMS\MyVulkanApp\[Link]"
Delete "$SMPROGRAMS\MyVulkanApp\[Link]"
RMDir "$SMPROGRAMS\MyVulkanApp"

; Remove install directory

RMDir "$INSTDIR"
SectionEnd

1.1.2. Linux Packaging

On Linux, common packaging formats include:

• DEB Packages: For Debian-based distributions (Ubuntu, Debian, etc.)

• RPM Packages: For Red Hat-based distributions (Fedora, CentOS, etc.)

• AppImage: Self-contained applications that run on most Linux distributions

• Flatpak: Sandboxed applications with controlled access to system resources

• Snap: Universal Linux packages maintained by Canonical

Here’s an example of creating a basic AppImage for a Vulkan application:

#!/bin/bash
# Script to create an AppImage for a Vulkan application

# Create AppDir structure

mkdir -p AppDir/usr/bin
mkdir -p AppDir/usr/lib
mkdir -p AppDir/usr/share/applications
mkdir -p AppDir/usr/share/icons/hicolor/256x256/apps
mkdir -p AppDir/usr/share/metainfo

462
# Copy application binary
cp build/MyVulkanApp AppDir/usr/bin/

# Copy dependencies (excluding system libraries)

ldd build/MyVulkanApp | grep "=> /" | awk '{print $3}' | xargs -I{} cp -v {}
AppDir/usr/lib/

# Copy Vulkan loader

cp /usr/lib/[Link].1 AppDir/usr/lib/

# Copy application data

cp -r assets AppDir/usr/share/MyVulkanApp/assets
cp -r shaders AppDir/usr/share/MyVulkanApp/shaders

# Create desktop file

cat > AppDir/usr/share/applications/[Link] << EOF
[Desktop Entry]
Name=My Vulkan Application
Exec=MyVulkanApp
Icon=MyVulkanApp
Type=Application
Categories=Graphics;
EOF

# Copy icon
cp [Link] AppDir/usr/share/icons/hicolor/256x256/apps/[Link]

# Create AppStream metadata

cat > AppDir/usr/share/metainfo/[Link] << EOF
<?xml version="1.0" encoding="UTF-8"?>
<component type="desktop-application">
<id>[Link]</id>
<name>My Vulkan Application</name>
<summary>A Vulkan-powered application</summary>
<description>
<p>
My Vulkan Application is a high-performance graphics application
built with the Vulkan API.
</p>
</description>
<url type="homepage">[Link]
<releases>
<release version="1.0.0" date="2023-01-01"/>
</releases>
</component>
EOF

# Create AppRun script

cat > AppDir/AppRun << EOF
#!/bin/bash
SELF=\$(readlink -f "\$0")

463
 HERE=\$(dirname "\$SELF")
export PATH="\${HERE}/usr/bin:\${PATH}"
export LD_LIBRARY_PATH="\${HERE}/usr/lib:\${LD_LIBRARY_PATH}"
export VK_LAYER_PATH="\${HERE}/usr/share/vulkan/explicit_layer.d"
export VK_ICD_FILENAMES="\${HERE}/usr/share/vulkan/icd.d/vulkan_icd.json"
"\${HERE}/usr/bin/MyVulkanApp" "$@"
EOF

chmod +x AppDir/AppRun

# Download appimagetool
wget -c
"[Link]
x86_64.AppImage"
chmod +x appimagetool-x86_64.AppImage

# Create the AppImage

./appimagetool-x86_64.AppImage AppDir MyVulkanApp-x86_64.AppImage

1.1.3. macOS Packaging

On macOS, common packaging formats include:

• Application Bundles (.app): The standard format for macOS applications

• Disk Images (.dmg): Mountable disk images containing the application

• Packages (.pkg): Installer packages for more complex installations

Here’s an example of creating a basic macOS application bundle structure for a Vulkan application
using MoltenVK:

#!/bin/bash
# Script to create a macOS application bundle for a Vulkan application

# Create bundle structure

mkdir -p [Link]/Contents/MacOS
mkdir -p [Link]/Contents/Resources
mkdir -p [Link]/Contents/Frameworks

# Copy application binary

cp build/MyVulkanApp [Link]/Contents/MacOS/

# Copy MoltenVK framework

cp -R $VULKAN_SDK/macOS/Frameworks/[Link]
[Link]/Contents/Frameworks/

# Copy application resources

cp -r assets [Link]/Contents/Resources/assets
cp -r shaders [Link]/Contents/Resources/shaders
cp [Link] [Link]/Contents/Resources/

464
 # Create [Link]
cat > [Link]/Contents/[Link] << EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
"[Link]
<plist version="1.0">
<dict>
<key>CFBundleExecutable</key>
<string>MyVulkanApp</string>
<key>CFBundleIconFile</key>
<string>[Link]</string>
<key>CFBundleIdentifier</key>
<string>[Link]</string>
<key>CFBundleInfoDictionaryVersion</key>
<string>6.0</string>
<key>CFBundleName</key>
<string>My Vulkan Application</string>
<key>CFBundlePackageType</key>
<string>APPL</string>
<key>CFBundleShortVersionString</key>
<string>1.0.0</string>
<key>CFBundleVersion</key>
<string>1</string>
<key>NSHighResolutionCapable</key>
<true/>
</dict>
</plist>
EOF

# Create DMG (optional)

hdiutil create -volname "My Vulkan Application" -srcfolder [Link] -ov -format
UDZO [Link]

1.2. Handling Vulkan Dependencies

One of the key considerations when packaging Vulkan applications is handling the Vulkan loader
and any required extensions.

1.2.1. Vulkan Loader

The Vulkan loader is the component that connects your application to the Vulkan implementation
on the user’s system. There are different approaches to handling the loader:

1. Rely on System-Installed Loader: Require users to have the Vulkan SDK or drivers installed

2. Bundle the Loader: Include the Vulkan loader with your application

3. Hybrid Approach: Check for a system-installed loader and fall back to a bundled one if not
found

465
Here’s an example of a hybrid approach:

import std;
import vulkan_raii;

class VulkanLoader {
public:
static bool initialize() {
try {
// First, try to use the system-installed Vulkan loader
if (try_system_loader()) {
std::cout << "Using system-installed Vulkan loader" << std::endl;
return true;
}

// If that fails, try to use the bundled loader

if (try_bundled_loader()) {
std::cout << "Using bundled Vulkan loader" << std::endl;
return true;
}

// If both approaches fail, report an error

std::cerr << "Failed to initialize Vulkan loader" << std::endl;
return false;
} catch (const std::exception& e) {
std::cerr << "Error initializing Vulkan loader: " << [Link]() <<
std::endl;
return false;
}
}

private:
static bool try_system_loader() {
try {
// Create a Vulkan instance to test if the system loader works
vk::raii::Context context;
vk::ApplicationInfo app_info{};
app_info.setApiVersion(VK_API_VERSION_1_2);

vk::InstanceCreateInfo create_info{};
create_info.setPApplicationInfo(&app_info);

vk::raii::Instance instance(context, create_info);

return true;
} catch (...) {
return false;
}
}

static bool try_bundled_loader() {

466
 try {
// Set the path to the bundled Vulkan loader
#if defined(_WIN32)
std::string loader_path = get_executable_path() + "\\[Link]";
SetDllDirectoryA(get_executable_path().c_str());
#elif defined(__linux__)
std::string loader_path = get_executable_path() + "/[Link].1";
setenv("LD_LIBRARY_PATH", get_executable_path().c_str(), 1);
#elif defined(__APPLE__)
std::string loader_path = get_executable_path() +
"/../Frameworks/[Link]";
setenv("DYLD_LIBRARY_PATH", (get_executable_path() +
"/../Frameworks").c_str(), 1);
#endif

// Check if the bundled loader exists

if (!std::filesystem::exists(loader_path)) {
return false;
}

// Try to create a Vulkan instance using the bundled loader

vk::raii::Context context;
vk::ApplicationInfo app_info{};
app_info.setApiVersion(VK_API_VERSION_1_2);

vk::InstanceCreateInfo create_info{};
create_info.setPApplicationInfo(&app_info);

vk::raii::Instance instance(context, create_info);

return true;
} catch (...) {
return false;
}
}

static std::string get_executable_path() {

#if defined(_WIN32)
char path[MAX_PATH];
GetModuleFileNameA(NULL, path, MAX_PATH);
std::string exe_path(path);
return exe_path.substr(0, exe_path.find_last_of("\\/"));
#elif defined(__linux__)
char result[PATH_MAX];
ssize_t count = readlink("/proc/self/exe", result, PATH_MAX);
std::string exe_path(result, (count > 0) ? count : 0);
return exe_path.substr(0, exe_path.find_last_of("/"));
#elif defined(__APPLE__)
char path[PATH_MAX];
uint32_t size = sizeof(path);
if (_NSGetExecutablePath(path, &size) == 0) {
std::string exe_path(path);

467
 return exe_path.substr(0, exe_path.find_last_of("/"));
}
return "";
#endif
}
};

1.2.2. Vulkan Layers and Extensions

If your application requires specific Vulkan layers or extensions, you need to handle them
appropriately:

1. Document Requirements: Clearly document which extensions your application requires

2. Check for Support: Always check if required extensions are available before using them

3. Provide Fallbacks: Implement fallback behavior for missing extensions when possible

4. Bundle Layers: For development tools, consider bundling validation layers

1.3. Shader Management

Shaders are a critical part of Vulkan applications, and they need special consideration during
packaging:

1. Pre-Compile Shaders: Package pre-compiled SPIR-V shaders rather than GLSL source

2. Shader Versioning: Implement a versioning system for shaders to handle updates

3. Shader Optimization: Consider optimizing shaders for different hardware targets

4. Shader Caching: Implement a shader cache to improve load times

Here’s an example of a shader management system for a packaged application:

import std;
import vulkan_raii;

class ShaderManager {
public:
ShaderManager(vk::raii::Device& device) : device(device) {
// Determine the shader directory based on the application's location
shader_dir = get_application_directory() + "/shaders";

// Create a shader module cache

shader_cache.reserve(100); // Reserve space for up to 100 shader modules
}

vk::raii::ShaderModule load_shader(const std::string& name) {

// Check if the shader is already in the cache
auto it = shader_cache.find(name);
if (it != shader_cache.end()) {

468
 return vk::raii::ShaderModule(nullptr, nullptr, nullptr); // Return a copy
of the cached module
}

// Load the shader from the package

std::string path = shader_dir + "/" + name + ".spv";
std::vector<char> code = read_file(path);

// Create the shader module

vk::ShaderModuleCreateInfo create_info{};
create_info.setCodeSize([Link]());
create_info.setPCode(reinterpret_cast<const uint32_t*>([Link]()));

// Create and cache the shader module

vk::raii::ShaderModule module(device, create_info);
shader_cache[name] = std::move(module);

return vk::raii::ShaderModule(nullptr, nullptr, nullptr); // Return a copy of

the cached module
}

void clear_cache() {
shader_cache.clear();
}

private:
std::string get_application_directory() {
// Platform-specific code to get the application directory
// ...
return "."; // Placeholder
}

std::vector<char> read_file(const std::string& path) {

std::ifstream file(path, std::ios::ate | std::ios::binary);
if (!file.is_open()) {
throw std::runtime_error("Failed to open shader file: " + path);
}

size_t file_size = static_cast<size_t>([Link]());

std::vector<char> buffer(file_size);

[Link](0);
[Link]([Link](), file_size);
[Link]();

return buffer;
}

vk::raii::Device& device;
std::string shader_dir;
std::unordered_map<std::string, vk::raii::ShaderModule> shader_cache;

469
 };

1.4. Automated Packaging with CI/CD

As we discussed in the CI/CD section, automating the packaging process can save time and reduce
errors. Here’s how to integrate packaging into your CI/CD pipeline:

1. Build Matrix: Set up a build matrix for different platforms and configurations

2. Packaging Scripts: Create scripts for each platform’s packaging process

3. Version Management: Automatically increment version numbers based on git tags or other
criteria

4. Artifact Storage: Store packaged applications as build artifacts

5. Release Automation: Automate the release process to distribution platforms

Here’s an example of a GitHub Actions workflow that includes packaging:

name: Build and Package

on:
push:
tags:
- 'v*'

jobs:
build-and-package:
runs-on: ${{ [Link] }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
include:
- os: ubuntu-latest
package-script: ./scripts/package_linux.sh
artifact-name: MyVulkanApp-Linux
artifact-path: MyVulkanApp-x86_64.AppImage
- os: windows-latest
package-script: .\scripts\package_windows.bat
artifact-name: MyVulkanApp-Windows
artifact-path: MyVulkanApp_Installer.exe
- os: macos-latest
package-script: ./scripts/package_macos.sh
artifact-name: MyVulkanApp-macOS
artifact-path: [Link]

steps:
- uses: actions/checkout@v3
with:
submodules: recursive

470
 - name: Install Vulkan SDK
uses: humbletim/install-vulkan-sdk@v1.1.1
with:
version: latest
cache: true

- name: Configure CMake

run: cmake -B ${{[Link]}}/build -DCMAKE_BUILD_TYPE=Release

- name: Build
run: cmake --build ${{[Link]}}/build --config Release

- name: Package
run: ${{ [Link]-script }}

- name: Upload Package

uses: actions/upload-artifact@v3
with:
name: ${{ [Link]-name }}
path: ${{ [Link]-path }}

create-release:
needs: build-and-package
runs-on: ubuntu-latest
steps:
- name: Download all artifacts
uses: actions/download-artifact@v3

- name: Create Release

uses: softprops/action-gh-release@v1
with:
files: |
MyVulkanApp-Linux/MyVulkanApp-x86_64.AppImage
MyVulkanApp-Windows/MyVulkanApp_Installer.exe
MyVulkanApp-macOS/[Link]

1.5. Conclusion
Packaging and distribution are critical steps in the lifecycle of a Vulkan application. By carefully
considering platform-specific requirements, handling dependencies appropriately, and automating
the packaging process, you can ensure a smooth experience for your users across different
platforms.

Remember that the goal of packaging is to make installation and updates as seamless as possible for
your users. Invest time in creating a robust packaging and distribution system, and your users will
benefit from a more professional and reliable application.

In the next and final section, we’ll summarize what we’ve learned throughout this chapter on

471
tooling for Vulkan applications.

Previous: Vulkan Extensions for Robustness | Next: Conclusion :pp: ++

Tooling: Conclusion
1. Conclusion
In this chapter, we’ve explored a comprehensive set of tools and techniques for developing,
debugging, and distributing Vulkan applications. Let’s summarize what we’ve learned and discuss
how to apply these techniques in your own projects.

1.1. What We’ve Learned

1.1.1. CI/CD for Vulkan Projects

We started by implementing a continuous integration and continuous deployment pipeline

specifically tailored for Vulkan applications. This included:

• Setting up a basic CI/CD pipeline with GitHub Actions

• Handling Vulkan-specific considerations like SDK installation and GPU availability

• Automating testing for Vulkan applications

• Packaging and distributing Vulkan applications across different platforms

A well-designed CI/CD pipeline helps ensure consistent quality across builds and platforms,
catching issues early in the development process.

1.1.2. Debugging with VK_KHR_debug_utils and RenderDoc

We then explored powerful debugging tools for Vulkan applications:

• Using the VK_KHR_debug_utils extension for in-application debugging

• Labeling objects, command buffers, and queue operations for better debugging

• Integrating RenderDoc for frame capture and analysis

• Combining these approaches for comprehensive debugging

These tools provide visibility into the complex operations happening on the GPU, making it easier
to identify and fix issues in your rendering pipeline.

1.1.3. Crash Handling and Minidumps

Next, we implemented robust crash handling mechanisms:

• Basic crash handling for exceptions and signals

472
 • Generating minidumps for detailed crash analysis

• Collecting Vulkan-specific information in crash reports

• Integrating with telemetry systems for production applications

Proper crash handling helps you diagnose and fix issues that occur in production environments,
leading to a more stable and reliable application.

1.1.4. Vulkan Extensions for Robustness

Finally, we explored Vulkan extensions that can help make your application more resilient to
undefined behavior:

• Using VK_EXT_robustness2 for handling out-of-bounds accesses and null descriptors

• Implementing other robustness extensions like VK_KHR_buffer_device_address and

VK_EXT_descriptor_indexing

• Combining robustness extensions with debugging tools for maximum effectiveness

These extensions provide valuable tools for making your application more robust against common
errors, though they should not be seen as a substitute for fixing the underlying issues.

1.2. Putting It All Together

Throughout this chapter, we’ve used modern C++20 modules and the vk::raii namespace for
resource management. This approach offers several advantages:

• Improved code organization with modules

• Automatic resource cleanup with RAII

• More readable and maintainable code

• Better error handling with exceptions

Let’s see how all these components can work together in a complete application:

import std;
import vulkan_raii;

class VulkanApplication {
public:
VulkanApplication() {
// Initialize crash handler
crash_handler::initialize("MyVulkanApp", "crash_logs");

// Initialize Vulkan with debugging and robustness

initialize_vulkan();
}

void run() {

473
 // Main application loop
while (!should_close()) {
try {
update();
render();
} catch (const vk::SystemError& e) {
// Handle recoverable Vulkan errors
std::cerr << "Vulkan error: " << [Link]() << std::endl;
if (!recover_from_error()) {
break;
}
}
}

cleanup();
}

private:
void initialize_vulkan() {
// Create instance with validation layers in debug builds
#ifdef _DEBUG
enable_validation_layers = true;
#else
enable_validation_layers = false;
#endif

// Create instance
instance = create_instance();

// Set up debug messenger if validation is enabled

if (enable_validation_layers) {
debug_messenger = create_debug_messenger(instance);
}

// Select physical device

physical_device = select_physical_device(instance);

// Check for robustness support

has_robustness2 = check_robustness2_support(physical_device);

// Create logical device with robustness if available

device = create_device(physical_device);

// Name Vulkan objects for debugging

if (enable_validation_layers) {
debug_utils::set_name(device, *device, "Main Device");
// Name other objects as they're created
}

// Initialize other Vulkan resources

// ...

474
 }

void render() {
// Begin frame
auto cmd_buffer = begin_frame();

// Label command buffer regions for debugging

if (enable_validation_layers) {
vk::DebugUtilsLabelEXT label_info{};
label_info.setPLabelName("Main Render Pass");
label_info.setColor(std::array<float, 4>{0.0f, 1.0f, 0.0f, 1.0f});
cmd_buffer.beginDebugUtilsLabelEXT(label_info);
}

// Record rendering commands

// ...

// End debug label

if (enable_validation_layers) {
cmd_buffer.endDebugUtilsLabelEXT();
}

// End frame
end_frame(cmd_buffer);

// Capture frame with RenderDoc if requested

if (capture_next_frame) {
if (renderdoc_api) {
renderdoc_api->TriggerCapture();
}
capture_next_frame = false;
}
}

// Vulkan objects
vk::raii::Context context;
vk::raii::Instance instance{nullptr};
vk::raii::DebugUtilsMessengerEXT debug_messenger{nullptr};
vk::raii::PhysicalDevice physical_device{nullptr};
vk::raii::Device device{nullptr};

// Flags
bool enable_validation_layers = false;
bool has_robustness2 = false;
bool capture_next_frame = false;

// RenderDoc API
RENDERDOC_API_1_4_1* renderdoc_api = nullptr;
};

475
1.3. Best Practices for Professional Vulkan
Development
Based on what we’ve covered in this chapter, here are some best practices for professional Vulkan
development:

1. Automate Your Workflow: Use CI/CD pipelines to automate building, testing, and packaging
your application.

2. Debug Early and Often: Use validation layers and debugging tools throughout development,
not just when issues arise.

3. Name Your Objects: Use VK_KHR_debug_utils to give meaningful names to Vulkan objects,
making debugging much easier.

4. Prepare for Crashes: Implement robust crash handling and reporting mechanisms from the
start of your project.

5. Consider Robustness: Evaluate the trade-offs of using robustness extensions based on your
application’s needs.

6. Test Across Platforms: Vulkan applications can behave differently across different hardware
and drivers, so test extensively.

7. Document Your Requirements: Clearly document which Vulkan extensions and features your
application requires.

8. Stay Updated: The Vulkan ecosystem is constantly evolving, so stay informed about new
extensions and tools.

1.4. Future Directions

As Vulkan continues to evolve, new tools and techniques will emerge for developing, debugging,
and distributing applications. Some areas to watch include:

• Improved Debugging Tools: Tools like RenderDoc continue to add new features for Vulkan
debugging.

• Ray Tracing Tooling: As ray tracing becomes more common, expect more specialized tools for
debugging and optimizing ray tracing pipelines.

• Machine Learning Integration: Tools that use machine learning to identify potential issues or
optimize performance.

• Cross-API Development: Tools that help manage development across multiple graphics APIs
(Vulkan, DirectX, Metal).

1.5. Final Thoughts

Developing professional Vulkan applications requires more than just understanding the API—it
requires a comprehensive tooling ecosystem that supports the entire development lifecycle. By
implementing the tools and techniques covered in this chapter, you’ll be well-equipped to develop,
debug, and distribute high-quality Vulkan applications.

476
Remember that tooling is an investment that pays dividends throughout the development process.
Time spent setting up good CI/CD pipelines, debugging tools, and crash handling mechanisms will
save you countless hours of troubleshooting and manual work later on.

1.6. Code Examples

The complete code for this chapter can be found in the following files:

• simple_engine/32_cicd_setup.cpp: Implementation of CI/CD for Vulkan projects

• simple_engine/33_debug_utils.cpp: Implementation of debugging with VK_KHR_debug_utils and

RenderDoc

• simple_engine/34_crash_handling.cpp: Implementation of crash handling and minidumps

• simple_engine/35_robustness_extensions.cpp: Implementation of Vulkan extensions for

robustness

CI/CD Setup C++ code Debug Utils C++ code Crash Handling C++ code Robustness Extensions C++
code

Previous: Packaging and Distribution | Next: Mobile Development :pp: ++

Tooling
This chapter covers essential tooling and techniques for developing, debugging, and distributing
Vulkan applications, with a focus on using modern C++20 modules and the vk::raii namespace.

• Introduction

• CI/CD for Vulkan Projects

• Debugging with VK_KHR_debug_utils and RenderDoc

• Crash Handling and Minidumps

• Vulkan Extensions for Robustness

• Packaging and Distribution

• Conclusion

Previous: Subsystems Conclusion | Back to Building a Simple Engine :pp: ++

Mobile Development: Introduction

1. Introduction to Mobile Development
In previous chapters, we’ve built a solid foundation for our simple engine, implementing core
components like the rendering pipeline, camera systems, model loading, essential subsystems, and
tooling. Now, we’re ready to explore how to adapt our engine for mobile platforms, specifically

477
Android and iOS.

Mobile development presents unique challenges and opportunities for Vulkan applications. The
constraints of mobile hardware—limited power, memory, and thermal capacity—require careful
optimization and consideration of platform-specific features. At the same time, mobile platforms
offer exciting possibilities for reaching a wider audience with your applications.

1.1. What We’ll Cover

This chapter will guide you through the complex landscape of mobile Vulkan development, where
desktop assumptions often don’t apply. We’ll start by examining the platform-specific requirements
of Android and iOS, which present unique challenges in setup, lifecycle management, and input
handling. Mobile applications face constraints that desktop applications rarely encounter—sudden
interruptions, battery concerns, and varying hardware capabilities all require careful
consideration in your engine design.

Performance optimization takes on critical importance in mobile environments where every watt
of power consumption and every millisecond of frame time affects user experience. We’ll explore
essential techniques like efficient texture formats, along with mobile-specific optimizations that can
mean the difference between smooth performance and user frustration.

Understanding the fundamental architectural differences between mobile and desktop GPUs
becomes essential for effective optimization. We’ll compare Tile-Based Rendering (TBR) and
Immediate Mode Rendering (IMR) approaches, helping you understand why techniques that work
well on desktop might perform poorly on mobile, and how to design rendering strategies that
leverage mobile GPU strengths.

Finally, we’ll explore the Vulkan extensions specifically designed for mobile platforms. Extensions
like VK_KHR_dynamic_rendering_local_read, VK_KHR_dynamic_rendering, and
VK_EXT_shader_tile_image unlock performance opportunities that can dramatically improve your
application’s efficiency on mobile hardware, transforming acceptable performance into
exceptional user experiences.

1.2. Prerequisites
This chapter represents the culmination of everything we’ve built throughout the previous
chapters, as mobile development requires deep integration with all engine systems. You’ll need
solid mastery of Vulkan fundamentals and the engine architecture we’ve developed, since mobile
optimization often requires fine-tuning at every level—from resource management and rendering
pipelines to memory allocation and synchronization.

Modern C++ expertise becomes particularly valuable in mobile development, where performance
constraints demand efficient code and careful resource management. C++17 and C++20 features like
constexpr, structured bindings, and concepts help create mobile-optimized code that performs well
under strict power and thermal limitations.

Understanding basic mobile development concepts will provide crucial context for the platform-
specific decisions we’ll make. Mobile applications operate under constraints that desktop

478
applications rarely face—app lifecycle events, varying screen densities, touch input paradigms, and
the need to preserve battery life all influence how we design and implement our Vulkan engine for
mobile platforms.

You should also be familiar with the following chapters from the main tutorial:

• Basic Vulkan concepts:

◦ Command buffers

◦ Graphics pipelines

• Vertex and index buffers

• Uniform buffers

• Compute shaders

Let’s begin by exploring the platform considerations for Android and iOS.

Previous: Tooling Conclusion | Next: Platform Considerations for Android and iOS :pp: ++

Mobile Development: Platform

Considerations
1. Platform Considerations for Android and
iOS
Developing Vulkan applications for mobile platforms requires understanding the specific
requirements and constraints of Android and iOS. In this section, we’ll explore the key
considerations for each platform and how to adapt your engine accordingly.

1.1. Android Platform Considerations

Android has supported Vulkan since version 7.0 (Nougat), but the level of support varies across
devices. Here are the key considerations for Android development:

1.1.1. Setting Up Vulkan on Android

To use Vulkan on Android, you need to:

• Declare Vulkan Support: In your [Link], declare that your application uses
Vulkan:

<manifest xmlns:android="[Link]
package="[Link]">
<uses-feature android:name="[Link]"

479
 android:version="0x400003" android:required="true" />
<uses-feature android:name="[Link]" android:version="0"
android:required="true" />
<!-- For devices that support Vulkan but don't declare it -->
<uses-feature android:name="[Link]"
android:required="false" />
<!-- ... -->
</manifest>

• Initialize Vulkan: Use the Android Native Development Kit (NDK) to initialize Vulkan. The
process is similar to desktop Vulkan, but you’ll need to obtain the native window handle from
Android:

// Get the native window handle from Android

ANativeWindow* native_window = ANativeWindow_fromSurface(env, surface);

// Create the Vulkan surface

VkAndroidSurfaceCreateInfoKHR create_info{};
create_info.sType = VK_STRUCTURE_TYPE_ANDROID_SURFACE_CREATE_INFO_KHR;
create_info.window = native_window;

VkSurfaceKHR vulkan_surface;
vkCreateAndroidSurfaceKHR(instance, &create_info, nullptr, &vulkan_surface);

1.1.2. Android Lifecycle Management

Android applications have a complex lifecycle that you need to handle properly:

1. Activity Pausing and Resuming: When your application is paused (e.g., when the user switches
to another app), you should release Vulkan resources and recreate them when the application
resumes.

2. Surface Changes: The surface can change due to configuration changes (e.g., rotation). You
need to handle these changes by recreating the swapchain.

3. Memory Pressure: Android can reclaim memory from your application at any time. Design
your engine to handle memory pressure gracefully.

1.1.3. Android Input Handling

Android input handling differs from desktop:

1. Touch Input: Instead of mouse input, you’ll need to handle touch events, including multi-touch
gestures.

2. Sensors: Android devices have various sensors (accelerometer, gyroscope, etc.) that you can use
for input.

480
1.1.4. Vendor-Specific Considerations

Different Android device manufacturers may have specific considerations:

1. Custom Android Versions: Many manufacturers use customized versions of Android. Test your
application on various devices to ensure compatibility.

2. GPU Architectures: Different vendors use different GPU architectures (Adreno, Mali, PowerVR,
etc.). Each has unique performance characteristics.

3. Alternative App Stores: Some devices may not have Google Play Services. Consider distributing
through alternative app stores when necessary.

4. SoC Variations: System-on-Chip variations affect performance. Most mobile GPUs are tile-based
renderers, so optimize your rendering pipeline accordingly using the techniques described in
the TBR section.

// Example of checking for specific device vendors

bool check_device_vendor(vk::PhysicalDevice physical_device, uint32_t vendor_id) {
vk::PhysicalDeviceProperties props = physical_device.getProperties();
return [Link] == vendor_id;
}

// Common vendor IDs

const uint32_t VENDOR_ID_QUALCOMM = 0x5143; // Adreno
const uint32_t VENDOR_ID_ARM = 0x13B5; // Mali
const uint32_t VENDOR_ID_IMAGINATION = 0x1010; // PowerVR
const uint32_t VENDOR_ID_HUAWEI = 0x19E5; // Kirin

// You can then apply vendor-specific optimizations if needed

void configure_for_device(vk::PhysicalDevice physical_device) {
if (check_device_vendor(physical_device, VENDOR_ID_HUAWEI)) {
// Apply Huawei-specific optimizations if needed
}
// Handle other vendors as needed
}

1.2. iOS Platform Considerations

iOS supports Vulkan through MoltenVK, a translation layer that maps Vulkan to Metal. Here are the
key considerations for iOS development:

1.2.1. Setting Up MoltenVK on iOS

To use Vulkan on iOS, you need to:

• Include MoltenVK: Add the MoltenVK framework to your Xcode project.

• Initialize MoltenVK: Initialize MoltenVK before creating your Vulkan instance:

481
 // Initialize MoltenVK
MVKConfiguration config{};
vkGetMoltenVKConfigurationMVK(nullptr, &config);
[Link] = true; // Enable debug mode during development
vkSetMoltenVKConfigurationMVK(nullptr, &config);

// Create Vulkan instance as usual

// ...

• Create a Metal-Compatible Surface: Create a Vulkan surface from a CAMetalLayer:

// Get the Metal layer from your UIView

CAMetalLayer* metal_layer = (CAMetalLayer*)layer;

// Create the Vulkan surface

VkMetalSurfaceCreateInfoEXT create_info{};
create_info.sType = VK_STRUCTURE_TYPE_METAL_SURFACE_CREATE_INFO_EXT;
create_info.pLayer = metal_layer;

VkSurfaceKHR vulkan_surface;
vkCreateMetalSurfaceEXT(instance, &create_info, nullptr, &vulkan_surface);

1.2.2. iOS Lifecycle Management

iOS applications also have a lifecycle that you need to handle:

1. Application State Changes: Handle applicationWillResignActive, applicationDidBecomeActive,

etc., by releasing and recreating Vulkan resources as needed.

2. Memory Warnings: iOS can send memory warnings when the system is low on memory.
Handle these by releasing non-essential resources.

1.2.3. iOS Input Handling

iOS input handling is similar to Android but with some differences:

1. Touch Input: iOS has its own touch event system that you’ll need to integrate with your engine.

2. Sensors: iOS devices also have various sensors that you can use for input.

1.3. Cross-Platform Considerations

To maintain a single codebase for both Android and iOS (and potentially desktop), consider:

• Abstraction Layers: Create platform-specific abstraction layers for window creation, input
handling, and other platform-specific functionality.

• Conditional Compilation: Use preprocessor directives to handle platform-specific code:

482
 #ifdef __ANDROID__
// Android-specific code
#elif defined(__APPLE__)
// iOS-specific code
#else
// Desktop-specific code
#endif

• Feature Detection: Use Vulkan’s feature detection mechanisms to adapt to the capabilities of
the device, rather than making assumptions based on the platform.

1.4. Best Practices for Mobile Platform Integration

1. Test on Real Devices: Emulators and simulators may not accurately represent the performance
and behavior of real devices.

2. Handle Different Screen Sizes and Aspect Ratios: Mobile devices come in various sizes and
aspect ratios. Design your UI and rendering to adapt accordingly.

3. Consider Battery Life: Mobile users are sensitive to battery drain. Optimize your engine to
minimize power consumption.

4. Respect Platform Guidelines: Follow the design and user experience guidelines for each
platform to ensure your application feels native.

In the next section, we’ll explore performance optimizations specifically tailored for mobile
hardware, focusing on texture formats and memory usage.

Previous: Introduction | Next: Performance Optimizations for Mobile :pp: ++

Mobile Development: Performance

Optimizations
1. Performance Optimizations for Mobile
Mobile devices have significantly different hardware constraints compared to desktop systems. In
this section, we’ll explore key performance optimizations that are essential for achieving good
performance on mobile platforms.

This chapter covers general mobile performance. For practices that arise
 specifically because the GPU is tile-based (TBR), see Rendering Approaches: Tile-
Based Rendering.

483
1.1. Texture Optimizations
We focus on mobile‑specific decisions here. For general Vulkan image creation,
staging uploads, and descriptor setup, refer back to earlier chapters in the engine
 series—Resource Management, Rendering Pipeline—or the Vulkan Guide
([Link]

Textures are often the largest consumers of memory in a graphics application. Optimizing them is
crucial for performance on both mobile and desktop.

1.1.1. Efficient Texture Formats

Choosing the right texture format is crucial across platforms; what differs is which formats are
natively supported by a given device/driver:

1. Compressed Formats: Use hardware-supported compressed formats whenever possible:

◦ ASTC (Adaptive Scalable Texture Compression): Widely supported on modern mobile GPUs
and increasingly available on desktop; excellent quality-to-size ratio with flexible block
sizes.

◦ ETC2/EAC: Required for OpenGL ES 3.0 and supported by most Android devices; commonly
available on Vulkan stacks, too.

◦ PVRTC: Primarily supported on iOS devices with PowerVR GPUs.

◦ BC (Block Compression, a.k.a. DXT/BCn): Ubiquitous on desktop; supported by some mobile

GPUs.

2. Format Selection Based on Content and Support: Choose formats based on the type of texture
and what the device reports:

◦ For high detail (normals, roughness): prefer ASTC 4x4 or 6x6 when supported; on desktop,
BC5/BC7 are common alternatives.

◦ For albedo/basecolor: ASTC 6x6–8x8 works well when available; on desktop, BC1/BC7 are
typical.

◦ For single-channel data: consider R8 or compressed single-channel alternatives when

available.

This guidance is not mobile-only: block compression reduces memory footprint

and bandwidth on all platforms. The Mobile chapter highlights it because
 bandwidth and power are tighter constraints on phones/tablets. On desktop, the
same benefits apply; the primary difference is which formats are commonly
available (e.g., BC on desktop, ASTC/ETC2 on many mobile devices).

Here’s how to check for and use compressed formats in Vulkan:

bool is_format_supported(vk::PhysicalDevice physical_device, vk::Format format,

vk::ImageTiling tiling,
vk::FormatFeatureFlags features) {

484
 vk::FormatProperties props = physical_device.getFormatProperties(format);

if (tiling == vk::ImageTiling::eLinear) {
return ([Link] & features) == features;
} else if (tiling == vk::ImageTiling::eOptimal) {
return ([Link] & features) == features;
}

return false;
}

vk::Format find_supported_format(vk::PhysicalDevice physical_device,

const std::vector<vk::Format>& candidates,
vk::ImageTiling tiling,
vk::FormatFeatureFlags features) {
for (vk::Format format : candidates) {
if (is_format_supported(physical_device, format, tiling, features)) {
return format;
}
}

throw std::runtime_error("Failed to find supported format");

}

1.2. Memory Optimizations

Memory is a precious resource on all platforms. It tends to be more performance‑critical on mobile
due to tighter bandwidth, power, and thermal budgets. Here are some key optimizations:

1.2.1. Minimize Memory Allocations

1. Pool Allocations: Use memory pools to reduce the overhead of frequent allocations and
deallocations.

2. Suballocate from Larger Blocks: Instead of creating many small Vulkan memory allocations,
allocate larger blocks and suballocate from them:

class VulkanMemoryPool {
public:
VulkanMemoryPool(vk::Device device, vk::PhysicalDevice physical_device,
vk::DeviceSize block_size, uint32_t memory_type_index)
: device(device), block_size(block_size), memory_type_index(memory_type_index)
{
allocate_new_block();
}

~VulkanMemoryPool() {
for (auto& block : memory_blocks) {
[Link]([Link]);

485
 }
}

struct Allocation {
vk::DeviceMemory memory;
vk::DeviceSize offset;
vk::DeviceSize size;
};

Allocation allocate(vk::DeviceSize size, vk::DeviceSize alignment) {

// Find a block with enough space
for (auto& block : memory_blocks) {
vk::DeviceSize aligned_offset = align(block.next_offset, alignment);
if (aligned_offset + size <= block_size) {
Allocation alloc;
[Link] = [Link];
[Link] = aligned_offset;
[Link] = size;

block.next_offset = aligned_offset + size;

return alloc;
}
}

// No block has enough space, allocate a new one

allocate_new_block();
return allocate(size, alignment); // Try again with the new block
}

private:
struct MemoryBlock {
vk::DeviceMemory memory;
vk::DeviceSize next_offset = 0;
};

void allocate_new_block() {
vk::MemoryAllocateInfo alloc_info;
alloc_info.setAllocationSize(block_size);
alloc_info.setMemoryTypeIndex(memory_type_index);

MemoryBlock block;
[Link] = [Link](alloc_info);
block.next_offset = 0;

memory_blocks.push_back(block);
}

vk::DeviceSize align(vk::DeviceSize offset, vk::DeviceSize alignment) {

return (offset + alignment - 1) & ~(alignment - 1);
}

486
 vk::Device device;
vk::DeviceSize block_size;
uint32_t memory_type_index;
std::vector<MemoryBlock> memory_blocks;
};

1.2.2. Reduce Bandwidth Usage

1. Minimize State Changes: Group draw calls by material to reduce state changes.

2. Use Smaller Data Types: Use 16-bit indices and half-precision floats where appropriate.

3. Optimize Vertex Formats: Use packed vertex formats to reduce memory bandwidth:

// Traditional vertex format (48 bytes per vertex)

struct Vertex {
glm::vec3 position; // 12 bytes
glm::vec3 normal; // 12 bytes
glm::vec2 texCoord; // 8 bytes
glm::vec4 color; // 16 bytes
};

// Optimized vertex format (16 bytes per vertex)

struct OptimizedVertex {
// Position: 3 components, 16-bit float each
uint16_t position[3]; // 6 bytes

// Normal: 2 components (can reconstruct Z), 8-bit signed normalized

int8_t normal[2]; // 2 bytes

// TexCoord: 2 components, 16-bit float each

uint16_t texCoord[2]; // 4 bytes

// Color: 4 components, 8-bit unsigned normalized

uint8_t color[4]; // 4 bytes
};

If you are targeting tile-based GPUs (TBR), bandwidth can be heavily impacted by
attachment load/store behavior and tile flushes. See Rendering Approaches —
 sections “Attachment Load/Store Operations on Tilers” and “Pipelining on Tilers:
Subpass Dependencies and BY_REGION” for concrete guidance.

1.3. Draw Call Optimizations

Mobile GPUs are particularly sensitive to draw call overhead:

1. Instancing: Use instancing to reduce draw calls for repeated objects.

2. Batching: Combine multiple objects into a single mesh where possible.

487
3. Level of Detail (LOD): Implement LOD systems to reduce geometry complexity for distant
objects.

On tile-based GPUs, reducing CPU overhead is important, but keeping work and
data on-chip via proper pipelining and subpasses often yields larger gains. See
 Rendering Approaches — “Pipelining on Tilers: Subpass Dependencies and
BY_REGION” for barrier/subpass patterns, and “Attachment Load/Store Operations
on Tilers” for loadOp/storeOp guidance that avoids external memory traffic.

1.4. Vendor-Specific Optimizations

Different mobile GPU vendors have specific architectures that may benefit from targeted
optimizations.

1.4.1. Vendor-Specific GPU Optimizations

Different mobile GPU vendors have specific architectures that benefit from targeted optimizations:

• Memory Management: Many mobile SoCs have unified memory architecture:

◦ Use VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT memory

when possible

◦ Take advantage of fast CPU-GPU memory transfers in unified memory architectures

• Texture Compression: Different devices support different texture compression formats:

// Check for texture compression format support

bool supports_texture_format(vk::PhysicalDevice physical_device, vk::Format format) {
vk::FormatProperties props = physical_device.getFormatProperties(format);
return ([Link] & vk::FormatFeatureFlagBits::eSampledImage);
}

// Get optimal texture format based on device capabilities

vk::Format get_optimal_texture_format(vk::PhysicalDevice physical_device) {
vk::PhysicalDeviceProperties props = physical_device.getProperties();
vk::PhysicalDeviceFeatures features = physical_device.getFeatures();

// Check for ASTC support (widely supported on modern mobile GPUs)

// Most games are written with knowledge of what the assets were compressed with
so it's standard practice to only ensure the required format is supported.
if (features.textureCompressionASTC_LDR) {
return vk::Format::eAstc8x8SrgbBlock;
}
}

• Performance Monitoring: Most vendors provide performance monitoring tools that can help
identify bottlenecks specific to their hardware.

488
1.5. Best Practices for Mobile Performance
1. Profile on Target Devices: Performance characteristics vary widely across mobile devices. Test
on a range of hardware from different manufacturers and with different GPU architectures.

2. Monitor Temperature: Mobile devices throttle performance when they get hot. Design your
engine to adapt to thermal throttling.

3. Balance Quality and Performance: Provide graphics settings that allow users to balance
quality and performance based on their device capabilities.

4. Implement Adaptive Resolution: Dynamically adjust rendering resolution based on

performance metrics.

In the next section, we’ll explore different rendering approaches for mobile GPUs, focusing on the
differences between Tile-Based Rendering (TBR) and Immediate Mode Rendering (IMR).

Previous: Platform Considerations | Next: Rendering Approaches :pp: ++

Mobile Development: Rendering

Approaches
1. Rendering Approaches for Mobile GPUs
Mobile GPUs typically use different rendering architectures compared to desktop GPUs.
Understanding these differences is crucial for optimizing your Vulkan application for mobile
platforms. In this section, we’ll explore the two main rendering approaches: Tile-Based Rendering
(TBR) and Immediate Mode Rendering (IMR).

1.1. Tile-Based Rendering (TBR)

Most modern mobile GPUs use a tile-based rendering architecture, also known as Tile-Based
Deferred Rendering (TBDR) in some implementations.

1.1.1. How TBR Works

1. Tiling Phase: The screen is divided into small tiles (typically 16x16 or 32x32 pixels).

2. Binning Phase: The GPU determines which primitives (triangles) affect each tile.

3. Rendering Phase: For each tile:

1. Load the primitives affecting that tile into on-chip memory.

2. Render the primitives to the tile.

3. Write the completed tile back to main memory.

489
1.1.2. Advantages of TBR

1. Reduced Memory Bandwidth: Since rendering happens in on-chip memory, there’s less traffic
to main memory.

2. Power Efficiency: Lower memory bandwidth means lower power consumption, which is
crucial for battery-powered devices.

3. Hidden Surface Removal: Many TBR GPUs perform early depth testing during the binning
phase, reducing overdraw.

1.1.3. Optimizing for TBR

To get the best performance from TBR GPUs, consider these optimizations:

• Transient Attachments: Use transient attachments for render targets that are only used within
a render pass:

vk::AttachmentDescription depth_attachment{};
depth_attachment.setFormat(depth_format);
depth_attachment.setSamples(vk::SampleCountFlagBits::e1);
depth_attachment.setLoadOp(vk::AttachmentLoadOp::eClear);
depth_attachment.setStoreOp(vk::AttachmentStoreOp::eDontCare); // Don't store the
result
depth_attachment.setStencilLoadOp(vk::AttachmentLoadOp::eDontCare);
depth_attachment.setStencilStoreOp(vk::AttachmentStoreOp::eDontCare);
depth_attachment.setInitialLayout(vk::ImageLayout::eUndefined);
depth_attachment.setFinalLayout(vk::ImageLayout::eDepthStencilAttachmentOptimal);

// When creating the image, mark the attachment as transient

vk::ImageCreateInfo image_info{};
image_info.setImageType(vk::ImageType::e2D);
image_info.setExtent(vk::Extent3D(width, height, 1));
image_info.setMipLevels(1);
image_info.setArrayLayers(1);
image_info.setFormat(depth_format);
image_info.setTiling(vk::ImageTiling::eOptimal);
image_info.setInitialLayout(vk::ImageLayout::eUndefined);
image_info.setUsage(vk::ImageUsageFlagBits::eDepthStencilAttachment |
vk::ImageUsageFlagBits::eTransientAttachment);
image_info.setSamples(vk::SampleCountFlagBits::e1);
// Prefer lazily allocated memory for transient attachments when supported
// Choose memory with vk::MemoryPropertyFlagBits::eLazilyAllocated

• Render Pass Structure: Design your render passes to take advantage of tile-based rendering:

◦ Use subpasses to keep rendering operations within the tile memory.

◦ Use the right load/store operations to minimize memory traffic.

// Create a render pass with multiple subpasses

490
 vk::SubpassDescription subpass1{};
[Link](vk::PipelineBindPoint::eGraphics);
[Link](color_attachment_refs);
[Link](&depth_attachment_ref);

vk::SubpassDescription subpass2{};
[Link](vk::PipelineBindPoint::eGraphics);
[Link](input_attachment_refs); // Use output from subpass1 as
input
[Link](final_color_attachment_refs);

// Create a dependency to ensure proper ordering

vk::SubpassDependency dependency{};
[Link](0);
[Link](1);
[Link](vk::PipelineStageFlagBits::eColorAttachmentOutput);
[Link](vk::PipelineStageFlagBits::eFragmentShader);
[Link](vk::AccessFlagBits::eColorAttachmentWrite);
[Link](vk::AccessFlagBits::eInputAttachmentRead);

// Create the render pass

vk::RenderPassCreateInfo render_pass_info{};
render_pass_info.setAttachments(attachments);
render_pass_info.setSubpasses({subpass1, subpass2});
render_pass_info.setDependencies(dependency);

vk::RenderPass render_pass = [Link](render_pass_info);

1.1.4. Best Practices for TBR

• Avoid External Framebuffer Reads: Avoid reading from images that require the tile to be
flushed to external memory and reloaded; this is expensive on TBR.

◦ Local, same-pixel reads from on-chip/tile memory are fine and encouraged on tile-based
GPUs.

◦ In Vulkan, use input attachments within subpasses or the

VK_KHR_dynamic_rendering_local_read capability to perform tile-local reads without leaving
tile memory. This is often referred to as pixel-local storage (PLS) on tile-based architectures.

• Optimize for Tile Size: Consider the tile size when designing your rendering algorithm. For
example, if you know the tile size is 16x16, you might organize your data or algorithms to work
efficiently with that size.

[Link]. Attachment Load/Store Operations on Tilers

On tile-based GPUs, correctly using loadOp and storeOp is one of the highest-impact optimizations:

• Clear attachments with loadOp = CLEAR and initialLayout = UNDEFINED when you don’t need
previous contents. This avoids an external memory read for the tile.

• Use storeOp = DONT_CARE for attachments whose results are not needed after the render pass

491
 (e.g., transient depth or intermediate color targets). This can prevent flushing the tile back to
main memory.

• For the swapchain image (or any image you will sample/transfer from later), use storeOp =
STORE and set finalLayout appropriately (e.g., PRESENT_SRC_KHR for the swapchain).

• For MSAA, resolve within the same render pass so the hardware can resolve from tile memory
and only store the resolved image to external memory.

// Color attachment that we clear and present

vk::AttachmentDescription color_attachment{};
color_attachment.setFormat(swapchain_format);
color_attachment.setSamples(vk::SampleCountFlagBits::e1);
color_attachment.setLoadOp(vk::AttachmentLoadOp::eClear);
color_attachment.setStoreOp(vk::AttachmentStoreOp::eStore); // we need to present
color_attachment.setStencilLoadOp(vk::AttachmentLoadOp::eDontCare);
color_attachment.setStencilStoreOp(vk::AttachmentStoreOp::eDontCare);
color_attachment.setInitialLayout(vk::ImageLayout::eUndefined); // no need to load
previous contents
color_attachment.setFinalLayout(vk::ImageLayout::ePresentSrcKHR);

// Depth attachment used only within the pass

vk::AttachmentDescription depth_attachment{};
depth_attachment.setFormat(depth_format);
depth_attachment.setSamples(vk::SampleCountFlagBits::e1);
depth_attachment.setLoadOp(vk::AttachmentLoadOp::eClear);
depth_attachment.setStoreOp(vk::AttachmentStoreOp::eDontCare); // don't flush depth to
memory
depth_attachment.setStencilLoadOp(vk::AttachmentLoadOp::eDontCare);
depth_attachment.setStencilStoreOp(vk::AttachmentStoreOp::eDontCare);
depth_attachment.setInitialLayout(vk::ImageLayout::eUndefined);
depth_attachment.setFinalLayout(vk::ImageLayout::eDepthStencilAttachmentOptimal);

If you use dynamic rendering, the same rules apply via

 vk::RenderingAttachmentInfo loadOp/storeOp fields. See Vulkan Guide for
background: Render Passes and Subpasses, Tile-based GPUs.

[Link]. Pipelining on Tilers: Subpass Dependencies and BY_REGION

Tile-based GPUs benefit from fine-grained synchronization that keeps work and data on-chip:

• Prefer subpasses with input attachments to keep producer/consumer within the same render
pass, enabling tile-local reads.

• Use vk::DependencyFlagBits::eByRegion to scope hazards to the pixel regions actually

written/read, avoiding unnecessary tile flushes.

• Avoid over-broad barriers (e.g., ALL_COMMANDS, MEMORY_READ/WRITE) that serialize the

pipeline and may force external memory traffic. Use precise stage/access masks.

Example: dependency from a color-writing subpass to a subpass that reads that color as an input

492
attachment.

vk::SubpassDependency dep{};
[Link](0);
[Link](1);
[Link](vk::PipelineStageFlagBits::eColorAttachmentOutput);
[Link](vk::PipelineStageFlagBits::eFragmentShader);
[Link](vk::AccessFlagBits::eColorAttachmentWrite);
[Link](vk::AccessFlagBits::eInputAttachmentRead);
[Link](vk::DependencyFlagBits::eByRegion);

Example: external dependency to the first subpass of a render pass, allowing pipelining with prior
pass while limiting scope by region.

vk::SubpassDependency externalDep{};
[Link](VK_SUBPASS_EXTERNAL);
[Link](0);
[Link](vk::PipelineStageFlagBits::eColorAttachmentOutput);
[Link](vk::PipelineStageFlagBits::eEarlyFragmentTests |
vk::PipelineStageFlagBits::eColorAttachmentOutput);
[Link](vk::AccessFlagBits::eColorAttachmentWrite);
[Link](vk::AccessFlagBits::eDepthStencilAttachmentWrite |
vk::AccessFlagBits::eColorAttachmentWrite);
[Link](vk::DependencyFlagBits::eByRegion);

With Synchronization2 (vkCmdPipelineBarrier2 and friends) avoid

ALL_COMMANDS and prefer the minimal set of stages/access that capture your
 hazard. Use render pass/subpass structure when possible—it’s the most tiler-
friendly way to express pipelining.

For further guidance, see the Vulkan Guide topics on Tile-based GPUs, Render Passes, and
Synchronization.

[Link]. Memory Management

To improve the efficiency of memory allocation in TBR architectures:

• Select Optimal Memory Types: Choose the best matching memory type (with the appropriate
VkMemoryPropertyFlags) when using vkAllocateMemory.

• Batch Allocations: For each type of resource (e.g., index buffer, vertex buffer, and uniform
buffer), allocate large chunks of memory with a specific size in one go when possible.

• Reuse Memory Resources: Let multiple passes take turns using the allocated memory through
time slicing.

• Use Cached Memory When Appropriate: Consider using

VK_MEMORY_PROPERTY_HOST_CACHED_BIT and manually flushing memory when it may be
accessed by the CPU. This is often more efficient than

493
 VK_MEMORY_PROPERTY_HOST_COHERENT_BIT because the driver can refresh a large block of
memory at once.

• Minimize Allocation Calls: Avoid frequent calls to vkAllocateMemory. The number of memory
allocations is limited by maxMemoryAllocationCount.

[Link]. Shader Optimizations

Optimizing shaders for TBR architectures can significantly improve performance:

• Vectorized Memory Access: Access memory in a vectorized manner to reduce access cycles
and bandwidth. For example:

// Recommended: Vectorized access

struct TileStructSample {
vec4 data;
};

void main() {
uint idx = 0u;
TileStructSample ts[3];
while (idx < 3u) {
ts[int(idx)].data = a;
idx++;
}
}

// Not recommended: Non-vectorized access

struct TileStructSample {
float data1;
float data2;
float data3;
float data4;
};

void main() {
uint idx = 0u;
TileStructSample ts[3];
while (idx < 3u) {
ts[int(idx)].data1 = a;
ts[int(idx)].data2 = b;
ts[int(idx)].data3 = c;
ts[int(idx)].data4 = d;
idx++;
}
}

• Optimize Uniform Buffers: Consider using push constants or macro constants instead of
uniform buffers for small data. Avoid dynamic indexing when possible.

• Minimize Branching: Reduce complex branch structures, branch nesting, and loop structures

494
 as they can harm parallelism.

• Use Half-Precision: When appropriate, use half-precision floats to reduce bandwidth and
power consumption. In SPIR-V, use relaxed-precision decoration on variables or results.

[Link]. Depth Testing Optimizations

Proper depth testing is crucial for TBR performance:

• Enable Depth Testing and Writing: This allows the GPU to cull hidden primitives and reduce
overdraw.

• Avoid Operations That Disable Early-Z: The following operations can prevent effective early
depth testing:

◦ Using the discard instruction in fragment shaders

◦ Writing to gl_FragDepth (GLSL) SV_Depth (slang) explicitly

◦ Using storage images or storage buffers

◦ Using gl_SampleMask (GLSL explicit way to turn on/off specific pixels)

◦ Enabling both depth bounds and depth write

◦ Enabling both blending and depth write

• Consistent Compare Operations: When using compareOp, try to keep the values consistent for
each draw in the render pass.

• Clear Attachments Properly: Attachments should be cleared at the beginning of the render
pass, or when no valid compareOp value is assigned to previous draw calls.

1.2. Immediate Mode Rendering (IMR)

Traditional desktop GPUs and some older mobile GPUs use an immediate mode rendering
architecture.

1.2.1. How IMR Works

1. Vertex Processing: Process vertices and assemble primitives.

2. Rasterization: Convert primitives to fragments.

3. Fragment Processing: Process each fragment and write the result directly to the framebuffer in
main memory.

1.2.2. Advantages of IMR

1. Simplicity: The rendering model is more straightforward and matches the traditional graphics
pipeline.

2. Flexibility: Some algorithms that require reading from the framebuffer are easier to
implement.

495
1.2.3. Optimizing for IMR

If your target device uses IMR, consider these optimizations:

1. Front-to-Back Rendering: Render opaque objects from front to back to minimize overdraw.

2. Early-Z: Use depth testing to reject fragments early in the pipeline.

3. Occlusion Culling: Implement occlusion culling to avoid rendering objects that won’t be visible.

1.3. Detecting Rendering Architecture

Vulkan doesn’t provide a direct way to determine if a GPU uses TBR or IMR. However, you can
make educated guesses based on the device vendor and model:

bool is_likely_tbr_gpu(vk::PhysicalDevice physical_device) {

vk::PhysicalDeviceProperties props = physical_device.getProperties();

// Most mobile GPUs from these vendors use TBR

if ([Link] == 0x5143) { // Qualcomm
return true;
}
if ([Link] == 0x1010) { // PowerVR (Imagination Technologies)
return true;
}
if ([Link] == 0x13B5) { // ARM Mali
return true;
}
if ([Link] == 0x19E5) { // Huawei
return true;
}

// Apple GPUs are also TBR

if ([Link] == 0x106B) { // Apple
return true;
}

// For other vendors, you might need to maintain a list of known TBR GPUs
// or just assume desktop GPUs are IMR and mobile GPUs are TBR

return false;
}

1.4. Adapting to Both Architectures

The best approach is to design your engine to work well on both TBR and IMR architectures:

• Detect the Architecture: Use heuristics to detect the likely architecture.

• Conditional Optimizations: Apply different optimizations based on the detected architecture:

496
 void configure_rendering_pipeline(vk::PhysicalDevice physical_device) {
bool is_tbr = is_likely_tbr_gpu(physical_device);

if (is_tbr) {
// TBR optimizations
use_transient_attachments = true;
prioritize_subpass_dependencies = true;
avoid_framebuffer_reads = true;
} else {
// IMR optimizations
use_front_to_back_sorting = true;
prioritize_early_z = true;
implement_occlusion_culling = true;
}
}

• Fallback Strategy: If you can’t determine the architecture, optimize for TBR, as those
optimizations generally don’t harm IMR performance significantly.

1.5. Best Practices for Both Architectures

Regardless of the rendering architecture, these practices will help optimize performance:

1. Minimize State Changes: Group draw calls by material to reduce state changes.

2. Batch Similar Objects: Use instancing or batching to reduce draw call overhead.

3. Use Appropriate Synchronization: Use the minimum synchronization required to ensure

correct rendering.

4. Profile on Target Devices: Always test your optimizations on actual target devices.

In the next section, we’ll explore Vulkan extensions that can help you optimize performance on
mobile devices, particularly those that leverage the tile-based architecture.

Previous: Performance Optimizations | Next: Vulkan Extensions for Mobile :pp: ++

Mobile Development: Vulkan

Extensions
1. Vulkan Extensions for Mobile
Vulkan’s extensibility is one of its greatest strengths, allowing hardware vendors to expose
specialized features that can significantly improve performance. For mobile development, several
extensions are particularly valuable as they can help optimize for the unique characteristics of
mobile GPUs. In this section, we’ll explore key Vulkan extensions that can enhance performance on

497
mobile devices.

1.1. VK_KHR_dynamic_rendering
Dynamic rendering is a game-changing extension that simplifies the Vulkan rendering workflow by
eliminating the need for explicit render pass and framebuffer objects.

1.1.1. Overview

The VK_KHR_dynamic_rendering extension (now part of Vulkan 1.3 core) allows you to begin and end
rendering operations directly within a command buffer, without creating render pass and
framebuffer objects. This benefits a wide range of platforms (desktop and mobile) because it:

1. Simplifies Code: Reduces the complexity of managing render passes and framebuffers.

2. Enables More Flexible Rendering: Makes it easier to implement techniques that don’t fit well
into the traditional render pass model.

3. Potentially Lowers API Overhead: Fewer objects to create and manage can simplify setup; any
CPU savings are usually small and workload-dependent.

1.1.2. Implementation (Step-by-step)

Let’s break the setup into a few small, focused steps.

[Link]. Enable the extension and load entry points

We first enable the device extension and, if you’re not on Vulkan 1.3 core, load the function
pointers.

// Enable the extension when creating the device

std::vector<const char*> device_extensions = {
VK_KHR_SWAPCHAIN_EXTENSION_NAME,
VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME
};

// Get function pointers (if not using Vulkan 1.3)

PFN_vkCmdBeginRenderingKHR vkCmdBeginRenderingKHR =
reinterpret_cast<PFN_vkCmdBeginRenderingKHR>(
vkGetDeviceProcAddr(device, "vkCmdBeginRenderingKHR"));
PFN_vkCmdEndRenderingKHR vkCmdEndRenderingKHR =
reinterpret_cast<PFN_vkCmdEndRenderingKHR>(
vkGetDeviceProcAddr(device, "vkCmdEndRenderingKHR"));

This prepares your device to use dynamic rendering and gives access to the commands needed to
begin/end a rendering scope without a traditional render pass.

498
[Link]. Describe attachments for this rendering scope

We define the color and depth attachments and package them into a VkRenderingInfoKHR. Think of
this as an inline, one-off description of what would normally be baked into render
pass/framebuffer objects.

VkRenderingAttachmentInfoKHR color_attachment{};
color_attachment.sType = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO_KHR;
color_attachment.imageView = color_image_view;
color_attachment.imageLayout = VK_IMAGE_LAYOUT_COLOR_ATTACHMENT_OPTIMAL;
color_attachment.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
color_attachment.storeOp = VK_ATTACHMENT_STORE_OP_STORE;
color_attachment.clearValue = clear_value;

VkRenderingAttachmentInfoKHR depth_attachment{};
depth_attachment.sType = VK_STRUCTURE_TYPE_RENDERING_ATTACHMENT_INFO_KHR;
depth_attachment.imageView = depth_image_view;
depth_attachment.imageLayout = VK_IMAGE_LAYOUT_DEPTH_ATTACHMENT_OPTIMAL;
depth_attachment.loadOp = VK_ATTACHMENT_LOAD_OP_CLEAR;
depth_attachment.storeOp = VK_ATTACHMENT_STORE_OP_DONT_CARE;
depth_attachment.clearValue = depth_clear_value;

VkRenderingInfoKHR rendering_info{};
rendering_info.sType = VK_STRUCTURE_TYPE_RENDERING_INFO_KHR;
rendering_info.renderArea = render_area;
rendering_info.layerCount = 1;
rendering_info.colorAttachmentCount = 1;
rendering_info.pColorAttachments = &color_attachment;
rendering_info.pDepthAttachment = &depth_attachment;

Each frame (or subpass-equivalent), you can tweak these descriptors directly (e.g., swapchain views
after resize), avoiding pipeline-wide re-creation.

[Link]. Begin rendering, draw, end rendering

With the attachments described, we open the rendering scope, record draws, then close the scope.

vkCmdBeginRenderingKHR(command_buffer, &rendering_info);

// Record drawing commands

vkCmdBindPipeline(command_buffer, VK_PIPELINE_BIND_POINT_GRAPHICS, pipeline);
vkCmdDraw(command_buffer, vertex_count, 1, 0, 0);

// End rendering
vkCmdEndRenderingKHR(command_buffer);

The begin/end pair replaces vkCmdBeginRenderPass/vkCmdEndRenderPass while providing more

flexibility for modern rendering flows.

499
[Link]. C++ bindings ([Link]) variant

If you’re using [Link] (vk::), the structure population is more ergonomic but follows the same
steps.

// Using [Link]
vk::RenderingAttachmentInfoKHR color_attachment;
color_attachment.setImageView(color_image_view);
color_attachment.setImageLayout(vk::ImageLayout::eColorAttachmentOptimal);
color_attachment.setLoadOp(vk::AttachmentLoadOp::eClear);
color_attachment.setStoreOp(vk::AttachmentStoreOp::eStore);
color_attachment.setClearValue(clear_value);

vk::RenderingAttachmentInfoKHR depth_attachment;
depth_attachment.setImageView(depth_image_view);
depth_attachment.setImageLayout(vk::ImageLayout::eDepthAttachmentOptimal);
depth_attachment.setLoadOp(vk::AttachmentLoadOp::eClear);
depth_attachment.setStoreOp(vk::AttachmentStoreOp::eDontCare);
depth_attachment.setClearValue(depth_clear_value);

vk::RenderingInfoKHR rendering_info;
rendering_info.setRenderArea(render_area);
rendering_info.setLayerCount(1);
rendering_info.setColorAttachments(color_attachment);
rendering_info.setPDepthAttachment(&depth_attachment);

Once the description is assembled, begin the rendering scope, submit draws, and end the scope.

command_buffer.beginRenderingKHR(rendering_info);

// Record drawing commands

command_buffer.bindPipeline(vk::PipelineBindPoint::eGraphics, pipeline);
command_buffer.draw(vertex_count, 1, 0, 0);

// End rendering
command_buffer.endRenderingKHR();

1.2. VK_KHR_dynamic_rendering_local_read
The VK_KHR_dynamic_rendering_local_read extension is particularly valuable for tile-based renderers
as it allows shaders to read from attachments without forcing a tile to main memory and back.

1.2.1. Overview

This extension enhances dynamic rendering by allowing fragment shaders to read from color and
depth/stencil attachments within the same rendering scope. On tile-based renderers, this means the
reads can happen directly from tile memory, avoiding expensive round trips to main memory.

500
Key benefits include:

1. Reduced Memory Bandwidth: Reads happen from on-chip memory rather than main memory,
reducing bandwidth usage for bandwidth-intensive operations.

2. Improved Performance: Particularly for algorithms that need to read from previously written
attachments.

3. Power Efficiency: Lower memory bandwidth means lower power consumption.

1.2.2. How It Reduces Memory Bandwidth

The VK_KHR_dynamic_rendering_local_read extension is particularly effective at reducing memory

bandwidth because:

1. Eliminates Tile Flush Operations: Without this extension, when a shader needs to read from a
previously written attachment, the GPU must flush the entire tile to main memory and then
read it back. This extension allows the shader to read directly from the tile memory, eliminating
these costly flush operations.

2. Supports Per-Pixel Local Reads: It enables fragment shaders to read the value written at the
same pixel from attachments within the current rendering scope/tile. This suits per-pixel
operations (e.g., tone mapping or reading depth/previous color).

3. Bandwidth Reduction Measurements: In real-world applications, this extension has been

shown to reduce memory bandwidth for workloads that benefit from per-pixel local reads. The
benefit is workload- and GPU-dependent.

4. Practical Example: Consider a deferred rendering pipeline that needs to read G-buffer data at
the same pixel for lighting. Without this extension, the G-buffer would need to be written to
main memory and then read back for the lighting pass. With this extension, the lighting pass
can read directly from the G-buffer in tile memory, saving bandwidth.

1.2.3. Implementation

To use this extension:

// Enable the extension when creating the device

std::vector<const char*> device_extensions = {
VK_KHR_SWAPCHAIN_EXTENSION_NAME,
VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME,
VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME
};

// Create a pipeline that reads from attachments

vk::PipelineRenderingCreateInfoKHR rendering_create_info;
rendering_create_info.setColorAttachmentCount(1);
rendering_create_info.setColorAttachmentFormats(color_format);
rendering_create_info.setDepthAttachmentFormat(depth_format);

// Set up the attachment local read info

vk::AttachmentSampleCountInfoAMD sample_count_info;

501
 sample_count_info.setColorAttachmentSamples(vk::SampleCountFlagBits::e1);
sample_count_info.setDepthStencilAttachmentSamples(vk::SampleCountFlagBits::e1);

vk::RenderingAttachmentLocationInfoKHR location_info;
location_info.setColorAttachmentLocations(0); // Location 0 for the color attachment

vk::RenderingInputAttachmentIndexInfoKHR input_index_info;
input_index_info.setColorInputAttachmentIndices(0); // Index 0 for the color
attachment

// Create the graphics pipeline

vk::GraphicsPipelineCreateInfo pipeline_info;
pipeline_info.setPNext(&rendering_create_info);
// ... set other pipeline creation parameters

// In your fragment shader, you can now read from the attachment
// using subpassLoad() or texture sampling with the appropriate extension
// Fragment shader example (GLSL):
// #extension GL_EXT_shader_tile_image : require
// layout(location = 0) out vec4 outColor;
// layout(input_attachment_index = 0, set = 0, binding = 0) uniform subpassInput
inputColor;
// void main() {
// vec4 color = subpassLoad(inputColor);
// outColor = color * 2.0; // Double the brightness
// }

1.3. VK_EXT_shader_tile_image
The VK_EXT_shader_tile_image extension provides direct access to tile memory in shaders, which can
significantly improve performance on tile-based renderers.

1.3.1. Overview

This extension allows shaders to:

1. Access Tile Memory Directly: Read and write to the current tile’s memory without going
through main memory.

2. Perform Tile-Local Operations: Execute operations that stay entirely within the tile memory.

3. Optimize Bandwidth-Intensive Algorithms: Particularly beneficial for post-processing effects.

4. Reduce Memory Bandwidth: Helps lower memory bandwidth by keeping data in tile-local
memory during multi-pass workloads.

1.3.2. How It Reduces Memory Bandwidth

The VK_EXT_shader_tile_image extension is particularly effective at reducing memory bandwidth for

these reasons:

502
1. Tile-Based Architecture Optimization: Mobile GPUs typically use tile-based rendering, where
the screen is divided into small tiles that are processed independently. This extension takes full
advantage of this architecture by allowing shaders to work directly with the tile data in fast on-
chip memory.

2. Eliminates Intermediate Memory Transfers: Without this extension, multi-pass rendering

requires writing results to main memory after each pass and reading them back for the next
pass. With VK_EXT_shader_tile_image, these intermediate results can stay in tile memory,
eliminating these costly transfers.

3. Bandwidth Savings Measurements: Testing on various mobile GPUs has shown meaningful
bandwidth reductions for complex multi-pass pipelines; actual gains are workload- and GPU-
dependent.

4. Practical Applications:

◦ Image Processing Filters: Applying multiple filters (blur, sharpen, color correction) can be
done without leaving tile memory.

◦ Deferred Rendering: G-buffer data can be kept in tile memory for the lighting pass.

◦ Shadow Mapping: Shadow calculations can be performed more efficiently by keeping depth
information in tile memory.

5. Power Efficiency: The reduction in memory bandwidth directly translates to lower power
consumption, which is critical for mobile devices. Tests have shown up to 20% power savings
for graphics-intensive applications.

1.3.3. Implementation

To use this extension:

// Enable the extension when creating the device

std::vector<const char*> device_extensions = {
VK_KHR_SWAPCHAIN_EXTENSION_NAME,
VK_EXT_SHADER_TILE_IMAGE_EXTENSION_NAME
};

// When creating your shader module, make sure your shader uses the extension
// GLSL example:
// #extension GL_EXT_shader_tile_image : require
//
// layout(tile_image, set = 0, binding = 0) uniform tileImageColor { vec4 color; }
tileColor;
//
// void main() {
// // Read from tile memory
// vec4 current_color = [Link];
//
// // Process the color
// vec4 new_color = process(current_color);
//
// // Write back to tile memory

503
 // [Link] = new_color;
// }

1.4. Combining Extensions for Maximum Performance

For the best mobile performance, consider using these extensions together:

// Enable all relevant extensions

std::vector<const char*> device_extensions = {
VK_KHR_SWAPCHAIN_EXTENSION_NAME,
VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME,
VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME,
VK_EXT_SHADER_TILE_IMAGE_EXTENSION_NAME
};

// Check which extensions are supported

auto available_extensions = physical_device.enumerateDeviceExtensionProperties();
std::vector<const char*> supported_extensions;

for (const auto& requested_ext : device_extensions) {

for (const auto& available_ext : available_extensions) {
if (strcmp(requested_ext, available_ext.extensionName) == 0) {
supported_extensions.push_back(requested_ext);
break;
}
}
}

// Create device with supported extensions

vk::DeviceCreateInfo device_create_info;
device_create_info.setPEnabledExtensionNames(supported_extensions);
// ... set other device creation parameters
vk::Device device = physical_device.createDevice(device_create_info);

// Now you can use the supported extensions in your rendering code
// ...

1.5. Device Extension Support

Different mobile vendors and devices vary in which Vulkan extensions they expose. Understanding
per-device support helps you pick features safely at runtime.

1.5.1. Device Extension Support Details

Different mobile GPU vendors have varying levels of support for Vulkan extensions:

• Dynamic Rendering Support: Many mobile GPUs have optimized implementations of

VK_KHR_dynamic_rendering. This can lead to significant performance improvements compared to

504
 traditional render passes, especially on tile-based renderers.

• Tile-Based Optimizations: On tile-based GPUs (e.g., Mali, PowerVR), VK_EXT_shader_tile_image

and VK_KHR_dynamic_rendering_local_read are effective because they keep reads and writes in
tile memory. See the extension sections above for details; benefits are workload- and GPU-
dependent.

• Checking for Extension Support (EXT/KHR) on the current device:

// Common vendor IDs (used here only for labeling/logging output)

const uint32_t VENDOR_ID_QUALCOMM = 0x5143; // Adreno
const uint32_t VENDOR_ID_ARM = 0x13B5; // Mali
const uint32_t VENDOR_ID_IMAGINATION = 0x1010; // PowerVR
const uint32_t VENDOR_ID_HUAWEI = 0x19E5; // Kirin
const uint32_t VENDOR_ID_APPLE = 0x106B; // Apple

bool log_device_extension_support(vk::PhysicalDevice physical_device) {

vk::PhysicalDeviceProperties props = physical_device.getProperties();
std::string vendor_name;

// Identify vendor for display purposes only

switch ([Link]) {
case VENDOR_ID_QUALCOMM: vendor_name = "Qualcomm"; break;
case VENDOR_ID_ARM: vendor_name = "ARM Mali"; break;
case VENDOR_ID_IMAGINATION: vendor_name = "PowerVR"; break;
case VENDOR_ID_HUAWEI: vendor_name = "Huawei"; break;
case VENDOR_ID_APPLE: vendor_name = "Apple"; break;
default: vendor_name = "Unknown"; break;
}

// Check for widely useful EXT/KHR extensions on this device

auto available_extensions = physical_device.enumerateDeviceExtensionProperties();
bool has_dynamic_rendering = false;
bool has_dynamic_rendering_local_read = false;
bool has_shader_tile_image = false;

for (const auto& ext : available_extensions) {

std::string ext_name = [Link];
if (ext_name == VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME) {
has_dynamic_rendering = true;
} else if (ext_name == VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME) {
has_dynamic_rendering_local_read = true;
} else if (ext_name == VK_EXT_SHADER_TILE_IMAGE_EXTENSION_NAME) {
has_shader_tile_image = true;
}
}

// Log the extension support

std::cout << vendor_name << " device detected with extension support:" <<
std::endl;
std::cout << " Dynamic Rendering: " << (has_dynamic_rendering ? "Yes" : "No") <<

505
 std::endl;
std::cout << " Dynamic Rendering Local Read: " <<
(has_dynamic_rendering_local_read ? "Yes" : "No") << std::endl;
std::cout << " Shader Tile Image: " << (has_shader_tile_image ? "Yes" : "No") <<
std::endl;

return has_dynamic_rendering || has_dynamic_rendering_local_read ||

has_shader_tile_image;
}

• Platform-Specific Optimizations: When developing for mobile devices, consider these

optimizations:

◦ Prioritize the use of dynamic rendering over traditional render passes on tile-based
renderers

◦ Use tile-based extensions whenever available

◦ Test different configurations to find the optimal settings for various device models

1.6. Best Practices for Using Extensions

1. Check for Support: Always check if an extension is supported before using it.

2. Fallback Paths: Implement fallback paths for when extensions aren’t available.

3. Test on Real Devices: Extensions may behave differently across vendors and devices. Test on a
variety of hardware from different manufacturers.

4. Stay Updated: Keep track of new extensions that could benefit mobile performance, as mobile
GPU vendors continue to enhance their Vulkan support.

In the next section, we’ll conclude our exploration of mobile development with a summary of key
takeaways and best practices.

Previous: Rendering Approaches | Next: Conclusion :pp: ++

Mobile Development: Conclusion

1. Conclusion
1.1. Putting It All Together
Let’s see how all these components can work together in a complete mobile-optimized Vulkan
application:

class MobileOptimizedEngine {
public:
MobileOptimizedEngine() {

506
 // Initialize platform-specific components
#ifdef __ANDROID__
initialize_android();
#elif defined(__APPLE__)
initialize_ios();
#else
initialize_desktop();
#endif

// Initialize Vulkan with mobile optimizations

initialize_vulkan();
}

void run() {
// Main application loop
while (!should_close()) {
handle_platform_events();
update();
render();
}

cleanup();
}

private:
void initialize_vulkan() {
// Create instance
vk::InstanceCreateInfo instance_info;
// ... set instance parameters
instance = vk::createInstance(instance_info);

// Select physical device

physical_device = select_physical_device(instance);

// Detect if we're on a TBR GPU

is_tbr_gpu = is_likely_tbr_gpu(physical_device);

// Check for extension support

auto available_extensions =
physical_device.enumerateDeviceExtensionProperties();
std::vector<const char*> supported_extensions = {
VK_KHR_SWAPCHAIN_EXTENSION_NAME };

// Add mobile-specific extensions if supported

if (check_extension_support(available_extensions,
VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME)) {
supported_extensions.push_back(VK_KHR_DYNAMIC_RENDERING_EXTENSION_NAME);
use_dynamic_rendering = true;
}

if (check_extension_support(available_extensions,

507
 VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME)) {

supported_extensions.push_back(VK_KHR_DYNAMIC_RENDERING_LOCAL_READ_EXTENSION_NAME);
use_dynamic_rendering_local_read = true;
}

if (check_extension_support(available_extensions,
VK_EXT_SHADER_TILE_IMAGE_EXTENSION_NAME)) {
supported_extensions.push_back(VK_EXT_SHADER_TILE_IMAGE_EXTENSION_NAME);
use_shader_tile_image = true;
}

// Create logical device with supported extensions

vk::DeviceCreateInfo device_info;
device_info.setPEnabledExtensionNames(supported_extensions);
// ... set other device parameters
device = physical_device.createDevice(device_info);

// Initialize other Vulkan resources

// ...
}

void render() {
// Begin frame
auto cmd_buffer = begin_frame();

if (use_dynamic_rendering) {
// Use dynamic rendering
vk::RenderingAttachmentInfoKHR color_attachment;
// ... set attachment parameters

vk::RenderingInfoKHR rendering_info;
// ... set rendering parameters

cmd_buffer.beginRenderingKHR(rendering_info);

// Record drawing commands

// ...

cmd_buffer.endRenderingKHR();
} else {
// Use traditional render passes
// ...
}

// End frame
end_frame(cmd_buffer);
}

// Platform-specific initialization
void initialize_android() {

508
 // Android-specific setup
// ...
}

void initialize_ios() {
// iOS-specific setup with MoltenVK
// ...
}

void initialize_desktop() {
// Desktop-specific setup
// ...
}

// Helper functions
bool check_extension_support(const std::vector<vk::ExtensionProperties>&
available, const char* extension_name) {
for (const auto& ext : available) {
if (strcmp(extension_name, [Link]) == 0) {
return true;
}
}
return false;
}

bool is_likely_tbr_gpu(vk::PhysicalDevice device) {

vk::PhysicalDeviceProperties props = [Link]();

// Most mobile GPUs from these vendors use TBR

if ([Link] == 0x5143 || // Qualcomm
[Link] == 0x1010 || // PowerVR
[Link] == 0x13B5 || // ARM Mali
[Link] == 0x19E5 || // Huawei
[Link] == 0x106B) { // Apple
return true;
}

return false;
}

// Vulkan objects
vk::Instance instance;
vk::PhysicalDevice physical_device;
vk::Device device;

// Flags
bool is_tbr_gpu = false;
bool use_dynamic_rendering = false;
bool use_dynamic_rendering_local_read = false;
bool use_shader_tile_image = false;

509
 };

1.2. Ship-Ready Checklist

1. Feature detection and fallbacks: Probe EXT/KHR support at startup, enable conditionally, and
maintain tested fallback paths.

2. Render path selection: Switch between TBR-friendly and IMR-neutral paths at runtime based on
a simple vendor/heuristic check.

3. Framebuffer read policy: Prefer tile-local, per-pixel reads (input attachments or dynamic
rendering local read). Avoid patterns that force external memory round-trips.

4. Textures and assets: Use KTX2 as the container; prefer ASTC when available with ETC2/PVRTC
fallbacks as needed. Generate mipmaps offline.

5. Memory/attachments: Use transient attachments where results aren’t needed after the pass;
suballocate to minimize fragmentation.

6. Thermal/perf governor: Implement dynamic resolution or quality tiers and sensible FPS caps to
keep thermals in check.

7. Instrumentation: Add GPU markers/timestamps, frame-time histograms, and bandwidth proxies

to track regressions.

8. Device matrix: Maintain a small, representative device lab (different vendors/tiers) and run
sanity scenes regularly.

1.3. Validation and Profiling Playbook

• Validate correctness:

◦ Swapchain details (present mode, min image count) per device.

◦ Layout transitions and access masks, especially when using local read.

◦ Synchronization between rendering scopes and compute/transfer work.

• Profile efficiently:

◦ Use platform tools (e.g., Android GPU Inspector, RenderDoc, Xcode GPU Capture) to identify
tile flushes, overdraw, and bandwidth hot spots.

◦ A/B test: classic render pass vs dynamic rendering, local read on/off, tile-image on/off.

◦ Track power and thermals over multi‑minute runs, not just single frames.

1.4. Next Steps

• Integrate a capability layer that exposes feature bits (dynamic rendering, local read, tile image)
to higher-level systems.

• Add automated startup probes that dump device/feature info to logs for field telemetry.

• Expand the regression scene suite to cover TBR‑sensitive and bandwidth‑heavy paths.

510
1.4.1. Explore Advanced Topics (Simple Engine Tutorials)

The following short, focused tutorials build directly on the Simple Engine and are great next steps:

• Tutorials Index — browse all topics

• Mipmaps and LOD — practical guidance on stable texture sampling and anisotropy.

• Dynamic Rendering Local Read — optimize same‑pass reads via tile/local memory when
supported.

1.5. Code Examples

The complete code for this chapter can be found in the following files:

Mobile Platform Integration C++ code Mobile Optimizations C++ code TBR Optimizations C++ code
Mobile Extensions C++ code

Previous: Vulkan Extensions for Mobile | Back to Building a Simple Engine :pp: ++

Mobile Development
This chapter covers the essential aspects of adapting your Vulkan engine for mobile platforms,
focusing on Android and iOS development, performance optimizations, rendering approaches, and
mobile-specific Vulkan extensions.

• Introduction

• Platform Considerations for Android and iOS

• Performance Optimizations for Mobile

• Rendering Approaches: TBR vs IMR

• Vulkan Extensions for Mobile

• Conclusion

Previous: Tooling Conclusion | Back to Building a Simple Engine :pp: ++

Advanced Topics (Simple Engine)

Welcome — this section collects short, conversational guides that explain what each feature is, why
we use it, and how it’s implemented in the Simple Engine.

Start anywhere that matches your interest:

• Planar Reflections

• Ray Query Rendering

• Ray Query Reflections and Transparency

511
 • Rendering Pipeline Overview

• Forward, Forward+, Deferred

• Forward+ Rendering

• Frustum Culling and Distance LOD

• Mipmaps and LOD

• glTF Animation & Transform Composition

• Push Constants (per‑object material)

• Descriptor Indexing & Stable Updates

• Separate Image/Sampler

• Synchronization & Streaming

• Synchronization 2 & Frame Pacing

• VK_EXT_robustness2

• Dynamic Rendering Local Read

• Shader Tile Image

Back to Building a Simple Engine = Frustum Culling and Distance‑based LOD

Culling is the simplest way to keep your GPU focused on what the camera can see. In this engine we
keep it intentionally pragmatic: CPU frustum tests plus a tiny “distance/size LOD” that skips objects
that would contribute only a handful of pixels.

• CPU frustum culling against per‑mesh AABBs

• A tiny distance/size LOD that skips very small objects (projected size threshold)

1. What we do
1. Extract the camera frustum planes from proj * view once per frame.

2. For each mesh instance, transform its local AABB to world space and test against the planes.

3. If enabled, estimate projected pixel size and skip objects below a threshold (separate thresholds
for opaque vs transparent).

2. Where to look in the code

• Plane extraction and AABB tests:

◦ renderer_rendering.cpp (helpers near the top of the file)

• Per-frame culling application:

◦ renderer_rendering.cpp (the render list building and per-pass filtering)

• UI controls:

◦ ImGui panel in renderer_rendering.cpp — “Frustum culling”, “Distance LOD”, and per-pass

512
 thresholds

3. Why it’s set up this way

• AABBs are cheap to transform and test; doing this on the CPU avoids sending obviously invisible
draws.

• A projected‑size cutoff is a practical alternative to a full LOD system for large scenes.

4. Tuning tips
• Start conservative (smaller thresholds), then increase until you can’t notice pop‑in while
moving.

• Transparent objects typically need a slightly higher threshold due to blending artifacts at tiny
sizes.

5. Future work ideas

If you want to push this further:

• Add per-material or per-layer culling rules (e.g., keep signage readable longer).

• Add hierarchical culling (BVH of AABBs) for very large scenes.

• Add GPU occlusion culling (HZB) once the pipeline grows beyond “readable sample” scale.

• Replace the projected-size heuristic with real mesh LODs (or meshlets).

6. What to read next

• Rendering Pipeline Overview

• Forward+ Rendering

• Ray Query Rendering = Descriptor Indexing and Stable Descriptor Updates

Vulkan descriptors are powerful, but they’re also one of the easiest places to accidentally violate
“frame in flight” lifetime rules.

In this engine we use one simple rule:

Only update descriptors at a known safe point.

That rule keeps streaming stable, keeps validation clean, and (most importantly) keeps the code
readable.

7. The safe point

Each frame‑in‑flight has a fence. At the start of a new frame, we wait for that fence. Once it signals,

513
the GPU is done with any work that referenced this frame’s descriptor sets. That’s the safe moment
to update this frame’s sets.

Why it matters: updating a set that’s still in use leads to invalid writes or so‑called
“update‑after‑bind” violations unless you deliberately opt into those behaviors and structure your
pipeline around them. The safe point pattern stays portable and clear.

8. What we update
• Material textures that finished streaming.

• The reflection texture binding (binding 10) for planar reflections.

• Per‑frame buffers for Forward+ (tile headers/indices, lights SSBO) when resized.

In Ray Query mode we also refresh the large texture table (the fixed-size sampler array) so that
newly streamed textures become visible without rebuilding the pipeline.

We refresh only the current frame’s sets at the safe point and leave other frames to update at their
own turn. This prevents cross‑frame “flip‑flop” where a texture looks different on alternating
frames.

9. Descriptor Indexing: when to use it

Descriptor Indexing opens features such as variable‑sized arrays and update‑after‑bind. It’s
powerful, but it shifts complexity to your synchronization and lifetime rules. In this sample we
emphasize clarity:

• We keep descriptor layouts simple and stable.

• We update at the safe point rather than while a command buffer might still be pending.

When we do use descriptor indexing features, it’s for one specific reason: large, non-uniformly
indexed descriptor arrays (e.g., Ray Query’s texture table). In that case, correctness depends on:

• enabling the descriptor indexing feature bits required by the GPU

• marking bindings with the correct binding flags (when supported)

• never caching stale Vulkan image/sampler handles across async streaming

If your project needs truly dynamic descriptor arrays or frequent mid‑frame updates, Descriptor
Indexing can be the right tool—just document the new invariants carefully.

10. Practical tips

• Centralize descriptor updates; don’t scatter writes across the frame.

• Use default textures for placeholders, then swap once—don’t bounce between real and default.

• Prefer combined image samplers for samples aimed at teaching; split image/sampler only when
you need the flexibility.

514
11. Where to look in the code
• Frame safe point + per-frame descriptor refresh:

◦ renderer_rendering.cpp

• Descriptor set layouts (including update-after-bind flags when enabled):

◦ renderer_pipelines.cpp

• Device feature enable for descriptor indexing:

◦ renderer_core.cpp

• Streaming-safe Ray Query texture table rebuild:

◦ renderer_ray_query.cpp

12. Future work ideas

If you want to explore more advanced descriptor patterns:

• Move to variable descriptor counts for texture tables (when device support is good enough for
your targets).

• Use separate image/sampler descriptors to share samplers across many textures.

• Add a “descriptor stress test” mode (development-only) that rapidly streams textures to validate
lifetime rules.

13. What to read next

• Synchronization and Stremaing

• Separate Image Sampler Descriptors

• Ray Query Rendering

This conservative approach avoids common pitfalls while keeping the code approachable. =
VK_KHR_dynamic_rendering_local_read — keeping color data in tile memory

Dynamic Rendering lets you render without full render passes/subpasses. The
VK_KHR_dynamic_rendering_local_read feature is a small but handy addition: it allows same‑pass
reads from attachments via tile/local memory paths on hardware that supports it.

14. Why it matters

Some post‑lighting effects and resolve‑like steps read the color you just wrote. With this feature,
drivers can service those reads from fast on‑chip memory instead of round‑tripping to VRAM.

515
15. How we approach it
• We enable the feature if present and keep codepaths compatible when it isn’t.

• We still end a rendering instance before doing layout transitions. The feature does not allow
arbitrary barrier misuse — regular Synchronization 2 rules apply.

16. Practical guidance

• Treat this as an optimization, not a new API surface.

• Keep stage/access masks precise. In this sample we keep transitions outside active rendering for
clarity.

17. Where to look in the code

• Feature detection and enablement:

◦ renderer_core.cpp (device feature enable path)

• Dynamic rendering setup + barriers:

◦ renderer_rendering.cpp

◦ renderer_pipelines.cpp

18. Future work ideas

If you want to demonstrate local-read more directly:

• Add a small “same-pass” effect that reads the current color attachment (e.g., a simple local
contrast or edge highlight).

• Add a debug HUD that prints whether the feature is enabled on the current device.

• Compare performance with and without local-read on tile-based GPUs (mobile) using a fixed
camera path.

19. What to read next

• Rendering Pipelne Overview

• Synchronization and Streaming

• Frame Pacing = Forward, Forward+, and Deferred — choosing the right path

Vulkan lets you build many kinds of pipelines. In practice, most real‑time engines gravitate toward
one of three shading architectures: Forward, Forward+, or Deferred.

This page explains what each one is, why this sample chooses Forward+, and where the relevant
pieces live in the code.

516
20. Forward rendering
Forward draws each object with its lighting in a single pass. It’s the most direct model: bind a
material, bind lights (uniforms or textures/SSBOs), draw. It’s easy to reason about and integrates
well with transparency and MSAA.

Pros:

• Simple and predictable.

• Good with transparent objects and MSAA.

• Great for small light counts or baked lighting.

Cons:

• Per‑pixel light loops can get expensive as the number of lights grows.

• You evaluate lights even when most don’t affect the pixel.

21. Forward+ (what we use for dynamic

lights)
Forward+ partitions the screen into tiles and assigns lights to those tiles with a compute pass. The
main pass then shades with only the lights relevant to the pixel’s tile. In this sample we use a
lightweight Forward+ that focuses on emissive/simplified lights to keep the code approachable.

Pros:

• Scales to many local lights; you only evaluate lights that might affect the pixel.

• Keeps forward’s strengths (transparency/MSAA friendliness).

Cons:

• Requires a pre‑pass or depth info and a compute dispatch to build the tile lists.

• More moving parts than plain forward.

22. Deferred shading (when to consider it)

Deferred writes material properties (G‑Buffer) in the first pass, then lights that buffer in a second
pass. That turns lighting cost into “cost per lighted pixel” and tends to excel with many lights, but it
makes transparency and MSAA trickier.

Pros:

• Many dynamic lights at high performance.

• Clear separation of material/write and light/evaluate.

517
Cons:

• Transparent objects must be handled separately (often with a forward pass).

• MSAA is more complex; memory bandwidth can be high.

23. What the sample uses (and why)

We use Forward+ for small, dynamic lights and a forward material path for everything else. That
keeps the code compact while still letting you place many little lights around the scene.
Transparency (glass) is shaded in a second forward pass so order and blending are correct.

If your project needs hundreds of shadowed lights and complex post‑lighting, explore a deferred
path or a hybrid: deferred for opaque, forward for transparent.

24. Implementation highlights in this

codebase
• A small compute pass builds per‑tile light lists.

• Per‑frame SSBOs hold tile headers/light indices; the main PBR pass reads those to loop only
relevant lights.

• Descriptor updates happen at the frame’s safe point so we don’t touch in‑use sets.

25. Where to look in the code

• Forward/Forward+ render loop integration:

◦ renderer_rendering.cpp

• Pipeline + descriptor layout setup:

◦ renderer_pipelines.cpp

• Main PBR shader (reads per-tile light lists when Forward+ is enabled):

◦ shaders/[Link]

The tile/cluster build shader is wired in renderer_pipelines.cpp. Start there and

 follow which compute pipeline is created for the Forward+ light assignment pass.

26. Choosing for your project

Use Forward if:

• Light count is low, transparency/MSAA are priorities, and you want the simplest pipeline.

Use Forward+ if:

518
 • You want many local lights but still want forward’s strengths.

Use Deferred if:

• You need to scale to many dynamic lights with complex lighting, and you’re ready to solve
transparency/MSAA separately.

There’s no one answer; pick the simplest that meets your needs. You can always grow the pipeline
later.

27. Future work ideas

If you want to expand the lighting system beyond “readable sample”:

• Add clustered Forward+ (3D clusters using depth slices) instead of 2D tiles.

• Add shadows (start with a single directional shadow map, then add point/spot shadows).

• Add a small deferred path for opaque only (keep transparent as forward).

• Add ray query helpers for selective effects (reflection rays, shadow rays, or AO probes) without
building a full RT pipeline.

28. What to read next

• Rendering Pipeline Overview

• Forward+ Rendering

• Frame Pacing = Forward+ Rendering in this Sample

Forward+ keeps the forward shading model you already know, but limits the per‑pixel light loop to
only the lights that might affect that pixel. It does this by dividing the screen into tiles (and
optionally Z‑slices) and building per‑tile light lists with a compute pass.

29. What we do
• Depth pre‑pass (optional): populates depth so the compute stage can cull by Z more effectively.

• Compute pass: assigns lights to tiles (and slices) and writes compact lists to SSBOs.

• Main PBR pass: for each pixel, fetch the tile header and iterate only those lights.

30. Where to look in code

• Buffers, per-frame state, and descriptor bindings:

◦ renderer_compute.cpp and renderer_resources.cpp (look for the ForwardPlusPerFrame data)

• Compute dispatch and per-frame parameters:

◦ renderer_rendering.cpp

519
 • Shader-side light list consumption:

◦ shaders/[Link] (Forward+ light loop)

31. Tips
• Tune tile size; 16×16 is a reasonable default for 1080p.

• If you pre‑pass depth, use depthWriteEnable=false and depthCompare=Equal in the subsequent

opaque color pass.

32. Future work ideas

If you want to take this beyond a compact sample:

• Upgrade from 2D tiles to clustered Forward+ (depth slicing and/or logarithmic Z).

• Add a small light “budget” UI and debug visualizations (tile heatmap) behind a development
build flag.

• Add shadowing (start with a single directional light shadow map) and extend the tile data to
include shadowed light indices.

33. What to read next

• Forward+ deferred rendering

• Rendering Pipeline Overview

• Frame Pacing = glTF Animation and Transform Composition

Animations bring life to 3D scenes — spinning ceiling fans, rotating gears, walking characters. This
page explains how this engine loads and plays animations from glTF files, with a strong focus on
transform composition (the thing that keeps objects animating in place instead of “drifting
away”).

34. What are glTF animations?

glTF (GL Transmission Format) is a standard for 3D assets that includes support for skeletal and
node-based animations. Animations in glTF consist of:

• Channels: Define which node property to animate (translation, rotation, or scale)

• Samplers: Provide keyframe data and interpolation methods (step, linear, or cubic spline)

• Timeline: Time values mapping to output values for smooth playback

When you export an animated model from Blender, Maya, or other 3D tools, the animation data
captures how nodes move over time relative to their initial transforms. This is crucial:
animations describe deltas (changes), not absolute world positions.

520
35. Understanding transform composition
Transform composition is the foundation of how animations work in 3D engines. Understanding
this concept is essential for implementing any animation system.

The core principle: Animation data in GLTF describes changes (deltas), not absolute positions.
When an artist animates a ceiling fan spinning in Blender, they’re defining how much it rotates
over time, not where it should be in world space.

Consider a ceiling fan at position (10, 5, 8) with an animation that rotates it. The animation
keyframes might specify: * Translation: (0, 0, 0) — no movement * Rotation: 0° → 360° around the Y
axis — spinning * Scale: (1, 1, 1) — no scaling

To display this correctly, we must compose the animation delta with the object’s base transform:

• Base transform: The object’s initial position/rotation/scale from the scene hierarchy

• Animation transform: The time-varying delta from the keyframes

• Final transform: Base composed with Animation

The composition rules are: * Translation: final = base + animDelta (addition) * Rotation: final =
base * animDelta (quaternion multiplication) * Scale: final = base * animDelta (component-wise
multiplication)

With proper composition, our ceiling fan remains at (10, 5, 8) and spins in place.

36. How it works in our engine

Our animation system has three key components:

36.1. 1. Loading: Extract base transforms and

animation data
When loading a glTF file (model_loader.cpp), we extract:

• Node transforms: Each GLTF node has a local transform matrix stored in
animatedNodeTransforms map

• Animation data: Channels, samplers, and keyframes stored in Animation objects

• Node-to-mesh mapping: Links node indices to mesh indices for entity matching

// In model_loader.cpp (conceptual)
std::unordered_map<int, glm::mat4> animatedNodeTransforms; // nodeIndex -> base
transform
std::unordered_map<int, int> animatedNodeMeshes; // nodeIndex -> meshIndex
std::vector<Animation> animations; // animation clips

521
36.2. 2. Scene setup: Create entities and apply base
transforms
In scene_loading.cpp, for each animated node:

• Create separate entities: If multiple nodes share the same mesh (like two ceiling fans), create
individual entities so each can animate independently

• Apply base transforms: Decompose the node’s transform matrix into position/rotation/scale
and set the entity’s TransformComponent

• Build nodeToEntity map: Links GLTF node indices to entity pointers for animation targeting

// For each animated node

glm::mat4 nodeTransform = animatedNodeTransforms[nodeIndex];
glm::vec3 position, scale;
glm::quat rotation;
glm::decompose(nodeTransform, scale, rotation, position, ...);

transform->SetPosition(position); // Base position (e.g., ceiling)

transform->SetRotation(eulerAngles(rotation));
transform->SetScale(scale);

Critical insight: animated nodes that share geometry must have separate entities. GPU instancing
(one entity, multiple transforms) doesn’t work for individual animation control.

37. Where to look in the code

If you want to follow the data end-to-end:

• glTF parsing (nodes, animations, samplers):

◦ model_loader.cpp

◦ model_loader.h

• Scene/entity creation and node→entity mapping:

◦ scene_loading.cpp

• Animation playback and transform composition:

◦ animation_component.cpp

◦ animation_component.h

• Transform storage and composition helpers:

◦ transform_component.cpp

◦ transform_component.h

522
38. Future work ideas
If you want to grow the animation system:

• Support animation blending (cross-fade between clips).

• Add skeletal skinning (vertex blending) if you want character animation.

• Add an animation debug UI that shows the active clip/time per entity (development-only).

• Add “bake transforms” options (useful for static meshes that only need a single animated pose).

39. What to read next

• Synchronization_and_Streaming.adoc (animation + streaming can interact in large scenes)

• Rendering_Pipeline_Overview.adoc

• Push_Constants_Per_Object.adoc

39.1. 3. Playback: Compose animation with base

transforms
In AnimationComponent::Update():

1. Capture base transforms on first frame: Store each entity’s initial position/rotation/scale
when animation starts

2. Sample keyframes: Interpolate animation data at current time

3. Compose transforms: Add/multiply animation deltas with base transforms

4. Apply to entity: Update the TransformComponent with the composed result

// Animation update logic

glm::vec3 basePos = basePositions[nodeIndex]; // e.g., (10, 5, 8)
glm::vec3 animTranslation = SampleVec3(sampler, time); // e.g., (0, 0, 0)
transform->SetPosition(basePos + animTranslation); // Result: (10, 5, 8)

glm::quat baseRot = baseRotations[nodeIndex]; // e.g., identity quaternion

glm::quat animRotation = SampleQuat(sampler, time); // e.g., 45° around Y
glm::quat finalRotation = baseRot * animRotation; // Compose using quaternion
multiplication
transform->SetRotation(glm::eulerAngles(finalRotation)); // Convert to Euler for
transform

40. Transform composition rules

Different transform properties compose differently:

523
Translation: Additive

finalPosition = basePosition + animationTranslation

Addition works naturally for positions in 3D space.

Rotation: Quaternion multiplication

finalRotation = baseRotation * animationRotation // quaternion math

finalEuler = eulerAngles(finalRotation) // convert for display

Rotations must be composed using quaternion multiplication to avoid gimbal lock and correctly
preserve rotation order. Always work in quaternion space during composition, then convert to
Euler angles only when setting the transform.

Scale: Multiplicative

finalScale = baseScale * animationScale // component-wise

Animation scale of (1, 1, 1) means "no change", (2, 1, 1) means "double X axis".

41. Handling multiple instances

When two GLTF nodes reference the same mesh (e.g., two identical ceiling fans), you need separate
entities for independent animation.

Why separate entities? * GPU instancing is designed for many identical, non-animated objects
(trees, rocks, grass) * Instance transforms are set once per frame; you cannot animate each instance
independently * Animation requires per-entity TransformComponents that update every frame

Implementation approach: Create separate entities

// First node reuses existing entity

nodeEntity = geometryEntities[meshIndex];

// Second node creates new entity with cloned geometry

nodeEntity = engine->CreateEntity("AnimNode_5");
mesh->SetVertices(sourceMesh->vertices); // Clone mesh data
mesh->SetIndices(sourceMesh->indices);

Each entity gets its own TransformComponent and can animate independently.

524
42. Keyframe interpolation
GLTF supports three interpolation modes:

Step: Jump instantly to next keyframe (no smoothing)

return [Link]; // Robotic, retro feel

Linear: Smooth linear blend between keyframes

return glm::mix(v0, v1, t); // Most common, looks natural

Cubic Spline: Smooth curves using tangents

// Hermite spline using in-tangent, value, out-tangent

// For production: implement full cubic interpolation for smoother motion

For rotations, use spherical linear interpolation (slerp) instead of mix:

return glm::slerp(q0, q1, t); // Avoids gimbal lock

43. Performance considerations

Animation Update Cost: O(channels × entities) * For 10 animated objects with 3 channels each
(translation, rotation, scale): ~30 transform updates per frame * This is cheap; transform math is
fast

Memory: Each animated entity needs: * Cloned mesh data (vertices, indices): ~100KB for a ceiling
fan * Transform storage: 3×vec3 = 36 bytes per node

Optimization tip: If you have hundreds of identical animated objects (e.g., grass blades), consider
GPU-side animation with compute shaders instead of per-entity CPU updates.

44. Alternatives and extensions

Skeletal animation (skinning)
• For characters with bones/joints

• Requires vertex skinning (blend multiple bone transforms per vertex)

• More complex than node animation but enables realistic deformation

Morph targets (blend shapes)

• For facial animation or smooth shape transitions

525
 • GLTF supports weights channel for morph targets

• Extends beyond node transforms to deform mesh vertices

Procedural animation
• Generate animation data at runtime (e.g., wind sway, noise-based motion)

• More flexible but requires custom authoring

45. What to read next

If you want to dive deeper:

• Transform Component: See transform_component.h for how we store and compute model
matrices

• GLTF Specification: GLTF Specification about animation

• Synchronization: How animation updates interact with render frame timing

The key takeaway: Always compose animation transforms with base transforms. This
fundamental principle is what makes objects animate in their correct world positions while the
animation data itself describes relative changes. Understanding this composition is essential for
any animation system.

Now you have the foundation to implement GLTF animations in your own projects. Happy
animating! Ὠ = Mipmaps and Level of Detail (LOD)

Mipmaps reduce aliasing and bandwidth by sampling pre‑filtered versions of a texture. LOD is
simply “which mip do we use right now?”.

In this sample the key idea is: we want stable, good-looking texture sampling while assets are
streaming in, without turning texture management into a giant subsystem.

46. What we do here

• Use mipmapped textures when available (KTX2 transcodes can include mips).

• For raw RGBA uploads, we cap auto‑generated mips to a small number to avoid large VRAM
spikes.

• Enable sampler anisotropy with a UI slider so you can see the trade‑offs quickly.

47. Where it lives in code

• Sampler creation and anisotropy slider:

◦ renderer_resources.cpp (sampler creation helpers)

◦ ImGui panel in renderer_rendering.cpp

• Upload path (staging → device image, then transition to SHADER_READ_ONLY_OPTIMAL):

526
 ◦ renderer_resources.cpp

◦ resource_manager.cpp / scene_loading.cpp (higher-level streaming/control flow)

48. Tips
• Prefer compressed formats (BC/ASTC/ETC) with mips for big scenes.

• Clamp the max anisotropy to what your device supports.

49. Future work ideas

If you want to take this farther:

• Add a per-material “mip bias” control (great for stylized looks and debugging shimmering).

• Add texture streaming by mip level (load low mips first, then refine).

• Add a small “texture residency” overlay (counts of textures by mip availability).

50. What to read next

• Descriptor Indexing UpdateAfterBind

• Synchronization and Streaming

• Ray Query Rendering = Planar Reflections in Our Engine

You’ve probably noticed shiny floors and windows in real‑world scenes. In games, we often fake
that look. In this engine we chose a practical, reliable technique: planar reflections. This page
explains what they are, why we use them, how they’re implemented, and when you might want
something else.

51. What are planar reflections?

Planar reflections render a mirror image of the scene across a single plane (e.g., a flat floor or a
window). Think of it as a “mirror camera” that looks into the scene from the other side of the
reflective surface. We render that mirrored view into a texture, then sample that texture when
drawing glass (or any reflective planar surface).

Planar reflections work great for:

• Flat mirrors, calm water, polished floors, glass panes.

• Scenes where you need stable, high‑quality reflections without heavy noise or temporal
instability.

They are not ideal for:

• Curved/rough surfaces that need glossy, view‑dependent blurs everywhere.

527
 • Arbitrary reflection directions (e.g., metals with complex micro‑geometry).

52. Why we chose planar reflections for the

sample
We want a reflection method that is:

• Easy to understand (one extra pass, one extra texture).

• Deterministic and stable (no “sparkles” or temporal accumulation headaches).

• Practical for a single dominant reflector (glass, floor) in a forward renderer.

Planar reflections deliver all three. They also scale well across GPUs without requiring ray tracing
hardware.

53. How it works in our engine

We add one small pass and one small blend in the main pass:

1. Mirror pass (off‑screen)

◦ Compute a mirrored view matrix by reflecting the camera across a plane (e.g., Y=0 for a
ground plane).

◦ Render the opaque scene with face culling disabled (or adjusted) into a reflection
color+depth target.

◦ Synchronize the reflection image for sampling in the next pass.

2. Main pass (normal camera)

◦ Draw opaque + transparent objects as usual.

◦ When drawing glass, sample the reflection texture and blend it with glass shading using
Fresnel + roughness + a user‑controlled “reflection intensity”.

That’s it. No special render graph magic, no ray queries, no temporal accumulation.

54. Where to look in the code

• Mirror camera math, reflection pass, and pass ordering:

◦ renderer_rendering.cpp

• Reflection render targets and pipeline setup:

◦ renderer_pipelines.cpp

• Reflection sampling + glass shading:

◦ shaders/[Link]

◦ shaders/pbr_utils.slang

528
 • Reflection binding and per-frame safe-point updates:

◦ renderer_rendering.cpp (reflection descriptor refresh)

55. The mirror math (short and sweet)

You define a plane in world space: ax + by + cz + d = 0.

From that plane you build a reflection matrix R. Apply R to the regular camera view to get the
mirrored view. In practice you’ll also flip culling or set cullMode = none for the mirrored pass
because the winding order changes under reflection.

We also pass the plane to shaders for optional clipping:

• A simple dot(product) with world position lets us discard fragments “behind” the plane in the
mirror pass.

56. The rendering steps in detail

Mirror pass:

• Create a reflection color image (format matches your main pass needs; we pick a color format
that the composite/glass pass can sample easily) and a reflection depth image.

• Before rendering: transition the reflection color image from SHADER_READ_ONLY_OPTIMAL to

COLOR_ATTACHMENT_OPTIMAL using Synchronization 2 (vkCmdPipelineBarrier2). Do the same for
depth to DEPTH_ATTACHMENT_OPTIMAL.

• Begin dynamic rendering, bind the PBR pipeline for opaque objects, and disable culling (or flip
front faces).

• Render opaque meshes. You can add a clip test against the plane if needed.

• End rendering. Transition the reflection color image to SHADER_READ_ONLY_OPTIMAL for sampling
in the main pass.

Main pass:

• Render opaque as usual (we use an off‑screen buffer to do tone‑mapped composite later).

• Transparent pass: when drawing glass, sample the reflection texture and blend:

◦ Use Fresnel (stronger at grazing angles) and reduce with roughness.

◦ Multiply by a small “reflection intensity” you can tune in the UI.

57. Synchronization and barriers (what

matters)
We keep it simple with Vulkan Synchronization 2:

529
 • Do not change image layouts inside an active dynamic render pass. End it first.

• Use vkCmdPipelineBarrier2 with: correct source/destination stage masks, access masks, and
old/new layouts.

• Reflection color: SHADER_READ_ONLY_OPTIMAL → COLOR_ATTACHMENT_OPTIMAL before mirror pass; back

to SHADER_READ_ONLY_OPTIMAL after.

• Swapchain image: transition to COLOR_ATTACHMENT_OPTIMAL for composite/transparent; transition

to PRESENT_SRC_KHR only after ending the last rendering pass.

58. Descriptors: where is the reflection

bound?
• We reserve binding 10 in the PBR set for the reflection sampler.

• At the per‑frame “safe point” (when previous frame’s work is done), we refresh binding 10 for
the current frame to point to the reflection image from the previous frame.

• The glass shader checks a UBO flag (reflectionEnabled) and samples only when a valid
reflection image exists.

59. Glass blending: an approachable model

Glass is mostly transmission, but we want vivid, plausible reflections. We use:

• Fresnel term (Schlick): stronger reflections at grazing angles.

• Roughness factor: more roughness → weaker, blurrier reflections (we keep it simple here and
just dim the strength).

• Reflection intensity slider: exposed in the UI so you can tune visibility in seconds.

This is not a full physical spectral model, and that’s fine. It’s readable and produces convincing
results.

60. Alternatives and when to choose them

Screen‑space reflections (SSR)
• Works without extra passes; uses existing color/depth from your frame.

• Great for puddles and local effects, but can miss off‑screen objects and suffers from temporal
instability.

• Choose SSR if you want quick reflections everywhere and can accept occasional artifacts.

Environment maps / cube maps / reflection probes

• Very fast; precomputed.

• Not view‑accurate for nearby objects; best for distant glossy reflections.

• Choose probes for general ambient reflections or when the surface isn’t a perfect mirror.

530
Ray tracing (hardware) / hybrid approaches
• Very accurate; supports complex reflections.

• Requires hardware and advanced denoising; more code and performance cost.

• Choose RT if you target high‑end GPUs and want “it just looks right” reflections everywhere.

Planar reflections (this sample)

• A single extra pass, deterministic and stable.

• Perfect for one or a few large planar reflectors (floor, windows, calm water).

• Choose this when you want high‑quality mirrors for specific surfaces without adopting ray
tracing.

61. Performance tips

• Render the mirror pass at a lower resolution (we provide a resolution scale slider).

• Cull aggressively (our CPU frustum culling works for both camera and mirrored camera).

• Disable the mirror pass when the reflective surface isn’t visible.

• Consider blurring the reflection sample for rough surfaces if you want softer looks.

62. Troubleshooting
“Reflections appear too weak”
• Increase the Reflection intensity slider and/or reduce roughness.

63. Future work ideas

If you want to push planar reflections further:

• Add a roughness-aware blur of the reflection texture (mip chain or separable blur).

• Add multiple reflection planes (useful for multi-floor scenes).

• Add a screen-space fallback (SSR) and blend with planar where valid.

• Add selective ray query reflections for non-planar surfaces (hybrid approach).

64. What to read next

If you’re curious about the rest of this sample:

• Synchronization and Streaming

• Forward+ Rendering

• Descriptor Indexing and Stable Descriptor Updates

• Rendering Pipeline Overview

531
Enjoy experimenting. This approach is intentionally straightforward so you can focus on learning
Vulkan’s moving parts without getting lost in a maze of techniques. = Push Constants — per‑object
material properties

Push constants are tiny pieces of data you can send to shaders without creating or updating buffers.
They’re perfect for per‑draw material knobs.

In this engine we use push constants for values that change per draw call (material factors and “is
there a texture?” flags). Anything that changes less frequently (camera data, light lists, big arrays)
stays in UBOs/SSBOs.

65. How we use them

We pack the common PBR controls (base color factor, metallic/roughness, texture presence flags,
emissive strength, transmission, IOR) into a single push constant block and update it before each
draw.

66. Where to look

• C++ push constant struct and update call:

◦ renderer.h (MaterialProperties)

◦ renderer_rendering.cpp (where we push per-draw material properties)

• Shader push constant block:

◦ shaders/[Link] ( block)

• PBR helper functions used by the shader:

◦ shaders/pbr_utils.slang

◦ shaders/lighting_utils.slang

67. Guidelines
• Keep the block small (Vulkan guarantees at least 128 bytes). This sample fits comfortably.

• Use push constants for values that change every draw call. Use UBO/SSBO for larger,
less‑frequent data.

68. Future work ideas

If you want to extend this pattern:

• Split “rarely changing per-material data” into a GPU material buffer and use push constants
only for the per-draw index.

• Add a second push constant block for per-draw debug visualizations (development-only) to keep
it out of hot UBO paths.

532
 • Add a material override system (force roughness/metallic for entire scene) by layering a global
UBO on top.

69. What to read next

• Rendering Pipeline Overview

• Descriptor Indexing UpdateAfterBind

• Ray Query Rendering == Ray Query Reflections and Transparency

Ray queries make it straightforward to add reflection and refraction to a renderer without adopting
a full ray tracing pipeline. In this engine, the Ray Query mode compute shader already computes
primary visibility; we extend that shader with secondary rays to handle reflective and
transmissive materials.

This page explains the design in a way you can reuse in your own projects.

69.1. Two toggles, one clear mental model

Ray Query mode exposes two feature toggles:

• Reflections: enables a reflection ray from the first hit.

• Transparency/Refraction: enables a refraction ray for transmissive materials.

There’s also a small quality knob:

• Max secondary bounces: 0 disables secondary rays entirely; 1 enables a single bounce.

The point of the bounce cap is to keep performance predictable while still demonstrating how ray
queries can be layered into a physically-based shading model.

69.2. Reflection rays (one bounce)

At the first surface hit we have:

• the outgoing view direction V

• the surface normal N

• material parameters (roughness, metallic, and Fresnel base reflectance)

The reflection direction is the standard geometric reflection:

R = reflect(-V, N)

In the compute shader we trace a new ray from a small offset along the normal to avoid self-
intersections:

• origin: P + N * eps

• direction: R

533
If the reflection ray hits something, we shade that hit using the same PBR path as the primary ray. If
it misses, we use a stable sky/background function.

The final reflection contribution is weighted by Fresnel and reduced by roughness:

• grazing angles reflect more

• rough surfaces reflect less strongly

This keeps the result intuitive and stable.

69.3. Thin-glass refraction (one bounce)

For transmissive materials we implement a “thin glass” model:

• a refraction ray gives you the view through the surface

• a reflection ray gives you the view on the surface

• Fresnel blends between them

We compute refraction using Snell’s law with a simple total internal reflection fallback.

The refraction ray uses:

• origin: P + refrDir * eps (offset along the refraction direction)

• direction: refrDir

The transmitted result is blended with reflection using Fresnel, and then mixed into the base
surface color using the material’s transmission factor.

69.4. Alpha-masked surfaces (foliage)

Many real scenes use alpha masking for foliage and thin geometry. Alpha masking is different
from regular blending:

• the surface is either present or absent per pixel

• the decision is driven by a baseColor alpha texture and an alphaCutoff

In a traditional ray tracing pipeline, alpha masking is often implemented in an any-hit shader. With
ray queries, we can implement the same idea by controlling which candidate intersections get
committed.

The approach is:

1. Allow non-opaque candidates for alpha-masked instances.

2. For each candidate triangle hit:

◦ compute the candidate UV

◦ sample baseColor alpha

534
 ◦ accept the candidate only when alpha >= alphaCutoff

This produces correct visibility for masked geometry in primary rays, and it also keeps
reflections/refractions consistent because they use the same traversal routine.

69.5. Where to look in the code

• Ray Query shader implementation:

◦ shaders/ray_query.slang

• Ray Query UI toggles and bounce cap:

◦ renderer_rendering.cpp == Ray Query Rendering

This engine includes a ray-traced rendering mode built on Vulkan’s ray queries. Instead of building
a full ray tracing pipeline (raygen / miss / hit shaders), ray queries let you perform intersection tests
directly from regular shaders.

In this sample we use ray queries from a compute shader to render the whole frame:

• Build BLAS (per mesh) and a TLAS (scene instances).

• Dispatch a compute shader that:

◦ Generates one primary ray per pixel from the camera.

◦ Uses TraceRayInline() to find intersections in the TLAS.

◦ Shades the hit using the same PBR utilities as the raster path.

• Write the result into a storage image, then composite to the swapchain.

69.6. Why ray queries?

Ray queries are a good fit for a “hybrid” renderer:

• You can call them from compute, fragment, or other shader stages.

• They reuse the standard descriptor system.

• They keep control flow in your shader code: you decide how to traverse, when to accept hits,
and how to shade.

69.7. High-level architecture

At a high level, Ray Query mode touches three areas:

• Acceleration structures: built from the scene’s vertex and index buffers.

• Descriptors: bind the TLAS, the output storage image, and the scene data needed for shading.

• Shader: generate rays, do the query, shade the hit.

The important idea is that the ray query shader does not “own” the scene. It reads the same scene
assets as rasterization (meshes, materials, textures), but through a separate descriptor set designed

535
for the compute path.

69.8. Acceleration structure build (BLAS/TLAS)

We build acceleration structures once the scene is ready:

• A BLAS is created per unique mesh.

• Each scene instance is added to the TLAS with its transform.

• Each TLAS instance encodes a custom instance index so the shader can index into a matching
GeometryInfo table.

The Ray Query shader uses that per-instance index to look up:

• device addresses for vertex and index buffers

• the material index

• a per-instance normal transform for correct world-space normals

69.9. Descriptor layout

Ray Query mode uses a dedicated descriptor set layout. The exact binding numbers matter because
they must match the shader.

Typical bindings in this engine are:

• Binding 0: a small Ray Query-specific UBO (camera matrices, exposure/gamma, toggles)

• Binding 1: the TLAS

• Binding 2: output storage image

• Binding 3: light buffer

• Binding 4: GeometryInfo buffer

• Binding 5: material buffer

• Binding 6: a large combined image sampler array used as a texture table

69.10. Streaming-safe texture access

This engine streams textures asynchronously. A key design choice for Ray Query mode is that the
shader indexes textures through a fixed-size array (a “texture table”).

At runtime:

• Materials store texture indices into the table (baseColor, normal, metallic-roughness, occlusion,
emissive).

• The renderer refreshes the table using the current texture handles.

• Slots 0..4 are reserved for shared default textures (so sampling always has a valid fallback).

536
This approach keeps shading simple in the shader: sampling uses NonUniformResourceIndex() and
SampleLevel(…, 0.0) (explicit LOD is important for compute).

69.11. Dispatch and presenting the result

The Ray Query compute shader writes to a storage image (typically HDR-capable).

After dispatch:

• A barrier transitions the Ray Query output image from GENERAL (write) to
SHADER_READ_ONLY_OPTIMAL (read).

• A fullscreen composite pass samples the output image and writes to the swapchain.

• A final transition prepares the swapchain for present.

This lets the engine reuse the same post-processing controls (exposure/gamma) for both raster and
ray query paths.

69.12. Where to look in the code

• Shader:

◦ shaders/ray_query.slang

• CPU-side Ray Query build and descriptors:

◦ renderer_ray_query.cpp

• Render loop integration + UI:

◦ renderer_rendering.cpp

• Descriptor indexing features (for large sampler arrays):

◦ renderer_core.cpp = Rendering Pipeline Overview (this sample)

This engine uses Vulkan Dynamic Rendering with a small, readable sequence of passes. The order is
deliberate: it keeps tone mapping explicit, keeps transparency sane, and makes it easy to plug in
optional features like planar reflections or the Ray Query compute path.

70. The pass order

1. Optional reflection pass (off‑screen):

◦ Mirror the camera across a plane (e.g., floors or windows).

◦ Render opaque geometry into a reflection render target (color + depth).

◦ Transition the reflection image to SHADER_READ_ONLY_OPTIMAL for the next frame.

2. Opaque to off‑screen color:

◦ Render all opaque objects into an off‑screen color image (opaqueSceneColor).

◦ Depth is read/write as usual (or read‑only if you ran a depth pre‑pass).

537
3. Composite to swapchain:

◦ End the opaque rendering.

◦ Transition opaqueSceneColor to SHADER_READ_ONLY_OPTIMAL.

◦ Begin a new rendering instance targeting the swapchain image and draw a full‑screen pass
that samples the off‑screen color (tone mapping included).

4. Transparent on top:

◦ Keep the swapchain image as the color attachment and bind the scene depth.

◦ Render transparent objects (glass/liquids) back‑to‑front.

◦ Glass samples the prior frame’s reflection texture when enabled.

5. UI:

◦ Render the UI on top.

◦ Transition the swapchain image to PRESENT_SRC_KHR after ending rendering.

71. Where to look in the code

• Main render loop + pass ordering:

◦ renderer_rendering.cpp

• Pipeline setup (dynamic rendering attachments, layouts, formats):

◦ renderer_pipelines.cpp

• Composite pass shader (tone mapping + presentation):

◦ shaders/[Link]

• PBR shading utilities shared by multiple pipelines:

◦ shaders/[Link]

◦ shaders/pbr_utils.slang

◦ shaders/lighting_utils.slang

72. Why this shape

• A single off‑screen buffer makes tone mapping explicit and avoids gamma‑incorrect copy paths.

• Transparent ordering stays simple because the swapchain is the current color attachment in
that pass.

• The reflection pass is optional and self‑contained.

73. Future work ideas

If you want to take this pipeline further:

• Add a depth pre-pass for heavy scenes (helps early-z and enables more accurate Forward+

538
 clustering).

• Add a lightweight bloom chain that runs between the off-screen opaque pass and the composite.

• Add a dedicated transparent resolve path (weighted blended OIT) if you need lots of
overlapping glass.

• Demonstrate hybrid rendering by calling ray queries from a raster shader (reflection probe,
shadow test, or glass-only reflections). = VK_EXT_robustness2 — safer defaults for real‑world
engines

Vulkan lets you run fast and close to the metal. That also means a bad index or out‑of‑range access
can produce undefined results. VK_EXT_robustness2 tightens that up so mistakes fail predictably
instead of corrupting memory or producing flicker.

74. What it gives you

• Robust buffer access 2 — out‑of‑bounds buffer reads return zero; writes are discarded.

• Robust image access 2 — out‑of‑range image coordinates clamp or return zero per the spec.

• Null descriptors — a descriptor can be left “null” and the shader sees a defined zero value
instead of UB.

These behaviors make the engine more forgiving while students iterate and while textures stream
in.

75. How we use it here

• We enable the extension and feature structs during device creation when available.

• Shaders are written assuming legal ranges, but if a streaming texture or optional binding is
temporarily missing, sampling a null descriptor is defined and safe.

• The Forward+/reflection paths avoid mid‑frame descriptor edits; robustness2 then acts as an
extra safety net.

76. When to enable

Always enable when the device supports it for teaching samples and tools. For shipping titles, you
can still keep it on; the performance cost is generally negligible on modern drivers, and the safety is
worth it.

77. Where to look in the code

• Device extension/feature enable:

◦ renderer_core.cpp

◦ vulkan_device.cpp

• Bounds checks and defensive indexing in the Ray Query shader:

539
 ◦ shaders/ray_query.slang (bounds checks for geometryInfoCount / materialCount)

• Safe descriptor update patterns (so you don’t rely on robustness for correctness):

◦ renderer_rendering.cpp (per-frame safe point)

◦ Descriptor_Indexing_UpdateAfterBind.adoc

78. Takeaways
• Robustness doesn’t replace good synchronization and lifetime rules; it complements them.

• Null descriptors and “safe zero” reads make streaming and feature toggles less fragile.

79. Future work ideas

If you want to stress-test robustness (without turning the engine into a debugging tool):

• Add a development-only “fault injection” toggle that intentionally feeds out-of-range indices in a
controlled shader path.

• Add a small runtime report that prints whether VK_EXT_robustness2 is enabled on the current
device.

• Add a unit-style GPU test scene that exercises missing textures / missing buffers while keeping
VVL clean.

80. What to read next

• Synchronization and Streaming

• Descriptor Indexing UpdateAfterBind

• Ray Query Rendering = Separate Image and Sampler Descriptors — when and why

Vulkan lets you bind an image view and a sampler either together (combined image sampler) or
separately. Combined bindings are simpler to teach and maintain. Separate bindings give you
flexibility (e.g., reuse one sampler across many images; change only the sampler states).

In this sample we default to combined image samplers because they keep the descriptor model
simple while we’re focused on bigger engine concepts (streaming, synchronization, pass structure).

81. Our default

We prefer combined image samplers in this sample for readability and because most materials
don’t swap samplers at runtime.

82. When to split

• You want to toggle sampler states (e.g., enable/disable anisotropy) across many textures without

540
 updating every descriptor.

• You have a library of sampler objects (point/linear/aniso/wrap/clamp) and want to

mix‑and‑match with images.

83. Practical guidance

• Keep layouts small and stable for teaching.

• If you introduce split bindings, document lifetime rules clearly: images and samplers can now
change independently.

84. Where to look in the code

• Texture and sampler creation:

◦ renderer_resources.cpp

• Descriptor layouts and bindings:

◦ renderer_pipelines.cpp

• Descriptor updates at the per-frame safe point:

◦ renderer_rendering.cpp

85. Future work ideas

If you want to demonstrate separate image/sampler descriptors concretely:

• Create a small sampler “library” (point/linear/aniso, wrap/clamp) and switch sampler indices
from the UI.

• Use shared samplers with a large texture table to reduce descriptor update volume during
streaming.

• Add per-material sampler selection (e.g., nearest for pixel art signage).

86. What to read next

• Descriptor Indexing UpdateAfterBind

• Mipmaps and LOD

• Synchronization and Streaming = VK_EXT_shader_tile_image — fast access to tile data

Some GPUs expose “tile” or “subpass” data paths that let shaders read from on‑chip color/depth
without a round trip to memory. VK_EXT_shader_tile_image is a portable way to tap into that.

In this sample we treat it as an optional optimization: the pipeline remains correct without it, and
we only take a “fast path” when the device advertises support.

541
87. What problem it solves
Post‑lighting texture reads from the just‑written color can be expensive. With tile image access,
certain patterns become cheaper and more deterministic on supported hardware.

88. How we handle it in this sample

• We enable the feature if present and expose a boolean you can check in code.

• The renderer still uses clean Synchronization 2 barriers and ends dynamic rendering before
formal layout transitions. That keeps the code understandable on devices that don’t support tile
reads.

89. Guidance
Use it as an optimization. Write code that’s correct everywhere, then add tile‑image fast paths when
available.

90. Where to look in the code

• Feature detection and enablement:

◦ renderer_core.cpp

• Dynamic rendering setup and attachment transitions (kept explicit for clarity):

◦ renderer_rendering.cpp

◦ renderer_pipelines.cpp

91. Future work ideas

If you want to demonstrate tile-image usage more directly:

• Add a small “local read” post effect that reads from the current color attachment and compares
with a regular sampled path.

• Add a device capability print (development-only) so students can see when the fast path is
active.

• Add a micro-benchmark scene and compare bandwidth on tile-based GPUs.

92. What to read next

• Dynamic Rendering Local Read

• Frame Pacing

• Rendering Pipeline Overview = Synchronization 2 and frame pacing in this engine

542
Vulkan Synchronization 2 makes barriers and submissions easier to read. This sample uses it to
keep uploads and rendering in step without stalls.

The goal here isn’t “maximum cleverness.” It’s predictable ordering:

• the transfer queue moves data onto the GPU

• the graphics queue draws using whatever is ready

• the CPU only mutates per-frame resources when it knows the GPU is done with them

93. The moving parts

• Timeline semaphore on the transfer queue — batches of texture uploads signal increasing
values.

• Graphics submit waits on the latest uploads value — by the time we draw, textures are ready to
sample.

• Frame fences — each frame‑in‑flight has a fence we wait on at the start of the next frame’s CPU
work.

94. Barriers we rely on

Uploads path:

• UNDEFINED → TRANSFER_DST_OPTIMAL (dstStage: TRANSFER, dstAccess: TRANSFER_WRITE)

• After copy: TRANSFER_DST_OPTIMAL → SHADER_READ_ONLY_OPTIMAL (srcStage: TRANSFER, dstStage:

FRAGMENT_SHADER)

Render path:

• Attachment images transition outside active dynamic rendering blocks using

vkCmdPipelineBarrier2.

• Swapchain transitions: to COLOR_ATTACHMENT_OPTIMAL before composite/transparent, to

PRESENT_SRC_KHR after ending the last rendering pass.

95. Descriptor updates at the safe point

At the start of a frame, after waiting on the frame fence, we refresh only this frame’s descriptor
sets. That avoids “update‑after‑bind” pitfalls and frame‑to‑frame flicker during streaming.

96. Takeaways
• Keep transitions outside active beginRendering/endRendering scopes.

• Use clear stage/access pairs; prefer Synchronization 2 for readability.

• Pair timeline semaphores with fences: timelines coordinate queues; fences bound the CPU turn.

543
97. Where to look in the code
• Upload submission and timeline semaphore signaling:

◦ renderer_resources.cpp

◦ renderer_utils.cpp

• Graphics submit waits (including “latest upload value”):

◦ renderer_rendering.cpp

• Image barriers for the render path (attachments + swapchain):

◦ renderer_rendering.cpp

• Swapchain and present integration:

◦ swap_chain.h

◦ renderer_rendering.cpp

98. Future work ideas

If you want to experiment with pacing and latency:

• Add a UI toggle for the frames-in-flight count and measure input latency vs throughput.

• Add a “fixed camera path” mode (development-only) to produce repeatable GPU timing
comparisons.

• Add GPU timestamp queries around the big passes to visualize where time goes.

• Add async compute experiments (if your device supports it) for things like Forward+ light list
building.

99. What to read next

• Synchronization and Streaming

• Descriptor Indexing UpdateAfterBind

• Rendering Pipeline Overview = Synchronization and Streaming

Modern Vulkan gives us powerful tools to keep the GPU busy while assets stream in. This engine
uses a background uploader, a dedicated transfer queue, and Synchronization 2 to avoid stalls and
flicker. Let’s walk through the moving parts and how they fit together.

100. The idea

• File I/O and staging happen off the render thread.

• GPU copies and layout transitions run on a transfer queue, not the graphics queue.

• A timeline semaphore lets graphics wait for “the latest finished upload” without

544
 micro‑managing per‑resource fences.

• We only update descriptors at a safe point (right after waiting for the in‑flight frame’s fence) so
we never write into sets that the GPU is still using.

This keeps the frame loop simple and responsive—even while large textures stream in.

101. The background uploader

We enqueue texture jobs (transcode/IO → staging buffer → device image). A dedicated thread:

1. Batches pending copies into a command buffer on the transfer queue.

2. Records layout transitions from TRANSFER_DST_OPTIMAL to SHADER_READ_ONLY_OPTIMAL using

Synchronization 2.

3. Submits once, signaling a monotonically increasing timeline value.

4. Notifies the renderer which textures are now “ready to sample.”

The render submit includes a wait on the latest uploads timeline value, so textures are available by
the time we draw.

102. The safe point for descriptor updates

Vulkan won’t let us mutate a descriptor set that’s currently in use. The engine does this instead:

• At the start of each frame, we wait for the fence associated with that frame‑in‑flight.

• Now it’s safe to update this frame’s descriptor sets (they aren’t in use).

• We refresh image bindings with the uploaded texture’s view/sampler at this point.

As a result there’s no texture “flip‑flop” or flicker: once a real texture replaces a placeholder, it
stays.

103. Synchronization 2 in practice

Uploads path uses vkCmdPipelineBarrier2 with clear, minimal scopes:

• Staging → image copy: make the destination image TRANSFER_DST_OPTIMAL.

• After the final copy: transition to SHADER_READ_ONLY_OPTIMAL (src stage = eTransfer, dst stage =
eFragmentShader).

• Ownership transfers only if the transfer and graphics queues use different families (most
desktop drivers share families).

On the graphics side, we keep attachment layout transitions outside of any dynamic render pass
instance and also use vkCmdPipelineBarrier2 for readability.

545
104. A typical texture’s journey
1. Job enqueued with a file path.

2. Background thread: load/transcode to staging, allocate device image, record copies.

3. Submit to transfer queue; signal timeline.

4. Renderer’s next frame begins; the per‑frame fence unblocks.

5. Descriptor for this frame updates to point at the uploaded image (safe point).

6. Draw: fragment shader samples the new texture without stalls.

105. Tips and pitfalls

• Keep descriptor updates at the safe point. Avoid updating in‑use sets.

• Use the transfer queue for bulk copies; keep the graphics queue focused on drawing.

• Prefer Synchronization 2 for clarity (stage/access pairs are explicit, transitions stand out).

• Batch uploads: the fewer submits, the lower the CPU overhead.

That’s all it takes to make streaming feel “invisible” to the player—and tidy to maintain.

106. Where to look in the code

• High-level scene load and job enqueue:

◦ scene_loading.cpp

◦ resource_manager.cpp

• Texture/image creation + upload path:

◦ renderer_resources.cpp

• Transfer queue submission + synchronization helpers:

◦ renderer_utils.cpp

◦ vulkan_device.cpp

• Frame “safe point” (per-frame fence wait) and descriptor refresh:

◦ renderer_rendering.cpp

• Descriptor update patterns (why we update at the safe point):

◦ Descriptor_Indexing_UpdateAfterBind.adoc

107. Future work ideas

If you want to push streaming further:

• Stream by mip level (low mips first), then refine in the background.

546
 • Add a small streaming HUD (bytes queued, bytes uploaded, textures ready) behind a
development build flag.

• Add per-resource priorities (camera distance, importance tags) so the most noticeable assets
arrive first.

• Add “hot reload” for textures to validate descriptor lifetime rules under rapid churn.

108. What to read next

• Synchronization_2_Frame_Pacing.adoc

• Descriptor_Indexing_UpdateAfterBind.adoc

• Ray_Query_Rendering.adoc :pp: ++

Appendix:
1. Detailed Architectural Patterns
This appendix provides in-depth information about common architectural patterns used in modern
rendering and game engines. These patterns are referenced in the main Engine Architecture
section, with a focus on Component-Based Architecture in the main tutorial.

1.1. Layered Architecture

One of the most fundamental architectural patterns is the layered architecture, where the system is
divided into distinct layers, each with a specific responsibility.

[Layered Architecture Diagram] | ../../../images/layered_architecture_diagram.png

1.1.1. Typical Layers in a Rendering Engine

1. Platform Abstraction Layer - Provides a consistent interface to platform-specific functionality.

2. Resource Management Layer - Manages loading, caching, and unloading of assets.

3. Rendering Layer - Handles the rendering pipeline, shaders, and graphics API interaction.

4. Scene Management Layer - Manages the scene graph, spatial partitioning, and culling.

5. Application Layer - Handles user input, game logic, and high-level application flow.

1.1.2. Benefits of Layered Architecture

• Clear separation of concerns

• Easier to understand and maintain

• Can replace or modify individual layers without affecting others

• Facilitates testing of individual layers

547
1.1.3. Implementation Example

// Platform Abstraction Layer

class Platform {
public:
virtual void Initialize() = 0;
virtual void* CreateWindow(int width, int height) = 0;
virtual void ProcessEvents() = 0;
// ...
};

// Resource Management Layer

class ResourceManager {
public:
virtual Texture* LoadTexture(const std::string& path) = 0;
virtual Mesh* LoadMesh(const std::string& path) = 0;
// ...
};

// Rendering Layer
class Renderer {
public:
virtual void Initialize(Platform* platform) = 0;
virtual void RenderScene(Scene* scene) = 0;
// ...
};

// Scene Management Layer

class SceneManager {
public:
virtual void AddEntity(Entity* entity) = 0;
virtual void UpdateScene(float deltaTime) = 0;
// ...
};

// Application Layer
class Application {
private:
Platform* platform;
ResourceManager* resourceManager;
Renderer* renderer;
SceneManager* sceneManager;

public:
void Run() {
platform->Initialize();
renderer->Initialize(platform);

// Main loop
while (running) {

548
 platform->ProcessEvents();
sceneManager->UpdateScene(deltaTime);
renderer->RenderScene(sceneManager->GetActiveScene());
}
}
};

1.2. Data-Oriented Design

Data-Oriented Design (DOD) focuses on organizing data for efficient processing, rather than
organizing code around objects.

[Data-Oriented Design Diagram] | ../../../images/data_oriented_design_diagram.svg

1.2.1. Key Concepts

1. Data Layout - Organizing data for cache-friendly access patterns.

2. Systems - Process data in bulk, often using SIMD instructions.

3. Entity-Component-System (ECS) - A common implementation of DOD principles.

1.2.2. Benefits of Data-Oriented Design

• Better cache utilization

• More efficient memory usage

• Easier to parallelize

• Can lead to significant performance improvements

1.2.3. Implementation Example

// A simple ECS implementation

struct TransformData {
std::vector<glm::vec3> positions;
std::vector<glm::quat> rotations;
std::vector<glm::vec3> scales;
};

struct RenderData {
std::vector<Mesh*> meshes;
std::vector<Material*> materials;
};

class TransformSystem {
private:
TransformData& transformData;

public:

549
 TransformSystem(TransformData& data) : transformData(data) {}

void Update(float deltaTime) {

// Process all transforms in bulk
for (size_t i = 0; i < [Link](); ++i) {
// Update transforms
}
}
};

class RenderSystem {
private:
RenderData& renderData;
TransformData& transformData;

public:
RenderSystem(RenderData& rData, TransformData& tData)
: renderData(rData), transformData(tData) {}

void Render() {
// Render all entities in bulk
for (size_t i = 0; i < [Link](); ++i) {
// Render mesh with transform
}
}
};

1.3. Service Locator Pattern

The Service Locator pattern provides a global point of access to services without coupling
consumers to concrete implementations.

[Service Locator Pattern Diagram] | ../../../images/service_locator_pattern_diagram.svg

1.3.1. Key Concepts

1. Service Interface - Defines the contract for a service.

2. Service Provider - Implements the service interface.

3. Service Locator - Provides access to services.

1.3.2. Benefits of Service Locator Pattern

• Decouples service consumers from service providers

• Allows for easy service replacement

• Facilitates testing with mock services

550
1.3.3. Implementation Example

// Audio service interface

class IAudioService {
public:
virtual ~IAudioService() = default;
virtual void PlaySound(const std::string& soundName) = 0;
virtual void StopSound(const std::string& soundName) = 0;
};

// Concrete audio service

class OpenALAudioService : public IAudioService {
public:
void PlaySound(const std::string& soundName) override {
// Implementation using OpenAL
}

void StopSound(const std::string& soundName) override {

// Implementation using OpenAL
}
};

// Service locator
class ServiceLocator {
private:
static IAudioService* audioService;
static IAudioService nullAudioService; // Default null service

public:
static void Initialize() {
audioService = &nullAudioService;
}

static IAudioService& GetAudioService() {

return *audioService;
}

static void ProvideAudioService(IAudioService* service) {

if (service == nullptr) {
audioService = &nullAudioService;
} else {
audioService = service;
}
}
};

// Usage example
void PlayGameSound() {
ServiceLocator::GetAudioService().PlaySound("explosion");
}

551
1.4. Comparative Analysis of Architectural Patterns
Below is a comparative analysis of the architectural patterns discussed in this appendix:

Pattern Strengths Weaknesses Best Used For

Layered Architecture • Clear separation of • Can lead to "layer • Smaller engines

concerns bloat"
• Educational
• Easy to understand • May introduce projects
unnecessary
• Good for beginners • When clarity is
indirection
more important
• Potential than performance
performance
overhead from
layer traversal

Component-Based • Highly flexible and • More complex to • Modern rendering

Architecture modular implement initially engines

• Promotes code • Can be harder to • Systems with

reuse debug diverse object types

• Avoids deep • Potential • Projects requiring

inheritance performance frequent extension
hierarchies overhead from
component lookups
• Easier to extend
with new features

Data-Oriented Design • Excellent • Less intuitive than • Performance-

performance OOP critical systems

• Cache-friendly • Steeper learning • Mobile platforms

memory access curve
• Systems processing
• Good for parallel • Can make code large amounts of
processing harder to read similar data

Service Locator Pattern • Decouples service • Can hide • Cross-cutting

providers from dependencies concerns
consumers
• Potential for • Systems requiring
• Facilitates testing runtime errors runtime
configuration
• Allows runtime • Global state
service swapping concerns • When loose
coupling is critical

2. Advanced Rendering Techniques

This section provides an overview of advanced rendering techniques commonly used in modern
rendering engines. For more comprehensive information, refer to these excellent resources:

552
 • Physically Based Rendering: From Theory to Implementation - [Link]

• Real-Time Rendering - [Link]

• GPU Gems series - [Link]

2.1. Deferred Rendering

Deferred rendering separates the geometry and lighting calculations into separate passes, which
can be more efficient for scenes with many lights:

1. Geometry Pass - Render scene geometry to G-buffer textures (position, normal, albedo, etc.).

2. Lighting Pass - Apply lighting calculations using G-buffer textures.

2.2. Forward+ Rendering

Forward+ (or tiled forward) rendering combines the simplicity of forward rendering with some of
the efficiency benefits of deferred rendering:

1. Light Culling Pass - Divide the screen into tiles and determine which lights affect each tile.

2. Forward Rendering Pass - Render scene geometry with only the lights that affect each tile.

2.3. Physically Based Rendering (PBR)

PBR aims to create more realistic materials by simulating how light interacts with surfaces in the
real world:

1. Material Parameters - Define materials using physically meaningful parameters (albedo,

metalness, roughness, etc.).

2. BRDF - Use a physically based bidirectional reflectance distribution function.

3. Image-Based Lighting - Use environment maps for ambient lighting.

2.4. Advanced Camera Techniques

This section covers advanced techniques for implementing sophisticated camera systems in 3D
applications:

• Camera Collision: Implement a collision volume for the camera to prevent it from passing
through walls

• Context-Aware Positioning: Adjust camera position based on the environment (e.g., zoom out
in large open areas, zoom in in tight spaces)

• Intelligent Framing: Adjust the camera to keep both the character and important objects in
frame

• Predictive Following: Anticipate character movement to reduce camera lag

• Camera Obstruction Transparency: Make objects that obstruct the view partially transparent

553
 • Dynamic Field of View: Adjust the FOV based on movement speed or environmental context

3. Conclusion
These architectural patterns and rendering techniques provide a foundation for designing your
rendering engine. In practice, most engines use a combination of these patterns to address different
aspects of the system.

When designing your engine architecture, consider:

1. Performance Requirements - Different patterns have different performance characteristics.

2. Flexibility Needs - How much flexibility do you need for future extensions?

3. Team Size and Experience - More complex architectures may be harder to work with for
smaller teams.

4. Project Scope - A small project may not need the complexity of a full ECS.

Back to Architectural Patterns Back to Rendering Pipeline

554