Low-Level Platform &

High-Performance Concurrency
Go / C++ / Linux

15 In-Depth Technical Questions & Answers

Covering lock-free data structures, goroutines, epoll, memory management,
kernel internals, signals, cache locality, and production debugging.
 Table of Contents
01 Lock-Free vs. Mutex Synchronization

02 High-Throughput Event Ingestion to Redis (Go)

03 Memory Management in Long-Running Daemons

04 Linux Virtual Memory Layout & Page Faults

05 Debugging Memory Leaks & Race Conditions in Kubernetes

06 Cooperative vs. Preemptive Multitasking

07 epoll and Multiplexed I/O

08 Thread-Safe Initialization & Double-Checked Locking

09 Cache Lines, False Sharing & Memory Alignment

10 Linux Signals & Graceful Shutdown

11 C++ Thread Callbacks vs. C Function Pointers

12 malloc/brk vs. mmap in Linux

13 Context Switch Cost & Scalable Architecture

14 Linux Page Cache, O_DIRECT & Trade-offs

15 Thread Affinity & CPU Cache Locality

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 2

 QUESTION 01

Lock-Free vs. Mutex Synchronization

When to favor lock-free (std::atomic, lock-free queues):

• Ultra-low latency paths where even a brief mutex block is unacceptable (market data tickers, packet
forwarding)
• High read-to-write ratios — atomic loads are essentially free on x86 due to TSO memory model
• Single-producer / single-consumer queues — a classic lock-free ring buffer needs zero locks
• Progress guarantees matter — lock-free structures guarantee system-wide progress even if a thread is
preempted; mutexes do not

When to favor std::mutex / shared_mutex:

• Complex invariants spanning multiple variables (atomics can't protect compound state atomically without
CAS loops)
• Low-contention paths — a mutex is simpler, easier to audit, and fast when uncontested (~20ns uncontested
pthread_mutex_lock)
• Reader-writer locks (shared_mutex) when reads vastly outnumber writes and read sections are non-trivial
The hidden cost of lock-free: ABA problem, memory reclamation (hazard pointers or epoch-based reclamation),
and substantially harder correctness proofs.

Benchmarking lock contention:

// 1. Use perf to count mutex-related events
// perf stat -e lock_imbalance,cycles,context-switches ./your_binary

// 2. Instrument with rdtsc to measure lock wait time

uint64_t before = __rdtsc();
[Link]();
uint64_t after = __rdtsc();
lock_wait_cycles.fetch_add(after - before, std::memory_order_relaxed);

// 3. Use Google Benchmark to sweep thread counts

BENCHMARK(BM_Queue)->ThreadRange(1, 64)->UseRealTime();

Key metrics: lock wait cycles, cache misses on the mutex cacheline (perf c2c), and the throughput cliff — the
thread count where throughput stops scaling and inverts.

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 3

 QUESTION 02

High-Throughput Event Ingestion to Redis (Go)

package ingest

import (
"context"
"sync"
"time"
"[Link]/redis/go-redis/v9"
)

const (
batchSize = 4096
batchTimeout = 50 * [Link]
workerCount = 8
chanBuffer = 64 * 1024 // large buffer absorbs upstream bursts
)

type Event struct {

Key string
Data []byte
}

type Ingester struct {

ch chan Event
rdb *[Link]
wg [Link]
}

func (ing *Ingester) worker(ctx [Link]) {

defer [Link]()
batch := make([]Event, 0, batchSize) // pre-allocated, reused
ticker := [Link](batchTimeout)
defer [Link]()

flush := func() {
if len(batch) == 0 { return }
pipe := [Link]()
for _, ev := range batch {
[Link](ctx, [Link], [Link])
}
flushCtx, cancel := [Link](ctx, 200*[Link])
defer cancel()
[Link](flushCtx)
batch = batch[:0] // reset length, keep underlying array
}

for {
select {
case <-[Link]():
for { // drain remaining items before exit
select {
case ev := <-[Link]:
batch = append(batch, ev)
default:
flush(); return
}
}
case ev := <-[Link]:
batch = append(batch, ev)
if len(batch) >= batchSize { flush() }
case <-ticker.C:

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 4

 flush() // time-triggered flush for low-volume periods
}
}
}

// Ingest is non-blocking with backpressure signal

func (ing *Ingester) Ingest(ctx [Link], ev Event) error {
select {
case [Link] <- ev: return nil
case <-[Link](): return [Link]()
default:
[Link]()
return ErrBackpressure // caller retries with exponential backoff
}
}

Channel blockage strategy:

• Upstream gets ErrBackpressure and can retry with exponential backoff
• A circuit breaker trips after N consecutive Redis failures so workers don't pile up
• [Link] on each Redis pipeline prevents a slow Redis from blocking forever

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 5

 QUESTION 03

Memory Management in Long-Running Daemons

Sources of heap fragmentation:

• Frequent small allocations of varying sizes create holes the allocator can't reuse
• Go's GC compacts to some degree but stop-the-world pauses still inflate P99
• C++ malloc (ptmalloc2) is particularly prone to fragmentation under mixed allocation sizes

Go — [Link]:
var eventPool = [Link]{
New: func() any {
s := make([]byte, 0, 4096)
return &s
},
}

func processEvent() {
buf := [Link]().(*[]byte)
*buf = (*buf)[:0] // reset length, keep capacity
defer [Link](buf)
// use buf ...
}

[Link] objects are GC'd between GC cycles — it reduces allocation pressure, not live memory. For objects
that must survive GC, use a channel-based free list instead.

C++ — Arena Allocator:

class Arena {
std::vector<std::byte> buf_;
size_t offset_ = 0;
public:
explicit Arena(size_t capacity) : buf_(capacity) {}

void* alloc(size_t size, size_t align = alignof(std::max_align_t)) {

size_t aligned = (offset_ + align - 1) & ~(align - 1);
if (aligned + size > buf_.size()) throw std::bad_alloc{};
offset_ = aligned + size;
return buf_.data() + aligned;
}

void reset() { offset_ = 0; } // O(1) — no individual frees

};

thread_local Arena request_arena{2 * 1024 * 1024}; // 2MB per thread

Situation Tool

Short-lived objects with uniform lifetime Arena allocator

Reusable buffers in hot path [Link] / object pool

Reducing GC pressure in Go [Link] + avoid pointer-heavy structs

C++ high-frequency / packet processing Arena per request, tcmalloc globally

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 6

 QUESTION 04

Linux Virtual Memory Layout & Page Faults

High addresses
+-----------------------------+ 0xFFFF_FFFF_FFFF_FFFF
| Kernel space (unmapped) | (user processes cannot access)
+-----------------------------+ 0xFFFF_8000_0000_0000
| Stack (grows downward) |
| ... |
+-----------------------------+
| Memory-Mapped Region | mmap, shared libs, file mappings
+-----------------------------+
| Heap (grows upward) | malloc / brk
+-----------------------------+
| BSS (uninit globals) |
| Data (init globals) |
| Text (code, read-only) |
+-----------------------------+ 0x0000_0000_0040_0000
Low addresses

Page Fault lifecycle:

• CPU dereferences a virtual address not present in the TLB
• Hardware Page Table Walker traverses the 4-level page table (PML4 → PDPT → PD → PT)
• If PTE is not present → CPU raises #PF exception, traps to kernel
• Kernel's do_page_fault() inspects the VMA for the address
• Minor fault: page exists in memory (e.g., stack growth, copy-on-write) → kernel maps it, returns. Cost:
~1µs
• Major fault: page must be fetched from disk → block I/O issued. Cost: ~10ms (devastating for P99)

Impact on high-throughput processing:

• Cold startup causes major faults on first access of large data structures — pre-fault with
madvise(MADV_WILLNEED) or mlock()
• Transparent Huge Pages (THP) reduce TLB pressure for large buffers (2MB pages vs 4KB) but can cause
latency spikes during compaction — disable for latency-sensitive services or use explicit mmap(...,
MAP_HUGETLB)
• Sequential access is TLB-friendly; pointer-chasing (linked lists, tree traversals) causes repeated TLB
misses

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 7

 QUESTION 05

Debugging Memory Leaks & Race Conditions in Kubernetes

Memory leak in a running pod (Go):

# Step 1: Confirm leak trajectory
kubectl top pod <pod> --containers # watch RSS grow over time

# Step 2: Expose pprof endpoint, then port-forward

kubectl port-forward pod/<pod> 6060:6060
go tool pprof [Link]

# Inside pprof interactive shell:

(pprof) top20 # top allocators by live bytes
(pprof) list <func> # annotated source for a specific function
(pprof) web # flame graph in browser (requires graphviz)

# Compare two heap snapshots to find growth:

curl localhost:6060/debug/pprof/heap > [Link]
# wait 60s
curl localhost:6060/debug/pprof/heap > [Link]
go tool pprof -diff_base [Link] [Link]

For C++ workloads:

# Valgrind (too slow for production; use in staging with reduced load)
valgrind --leak-check=full --track-origins=yes ./service

# AddressSanitizer (compile-time, faster alternative)

g++ -fsanitize=address,leak -g [Link] -o service
# Run normally — ASAN reports leaks on exit with full stack traces

# Heaptrack for production-closer profiling:

heaptrack ./service
heaptrack_gui [Link].*.gz

Race condition debugging:

# Go — race detector (2-20x slowdown, but catches all races)
go build -race ./...

# C++ — ThreadSanitizer
g++ -fsanitize=thread -g [Link] -o service
# Reports: "READ of size N by thread T1, WRITE by thread T2"

# GDB on a live pod

kubectl exec -it <pod> -- bash
gdb -p $(pgrep service)
(gdb) thread apply all bt # stack trace for all threads

Reading a pprof flame graph: Width = cumulative time/allocations in that call path. A wide flat top means that
function itself (not its callees) is the bottleneck. Narrow tall stacks indicate deep call chains.

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 8

 QUESTION 06

Cooperative vs. Preemptive Multitasking

Property Go Goroutines Linux OS Threads

Scheduler location User space (Go runtime) Kernel

Stack size 2KB initial, grows dynamically 8MB fixed default

Context switch cost ~100ns (no syscall) ~1-10µs (syscall + TLB flush)

Preemption mechanism Signal-based (Go 1.14+) + yield at Timer interrupt (~4ms quantum)
function calls

Scale Millions over GOMAXPROCS OS threads 1 OS thread per thread

The tight CPU loop problem:

// BAD: reduces scheduling responsiveness even post-Go 1.14
go func() {
for {
heavyComputation() // no function calls that yield
}
}()

// GOOD: yield explicitly in CPU-bound loops

go func() {
for {
heavyComputation()
[Link]() // explicit yield point
}
}()

For OS threads, a tight loop is preempted by the kernel timer interrupt regardless — but it monopolizes a CPU
core and causes cache pollution for other threads sharing the same physical core (SMT/Hyperthreading
contention).

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 9

 QUESTION 07

epoll and Multiplexed I/O

The C10K problem: 1 thread per connection means 100K connections = 100K threads = ~800GB stack memory
+ crushing context switch overhead. epoll solves this.
// Setup
int epfd = epoll_create1(EPOLL_CLOEXEC);

// Register a non-blocking socket

struct epoll_event ev = {
.events = EPOLLIN | EPOLLET, // Edge-triggered: notify only on state change
.[Link] = client_fd
};
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

// Event loop — single thread handles N connections

struct epoll_event events[MAX_EVENTS];
while (true) {
int n = epoll_wait(epfd, events, MAX_EVENTS, -1); // blocks until ready
for (int i = 0; i < n; i++) {
if (events[i].events & EPOLLIN) {
handle_read(events[i].[Link]); // guaranteed non-blocking
}
}
}

Mode Behavior Risk

Level-triggered (LT) epoll_wait keeps returning if data remains unread Repeated wakeups if drain is slow

Edge-triggered (ET) Notifies only on state change (new data arriving) Must read until EAGAIN or silent hang

Why it's fast: All 100K connections share one epoll_wait syscall. The kernel maintains a red-black tree of
watched fds and a ready list. When a packet arrives, the NIC interrupt handler marks the fd ready in O(1). Go's
net package, Nginx, [Link], and Redis all use epoll.

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 10

 QUESTION 08

Thread-Safe Initialization & Double-Checked Locking

The broken naive version:

// BROKEN on multi-core without memory barriers
Singleton* getInstance() {
if (instance == nullptr) { // Check 1 — no barrier
std::lock_guard lk(mtx);
if (instance == nullptr) { // Check 2
instance = new Singleton(); // CPU can reorder:
// 1. Allocate memory
// 2. Write pointer to `instance` <- reordered before constructor!
// 3. Call constructor
// Another thread sees non-null pointer to unconstructed object
}
}
return instance;
}

Fixed with std::atomic acquire/release:

std::atomic<Singleton*> instance{nullptr};
std::mutex mtx;

Singleton* getInstance() {
Singleton* p = [Link](std::memory_order_acquire);
if (p == nullptr) {
std::lock_guard lk(mtx);
p = [Link](std::memory_order_relaxed);
if (p == nullptr) {
p = new Singleton();
[Link](p, std::memory_order_release); // publish after construction
}
}
return p;
}

Best practice — static local (C++11) or [Link] (Go):

// C++: guaranteed thread-safe initialization since C++11
Singleton& getInstance() {
static Singleton instance; // zero overhead after first call
return instance;
}

// Go: [Link]
var (
instance *Service
once [Link]
)
func GetService() *Service {
[Link](func() { instance = &Service{} })
return instance
}

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 11

 QUESTION 09

Cache Lines, False Sharing & Memory Alignment

Modern CPUs load memory in 64-byte cache lines. If two threads write to different variables sharing a cache
line, every write invalidates the other CPU's cached copy — bouncing the line between cores via the MESI
protocol.

False sharing — broken example:

// BAD: counters[0] and counters[1] share a single cache line
// Thread 0 writes counters[0], Thread 1 writes counters[1]
// Result: cache line bounces between cores at full memory bandwidth cost
struct Counters {
std::atomic<uint64_t> counters[8]; // 64 bytes total — one cache line
};

Fixed with padding:

// C++: each counter on its own 64-byte cache line
struct alignas(64) PaddedCounter {
std::atomic<uint64_t> value;
char padding[64 - sizeof(std::atomic<uint64_t>)];
};
PaddedCounter per_thread_counters[MAX_THREADS];

// Go equivalent:
type PaddedCounter struct {
value uint64
_ [56]byte // pad to 64 bytes
}

Detection:
perf c2c (cache-to-cache) directly shows false sharing hotspots — it reports lines with high 'hitm' (hit in modified
state) counts, meaning one core read a line that another core had modified.

Structure type Strategy

Hot read-only config structs Pack tightly (no wasted cache lines)

Hot per-thread mutable counters One counter per cache line (padded)

Ring buffer head/tail pointers Head and tail on separate cache lines

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 12

 QUESTION 10

Linux Signals & Graceful Shutdown

Signal execution model:

• Signals are asynchronous — delivered by the kernel at arbitrary points in execution
• Only async-signal-safe functions can be called from signal handlers (write, _exit, sigaction — NOT malloc,
printf, mutex lock)
• Signals are per-process; pthread_kill targets a specific thread; kill() delivers to any thread

Graceful shutdown — Go:

func main() {
ctx, stop := [Link]([Link](),
[Link], [Link])
defer stop()

engine := NewEngine()
[Link](ctx)

<-[Link]() // block until signal received

[Link]("shutdown signal received")

shutdownCtx, cancel := [Link]([Link](), 30*[Link])

defer cancel()

if err := [Link](shutdownCtx); err != nil {

[Link]("forced shutdown after timeout: %v", err)
[Link](1)
}
[Link]("clean shutdown complete")
}

Graceful shutdown — C++ (self-pipe trick):

int shutdown_pipe[2];

void signal_handler(int sig) {

char c = 1;
write(shutdown_pipe[1], &c, 1); // async-signal-safe
}

int main() {
pipe2(shutdown_pipe, O_NONBLOCK | O_CLOEXEC);
signal(SIGTERM, signal_handler);

// Add shutdown_pipe[0] to epoll watch set

// On event: 1) stop accepting work
// 2) wait for in-flight transactions (with timeout)
// 3) flush write-ahead log / Redis pipeline
// 4) exit
}

Kubernetes context: SIGTERM is sent when a pod is evicted. terminationGracePeriodSeconds (default 30s) is
your window. After that, SIGKILL — no handler is possible.

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 13

 QUESTION 11

C++ Thread Callbacks vs. C Function Pointers

C function pointers — no closure state:

// Can only call free functions or static members
void on_event(void* userdata, int event_type) { /* ... */ }
register_callback(on_event, userdata); // userdata is manual "closure"

C++ std::function + lambdas — captures state (with lifetime danger):

class EventHandler {
std::string name_;
public:
void register_unsafe(EventLoop& loop) {
// DANGER: if EventHandler is destroyed before callback fires
loop.on_event([this](int ev) {
process(ev); // use-after-free if `this` is gone
});
}
};

Safe lifetime management with weak_ptr:

class EventHandler : public std::enable_shared_from_this<EventHandler> {
public:
void register_safe(EventLoop& loop) {
std::weak_ptr<EventHandler> weak = shared_from_this();
loop.on_event([weak](int ev) {
if (auto self = [Link]()) { // nullptr if already destroyed
self->process(ev);
}
// else: silently drop — no crash, no UB
});
}
};

Rules:
• Never capture raw this in callbacks that outlive the object
• Prefer weak_ptr → lock() pattern for async callbacks
• For std::thread with member functions: ensure object outlives the thread or use join() in destructor

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 14

 QUESTION 12

malloc/brk vs. mmap in Linux

Property brk / sbrk mmap (anonymous)

Mechanism Extends heap segment linearly Creates new VMA anywhere in address
space

Granularity Byte-level (kernel rounds to page) Page-aligned only

Return to OS Fragmentation prevents it — holes stay Each munmap returns pages immediately
mapped

Overhead Lower for small allocations Higher per-call (kernel VMA management)

Fragmentation High (long-lived heap) None (each mapping independent)

By default, allocations >= 128KB (M_MMAP_THRESHOLD) use mmap; smaller ones use the brk heap. Each
mmap allocation is independently returned to the OS on free() — preventing the high-water mark problem where
brk never shrinks.
// Override threshold for your workload
mallopt(M_MMAP_THRESHOLD, 64 * 1024); // mmap anything >= 64KB
mallopt(M_TRIM_THRESHOLD, 128 * 1024); // trim heap more aggressively

System overhead implications:

• Many mmap allocations → many VMAs → /proc/pid/maps grows → mmap_sem contention under
concurrent allocation
• Linux default VMA limit: 65536 (vm.max_map_count) — can be hit by JVMs or systems with many threads
+ large allocations
• jemalloc and tcmalloc use large mmap slabs internally, then sub-allocate — best of both worlds

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 15

 QUESTION 13

Context Switch Cost & Scalable Architecture

Context switch costs:

• Direct cost: save/restore CPU registers (~100-200 cycles), kernel entry/exit
• Indirect cost (dominant): TLB flush, L1/L2 cache eviction — next thread starts cold. A full cache miss
costs 200+ cycles vs 4 cycles for L1 hit
• Measured total: 1–10µs per switch depending on cache footprint

Architecture patterns to avoid thread proliferation:

Pattern 1: Event Loop + Worker Pool
Acceptor Thread -> epoll -> Event Queue -> Fixed Worker Pool (N = CPU cores)
N stays constant regardless of connection count

Pattern 2: Actor Model (Go channels / C++ message passing)

Each "actor" is a goroutine/fiber with a mailbox
Communication only via messages — no shared state, no locks
Runtime multiplexes millions of actors over O(CPU) OS threads

Pattern 3: Work Stealing (Go runtime, Intel TBB)

Each OS thread has a local run queue
Idle threads steal from busy threads' queues
Minimizes cross-thread communication while keeping all cores busy

Rule of thumb: Thread count = CPU cores for CPU-bound work. For I/O-bound: use async I/O (epoll +
coroutines), not more threads.

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 16

 QUESTION 14

Linux Page Cache, O_DIRECT & Trade-offs

Page cache behavior on writes:

• write() copies data into kernel page cache — returns immediately to application
• Kernel marks pages dirty and writes to disk asynchronously (pdflush/writeback)
• Subsequent reads of same data served from cache — no disk I/O
• fsync() forces dirty pages to disk synchronously before returning

O_DIRECT bypass:
// Requires: buffer aligned to 512B (or logical block size),
// size must be a multiple of block size
int fd = open("[Link]", O_WRONLY | O_DIRECT | O_CREAT, 0644);

void* buf;
posix_memalign(&buf, 512, 4096); // alignment is mandatory
write(fd, buf, 4096); // bypasses page cache, goes to disk

Page Cache (default) O_DIRECT

Write latency Low (write to RAM) High (wait for disk)

Read latency Low if cached Always hits disk

CPU overhead Extra copy (user -> kernel buffer) Zero-copy possible

Consistency Kernel manages coalescing Application controls I/O scheduling

Use case General workloads Databases with their own buffer pool

Why databases use O_DIRECT: Their buffer pool is the cache. Double-caching wastes RAM. They also want
precise control over fsync and write ordering for ACID guarantees (PostgreSQL, MySQL InnoDB).

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 17

 QUESTION 15

Thread Affinity & CPU Cache Locality

Without pinning, the OS scheduler may migrate threads between cores. Each migration cold-starts L1/L2
caches (~100 cycle penalty per warm cache line). For hot loops processing packet buffers, this can tank
throughput by 30-50%.
#include <sched.h>
#include <pthread.h>

void pin_thread_to_core(int core_id) {

cpu_set_t cpuset;
CPU_ZERO(&cpuset);
CPU_SET(core_id, &cpuset);

pthread_t thread = pthread_self();

pthread_setaffinity_np(thread, sizeof(cpu_set_t), &cpuset);
}

void* packet_worker(void* arg) {

int core = *(int*)arg;
pin_thread_to_core(core);
// Hot buffers, DMA memory, lookup tables now stay in L1/L2
while (true) { process_packet_batch(); }
}

NUMA-aware affinity:
#include <numa.h>

// Pin threads to cores on the same NUMA node as the NIC's memory
// Cross-NUMA memory access: ~100ns vs ~30ns local
int nic_numa_node = get_nic_numa_node();
// reads /sys/class/net/<iface>/device/numa_node
numa_run_on_node(nic_numa_node);
numa_set_preferred(nic_numa_node);

Isolating cores from the OS scheduler (grub cmdline):

GRUB_CMDLINE_LINUX="isolcpus=2,3,4,5 nohz_full=2,3,4,5 rcu_nocbs=2,3,4,5"
# Removes cores 2-5 from the general scheduler pool
# Used in DPDK networking, HFT, and real-time audio

Verification:
# Confirm thread is pinned
taskset -p <tid>
cat /proc/<pid>/status | grep Cpus_allowed

# Measure cache efficiency

perf stat -e L1-dcache-load-misses,LLC-load-misses ./service
# After pinning: L1 miss rate should drop significantly on hot loops

Low-Level Platform & High-Performance Concurrency — Go / C++ / Linux Page 18