From Junior to Senior

The Missing Pieces: Full-Stack, System Design & LLM/AI Integration

This extends your Production Engineering Reference (API styles, Redis, DB layer, languages,
testing). That doc assumes you already think like a senior engineer. This one is the bridge: you
know basic DSA, basic web dev, basic backend — this fills in why things are structured the way
they are, how LLM/AI fits into a real stack, what data structures actually show up in production
code, and how to organize folders so a codebase survives more than 3 months.
A note on accuracy: the architectural patterns and tradeoffs below reflect general, widely-used
industry practice as I understand it. Specific tool comparisons, pricing, and “current best
practice” in the LLM tooling space in particular move fast — verify against current vendor docs
before relying on those specifics.
Table of Contents
TOC \h \o "1-3"
1. The Mental Model You're Missing
A junior thinks: “user clicks button → my function runs → response comes back.” A senior thinks
in layers, each with a failure mode and a tradeoff:
Client (React/mobile)
→ CDN/Edge (static assets, sometimes edge functions)
→ Load Balancer (round robin / least-conn / consistent hash)
→ API Gateway / BFF (auth, rate limit, routing, response shaping)
→ Service layer (business logic — REST/gRPC/GraphQL internally)
→ Cache layer (Redis — checked before DB)
→ Data layer (Postgres = truth, OpenSearch = search, S3 = blobs, Vector DB =
embeddings)
→ Async layer (Kafka/SQS — anything that doesn't need to block the response)
→ Observability (every layer above emits logs/metrics/traces to this)

The senior-level skill isn't knowing every box — it's knowing which box should own a given
piece of state, and what breaks if you put it in the wrong one. Example junior mistake: storing a
user's shopping cart in application memory (lost on every restart/deploy). Senior fix: Redis with
TTL, because cart data is semi-durable but disposable, and a DB write on every “add to cart”
click is wasteful.

2. Folder Structures — What Goes Where and Why

2.1 Single Backend Service (Node/TS example — applies similarly to
Go/Python/Java)
service-name/
├── src/
│ ├── api/ # HTTP/gRPC entrypoints — thin, no business logic
│ │ ├── routes/
│ │ └── grpc/
│ ├── domain/ # Core business logic — pure functions, no framework
imports
│ │ └── student/
│ │ ├── [Link]
│ │ ├── [Link]
│ │ └── [Link]
│ ├── infra/ # Everything that talks to the outside world
│ │ ├── db/ # repositories, migrations
│ │ ├── cache/ # redis client wrapper
│ │ ├── queue/ # kafka/sqs producer-consumer
│ │ └── external/ # 3rd-party API clients (incl. LLM providers)
│ ├── middleware/ # auth, rate-limit, error handling
│ └── config/ # env validation, feature flags
├── test/
│ ├── unit/
│ ├── integration/ # spins up Testcontainers (real Postgres/Redis)
│ └── e2e/
├── migrations/
├── Dockerfile
├── [Link] # local dev: db + redis + this service
└── .github/workflows/[Link]
Why this shape: api/ and domain/ are separated so business logic isn't tangled with HTTP
concerns — you can unit-test domain/ with zero mocks. infra/ is the only layer allowed to import
a DB driver or HTTP client; this is called the hexagonal / ports-and-adapters pattern. Juniors
usually put DB calls directly inside route handlers — works for a demo, becomes untestable at
50 endpoints.

2.2 Monorepo (multiple services + frontend, what most real companies run)
company-platform/
├── apps/
│ ├── web/ # React/[Link] frontend
│ ├── admin-portal/
│ └── mobile/
├── services/
│ ├── identity-svc/ # Go
│ ├── academics-svc/ # Node
│ ├── analytics-svc/ # Python
│ ├── llm-gateway-svc/ # NEW — see section 4 below
│ └── notification-svc/ # Go
├── packages/ # shared code — the actual reason monorepos exist
│ ├── proto/ # shared .proto contracts (gRPC)
│ ├── ui-components/
│ ├── tsconfig-base/
│ └── shared-types/
├── infra/
│ ├── terraform/
│ └── k8s/
└── [Link] (or [Link]) # build orchestration

When to use monorepo vs polyrepo: monorepo wins when services share types/contracts
heavily and one team owns most of them (atomic cross-service PRs, one CI pipeline). Polyrepo
wins when teams are large/independent and you want hard deploy boundaries — a bad commit
in one repo can't accidentally break another team's build.

2.3 An LLM-Integrated Service Specifically

llm-gateway-svc/
├── src/
│ ├── api/ # exposes a clean internal API to other services
│ ├── providers/ # adapter per LLM vendor (Anthropic, OpenAI, local)
│ │ ├── [Link]
│ │ └── [Link] # so swapping vendors = swapping one file
│ ├── prompts/ # versioned prompt templates, NOT inline strings in
code
│ │ └── [Link]
│ ├── rag/
│ │ ├── [Link]
│ │ ├── [Link]
│ │ └── [Link]
│ ├── tools/ # function-calling tool definitions
│ ├── guardrails/ # input/output validation, PII scrubbing
│ └── cache/ # semantic + exact-match response caching
└── eval/ # offline eval set — golden Q/A pairs, run in CI
└── [Link]

The eval/ folder is the thing juniors never build and seniors never ship without. More on this in
section 4.6.
3. Data Structures — Where They Actually Show Up (Not
LeetCode, Production)
DSA Concept Where It Lives in a Real System
Hash map Redis itself is essentially a distributed hash map; in-memory LRU
caches inside a service; deduplication sets
Heap / priority queue Job schedulers, grading queues, load-test result aggregation,
shortest-path routing
Trie Autocomplete/search-as-you-type, IP routing tables
Graph Recommendation engines, org charts, course-prerequisite chains
(Neo4j), dependency resolution
LRU cache Redis eviction policy; in-process hot-config caching; CDN cache
eviction
Bloom filter “Has this user seen this notification?” at scale without storing every
ID; used by CDNs/DBs to skip disk reads
Consistent hashing Load balancer routing; distributed cache sharding (adding/removing
a node doesn't invalidate everything)
B-tree / B+tree How Postgres indexes work under the hood — explains why “add an
index” sometimes doesn't help
Merkle tree Git's object model; verifying data integrity across distributed nodes
The senior insight: you rarely implement these from scratch in production — you recognize “oh,
this is a heap problem” and reach for PriorityQueue/heapq/a Redis ZSET (which is a skip-list-
backed sorted set — a production-grade ordered structure handed to you for free). Knowing
DSA in production is pattern-recognition, not implementation.

4. LLM / AI Integration — The Part Missing From the Base

Doc
4.1 The Core Decision: Prompting vs RAG vs Fine-Tuning vs Agents
Need Use Why
Model needs general behavior System prompt Cheapest, fastest to iterate, no infra
change
Model needs facts not in training RAG (Retrieval-Augmented Keeps facts out of model weights
data Generation) — update a doc, no retraining
Model needs to take actions Tool use / function calling Model emits structured intent; your
(APIs, DB, email) code executes it
Domain-specific style/format at Fine-tuning (rare for most Expensive, slow to iterate, easy to
scale, cost-sensitive teams) overfit — last resort
Multi-step tasks needing self- Agent (model → tool call → Use only when a single
correction observe → repeat) prompt+tools call can't do it
I'd treat “fine-tuning vs RAG” as the single most over-asked, under-needed decision in this
space — most teams reach for fine-tuning before they've maxed out a good RAG + prompt
setup. I'm fairly confident in that general pattern, but verify against current vendor guidance
since this is exactly the kind of best practice that shifts as tooling matures.

4.2 RAG Pipeline — Concretely

Document → Chunker → Embedder → Vector DB
↓
User query → Embedder → similarity search → top-K chunks → prompt template → LLM →
response

// rag/[Link] — simplified
async function retrieveContext(query: string, k = 5) {
const queryEmbedding = await [Link](query);
const results = await [Link]({
vector: queryEmbedding,
topK: k,
filter: { tenant_id: currentTenant }, // never skip tenant isolation
});
return [Link](r => [Link]);
}

async function answerWithRAG(query: string) {

const context = await retrieveContext(query);
const prompt = buildPrompt(STUDENT_QA_TEMPLATE, { context, query });
return [Link](prompt);
}

Chunking tradeoffs (this is where most RAG quality problems actually live, not the model):
• Too small (e.g. 100 tokens): loses surrounding context, retrieval gets fragments that
don't make sense alone.
• Too large (e.g. 2000 tokens): wastes context window, dilutes relevance, fewer distinct
chunks fit in top-K.
A common middle ground people use is a few hundred tokens per chunk with roughly 10–20%
overlap — but I'm not fully certain of an authoritative “correct” number; this varies by document
type and is worth A/B testing against your own eval set rather than trusting a default.
Vector DB choices: Pinecone, Weaviate, Qdrant, pgvector (Postgres extension), Chroma.
Senior-level take: if your scale is moderate and you already run Postgres, pgvector avoids
adding a whole new system to operate — only reach for a dedicated vector DB when you
actually hit its limits (very large corpora, hybrid search at scale, multi-region replication). Check
each vendor's current docs before picking; this space changes often.

4.3 Function/Tool Calling — How the Model Actually “Does Things”

const tools = [{
name: "get_attendance_rate",
description: "Fetch a student's attendance percentage for the current term",
input_schema: {
type: "object",
properties: { student_id: { type: "string" } },
required: ["student_id"],
},
}];
const response = await [Link]({ messages, tools });
if (response.stop_reason === "tool_use") {
const toolCall = [Link](b => [Link] === "tool_use");
const result = await executeToolSafely([Link], [Link]); // your code
runs this, not the model
// feed result back as a tool_result message, then call the model again
}

Critical production rule: the model never gets raw DB/API access. It emits a structured request
to call a tool; your backend validates the input, checks authorization (can this user query this
student's data?), executes, and only then returns the result. Skipping authorization here is the
#1 way “AI features” turn into data leaks.

4.4 Streaming Responses (Why Every Chat UI Streams Token-by-Token)

// SSE streaming from your backend to the browser
[Link]("/chat/stream", async (req, res) => {
[Link]("Content-Type", "text/event-stream");
const stream = await [Link]({ messages });
for await (const chunk of stream) {
[Link](`data: ${[Link]({ text: [Link] })}\n\n`);
}
[Link]();
});

Without streaming, the user stares at a spinner for the full generation time (can be 5–20+
seconds for long responses). Streaming gets the first token to the user in under a second and
dramatically improves perceived latency even though total time is the same.

4.5 Cost & Latency Tradeoffs (The Thing Juniors Never Budget For)
• Bigger/smarter models cost more per token and are slower — don't route every request
to your most expensive model. A common pattern is a router: a cheap/fast model
handles simple requests, escalates to a stronger model only when needed.
• Cache aggressively: exact-match caching (same prompt → same cached response) and
semantic caching (similar-enough prompt → reuse response) both meaningfully cut cost
on high-traffic endpoints.
• Token usage is the unit of cost — context stuffed into every prompt (long system
prompts, large RAG context) is recurring cost on every call. Pull current per-token pricing
from the provider's pricing page rather than relying on a remembered figure.

4.6 Evals — The Part Nobody Does Until Production Breaks

A junior ships a prompt that “looks good” on 3 manual tests. A senior maintains a golden eval
set: a few dozen to a few hundred representative input/expected-output pairs, run automatically
(ideally in CI) every time a prompt or model changes, scored by exact/fuzzy match, a rubric, or
an LLM-as-judge call. This is what catches “I improved the prompt for case X and silently broke
case Y” before a user does.

4.7 Guardrails
• Input: strip/flag prompt-injection attempts, validate structure before it ever reaches the
model.
 • Output: validate structured outputs against a schema before acting on them — never
trust a parsed response blindly, wrap parsing in try/catch and validate shape.
• PII: scrub or redact sensitive fields before they go into logs, and be deliberate about
what's allowed to leave your infrastructure to a third-party model provider at all — this is
as much a compliance question as an engineering one, so check your org's data policy
rather than assuming.

5. System Design Concepts the Base Doc Assumes You

Already Know
5.1 CAP Theorem, Plainly
In a distributed system, when there's a network partition, you must choose: stay Consistent
(every read sees the latest write, but might fail/block) or stay Available (always respond, but
might serve stale data). Postgres (single primary) leans CP. Many NoSQL stores (Cassandra,
DynamoDB) lean AP by default but are tunable. This is why “just use a distributed DB” isn't a
free upgrade — you're trading away a guarantee.

5.2 Message Queues — Kafka vs RabbitMQ vs SQS

Best For Why
Kafka High-throughput event streams, multiple Log-based, retains
consumers replaying history messages, built for fan-out
at scale
RabbitMQ Task queues, complex routing, lower latency Simpler mental model, great
per message for “do this job once”
SQS (managed You don't want to operate the queue infra Managed, scales
equivalents) yourself automatically, less control
over ordering/retention
Junior mistake: reaching for Kafka for a simple “send this email async” job — that's
RabbitMQ/SQS territory. Kafka's value is replayable event history and multi-consumer fan-out
(e.g., one “attendance marked” event consumed independently by Notification, Analytics, and
Audit services).

5.3 Horizontal vs Vertical Scaling

Vertical = bigger machine (simpler, has a ceiling, single point of failure). Horizontal = more
machines (requires statelessness — anything sticky like in-memory sessions breaks this unless
you externalize that state to Redis). Production systems default to horizontal scaling for the
service layer and accept that the database is the harder thing to horizontally scale (read
replicas, sharding, or managed solutions).

5.4 Deployment Strategies

Strategy How Risk Profile
Rolling Replace instances gradually Brief mixed-version window
 Strategy How Risk Profile
Blue-green Two full environments, switch traffic at once Instant rollback, costs 2x infra
during cutover
Canary Route a small % of traffic to new version first Catches bugs before full
exposure, more complex
routing
This connects directly to the base doc's mention of Argo Rollouts/Flagger/LaunchDarkly —
those tools implement canary/progressive delivery for you rather than you hand-rolling traffic
splitting.

6. Observability — Logging, Metrics, Tracing

The thing that tells you why production broke.
• Logs: structured (JSON), not [Link](“here”). Every log line should be filterable by
request ID, service, severity.
• Metrics: numeric time series — request rate, error rate, latency percentiles (p50/p95/p99
— averages hide the bad tail). Commonly Prometheus + Grafana.
• Tracing: follows one request across every service it touches (gateway → gRPC call →
DB query → cache check), so you can see exactly which hop added the latency. Tools:
OpenTelemetry, Jaeger, Datadog APM.
A senior debugging a “the app is slow” report doesn't guess — they pull a trace for a slow
request and look at which span took the time.

7. Security Basics You're Expected to Already Apply

• Never trust client input — validate/sanitize at the API boundary, every time, even for
“internal” services.
• Auth vs authz: authentication (who are you) is separate from authorization (what are you
allowed to do) — check both, every request, not just at login.
• Secrets live in a secrets manager (Vault, AWS Secrets Manager, etc.), never in code
or .env files committed to git.
• Rate limit and input-validate any endpoint that calls an LLM — it's both a cost-control
and abuse-prevention measure (see base doc's Redis rate-limiting example).
• Least privilege everywhere — a service's DB user should only have permissions for the
tables it actually needs.

8. The Actual Junior → Senior Checklist

1. Can explain why a piece of state lives where it lives (DB vs cache vs queue vs in-
memory), not just that it works.
2. Writes code with business logic separated from framework/IO code, so it's testable
without spinning up a real server.
3. Never lets an LLM call a side-effecting action without a validation + authorization layer in
between.
 4. Has an opinion on tradeoffs (REST vs gRPC vs GraphQL, SQL vs NoSQL, fine-tune vs
RAG) backed by the actual constraint of the problem, not by what's trendy.
5. Builds for the failure case first — what happens when Redis is down, when the LLM call
times out, when a partition happens — not just the happy path.
6. Has logs/metrics/traces in place before something breaks, not added reactively after an
incident.
7. Treats prompts and infra config the same way as code — versioned, reviewed, tested
against a golden set or test suite.
8. Knows the cost of what they're shipping (token cost, infra cost, query cost) and chooses
the cheapest option that meets the bar, not the most powerful one available.

A note on this document, per a preference for flagged uncertainty: the architectural patterns and
tradeoffs above reflect general, widely-used industry practice as I understand it, but specific tool
comparisons, pricing, and “current best practice” in the LLM tooling space in particular should
be verified against current vendor docs — that ecosystem moves faster than most of the rest of
this stack.