Production Engineering Reference

API styles, caching, databases, languages — what to use, when, and why

Audience: backend / platform engineers making real architecture decisions.

Covers: REST vs gRPC vs GraphQL (with code), Redis caching patterns, the database layer (SQL/NoSQL/cache/search),
language tradeoffs across the stack, and a testing layer — each with concrete examples and "use this when / not this when"
guidance.

1. API Styles in Production: REST vs gRPC vs GraphQL

2. gRPC In Depth — Protobuf, Streaming, Code Example
3. GraphQL In Depth — Schema, Resolvers, N+1, Code Example
4. Redis In Depth — Patterns, Data Structures, Pitfalls
5. The Database Layer — SQL, NoSQL, Search, Object Storage
6. Language Tradeoffs Across the Stack (with examples)
7. Testing Layer — Tools per Language, Pyramid
8. Decision Cheat-Sheets

Production Engineering Reference — Internal Dev Docs Page 1

 1. API Styles in Production: REST vs gRPC vs GraphQL
These three solve the same basic problem — letting clients/services call your backend — but optimize for different things.
Production systems often run all three simultaneously, each at the layer it fits.

REST (JSON/HTTP) gRPC (Protobuf/HTTP2) GraphQL

Best for Public APIs, simple CRUD, Service-to-service (internal Aggregating many sources for a
browser-friendly microservices) flexible frontend

Payload format JSON (text, verbose) Protobuf (binary, compact) JSON, but client picks the shape

Transport HTTP/1.1 or 2 HTTP/2 (mandatory) Usually HTTP/1.1 POST

Typing Loose (OpenAPI optional) Strong, code-generated from Strong, schema-first

.proto

Streaming Limited (SSE/websockets Native (unary, server, client, bidi Subscriptions (via websockets,
bolted on) streams) bolted on)

Browser Native Needs grpc-web proxy (Envoy) Native

support

Caching Easy (HTTP caching, CDNs) Hard (binary, point-to-point) Hard (single endpoint, custom
caching needed)

Tooling maturity Universal Excellent in backend/infra, weaker Strong, growing

for browsers

Where each one actually shows up in the architecture from your diagram
• Clients → API Gateway/BFF: REST or GraphQL — browsers and mobile apps need something that works over plain
HTTP/1.1 without extra proxies, and GraphQL specifically helps the BFF avoid over-fetching when the web app, mobile app,
and admin portal each need different shapes of the same data.
• API Gateway/BFF → Microservices (Identity, Academics, Scheduling, Finance, Examination, Attendance,
Notification, Analytics): gRPC — this is pure internal service-to-service traffic, so binary Protobuf + HTTP/2 multiplexing
gives lower latency and higher throughput than JSON/REST, and the .proto contract keeps every service's interface strongly
typed and version-checked at build time.
• Public/partner-facing API (if VivekLab ERP ever exposes one): REST — third parties expect REST + OpenAPI; it is the
lowest-friction option for external integrators.

Rule of thumb: REST at the edge (talking to humans and third parties), gRPC in the middle (talking to your own services), GraphQL
when one endpoint needs to flexibly aggregate many backends for a UI that changes a lot.

Production Engineering Reference — Internal Dev Docs Page 2

 2. gRPC In Depth
2.1 What it actually is
gRPC is an RPC framework from Google built on HTTP/2 and Protocol Buffers (Protobuf). You define a service contract
once in a .proto file, and gRPC generates client + server code in every supported language from that single source of truth
— so a Go service and a Node service can call each other with full type safety and neither has to hand-write a client SDK.

2.2 The four call types

Type Shape Real example

Unary 1 request → 1 response GetStudentById(id) → Student

Server streaming 1 request → stream of StreamAttendanceUpdates(classId) → stream of

responses AttendanceEvent, as they happen

Client streaming stream of requests → 1 UploadExamSheetsBatch(stream Sheet) →

response UploadSummary, client pushes many files, server acks
once

Bidirectional streaming stream ↔ stream, both ways, LiveNotificationChannel(stream Ack) ↔ stream Notification
concurrently — used for the Notification service pushing live alerts

2.3 Example: defining a service contract

// [Link]
syntax = "proto3";
package [Link].v1;

service AttendanceService {
rpc MarkAttendance (MarkAttendanceRequest) returns (MarkAttendanceResponse);
rpc StreamClassAttendance (ClassRequest) returns (stream AttendanceEvent);
}

message MarkAttendanceRequest {
string student_id = 1;
string class_id = 2;
bool present = 3;
int64 timestamp = 4;
}

message MarkAttendanceResponse {
bool success = 1;
string message = 2;
}

message ClassRequest { string class_id = 1; }

message AttendanceEvent {
string student_id = 1;
bool present = 2;
int64 timestamp = 3;
}

2.4 Server implementation (Go) — matches the diagram's Attendance Node (Go)
func (s *attendanceServer) MarkAttendance(ctx [Link],
req *[Link]) (*[Link], error) {

if err := [Link](ctx, [Link], [Link], [Link]); err != nil {

return nil, [Link]([Link], "failed to save attendance: %v", err)
}
[Link]("[Link]", req) // -> Kafka, fans out to Notification svc
return &[Link]{Success: true}, nil
}

Production Engineering Reference — Internal Dev Docs Page 3

 2.5 Client call (Node/TypeScript) — from the API Gateway/BFF
import { AttendanceServiceClient } from "./generated/attendance_grpc_pb";
import { MarkAttendanceRequest } from "./generated/attendance_pb";

const client = new AttendanceServiceClient("attendance-svc:50051", [Link]());

const req = new MarkAttendanceRequest();

[Link]("S1029");
[Link]("10A");
[Link](true);

[Link](req, (err, response) => {

if (err) return handleGrpcError(err);
[Link]([Link]());
});

2.6 When to use gRPC / when not to

Use gRPC when Avoid gRPC when

Service-to-service calls inside your own cluster Calling directly from a browser (needs grpc-web + Envoy
proxy — extra moving part)

You need streaming (live attendance feed, chat, telemetry) Your consumers are third-party partners expecting plain
REST/JSON

You want compile-time-checked contracts across many The API needs to be human-browsable/debuggable
languages/teams casually (curl-friendly REST wins here)

Latency and payload size matter (binary Protobuf is far Team has no protobuf tooling experience and the deadline
smaller than JSON) is tight

Production Engineering Reference — Internal Dev Docs Page 4

 3. GraphQL In Depth
3.1 What it actually is
GraphQL is a query language for APIs plus a runtime for executing those queries against a schema. Instead of many fixed
REST endpoints (/students/:id, /students/:id/grades, /students/:id/attendance), the client sends one
query describing exactly the fields it needs, and the server resolves each field — often by fanning out to multiple
microservices in parallel.

3.2 Schema example (matches Academics + Examination + Attendance services)

type Student {
id: ID!
name: String!
grades: [Grade!]! # resolved by Academics service
examResults: [ExamResult!]! # resolved by Examination service
attendanceRate: Float! # resolved by Attendance service
}

type Grade { subject: String!, score: Float! }

type ExamResult { examName: String!, score: Float!, maxScore: Float! }

type Query {
student(id: ID!): Student
}

3.3 Resolver example (Node/TypeScript, in the BFF)

const resolvers = {
Query: {
student: async (_, { id }, { dataSources }) => [Link](id),
},
Student: {
grades: (student, _, { dataSources }) => [Link]([Link]),
examResults: (student, _, { dataSources }) => [Link]([Link]),
attendanceRate: (student, _, { dataSources }) => [Link]([Link]),
},
};
// One GraphQL query to the BFF fans out to 3 backend gRPC calls in parallel,
// instead of the mobile app making 3 separate REST round-trips.

3.4 Client query — exactly what the Admin portal asks for, nothing more
query StudentSummary($id: ID!) {
student(id: $id) {
name
attendanceRate
grades { subject score }
}
}
# Notice: examResults is NOT requested here, so that resolver never even runs.
# A REST endpoint would have returned it anyway (over-fetching) unless you
# built a bespoke /summary endpoint just for this one screen.

3.5 The N+1 problem — GraphQL's most common production bug

If `grades` is resolved per-student with a separate DB call, and you query 100 students at once, naive resolvers fire 100 separate
queries (N+1) instead of 1 batched query. Fix this with a DataLoader that batches and caches calls within a single request tick.
const gradesLoader = new DataLoader(async (studentIds) => {
const rows = await [Link](
'SELECT * FROM grades WHERE student_id = ANY($1)', [studentIds]
);
return [Link](id => [Link](r => r.student_id === id)); // batched, 1 query total
});

Production Engineering Reference — Internal Dev Docs Page 5

 3.6 When to use GraphQL / when not to
Use GraphQL when Avoid GraphQL when

Multiple very different clients (web, mobile, admin) need You have one simple client and one simple backend —
different shapes of the same data REST is less ceremony

You're aggregating data from many microservices for a Strong HTTP-level caching (CDN, browser cache) is the
single screen priority — GraphQL's single endpoint defeats this

Frontend teams need to iterate without waiting on new File uploads / binary-heavy operations (REST/multipart
backend endpoints handles these more naturally)

Reducing mobile over-fetching matters for bandwidth/battery Your team has no budget to manage schema evolution,
N+1 issues, and query-cost limiting

Production Engineering Reference — Internal Dev Docs Page 6

 4. Redis In Depth
Redis is an in-memory data store used for caching, session storage, queues, rate limiting, leaderboards, and pub/sub — not
a replacement for your system-of-record database. Treat anything in Redis as disposable: if it's lost, the system should
recover by recomputing or refetching from Postgres.

4.1 Data structures and what each is actually for

Structure Use Case Example

String Simple cache value, counters SET session:abc123 '{...user...}' EX 3600

Hash Object-like cache entry (partial field updates) HSET student:S1029 name 'Aritra' attendance_pct
92

List Simple FIFO queue, recent-activity feeds LPUSH notif:queue:user42 '{...}'

Set Unique membership checks (e.g. "has this SADD notice:seen:NTC55 user42
user seen this notice")

Sorted Set (ZSET) Leaderboards, rate limiting windows, priority ZADD exam:rank:EX10 92 'S1029'
queues

Stream Lightweight event log (simpler alternative to XADD attendance:stream * student S1029 present
Kafka for smaller volume) true

Pub/Sub Real-time fan-out (e.g. push live notification PUBLISH notifications:live '{...}'
to connected admin dashboards)

4.2 Caching patterns — pick the right one per use case
Pattern How it works Best for

Cache-aside (lazy load) App checks Redis first; on miss, reads DB, then Most read-heavy endpoints (student profile,
writes to Redis course catalog) — simplest, most common
pattern

Write-through App writes to Redis and DB at the same time, Data that must never be stale right after a
synchronously write (e.g. exam result just published)

Write-behind App writes to Redis immediately, DB updated High-write-volume, latency-critical paths

(write-back) asynchronously later where some risk of data loss is acceptable
(analytics counters)

TTL-based expiry Every key has an expiry; stale data self-heals Session tokens, rate-limit counters,
anything time-bound by nature

4.3 Code example: cache-aside for a hot read endpoint

async function getStudentProfile(studentId: string) {
const cacheKey = `student:profile:${studentId}`;
const cached = await [Link](cacheKey);
if (cached) return [Link](cached); // cache HIT

const student = await [Link]( // cache MISS -> hit Postgres

'SELECT * FROM students WHERE id = $1', [studentId]
);
await [Link](cacheKey, [Link](student), 'EX', 300); // cache for 5 min
return student;
}

// Invalidate on write, don't wait for TTL, to avoid serving stale data:
async function updateStudentProfile(studentId: string, data: object) {
await [Link]('UPDATE students SET ... WHERE id = $1', [studentId]);
await [Link](`student:profile:${studentId}`); // invalidate
}

Production Engineering Reference — Internal Dev Docs Page 7

 4.4 Code example: rate limiting the API Gateway with Redis
-- Lua script run atomically inside Redis (avoids race conditions)
local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])

local current = [Link]("INCR", key)

if current == 1 then
[Link]("EXPIRE", key, window)
end
if current > limit then
return 0 -- reject: too many requests
end
return 1 -- allow
-- Gateway calls this per-request with key = "ratelimit:" .. clientIP, limit=100, window=60

4.5 Common Redis pitfalls in production

• Treating Redis as a database: no durability guarantee by default (AOF/RDB persistence helps but it's not a transactional
system of record — keep Postgres as the source of truth).
• Cache stampede: when a hot key expires, thousands of requests simultaneously miss and hammer the DB at once —
mitigate with jittered TTLs or a "lock + recompute once" pattern.
• Unbounded key growth: always set a TTL or you'll silently fill memory until eviction (or OOM) kicks in.
• Big keys: storing huge JSON blobs in one key blocks other operations during access — shard into smaller hashes when
an object gets large.

Production Engineering Reference — Internal Dev Docs Page 8

 5. The Database Layer — SQL, NoSQL, Search, Object
Storage
Production systems don't use one database for everything — they use "polyglot persistence": the right storage engine per
access pattern, all feeding from/to the same event bus to stay consistent.

5.1 PostgreSQL — system of record

Used for Identity, Academics, Finance, Examination, Attendance — anywhere you need ACID transactions, foreign-key
integrity, and the ability to ask arbitrary relational questions later. "DB per service" (as in the diagram) means each
microservice owns its own schema/database and never reaches into another service's tables directly — cross-service reads
happen via gRPC calls or the event bus.
-- A finance transaction MUST be atomic: deduct fee balance and record payment together,
-- or neither happens. This is exactly what Postgres transactions guarantee and Redis cannot.
BEGIN;
UPDATE student_accounts SET balance = balance - 5000 WHERE student_id = 'S1029';
INSERT INTO payments (student_id, amount, method) VALUES ('S1029', 5000, 'upi');
COMMIT;

5.2 When you'd add a NoSQL store (not in the original diagram, but common)
Store Data shape Production use case

MongoDB / Flexible, nested JSON documents Storing arbitrary form submissions, audit logs with varying shape
DocumentDB per event type

DynamoDB / Key-value / wide-column, massive Append-only event history, IoT-style sensor data, very high write
Cassandra write scale throughput with simple access patterns

Neo4j Graph (nodes + relationships) Modeling complex relationships, e.g. course prerequisite chains,
org reporting structures

5.3 OpenSearch — full-text and log search

Postgres can do basic full-text search, but it doesn't scale well for fuzzy matching, relevance ranking, faceted filters, or
searching across millions of log lines. OpenSearch (or Elasticsearch) indexes data for exactly that: "find all students whose
name sounds like Aritra," or "find every ERROR log across all services in the last hour mentioning timeout."
GET /students/_search
{
"query": { "match": { "name": "aritra" } },
"highlight": { "fields": { "name": {} } }
}

5.4 S3 / Object storage

For anything that isn't structured/queryable data: exam sheet PDFs, profile photos, generated report cards, presentation
exports. Store the file in S3, store only the S3 key/URL in Postgres — never put binary blobs directly in a relational database
(bloats backups, kills query performance).

5.5 Choosing the right store — decision table

I need to... Use

Guarantee a transaction either fully happens or fully rolls back PostgreSQL

Cache a hot read to avoid hitting the DB on every request Redis

Search free-text or fuzzy-match across millions of records/logs OpenSearch

Store a PDF, image, or other binary file S3 / Object storage

Production Engineering Reference — Internal Dev Docs Page 9

 I need to... Use

Record a high-volume, append-only stream of events for later processing Kafka topic → (optionally)
Cassandra/DynamoDB

Store deeply nested, schema-flexible documents MongoDB

Production Engineering Reference — Internal Dev Docs Page 10

 6. Language Tradeoffs Across the Stack
No language is universally "best" — production teams pick per-service based on the property that matters most for that
workload. Below are the languages that actually show up in this architecture, with concrete tradeoffs.

TypeScript / [Link]
Strengths: Huge ecosystem (npm), same language as the React frontend (shared types/DTOs possible), excellent for
I/O-bound workloads (API gateway, BFF, CRUD services) due to its non-blocking event loop.

Weaknesses: Single-threaded by default for CPU-bound work (heavy computation blocks the event loop unless offloaded to
worker threads); runtime type errors still possible despite TS if discipline slips.

Used for in this architecture: API Gateway/BFF, Identity, Academics, Finance, Examination services.

Example win: A `/dashboard` BFF endpoint making 4 parallel downstream gRPC calls via [Link]() — Node's async
model handles this with minimal code and low memory overhead.

Go
Strengths: Goroutines make massive concurrency cheap (handle 10,000 simultaneous notification sends with one
process); fast cold start and tiny container images — ideal for Kubernetes autoscaling; compiles to a single static binary,
simple deploys.

Weaknesses: More verbose than Python/TS for everyday CRUD (no generics-heavy ORMs, more boilerplate error handling
`if err != nil`); smaller ecosystem for, e.g., ML or complex business-rule DSLs.

Used for in this architecture: Scheduling, Attendance, Notification services.

Example win: Notification service fanning out 50,000 push notifications concurrently with a worker-pool pattern, finishing in
seconds instead of minutes, using a fraction of the memory a thread-per-request model would need.

Python
Strengths: Best ecosystem on Earth for data/ML (pandas, NumPy, scikit-learn, PyTorch); extremely fast to prototype and
iterate; readable, lowers onboarding cost for analytics-focused hires.

Weaknesses: Slow at raw CPU-bound execution compared to Go/Java (GIL limits true multi-threading); not the first choice
for high-throughput low-latency services.

Used for in this architecture: Analytics service.

Example win: Computing per-class attendance trend predictions with pandas + scikit-learn in 20 lines of code — would take
far longer to hand-roll in Go.

Java / Kotlin (JVM)

Strengths: Extremely mature ecosystem for enterprise/transactional systems (Spring Boot); JVM JIT gives strong sustained
throughput for long-running services; best-in-class Kafka client libraries (Kafka Streams); strong static typing and tooling.

Weaknesses: Higher memory footprint and slower cold start than Go — less ideal for serverless/very elastic autoscaling;
more verbose/ceremony-heavy than Node or Python for simple services.

Used for in this architecture (common alternative choice): Could replace Node for Finance/Identity if the team wants
Spring Security's maturity for auth/payment flows, or for Kafka Streams consumers.

Example win: A Finance service using Spring's @Transactional + strong typing to make a half-applied payment transaction
structurally hard to ship.

Production Engineering Reference — Internal Dev Docs Page 11

 SQL / HCL / YAML (declarative, not general-purpose)
Strengths: Declarative languages describe *what* you want, not *how* — this is exactly right for queries (SQL),
infrastructure state (Terraform/HCL), and pipeline/deployment config (YAML), because it makes the system diff-able and
reviewable.

Weaknesses: Not Turing-complete by design (intentional) — anything genuinely procedural (loops with complex branching)
needs to drop into a real language (e.g. Terraform's `for_each`/modules only go so far before you need a wrapper script).

Used for in this architecture: PostgreSQL queries, Terraform (Platform Infrastructure), Kubernetes manifests, GitHub
Actions pipelines.

Example win: `terraform plan` showing exactly what infrastructure will change before it happens — impossible to get that
kind of safety from an imperative bash script doing the same job.

6.1 Side-by-side summary

Language Concurrency model Typical latency profile Ecosystem strength Container
cold-start

Node/TS Single-threaded event loop Low for I/O-bound, poor for Web/API tooling, npm Fast
+ libuv CPU-bound

Go Goroutines (M:N green Low, consistent under Cloud-native/infra Very fast

threads) concurrency tooling

Python GIL-limited threads, async Higher (interpreted) Data science/ML, Fast

via asyncio unmatched

Java/Kotlin OS threads + JVM thread Low after JIT warm-up Enterprise/transactional, Slow(er), JVM
pools Kafka startup

Production Engineering Reference — Internal Dev Docs Page 12

 7. Testing Layer — Tools per Language, and the Pyramid
Each test type catches a different bug class at a different cost. Production teams enforce this in CI: unit + contract tests gate
every PR; integration/E2E gate merges to main; performance/chaos/security gate releases; synthetic monitoring and
canaries run continuously after release.

Test Type Tools (real-world) Catches

Unit Jest/Vitest (TS), pytest (Python), go test Logic bugs in one function/class, isolated
(Go), JUnit5 (Java)

Contract Pact, Spring Cloud Contract Breaking API changes between services without spinning
up the whole system

Integration Testcontainers, Supertest Bugs in how a service talks to its real DB/Kafka/Redis

E2E / UI Playwright, Cypress Multi-service user-journey breakage (login, checkout, exam

submission)

Performance/load k6, Gatling, Locust N+1 queries, connection pool exhaustion under real traffic

Chaos Gremlin, Chaos Mesh Cascading failure when a dependency dies or network
degrades

Security OWASP ZAP, Snyk, Trivy, Semgrep Vulnerable deps, container CVEs, SQLi/XSS patterns

Synthetic monitoring Checkly, Datadog Synthetics Live production outages before users report them

Canary/progressive Argo Rollouts, Flagger, LaunchDarkly Bad deploys, limited to a small % of traffic before full rollout
delivery

Language Unit Integration/API E2E/UI Mocking

TS/Node Jest, Vitest Supertest, Testcontainers Playwright, Cypress MSW, Sinon

Go go test + testify Testcontainers-go Playwright (via API) gomock

Python pytest pytest + Testcontainers Playwright-python [Link],

responses

Java/Kotlin JUnit5, Kotest Spring Boot Test, Selenium, Playwright-java Mockito, WireMock
Testcontainers

Production Engineering Reference — Internal Dev Docs Page 13

 8. Decision Cheat-Sheets
8.1 API style
Scenario Pick

Browser/mobile client calling your backend directly REST or GraphQL

One microservice calling another inside the cluster gRPC

Live/streaming updates between services gRPC streaming

One UI screen needs flexible data from 4 different services GraphQL

Public/partner-facing API for external integrators REST + OpenAPI

8.2 Storage
Scenario Pick

Needs ACID transaction guarantees PostgreSQL

Hot, frequently-read, rarely-changing data Redis (cache-aside)

Free-text/fuzzy search across millions of rows OpenSearch

Binary files (PDFs, images) S3

Massive write throughput, simple key access DynamoDB/Cassandra

8.3 Language for a new service

Scenario Pick

CRUD-heavy, business logic, same team as frontend Node/TypeScript

High concurrency, low latency, fan-out jobs Go

Data processing, ML, analytics Python

Heavy transactional/enterprise logic, mature security needs Java/Kotlin

End of document.

Production Engineering Reference — Internal Dev Docs Page 14