■ End-to-End System Blueprint

Distributed Search Engine

& Web Crawler
C++ | Python | FastAPI | NLP | IR | System Design

Covers: Socket Programming · Multithreading · DBMS · Inverted Index · BM25 · PageRank · nDCG · Indian

Languages · Docker · Nginx · Hosting

Table of Contents
Chapter 1 System Architecture & Design Philosophy
Chapter 2 C++ Crawler — Socket, DNS, HTTP, Multithreading
Chapter 3 Python Processing Pipeline — NLP & Text Extraction
Chapter 4 Database Design — SQL + NoSQL Strategy
Chapter 5 Inverted Index & Information Retrieval
Chapter 6 Query Engine — BM25, PageRank, Snippets
Chapter 7 Indian Language Support & Multilingual NLP
Chapter 8 Evaluation — Offline & Online Metrics
Chapter 9 Frontend — FastAPI + Jinja2
Chapter 10 Deployment — Docker, Nginx & Hosting Hacks
Chapter 11 Full Project Checklist & Milestones

Distributed Search Engine Blueprint — Page 2

Chapter 1 — System Architecture & Design Philosophy

1.1 High-Level Architecture

The system is a microservices-based, distributed search engine that ingests web pages via a C++ crawler,
processes and indexes them with a Python NLP pipeline, stores indexes in a hybrid SQL/NoSQL setup, and
serves queries through a FastAPI backend with Jinja2 frontend — deployed on Docker behind Nginx.

Data Flow (End-to-End)

[ Seed URLs ]
|
v
[ URL Frontier ] <--- Redis Priority Queue (politeness + priority)
|
v
[ C++ Crawler Workers ] --- multithreaded, socket-based HTTP fetcher
| [Link] check, DNS resolution, retry logic
v
[ Raw HTML Store ] --- MinIO / local FS (content-addressed, SHA256 key)
|
v
[ Python Processing Pipeline ] (Kafka consumer)
|-- HTML parsing (BeautifulSoup / lxml)
|-- Language detection (langdetect / fastText)
|-- Tokenization + Stemming (NLTK, Stanza for Indian langs)
|-- NER, entity linking
|-- Link extraction -> back to URL Frontier
|
v
[ Inverted Index Builder ] --- partial indexes -> merge -> sharded posting lists
|
v
[ Index Shards ] --- PostgreSQL (metadata) + RocksDB/Redis (postings)
|
v
[ Query Engine ] --- BM25 + PageRank rerank + snippet gen
|
v
[ FastAPI Backend ] --- query parsing, caching (Redis), result serving
|
v
[ Jinja2 Frontend ] --- search UI, pagination, snippet display
|
v
[ Nginx Reverse Proxy ] --- SSL, load balance, rate limit

1.2 System Design Concepts Applied

Distributed Search Engine Blueprint — Page 3

 Concept Where Used Tool/Pattern

Load Balancing Crawler workers, Query replicas Nginx upstream / Round-robin

Caching Query results, DNS, [Link] Redis (TTL-based)

Message Queue Crawler → Processor pipeline Apache Kafka / RabbitMQ

Microservices Crawler, Indexer, Query, Frontend Docker Compose / K8s

Sharding Inverted index partitioning Hash sharding on term

Consistency URL dedup, Seen-set Redis Bloom Filter

Fault Tolerance Crawler retries, index replication Retry queues, replica sets

Autoscaling Crawler workers Docker Compose scale / K8s HPA

Reverse Proxy Entry point, SSL termination Nginx

CDN / Static Frontend assets Cloudflare / Nginx static

Rate Limiting Crawler politeness, API abuse Token bucket (Redis)

1.3 CAP Theorem Choices

The search engine deliberately trades strict consistency for availability and partition tolerance (AP system).
Stale index entries are acceptable; a 404 crawl result simply marks the URL for re-crawl. The URL frontier
(Redis) is CP — we must not double-crawl the same URL excessively.

Distributed Search Engine Blueprint — Page 4

Chapter 2 — C++ Crawler: Socket, DNS, HTTP,
Multithreading

2.1 Directory Structure

crawler/
■■■ [Link]
■■■ include/
■ ■■■ dns_resolver.hpp
■ ■■■ http_fetcher.hpp
■ ■■■ url_frontier.hpp
■ ■■■ robots_checker.hpp
■ ■■■ thread_pool.hpp
■ ■■■ bloom_filter.hpp
■■■ src/
■ ■■■ [Link]
■ ■■■ dns_resolver.cpp
■ ■■■ http_fetcher.cpp
■ ■■■ url_frontier.cpp
■ ■■■ robots_checker.cpp
■ ■■■ thread_pool.cpp
■■■ tests/
■■■ test_fetcher.cpp

2.2 DNS Resolution (C++)

Raw DNS lookups via getaddrinfo() with caching in an LRU map (TTL-aware). We cache resolved IPs per
hostname for the crawl duration, refreshing on TTL expiry.
// dns_resolver.hpp
#include <netdb.h>
#include <arpa/inet.h>
#include <unordered_map>
#include <chrono>

struct DNSEntry {
std::string ip;
std::chrono::steady_clock::time_point expiry;
};

class DNSResolver {
public:
std::string resolve(const std::string& host);
private:
std::unordered_map<std::string, DNSEntry> cache_;
std::mutex mtx_;
};

// dns_resolver.cpp

Distributed Search Engine Blueprint — Page 5

std::string DNSResolver::resolve(const std::string& host) {
std::lock_guard<std::mutex> lk(mtx_);
auto it = cache_.find(host);
if (it != cache_.end() &&
std::chrono::steady_clock::now() < it->[Link])
return it->[Link]; // cache hit

addrinfo hints{}, *res;

hints.ai_family = AF_INET;
hints.ai_socktype = SOCK_STREAM;
if (getaddrinfo(host.c_str(), nullptr, &hints, &res) != 0)
throw std::runtime_error('DNS failed: ' + host);

char buf[INET_ADDRSTRLEN];
inet_ntop(AF_INET,
&((sockaddr_in*)res->ai_addr)->sin_addr, buf, sizeof(buf));
freeaddrinfo(res);

cache_[host] = {buf, std::chrono::steady_clock::now()

+ std::chrono::seconds(300)};
return buf;
}

2.3 HTTP Fetcher via Raw Sockets

// http_fetcher.cpp (simplified; use OpenSSL for HTTPS)
std::string HTTPFetcher::fetch(const std::string& url, int timeout_ms) {
ParsedURL pu = parse_url(url); // scheme, host, path, port
std::string ip = dns_.resolve([Link]);

int fd = socket(AF_INET, SOCK_STREAM, 0);

sockaddr_in addr{};
addr.sin_family = AF_INET;
addr.sin_port = htons([Link]);
inet_pton(AF_INET, ip.c_str(), &addr.sin_addr);

// Non-blocking connect with select() for timeout

fcntl(fd, F_SETFL, O_NONBLOCK);
connect(fd, (sockaddr*)&addr, sizeof(addr));
fd_set wfds; FD_ZERO(&wfds); FD_SET(fd, &wfds);
timeval tv{timeout_ms/1000, (timeout_ms%1000)*1000};
if (select(fd+1, nullptr, &wfds, nullptr, &tv) <= 0)
throw std::runtime_error('Connect timeout');

// Send HTTP/1.1 request

std::string req = 'GET ' + [Link] + ' HTTP/1.1\r\n'
+ 'Host: ' + [Link] + '\r\n'
+ 'User-Agent: IndiaBot/1.0\r\n'
+ 'Connection: close\r\n\r\n';
send(fd, req.c_str(), [Link](), 0);

Distributed Search Engine Blueprint — Page 6

// Read response with buffer
std::string response;
char buf[4096];
ssize_t n;
while ((n = recv(fd, buf, sizeof(buf), 0)) > 0)
[Link](buf, n);
close(fd);
return response; // caller strips headers
}

2.4 Thread Pool & Concurrency

// thread_pool.hpp
#include <thread>
#include <queue>
#include <functional>
#include <condition_variable>

class ThreadPool {
public:
explicit ThreadPool(size_t n);
~ThreadPool();
void enqueue(std::function<void()> task);
private:
std::vector<std::thread> workers_;
std::queue<std::function<void()>> tasks_;
std::mutex mtx_;
std::condition_variable cv_;
bool stop_ = false;
};

// Constructor spawns N threads

ThreadPool::ThreadPool(size_t n) {
for (size_t i = 0; i < n; ++i)
workers_.emplace_back([this] {
for (;;) {
std::function<void()> task;
{
std::unique_lock<std::mutex> lk(mtx_);
cv_.wait(lk, [this]{ return stop_ || !tasks_.empty(); });
if (stop_ && tasks_.empty()) return;
task = std::move(tasks_.front());
tasks_.pop();
}
task();
}
});
}

// [Link] — 64 crawler threads

ThreadPool pool(64);

Distributed Search Engine Blueprint — Page 7

while (true) {
auto url = [Link](); // blocking Redis BLPOP
if ([Link](url)) continue; // already seen
[Link](url);
[Link]([url, &fetcher, &kafka_producer]() {
auto html = [Link](url, 5000);
kafka_producer.send('raw_pages', url, html);
});
}

2.5 [Link] Compliance & Politeness

Parse [Link] using a simple rule-matcher. Enforce per-domain delay (default 1 s) via a domain-keyed
timestamp map protected by a shared mutex. Use a Bloom Filter (murmur3 hash, 2 GB bit-array) for URL
deduplication — false-positive rate ~0.1% at 1 billion URLs.

Distributed Search Engine Blueprint — Page 8

Chapter 3 — Python Processing Pipeline

3.1 Pipeline Architecture

pipeline/
■■■ [Link] # Kafka consumer (raw_pages topic)
■■■ html_parser.py # lxml + BeautifulSoup
■■■ lang_detector.py # fastText [Link]
■■■ [Link] # NLTK/Stanza; IndicNLP for Indian scripts
■■■ [Link] # spaCy en_core_web_sm + custom NER
■■■ link_extractor.py # Outbound URLs -> URL Frontier
■■■ [Link] # OG tags, [Link], dates
■■■ index_emitter.py # Sends (term, docID, positions) to Indexer
■■■ models/
■■■ [Link] # fastText language ID model
■■■ hi_core.model # Stanza Hindi model

3.2 HTML Parsing & Text Cleaning

# html_parser.py
from [Link] import Cleaner
from bs4 import BeautifulSoup
import re

NOISE_TAGS = ['script','style','nav','footer','header','aside','form']

def extract_text(raw_html: str) -> dict:

soup = BeautifulSoup(raw_html, 'lxml')
for tag in soup(NOISE_TAGS):
[Link]()

title = [Link]() if [Link] else ''

meta_d = [Link]('meta', attrs={'name':'description'})
desc = meta_d['content'] if meta_d else ''

# Extract main content heuristically (largest text block)

body_text = ' '.join(soup.stripped_strings)
body_text = [Link](r'\s+', ' ', body_text).strip()

h1s = [h.get_text() for h in soup.find_all('h1')]

h2s = [h.get_text() for h in soup.find_all('h2')]

return {'title': title, 'description': desc,

'body': body_text, 'h1': h1s, 'h2': h2s}

3.3 Language Detection

# lang_detector.py
import fasttext
model = fasttext.load_model('models/[Link]')

Distributed Search Engine Blueprint — Page 9

def detect_lang(text: str) -> str:
# Returns ISO 639-1 code: 'en','hi','ta','te','ml','bn' etc.
label, conf = [Link](text[:500].replace('\n',' '), k=1)
lang = label[0].replace('__label__', '')
return lang if conf[0] > 0.7 else 'unknown'

3.4 Tokenization & Stemming

# [Link]
import nltk
from [Link] import SnowballStemmer
import stanza
from [Link] import indic_tokenize

STANZA_LANGS = {'hi','ta','te','ml','bn','mr','gu','pa','kn'}
stanza_pipelines = {}

def get_stanza(lang):
if lang not in stanza_pipelines:
stanza_pipelines[lang] = [Link](
lang=lang, processors='tokenize,lemma', use_gpu=False)
return stanza_pipelines[lang]

def tokenize(text: str, lang: str) -> list[str]:

if lang in STANZA_LANGS:
nlp = get_stanza(lang)
doc = nlp(text)
return [[Link] for s in [Link]
for w in [Link] if [Link]]
elif lang == 'en':
tokens = nltk.word_tokenize([Link]())
stemmer = SnowballStemmer('english')
return [[Link](t) for t in tokens
if [Link]() and t not in STOPWORDS]
else:
# Fallback: unicode-aware whitespace split
return indic_tokenize.trivial_tokenize(text, lang)

3.5 Named Entity Recognition

# [Link] — multilingual NER
import spacy
nlp_en = [Link]('en_core_web_sm')

def extract_entities(text: str, lang: str) -> list[dict]:

if lang == 'en':
doc = nlp_en(text[:10000]) # spaCy limit
return [{'text': [Link], 'label': e.label_}
for e in [Link]]
elif lang in STANZA_LANGS:
# Stanza NER for supported Indian langs
nlp = [Link](lang, processors='tokenize,ner')
doc = nlp(text[:5000])

Distributed Search Engine Blueprint — Page 10

return [{'text': [Link], 'label': [Link]}
for s in [Link] for e in [Link]]
return []

Distributed Search Engine Blueprint — Page 11

Chapter 4 — Database Design: SQL + NoSQL Strategy

4.1 Choice Rationale

Data Type Store Why

Document metadata (URL, title, desc,

PostgreSQL
lang, date) Relational, ACID, full-text indexing via tsvector

Inverted posting lists (term → docID,

RocksDB
positions)
(LSM-tree) Write-optimized, compressed, SST merges

URL frontier & seen-set Redis In-memory speed, BLPOP queue, Bloom Filter via RedisBloom

PageRank scores PostgreSQL (urls table) Joins with metadata for reranking

Raw HTML blobs MinIO (S3-compatible) Content-addressed, cheap object storage

Session / query cache Redis TTL-based result caching

Analytics & A/B logs ClickHouse Columnar OLAP for CTR, dwell-time analysis

4.2 PostgreSQL Schema

-- urls: master document table
CREATE TABLE urls (
id BIGSERIAL PRIMARY KEY,
url TEXT UNIQUE NOT NULL,
url_hash CHAR(64) NOT NULL, -- SHA256
domain TEXT NOT NULL,
lang CHAR(5) DEFAULT 'en',
title TEXT,
description TEXT,
crawled_at TIMESTAMPTZ,
pagerank DOUBLE PRECISION DEFAULT 0.0,
http_status SMALLINT,
content_type TEXT,
word_count INTEGER
);
CREATE INDEX idx_urls_domain ON urls(domain);
CREATE INDEX idx_urls_lang ON urls(lang);
CREATE INDEX idx_urls_rank ON urls(pagerank DESC);

-- links: for PageRank computation

CREATE TABLE links (
src_id BIGINT REFERENCES urls(id),
dst_id BIGINT REFERENCES urls(id),
PRIMARY KEY (src_id, dst_id)
);

-- entities: NER output

CREATE TABLE entities (
doc_id BIGINT REFERENCES urls(id),

Distributed Search Engine Blueprint — Page 12

text TEXT,
label TEXT, -- PERSON, ORG, LOC, GPE …
PRIMARY KEY (doc_id, text, label)
);

-- queries: search log for online eval

CREATE TABLE query_log (
id BIGSERIAL PRIMARY KEY,
query_text TEXT,
lang CHAR(5),
results JSONB,
clicked_pos INTEGER,
dwell_ms INTEGER,
session_id UUID,
ts TIMESTAMPTZ DEFAULT NOW()
);

4.3 RocksDB Posting List Format

# Key: b'term:en:run'
# Value: packed binary posting list
import struct

def encode_postings(postings: list[tuple]) -> bytes:

# postings = [(doc_id, tf, [pos1, pos2, ...]), ...]
buf = bytearray()
prev_id = 0
for doc_id, tf, positions in sorted(postings):
delta = doc_id - prev_id # delta encoding
prev_id = doc_id
buf += [Link]('>IH', delta, tf)
buf += [Link](f'>{len(positions)}H', *positions)
return bytes(buf)

# Compression: apply zstd on top

import zstandard as zstd
cctx = [Link](level=3)
compressed = [Link](encode_postings(postings))

Distributed Search Engine Blueprint — Page 13

Chapter 5 — Inverted Index & Information Retrieval

5.1 Inverted Index Structure

Term Dictionary (sorted): Posting Lists:
■■■■■■■■■■■■■■■■■■■■■■■■■■■ term 'india':
■ Term ■ Offset ■ ---------> [doc3:tf=5:pos[2,18,44]
■ 'india' ■ 0x00A1 ■ doc7:tf=2:pos[1,9]
■ 'search' ■ 0x00B3 ■ doc12:tf=8:pos[0,3,7,...]]
■ 'engine' ■ 0x00C7 ■
■■■■■■■■■■■■■■■■■■■■■■■■■■■

Field weights (used in BM25 variant):

title weight = 3.0
h1/h2 weight = 2.0
body weight = 1.0
URL tokens weight = 1.5
description weight = 1.2

5.2 Index Builder (Python)

# index_builder.py
import rocksdb
from collections import defaultdict

class SegmentWriter:
'''Builds in-memory partial index, flushes to RocksDB SST.'''
def __init__(self, shard_id: int):
self.shard_id = shard_id
[Link]: dict[str, list] = defaultdict(list)
self.doc_lengths: dict[int, int] = {}

def add_document(self, doc_id: int, tokens: list[str],

field: str, weight: float):
tf_map = defaultdict(list)
for pos, token in enumerate(tokens):
tf_map[token].append(pos)
self.doc_lengths[doc_id] = len(tokens)
for term, positions in tf_map.items():
[Link][term].append(
(doc_id, len(positions) * weight, positions))

def flush(self, db: [Link]):

batch = [Link]()
for term, posting_list in sorted([Link]()):
key = f'p:{self.shard_id}:{term}'.encode()
val = compress(encode_postings(posting_list))
[Link](key, val)
[Link](batch)
[Link]()

Distributed Search Engine Blueprint — Page 14

5.3 Index Sharding Strategy
# Hash sharding on term (consistent hashing)
NUM_SHARDS = 8

def get_shard(term: str) -> int:

import hashlib
return int(hashlib.md5([Link]()).hexdigest(), 16) % NUM_SHARDS

# Each shard is a separate RocksDB instance

# Can be deployed on different machines
# Query fan-out: broadcast to all shards, merge results

Distributed Search Engine Blueprint — Page 15

Chapter 6 — Query Engine: BM25, PageRank, Snippets

6.1 BM25 Scoring

# BM25 formula:
# score(D,Q) = SUM_t [ IDF(t) * (tf(t,D)*(k1+1)) / (tf(t,D) + k1*(1-b+b*|D|/avgdl)) ]

import math

class BM25:
K1 = 1.5 # term frequency saturation
B = 0.75 # length normalization

def __init__(self, N: int, avgdl: float):

self.N = N
[Link] = avgdl

def idf(self, df: int) -> float:

return [Link]((self.N - df + 0.5) / (df + 0.5) + 1)

def score(self, tf: float, df: int, doc_len: int) -> float:
idf = [Link](df)
norm = tf * (self.K1 + 1) / (
tf + self.K1 * (1 - self.B + self.B * doc_len / [Link]))
return idf * norm

def query_score(self, query_terms: list[str], doc_id: int,

postings_db, doc_meta) -> float:
total = 0.0
dl = doc_meta[doc_id]['word_count']
for term in query_terms:
postings = postings_db.get(term)
if not postings: continue
entry = [Link](doc_id)
if not entry: continue
total += [Link](entry['tf'], len(postings), dl)
return total

6.2 PageRank Computation

# [Link] — iterative computation on the links table
import psycopg2, numpy as np

def compute_pagerank(conn, damping=0.85, iterations=50):

cur = [Link]()
[Link]('SELECT COUNT(*) FROM urls')
N = [Link]()[0]
pr = [Link](N+1, 1.0/N) # initial uniform

# Build adjacency as {dst: [srcs]}

Distributed Search Engine Blueprint — Page 16

[Link]('SELECT src_id, dst_id FROM links')
in_links = {}
out_count = {}
for src, dst in [Link]():
in_links.setdefault(dst, []).append(src)
out_count[src] = out_count.get(src, 0) + 1

for _ in range(iterations):
new_pr = [Link](N+1, (1 - damping) / N)
for node, srcs in in_links.items():
new_pr[node] += damping * sum(
pr[s] / out_count.get(s, 1) for s in srcs)
pr = new_pr

# Write back
[Link](
'UPDATE urls SET pagerank=%s WHERE id=%s',
[(float(pr[i]), i) for i in range(1, N+1)])
[Link]()

6.3 Query Processing Pipeline

# query_engine.py
class QueryEngine:
def search(self, raw_query: str, lang: str = 'en',
top_k: int = 10) -> list[dict]:

# 1. Parse & normalize

q = [Link](raw_query) # AND/OR/phrase detection
tokens = tokenize([Link], lang) # same pipeline as indexing

# 2. Query expansion (WordNet synonyms for English)

expanded = [Link](tokens, lang)

# 3. Retrieve candidates from all shards (fan-out)

candidates = [Link](expanded) # {doc_id: bm25_score}

# 4. Rerank: combine BM25 + PageRank

for doc_id in candidates:
pr = [Link][doc_id]['pagerank']
candidates[doc_id] = 0.7 * candidates[doc_id] + 0.3 * pr

# 5. Sort, paginate
ranked = sorted(candidates, key=[Link], reverse=True)
top = ranked[:top_k]

# 6. Snippet generation
results = []
for doc_id in top:
meta = [Link][doc_id]
snippet = self.generate_snippet(doc_id, tokens)
[Link]({**meta, 'snippet': snippet,

Distributed Search Engine Blueprint — Page 17

'score': candidates[doc_id]})
return results

def generate_snippet(self, doc_id, query_terms,

window=40) -> str:
'''Extract best passage containing query terms.'''
body = self.raw_text[doc_id]
words = [Link]()
best_start, best_score = 0, 0
for i, w in enumerate(words):
score = sum(1 for t in query_terms if t in [Link]())
if score > best_score:
best_score, best_start = score, i
snippet = ' '.join(words[best_start:best_start+window])
return snippet + '...'

Distributed Search Engine Blueprint — Page 18

Chapter 7 — Indian Language Support & Multilingual NLP

7.1 Supported Languages

Language Script ISO Code Tokenizer Stemmer/Lemma

Hindi Devanagari hi Stanza / IndicNLP Stanza lemma

Tamil Tamil ta Stanza Stanza lemma

Telugu Telugu te Stanza Stanza lemma

Malayalam Malayalam ml Stanza Stanza lemma

Bengali Bengali bn Stanza Stanza lemma

Kannada Kannada kn IndicNLP Rule-based

Gujarati Gujarati gu IndicNLP Rule-based

Marathi Devanagari mr Stanza Stanza lemma

Punjabi Gurmukhi pa IndicNLP Rule-based

Urdu Nastaliq ur IndicNLP Rule-based

7.2 Unicode Normalization

import unicodedata
from [Link].indic_normalize import IndicNormalizerFactory

factory = IndicNormalizerFactory()

def normalize_indic(text: str, lang: str) -> str:

# NFC normalization first
text = [Link]('NFC', text)
# Language-specific normalization
normalizer = factory.get_normalizer(lang)
return [Link](text)

# Transliteration: Hindi -> Latin (for cross-lingual query)

from indictrans import Transliterator
trn = Transliterator(source='hin', target='eng')
latin = [Link]('■■■■') # -> 'bhaarat'

7.3 Cross-Language Retrieval

When a user types a transliterated query like 'bharat ki khoj', the engine detects it as a Roman-script Hindi
query, converts it to Devanagari using the Indic Transliterator, tokenizes it with the Hindi pipeline, and merges
results from both the English and Hindi shards with score blending.

7.4 Stopwords for Indian Languages

Distributed Search Engine Blueprint — Page 19

# [Link]
from [Link] import stopwords as nltk_sw
from [Link] import STOPWORDS_LIST

STOPWORDS = {
'en': set(nltk_sw.words('english')),
'hi': set(STOPWORDS_LIST.get('hi', [])),
'ta': set(STOPWORDS_LIST.get('ta', [])),
# ... etc
}

def remove_stopwords(tokens: list[str], lang: str) -> list[str]:

sw = [Link](lang, set())
return [t for t in tokens if t not in sw]

Distributed Search Engine Blueprint — Page 20

Chapter 8 — Evaluation: Offline & Online Metrics

8.1 Offline Metrics

# evaluation/offline_metrics.py
import numpy as np

def precision_at_k(relevant: set, retrieved: list, k: int) -> float:

'''P@k = |relevant ∩ retrieved[:k]| / k'''
top_k = retrieved[:k]
return sum(1 for d in top_k if d in relevant) / k

def recall_at_k(relevant: set, retrieved: list, k: int) -> float:

'''R@k = |relevant ∩ retrieved[:k]| / |relevant|'''
top_k = retrieved[:k]
if not relevant: return 0.0
return sum(1 for d in top_k if d in relevant) / len(relevant)

def average_precision(relevant: set, retrieved: list) -> float:

'''AP = (1/|R|) * SUM_k [ P@k * rel(k) ]'''
score, hits = 0.0, 0
for k, doc in enumerate(retrieved, 1):
if doc in relevant:
hits += 1
score += hits / k
return score / max(1, len(relevant))

def mean_average_precision(queries: list[dict]) -> float:

'''MAP = mean of AP over all queries'''
return [Link]([
average_precision(q['relevant'], q['retrieved'])
for q in queries])

def dcg_at_k(relevances: list[int], k: int) -> float:

'''DCG@k = SUM_i [ rel_i / log2(i+2) ] for i in 0..k-1'''
relevances = relevances[:k]
return sum(r / np.log2(i + 2) for i, r in enumerate(relevances))

def ndcg_at_k(relevances: list[int], k: int) -> float:

'''nDCG@k = DCG@k / IDCG@k'''
ideal = sorted(relevances, reverse=True)
idcg = dcg_at_k(ideal, k)
if idcg == 0: return 0.0
return dcg_at_k(relevances, k) / idcg

def f1_at_k(relevant, retrieved, k):

p = precision_at_k(relevant, retrieved, k)
r = recall_at_k(relevant, retrieved, k)
if p + r == 0: return 0.0
return 2 * p * r / (p + r)

Distributed Search Engine Blueprint — Page 21

8.2 Offline Evaluation Procedure
Use the TREC Web Track or a custom annotated dataset. For Indian language queries, create a 200-query
benchmark with 3 human judges per query, using majority vote for relevance labels. Compute nDCG@10 as
the primary offline metric.

8.3 Online Metrics (Post-Deployment)

Metric Formula / Definition Collection Method

Click-Through Rate (CTR) clicks / impressions per rank position JS beacon on result click

Dwell Time time between click and back-button press JS page unload event + timestamp diff

Mean Reciprocal Rank (MRR)

mean of 1/rank of first click query_log table

Zero-Result Rate queries with no clicks / total queries query_log + click_log JOIN

Query Latency P95 95th percentile response time (ms) Prometheus + Grafana

Abandonment Rate queries with no click at all session analysis in ClickHouse

Long-Click Rate dwell time > 30 s / total clicks ClickHouse aggregation

8.4 A/B Testing Framework

# ab_testing.py
import hashlib, random

def assign_variant(session_id: str,

experiment: str,
variants: list[str]) -> str:
'''Stable hash-based assignment — same session always same variant.'''
h = int(hashlib.sha256(
f'{experiment}:{session_id}'.encode()).hexdigest(), 16)
return variants[h % len(variants)]

# Condition: run A/B after 1 week post-deployment, min 10k queries/variant

# Control (A): BM25 only
# Treatment (B): BM25 + PageRank reranking
# Primary metric: nDCG@10 from implicit feedback (clicks + dwell)
# Statistical test: Mann-Whitney U (non-parametric, no normal assumption)
# Minimum detectable effect: 0.5% nDCG improvement

Distributed Search Engine Blueprint — Page 22

Chapter 9 — Frontend: FastAPI + Jinja2

9.1 FastAPI Application Structure

backend/
■■■ [Link] # FastAPI app entry
■■■ routers/
■ ■■■ [Link] # GET /search, GET /autocomplete
■ ■■■ index_admin.py # POST /index (admin trigger)
■■■ services/
■ ■■■ query_engine.py # wraps QueryEngine
■ ■■■ [Link] # Redis cache wrapper
■■■ templates/
■ ■■■ [Link] # Jinja2 base layout
■ ■■■ [Link] # Homepage / search box
■ ■■■ [Link] # SERP (search results page)
■ ■■■ [Link]
■■■ static/
■■■ css/[Link]
■■■ js/[Link] # autocomplete, click tracking

9.2 FastAPI [Link]

# [Link]
from fastapi import FastAPI, Request, Query
from [Link] import Jinja2Templates
from [Link] import StaticFiles
from [Link] import JSONResponse
from services.query_engine import engine
from [Link] import redis_cache
import time

app = FastAPI(title='IndiaSearch')
[Link]('/static', StaticFiles(directory='static'), name='static')
templates = Jinja2Templates(directory='templates')

@[Link]('/')
async def home(request: Request):
return [Link]('[Link]',
{'request': request})

@[Link]('/search')
async def search(request: Request,
q: str = Query(..., min_length=1),
lang: str = 'auto',
page: int = 1):
cache_key = f'search:{q}:{lang}:{page}'
cached = await redis_cache.get(cache_key)
if cached:
results = cached
else:

Distributed Search Engine Blueprint — Page 23

t0 = time.perf_counter()
results = [Link](q, lang=lang,
top_k=10, offset=(page-1)*10)
latency_ms = (time.perf_counter() - t0) * 1000
await redis_cache.set(cache_key, results, ttl=300)
return [Link]('[Link]', {
'request': request, 'query': q,
'results': results, 'page': page,
'latency_ms': round(latency_ms, 1)
})

@[Link]('/autocomplete')
async def autocomplete(q: str) -> JSONResponse:
suggestions = [Link](q, limit=8)
return JSONResponse(suggestions)

9.3 Jinja2 Results Template ([Link])

{% extends '[Link]' %}
{% block content %}
<div class='search-bar'>
<form action='/search' method='get'>
<input name='q' value='{{ query }}' autocomplete='off'
id='search-input'>
<select name='lang'>
<option value='auto'>Auto</option>
<option value='en'>English</option>
<option value='hi'>Hindi</option>
<option value='ta'>Tamil</option>
</select>
<button type='submit'>Search</button>
</form>
<p class='meta'>{{ results|length }} results
({{ latency_ms }} ms)</p>
</div>

{% for r in results %}
<div class='result' data-doc-id='{{ [Link] }}'
data-pos='{{ [Link] }}'>
<a class='result-title' href='{{ [Link] }}'
onclick='trackClick({{ [Link] }}, {{ [Link] }})'>
{{ [Link] }}</a>
<span class='result-url'>{{ [Link] }}</span>
<p class='result-snippet'>{{ [Link] }}</p>
{% if [Link] %}
<div class='entities'>
{% for e in [Link][:3] %}
<span class='tag tag-{{ [Link]|lower }}'>{{ [Link] }}</span>
{% endfor %}
</div>
{% endif %}
</div>

Distributed Search Engine Blueprint — Page 24

{% endfor %}
{% endblock %}

Distributed Search Engine Blueprint — Page 25

Chapter 10 — Deployment: Docker, Nginx & Hosting

10.1 [Link]
version: '3.9'
services:
crawler:
build: ./crawler
environment:
- REDIS_URL=redis://redis:6379
- KAFKA_BROKERS=kafka:9092
depends_on: [redis, kafka]
deploy:
replicas: 4

pipeline:
build: ./pipeline
environment:
- KAFKA_BROKERS=kafka:9092
- POSTGRES_DSN=postgresql://user:pass@postgres/searchdb
depends_on: [kafka, postgres]
deploy:
replicas: 2

backend:
build: ./backend
ports: ['8000:8000']
environment:
- POSTGRES_DSN=postgresql://user:pass@postgres/searchdb
- REDIS_URL=redis://redis:6379
depends_on: [postgres, redis]

nginx:
image: nginx:alpine
ports: ['80:80', '443:443']
volumes:
- ./[Link]:/etc/nginx/[Link]
- ./certs:/etc/nginx/certs
depends_on: [backend]

postgres:
image: postgres:16-alpine
volumes: ['pgdata:/var/lib/postgresql/data']
environment:
POSTGRES_DB: searchdb
POSTGRES_USER: user
POSTGRES_PASSWORD: pass

redis:
image: redis/redis-stack:latest # includes RedisBloom

Distributed Search Engine Blueprint — Page 26

kafka:
image: confluentinc/cp-kafka:7.6.0
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092

volumes:
pgdata:

10.2 Nginx Configuration

# [Link]
upstream backend {
server backend:8000;
keepalive 32;
}

server {
listen 80;
server_name [Link];
return 301 [Link]
}

server {
listen 443 ssl http2;
server_name [Link];
ssl_certificate /etc/nginx/certs/[Link];
ssl_certificate_key /etc/nginx/certs/[Link];

gzip on;
gzip_types text/html application/json;

location /static/ {
alias /app/static/;
expires 30d;
}

location / {
proxy_pass [Link]
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
limit_req zone=search burst=20 nodelay;
}
}

10.3 Hosting Hacks (Free / Cheap Tier)

Budget hosting strategy for a student/demo deployment. Full production deployment requires a VPS or cloud VM.

Distributed Search Engine Blueprint — Page 27

 Service What to Host Free Tier Limits Hack

[Link] FastAPI backend + Postgres

500 hr/mo, 1 GB RAM Sleep on inactivity; use UptimeRobot ping to keep aliv

[Link] FastAPI backend 750 hr/mo, 512 MB Deploy Docker image directly from GitHub

[Link] Docker Compose app 3 shared VMs free Use 'flyctl deploy'; supports persistent volumes

Cloudflare Tunnel Expose local server Always free 'cloudflared tunnel' — no port forwarding needed

Ngrok Quick demo Free tier 1 tunnel Instant HTTPS for demo; not permanent

Oracle Cloud Free Always-Free VM (4 OCPU, Forever

24 GB) free Best hack: full VPS, run Docker Compose, attach dom

Cloudflare Pages Static frontend Unlimited Serve Jinja-rendered HTML as static if queries are pre

Supabase PostgreSQL 500 MB free Drop-in Postgres replacement; REST API included

10.4 Getting a Free Domain & SSL

# 1. Get free domain: [Link] (.tk/.ml) or use subdomain of render/railway
# 2. Cloudflare DNS (free) — add A record pointing to your server IP
# 3. Let's Encrypt SSL via certbot:
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d [Link]
# Auto-renews every 90 days via systemd timer

# 4. Or use Cloudflare's free SSL proxy (orange cloud on) —

# no cert needed on server, Cloudflare terminates SSL

Distributed Search Engine Blueprint — Page 28

Chapter 11 — Full Project Checklist & Milestones
Phase 1: Foundation (Weeks 1-2)
■ [Link] build system for C++ crawler
■ DNS resolver with LRU cache
■ Raw socket HTTP/1.1 fetcher (IPv4)
■ HTTPS via OpenSSL (SNI support)
■ Thread pool (64 workers) with task queue
■ Bloom filter (murmur3, 2 GB bit-array) for URL dedup
■ [Link] parser & per-domain delay enforcement
■ Redis URL frontier (priority queue with politeness buckets)
■ Kafka producer for raw HTML

Phase 2: Pipeline & Index (Weeks 3-4)

■ Kafka consumer in Python
■ lxml HTML parser + noise tag removal
■ fastText language detection (176 languages)
■ NLTK/Stanza tokenizer + SnowballStemmer
■ IndicNLP integration (Hindi, Tamil, Telugu, Bengali, etc.)
■ spaCy NER (English) + Stanza NER (Indian languages)
■ Link extractor feeding back to URL frontier
■ PostgreSQL schema: urls, links, entities, query_log
■ RocksDB posting list writer with zstd compression
■ Delta encoding for doc IDs in posting lists
■ 8-shard consistent-hash index

Phase 3: Query Engine (Weeks 5-6)

■ BM25 scorer with field weighting
■ Iterative PageRank computation (50 iterations)
■ BM25 + PageRank score fusion (0.7/0.3 blend)
■ Query parsing: AND/OR/phrase/negation
■ Query expansion via WordNet (English)
■ Transliteration-based cross-lingual retrieval
■ Best-passage snippet generation (40-word window)
■ Redis result caching (TTL=300s)
■ Autocomplete via prefix trie or PostgreSQL trigram

Phase 4: Frontend & API (Week 7)

Distributed Search Engine Blueprint — Page 29

■ FastAPI app with /search, /autocomplete, /admin
■ Jinja2 templates: home, results, error pages
■ Language selector (auto-detect + manual override)
■ Pagination (10 results/page)
■ Click tracking JS beacon (sends to /api/click)
■ Dwell time measurement (unload event)
■ Entity tag display on result cards
■ Mobile-responsive CSS (Tailwind or vanilla)

Phase 5: Deployment (Week 8)

■ Dockerfiles for crawler, pipeline, backend
■ [Link] with all services
■ Nginx reverse proxy config with SSL
■ Deploy to Oracle Cloud Free Tier VM
■ Cloudflare DNS + free SSL
■ UptimeRobot monitoring (free)
■ Prometheus + Grafana for latency/throughput
■ ClickHouse for query analytics

Phase 6: Evaluation (Ongoing)

■ Build 200-query annotated benchmark (10 Indian language queries)
■ Compute P@5, P@10, R@10, MAP, nDCG@10 offline
■ Set up A/B test: BM25 only vs BM25+PageRank
■ Track CTR by rank position (position bias analysis)
■ Track dwell time per query category
■ Monitor zero-result rate and query abandonment
■ Run Mann-Whitney U test after 10k queries/variant

Advanced / Extra Credit

■ HTTPS crawling with OpenSSL (SNI, cert verification)
■ Distributed crawler: multiple machines, shared Redis frontier
■ Learning-to-Rank (LTR): train LambdaMART on click logs
■ Vector search: embed docs with IndicBERT, FAISS index
■ Knowledge graph sidebar (entity linking to Wikidata)
■ Spell correction: SymSpell or custom n-gram model
■ Search-as-you-type: ElasticSearch prefix completion
■ [Link] parser for seed URL discovery
■ GPU-accelerated BM25 via CuPy on Oracle A10 instance

Distributed Search Engine Blueprint — Page 30

Conclusion
This blueprint covers a production-grade distributed search engine from raw socket crawling in C++ to a
publicly accessible FastAPI website. The project integrates: system design (CAP, sharding, caching,
microservices), low-level networking (DNS, sockets, multithreading), database engineering (PostgreSQL +
RocksDB + Redis), information retrieval (BM25, PageRank, inverted index), NLP (tokenization, NER,
stemming, 10 Indian languages), rigorous evaluation (nDCG, MAP, A/B testing), and real hosting on free-tier
infrastructure.

Each chapter maps directly to a learnable module. Implement iteratively — get a single-threaded crawler
working first, then add multithreading, then the pipeline, and so on. The checklist provides a
milestone-by-milestone roadmap to a fully deployed search engine in approximately 8 weeks.

Key libraries: libssl (C++), RocksDB (C++), Kafka-cpp, BeautifulSoup4, fastText, Stanza, IndicNLP, spaCy, NLTK,
FastAPI, Jinja2, psycopg2, redis-py, confluent-kafka-python, zstandard, reportlab.

Distributed Search Engine Blueprint — Page 31