NLP Crash Course (Comprehensive Interview Sheet)
What is NLP?
Natural Language Processing (NLP) is a branch of artificial intelligence (AI) focused on enabling machines to
understand, interpret, generate, and respond to human language. NLP powers many applications including
text classification, named entity recognition, language generation, question answering, and machine
translation.
Text Preprocessing (Tokenization and Normalization)
Tokenization is the process of splitting raw text into smaller units such as words, subwords, or sentences. For
example, the sentence 'The quick brown fox' becomes ['The', 'quick', 'brown', 'fox'].
Normalization involves standardizing text to reduce variability. Common steps include converting text to
lowercase, removing punctuation, and optionally removing stop words such as 'the' or 'is'.
Lemmatization and Stemming
Stemming reduces words to their root forms by chopping off suffixes. This is a fast and rough process that
may not always produce valid words. For example, 'fishing' becomes 'fish'.
Lemmatization, on the other hand, reduces words to their base or dictionary forms, considering the words
meaning and context. For example, 'was' becomes 'be' and 'running' becomes 'run'.
Lemmatization is generally preferred in NLP tasks where preserving meaning is important.
Bag-of-Words and TF-IDF
Bag-of-Words (BoW) represents text by counting how many times each word appears in a document. It
ignores word order and context but is simple and effective for many tasks.
Term Frequency-Inverse Document Frequency (TF-IDF) improves upon BoW by weighting words based on
their frequency across documents. Words that are common across documents get lower weights, while rare
but important words get higher weights.
Word Embeddings (Dense Vector Representations)
Word embeddings are dense vector representations that capture the semantic meaning of words. Models like
Word2Vec and GloVe learn these representations such that similar words are close together in vector space.
For example, the vectors for 'king' and 'queen' would be closer together than 'king' and 'apple'. However,
these embeddings are static, meaning the same word has the same vector regardless of context.
Contextual Embeddings and Transformers
Contextual embeddings are generated dynamically based on the surrounding text. This means that the word
'bank' in 'river bank' and 'investment bank' would have different vector representations.
Transformer models like BERT, GPT, and others use attention mechanisms to create these embeddings.
This advancement significantly improved NLP by capturing context-dependent meaning.
Large Language Models (LLMs)
Large Language Models (LLMs) are trained on massive text corpora and can perform a variety of language
tasks. Examples include OpenAI's GPT series and Meta's LLaMA.
These models can be fine-tuned on specific tasks or domains for better performance. Prompt engineering is
also widely used to guide their responses by crafting specific input formats or instructions.
NLP in Production (Real-world Applications and Challenges)
Deploying NLP models in production requires considerations beyond accuracy:
- Latency: Models must respond quickly enough for user-facing applications.
- Observability: Performance metrics, drift detection, and error monitoring are needed to ensure reliability.
- Explainability: Especially in sensitive domains like law and healthcare, understanding why a model made a
decision is important.
- Scalability: Systems must handle large volumes of data and user requests.
- Human-in-the-loop: AI often needs human oversight to verify or correct outputs, ensuring quality and trust.
Summary of NLP Workflow
Traditional NLP Workflow:
Raw text -> Tokenization -> Lemmatization or Stemming -> Vectorization (e.g. TF-IDF) -> Model Training ->
Prediction/Output
Modern NLP Workflow with Transformers:
Raw text -> Tokenization -> Contextual Embeddings (via Transformer) -> Model -> Output
While transformers dominate modern NLP, classical preprocessing steps are still useful for simpler models,
rule-based systems, or when computational resources are limited.