No Topic Date

W17L1 AI Product Development Lifecycle 25 Sept 2025

W17L2 LLM Evaluation, AI Metrics, and User Testing Best Practices (AN) 27 Sept 2025
W17L3 Hugging Face for PMs - tools, capabilities and applications (AN) 29 Sept 2025

W18L1 Tech Basics: Databases, APIs, Frontend, Backend, and Agentic AI (Prof) 16 Oct 2025
AI & Agents : Lovable, Replit, Cursor, GHCP and Bolt (AN)
W18L2 17 Oct 2025
[Link]
4D Product Development with GenAI Tools
W18L3 18 Oct 2025
[Link]

AI Model Categories, Advanced LLM Metrics, and Production Hosting

W19L1 23 Oct 2025
Basics
Stakeholder Managment for AI PM and AI Product Lifecycle, and Scaling
W19L2 Case Studies 25 Oct 2025
[Link]
Tech Debt in AI Systems: Model Upkeep, Agile Iteration, and Monitoring
W19L3 Tools 27 Oct 2025
[Link]

RAG Fundamentals: Concepts, Use Cases, and Tools for Product

W20L1 30 Oct 2025
Managers
Type of LLM Models, LLM Benchmarks, GAN and other Image Models
W20L2 31 Oct 2025
[Link]
Github Copilot for Product Managers - understanding and contributing to
W20L3 Code 03 Nov 2025
[Link]

Prepared by Vaibhav Yendole 1

 W17L1 AI Product Development Lifecycle (Prof. Saravana)
Summary

AI products follow a lifecycle that looks similar to traditional software but adds data, models, evaluation,
and safety as first-class citizens. The PM’s job is to align user value with technical
feasibility and operational responsibility, while shipping iteratively.

Lifecycle map:

1.) Problem framing → 2) Feasibility & approach → 3) Data strategy → 4) Prototype & evaluate → 5) Safety &
governance → 6) Productionization → 7) Post-launch monitoring → 8) Improvement loop.

Frame the Problem

 What job are we helping the user do, and how do they do it today without AI?

 What lift do we expect (e.g., higher NPS, better conversion, lower handling time)?

 Note hard limits: domain rules, privacy/safety/compliance, languages, accessibility.

 Ask: could simple rules or heuristics deliver ~80% of the value?

Check Feasibility & Choose an Approach

 Build vs. buy: hosted closed model (fast), open-source (control/cost), fine-tune/custom
(specialized).

 Design path: basic prompting → RAG (retrieve facts) → fine-tuning → tool use/agents; single-turn or
chat; on-device or server.

 Plan for trade-offs: latency targets (e.g., P95), token/cost budgets, throughput.

Plan the Data

 Sources: product logs, domain docs/KBs, labeled feedback, red-team examples.

 Quality: coverage, freshness, bias checks, PII handling, consent, retention.

 Governance: access controls, audit trails, data lineage.

Prototype & Evaluate

 Build in steps: prompt trials → lightweight RAG → sandboxed tool calls → gated beta.

 Offline tests: gold answers, adversarial cases, rubric scores, pairwise preferences.

 Online tests: A/B or interleaving, plus guardrail monitors.

 Track metrics:

 User: task success, time to complete, CSAT/NPS.

 System: accuracy, faithfulness, refusal rate, P95 latency, hallucination/toxicity, cost per
session.

 Business: conversion, retention, revenue, support deflection.

Safety, Risk & Compliance

Prepared by Vaibhav Yendole 2

  Guardrails: input filters, output moderation, policy-aware prompts, retrieval allowlists.

 Fairness: evaluate by segment (age, device, language) and consider error severity.

 Compliance: minimize PII, get consent, encrypt, do DPIA/TRA, respect data residency.

Productionize & Launch

 Infra: vector store/RAG service, feature flags, canaries, circuit breakers, fallbacks.

 Reliability: rate limits, timeouts/retries, caching, idempotency, solid observability.

 Rollout: private preview → allowlisted beta → staged GA, with kill-switches ready.

Monitor After Launch

 Dashboards: SLO/SLA (e.g., P95 < 900 ms), quality scorecards, cost per 1k requests.

 Watch drift/freshness: data drift alerts, retriever recall, stale knowledge.

 Feedback loops: human review, triage, auto-label pipelines.

Improve in Loops

 Levers: better prompts, tuned retrieval, new/changed tools, fine-tuning, new data, small UI nudges.

 Change safely: shadow tests, targeted red-team bursts, safety gates in CI/CD.

Case Study

Multilingual Loan Support Assistant with Strict SLAs A fintech needs a chat assistant for policy,
eligibility, and documentation queries across languages. Constraints: P95 ≤ 900ms, privacy &
auditability, limited context window, fairness issues for older low-income cohort, cost control.

Questions as a Product Manager:

 What is the minimum viable approach? (e.g., RAG with chunked docs + strict ACLs)

 How do we meet latency targets while maintaining quality? (e.g., tiered fallback, caching,
streaming)

 How to measure and fix fairness issues? (segment audits, targeted data collection)

 What does the rollout and kill-switch plan look like?

Glossary

 RAG (Retrieval-Augmented Generation): Augment model outputs with retrieved, vetted

knowledge.

 P95 Latency: Time under which 95% of requests complete.

 Drift: Degradation as data/distribution changes over time.

 Guardrails: Programmatic constraints to reduce unsafe/irrelevant outputs.

 SLO/SLA: Internal target vs. external commitment for reliability/latency/quality.

------------------------------------------------------------------------------------------------------------------------

Prepared by Vaibhav Yendole 3

Why this matters

As a PM, your real edge isn’t just building features—it’s compressing the time from idea to impact.
Generative AI and no-code change the game by letting you validate concepts, assemble working
prototypes, and ship marketing collateral days—not months—ahead of schedule. Today’s lecture shows
how to plug AI (including custom GPTs) directly into the product lifecycle so you can de-risk bets faster,
align stakeholders with evidence, and scale with fewer engineering dependencies.

Foundations you stand on (quick review)

Solid PM craft still anchors AI work.

 Stakeholder alignment: Like Southwest Airlines, keep everyone rallied around a clear, customer-
centric strategy so AI work ladders to brand promise and service model.

 AARRR metrics: With Decathlon, track Acquisition → Activation → Retention → Referral →

Revenue to see exactly where AI boosts funnel performance (e.g., better activation via smarter
onboarding content).

 Business models: Pick from freemium, subscription, licensing, marketplace, etc., to shape
pricing and distribution choices your AI features must support.

 Educating consumers: The Scrum Daddy meditation app example reminds us: highlight actual
vs. perceived value so AI outputs (e.g., recommendations) are trusted and adopted.

 GTM and branding: Case studies like Pickmagic and Nexora show how positioning and brand
architecture (branded house vs. house of brands) guide naming, visuals, and channel
messaging—crucial when you automate content with AI.

Analytics as your PM lens

AI multiplies value when it’s tied to analytics:

 Descriptive (what happened) and Diagnostic (why it happened) frame the problem.

 Predictive (what will happen) powers planning.

 Prescriptive (what we should do) informs actions. In a tech firm’s customer acquisition funnel,
you might describe drop-offs, diagnose misaligned messaging, predict conversion by segment, and
prescribe next-best actions (e.g., personalized trials).

Supervised learning in practice

Use classification (bucket outcomes) and regression (predict numbers) to drive decisions. Example:
classify whether a lead will convert; regress to estimate expected first-order value. Then evaluate with
appropriate metrics before you tie these models to onboarding flows or sales ops.

Generative AI and custom GPTs

Foundational models (broadly trained) can be steered into task-based models via prompting. With one-
shot or few-shot examples, you turn a general LLM into a custom GPT that understands your domain and
style. For PMs, this means rapidly prototyping user journeys, support replies, or content variants and
testing them with real users—before committing roadmap capacity.

Prepared by Vaibhav Yendole 4

Case study: Amazon product listing automation (“Bangle Studio”)

Problem: Manual Amazon listings from product images are slow and costly.

Prototype: A custom GPT analyzes an uploaded image (e.g., Rajasthan bangles), asks clarifying
questions, and outputs SEO-optimized titles, descriptions, key features, pricing analysis, and
recommendations.

Application: Evolved into a no-code web app (Bangle Studio) that automates listing generation, market
analysis, and pricing insights with simple upload + chat.

Marketing collateral: Integrated generative tools—NanoBanana for lifestyle images and VO3 for short
product videos—so sellers get photos and reels without creative teams.

Why it matters for PMs: You compress prototype → validate → commercialize: test copy variants,
compare pricing, and launch pages faster—then loop metrics back into prompts for continuous lift.

From playground to production with no-code

Early concept tests live in custom GPT playgrounds—cheap, quick, and great for “is there signal?”
checks. When you see traction, platforms like Base44 let you generate a working AI app from a single
prompt, then:

 Customize UI and on-the-fly behavior,

 Wire up databases and cloud storage,

 Use simplified versioning for safe iteration. You keep control of speed and scope while
engineering partners focus on durability and scale.

What you practiced

You built prompts for an image-to-marketing app that outputs Amazon listings, press releases, website
copy, and social posts. You also saw a travel co-pilot prototype—proof that ChatGPT + NanoBanana +
VO3 can be composed into one experience, with storage, hosting, and maintenance aligned to PM realities
(cost, latency, and reliability).

Key takeaways

 AI shrinks the lifecycle: Idea → prototype → launch can be parallelized with generative AI and no-
code.

 Custom GPTs de-risk early: Use few-shot prompts to validate concepts before writing specs.

 No-code = speed to value: Tools like Base44 convert prompts into apps; iterate UI, data, and logic
quickly.

 Own the funnel with AARRR: Instrument activation and retention so AI outputs prove business
impact.

 Ship collateral faster: Listings, images, and videos via ChatGPT, NanoBanana, VO3 reduce cost
and time.

 PM superpower: Pair classic PM skills (alignment, models, GTM) with AI + no-code literacy to lead
from idea to measurable impact.

Prepared by Vaibhav Yendole 5

Overview of the Lecture

The lecture focused on the AI Product Development Lifecycle, emphasizing the integration of generative AI
and no-code platforms into the traditional product development stages. It covered prior foundational
topics in product management and then dove deeply into practical applications of AI, especially custom
GPTs, for accelerating product development and commercialization.

Review of Previous Topics

 Stakeholder Alignment: Discussed the importance of aligning all stakeholders around a

customer-centric integrated marketing strategy, using Southwest Airlines as a case study.

 Metrics and Analytics: Introduced the AAARRR framework (Acquisition, Activation, Retention,
Referral, Revenue) for tracking product success, exemplified with Decathlon.

 Business Models: Explored eight business models including freemium, subscription, licensing,
and marketplace models, with activities to apply these models to products.

 Educating Consumers: Emphasized educating customers about actual versus perceived value,
using examples like the Scrum Daddy meditation app.

 Go-to-Market Strategy and Branding: Covered segmentation, persona creation, product

positioning, and branding strategies with case studies like Pickmagic and Nexora, including
branded house and house of brands models.

AI and Machine Learning in Product Lifecycle

 Introduction to AI/ML: Covered four types of analytics — descriptive, diagnostic, predictive, and
prescriptive — and their role in product management.

 Supervised Learning: Delved into classification and regression problems, model evaluation, and
practical use cases such as a tech firm’s customer acquisition funnel.

 Generative AI and Custom GPTs: Explained foundational models, task-based models, prompting
techniques (one-shot, few-shot), and how custom GPTs can be created to address specific product
problems.

Custom GPT Use Case: Amazon Product Listing Automation

 Problem Identified: Traditional manual process of creating detailed Amazon product listings from
product images is time-consuming and expensive.

 Custom GPT Prototype: Developed a prototype that analyzes uploaded images (e.g., Rajasthan
bangles), asks follow-up questions, and automatically generates Amazon-optimized product titles,
descriptions, key features, pricing analysis, and recommendations.

 Application Development: Expanded from prototype to a full web application ("Bangle Studio")
that automates listing generation, market analysis, and pricing insights with upload and user
interaction features, without traditional coding.

 Additional AI Integrations: Incorporated AI-based lifestyle image generation (using platforms like
NanoBanana) and short product video creation (using VO3), enhancing product marketing
collateral easily and affordably.

AI-Driven Product Development Lifecycle Transformation

Prepared by Vaibhav Yendole 6

  Traditional vs AI-Enabled Process: Traditional product lifecycle (idea, concept, design, prototype,
test, validate, launch) can now be compressed, automated, and parallelized using generative AI
and no-code tools, drastically reducing time and resource requirements.

 Prototyping in Custom GPT: Early concept testing and validation can be done inside custom GPT
playgrounds quickly and cheaply.

 No-Code Application Development: Tools like Base44 enable creation of full-fledged AI-powered
applications from a single prompt, eliminating the need for deep development expertise.

 Iterative Testing and Refinement: AI facilitates iterative development with on-the-fly debugging, UI
customization, and functionality expansion.

 Database and Version Control: No-code platforms support database linking, cloud integration,
and simplified app versioning for ongoing maintenance and scaling.

Practical Activity: Developing AI-Powered Applications Using No-Code Platforms

 App Idea: User uploads a product image; app generates comprehensive marketing collateral
automatically (Amazon listing, press release, website content, social media posts).

 Prompt Engineering: Students were tasked to develop detailed, non-technical prompts using
ChatGPT or similar models to describe app behavior for platforms like Base44.

 Live Demonstrations: Participants shared their created prompts and app prototypes for product
listing and a travel co-pilot app, illustrating AI’s role in simplifying app creation and user experience
customization.

 Integration and Scalability: Discussion on combining different AI services (ChatGPT,

NanoBanana, VO3) within one app, cloud integration for storage and model hosting, and
maintaining apps over time.

Key Takeaways

 The product development lifecycle is drastically simplified with generative AI and no-code tools.

 Custom GPTs enable rapid prototyping to validate ideas efficiently before full product
development.

 No-code platforms like Base44 translate AI-driven prompts directly into usable applications,
democratizing product creation.

 AI-generated marketing assets (listings, images, videos) can drastically reduce time and costs for
e-commerce sellers and marketers.

 Product managers should acquire AI and basic no-code skills to better manage and directly build
AI-enhanced products in the future.

Stakeholder Alignment in Product Lifecycle Management

Stakeholder alignment is the initial and crucial phase in product lifecycle management where all team
members and stakeholders are brought on the same page regarding the product strategy. It involves
focusing on delivering one particular value proposition to the target segment. For example, in the case of

Prepared by Vaibhav Yendole 7

Southwest Airlines, the focus was on customer centricity and bringing the entire team onboard to deliver
unified value to customers.

Metrics and AAAARR Framework

Metrics tracking is vital to assess the performance and growth of a product. The AAAARR framework is a
commonly used metric framework comprising Acquisition, Activation, Retention, Referral, and Revenue.
For example, Decathlon uses these metrics to understand user behavior and business success.

Example:

metrics = {

acquisition: trackVisitors(),

activation: trackUserSignups(),

retention: trackReturningUsers(),

referral: trackUserReferrals(),

revenue: trackUserPurchases()

Business Models in Product Management

Understanding and selecting appropriate business models is fundamental in product management.

Models like freemium, subscription, licensing, transactional, and marketplace dictate how revenue is
generated from the product. For example, a freemium model offers free basic features while charging for
premium features

Educating Consumers on Product Value

Creating a good product is insufficient; educating consumers about the actual and perceived value is
necessary. This bridges the gap between what a product offers and what customers understand about it.
For instance, using the Scrum Daddy app, efforts were made to educate consumers on the hidden value of
the app.

Go-To-Market Strategy and Positioning

Go-To-Market (GTM) involves choosing target customer segments, creating buyer personas, and
positioning the product accordingly. For example, using Pickmagic and Nexora cases, the importance of
choosing one or multiple segments and branding strategies like brand house or house of brands was
discussed.

Introduction to AI and Machine Learning in Product Management

Prepared by Vaibhav Yendole 8

AI and Machine Learning (ML) are integrated into product management to advance analytics capabilities.
The four types of analytics discussed were descriptive, diagnostic, predictive, and prescriptive. For
example, using loan application data to predict customer behavior involves predictive analytics and
supervised learning.

Example :

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LogisticRegression

X_train, X_test, y_train, y_test = train_test_split(data, labels)

model = LogisticRegression()

[Link](X_train, y_train)

predictions = [Link](X_test)

Supervised Learning - Classification and Regression

Supervised learning involves training models on labeled data. Classification problems predict categorical
variables, while regression addresses continuous variables. Model evaluation methods determine how
well the model performs. For example, predicting customer demo attendance and conversion rates falls
under supervised learning.

Generative AI and Custom GPTs

Generative AI uses foundation models capable of performing multiple tasks using the same architecture.
Custom GPTs are customized versions embedding specific data plus instructions to address particular
use cases. For example, creating a course assistant custom GPT answering course-specific questions.

Custom GPTs control behavior through:

 Title

 Description

 Instructions

 Knowledge base (uploaded files)

 Create actions (integration with external resources)

Example:

Custom GPT Instructions = "Only answer questions based on the uploaded course syllabus documents. If
the answer is not in the documents, respond with 'Information not available'."

Custom GPTs enable tailored AI-driven applications.

Prepared by Vaibhav Yendole 9

Example Use Case: Amazon Product Listing Generator Using AI

An AI-powered product, termed 'Bangle Studio', automates the creation of Amazon listings from product
images. It analyzes images, asks follow-up questions to clarify ambiguous information, generates
optimized product names, descriptions, key features, and pricing strategies.

Features:

 Image analysis to identify materials, colors, and design styles

 Interactive Q&A for precise product details

 SEO optimized product listing generation

 Market analysis with price recommendations

 Integration with Amazon APIs for automated listing upload

Example of prompt interaction:

User: Upload 3 images of bangles.

AI: Detected colors: pink, silver-tone metal. Is the metal brass?

User: Yes, brass with silver polish.

AI: Generating product title and description...

Extending Product Listings: Lifestyle Image and Video Generation

Beyond static product listings, AI can generate lifestyle images and short video clips showing products in
use, enhancing e-commerce appeal. Platforms like Nanobanana and VO3 enable generation of high-
resolution lifestyle images and 7-10 second product videos.

Process:

 Use product image and description to generate detailed prompts

 Send prompts to image generation services (e.g. Nanobanana)

 Create videos using animation models

Example:

prompt = "Generate a high-resolution lifestyle image featuring elegant pink bangles styled with traditional
Indian attire."

Product Development Lifecycle Transformation through AI

Traditional product lifecycle stages like idea generation, concept screening, prototyping, testing,
validation, and go-to-market are drastically shortened with AI:

 Idea and concept validation through custom GPT prototypes

 Automated app development using low-code/no-code platforms powered by AI (e.g., [Link])

 Parallelized development, testing, and validation enabled by AI responses

Prepared by Vaibhav Yendole 10
  Faster go-to-market through integrated AI capabilities

Example workflow:

1. Identify problem -> 2. Prototype with custom GPT -> 3. Develop no-code app (base44) -> 4. Test &
validate quickly -> 5. Launch & iterate

Developing AI-Powered Applications Using Low-Code Platforms

Low-code platforms like [Link] allow creation of full applications by providing detailed text prompts
rather than programming.

Steps:

 Define app functionality in a clear and detailed natural language prompt

 Use generative AI (e.g., ChatGPT, Gemini) to refine the prompt

 Input the prompt into the low-code platform

 The platform generates code and UI for a working app

Example prompt snippet:

Develop an e-commerce product marketing app where user uploads a product image and the app
generates Amazon listing, press release, website content, and social media text posts.

This approach reduces dependency on traditional coding and speeds up product deployment.

Managing AI Applications and Databases

AI applications built on low-code platforms can integrate with databases for data persistence and version
control.

 Built-in database functions in low-code platforms support storing user inputs and outputs

 Integration with cloud databases (AWS, Azure, GCP) allows scalability and backend processing

 Version control through app duplication enables iterative improvements without downtime

Example:

// Example of storing product listing into database (pseudo code)

[Link]('listing123', productListingData)

Proper data management ensures seamless app functionality and maintenance.

Prepared by Vaibhav Yendole 11

 W17L2 LLM Evaluation, AI Metrics, and User Testing Best Practices (AN)
Core Concepts: Moving Beyond "It Feels Right"

As a product manager, you're used to defining and tracking metrics to measure success. But how do you
do that when the "product" is a constantly evolving, non-deterministic LLM? Here are the foundational
concepts you need to know:

1. LLM Evals: Testing Product Outputs

At its core, an LLM evaluation (or "eval") is a systematic process for assessing the quality and reliability of
an LLM's output for a specific task. For product managers, this is your toolkit for ensuring the AI is actually
solving the user's problem effectively and safely. Think of it as user acceptance testing (UAT) for your AI.

2. Evals vs. Benchmarks vs. Metrics

These terms are often used interchangeably, but they have distinct meanings:

 Benchmarks: These are standardized tests that measure an LLM's performance on a broad range
of general tasks (e.g., answering trivia questions, summarizing text). They are useful for comparing
different models before you've integrated them into your product.

 Evals: These are custom tests that you design to measure an LLM's performance on tasks that
are specific to your product and your users. For example, if you have an AI-powered customer
support bot, an eval would test its ability to answer questions based on your company's knowledge
base.

 Metrics: These are the specific, quantifiable measures you use within your evals and benchmarks
to score the LLM's performance. Examples include accuracy, relevance, helpfulness, and toxicity.

Analogy: Think of building a car.

 Benchmarks are like the standardized fuel efficiency and safety ratings (e.g., MPG, crash test
scores) that are the same for all cars.

 Evals are like taking a specific car on a test drive on the exact roads your customers will be driving
on, with the specific cargo they'll be carrying.

 Metrics are the specific things you measure during that test drive: gas mileage on hills, how
smoothly it handles bumps, etc.

3. Evaluator Types: Reference-Based vs. Reference-Free

There are two main ways to evaluate an LLM's output:

 Reference-Based Evaluation: This method compares the LLM's output to a pre-written "golden" or
"reference" answer. This is useful for tasks with a single correct answer, like factual question-
answering.

 Reference-Free Evaluation: This method assesses the LLM's output based on a set of criteria
without a specific "correct" answer to compare it to. This is more common for creative or open-
ended tasks, like summarization or content creation. Here, you might use another LLM to act as a
"judge" and score the output on metrics like coherence, creativity, or adherence to brand voice.

Prepared by Vaibhav Yendole 12

4. AI Product Feature Metrics

When an LLM is a feature within a larger product, you need to measure its impact on user behavior and
business goals. These metrics go beyond the quality of the LLM's output and focus on the value it delivers.
Examples include:

 Task Success Rate: Are users able to successfully complete their task using the AI feature?

 User Adoption and Engagement: Are users choosing to use the AI feature? How often?

 Reduction in Support Tickets: If the AI is meant to help users self-serve, is it reducing the load on
your support team?

 Conversion Rate: If the AI is part of a sales or marketing funnel, is it helping to convert users?

5. The Three Pillars of LLM Evaluation

A robust LLM evaluation system has three key components:

 Dataset: This is a collection of inputs (and sometimes, corresponding ideal outputs) that you use
to test the LLM. A good dataset should be representative of the real-world scenarios your users will
encounter.

 Evaluators: These are the methods or "judges" you use to score the LLM's outputs. This can be a
human, another LLM, or a programmatic check.

 Applications: This refers to the specific product or use case you are evaluating the LLM for. The
same LLM might perform well in one application (e.g., creative writing) but poorly in another (e.g.,
medical diagnosis).

A Glimpse into the Toolkit

Here's a brief overview of the tools we'll be exploring in the lecture:

 OpenAI Playground Evals: A framework provided by OpenAI that allows you to create and run
evaluations on their models. It's a great starting point for hands-on experimentation.

 LMQL (Language Model Query Language): A programming language that allows you to interact
with LLMs in a more structured and constrained way. Think of it as "SQL for LLMs," enabling you to
guide the model's output to fit specific formats or rules, which is incredibly useful for ensuring
reliable AI features.

 LangSmith: A platform from the creators of LangChain that is designed for debugging, testing, and
monitoring LLM applications. It gives you deep visibility into what your LLM is doing, making it
easier to identify and fix issues. It's built for developers but provides valuable insights for product
managers to understand the application's behavior.

 Evidently AI: An enterprise-grade platform that provides a comprehensive solution for LLM
evaluation and testing. It helps teams assess AI risks, define evaluation criteria, and set up a robust
testing process to ensure the reliability and safety of your AI systems.

Prepared by Vaibhav Yendole 13

Why this matters

As a PM building agentic AI products, your job isn’t done when the model “works.” You’re accountable
for quality, safety, and user trust in production. This lecture gives you a playbook to evaluate LLM
applications end-to-end—so you can decide which model to ship, spot regressions, prioritize fixes, and
communicate results clearly to stakeholders. In short: this is how you turn a clever demo (the Amazon
support bot) into a reliable product.

Evaluations, benchmarks, and metrics

Evaluations check whether the application solves the user’s problem in context (e.g., “Did the Amazon
chatbot resolve a damaged-item complaint correctly and empathetically?”). Benchmarks test
the model’s capabilities on standardized datasets to help you select a model before
deployment. Metrics are the numbers—accuracy, factuality, hallucination rate, etc.—that quantify
response or system quality.

PM lens: Use benchmarks for model selection, evaluations for product quality, and metrics to track
progress and justify roadmap calls.

Example: Before launch, you compare several LLMs on coding/reasoning benchmarks to pick a base
model. Post-launch, you evaluate your chatbot on real refund queries and track semantic
match and helpfulness scores weekly.

Common challenges in LLM apps

LLMs are probabilistic and contextual, so failures can be subtle: responses may be helpful but not
factual, or factual but unempathetic. You also face model drift (provider updates, fine-tuning
changes), safety risks (toxic content, prompt injection), and odd behaviors (e.g., the bot writes a poem
when asked for order status).

PM move: Define failure modes up front and make them visible in dashboards.

Types of evaluations

 Human evaluation: People label outputs as good/neutral/bad. Great for empathy and nuanced
judgment. Example: Raters compare two responses to a damaged-product complaint and prefer
the one that apologizes, confirms policy, and offers a replacement link.

 Automatic evaluation: A model scores outputs using a rubric (helpfulness, empathy, factual
match). Example: Use GPT-4 to score whether a reply includes the correct refund window and
resolution steps.

 A/B testing: Show users two versions in production and compare real behavior. Example: Variant
B with clearer action links reduces re-contacts by 8%.

 Unit test cases / rule checks: Hard guards independent of LLM behavior. Example: Validate order
IDs are numeric, URLs are HTTPS, and PII is masked.

PM lens: Combine all four for coverage: guards prevent obvious errors; human/automatic/A-B reveal
quality and preference.

Building a useful evaluation dataset

Target a balanced mix: 70% real user logs, 20% PM/dev curated, 10% synthetic. Label
by intent/category (order status, payment, refunds) to filter results and find weak spots fast.
Prepared by Vaibhav Yendole 14
Example: After labeling, you discover low scores on “partial refunds,” so you prioritize scripted guidance
and unit checks for that flow.

Semantic vs. syntactic matching (and metrics)

 Syntactic (exact) match: Word-for-word—useful for translation or strict retrieval.

 Semantic match: Meaning-level comparison—best for chatbots.

 Fuzzy match, cosine similarity, BLEU, ROUGE: Tools to handle partial overlap or n-gram similarity.

PM lens: Default to semantic for conversational tasks; track syntactic or fuzzy metrics where precision
matters (e.g., policy clauses).

Practical workflow and tools

Inputs: user query → model response → golden set (expected response). Define rubrics (helpfulness,
empathy, factual match), run manual, automatic, and side-by-side comparisons. Use platforms to
upload datasets and run evaluations; leverage semantic similarity, fuzzy matching, BLEU/ROUGE where
appropriate.

Example: For an airline cancellation (Indigo-style) query, your golden set includes eligibility, fees, and a
CTA. Automatic scoring flags missing fees—your team patches the prompt template.

Continuous monitoring and PM ownership

Set up telemetry to evaluate all user queries in real time. Review low-scoring items weekly, grouped by
labels, to guide fixes, fine-tuning decisions, and comms with stakeholders.

Example: A weekly review shows empathy dips on delayed-delivery cases—add a templated apology
sentence and re-evaluate.

Regression testing and model comparisons

Whenever the provider updates the model or you fine-tune, run regression tests against your golden set
to detect quality regressions—even strong models can backslide on specific intents.

Example: New model version improves reasoning but regresses on refund policy accuracy; you delay
rollout and adjust system instructions.

Best practices for human evaluation

Keep tasks simple and binary (good/neutral/bad). Show full context so judgments are fair. Use clear
rubrics for empathy, helpfulness, and factuality.

Key takeaways

 Use benchmarks to choose models; use evaluations to ship quality.

 Blend methods: human, automatic, A/B, plus unit guards for safety and correctness.

 Curate datasets (70/20/10) and label by intent to scale analysis.

 Prefer semantic matching for chatbots; add syntactic/fuzzy metrics where needed.

 Monitor continuously with telemetry and fix low-scoring patterns first.

 Run regression tests on every model change to prevent silent quality drops.

Prepared by Vaibhav Yendole 15

  Own the rubric: define helpfulness, empathy, and factual match so the team ships with
confidence.

Overview of the Lecture

The lecture focused on Large Language Model (LLM) evaluation, AI metrics, and user testing best
practices. It explored foundational concepts of AI and LLMs, practical approaches for evaluating the
quality of LLM-based applications, the differences between evaluations, benchmarks, and metrics, and
hands-on examples primarily using an Amazon customer support chatbot case study.

1. Recap of Previous Content and LLMs

 The instructor briefly reviewed previous sessions covering AI fundamentals, types of AI (AI, ML,
Deep Learning), prompt engineering basics, advanced prompting techniques (chain of thought,
tree of thought, self-reflection), and foundational understanding of LLMs including self-attention
and softmax.

 Hands-on work was done on fine-tuning models using Hugging Face and building applications with
LLMs.

 A primary question was introduced: How to evaluate if an LLM's response is good, factual, helpful,
or hallucinating?

2. Introduction to LLM Evaluations, Benchmarks, and Metrics

 Evaluations: Focus on assessing the quality of the application output, i.e., how well the LLM-based
solution solves the intended problem at the use-case level. LLM evaluation is context-specific,
such as evaluating an Amazon chatbot’s response to customer queries.

 Benchmarks: Measures the model's capabilities on standardized tasks or datasets (e.g., coding,
reasoning, image generation). Used to select which LLM to use before deployment.

 Metrics: Quantitative measurements at the response or system level, indicating accuracy,

factuality, hallucination rates, etc. Metrics are specialized for generative AI and differ from
traditional product metrics like NPS or retention.

 Clarification was provided on the differences and use cases of evaluations, benchmarks, and
metrics in product management and LLM application deployment.

3. Common Challenges in Evaluating LLM Applications

 LLMs are probabilistic and contextual, unlike rule-based systems, making failure detection
difficult.

 Challenges include:

 Helpfulness of output to users

Prepared by Vaibhav Yendole 16
  Factual correctness and hallucinations

 Model drift over time with updated models or fine-tuning

 Detection of harmful or toxic content and prompt injection attacks

 Handling unexpected behaviors (e.g., generation of irrelevant content like poems in a

customer support chatbot)

4. Types of LLM Evaluations

 Human Evaluation: Humans review model outputs and label responses as good, neutral, or bad.

 Automatic Evaluation: Using one model to evaluate the outputs of another, based on predefined
scoring rubrics.

 A/B Testing: Comparing two model responses side-by-side to understand user preferences during
real interactions.

 Unit Test Cases / Rule-based Checks: Hard-coded validations such as ensuring order IDs are
numeric, email formats are correct, URLs are safe (HTTPS), or personal information is masked.

These evaluations can be combined to form a robust assessment framework for LLM applications.

5. Hands-on Examples and Case Study: Amazon Customer Support Chatbot

 A sample customer query about a damaged product was used to collect human responses and
model responses.

 Participants rated responses and discussed alignment between human and model feedback.

 Examples showed how human judgments might differ due to domain knowledge or empathy,
emphasizing the importance of alignment between model responses and human expectations.

 Real-life examples from companies like Mintra and Indigo were discussed to illustrate domain-
specific challenges.

6. Preparing Evaluation Datasets

 Evaluation datasets are compiled from three sources:

 70% from actual user queries (user logs)

 20% curated by PMs or developers

 10% synthetically generated by LLMs

 PMs play a key role in curating quality data and filtering noise.

 Labeling queries by categories (e.g., order status, payment, refunds) helps filter evaluation results
efficiently.

 Importance of maintaining a broad and representative dataset rather than relying solely on user
queries was highlighted.
Prepared by Vaibhav Yendole 17
7. Unit Test Cases as Hard Guards

 Rules like validating order ID format, checking safe URLs, or masking personal identifiable
information ensure fundamental correctness and safety.

 Unit test cases are critical guardrails independent of LLM behavior and protect against obvious
mistakes or data leaks.

8. Evaluation Workflow and Tools

 Evaluation involves inputs: user query, model response, and a golden set or expected response.

 Scoring criteria and rubrics are defined (e.g., helpfulness, empathy, factual match).

 Tools allow running manual, automatic, and side-by-side comparisons of model responses.

 Models like GPT-4 can be used to score responses automatically without training on the evaluation
data.

 Evaluation setups can include semantic similarity scoring, fuzzy matching, BLEU, ROUGE metrics.

 Hands-on with OpenAI platform showing how to upload datasets and run evaluations.

9. Semantic vs Syntactic Matching and Evaluation Metrics

 Syntactic (Exact) Match: Word-for-word comparison, useful for tasks like translation or retrieval.

 Semantic Match: Compares the meaning behind sentences, ideal for chatbot or customer support
scenarios.

 Fuzzy Match, Cosine Similarity, BLEU, ROUGE: Various metrics to handle partial matches, n-
gram overlaps, or vector similarity.

 Semantic matching was emphasized as the most practical approach for most LLM applications.

10. Role of PMs and Continuous Monitoring

 PMs monitor evaluation outputs regularly, focusing on low-scoring queries to identify failures and
improvement areas.

 Filtering is done using labels and user query categories for scaling.

 Continuous evaluation (telemetry) runs on all user queries, giving real-time metrics for model
performance.

 Evaluations help identify when to fine-tune models or make product decisions.

11. Regression Testing and Model Comparisons

Prepared by Vaibhav Yendole 18

  Regression testing compares new model versions against older ones to ensure improvements or
detect regressions in response quality.

 Example provided where GPT-5 sometimes regressed compared to GPT-2.

 Evaluations can also include detailed assessments of model changes over time.

12. Best Practices in Human Evaluation and Data Presentation

 Keep human evaluation tasks simple and binary: good, neutral, bad.

 Show complete context to evaluators for fairness.

 Avoid partial or masked data that could impede judgment.

 Use clear rubrics for scoring empathy, helpfulness, or factual correctness.

13. Future Sessions and Hands-on Exercises

 Upcoming sessions will deep-dive into benchmarks and metrics like BLEU, ROUGE, perplexity.

 Hands-on exercises planned using Amazon chatbot example, including:

 Setting up golden sets

 Running semantic and fuzzy matching evaluations

 Configuring unit tests and A/B testing

 Use of open-source tools such as Langsmith, OpenAI playground, and Langchain for evaluation
setup.

 Encouragement for students to experiment on their own and share results for feedback.

Summary

The lecture provided a comprehensive understanding of how to evaluate LLM applications effectively. It
clarified the distinction between evaluations, benchmarks, and metrics, explored multiple evaluation
types including human-in-the-loop and automatic scoring, introduced unit test cases as essential safety
checks, and demonstrated practical workflows for dataset preparation and continuous evaluation. These
concepts are critical for managing LLM deployments in production, ensuring quality, safety, and alignment
with user expectations, especially in complex, real-world applications like customer support chatbots.

Introduction to LLM Evaluations

LLM evaluations are critical for assessing the quality of outputs produced by Large Language Model-based
applications. Unlike traditional rule-based systems, LLMs produce probabilistic and context-dependent
outputs, making it essential to evaluate their responses in real scenarios. Evaluations help determine if
the model's output is useful, factual, empathetic, and aligns with the intended use case.

Prepared by Vaibhav Yendole 19

Example A customer asks an Amazon chatbot: "The product I ordered arrived damaged, and I want a
replacement."

 The human support agent replies empathetically with specific instructions.

 The model replies with a generic statement asking the user to return or replace the item through the
orders page.

Evaluations determine the suitability of these responses by rating them as good, neutral, or bad.

Differences Between Evaluations, Benchmarks, and Metrics

Evaluations, Benchmarks, and Metrics are key terminologies in the context of LLM applications but
represent different concepts:

 Evaluations: Assess the quality of the output of an LLM application in a specific use case. E.g.,
human or automatic assessment of an Amazon chatbot's response to a delivery query.

 Benchmarks: Evaluate an LLM model itself, independent of applications, by scoring its

performance on standard tasks (e.g., MMLU, coding benchmarks).

 Metrics: Quantitative measurements (e.g., accuracy, precision, BLEU score, perplexity) used to
quantify the performance of models or applications continuously.

Example:

 When GPT-5 claims higher accuracy on coding benchmarks, that is a benchmark.

 Evaluating whether the GPT-5-powered Amazon chatbot responds accurately to customer queries
is an evaluation.

 Monitoring the daily exact match of chatbot answers to reference answers is a metric.

This differentiation is crucial for product managers and AI developers to understand what to focus on
during model development and deployment.

Types of Evaluations

There are multiple types of evaluations used to assess LLM applications:

1. Human Evaluation: Humans rate model responses (good, neutral, bad) by comparing them to
ideal or expected responses.

2. Automatic Evaluation: Another model evaluates the responses of the primary model using criteria
such as helpfulness, factual accuracy, and empathy.

3. A/B Testing: Two responses from different models or model settings are compared by users to
select the better one.

4. Unit Test Cases (Rule-Based Checks): Predefined rules or functions verify structural integrity or
certain properties, such as correct format of order IDs and masking of personal identifiable
information.

Prepared by Vaibhav Yendole 20

 5. Regression Testing: Comparing model outputs across different versions to check for
improvements or regressions.

Data Preparation for Evaluations

Evaluations rely on properly prepared datasets, which include:

 User Logs (70%): Real queries and interactions collected from actual users.

 Developer/PM Curated Examples (20%): Manually created examples to cover specific scenarios,
including edge cases.

 LLM Generated Examples (10%): Synthetic data generated to increase coverage.

This balanced mix ensures real-world relevance while providing coverage of less frequent cases.

Example dataset entry:

User Input Expected Response Source

"Where is my package?" "You can track your package via orders page." User log

PMs curate this dataset to ensure quality and relevance before running evaluations.

Human and Model-in-the-Loop Evaluations

When evaluating model responses, both human evaluators and automated models (evaluators) can be
used to assign ratings (good, neutral, bad).

The process can be automated:

 Input prompt is sent to the model.

 The model generates a response.

 An evaluation model assesses this response against a reference answer or based on an evaluation
prompt.

A/B Testing

A/B testing involves presenting two model-generated responses (A and B) to users without revealing the
source and recording user preference.

This is critical for:

 Comparing different models (e.g., GPT-3.5 vs GPT-4).

 Comparing different prompt engineering or fine-tuning strategies.

The outcome helps select the model/configuration providing better user satisfaction.

Example scenario:

 User query: "How do I track my order?"

 Response A from GPT-3.5

Prepared by Vaibhav Yendole 21
  Response B from GPT-4

 User selects the more helpful and empathetic answer.

The system logs these preferences to compute aggregated feedback.

Regression Testing in LLMs

Regression testing compares the outputs of different model versions on the same queries to detect:

 Improvements (better responses).

 Regressions (worse responses).

It is especially useful when upgrading the underlying model or changing prompt instructions.

Example comparison:

Model V1
User Query Response Model V2 Response Outcome

"Why was my order "Canceled due to stock. We'll Regression (less

canceled?" "Out of stock." notify you." clear)

Continuous regression testing ensures that newer models maintain or improve quality.

Evaluation Metrics

Metrics quantify aspects of model output quality for continuous monitoring and benchmarking. Some
common metrics include:

 Exact Match: Word-by-word comparison between model output and reference.

 Semantic Similarity (Cosine Similarity): Measures meaning similarity using embeddings.

 Fuzzy Match: Partial matching of word groups to handle paraphrasing.

 BLEU Score: Measures n-gram overlap for translation or generation tasks.

 ROUGE Score: Measures recall-based n-gram overlap used in summarization.

 Perplexity: Language model's uncertainty measure.

Creating Golden Sets

Golden sets are datasets where each input has an 'ideal' or expected output (reference answer).

They are the foundation for reference-based evaluations which compare model outputs against these
golden references.

Example golden set:

Prepared by Vaibhav Yendole 22

User Query Golden Reference Response

"How to return a damaged item?" "You can initiate a return via the orders page."

When a model output matches or semantically aligns with the golden reference, it is considered a pass in
evaluation.

When golden sets are not available, evaluations focus on reference-free metrics such as tone, length,
safety, and toxicity.

Reference-Free Evaluations

When golden sets are unavailable, reference-free evaluation methods assess the output based on
qualitative attributes like:

 Tone: Empathy, helpfulness, politeness.

 Length: Reasonableness of response length.

 Hallucination: Presence of fabricated or incorrect information.

 Safety and Toxicity: Detect harmful or inappropriate language.

These assessments help filter responses that may be factually incorrect or inappropriat

Setting Up Evaluation Workflows

An effective evaluation setup combines several components:

 Prepare balanced datasets (user logs, curated, synthetic).

 Define unit test cases to enforce hard rules.

 Design human or multiple model evaluators with clear rating criteria.

 Configure A/B and regression testing pipelines.

 Define evaluation metrics and thresholds.

 Automate continuous evaluation runs and filtering of low-quality responses.

Example evaluation workflow:

1. User query is received.

2. Input validation and unit tests run.

3. Model generates response.

4. Automated evaluation models rate response.

5. Responses failing unit tests or evaluations are flagged.

6. Sample flagged responses reviewed by PMs or human evaluators.

Such workflows facilitate rapid feedback and model improvements.

Prepared by Vaibhav Yendole 23
Using Eval Tools and Platforms

Tools like OpenAI's Eval framework and Langsmith assist in setting up and running evaluations efficiently.

 Upload datasets (CSV, Excel) with inputs and references.

 Define evaluation prompts, criteria, and metric thresholds via UI or APIs.

 Generate responses and automatically score them.

 Monitor results via dashboards, including overall and per-category performance.

Prepared by Vaibhav Yendole 24

 W17L3 LLM Evaluation, AI Metrics, and User Testing Best Practices (AN)

Why this matters

As a PM building agentic AI features, you’re accountable for outcomes—helpfulness, safety, and business
impact—not just cool demos. LLM evaluations (Evals) give you a repeatable way to measure whether your
models meet product standards before launch and during runtime. With Evals, you can turn fuzzy “seems good”
into evidence you can defend to engineers, compliance, and leadership.

What “evals” mean in practice

Evals are structured tests for LLM behavior using tools like OpenAI’s Eval framework. You design inputs,
expected outputs or criteria, and thresholds for pass/fail. This shifts discussions from opinion to data-backed
decisions about readiness, regression, and model/version choices.

Types of evaluations you’ll use

Unit tests These are simple pass/fail checks: does a reply include a product ID, mask PII, or return a secure URL?
Think of them as guardrails that keep experiences safe-by-default. Example: in a returns workflow, fail any
response that exposes an email or a non-HTTPS link.

Human vs model evaluations You compare a model’s answer to a reference human answer and rate
quality/alignment. This is great for support macros or classification labels where a vetted ground truth exists.

Model vs model (A/B) evaluations You run the same prompts across multiple models and pick the winner.
Useful when you’re deciding whether to upgrade a model or switch providers for better resolution quality on
customer chats.

Reference-based vs reference-free

 Reference-based compares to a “golden” answer—perfect for well-defined tasks like policy

classification.

 Reference-free scores against criteria (helpfulness, politeness, toxicity, length) when no single truth
exists—ideal for conversational assistants.

Preparing the data

Your golden set includes user queries, reference answers, and expected outputs. Sources can be user
logs, developer-curated examples, or synthetic cases generated by an LLM to expand coverage. In enterprise
settings, you’ll often assemble this via Power BI, Cosmos DB, or data lakes, making it easier to refresh evals as
behavior and volumes change.

Metrics you’ll actually track

Accuracy tells you overall correctness, but can hide skew. Precision answers: when we say “yes,” how often are
we right? In support or finance, this is often more critical than accuracy. Recall asks: of all the true “yes” cases,
how many did we catch? F1 Score balances precision and recall when both matter. Example: for a “refund-
eligible” classifier, high precision prevents falsely promising refunds; adequate recall ensures you don’t miss
eligible customers.

False positives vs false negatives

Prepared by Vaibhav Yendole 25

  False positive: The system confidently says refund processed when it wasn’t. This erodes trust and can
trigger costly remediation.

 False negative: Says no refund when one was actually due—customers churn quietly. Root causes often
include wrong domain rules, faulty workflows, or bad knowledge grounding. As a PM, you’ll
prioritize monitoring and reducing false positives in high-impact domains (e.g., finance).

Text similarity for reference-based checks

When ground truths exist, you can score match quality:

 Exact match (string-equal) for high-precision domains like compliance text.

 Fuzzy match tolerates formatting/word-order variance while preserving meaning.

 Semantic match focuses on meaning—ideal for support answers. Popular metrics: BLEU (precision via
n-grams), ROUGE (recall-oriented), METEOR (synonyms-friendly), Cosine similarity (semantic vectors).
You’ll set a cutoff (e.g., 65%) and dig into fails (extra/missing words) to refine prompts or policies.

Hands-on workflow with OpenAI Eval platform

You’ll upload reference data, define unit tests + semantic checks, and set thresholds for pass/fail. Run
automated evaluations (e.g., with GPT-4.1) and review fail cases to decide: prompt tweak, rule change, or model
swap. This becomes your pre-release gate and post-release canary.

Industry applications and prioritization

Evals are domain-specific:

 Finance/healthcare: prefer exact or tightly fuzzy matches and strict guardrails.

 Customer support/conversational AI: rely more on semantic match and courtesy/toxicity checks.

 Recommendations: track precision@K to ensure the top results delight users. You can also
test multiple models in parallel and use a meta-evaluator to select the best output per request.

Integration and continuous improvement

Enterprise SDKs (e.g., Azure Eval SDK) let you embed evals into CI/CD. Telemetry and logging with tools
like Evidently AI help monitor drift and catch regressions in real time. The loop is: measure → analyze → fine-
tune → re-evaluate, feeding directly into roadmaps and incident reviews.

Key takeaways

 Evals turn opinions into evidence, aligning stakeholders on launch/readiness.

 Use unit tests for safety guardrails and reference(-free) checks for quality at scale.

 Optimize precision/recall based on domain risk; watch false positives in high-impact flows.

 Pick Exact/Fuzzy/Semantic matching to fit the task; set clear thresholds (e.g., 65%).

 Treat evals as a living system with CI/CD integration, telemetry, and ongoing fine-tuning.

 Always connect eval results to product decisions: model choice, prompt policy, and roadmap priorities.

Prepared by Vaibhav Yendole 26

 Lecture Summary

Overview of the Lecture

The lecture focused on understanding the evaluation (Evals) of large language models (LLMs),
particularly using tools like OpenAI’s Eval framework. It covered types of evaluations, metrics for
assessment, practical demonstrations, and discussed how these evaluations apply to real-world
applications such as customer support, recommendations, and classification models.

Types of Evaluations in LLMs

 Unit Tests: These are rule-based checks like verifying if the model’s response contains a product
ID, masks personal information, or returns a secure URL. They work as simple pass/fail tests for
specific criteria.

 Human vs Model Evaluations: Compares a model-generated response against a reference human

response, rating the quality and alignment.

 Model vs Model Evaluations (A-B Testing): Compares outputs from different models on the same
input to decide which performs better.

 Reference-Based vs Reference-Free Evaluations:

 Reference-based evaluation compares model output to a known “golden” answer.

 Reference-free evaluation assesses output using criteria like helpfulness, politeness,

toxicity, or length when no ground truth is available.

Data Preparation for Evaluations

 The golden set of data consists of user queries, reference answers, and expected model outputs.

 Data can come from user logs, developer-curated records, or synthetic LLM-generated content.

 In enterprises, data collection uses tools like Power BI, Cosmos DB, or data lakes to aggregate
queries and responses for analysis.

Key Metrics for LLM Evaluation

 Accuracy: Ratio of total correctly predicted responses (true positives and true negatives) to total
responses.

 Precision: Measures how many of the predicted positives were actually correct; focuses on the
relevance of positive predictions. Often more critical than accuracy.

 Recall: Measures how many actual positives were correctly predicted by the model.

 F1 Score: Harmonic mean of precision and recall, balancing both metrics.

Understanding False Positives and False Negatives

 False Positive: Model incorrectly predicts a positive response when it is wrong (e.g., model says
refund processed but it was not).

 False Negative: Model predicts negative (e.g., no refund processed) while the refund was actually
processed or correct information was withheld.

Prepared by Vaibhav Yendole 27

  False positives tend to have a higher impact on user satisfaction because they deliver inaccurate
information confidently.

 Root cause of false positives usually involves incorrect domain knowledge, faulty workflows, or
wrong business rules.

Text Similarity Metrics for Reference-Based Evaluations

 Exact Match: All words must be exactly the same; used in high precision domains like finance,
healthcare, or fraud detection.

 Fuzzy Match: Allows minor variations in formatting or word order but expects the meaning to
remain the same.

 Semantic Match: Focuses on the similarity of meaning, not exact words; used extensively in
conversational AI and customer support.

 Popular scoring metrics include:

 BLEU (Bilingual Evaluation Understudy): Focuses on precision based on n-gram overlap.

 ROUGE (Recall-Oriented Understudy for Gisting Evaluation): Measures recall based on n-

gram overlap, longest common subsequence, etc.

 METEOR: Allows synonym matching, weighting meaning more flexibly than BLEU or ROUGE.

 Cosine Similarity: Measures semantic similarity between vectors representing sentences.

Practical Demonstration Using OpenAI Eval Platform

 Hands-on creation of evaluations based on unit tests and semantic similarity metrics.

 Setting thresholds for passing/failing evaluations, e.g., 65% similarity cutoff.

 Analysing detailed fail cases to understand why an evaluation failed (e.g., extra or missing words).

 Demonstrated how to upload reference data and run automated evaluations using GPT-4.1.

Application of Models and Evals in Industry Contexts

 Evals are domain-specific and often mixed with custom tests tailored to business standards and
workflows.

 Industry type (e-commerce, finance, healthcare) influences which evaluation metrics are
prioritized (e.g., exact match for finance, fuzzy or semantic matches for customer support).

 Multiple models may be tested in parallel with meta-evaluators selecting the best-performing
model output.

 Monitoring false positives is critical for high-impact domains like finance to avoid misleading users.

 Recommendations systems use metrics like “precision at K” to assess quality of top results shown
to users.

Integration and Tooling

 SDKs (e.g., Azure Eval SDK) enable integration of evaluation frameworks into enterprise
applications for end-to-end testing.

Prepared by Vaibhav Yendole 28

  Telemetry and logging tools like Evidently AI help track performance of AI workflows in real-time.

 Metrics and evals feed into fine-tuning and continuous improvement processes for models and
agents.

 The lecture emphasized ongoing hands-on practice and applying these evaluation concepts in real
projects and interview scenarios.

Data Preparation for Evaluations

An essential first step in setting evaluations is preparing the dataset. This involves collecting user queries
and pairing them with the expected correct answers (golden set). Sources for this data include:

 User logs

 Developer-curated records

 LLM-generated content verified by humans

Preparing this labeled dataset allows for reference-based evaluations, where model responses are
compared against known correct answers.

Types of Evaluations (Evals)

There are primarily three types of evals discussed:

1. Unit Test-based Evaluations: Check simple rule-based correctness, e.g., whether the response
contains an order ID or masks personal information correctly.

2. Human vs Model Evaluations: Comparing LLM responses against human-generated answers and
rating their quality.

3. Model vs Model Evaluations (A/B Testing): Comparing different models' responses to the same
query to select the better performing model.

Reference-Based and Reference-Free Evaluations

Evaluations can be categorized into two types based on the availability of reference answers:

 Reference-Based: Known correct answers are available. Evals compare model's output directly
with the reference.

 Reference-Free: No known correct answers exist. Evaluation focuses on other metrics such as
politeness, empathy, helpfulness, length, toxicity, or escalation behavior.

Example:

 Reference-Based: The customer asks "Where is my order?" and we have the expected answer.

 Reference-Free: Customer says "Amazon sucks," and we evaluate if the model responds politely
without toxic language.

Prepared by Vaibhav Yendole 29

 Evaluation Metrics: Precision, Recall, Accuracy, and F1 Score

Fundamental metrics used in evaluating models include:

 True Positive (TP): Model predicts correctly.

 False Positive (FP): Model predicts something is true but it is false.

 False Negative (FN): Model fails to predict true cases.

 True Negative (TN): Correctly identifies negatives.

Metrics:

 Accuracy: Proportion of total correct predictions (TP + TN) to all predictions.

 Precision: Proportion of true positives among predicted positives (TP / (TP + FP)). Focuses on
correctness among positive predictions.

 Recall: Proportion of true positives detected among all actual positives (TP / (TP + FN)).

 F1 Score: Harmonic mean of precision and recall, balances both.

Example Calculation:

precision = TP / (TP + FP)

recall = TP / (TP + FN)

accuracy = (TP + TN) / (TP + TN + FP + FN)

f1_score = 2 * (precision * recall) / (precision + recall)

Understanding False Positives and False Negatives in LLM Responses

Understanding FP and FN with customer support examples:

 False Positive: Model answers a question incorrectly with confidence (e.g., says refund processed
when it is not). This is risky as it misleads the user.

 False Negative: Model refrains from giving an answer or misses a correct answer.

False positives often have a bigger negative impact as they give users wrong information.

Example:

User asks if refund is processed.

 Model incorrectly replies 'Refund processed' (False Positive)

 Model correctly replies 'Refund not processed yet' (True Negative)

Note: False positives are critical to minimize to build trust.

Text Similarity Metrics in Reference-Based Evaluations

Several metrics help compare model output to reference text:

Prepared by Vaibhav Yendole 30

  Exact Match: Checks word-by-word exactness. Useful in financial, medical contexts.

 Fuzzy Match: Allows minor variations in spacing, word order.

 BLEU (Bilingual Evaluation Understudy): Measures n-gram precision, sensitive to exact word
overlap.

 METEOR: Considers synonyms and stemming for better semantic matching.

 ROUGE: Measures recall-oriented overlap, useful for summarization.

 Cosine Similarity: Measures vector similarity of embeddings, capturing semantic similarity.

Unit Test Case Example for LLM Output Validation

Unit tests automate checking output properties. For example, to validate whether the model output
contains an order ID matching a regex pattern.

A/B Testing for Model vs Model Evaluation

A/B testing evaluates outputs of two models on the same input data to identify which performs better. An
evaluator model or human raters score responses to select the superior one.

Example:

Input: "My product arrived damaged, what can I do?"

 Model A Response: "Please initiate a return via your account."

 Model B Response: "Contact support to return or replace your product."

Evaluator assigns a score to each response and identifies the better answer, facilitating model
improvements.

Reference-Free Metrics: Politeness, Toxicity, Helpfulness

When no reference answers are available, models are evaluated on qualitative metrics:

 Politeness: Is the tone respectful and empathetic?

 Toxicity: Does the response contain offensive or harmful language?

 Helpfulness: Does the output address the user's needs suitably?

 Length and Readability: Is the response neither too brief nor verbose, and is it easy to read?

Integration and Automation of Evaluations

Evals run in parallel with the assistant model and are separate systems:

 Evaluations collect model responses and run various checks.

 SDKs are used to integrate evaluations into the data pipeline.

Prepared by Vaibhav Yendole 31

  Continuous logging of model responses enables end-to-end chain testing, crucial for workflows
involving multiple agents or systems.

Example: Using OpenAI Eval platform or Evidently AI for telemetry and evaluation SDK integration.

This separation ensures unbiased and robust measurement of model performance.

Domain-Specific Evaluation Customization

Evaluations often need customization to suit specific industry requirements:

 Financial and healthcare domains require strict exact match and high precision.

 Customer support chatbots may rely more on fuzzy or semantic matches and qualitative
measures.

 Customized evals can include checking if the answer escalates to a human when necessary or
complies with internal policies.

Teams experiment to find suitable metrics balancing precision, recall, and user satisfaction.

Example: For a financial chatbot: exact match for transaction references and masking of personal info.
For a customer support bot: empathy and resolution measured instead.

Practical Hands-On Exercises and Tools

Hands-on practice is recommended to reinforce theoretical concepts. Suggested tools and platforms
include:

 OpenAI Eval platform: Create custom evals with uploaded reference data and Python unit tests.

 LMQL and LangSmith: Advanced prompt engineering and evaluation frameworks.

 Detoxify: Toxicity detection.

 Integration with Power BI, Cosmos DB, or data lakes for logging and dataset preparation.

Instructions provided help users create evals, run semantic similarity checks, and automate unit tests.

Engaging in coding exercises combining UI/UX prototyping, agent creation, and eval integration is
encouraged.

Prepared by Vaibhav Yendole 32

 W18L1 Tech Basics: Databases, APIs, Frontend, Backend, and Agentic AI (Prof S)
Pre-class resources: Tech Basics: Databases, APIs, Frontend, Backend, and Agentic AI

What You'll Gain from This Pre-Read

After reading, you'll be able to:

 Recognize the four main building blocks of technology: frontend, backend, APIs, and databases

 Understand what AI Agents and Agentic AI mean, and how they differ

 Explain what Vibe Coding is and how tools like Lovable change software creation

 Identify how agentic tools are transforming SaaS development

 Engage confidently in discussions about modern tech and AI

Think of this as: Learning the alphabet of modern technology — not fluency yet, but enough to recognize
words and meaning when you see them again.

What This Pre-Read Covers (and What It Doesn't)

This pre-read will:

 Introduce core concepts and vocabulary that power modern apps and AI systems

 Explain why these technologies matter in real products and jobs

 Show foundational examples of AI-driven tools in action

 Build curiosity and orientation

This pre-read will NOT:

 Teach you to code

 Go deep into complex architectures or algorithms

 Replace hands-on practice or project experience

Part 1: The Big Picture - Why Does This Matter?

Imagine opening Netflix — within seconds, personalized recommendations load instantly.

Behind this simple experience are four key components:

 Frontend: What users see and interact with

 Backend: The server logic processing requests

 API: The messenger connecting systems

 Database: Where your data lives

Now, a fifth layer — AI Agents — makes systems smarter, automating decisions and learning
continuously.
Prepared by Vaibhav Yendole 33
 Why this matters:
Every digital product relies on these building blocks. Understanding them helps you think like a
technologist and see how AI is transforming the stack.

Where You'll Use This:

 Job Roles: Product Managers, Full Stack Developers, AI Designers

 Real Products: Netflix, Notion, Spotify, Replit

 What You Can Build: Chatbots, dashboards, automation tools

Think of it like this:

A restaurant analogy —
Frontend is the waiter, Backend the kitchen, API the order slip, Database the pantry, and AI Agent the
chef who learns from every dish.

Limitation: Unlike restaurants, software scales instantly to millions.

Part 2: Your Roadmap Through This Topic

Here’s what we’ll explore together:

1. Core Layers of Technology — You’ll discover how frontend, backend, APIs, and databases
interact.

2. AI Agents vs Agentic AI — Learn how AI moves from automation to autonomy.

3. Vibe Coding — Explore how AI-powered development tools simplify building software.

4. Agentic Tools for SaaS — See how AI tools transform SaaS creation and deployment.

The journey: From understanding the stack → to AI augmentation → to agentic innovation.

Part 3: Key Terms to Listen For

Frontend: The visible part of an app you interact with.

Backend: The unseen logic and processes that make apps work.

API (Application Programming Interface): The bridge that lets systems communicate.

Database: Structured storage for user data and content.

AI Agent: A program that takes actions toward goals autonomously.

Agentic AI: Advanced AI that can plan, act, and adapt dynamically.

Key Insight: Every modern product combines these layers — agentic AI is now managing them for us.

Part 4: Concepts in Action

Example 1: Agentic AI in SaaS

Prepared by Vaibhav Yendole 34

 The Situation:

A SaaS company automates weekly updates.

The Challenge:

Manual testing and deployment caused delays.

How Agentic AI Applied:

1. Detects new code and plans steps.

2. Tests and deploys automatically.

3. Learns from past outcomes.

Outcome: Faster releases, fewer errors, and creative freedom for engineers.

What This Shows: Agentic AI improves SaaS reliability and speed.

Example 2: Vibe Coding with Lovable

The Situation: Developers describe projects in plain language.

Process: AI interprets goals, writes code, shows previews, and accepts refinements.

Result: Code built in minutes — no syntax mastery required.

What This Shows: Coding becomes collaboration, not instruction.

Part 5: How This Topic Connects

Builds On: None — start here!

Enables:

 Understanding system architecture

 Communicating with engineers

 Exploring AI-assisted software creation

Related Concepts:

 Full Stack Development

 LangChain and Crew AI

 SaaS Product Architecture

Questions to Keep in Mind

1. How does understanding these layers change how you see apps?

2. What happens when AI starts writing and deploying software autonomously?

3. How might agentic tools reshape careers in tech?

Prepared by Vaibhav Yendole 35
 Reflect: Think of one app you use daily — which parts might already be AI-driven?

Quick Self-Check: Did It Click?

After reading, you should be able to:

 Explain how frontend, backend, APIs, and databases connect

 Define what makes an AI Agent “agentic”

 Describe what Vibe Coding means

 Give one example of how agentic tools improve SaaS

 Identify what you’re curious to explore deeper

How to Read These Notes Effectively

DO:

 Read actively and connect concepts to real apps

 Skim once, then reread carefully

 Ask “how would I use this?”

DON’T:

 Try to memorize terms

 Skip examples or analogies

What's Next?

Explore Deeper:

 Build a small app with Replit or Lovable.

 Watch: Merlin AI – AI Agents Fundamentals (YouTube)

Final Thought

You’ve mapped the foundation of modern tech — from databases and APIs to AI and vibe coding.

Every time you use an app, you’re seeing these layers in action.

The next generation of software won’t just run — it will think.

And now, you’re ready to think with it.

Prepared by Vaibhav Yendole 36

 Lecture Note: Tech Basics — Databases, APIs, Frontend, Backend, and Agentic AI

What You’ll Be Able to Do:

 Explain how Databases, APIs, Frontend, and Backend work together in applications.

 Identify where Agentic AI fits into the modern tech ecosystem.

 Map the flow of information across systems using real examples.

 Describe common challenges and design trade-offs in digital product systems.

1. Introduction: Why Understanding the Tech Stack Matters

Core Definition

Every modern product—Netflix, Zomato, WhatsApp—is powered by a tech stack, a system of layers that
work together to deliver seamless digital experiences.
These layers include the Frontend (what users see), Backend (how things work), Database (where data
lives), and APIs (how everything talks).

A new evolution, Agentic AI, adds a layer of intelligence that can automate, reason, and act across these
systems.

A Simple Analogy

Imagine a restaurant:

 Frontend = The waiter and menu — what customers interact with.

 Backend = The kitchen — where orders are processed.

 Database = The pantry — where ingredients and recipes are stored.

 API = The order slip — how the waiter and kitchen communicate.

 Agentic AI = The smart restaurant manager — predicts customer demand and automates
restocking.

Limitation: This analogy ignores technical details like concurrency and scaling, but it helps you form a
mental model of the system’s flow.

Why This Matters

 Problem it solves: Helps you bridge the gap between business and tech.

 What you gain: Ability to visualize systems end-to-end, talk with engineers confidently, and design
more intelligent products.

 Real-world use: Every feature you use—from Instagram reels to ChatGPT—depends on this
interconnected system.

2. The Foundation: Understanding Each Layer

Concept A: Database — The Memory

Prepared by Vaibhav Yendole 37

 Definition:
A database stores and retrieves information for applications. It’s the persistent memory of the system.

Key Characteristics:

 Stores user, product, and transaction data permanently.

 Uses structured (SQL) or flexible (NoSQL) formats.

 Ensures reliability and data integrity.

Example:
In an e-commerce app, the database holds products, orders, and customer data.

Common Confusion:
The database stores data but does not process logic. Beginners often confuse it with the backend.

Mini Exercise:
Name one app you use daily. What kind of data do you think its database stores?

Concept B: Backend — The Logic Engine

Definition:
The backend is the system’s brain—it processes requests, applies rules, and interacts with the database.

Key Characteristics:

 Built using Python, [Link], Java, or similar languages.

 Executes business logic like authentication, payments, and calculations.

 Returns data through APIs.

Example:
When you log into Instagram, the backend checks your credentials, fetches your posts, and sends them
to your device.

Check Your Understanding:

Why does the backend handle validation instead of the frontend?
→ Because it controls access, preventing tampering or misuse.

Concept C: API — The Bridge

Definition:
An API (Application Programming Interface) defines how two systems communicate. It’s like a contract
between them.

Key Characteristics:

 Uses methods like GET (read) and POST (create).

 Returns structured data, often in JSON.

 Enables modular development and third-party integration.

Prepared by Vaibhav Yendole 38

 Example:
When a food delivery app shows maps, it calls the Google Maps API to fetch location data.

Common Confusion:
APIs don’t store or compute—they only deliver information.

Concept D: Frontend — The Face

Definition:
The frontend is what users see and interact with—buttons, menus, text, and visuals.

Key Characteristics:

 Built using HTML, CSS, and JavaScript frameworks like React.

 Handles user input and displays backend responses.

 Focuses on design, speed, and accessibility.

Example:
When you click “Add to Cart,” the frontend sends an API call to the backend to update your order.

Quick Check:
What happens after you click “Login” on a website?
→ Frontend collects credentials → API sends data → Backend verifies → Database confirms.

Concept E: Agentic AI — The Autonomous Layer

Definition:
Agentic AI is the next evolution of software systems—AI that can plan, decide, and act using multiple
tools autonomously.

Key Characteristics:

 Works with goals rather than fixed commands.

 Uses APIs and databases like a human assistant would.

 Learns from results to improve over time.

Example:
A finance AI that checks your expenses, identifies overspending, and automatically moves money into
savings.

Common Confusion:
Agentic AI ≠ Chatbot.
Chatbots respond; Agentic AI acts by executing multi-step workflows.

Reflection:
Can you think of an AI app that “acts” for you instead of just chatting?

3. Seeing It in Action: Worked Example

Prepared by Vaibhav Yendole 39
 Example 1: A Travel Planner App

Scenario:
You’re designing an app that helps users plan trips.

Flow:

1. Frontend: User enters destination and dates.

2. Backend: Validates data and fetches suggestions.

3. API: Connects to external services (Flights, Hotels).

4. Database: Stores preferences and past trips.

5. Agentic AI: Builds a personalized itinerary using APIs and user data.

Key Takeaway:
Each layer handles a unique part of the journey—Agentic AI connects them intelligently.

Check Your Understanding:

Which layer retrieves flight data?
→ The API.

Example 2: Real-World Use — Netflix

 Frontend: Movie catalog interface.

 Backend: Decides recommendations and fetches video URLs.

 Database: Stores viewing history.

 API: Bridges the player and server.

 Agentic AI: Predicts what you’ll like next and adjusts the homepage dynamically.

Learning Point:
Even global platforms follow this same structure—the difference is scale and intelligence.

4. Common Pitfalls and How to Avoid Them

Mistake Why It Happens How to Fix It

Confusing backend and Backend applies logic; database

Both deal with data
database stores data

Ignoring API Miscommunication between

Always define endpoints clearly
documentation systems

Keep frontend lightweight; delegate

Overloading frontend Trying to add too much logic
work to backend

Prepared by Vaibhav Yendole 40

 Mistake Why It Happens How to Fix It

Misunderstanding Agentic Treat it as a system of connected

Thinking it’s magic
AI reasoning tools

If You’re Confused:
Revisit the “Foundation” section and draw a diagram—seeing data flow helps fix mental gaps.

5. Practice & Self-Assessment

Practice Task (15–20 minutes)

Challenge:
Design a “Smart Library App” architecture using these 5 layers.

Include:

 Database: What data does it store?

 API: Example endpoint (GET /books/available).

 Backend: What rules does it apply?

 Frontend: What features does the user see?

 Agentic AI: How can it personalize recommendations?

Hint:
Start with the user action (“Search a book”) and map each layer’s role.

Quick Check Questions

1. What is the role of an API in the system?

2. Why can’t frontend and database talk directly?

3. How does Agentic AI extend automation?

4. Give one real-world example where multiple APIs work together.

5. Which part of the stack would handle data validation?

Sample Answers:

1. Acts as a communication bridge.

2. Security and data structure enforcement.

3. By reasoning and acting autonomously.

4. Food apps combining Maps + Payment APIs.

5. Backend — it applies business rules.

Prepared by Vaibhav Yendole 41

 Self-Assessment Checklist

You’ve mastered this if you can:

 Explain each layer clearly and sequentially.

 Visualize how APIs link components.

 Identify Agentic AI’s distinct role.

 Create a basic app flow diagram.

 Spot and correct common misconceptions.

If you checked fewer than 4 boxes, review Sections 2 and 3 before moving forward.

6. Consolidation: Key Takeaways & Next Steps

Key Recap

 Database: Stores data persistently.

 Backend: Processes logic and connects systems.

 API: Enables communication.

 Frontend: Delivers interaction.

 Agentic AI: Automates and optimizes across systems.

Mental Model

Think of an app as a human body:

 Frontend = Eyes and mouth

 API = Nerves

 Backend = Brain

 Database = Memory

 Agentic AI = Conscious intelligence

What You Can Now Do

You can:

 Map how any app works internally.

 Identify where to use automation.

 Explain how data moves through a tech system.

Next Steps

 Explore real APIs like OpenWeather or Google Maps.

 Learn simple CRUD operations in databases.

 Experiment with Agentic AI tools such as Lovable, [Link], and Replit Agents.
Prepared by Vaibhav Yendole 42
 Overview of the Lecture

The lecture focused on the comprehensive design and development of a technology product aimed at
automating the evaluation of handwritten exam papers using multiple technical components including
front end, back end, APIs, Optical Character Recognition (OCR), and AI models. It explored the product
management perspective of breaking down the user journey, understanding the interaction between
various parts of the product, and incorporating AI services and cloud infrastructure to build a scalable,
reliable product.

Product Idea and Use Case

 The core problem addressed is the time-consuming manual evaluation of handwritten exam
papers.

 The product aims to automate evaluation by:

 Scanning all answer sheets into one multi-page PDF using a high-speed scanner.

 Using OCR to convert handwritten text into digital text.

 Applying AI for automatic scoring based on a rubric or instructor's answer keys.

 The product is designed to be user-friendly, requiring minimal manual work from instructors (e.g.,
just uploading the scanned PDF).

 Essential to the product is flexibility for different exam scenarios, including handling ambiguous or
missing registration numbers on pages and non-text answers such as diagrams.

Front End Design and User Journey

 The product interface consists of about 7-8 screens:

i. Program Details: Input program name, batch, course.
ii. Upload Scanned Documents: Upload the large PDF from the scanner.
iii. Upload Rubrics and Answer Key: Enter or upload grading criteria.
iv. Processing Screen: Display process status (OCR, splitting).
v. Review and Results Screen: View AI-generated scores and reports.
 Additional screens for manual intervention during ambiguous or unassigned pages.
 The front end prioritizes straightforward input with minimal manual data organization.

Back End Processing

 Folder and File Management: Creates structured cloud storage folders (e.g., Azure blob storage)
by program, batch, course, timestamp to store uploaded and processed files.

 OCR (Optical Character Recognition):

Uses cloud-based document intelligence services to convert handwritten PDFs to digital


text efficiently.
Prepared by Vaibhav Yendole 43
  Supports concurrent worker threads for processing large documents quickly.

 Document Splitting:

 Identifies registration numbers written on each page using pattern recognition via regular
expressions.

 Splits the large PDF into individual student PDFs, handling ambiguous and missing
registration numbers with manual override features.

 AI-based Evaluation:

 Converts unstructured answer keys into structured rubrics using AI (e.g., GPT-4).

 Each student’s answer sheet is compared against the rubric to assign scores and provide
detailed feedback.

 Multiple AI models (GPT-4, LLaMa, Grok) evaluate responses independently to ensure

reliability.

 A meta AI (GPT-5) aggregates evaluations and generates confidence scores for each grading
to flag uncertainties for manual review.

 Reporting:

 Produces detailed reports, including marks per question, rationale for scores, and overall
performance.

 Allows exporting results (e.g., Excel format).

Use of APIs and Cloud Services

 Utilizes APIs from cloud providers such as Microsoft Azure for:

 Storage and backup.

 OCR via document intelligence.

 Hosting and calling AI models for grading.

 Product management involves deciding which services to build in-house versus using existing APIs
based on cost, speed, accuracy, and scalability.

 Emphasizes research on various API providers and auditing terms of service for data privacy and
security (important since student data is sensitive).

 Discussed concurrency and optimizing API calls for scalability when processing thousands of
pages.

Product Management Insights

 The product design starts with understanding the entire user journey, including actions outside the
application, like exam paper collection and scanning.

 The user experience must minimize manual effort while the backend handles complexity.
Prepared by Vaibhav Yendole 44
  Testing is critical to validate AI model selection and weighting for accuracy and consistency before
market launch.

 Product must be flexible to handle real-world exam scenarios such as out-of-order pages, extra
pages, and diagrams.

 Importance of maintaining data privacy and selecting APIs with appropriate data usage policies.

 Iterative product design: Begin with rough wireframes/textual flow, refine with backend logic
integration.

 Ensure the product is scalable, secure, and commercially viable.

Technical Challenges and Solutions Discussed

 Handwriting recognition accuracy: Using cloud-based OCR and AI to handle difficult or poor
handwriting.

 Page sequence and ambiguous pages: AI-assisted sequence reordering and manual correction
interfaces.

 Multi-AI model grading: Combining multiple model outputs to improve reliability and confidence.

 Handling diagrams and images in exams: Future feature to detect and process non-text answers.

 Optimization of OCR speed: Using parallel processing with concurrency/workers.

 Cost considerations: Balancing between API usage costs and in-house development.

Next Steps and Upcoming Topics

 Further exploration of APIs will be covered in the next session.

 Hands-on exercises creating API connections using platforms like base44.

 Continued emphasis on developing the product with scalable and secure architecture.

 Encouragement for students to experiment with app creation and API usage.

This lecture provides a thorough framework for product managers and developers on how to
conceptualize, design, and implement an AI-powered exam evaluation system, integrating frontend UI
design, backend processing, APIs, and multiple AI services efficiently.

Product Design and User Journey

Product design begins with understanding the complete user journey from start to finish, not just
focusing on the technical implementation. For example, in the evaluation automation product:
 The journey starts even before the exam answer sheets reach the system (students writing exams).

 The system should simplify the manual tasks for the end user, such as instructors uploading
documents.

Prepared by Vaibhav Yendole 45

  The product should handle scenarios like bulk uploading of hundreds of scanned pages into a
single PDF and automate splitting and evaluation.

A product manager should think about all possible scenarios, edge cases, and user pain points to make
a product truly helpful and commercially viable.

Optical Character Recognition (OCR)

OCR technology converts handwritten or scanned documents into machine-readable digital text.

 This system uses cloud-based OCR services like Azure Document Intelligence.

 OCR is run on the entire scanned PDF.

 The converted document is stored separately in a digital format.

 To speed up the OCR process for large documents, it is split into smaller batches and processed
concurrently using worker threads.

 If OCR fails or returns partial results, manual intervention or AI-assisted correction is employed.

AI-Generated Rubric and Grading

Rubric creation and grading are automated using AI models.

 Instructor uploads an unstructured or structured answer key and rubric.

 The system sends these inputs to AI models (e.g., GPT-4), via API, with detailed prompts to generate
structured evaluation rubrics.

 Student answers are then graded by sending both the student's answer and the rubric to multiple AI
models for assessment.

 Responses from different models (GPT-4, Llama, Grok, etc.) are combined to improve accuracy and
reliability.

 A final evaluation is generated with a confidence score, highlighting scores that need manual
review.

Product Manager Role and Design Thinking

Throughout the product development process, the product manager must:

 Understand the problem thoroughly and consider real user scenarios.

 Design the user journey focusing on automation and reduction of manual workload.

 Decide when to build in-house features versus use external APIs based on cost, reliability, and
speed.

 Ensure the product can scale and handle irregularities (ambiguous pages, missing registration
numbers).

 Incorporate validation mechanisms like confidence scoring and multi-model agreement to improve
trust.

 Consider security and data privacy when selecting third-party services.

Prepared by Vaibhav Yendole 46
 W18L2 AI & Agents : Lovable, Replit, Cursor, GHCP and Bolt (AN)
Lecture Summary

Overview of the Lecture

The lecture focused on the emerging concept of "vibe coding" — an AI-assisted approach to software
development that enables product managers (PMs) and others with little to no coding knowledge to
create fully functional applications using AI tools. The session covered foundational technology
concepts, different vibe coding tools, workflows, best practices, and practical steps to start building
applications via AI agents. The lecture also included live demonstrations, Q&A, and shared valuable
learning resources.

Foundational Technology Concepts

 Client-side (Front-end): What the user interacts with — buttons, colors, UI elements. Built
typically with HTML, CSS, JavaScript, React.

 Server-side (Back-end): Backend stores data, business logic, user authentication, and security.
Common back-end technologies include [Link], Python, databases like SQL, MongoDB,
Supabase.

 API: Acts as a messenger between front-end and back-end (like a waiter taking orders from client
and delivering them to the server).

 Full-stack Development: Involves both front-end and back-end development to create complete
applications.

 Databases:

 Relational (SQL) databases are structured with relations and avoid redundancy.

 Non-relational (NoSQL like MongoDB) are better for high-volume, rapid, unstructured data
(e.g., chat messages). Hybrid approaches are common in real-world applications (e.g.,
fintech).

 Authentication vs Security: Authentication verifies user identity (via OTP, email, biometrics), while
security protects the data and access rights. Security methods include tokens, OAuth, HTTPS
encryption, and end-to-end encryption (e.g., WhatsApp).

Introduction to Vibe Coding

 Defined by Andrukapathy, vibe coding is a paradigm that focuses on what to build using AI agents
while abstracting away the how (the actual coding).

 Encourages PMs and non-technical users to build end-to-end applications from ideas, sketches,
PRDs, or screenshots without writing traditional code.

 Supports quick prototyping, iteration, and experimentation, enabling faster product validation and
cross-functional alignment between product, design, and engineering.

Prepared by Vaibhav Yendole 47

  Effective vibe coding requires a clear and well-defined what & why; unclear ideas lead to poor
outputs.

Tools and Platforms for Vibe Coding

 Cloud-based Platforms: Replit, Lovable, Bolt, Vercel. These provide full-stack environments
accessible via browsers with no local setup.

 Replit: Excellent for data-heavy apps, supports Python, JavaScript.

 Lovable: Great for aesthetic, creative UI, easy GitHub integration & deployment.

 Bolt: Ideal for converting screenshots/sketches to prototypes quickly.

 Vercel: Primarily front-end focused and hosting platform.

 Local AI Assistants: Cursor, Windsurf, Kiro from Amazon, GitHub Copilot, GitHub Spark. These run
on desktops, suitable for developers who want more control over code and deployment.

 Chatbots (GPT, Claude, Poe, etc.): Useful for ideation, debugging, understanding code snippets,
less for full production apps.

Vibe Coding Workflows and Use Cases

 Four Main Approaches to Vibe Coding:

i. Screenshot to Prototype: Upload a screenshot of an app and ask AI to add features or

create prototypes.

ii. Sketch to Prototype: Draw app screens by hand, take a photo and convert into interactive
UI.

iii. PRD to Prototype: Upload product requirement documents which AI can use to generate
detailed wireframes, UI and eventually working apps.

iv. Figma to Prototype: Use Figma design files to automatically generate code and apps, often
via plugins like [Link] that export designs to platforms like Lovable.

 Demonstrated creating a Spotify-like voice search app, adding "undo send" email feature in Gmail,
building female safety feature in a ride-hailing app.

Best Practices and Interview Preparation

 Vibe coding interviews evaluate structured thinking and prompt creation — candidates must
clearly define the product’s problem, user, journey, and functional/non-functional requirements
before asking AI to generate code.

 Shared an 8-prompt step-by-step guide supporting product discovery to delivery phases that PMs
should follow for systematic vibe coding.

 Encouraged experimentation with multiple platforms using the same project to identify which suits
one’s needs and preferred aesthetics.
Prepared by Vaibhav Yendole 48
  Emphasized continuous iteration, testing features independently (front-end only, back-end only,
full stack, authentication, security).

 Demonstrated managing version control and GitHub integration for deploying, securing, and
maintaining projects.

Q&A and Additional Insights

 Explained differences and examples of SQL Vs NoSQL databases and use in domains like fintech.

 Discussed hybrid database use cases combining NoSQL for rapid ingestion and SQL for relational
processing.

 Covered security concerns of using personal cloud accounts and recommended enterprise license
usage for organizational code privacy.

 Discussed deployment options, including exporting code to GitHub and using CI/CD pipelines with
cloud platforms like AWS and Azure.

 Clarified that all vibe coding tools operate through natural language prompts; deep coding
knowledge is not mandatory.

 Recommended certification courses for vibe coding to build technical credibility.

Summary of Resources and Next Steps

 Shared a document with detailed prompts and instructions for hands-on vibe coding assignments.

 Encouraged participants to complete a full application using vibe coding tools before the next
session.

 Next session planned as a two-hour hands-on practical workshop to build and debug vibe coding
applications together.

 Provided pointers to installation of Figma plugins, GitHub integration, and best practices to
optimize workflow.

This comprehensive session significantly demystifies AI-assisted no-code development, empowering

PMs and non-developers to create and iterate digital products quickly and effectively. It bridges the gap
between idea and deployment through structured workflows and powerful AI tools.

Lecture Note: AI & Agents — Lovable, Replit, Cursor, GitHub Codespaces, and [Link]

What you’ll be able to do:

 Explain what Lovable, Replit, Cursor, GitHub Codespaces, and [Link] are.

Prepared by Vaibhav Yendole 49

  Identify how these browser-based IDEs enable agentic development (and where a desktop AI IDE
like Cursor fits).

 Understand how to build a simple tool that connects frontend, API, and database.

 Learn to publish and share your project using these tools.

1. Introduction: Why Learn AI & Agents in Browser-Based IDEs?

Let’s start simple.

Modern software development has shifted to the cloud — from writing code on your desktop to building
full apps directly in your browser.

With AI-powered agents, developers can now generate, test, and deploy projects faster than ever
before.

What Are Browser-Based IDEs?

IDE stands for Integrated Development Environment — a place where you can write, test, and deploy
code.
Browser-based IDEs like Lovable, Replit, GitHub Codespaces, and [Link] combine coding, AI
assistance, and instant deployment in one place.

Note: Cursor is a desktop AI IDE that pairs well with these cloud tools for local workflows.

Why This Matters

 Reduces setup time — minimal installation or configuration.

 Integrates AI co-pilots for real-time suggestions and code generation.

 Allows seamless collaboration and sharing.

A Simple Analogy

Imagine working on Google Docs — but for code.

You write, the AI suggests, and your project runs instantly — all online.

Limitations: Browser-based IDEs are ideal for web-focused projects. Apps needing heavy native
dependencies, GPUs, or proprietary local drivers may be better suited to a desktop IDE (e.g., Cursor) or a
containerized cloud dev environment (e.g., Codespaces).

2. The Foundation: Core Concepts Explained

Let’s explore each tool one by one and understand what makes them unique.

Concept 1: Lovable

Definition:
Lovable is an AI-first development platform where you can build, connect, and deploy apps directly in
the browser.

Prepared by Vaibhav Yendole 50

 Key Features:

 Built-in AI assistance (e.g., Claude, OpenAI).

 Automatic UI generation from natural language prompts.

 Native integrations for APIs and data (commonly via Supabase for auth and Postgres).

Example:
Type “Create a book tracker with a search bar and database connection” and Lovable scaffolds the UI,
routes, and data hooks.

Common Confusion:
Lovable isn’t a low-code toy — it’s a full development environment with agentic capabilities.

Check Your Understanding:

What makes Lovable different from traditional code editors?

Concept 2: Replit

Definition:
Replit is a collaborative online IDE that lets users code, run, and deploy instantly.

Key Features:

 Supports dozens of programming languages (via Nix).

 Built-in AI assistant called Replit AI (formerly Ghostwriter).

 Real-time collaboration (multiplayer editing and sharing).

 Integrated hosting and deployment.

 Data options like Replit Database and a simple Key-Value store, plus project Secrets for
environment variables.

Example:
Students can build and share a Python or [Link] web server in minutes — no local setup required.

Common Confusion:
Replit isn’t only for beginners — teams use it for prototypes, teaching, workshops, and hackathons.

Check Your Understanding:

How does Replit simplify the process of working on group coding projects?

Concept 3: Cursor

Definition:
Cursor is an AI-powered desktop code editor optimized for deep LLM integration.

Key Features:

 Context-aware coding suggestions beyond standard autocomplete.

 Built-in AI chat for debugging, refactoring, tests, and docs.

Prepared by Vaibhav Yendole 51
  Tight Git/GitHub workflows for local or remote repos.

Example:
While writing a function, Cursor can propose an optimized implementation and explain the reasoning,
then help generate tests.

Common Confusion:
Cursor is not a browser-based IDE. It bridges local development with AI, and complements tools like
Codespaces or Replit for cloud runs.

Check Your Understanding:

What benefit does Cursor’s AI assistant provide compared to standard auto-complete?

Concept 4: GitHub Codespaces

Definition:
GitHub Codespaces provides a full VS Code experience in your browser backed by a configurable dev
container for instant, reproducible environments.

Key Features:

 Containerized cloud environments with instant setup from a repo.

 Consistent toolchains across teams via devcontainer configuration.

 Works seamlessly with GitHub repos, issues, and PRs.

 GitHub Copilot available via the standard VS Code extension.

Example:
Open any GitHub repo in a Codespace, code in the browser, run tests, and push changes — no local
setup.

Common Confusion:
Codespaces isn’t a hosting provider — it’s a development environment integrated with GitHub
workflows.

Check Your Understanding:

Why is Codespaces valuable for teams managing multiple projects?

Concept 5: [Link]

Definition:
[Link] is an AI-native browser IDE to build React/[Link] apps rapidly using natural language prompts.

Key Features:

 Create React or [Link] apps from prompts, then edit in an in-browser editor.

 Integrate APIs and connect to databases (e.g., Supabase).

 One-click publishing to a shareable live URL (e.g., *.[Link]).

Prepared by Vaibhav Yendole 52

 Example:
Prompt: “Build a to-do app with authentication and cloud sync.”
[Link] scaffolds the app with pages, routes, and API handlers and lets you publish immediately.

Common Confusion:
Bolt isn’t just for demos — you can publish working apps quickly, then iterate.

Check Your Understanding:

What makes [Link] ideal for rapid experimentation?

3. Seeing It in Action: Worked Example

Let’s build a small project using Lovable or [Link].

Scenario:
You want to build a “Task Manager” web app that saves tasks and displays them to users.

Approach:

1. Generate the app:

In Lovable (or [Link]), prompt: “Create a task manager with a list view, add/edit forms, and a
REST API.”

2. Build a screen:
Use the generated components for Task List and Task Form; wire basic routes.

3. Integrate API:
Add endpoints like POST /tasks, GET /tasks, PUT /tasks/:id, DELETE /tasks/:id.

4. Connect the database:

Prefer a cloud Postgres via Supabase (auth + tables) for quick setup and production-ready data.

5. Publish:
Deploy from the platform and share the live URL for feedback.

Result:
You’ve built a full-stack app using AI agents with minimal setup.

Key Takeaway:
Browser-based IDEs combined with AI dramatically reduce friction between idea and product.

4. Common Pitfalls: What Can Go Wrong

Mistake Why It Happens Correct Approach

Over-relying on AI Vague instructions produce poor Write clear, step-by-step

prompts output prompts

Ignoring integrations APIs or DBs left unconnected Test connections early

Prepared by Vaibhav Yendole 53

 Mistake Why It Happens Correct Approach

Skipping version control Losing track of edits Use Git/GitHub sync or exports

Publishing without
Deployed apps may break Always preview before sharing
testing

Tip:
When prompting AI tools, always specify what you want, how it connects, and the desired output (e.g.,
framework, routes, data schema).

5. Your Turn: Practice and Self-Assessment

Practice Task

Build a Simple Notes App using Lovable or [Link].

Requirements:

 A screen with a title input and note text area.

 API endpoints (e.g., /notes with create/read/update/delete).

 Database connection (e.g., Supabase table notes).

 Publish and share the app link.

Hint:
Use a detailed prompt like:

“Build a note-taking app with a notes list and editor, store data in Supabase, and expose CRUD
endpoints.”

Check Your Understanding

1. How is Lovable different from [Link]?

2. What role does an API play in connecting frontend and backend?

3. How does Cursor help improve code quality?

4. Why are AI-assisted IDEs faster for beginners?

5. What is the biggest advantage of browser-based development?

6. Summary and Key Takeaways

Recap:

 Lovable: AI-first app builder (web) with prompt-to-UI and Supabase-friendly integrations.

 Replit: Collaborative IDE + deployments; Replit AI for assistance; easy sharing and secrets.

Prepared by Vaibhav Yendole 54

  Cursor: Desktop AI IDE with deep LLM integration for local workflows.

 GitHub Codespaces: Browser VS Code backed by dev containers; great for repo-centric,
reproducible environments with Copilot.

 [Link]: AI-native browser IDE for rapid React/[Link] creation and instant publishing.

Key Lesson:
These platforms use Agentic AI to accelerate coding, integration, and deployment — letting developers
focus on ideas instead of setup.

Mental Model:
Think of these platforms as “AI teammates” — they handle scaffolding, wiring, and repetitive tasks while
you steer direction and quality.

Next Steps:

 Build a small project in Lovable or [Link] to experience prompt-to-product flow.

 Explore Replit or Codespaces for collaborative/team setups.

 Practice writing precise prompts that state frameworks, routes, data models, and deployment
targets.

Overview of the Lecture

The lecture focused on the emerging concept of "vibe coding" — an AI-assisted approach to software
development that enables product managers (PMs) and others with little to no coding knowledge to
create fully functional applications using AI tools. The session covered foundational technology
concepts, different vibe coding tools, workflows, best practices, and practical steps to start building
applications via AI agents. The lecture also included live demonstrations, Q&A, and shared valuable
learning resources.

Foundational Technology Concepts

 Client-side (Front-end): What the user interacts with — buttons, colors, UI elements. Built
typically with HTML, CSS, JavaScript, React.

 Server-side (Back-end): Backend stores data, business logic, user authentication, and security.
Common back-end technologies include [Link], Python, databases like SQL, MongoDB,
Supabase.

 API: Acts as a messenger between front-end and back-end (like a waiter taking orders from client
and delivering them to the server).

 Full-stack Development: Involves both front-end and back-end development to create complete
applications.

 Databases:

 Relational (SQL) databases are structured with relations and avoid redundancy.

Prepared by Vaibhav Yendole 55

  Non-relational (NoSQL like MongoDB) are better for high-volume, rapid, unstructured data
(e.g., chat messages). Hybrid approaches are common in real-world applications (e.g.,
fintech).

 Authentication vs Security: Authentication verifies user identity (via OTP, email, biometrics), while
security protects the data and access rights. Security methods include tokens, OAuth, HTTPS
encryption, and end-to-end encryption (e.g., WhatsApp).

Introduction to Vibe Coding

 Defined by Andrukapathy, vibe coding is a paradigm that focuses on what to build using AI agents
while abstracting away the how (the actual coding).

 Encourages PMs and non-technical users to build end-to-end applications from ideas, sketches,
PRDs, or screenshots without writing traditional code.

 Supports quick prototyping, iteration, and experimentation, enabling faster product validation and
cross-functional alignment between product, design, and engineering.

 Effective vibe coding requires a clear and well-defined what & why; unclear ideas lead to poor
outputs.

Tools and Platforms for Vibe Coding

 Cloud-based Platforms: Replit, Lovable, Bolt, Vercel. These provide full-stack environments
accessible via browsers with no local setup.

 Replit: Excellent for data-heavy apps, supports Python, JavaScript.

 Lovable: Great for aesthetic, creative UI, easy GitHub integration & deployment.

 Bolt: Ideal for converting screenshots/sketches to prototypes quickly.

 Vercel: Primarily front-end focused and hosting platform.

 Local AI Assistants: Cursor, Windsurf, Kiro from Amazon, GitHub Copilot, GitHub Spark. These run
on desktops, suitable for developers who want more control over code and deployment.

 Chatbots (GPT, Claude, Poe, etc.): Useful for ideation, debugging, understanding code snippets,
less for full production apps.

Vibe Coding Workflows and Use Cases

 Four Main Approaches to Vibe Coding:

i. Screenshot to Prototype: Upload a screenshot of an app and ask AI to add features or

create prototypes.

ii. Sketch to Prototype: Draw app screens by hand, take a photo and convert into interactive
UI.

Prepared by Vaibhav Yendole 56

 iii. PRD to Prototype: Upload product requirement documents which AI can use to generate
detailed wireframes, UI and eventually working apps.

iv. Figma to Prototype: Use Figma design files to automatically generate code and apps, often
via plugins like [Link] that export designs to platforms like Lovable.

 Demonstrated creating a Spotify-like voice search app, adding "undo send" email feature in Gmail,
building female safety feature in a ride-hailing app.

Best Practices and Interview Preparation

 Vibe coding interviews evaluate structured thinking and prompt creation — candidates must
clearly define the product’s problem, user, journey, and functional/non-functional requirements
before asking AI to generate code.

 Shared an 8-prompt step-by-step guide supporting product discovery to delivery phases that PMs
should follow for systematic vibe coding.

 Encouraged experimentation with multiple platforms using the same project to identify which suits
one’s needs and preferred aesthetics.

 Emphasized continuous iteration, testing features independently (front-end only, back-end only,
full stack, authentication, security).

 Demonstrated managing version control and GitHub integration for deploying, securing, and
maintaining projects.

Q&A and Additional Insights

 Explained differences and examples of SQL Vs NoSQL databases and use in domains like fintech.

 Discussed hybrid database use cases combining NoSQL for rapid ingestion and SQL for relational
processing.

 Covered security concerns of using personal cloud accounts and recommended enterprise license
usage for organizational code privacy.

 Discussed deployment options, including exporting code to GitHub and using CI/CD pipelines with
cloud platforms like AWS and Azure.

 Clarified that all vibe coding tools operate through natural language prompts; deep coding
knowledge is not mandatory.

 Recommended certification courses for vibe coding to build technical credibility.

Summary of Resources and Next Steps

 Shared a document with detailed prompts and instructions for hands-on vibe coding assignments.

 Encouraged participants to complete a full application using vibe coding tools before the next
session.

Prepared by Vaibhav Yendole 57

  Next session planned as a two-hour hands-on practical workshop to build and debug vibe coding
applications together.

 Provided pointers to installation of Figma plugins, GitHub integration, and best practices to
optimize workflow.

This comprehensive session significantly demystifies AI-assisted no-code development, empowering

PMs and non-developers to create and iterate digital products quickly and effectively. It bridges the gap
between idea and deployment through structured workflows and powerful AI tools.

Introduction to Vibe Coding

Vibe coding is a new paradigm of building digital products where the focus shifts from writing actual
code to giving natural language instructions or prompts to AI agents. Instead of worrying about the
syntactical details of coding, developers and product managers express what they want to build, and AI
tools generate the code and build the application.

This concept is centered around the idea of "forgetting the how" of coding and embracing exponential
technologies represented by Large Language Models (LLMs) such as GPT, Claude, or Gemini.

Example:

Imagine you want to build a to-do app but instead of coding React and backend logic, you just tell the AI:
"Create a to-do list app with add, edit, delete tasks, and user login."

This approach accelerates product development and bridges gaps between design, engineering, and
marketing by quickly providing functional prototypes that can be tested and deployed.

Fundamentals of Front End and Back End

Front End (Client Side): This is what the user interacts with—the UI components like buttons, layouts,
colors, and user inputs.

Back End (Server Side): This pertains to the logic, data storage, business rules, authentication, and APIs
that serve and process user requests.

A common analogy is to a restaurant:

 Client side is the dining area where customers place orders.

 Server side is the kitchen that prepares the food.

 The API is like the waiter who takes the order and brings back the food.

Technologies typically used:

 Front end: HTML, CSS, JavaScript, React.

 Back end: [Link], Python, Java.

 Databases: MongoDB (NoSQL), MySQL, SQL Server (SQL), Supabase.

Prepared by Vaibhav Yendole 58

 Databases: SQL vs NoSQL

Relational Databases (SQL):

 Use structured tables with clear relationships (foreign keys).

 Optimize for data consistency and reduce redundancy.

 Examples: MySQL, PostgreSQL, Microsoft SQL Server.

Non-relational Databases (NoSQL):

 Use flexible document or key-value storage.

 Ideal for high volume, fast, unstructured data like chat messages.

 Examples: MongoDB, Firebase.

When to use:

 Use SQL when data consistency and relationships are critical.

 Use NoSQL for high-speed or unstructured data.

APIs: The Bridge Between Client and Server

APIs (Application Programming Interfaces) act as the messengers or waiters that accept requests from
the client side and fetch or update data from the server or database.

They enable communication between front end and back end. Common API types used are RESTful APIs.

Example:

A REST API endpoint to fetch all products:

Authentication and Security

Authentication: This is the process of verifying user identity before granting access. Common methods:
OTP via mobile, email/password, biometric (fingerprint, facial recognition).

Security: Protects the data from unauthorized access and ensures data integrity. Techniques include
encryption (e.g., HTTPS, end-to-end encryption), tokens, OAuth.

Example:

Simple mobile OTP authentication flow:

1. User inputs phone number.

2. Server sends OTP (One Time Password).

3. User submits OTP to server.

4. Server verifies OTP to authenticate user.

Security example - JWT token generation:

Prepared by Vaibhav Yendole 59

 Full Stack Developer and Role in Vibe Coding

Full Stack Developer: Someone who can build both front end and back end components of an
application including database and APIs.

In vibe coding, the AI acts as a full stack developer, converting your natural language ideas into a
complete working application end-to-end including UI, logic, database, and deployments.

Example: You instruct an AI tool: "Build a blog website with user login, post creation, and comments."
The AI produces:

 Front end React components for the UI

 Backend in [Link] for processing and serving data

 Database schema for storing users, posts, comments

Tools and Platforms for Vibe Coding

There are various types of vibe coding platforms:

1. Cloud Platforms (no setup needed, full stack support):

 Lovable: Excels at beautiful, aesthetic websites and integrates with GitHub for deployment.

 Replit: Suitable for data-heavy applications, supports Python, JavaScript.

 Bolt & Versal: Bolt great for screenshots/sketches to prototypes; Versal focuses on front
end.

2. Chatbot-based Agents (good for ideation and debugging):

 ChatGPT, Claude, etc. These can generate code snippets and help explain or debug code.

3. Local Desktop Assistants (for more control, coding experience needed):

 Cursor, Windsurf, Kiro — run locally, require coding competency.

Example: Using Lovable to transform a Figma design to working UI:

 Upload your Figma file

 AI generates code

 Preview and publish

Vibe Coding Workflows / Use Cases

Four main workflows to build with vibe coding:

1. Screenshot to Prototype: Take a screenshot of an existing app or design, add new features by
describing your requirements, and get a functional prototype.

2. Sketch to Prototype: Draw rough UI sketches on paper, upload the image, and convert it into
working UI.

Prepared by Vaibhav Yendole 60

 3. PRD to Prototype: Upload Product Requirement Document, AI reads it and creates prototypes and
application.

4. Figma Prototype to App: Import Figma design, AI generates responsive front end with optional
backend integration.

Example: Sketch to Prototype:

 Sketch app screens on paper

 Take photo and upload to a vibe coding platform

 Add instructions like "make this button clickable to add item to cart"

 Platform generates interactive prototype

PRD (Product Requirement Document) to Application

Vibe coding tools can directly interpret natural language PRDs to create working prototypes and
applications.

By feeding a detailed PRD, including user personas, features, and flows, AI generates UI wireframes,
writes the front and back end code, and integrates databases.

Example:

Given a PRD describing a coupon wallet app:

 Users receive coupons

 Coupons can be added to wallet

 Coupons can be redeemed at stores

The tool generates screens for coupon list, coupon details, add to wallet button, and backend APIs for
managing coupon status.

This enables rapid prototyping and validation without writing code manually.

 Discover: Is the problem worth solving? Define the market and user pain points.

 Define: Gather user journeys, prioritize features.

 Develop: Create UI wireframes, prototype screens.

 Deliver: Build and deploy fully functional apps.

Discover Phase: Analyze market potential for a food delivery app targeting college students in Tier 1
cities.

Define Phase: Create user journey map for ordering, payment, and delivery.

Develop Phase: Generate wireframes for home screen, cart screen, and checkout screen.

Deliver Phase: Build front-end and back-end code with database integration.

Prepared by Vaibhav Yendole 61

 W18L2 4D Product Development with GenAI Tools (AN)
Pre-class resources: 4D Product Development with GenAI Tools

What you’ll learn

In this session we’ll use a simple, end-to-end workflow—Discover → Define → Develop → Deliver—and
supercharge each stage with GenAI tools. By the end, you’ll know how to:

 run credible deep research for a market, problem, or user segment,

 frame the problem and communicate it with lean artifacts,

 prototype quickly to test value and usability, and

 deliver a focused plan with a prioritized backlog.

We’ll keep things practical and PM-friendly, showing prompts, outputs, and how to avoid common
pitfalls.

The 4D Framework at a Glance

 Discover: Gather signal (not noise) about users, competitors, and the problem space.

 Define: Turn research into a clear problem statement, success metrics, and a narrative your team
can rally behind.

 Develop: Explore solution options and prototype the most promising one fast.

 Deliver: Write a crisp PRD, then sequence work using a lightweight prioritization method.

Discover — Run credible deep research with AI

Goal: Build conviction about the problem, users, and constraints.

Tools you’ll see in class

 Perplexity Deep Research for multi-step, source-backed web research (great for competitor
scans, trend checks, and synthesis). (Perplexity AI)

 ChatGPT Deep Research to plan and execute multi-step investigations across the open web (good
for broad questions that need stitched evidence). (OpenAI)

 Julius AI to interrogate your own CSVs/spreadsheets, generate quick charts, and spot patterns in
product data (activation, retention, funnel drop-offs). (Julius AI)

How this looks in practice

 Start with a falsifiable problem hypothesis (e.g., “New users abandon the checkout because
delivery fees are unclear”).

 Prompt Perplexity/ChatGPT Deep Research to triangulate evidence (industry reports, forums, app
reviews). Ask for contradictions and caveats, not just summaries.

Prepared by Vaibhav Yendole 62

  Drop your internal data (exports) into Julius AI to validate patterns: “Show conversion by device
type and first-session source; visualize week-over-week change.”

Watch-outs

 Treat AI as a research assistant, not an oracle—verify sources and

capture assumptions alongside findings.

Define — Turn signal into a crisp narrative

Goal: Align stakeholders on the problem, user, and the “why now.”

Tools you’ll see in class

 Gamma to spin up clear, minimal decks or docs for the problem narrative, options, and next steps.
(Gamma)

 PopAi ([Link]) to draft visual slides from text or links for quick stakeholder readouts. ([Link])

Working pattern

 Write a Problem One-Pager (Problem, Audience, Evidence, Constraints, Success Metrics).

 Ask Gamma/PopAi to generate a 3–5 slide storyboard: “Why this matters,” “What we learned,”
“What we’ll test.” Keep metrics and risks on the slide—don’t bury them.

Example If research shows drop-off due to fee uncertainty, your narrative might:

 quantify lost revenue,

 highlight angry review snippets, and

 set a north-star (e.g., +8% checkout completion in 30 days) with a guardrail (NPS unchanged or
better).

Develop — Prototype to learn fast

Goal: Put something in front of users quickly to test value and usability.

Tools you’ll see in class

 Google Stitch (Google Labs) to generate UI and front-end code from prompts or sketches—handy
for rapid interface ideation. (Stitch)

 Visily to move from low-fi to high-fi wireframes and interactive prototypes—great for PMs/non-
designers. (Visily)

Working pattern

 In Stitch, prompt a few variants of the same idea (e.g., “show fees up front,” “progressive fee
reveal,” “fee toggle with breakdown”).

 In Visily, refine the winning flow, annotate your assumptions, and link screens for a mini-usability
pass.

Prepared by Vaibhav Yendole 63

  Aim for a lean user test (5–7 users or internal proxies). Capture where users hesitate and what
they say vs. do.

Deliver — Ship with focus and defend your choices

Goal: Communicate scope, de-risk, and sequence work.

Tools you’ll see in class

 ChatPRD to generate and iterate a Product Requirements Document (Problem, Goals, User
Stories, Acceptance Criteria, Risks). (ChatPRD)

 Claude to prioritize features using a framework like RICE—ask it to compute and explain scores
for each backlog item, then pressure-test the ranking. ([Link])

A lightweight prioritization flow

1. List candidate solutions (e.g., Up-front Fee Card, Tooltip Breakdown, Checkout Fee
Toggle, Coupon Education Nudge).

2. For each, estimate Reach, Impact, Confidence, Effort.

3. Have Claude compute a RICE score and explain outliers (“High impact but low confidence—what
data would increase confidence?”).

4. Sanity-check with constraints (compliance, performance, timeline) and finalize a thin slice to ship
first.

PRD essentials to include

 Non-goals to prevent scope creep,

 Instrumentation (what events/metrics you’ll capture), and

 Rollout plan (A/B, kill-switch, support comms).

Key takeaways

 AI won’t make decisions for you, but it compresses cycle time—from weeks to hours—when you
ask precise questions and validate results.

 The 4D rhythm keeps you honest: evidence first, then clear framing, cheap learning via
prototypes, and focused delivery.

 Tools change; the thinking doesn’t. Use GenAI to sharpen your judgment, not replace it.

Prepared by Vaibhav Yendole 64

 Lecture Summary

Overview of the Lecture

The session focused on practical, hands-on product development using Generative AI tools, particularly
through a step-by-step guided process known as "vibe coding." The goal was to take participants through
the phases of discovering a product idea, defining user personas and journeys, establishing market
research, building UX/UI designs, and finally integrating backend functionalities using tools like
SuperBase and Lovable.

1. Introduction to Vibe Coding and In-Context Engineering

 Vibe Coding Process: The instructor introduced vibe coding, a method where multiple AI prompts
are run in a single context window/chat, maintaining continuity and context for the AI to understand
and build upon.

 In-Context Engineering: Emphasized the importance of running all prompts related to a product
idea within the same conversation window to maintain context, allowing the AI to reference
previous outputs effectively.

 Participants were instructed to copy all prompts and AI outputs into a single document to preserve
context and streamline their workflow.

2. Discover Phase: Problem and Market Validation

 Prompt 1 (Market Research): Participants shared their problem statements, target regions, and
customer segments (e.g., ride-hailing in tier 2 and 3 Indian cities).

 LLMs evaluated if the problem was worth solving. If the AI indicated "go/no go," this decision was
crucial for project viability.

 Participants learned to avoid overly narrow problem definitions to prevent biasing AI responses.

 The process included copying outputs (market summary, PRD summary) into designated sections
of their working document.

3. Discover Phase: Customer Persona and Pain Points

 Prompt 2 (User Persona): AI would generate a detailed customer persona, demographics,

motivations, and pain points.

 Participants learned to critically evaluate AI-generated personas and pain points, cross-checking
whether these were relevant or realistic for their customer base.

 Emphasis on cross-referencing with real customer data or domain knowledge to refine or exclude
inaccurate pain points.

 Avoid blindly accepting AI outputs; personalize and tweak the details based on domain knowledge.

Prepared by Vaibhav Yendole 65

 4. Define Phase: User Journey and Functional Requirements

 Prompt 3 and 4 (User Journey & Core Problems): AI-generated user journey maps and core
problem statements based on prior phases.

 Focused on extracting specific problems to be addressed, such as consolidating income for gig
workers.

 Learned to remove unnecessary features (e.g., tax insights) for MVP focus to avoid
overcomplicating the product.

 Defined functional requirements such as login, profile setup, income aggregation, and
notifications.

 Prioritization of features was shown as part of AI output.

5. Define Phase: Brand and UI/UX Design Principles

 Prompt 5 (Branding): Participants gave AI inputs on brand name, mission statements, color
schemes, and design guidelines.

 AI generated a brand identity reflecting clarity, simplicity, and focus aligned with user needs.

 Discussion on typography, colors, and UI tone to ensure branding consistency.

 Strong emphasis on verifying AI-generated branding to fit product vision.

6. Develop Phase: UI Wireframes and Design Prototyping

 Prompt 6 (UX Wireframes): AI generated detailed wireframes and screen flows for onboarding,
dashboard, etc.

 Encouraged cross-referencing within the chat to pull all previous context instead of duplicating
manual inputs.

 Demonstrated stitching AI-generated screen descriptions with tools like Google Stitch to create
clickable design prototypes.

 Noted some limitations in AI tools requiring manual fine-tuning to achieve expected UI quality.

 Highlighted the importance of iterative prompting to refine the UI screens.

7. Deliver Phase: Creating the Frontend and Backend

 Prompt 7 (Frontend Definition): Conversion of UX wireframes to frontend interactive mockups

using Lovable.

 Participants learned how to upload screenshots and ensure design consistency between AI
outputs and prototypes.

Prepared by Vaibhav Yendole 66

  Prompt 8 (Backend Setup): AI generated SQL schemas, API definitions, and security
configurations for the product’s backend.

 SuperBase platform integration was demonstrated for database, authentication, and API
management.

 Steps included signing up for SuperBase, connecting it to Lovable, enabling cloud backend, and
configuring authentication methods like Google sign-in and OTP.

 AI-generated the necessary SQL queries and backend logic to deploy the database and APIs
seamlessly.

8. Best Practices and Advanced Concepts

 Cross-referencing and Context Management: Demonstrated how to reuse context effectively

across prompts to save tokens and ensure consistency.

 Critical Human Input: In each stage, participants were encouraged to validate AI responses
against real-world knowledge and adjust accordingly rather than blindly trusting AI.

 Multiple AI Tools: Comparisons between models like Claude, ChatGPT, Perplexity, etc., with
recommendations to choose based on problem type (UX vs data focus).

 Tips on reducing token usage by instructing AI to update existing context silently without verbose
responses.

9. Q&A and General Guidance

 Discussion on job interviews and standing out with a strong portfolio and networking.

 Clarifications on prototyping versus building fully functional production-ready applications.

 Advice to use advanced vibe coding platforms like Cursor, Windsurf, or Kiro for scaling projects.

 Warnings on API key safety and avoiding sharing sensitive data during public sessions.

 Encouragement to complete the Deep Learning AI vibe coding certification to bolster resumes and
understanding.

Summary

This session provided an end-to-end walkthrough of building a product leveraging Generative AI tools
through vibe coding:

 Starting with problem discovery and market validation.

 Defining user personas, journeys, and functional requirements.

 Creating brand identities and detailed UX wireframes.

 Developing frontend mockups and backend services using SuperBase and Lovable platforms.

 Leveraging in-context engineering for continuity and efficiency.

Prepared by Vaibhav Yendole 67
  Emphasizing human oversight and iterative refinement to maintain real-world relevance.

 Equipping participants with practical knowledge to build production-grade applications for

interviews and real projects using advanced AI-driven workflows.

Participants were encouraged to practice and experiment with these prompts and tools to gain mastery
over AI-augmented product development.

Evaluating and Refining Problem Statements and Customer Personas

When using LLMs to generate problem statements, market research, or customer personas, it is
important not to blindly accept the outputs. Instead:

 Validate if the identified pain points or customer segments align with your understanding or real
user data.

 Remove or ask the model to ignore irrelevant or incorrect pain points.

 Narrow or broaden the scope carefully to avoid bias that might limit innovative solutions.

Master Prompt for Vibe Coding Interview

In a vibe coding interview, you start with a single master prompt that orchestrates the entire product
development flow:

 Discover market and customer

 Define persona and journeys

 Develop functional requirements and brand

 Deliver UX/UI and backend

This structured approach with multiple detailed prompts ensures comprehensive coverage.

Sample Master Prompt:

"Create a guide to develop a full PRD and a functional prototype with

- Market size discovery

- Customer persona definition

- User journey mapping

- Functional and non-functional requirements

- Brand development

- UX/UI design

- Front-end delivery

- Backend and API setup with database and authentication"

Prepared by Vaibhav Yendole 68

W19L1 AI Model Categories, Advanced LLM Metrics, and Production Hosting
Basics (Prof. Saravana)

Pre class resources: AI Model Categories, Advanced LLM Metrics, and Production Hosting Basics

Estimated Reading Time: 15–20 minutes

Learning Outcomes

After completing this pre-read, you will be able to:

 Recognize the major AI model categories and when to use each.

 Explain key LLM evaluation metrics — accuracy, perplexity, bias/hallucination signals, and
latency.

 Describe production hosting concepts — environments, CI/CD, scaling, and monitoring.

 Ask informed questions about model fit and deployment trade-offs.

Think of this as building your foundation before deep diving into AI systems.

Overview

This pre-read covers:

 Core model families (discriminative, generative, sequence, image, embedding, chat/instruct,

multimodal).

 Practical LLM metrics (accuracy, perplexity, hallucination/bias indicators, latency).

 Production hosting essentials (dev → staging → prod, autoscaling, observability, rollbacks).

 A simple hands-on code example for evaluating predictions and measuring latency.

Part 1: The Big Picture — Why It Matters

An AI model can sound confident yet be wrong. Picking the right model type, measuring it correctly,
and deploying it safely is what turns a prototype into a reliable product.

Why It Matters

 Model choice impacts performance, cost, and fairness.

 Evaluation metrics show if the model meets real-world needs.

 Hosting design ensures reliability, speed, and scalability.

Where You’ll Apply This

Roles:

Prepared by Vaibhav Yendole 69

  ML/AI Engineers — design and evaluate models

 Product Managers — define goals and quality benchmarks

 DevOps/Platform Engineers — deploy, monitor, and maintain systems

Real-world examples:

 Search recommendations (YouTube, Spotify)

 AI assistants (email replies, chatbots)

 Computer vision (defect detection, ID verification)

Analogy:
Models are vehicles — hosting is the road network.
A large model is a race car: fast but expensive.
A smaller model is a scooter: cheap and agile.
Good roads, signals, and guardrails (infrastructure and metrics) keep them efficient and safe.

Part 2: Roadmap Through This Topic

1. Model Categories — understanding types and use cases.

2. LLM Metrics — measuring quality, reliability, and risk.

3. Production Hosting Basics — building for performance and safety.

The sequence follows how AI systems evolve: choose → measure → deploy.

Part 3: Key Concepts and Definitions

Discriminative Models – Learn to separate categories based on features.

Example: Classify spam vs. not-spam emails.

Generative Models – Learn to create new data similar to training data.

Example: Generate text, images, or code.

Perplexity – Measures how “surprised” a model is by data.

Lower values indicate better fluency, but not necessarily factual accuracy.

Hallucination – When an AI produces confident but false information.

Example: “According to Harvard, this was proven in 2018.” (when no such study exists)

Bias – Systematic unfairness in responses or predictions across groups or topics.

Latency – The time between input and response.

Crucial for user experience and system cost.

Insight: The “best” model is one that achieves the right balance between accuracy, reliability,
latency, and cost — not necessarily the largest one.

Part 4: Concepts in Action

Prepared by Vaibhav Yendole 70
 Scenario:
An AI email assistant slows down during peak usage and occasionally fabricates company policies.

Challenges:

 Latency spikes at high load

 Fabricated or misleading answers

How hosting fixes it:

1. Environment separation: Test in staging before production.

2. Autoscaling: Increase compute when usage spikes.

3. Monitoring: Track P50/P95 latency, error rates, and hallucination flags.

4. Guardrails: Require sources for claims (“according to…” must have URLs).

5. Rollbacks: Quickly revert when new versions increase risk.

Outcome:
Stable performance, fewer hallucinations, and improved reliability.

What this shows:

Infrastructure and evaluation discipline amplify model impact. Reliability isn’t accidental — it’s
engineered.

Real-World Tip: The best “upgrade” often isn’t a bigger model, but better monitoring, guardrails,
and prompt design.

Part 5: How This Connects to the Bigger Picture

Builds on:

 Fundamentals of model training and task framing.

Enables:

 Selecting suitable models for goals.

 Designing meaningful evaluation methods.

 Shipping safe, reliable, and scalable AI systems.

Links to:

 RAG (Retrieval-Augmented Generation): Uses external data to reduce hallucinations.

 Prompt Engineering: Steers model responses without retraining.

 A/B Testing: Compares prompt or model versions with real users.

Career Relevance:
ML Engineer • Data Scientist • AI Product Manager • Platform Engineer

Prepared by Vaibhav Yendole 71

 Part 6: Reflection Questions

1. How do model type and hosting design together affect user trust and system cost?

2. For a chatbot or triage bot, what guardrails and metrics would you implement first?

3. When is a smaller, faster model better than a larger, more accurate one?

4. Which product you use daily could improve most from better latency or grounding?

Self-Check

You should now be able to:

 Explain why model selection and hosting both matter.

 Define key terms: discriminative, generative, perplexity, hallucination, bias, latency.

 Provide an example where a metric other than accuracy drives success.

 Describe a dev → staging → prod flow with rollbacks.

 Suggest one guardrail to improve AI reliability.

If not, revisit the above and focus on why each step matters.

How to Read and Apply This

DO:

 Read actively — focus on relationships, not memorization.

 Experiment with the code — even small tweaks help learning.

 Relate examples to systems you use or build.

DON’T:

 Memorize definitions in isolation.

 Ignore latency or deployment factors.

 Skip reflection questions — that’s where mastery develops.

What’s Next?

Continue learning:

 Basics of RAG (Retrieval-Augmented Generation)

 Prompt evaluation and A/B testing techniques

 MLOps pipelines and model monitoring

Prepared by Vaibhav Yendole 72

 Final Thought

You now have the foundation — what models exist, how to evaluate them, and how to host them
responsibly.
Real AI excellence comes from combining curiosity, structure, and iteration.
Small, steady improvements turn demos into dependable systems.

Lecture Notes: AI Model Categories, Advanced LLM Metrics, and Production Hosting Basics

What you’ll be able to do:

 Identify the major AI model categories and where each fits.

 Explain core LLM evaluation metrics (accuracy, perplexity, bias/hallucination signals).

 Compare common deployment strategies (blue–green vs. canary) and apply a basic monitoring
checklist.

1) Introduction — What is this and why should you care?

Core definition

An AI model is a system that has learned patterns from data in order to classify, predict, or
generate outputs (text, images, decisions). Modern large language models (LLMs) specialize
in generating and understanding text. Effective AI practice means choosing the right
model, measuring what matters, and deploying it safely so users can rely on it.

A simple analogy

AI models are like kitchen appliances built for different jobs:

 A blender (generative model) mixes ingredients into something new.

 A toaster (discriminative model) judges “done” vs. “not done.”

 A fridge organizer (embedding model) helps you find things quickly.

Limitation: Appliances do one thing by design; AI behavior changes with data, instructions, and
updates—so measurement and monitoring are essential.

Why this matters to you

Problem it solves: Avoiding “cool demo, poor product.” You’ll connect model choice → evaluation →
safe hosting, so your AI features are useful, fair, and reliable.

What you’ll gain (concrete benefits):

 Clarity on which model type to use for a task (search, classify, generate).

 A small set of metrics that reveal quality and risk (not just accuracy).

 A mental checklist for deployment (separate environments, rollbacks, monitoring).

Real-world context: These ideas power recommendations, search, chat assistants, content
moderation, and more across products you already use.
Prepared by Vaibhav Yendole 73
 2) The Foundation

Concept A — AI model categories (Task vs. Data Type)

Definition: We categorize AI models along two main axes: what they do (their task) and what data they
use (their architecture).

1. By Task:

 Discriminative: These models decide which label fits an input. They classify, judge,
or choose from a set of predefined options (e.g., spam vs. not spam, positive vs. negative).

 Generative: These models produce new data (text, images, audio) that resembles the data
they were trained on. They create or generate.

2. By Data Type:

 Sequence models: Process ordered inputs, where the order matters (e.g., sentences,
stock market time-series).

 Image models: Process grids of pixels and spatial patterns (e.g., photos from a factory line,
medical scans).

Key characteristics: A model is almost always a combination of these axes.

 An LLM (like a chatbot) is a Generative Sequence model: It creates new text.

 A spam filter is a Discriminative Sequence model: It classifies text as "spam" or "not spam."

 A factory-line defect scanner is a Discriminative Image model: It classifies an image as "pass" or

"fail."

A concrete example:

 Task: Route a support ticket to the right department.

 Model: Discriminative Sequence model (it classifies the text).

 Output: "Billing" or "Tech Support."

 Task: Draft a polite reply to that support ticket.

 Model: Generative Sequence model (it creates new text).

 Output: "We are sorry to hear you are having trouble with..."

 Task: Check a factory photo for a scratch.

 Model: Discriminative Image model (it classifies the image).

 Output: "Pass" or "Fail (Scratch detected)."

Common confusion & fix: “Generative models are always better.” → Use discriminative models for fast,
reliable decisions; use generative when you need flexible language/image creation. A small, fast
discriminative model is often far cheaper and more accurate for a simple classification job than a large
generative one.

Check your understanding:

Prepared by Vaibhav Yendole 74
  Which task type (discriminative or generative) fits “route a ticket to the right team” vs. “draft a
friendly reply”?

 What data type (sequence or image) would a model that classifies customer reviews by topic use?

 Why might a smaller discriminative model beat a large generative one on speed and cost for a
simple classification task?

Concept B — LLM evaluation & metrics

Definition:
LLM evaluation combines task success (did it answer the question?), language quality (fluency,
coherence), and risk (bias, hallucination). No single metric is sufficient; we use a small bundle that
matches the job.

How it relates to Concept A:

Once you choose a model type, you measure it differently:

 Discriminative → accuracy, precision/recall, confusion matrix

 Generative → task success, perplexity (training-time fluency proxy), human/automatic text quality
checks

Key metrics:

 Accuracy / Precision / Recall: How often the decision is right—and right for the right class.

 Perplexity: A model’s surprise at text; lower means it predicts text more confidently (useful in
training analysis, not a full product metric).

 Hallucination signal: Flags when the model states facts without sources or contradicts provided
context.

 Bias indicators: Differences in outcomes across groups (e.g., gender) for similar inputs.

 Latency: Time to respond; matters as much as correctness for user experience.

Counter-example (“what it’s NOT”):

“High perplexity means the model is unsafe.” → Not necessarily. Perplexity is about fluency; safety
needs grounding and content checks.

Mini self-check:

 When would you prefer recall over precision? (e.g., safety or escalation tasks)

 Why doesn’t perplexity alone prove a chatbot answers factually?

Concept C — Production hosting & monitoring basics

Definition:
Production hosting delivers your model to users safely: separate environments (dev → staging → prod),
controlled releases (blue–green, canary), observability (latency, errors, flagged outputs), and rapid
rollback.

Prepared by Vaibhav Yendole 75

 Key characteristics:

 Blue–green: Two identical environments; switch traffic when “green” passes tests → easy rollback.

 Canary: Release to a small % of users first; promote if healthy.

 Monitoring: Track P50/P95 latency, failure rate, hallucination/bias flags, and user feedback.

A concrete example:

 Stage a new prompt version in green, run scripted test questions, check latency and grounding
flags, then move 100% traffic to green. Keep blue ready to roll back.

Common confusion & fix:

“Deployment is just pushing a file.” → It’s a process with guards: versioning, staged validation, metrics,
and rollback.

Check your understanding:

 When would you choose canary over blue–green?

 Name two alerts you would set for a chatbot release.

3) How A & B work together

 Pick a model (Concept A) → Decide the metric bundle (Concept B) that reveals the right risks for
the user task.

 Example: For a support triage assistant, track recall (don’t miss urgent cases), hallucination
flags (don’t invent policies), and P95 latency (don’t keep users waiting).

4) Worked example

Problem statement:
You run a help-center assistant that drafts replies. Some replies are slow and occasionally include
unsourced claims.

Approach:

1. Define success: “Accurate, grounded, under 2.0s P95 latency.”

2. Add checks: If the text contains “according to” without a source link → flag.

3. Run a small gold set: 20 real questions with approved reference answers.

4. Measure: Task success rate (rubric), average & P95 latency, number of grounding flags.

5. Release via canary: 5% traffic for 24h; promote if metrics stay green.

6. Observe: Dashboard shows task success ↑, flags ↓, P95 latency < 2s. Roll back if any regress.

Result:
Reliable replies, faster responses, measurable reduction in unsourced claims.

Key lesson:
Production readiness = quality + speed + risk control. All three matter.
Prepared by Vaibhav Yendole 76
 5) Check your understanding

 Explain: Why can perplexity look good while answers are still wrong?

 Apply: Which release is safer for frequent small updates—canary or blue–green—and why?

 Compare: When would you favor precision vs. recall in moderation?

 Analyze: A model’s accuracy is fine but P95 latency doubled. What user impact and what’s your
fix?

(Sample answers at the end of this note.)

6) Practice task

Scenario: You’re adding an FAQ assistant to your website.

Your task:

1. Choose one model type for each need:

 (a) Detect the question’s topic → [your choice]

 (b) Draft a polite answer → [your choice]

2. Pick three metrics for day-1: one quality, one risk, one latency metric.

3. Write a 3-step rollout (staging test → canary 5% → full release + rollback plan).

4. Define two alerts you’d set (e.g., hallucination-flag rate, P95 > target).

Hint: Keep it small and measurable—you can add more metrics later.

7) Common pitfalls

Pitfall Why it happens What to do instead Why it works

“Only Mirrors real

Track quality + risk +
accuracy Ignores risk and latency. user
speed together.
matters.” experience.

“Perplexity
Measures
proves Add grounding checks &
Confuses fluency with factuality. what users
product human rubric.
value.
quality.”

“Push straight Use staging, Safe, quick

Skips validation.
to prod.” then canary/blue–green. rollback.

Prepared by Vaibhav Yendole 77

 Pitfall Why it happens What to do instead Why it works

Dashboards: P50/P95 See

“We’ll eyeball
No trend visibility. latency, flags, task regressions
outputs.”
score. early.

If you’re stuck: Revisit Concept B (metrics) and ensure each metric maps to a user risk or goal.

8) Progressive complexity markers

 Let’s start simple… Choose the model family by task.

 Now add… A metric bundle that fits your task.

 For a complete picture… Add hosting strategy + monitoring.

 Advanced note: Combine with RAG to reduce hallucinations further.

9) Mini-summaries

 Models: Pick by job (decide vs. generate; text vs. image).

 Metrics: Use a bundle (quality + risk + speed).

 Hosting: Prefer staged releases and observable systems.

10) Sample answers (for Section 5)

 Perplexity ≠ truth: It reflects predictive fluency, not grounded correctness. Need grounding
checks and human/task rubrics.

 Safer for small updates: Canary—gradual rollout limits blast radius.

 Precision vs. recall:

 Precision for false-positive cost (e.g., blocking legitimate content).

 Recall for miss cost (e.g., missing harmful content).

 High P95 latency: Users feel slowness and abandon tasks; fix with caching, smaller model,
streaming outputs, autoscaling.

Key takeaways & next steps

 Choose wisely: Model families are tools—pick by task.

 Measure what matters: Pair quality + risk + speed from day one.

 Ship safely: Staging → Canary/Blue–Green → Monitor → Roll back.

Prepared by Vaibhav Yendole 78

 Next steps:
Try adding grounding rules (“claims require a source”), set two alerts (hallucination-flag rate, P95
latency), and rehearse a rollback.

Overview of the Lecture

The lecture covered a detailed review of the previous session about API usage in AI-powered
applications, followed by an introduction to creating AI-integrated applications using platforms like
Microsoft Azure and Base 44. It focused on designing product flows, API integration, AI model hosting,
cost considerations, and the use of no-code platforms for AI application development.

Recap of Last Week's Session: APIs in AI Applications

 Discussed building AI-based digital products including front-end, back-end, and API components.

 Focus on a sample product for exam evaluation using AI, involving multiple steps:

 Instructor enters course details → folder creation in cloud (Azure).

 Uploading scanned exam PDFs, storing via API calls.

 Running Optical Character Recognition (OCR) on handwritten exam papers using an API.

 Splitting PDFs by registration numbers into individual student exam sheets.

 Uploading instructor’s answer key and creating a standardized marking rubric with AI
assistance.

 Evaluating student answers by sending answer key, student paper, and prompts to AI
through API calls.

 Emphasized the design of product flows: front-end, back-end, and API calls at each step.

 API analogy explained with restaurant waiters as interfaces between customers and kitchens.

Understanding APIs and Their Role

 APIs act as intermediaries allowing separate applications/services to communicate seamlessly.

 Example: Application communicates with Azure cloud services via API key and endpoint (a URL).

 Customers (applications) do not need to know the back-end workings (kitchen) but only how to
send requests (order from menu) and receive responses (food).

 APIs are language-agnostic; any programming language can invoke APIs by sending requests with
keys and endpoints.

 Product managers must evaluate:

 Whether to develop functionalities in-house or use third-party APIs.

 Costs associated with API usage (per request or subscription) versus long-term in-house
development costs.
Prepared by Vaibhav Yendole 79
  Technical requirements like server hosting come into play if self-hosting AI models rather than
using third-party APIs.

AI Model Hosting and Cost Considerations

 AI models can be accessed via cloud APIs (e.g., Microsoft Azure’s AI services) or self-hosted open-
source models (LLaMA, DeepSeek).

 Hosting open-source models locally avoids per-request API costs but involves high initial
infrastructure costs and maintenance effort.

 Proprietary models like GPT-4 or GPT-5 must be accessed via third-party APIs with associated
costs.

 Price varies significantly with model capability —

 GPT-4.1 Nano: low-cost (~₹8 per million tokens), suitable for testing and basic applications.

 GPT-5 Pro: high-end ($15 input, $120 output per million tokens), offering advanced
reasoning and features.

 API usage and billing are managed through subscriptions, API keys, and endpoints within cloud
portals (e.g., Azure Portal).

 Budget setup and cost alert management are crucial to avoid unexpected billing.

Microsoft Azure Portal and Service Setup

 Azure Portal used to create accounts, manage subscriptions, set budgets, and create resource
groups.

 Services like Document Intelligence (for OCR), Azure OpenAI service (for LLMs), and storage are
created as instances/resources.

 Each Azure AI service provides an API key and endpoint that applications use to communicate with
these services.

 Multiple instances or services can be created to organize applications and manage billing per client
or project.

 Support for multiple models within one AI service deployment, allowing calls to different LLMs via
the same endpoint using model names.

No-Code Platforms for AI App Development: Base 44, Lovable, Replit

 Base 44 demonstrated as a platform that allows app creation using prompts, automatically
generating backend tables, dashboards, and UI components.

 API keys from cloud services like Azure can be integrated into Base 44 apps to enable
functionalities like OCR and AI-driven analytics.

 Advantages of no-code platforms:

Prepared by Vaibhav Yendole 80
  Rapid prototyping without coding.

 Built-in AI capabilities and database management.

 UI/UX customization possible by uploading screenshots for design inspiration.

 Limitations and strengths of no-code platforms:

 Base 44 is good for prototyping with inbuilt AI but limited in handling multiple complex API
changes after initial setup.

 Lovable requires incremental prompting for complex apps.

 Replit offers comprehensive capabilities but requires more time for app generation (up to 90
minutes) and is suitable for larger scale or complex applications.

 Integration with external databases (e.g., MongoDB, PostgreSQL on Azure) possible via API calls.

Practical Application: Creating "Pinmate" - A Personal Finance Tracker

 Task assigned to create an AI-assisted personal finance tracking app using Base 44 integrated with
Azure AI services.

 App features include:

 Data input via manual entry or scanned receipts/documents requiring OCR.

 Automatic categorization of income and expenses using AI.

 Financial analysis and actionable recommendations.

 Goal monitoring and visualization of spending trends.

 Workflow involved: add data → OCR → categorize → analyze → recommend → report generation.

 Students designed detailed prompts to describe the user journey, front-end/back-end needs, and
AI touchpoints for automatic app generation.

 Demonstration of Base 44 auto-generating necessary tables, forms, and dashboards from

prompts.

 Preview of the incomplete app highlighted the need to connect to Azure’s OCR and AI services for
full functionality (planned for next session).

Guidance for Product Managers

 Product managers must balance between in-house development and third-party API use based on
cost, volume, and technical team capability.

 Importance of designing clear product flows defining front-end, back-end, and AI/third-party API
integration points.

 Monitoring API usage and cost with budget controls in cloud portals is essential.

 For large-scale/multi-module applications, modular development via stepwise prompts and

managing interdependencies is advisable.
Prepared by Vaibhav Yendole 81
  UI/UX designs can be imported via screenshots for automated styling in platforms like Base 44.

Next Steps and Follow-Up

 Next class will focus on setting up Azure AI and document intelligence services from scratch.

 Students will learn to link API keys and endpoints securely in no-code platforms for OCR and AI
evaluation.

 Hands-on app deployment and integration of AI services will be demonstrated.

 Continued practice with prompt refinement and app flow development recommended.

 Exploration recommended on AWS and Google Cloud alternatives for similar AI services.

This summary captures critical aspects of the lecture on AI model categories, advanced LLM metrics,
API integration, and production hosting basics using Microsoft Azure and no-code platforms like Base 44.

Overview of APIs and Their Role in Application Development

APIs (Application Programming Interfaces) act as intermediaries that allow different software
applications to communicate with each other. In the context of digital product development, APIs
connect the front-end and back-end modules and also integrate with external services.

Key Points:

 APIs abstract the complexity of back-end or external service operations from the user.

 APIs are language-agnostic, meaning developers can invoke APIs using any programming language.

 An example analogy: a waiter (API) takes your order (request) to the kitchen (back-end service) and
brings your food (response).

Example: Calling an Azure API to create a folder in cloud storage.

import requests

Breakdown of a Digital Product Workflow for AI-powered Exam Evaluation

The lecture illustrated a product workflow involving multiple steps integrating front-end, back-end, and
APIs using AI services. The example product was an AI-powered exam paper evaluation system.

Workflow Steps:

1. Instructor enters course details -> creates cloud folder (via API).

2. Upload scanned multi-page PDF of exam papers -> stored via API.

3. Optical Character Recognition (OCR) to convert scans to digital text (via API to document
intelligence service).

4. Split large PDF into individual student PDFs based on registration numbers.

Prepared by Vaibhav Yendole 82

 5. Upload answer key by instructor.

6. Generate a standardized marking rubric by sending answer key + prompt to AI API.

7. Evaluate individual student answers by sending answer sheet + answer key + prompt to AI API.

8. Generate detailed reports.

API Usage:

 API calls are made to Azure services for storage, OCR, and AI evaluation.

 AI APIs handle rubric generation and student answer evaluation.

This multi-step process demonstrates the integration of APIs in handling complex business workflows
with AI support.

Understanding API Keys and Endpoints in Cloud Services

API keys and endpoints are essential to interact securely with cloud services.

 API Key: A unique identifier used to authenticate requests.

 API Endpoint: The URL where API requests are sent.

How it works:

 When an application calls a third-party service (like Azure), it sends the API key along with the
request to the API endpoint.

 The endpoint URL determines which service/function to invoke.

 Cloud providers offer APIs with different keys and endpoints for different services.

Considerations for Choosing Third-Party APIs as a Product Manager

Product managers should evaluate third-party APIs based on multiple factors:

 Cost: APIs often charge per request or per usage (e.g., per million tokens for AI).

 Example: GPT 4.1 nano costs rupees 8 per million tokens; GPT 5 costs much higher.

 Tech Stack Compatibility: API calls are generally language-agnostic, but product team’s tech
stack should influence the choice.

 Scalability: High usage might justify building in-house solutions to reduce long-term API costs.

 Service Availability: Some cloud services may not be available in all data center regions.

 Maintenance and Hosting:

 Using third-party APIs means offloading maintenance.

 Hosting open-source models locally requires server setup and ongoing maintenance.

Prepared by Vaibhav Yendole 83

 Example Cost Consideration: Suppose 10 million OCR requests per month cost $0.05 each; total
$500,000 per month. Building an in-house OCR system costing $200,000 one-time may be more
economical.

Deploying and Using AI Models on Microsoft Azure

Microsoft Azure provides cloud AI services with modular deployment options:

 Resource Group: A container to organize related services/resources.

 Instance: A deployed service under the resource group (e.g., an AI model endpoint).

 Models Catalog: Variety of AI models available for deployment (GPT 4.1 nano, GPT 5 Pro, Grok,
etc.).

Deploying a model:

 Create Resource Group.

 Create AI Service Instance (e.g., ‘PMAI app development’).

 Deploy a model instance within the AI Service (e.g., GPT 4.1 nano).

 Acquire API Key and Endpoint to integrate with your application.

Integrating AI and OCR Services within No-Code Platforms like Base 44

No-code platforms such as Base 44 allow rapid app development with AI integrations:

 Base 44 auto-generates back-end tables and UI based on prompts.

 You can connect Base 44 with external APIs for AI and OCR (e.g., Azure AI services).

 Using API keys and endpoints, Base 44 apps can invoke AI services for tasks like OCR extraction,
categorization, and analytics.

 Base 44 supports adding styling or UI tweaks via prompt instructions.

Example scenario in Base 44:

 User uploads a receipt image.

 App sends image to Azure Document Intelligence OCR API.

 OCR output is returned and parsed into expense data.

 AI service categorizes and analyzes spending.

 Budget and insights generated in the app.

No-code example (conceptual):

Developing AI-Powered Personal Finance Tracker Application (Pinmate) Workflow

Pinmate is a conceptual app for young professionals to track personal finance via AI and OCR.

Prepared by Vaibhav Yendole 84

 User Journey and Functionality:

1. Add Data: User enters expense/income manually or uploads scanned receipts, invoices, or bank
SMS.

2. OCR Processing: For non-digital docs, OCR extracts textual data.

3. Categorization: AI categorizes entries as income or various expense categories.

4. Storage: Organized data gets stored in a backend database.

5. Analysis and Reporting: AI generates summary reports, trends, and actionable recommendations.

6. Notifications & Tips: Personalized AI coach provides financial advice and goal tracking alerts.

Best Practices and Challenges Using LLM-Driven No-Code App Builders

No-code AI app development platforms (Base 44, Lovable, Replit) come with specific best practices and
limitations:

 Prompt Design: Incremental prompting (stepwise) works better for complex functionalities rather
than large single prompts.

 AI Service Declaration: Declare usage of AI services early in the prompt to avoid errors.

 Platform Differences:

 Base 44 offers built-in AI and auto database generation.

 Lovable requires careful incremental prompting; may skip complex features if prompt is too
large.

 Replit Agent can generate more comprehensive applications but takes longer (up to 90
mins).

 Integration Flexibility: All platforms support connecting external APIs; choice depends on project
complexity.

Cost Management and Billing Controls in Cloud AI Services

Managing cost is crucial when using cloud AI services like Azure.

 During account setup, attach credit card but configure budget limits.

 Set up alert thresholds to notify spending near limits.

 Different AI models have varying cost structures; example from Azure:

 GPT 4.1 nano: low cost (~₹8 per million tokens).

 GPT 5 Pro: significantly more expensive (~₹1500+ per million tokens).

 Monitoring usage regularly prevents unexpected high bills.

Prepared by Vaibhav Yendole 85

 W19L2 Stakeholder Managment for AI PM and AI Product Lifecycle, and
Scaling Case Studies(AN)

What You'll Gain from This Pre-Read

 Recognize the key stakeholders in AI product development

 Understand the AI product lifecycle: pre-training → fine-tuning → deployment → monitoring

 Learn from real scaling examples (Tesla Autopilot & Grammarly GenAI)

 Appreciate how stakeholder management drives safe, trusted AI growth

 Ask informed questions and explore deeper independently

Think of this as: Learning the alphabet before writing essays — you’re building the foundation.

What This Pre-Read Covers

This pre-read will:

 Introduce the key stakeholders in AI product development

 Explain each phase of the AI product lifecycle

 Show real-world scaling lessons from Tesla and Grammarly

 Build curiosity, context, and foundational awareness

Part 1: The Big Picture — Why Does This Matter?

Imagine launching an AI model that writes customer emails. It works great today — but a month later, it
starts generating confusing replies. What happened? The world changed, the data drifted, and the model
didn’t adapt in time. Now, users and executives want answers.

AI PMs live in this constant motion — balancing innovation with reliability, managing not just code and
data, but people, expectations, and accountability.

Where You’ll Use This:

 Job roles: AI Product Managers, Machine Learning Engineers, Data Science Leads, Responsible AI
specialists

 Real products: Tesla Autopilot, Grammarly, ChatGPT, Google Photos

 What you can build: AI features that continuously improve while staying safe, transparent, and
compliant

Think of it like this: Managing an AI product is like steering a self-driving ship — the crew (engineers,
designers, legal, users) all pull in different directions. Your role is to set the course and keep everyone
aligned, even when the waters change.

Prepared by Vaibhav Yendole 86

 Limitation: Unlike ships, AI systems learn from the environment itself — your crew’s work changes the
sea you’re sailing on.

Part 2: Your Roadmap Through This Topic

1. Stakeholders for AI Product Development

You’ll discover who contributes to AI products and why alignment matters — from engineers and data
teams to ethics and compliance officers.

2. The AI Product Lifecycle

We’ll trace the evolution of an AI model from data collection and pre-training, through fine-tuning and
deployment, to real-time monitoring and retraining.

3. Case Study: Tesla Autopilot Scaling

You’ll learn how Tesla’s rapid iteration cycles balance speed, safety, and regulation while retraining
models on millions of miles of driving data.

4. Case Study: Grammarly GenAI Scaling

You’ll explore how Grammarly expanded from grammar checking to generative writing while maintaining
infrastructure stability and user trust.

5. The Common Thread

We’ll tie it together — showing how stakeholder collaboration enables the AI lifecycle to run smoothly at
scale.

The journey: We’ll start with people, move through process, and end in practice.

Part 3: Key Terms to Listen For

Stakeholder

Anyone who affects or is affected by an AI product — from developers to end users.

Example: A legal counsel ensuring compliance before deployment.

AI Product Lifecycle

The repeating stages through which an AI model evolves — data collection, training, deployment, and
monitoring.
Think of it as: The “life story” of an AI system.

Pre-Training

Training a model on massive datasets to give it general knowledge before fine-tuning.

Example: GPT models pre-trained on web-scale text data.

Prepared by Vaibhav Yendole 87

 Fine-Tuning

Customizing a pre-trained model for a specific purpose using smaller, targeted datasets.
Example: Teaching a language model to write legal summaries.

Model Drift

When a model’s accuracy or reliability declines as real-world data changes.

Think of it as: Your AI “forgetting” how the world now works.

Monitoring

Ongoing observation of model performance, safety, cost, and fairness after deployment.
Example: Tracking how often Grammarly’s AI suggestions are accepted or rejected.

Key Insight:
The AI lifecycle never ends — it’s a loop, not a line. Stakeholders close the loop through data, feedback,
and decision-making.

Part 4: Concepts in Action

Example 1: Stakeholder Alignment in AI Product Development

The Situation:
An AI team wants to deploy a recommendation engine for personalized learning paths.

The Challenge:
Data scientists want accuracy, UX wants simplicity, compliance wants fairness, and leadership wants
ROI.

How Stakeholder Management Applied:

1. Mapping Roles: The AI PM identifies everyone’s goals — from engineering to legal.

2. Translating Language: Converts technical metrics (precision/recall) into product KPIs (user
satisfaction, retention).

3. Shared Decision Framework: Creates alignment through “bias budgets” and measurable trade-
offs.

The Outcome:
All teams move in sync, understanding both risks and opportunities. The system launches faster — with
fewer surprises.

What This Shows:

AI success depends on managing relationships and expectations, not just models.

Real-World Application:
Stakeholder mapping is the AI PM’s compass — it turns conflict into coordination.

Prepared by Vaibhav Yendole 88

 Example 2: Tesla Autopilot — Balancing Iteration and Safety

The Situation:
Tesla trains self-driving AI using billions of miles of fleet driving data.

The Challenge:
Deliver continuous improvements while meeting safety regulations.

How Tesla Applies the Lifecycle:

1. Pre-Training: Use massive, diverse datasets from vehicles worldwide.

2. Fine-Tuning: Focus on edge cases like unmarked roads and poor lighting.

3. Deployment: Roll out updates over-the-air (OTA) in controlled waves.

4. Monitoring: Collect feedback and driving telemetry for retraining.

The Outcome:
Tesla balances fast innovation with risk controls, although it faces scrutiny for transparency and testing
rigor.

What This Shows:

Speed without oversight can erode trust. Stakeholder alignment (engineering, legal, regulators) is as vital
as the model itself.

Real-World Application:
For AI PMs, regulators and users are critical stakeholders, not afterthoughts.

Example 3: Grammarly GenAI — Scaling with User Trust

The Situation:
Grammarly expanded from simple grammar correction to full generative writing and rewriting features.

The Challenge:
Introduce powerful features without confusing users or damaging credibility.

How Grammarly Applies Stakeholder Principles:

1. Transparency: Clear UI labeling for AI-generated content.

2. Phased Rollout: Gradual releases with monitoring of user engagement and feedback.

3. Ethics by Design: Human review and content filters ensure safe, non-harmful text.

The Outcome:
Users feel empowered, not replaced. Trust and adoption increase steadily.

What This Shows:

Scaling GenAI responsibly depends on transparency, opt-ins, and shared accountability across product,
infra, and ethics teams.

Real-World Application:
Stakeholder management builds the foundation for trust-driven innovation.
Prepared by Vaibhav Yendole 89
 Part 5: How This Topic Connects

Builds on:

 Basic product development knowledge (hypothesis → test → iterate)

 General awareness of AI and data systems

Enables:

 Better collaboration with technical and non-technical teams

 Leading responsible AI projects confidently

 Translating between AI performance and product outcomes

Related concepts:

 MLOps: The automation and tooling layer supporting the AI lifecycle

 Responsible AI: Fairness, transparency, and accountability frameworks

 Human-in-the-Loop (HITL): Humans providing corrective feedback to models

Part 6: Questions to Keep in Mind

1. How can AI PMs balance fast iteration with ethical and regulatory oversight?

2. If you were building an AI product, how would you ensure transparency and user trust?

3. What trade-offs exist between innovation speed, model safety, and stakeholder alignment?

Reflect:
Which stakeholder role do you personally relate to most — technical, ethical, or user-focused?

Quick Self-Check:

After reading, you should be able to:

 Explain why stakeholder management matters in AI

 Describe the AI product lifecycle in simple terms

 Identify key phases: pre-training, fine-tuning, deployment, monitoring

 Give one example (Tesla or Grammarly) of scaling AI responsibly

 Name one way stakeholders contribute to AI success

If not yet: Revisit the case studies and focus on who influenced what — that’s where the lifecycle truly
lives.

What’s Next?

Prepared by Vaibhav Yendole 90

 Explore deeper:

 Map the stakeholders in your current AI project.

 Identify potential conflicts or misalignments.

Practice:

 Sketch a mini lifecycle chart for your product idea.

 Highlight who is responsible for each phase.

Continue learning:

 Google AI PM Learning Path — foundational resource for PMs

 Tesla AI Day Talks — learn how iteration meets safety

 Grammarly Engineering Blog — insights on scaling GenAI systems

Final Thought

Building AI products isn’t just about smarter models — it’s about smarter collaboration.
Your role as an AI PM is to connect data, humans, and ethics into one aligned system.
Every improvement in AI starts with one principle: people build trust, and trust scales intelligence.

You’re now ready to dive deeper into AI product leadership

Lecture notes: Stakeholder Management for AI PM and AI Product Lifecycle, and Scaling Case
Studies

Why this matters

As an AI-PM, you win or lose not on a single model choice, but on how well you align stakeholders, pick
the right generation approach, and ship within cost, speed, and quality constraints. Today’s image
and video models can power virtual try-ons, creative tooling, and support experiences—but they’re also
compute-hungry and fickle. This session gives you the mental model to explain options to leadership,
brief design/engineering, and make roadmap trade-offs you can defend.

From recognition to generation

Early computer vision focused on image classification with CNNs—great at recognizing what’s in a
photo (think biometric checks in DigiYatra or Fast Tags), but not at creating new visuals. Generative
models changed that. GANs pair a generator that fabricates images with a discriminator that judges
realism; the adversarial loop sharpens quality. Variants like conditional GANs (guide with rules), pixel-
to-pixel GANs (precise edits), and CycleGANs (style/background transfer—hello, “magic eraser”
workflows) unlocked targeted creative control.

The current state of the art for realism is diffusion: start with noise and iteratively remove it guided by a
prompt until the picture matches your description. Stable Diffusion and DALL·E are examples. PM
lens: GANs are generally faster—useful for rapid mockups or stylized
art. Diffusion is slower/costlier but wins on photorealism. Choose based on your latency, cost, and
quality targets.
Prepared by Vaibhav Yendole 91
 Prompting that delivers

Great images begin with great prompts. Anchor to six

elements: subject, action, medium, material, photography style, and color palette/mood.
Add negative prompts (e.g., “no watermarks, no text”) to steer away from artifacts.

PM use case: Creative and growth teams can standardize prompt templates for brand-safe campaigns;
product can reuse them for consistent UI assets across markets.

Video generation

Today’s video models typically produce 8–15s clips without integrated audio (sound is added
separately). Architectures blend diffusion and transformers to predict the next frame. Popular tools
include VO2, Runway 2, SORA, and PICA labs. Enterprise-length or higher-fidelity outputs usually
need paid licenses and heavier infrastructure.

PM move: Coach your team on camera motion, lighting, perspective, and temporal cues to get
coherent motion from short prompts. Timebox experiments; track cost per second and render
latency as you iterate.

Choosing the right model class

When evaluating options, balance price, speed, and generalization (how broadly and accurately a
model handles varied requests). You’ll commonly weigh:

 Chat models (e.g., ChatGPT, Claude) for support/chatbots where accuracy and
factuality dominate.

 Completion models for autocomplete and content generation—optimize for relevance and
originality.

 Reasoning models for structured analysis—optimize for consistency.

 Multimodal models to mix text, image, audio, video—expect higher costs.

 Instruction-tuned models tailored to industries/tasks for better out-of-the-box alignment.

 RAG to ground outputs in external data for context and correctness.

A PM workflow

Co-pilot for support tickets is a solid pattern: categorize data, sample, and evaluate multiple
LLMs for accuracy before rollout. Ship, watch cost and performance, and iterate. Expect diminishing
returns from fine-tuning past a point; keep a toggle to fall back or switch models as pricing and quality
shift.

Technical knobs worth knowing

You don’t need to train models, but you should speak the language: learning rate and batch size affect
training stability/speed; temperature, top-p, and top-k control creativity and determinism at inference.
For GANs, remember the generator vs. discriminator dynamic; for diffusion, think prompt-guided
denoising from noise to image. This fluency builds stakeholder trust and sharpens your product specs.

Applications, limits, and what to watch

Prepared by Vaibhav Yendole 92

 Practical wins include virtual try-ons, color/style replacements, and other e-
commerce enhancements. Biometrics (face/iris) work well but have real-world limitations; be explicit
about confidence and fallback flows. Expect inconsistency across iterative edits and API
limitations in public services. Track tool pricing and licensing early so finance is not a late-stage
blocker. Upcoming focus areas: LLM evaluation, use-case-first model selection, stakeholder
management, and tech-debt planning.

Key takeaways

 Match model to job: GANs for speed and stylization; diffusion for realism when you can afford the
compute.

 Systematize prompts: Use the six-element template and negative prompts to hit brand and
quality bars.

 Plan for video constraints: Short clips, separate audio, costs that scale with seconds—measure
and communicate.

 Evaluate like a PM: Prioritize price, speed, and generalization; pick among chat, completion,
reasoning, multimodal, instruction-tuned, RAG by use case.

 Operationalize learning: Run side-by-side evaluations, monitor cost/performance, and

expect diminishing fine-tune gains.

 Speak stakeholder language: Know hyperparameters and core GAN/diffusion mechanics to

write tighter PRDs and explain trade-offs.

 Mind the edges: Biometric limits, edit inconsistency, and licensing constraints belong in your risk
log and roadmap.

Overview of the Lecture

The lecture focused on stakeholder management for AI product management and the AI product
lifecycle, with a special emphasis on image and video generation models, their types, workings, and
practical use cases. It covered foundational concepts, hands-on demonstrations, and the decision-
making process for product managers (PMs) regarding AI model selection for applications such as image
generation, video generation, and chatbot deployment.

Understanding Image Models in AI

 Image Classification and Early AI:

Initially, AI handled image classification tasks using convolutional neural networks (CNNs) to
recognize objects and patterns in images but could not generate images. Examples include
biometric identification in DigiYatra and Fast Tags.

 Generative Adversarial Networks (GANs):

GANs consist of two models – a generator and a discriminator – competing against each other to
generate realistic images. The generator creates images, and the discriminator judges their realism

Prepared by Vaibhav Yendole 93

 against real images, iteratively improving output. GANs can be conditioned via textual inputs to
guide image creation.

 Diffusion Models:
Current state-of-the-art image generation models use diffusion, where noise is gradually added to
an image and then removed while guided by a prompt, effectively reconstructing an image
matching the input description. Diffusion models produce highly realistic images but are
computationally intensive and costly. Examples include stable diffusion, DALL·E, and models
behind Nano Banana.

 Additional GAN Variants:

 Conditional GANs use rules to guide generation (e.g., ensuring facial features).

 Pixel-to-pixel GANs focus on precise pixel comparisons and replacements.

 Cycle GANs enable background replacement and style transfer (used in Photoshop's magic
eraser).

 Model and Cost Trade-offs:

GANs are faster and suitable for mockups or art, while diffusion models prioritize image quality and
realism but at higher computational cost. PMs must balance cost, speed, and quality demands.

Image Generation Prompting Techniques for PMs

 Successful image generation depends heavily on prompt engineering, focusing on six core
elements:

i. Subject: The main object (e.g., "young woman").

ii. Action: What the subject is doing.

iii. Medium: The type of artistic medium (e.g., watercolor, pencil sketch).

iv. Material: What the subject is made of (e.g., candies, bubbles).

v. Photography Style: Lens type, focus, lighting (e.g., close focus, volumetric light).

vi. Color Palette and Mood: Defines ambiance and textures.

 Using negative prompts can exclude unwanted elements like watermarks or text from generated
images.

 PMs can leverage these techniques to create realistic images aligned with product requirements,
marketing, and branding campaigns.

Video Generation Models

 Video generation AI models generate clips typically between 8-15 seconds, with no audio capability
integrated yet (audio is created separately).

 Common architectures combine diffusion and transformer models to predict the next video frame
based on previous frames.

Prepared by Vaibhav Yendole 94

  Popular tools include VO2, Runway 2, SORA, and PICA labs.

 Longer or enterprise-grade video generation requires paid licenses and more advanced
infrastructure.

 PMs should understand camera motion, lighting, perspective, and temporal cues to generate
coherent videos using limited prompt lengths.

Practical Insights on AI Product Management

 Model Selection for PMs:

PMs must evaluate models based on:

 Price: Cost per inference/training.

 Speed: Latency or throughput of outputs.

 Generalization: Ability to answer a wide variety of queries accurately.

 Types of AI Models used by PMs include:

[Link] Models (e.g., ChatGPT, Claude): For customer support and conversational agents. Accuracy and
factual correctness are paramount.

[Link] Models: Autocomplete or content generation models. Relevance and innovativeness matter.

[Link] Models: Perform higher-order thinking and analysis, requiring reliable and consistent answers.

[Link] Models: Handle multiple data types (text, image, audio, video). These are more expensive.

[Link]-tuned Models: Finetuned for specific industries or tasks for better performance.

[Link] Augmented Generation (RAG): Uses external data sources dynamically to improve accuracy
and context.

 PM workflow example:
Arun described building a Microsoft co-pilot AI to handle support tickets, involving data
categorization, sampling, evaluating multiple LLMs for accuracy, and iterating deployment for
performance and cost. Fine-tuning models yield limited accuracy improvements after a point.

Technical Concepts for PM Awareness

 Hyperparameters in AI Models:
Understanding parameters like learning rate, batch size, temperature, top-p, and top-k that
influence model training precision, speed, and creativity.

 GAN Training Dynamics:

The adversarial process of generator and discriminator improving iteratively until realistic output is
generated.

 Diffusion Model Principles:

Noise addition and removal at pixel level to generate new images from random noise guided by
prompts.

Prepared by Vaibhav Yendole 95

 Additional Topics Covered

 Differences between GAN variants and their use cases.

 How image recognition (face, iris scans) works practically with biometric data, including
limitations.

 Using AI for virtual try-ons, product color and style replacements, and e-commerce enhancements.

 Challenges in maintaining image consistency during iterative edits and the current limitations of
public AI APIs.

 Pricing and API references for various AI image and video generation tools.

 Upcoming sessions will cover LLM evaluation, choosing models based on use cases, stakeholder
management in AI PM, and tech debt management.

Summary

This lecture gave a comprehensive view on how AI models, especially image and video generation, work,
distinguishing between GAN and diffusion models, and how PMs need to approach AI product lifecycle
including model selection, trade-off considerations, and practical deployment strategies. It emphasized
the importance of prompt engineering, understanding model capabilities, costs, and the evolving
landscape of AI tools useful for product innovation and scaling.

Understanding Large Language Models (LLMs) and Model Types

LLMs encompass a variety of model types suited for different purposes:

 Chat Models: Provide customer support through conversational interactions. They require factual
accuracy.

 Completion Models: Used for tasks such as auto-completion in emails or documents, focusing on
relevance and creativity.

 Reasoning Models: Perform market research, problem-solving with reliable and consistent logical
outputs.

 Multi-modal Models: Capable of generating or understanding multiple data types (text, images,
video).

 Instruction-Tuned Models: Specialized for particular industries or use cases.

 Retrieval Augmented Generation (RAG): Enhance outputs by incorporating specific up-to-date

external data sources.

Additionally, models can be categorized by size:

 Large Language Models (LLMs): Several billion parameters, require considerable computation.

 Small Language Models (SLMs): Domain-specific, run on edge devices with lower latency and
offline capabilities.
Prepared by Vaibhav Yendole 96
 As PMs, understanding these distinctions and use cases enables better decision-making in selecting
appropriate AI models.

Generative Adversarial Networks (GANs)

GANs are a type of generative model involving two neural networks: a Generator and a Discriminator.

 The Generator creates images trying to mimic real images.

 The Discriminator evaluates whether an image is real or generated.

The Generator improves over time by trying to fool the Discriminator, leading to increasingly realistic
images.

Key points:

 Training involves iterative feedback from the Discriminator to the Generator.

 GANs are faster but less precise than diffusion models.

 Used for creating product mockups and synthesized faces.

Diffusion Models for Image Generation

Diffusion models generate images by gradually adding noise to an image and then learning to reverse this
noising process to reconstruct the original image or generate new images based on a prompt.

 Begin with a noisy image.

 Apply reverse diffusion steps conditioned on input text prompts.

 The model removes noise in a controlled manner to generate high-fidelity images.

Advantages over GANs:

 More stable and predictable training.

 Higher quality and realistic outputs.

Image Captioning and Image Classification in AI

Before image generation, AI focused mainly on image classification and captioning.

 Image classification identifies objects within images using convolutional neural networks (CNNs),
detecting edges, shapes, and colors.

 Image captioning pairs images with descriptive text, enabling models to generate textual
descriptions of images.

Use Cases:

 Biometric identification at airports (e.g., Digiatra uses facial recognition based on iris scans and
biometrics).

 Automatic tagging in photo storage (Google Photos).

 Vehicle license plate detection using object recognition.

Prepared by Vaibhav Yendole 97
GAN Variants and Applications

There are different GAN architectures designed for specific tasks:

 Conditional GANs: Use conditioning information (such as text prompts) to guide image generation.

 Pixel-to-Pixel GANs: Focus on generating images pixel-by-pixel.

 CycleGANs: Capable of translating images from one style to another (e.g., horse to zebra), often
used in background replacement.

These variants improve control and precision depending on the use case.

Prompt Engineering for Image and Video Generation

Effective AI image and video generation requires detailed and specific prompts that guide the model in
producing high-quality outputs. Key components to include in a prompt:

 Subject/Action: Define what the image/video should depict.

 Medium/Material: Specify the artistic medium or material (e.g., watercolor, candies).

 Photography/Lighting Style: Describe lighting conditions, lens type, and mood.

 Color Palette: Indicate desired color schemes.

 Negative Prompts: Explicitly specify what should not appear (e.g., no text or watermarks).

Good prompt engineering can significantly enhance generated content quality.

AI-Powered Video Generation Models

Popular tools include:

 Veo (Google VO2/VO3)

 Sora

 Runway ML

 Stable Diffusion video extensions

A dog diving into a pool wearing swimming goggles, video shot with a drone, bright daylight, smooth slow
motion.

Choosing AI Models: Considerations for Product Managers

Product Managers must choose AI models based on various criteria:

 Use Case Alignment: Does the model fit the specific requirement (chat, completion, reasoning,
multimodal, etc.)?

 Performance Metrics: Accuracy, relevance, consistency, and creativity.

 Speed and Latency: The response time suitable for the user experience.

Prepared by Vaibhav Yendole 98

  Cost: Expense per inference or training, budget constraints.

 Privacy and Data Handling: Compliance with data policies.

Example of decision process:

 Identify use case.

 Review model accuracy and cost.

 Test candidate models with representative data.

 Launch pilot, monitor performance and user feedback.

Example

PM Decision:

- Use Case: Customer support chatbot.

- Priority: Accuracy and factual responses.

- Model Options: GPT-based chat models, Claude.

- Selected Model: Claude for higher factuality with acceptable latency.

Iterative Improvement and Fine-Tuning of AI Models

Fine-tuning pretrained AI models improves performance but has practical limits.

 Small to moderate accuracy gains (typically up to ~12%) with fine-tuning.

 Larger improvements require newer or fundamentally different models.

 Continuous feedback loops with real user data are essential for relevance.

Fine-tuning approaches:

 Adding domain-specific training data.

 Reinforcement learning from human feedback.

Example Scenario:

- Base model accuracy: 65%

- Fine-tuning on domain data: improves to 75%

- Adding more data or new models needed beyond that.

AI in Practical Product Management: Case Study at Microsoft

Example of AI PM lifecycle:

 Analyze historical support ticket data.

 Categorize tickets and identify recurring issues.

Prepared by Vaibhav Yendole 99
  Develop dataset with questions, existing answers, and customer satisfaction.

 Deploy multiple LLMs (e.g., Gemini, Claude) to evaluate accuracy.

 Select model and fine-tune with organization-specific data.

 Launch AI co-pilot to reduce support workload.

 Monitor effectiveness and iterate.

Tools like Azure AI Foundry can host models with legal data controls.

Example Workflow:

1. Data Collection: 40,000 tickets/month

2. Categorization: 5 main issue types

3. Sample & Label: 50,000 samples labeled by satisfaction

4. Model Testing: 1000 test samples with multiple LLMs

5. Deployment: Selected best-performing model

6. Result: 62% reduction in task tickets over 2 months

Prepared by Vaibhav Yendole 100

 W19L3 Tech Debt in AI Systems: Model Upkeep, Agile Iteration, and
Monitoring Tools (AN)

1. Introduction: What is Technical Debt in AI and Why Should You Care?

Core Definition

Technical debt is the future cost of reworking shortcuts taken today. In AI systems, debt does not live
only in code; it also accumulates in data pipelines, models/metrics, and infrastructure. Unpaid debt
compounds as bugs, slower iteration, quality regressions, and outages, especially once a model is in
production and serving users.

A Simple Analogy

Think of AI systems like gardens. Shipping a model is planting; weeding, watering, pruning are the
monitoring and upkeep. If you skip upkeep, weeds (drift, data issues, stale features) take over.
Limitation of the analogy: Gardens change with seasons; AI drift can be abrupt (policy change, new
user segment) or silent and slow—your “weeds” may be invisible without instrumentation.

Why This Matters to You

 Problem it solves: Teams ship a promising model that degrades after launch. Six weeks later, fire
drills. With explicit model upkeep + monitoring, you spot drift early and fix root causes.

 What you’ll gain:

 A mental model for spotting debt (data/model/infra).

 A repeatable iteration cycle for maintenance.

 A tooling map (Evidently AI, WhyLabs) to monitor what matters.

 Real-world context: The same loop runs behind product teams at recommendation engines, fraud
detection, and content moderation systems.

Remember: AI success is not the day you deploy. It’s the months after, when reality meets your
assumptions.

2. The Foundation: Core Concepts Explained

Concept A: Types of Technical Debt in AI

Definition. Technical debt in AI spans (1) data debt, (2) model debt, and (3) infrastructure/process
debt. These interact: poor data logging (data debt) hides drift (model debt), and weak CI/CD (infra debt)
slows safe fixes.

Key characteristics:

 Data debt: Missing lineage, weak validation, untracked schema changes, no sampling strategy,
biased or stale labels.

 Model debt: Overfit baselines, brittle features, silent drift, unlabeled segments, unclear
thresholds/alerts.
Prepared by Vaibhav Yendole 101
  Infrastructure/process debt: Manual deploys, no reproducible pipelines, missing tests, no
canaries/rollbacks.

Concrete example: A schema change renames user_id → uid in production logs. Your feature job still
expects user_id, silently producing nulls → performance drops (data debt cascading into model debt).

Common confusion: “We refactored code, so debt is gone.” Correction: In AI, debt also
lives outside code—in data contracts, feature stores, labeling, and monitoring.

Concept B: The AI Product Lifecycle (Pre-train → Fine-tune → Deploy → Monitor)

Definition. A minimal lifecycle continuously collects/curates data, trains or fine-

tunes, validates, deploys with guardrails, then monitors live signals (performance, drift, data health)
and feeds findings back into the next iteration.

How it relates to Concept A: Each stage can introduce debt (e.g., shortcut labels in fine-tuning, rushed
deploys) or pay down debt (e.g., add validation checks, adopt canary releases).

Key characteristics:

 Pre-train/fine-tune: Establish baselines, label quality checks, hold-out policies.

 Deploy: Version models and features; add gates and rollbacks.

 Monitor: Instrument prediction quality, data drift, feature health, latency, and cost.

Concrete example: Weekly loop—ingest new labels, retrain, validate on a holdout, canary 10%, check
dashboards for drift/anomalies, then ramp to 100%.

Note (prerequisite connection): If you know train/validation/test splits, monitoring is like a new “post-
deployment test” that never stops.

How A and B Work Together

Think of Concept A (debt types) as what can go wrong, and Concept B (lifecycle) as when to
prevent/fix it. The lifecycle is your calendar; the debt types are your checklist at each step.

3. Seeing It in Action: Worked Examples

Tip: Read the “why” in each step. That’s what makes these habits stick.

Example 1 — The Basic Case (Detecting Feature Drift with Evidently AI)

Scenario. A binary classifier predicts “churn risk.” Last month’s AUC: 0.84. Now product managers
report more false positives.

Our approach:

1. Check data drift first; it’s the most common cause of sudden performance changes.

2. Use a drift analysis tool (e.g., Evidently AI) to compare current feature distributions vs. training.

Step-by-step:
Prepared by Vaibhav Yendole 102
 1. Prepare two datasets: a reference sample from training/validation time and a current sample
from recent production traffic.

2. Select the same features used by the model (e.g., age, tenure, session counts, plan type).

3. Run a data-drift report: compare distributions (means, histograms, category mix) and receive a
drift score/flag per feature.

4. Review the summary: identify which features show significant drift (e.g., plan_type mix
changed; avg_sessions shifted).

5. Share the report (HTML/PDF/dashboard) with the team and log decisions in a runbook.

Output: A clear list of which features drifted and by how much.

What just happened: We validated the fastest hypothesis—data shifted. If key features drift, the
model’s learned boundaries no longer reflect reality.

Check your understanding: Why do we check data drift before rebuilding the model?

Example 2 — Adding Complexity (Closing the Loop with WhyLabs)

Scenario. You’ve confirmed drift in plan_type and occasional missing values in avg_sessions. You
need continuous monitoring to prevent surprise regressions.

What’s different: We move from one-off forensics (a drift report) to always-on monitoring (e.g.,
WhyLabs Free Tier).

Solution:

 Instrument data health summaries in your prediction service (counts, nulls, ranges, rare-value
rates).

 Connect to WhyLabs (or a similar platform) to create anomaly detections and alerts on feature
health and error spikes.

 Set actions: If nulls exceed 2% for avg_sessions, auto-trigger a canary rollback.

Output: A dashboard with feature-level anomaly alerts, aggregated over time windows,
and notifications to Slack/Email.

Key lesson: Tools complement each other. Use a drift tool (Evidently) for analysis and a monitoring
platform (WhyLabs) for live anomaly and error monitoring.

Check your understanding: What alert threshold would you set for nulls or drift? How would you prevent
alert fatigue?

Example 3 — Real-World Application (Agile Loop with Debt Budget)

Background. Your team ships ranking models bi-weekly. Debt grew (manual deploys, flaky tests,
undocumented features).
The challenge: Improve stability without slowing feature velocity.

Approach: Add a debt budget to the sprint (e.g., 20% capacity) and explicit gates:
Prepared by Vaibhav Yendole 103
  Before merge: unit + data validations pass; feature schemas checked.

 Before deploy: canary at 5–10% + guardrails (latency, error rate, drift).

 After deploy: dashboards (WhyLabs) + weekly drift review (Evidently or equivalent).

Outcome: Fewer fire drills, predictable releases, more trust from stakeholders.

Caution: Skipping documentation is tempting. But undocumented features block incident response
and new hires—debt interest you will pay later.

4. Common Pitfalls (and How to Avoid Them)

The Mistake Why It’s a Problem The Right Approach Why This Works

Treating tech Makes non-code

Ignores data, model, Maintain a debt register with
debt as “just risks visible and
and infra risks categories (data/model/infra)
code” owned

Prevents
No data Silent schema breaks Define feature schemas + CI
regressions
contracts cause null features checks
before prod

Always-on dashboards Early detection

One-off You see
(WhyLabs) + periodic drift reduces blast
monitoring issues after users do
reviews (Evidently) radius

No canary or Bad models hit 100% of Canary 5–10% + auto-rollback Limits user
rollback traffic on guardrail breach impact

Solve the cause,

Rebuild-first Retrain hides root- Check data
not just
reflex cause data issues drift/anomalies before retrain
symptoms

Caution: Alert fatigue is real. Tie alerts to actionable thresholds (e.g., “drift p-value < 0.01 for 3 hours”
triggers rollback).

Tip: Add explainers to dashboards (“What this alert means” + “How to respond”) to help on-call
engineers.

If you’re stuck: Revisit Section 2 (debt types) and Section 3 (worked examples). Align your incident with
the right category and tool.

5. Your Turn: Practice & Self-Assessment

Practice Task

Prepared by Vaibhav Yendole 104

 The challenge: You own a text-classification API that flags harmful content. Over two weeks, false
negatives spike.
Specifications:

1. Draft a runbook with 3 checks in this order: (a) data/feature health, (b) distribution drift, (c)
performance by segment.

2. Propose two alerts in WhyLabs (one anomaly, one error-rate) with thresholds.

3. Describe one drift/quality report you’ll run weekly (e.g., with Evidently) and what decision it
informs.

Hint: Start with input health first (nulls, rare-token rates), then check drift on top features. Segment
performance by language/region to catch hidden pockets.

Extension (optional): Add a canary + rollback policy and define guardrail metrics (latency, error rate,
drift p-value).

Check Your Understanding

1. Explain: Why is data drift often the first hypothesis when performance drops?

2. Apply: In a weekly loop, where do Evidently AI and WhyLabs each fit?

3. Error analysis: Your alert triggers on drift, but performance is steady. What might be wrong with
your thresholding?

4. Transfer: How would you adapt this playbook for a generative AI use case (e.g., hallucination rate,
prompt drift)?

Answers:

1. Drift changes the input distribution; the model’s learned boundaries no longer match reality—
most common cause.

2. Evidently → in-depth drift/quality analysis; WhyLabs → always-on anomaly/error monitoring.

3. Possibly too-sensitive thresholds or features with low importance; combine drift with impact
metrics before paging.

4. Track LLM-specific signals: refusal rate, toxicity, hallucination evals, and prompt/response drift;
apply the same canary/rollback/monitor loop.

Self-Assessment Checklist (you’ve mastered this if you can):

 Name the three AI debt categories with examples.

 Sketch a pre-train → deploy → monitor loop with specific checks.

 Explain when to use Evidently vs. WhyLabs.

 Propose actionable alert thresholds tied to responses.

 Add canary + rollback without slowing the team.

Prepared by Vaibhav Yendole 105

 Quick Reference Card

Debt categories:

 Data: schema, quality, bias, freshness

 Model: drift, thresholds, segmentation gaps

 Infra/process: CI/CD, tests, canaries, rollbacks

Lifecycle: Pre-train → Fine-tune → Deploy → Monitor → Feedback

Tooling map: Evidently (deep reports) + WhyLabs (live anomalies/errors)

Guardrails: Latency, error rate, drift p-value, null rates, segment performance

Lecture Summary

Overview of the Lecture

The lecture primarily focused on understanding the technical debt in AI systems with an emphasis on
model selection, upkeep, agile iteration, and monitoring tools. It covered how product managers (PMs)
should evaluate AI models based on cost, speed, intelligence, and use-case specificity, alongside
understanding various benchmarking systems used to measure AI model performance. It also explored
the differences between open-source and proprietary models and how companies can fine-tune models
based on their needs.

Model Selection Criteria for AI Applications

 Cost Efficiency: PMs must evaluate the pricing models of AI APIs, typically charged per million
tokens. Understanding token usage and translating it into costs is crucial to avoid overspending,
especially at scale.

 Response Speed: Speed is vital for end-user experience; models must balance between speed
and cost. Ideal token generation speed is around 48 to 60 tokens per second.

 Intelligence and Capabilities: The model must meet the required knowledge or reasoning
capabilities tailored to the specific use case (e.g., customer support chatbot, story generation).

 Provider Restrictions: Companies may have preferences or restrictions on providers due to

compliance or security, influencing model choice.

 Model Type: Deciding between Small Language Models (SLM), Large Language Models (LLM),
open-source models (downloadable and fine-tunable), and proprietary APIs (restricted fine-tuning
mostly through vendor platforms).

Benchmarks and Testing AI Model Intelligence

 Common Benchmarks:

 MMLU (Massive Multitask Language Understanding) tests broad multitasking knowledge

across many subjects.

 Graduate-level reasoning benchmarks focus on scientific and logical problem-solving.

Prepared by Vaibhav Yendole 106

  Human-level exams (HLE / AGI measures) test real-world decision-making, ethics, and
complex reasoning.

 Mathematical reasoning and live coding benchmarks evaluate models’ performance on

solving equations and debugging code.

 Truthfulness benchmarks like TruthfulQA measure the model's honesty and reduction of
hallucination.

 HellaSwag tests common sense reasoning in story continuation tasks.

 Benchmark Use: Benchmarks provide objective quantitative scores (percentages) evaluating how
many questions or tasks the model can solve correctly. These scores aid PMs in deciding which
model best suits their product needs.

 Domain-Specific Benchmarks: Industries like healthcare, finance, or legal have specialized

benchmarks, and PMs should incorporate these for their specific use cases.

 Model Drift Considerations: Newer models might not always be better; PMs must monitor
benchmarks over time to check if updating to a new model improves or degrades performance.

Model Fine-tuning and Customization

 Proprietary models typically cannot be fully downloaded and fine-tuned on private data directly,
but fine-tuning can happen through platforms like OpenAI’s playground or Azure.

 Open-source models can be downloaded, hosted locally, and fine-tuned extensively, allowing for
customized improvements and better privacy control.

 Whether to fine-tune a model locally or use a proprietary API depends on budget and company
resources (buy vs. build decisions).

Tools and Websites for Model Evaluation

 [Link]: Provides a simple UI to compare price, speed, model capabilities, and

evaluation benchmarks across different AI models for various use cases like text-to-speech, image
generation, and intelligence.

 LLM Stats: Offers detailed leaderboard-style statistics on multiple benchmarks, performance

metrics, latency, uptime, and model parameters (e.g., open source vs. proprietary, knowledge
cutoff dates).

 Image Arena: Specialized for image generation model benchmarking, validating models against
scores like ELO and confidence intervals.

 These platforms allow filtering by use case, cost limits, provider, and model capabilities, helping
PMs make informed decisions.

Additional Important Concepts Discussed

 Tokenization: Tokens count instead of characters or words; different models use specific
tokenizers. Token consumption must be estimated carefully for cost calculation.

 Knowledge Cutoff Date: Knowing the last training data date is critical, especially for fast-evolving
domains, to understand what the model knows.

Prepared by Vaibhav Yendole 107

  Safety and Uptime: API reliability is quantified by uptime and failure rates; these are important for
enterprise decisions. Contractual SLAs for uptime can influence vendor selection.

 Trade-offs in Model Adoption: Switching models frequently can be challenging; PMs need to
weigh the benefits of new models vs. stability and cost management.

 Choosing Models for Company Scale: Larger companies might have higher budgets but also
higher demands for consistency and cost control. Smaller startups might experiment more but
must be cautious about budget burn.

 Industry Benchmarks and Custom Metrics: Use case-specific benchmarks or metrics must be
developed for a real-world evaluation that aligns with business goals.

Summary

 Start evaluating AI models with foundational criteria: cost per million tokens, response speed
(ideally 48-60 tokens/second), and basic intelligence aligned with the use case.

 Use authoritative benchmarking datasets (MMLU, TruthfulQA, HellaSwag, coding challenges,

scientific reasoning) to quantify model intelligence and truthfulness.

 Understand the difference between open-source and proprietary models regarding customization
and fine-tuning capabilities.

 Utilize online tools like [Link] and LLM Stats for live comparative analysis across
models.

 Consider provider restrictions, data privacy needs, and operational stability (API uptime, latency)
as critical to model selection.

 Incorporate domain-specific benchmarks for more accurate performance assessment and product
relevance.

 Monitor model drift over time before migrating to newer models to ensure performance
consistency.

 Tokenization and knowledge cutoffs are key technical points for budgeting and evaluating model
applicability.

 Safety, contract terms, and API reliability influence enterprise readiness and should be included in
decision making.

 PMs should balance technical metrics with practical business constraints to select the best AI
model for their product.

This comprehensive approach will help product managers systematically select, evaluate, and maintain
AI models aligned with product goals and technical feasibility.

Primary Evaluation Criteria for Model Selection: Cost, Speed, and Provider Constraints

The first level of evaluation for selecting an AI model focuses on practical considerations:

Prepared by Vaibhav Yendole 108

  Cost: Calculate expected monthly usage (queries or tokens) and the associated cost with the
chosen model to fit the budget.

 Speed: Ensure the model provides responses within an acceptable latency, usually close to sub-
second for user queries.

 Provider Constraints: Some companies restrict usage to certain providers or models based on
policies or security compliance (e.g., using only OpenAI models).

Tokenization and Understanding Token Costs

Tokens are units of text that models process; they are not directly equivalent to characters or words. Cost
and usage are typically calculated based on tokens.

 Each API has its own tokenizer for splitting text into tokens.

 The number of tokens in a query and response determines the cost.

 Estimating token usage helps forecast expenses.

Model Drift and Continuous Evaluation

Models evolve over time with new versions released regularly. Model drift refers to performance
degradation or unexpected changes when switching to newer versions.

 PMs must evaluate if newer models outperform existing ones on relevant benchmarks before
migration.

 Continuous monitoring ensures the AI application maintains or improves quality.

Open Source vs. Proprietary Models and Fine-tuning

Models can be open source or proprietary with different capabilities for customization.

 Open Source Models: Can be downloaded, hosted locally, and fine-tuned on private data for
customized performance.

 Proprietary Models: Offered via APIs; fine-tuning is possible but must be done on the provider's
platform (e.g., OpenAI fine-tuning on Azure).

 Fine-tuning allows adapting a general model to specific datasets, improving relevance and
reducing costs.

Knowledge Cutoff Date and Context Window Size

Two critical model properties influence utility:

 Knowledge Cutoff Date: The latest date until which the model was trained on data. Affects the
model’s relevance to recent events.

Prepared by Vaibhav Yendole 109

  Context Window Size: The maximum amount of input (tokens) the model can attend to at once.
Larger context windows enable understanding of longer documents.

PMs must consider these properties when selecting a model.

// Model A knowledge cutoff: 2023-09-01

// Model B knowledge cutoff: 2024-01-31

// Model A context window: 4,096 tokens

// Model B context window: 32,768 tokens (better for long documents)

Tools and Websites to Analyze and Compare Models

Several tools and websites help PMs compare AI models on cost, speed, benchmarks, and capabilities.

 [Link]: Allows filtering models by use case and comparing cost and performance.

 [Link]: Provides detailed benchmark scores and model metadata like context window,
open-source status, and knowledge cutoff dates.

 [Link]: Specialized for image generation model comparisons.

Prepared by Vaibhav Yendole 110

 W20L1 RAG Fundamentals: Concepts, Use Cases, and Tools for Product
Managers (Prof  AN)
Why RAG Matters

Imagine your product chatbot confidently gives users answers — but occasionally, it makes things up.
A customer asks about warranty coverage, and your bot assures them of a benefit that doesn’t exist.
This happens because traditional Large Language Models (LLMs) rely only on their training data, which
may be outdated or incomplete.

That’s where Retrieval-Augmented Generation (RAG) transforms the game.

RAG allows AI systems to look up real data from external sources before generating a response. It
combines the creativity of generation with the reliability of retrieval, ensuring that every answer is
factually grounded and contextually accurate.

In short: RAG makes AI systems accurate, current, and context-aware.

Concept: What Is Retrieval-Augmented Generation (RAG)?

Definition:
Retrieval-Augmented Generation (RAG) is an architecture that enhances large language models by
connecting them with external data sources. It retrieves the most relevant information from these
sources and uses it to generate accurate, contextual responses.

Core Components:

1. Retriever: Finds relevant data or documents based on a user query.

2. Generator: Produces a natural language response using both the retrieved data and the model’s
internal knowledge.

3. External Knowledge Base: The “library” from which the system retrieves information (e.g., PDFs,
APIs, product databases).

Analogy:
Think of RAG as a student answering a test. Instead of relying only on memory (LLM), they are allowed to
check verified notes (retrieval) before answering.

Why It Matters — The Need for RAG

Challenge with Traditional LLMs How RAG Fixes It

Outdated information due to training data Retrieves live, updated data from verified
cutoffs. sources.

Hallucinations — generating false or made-up

Cross-verifies data before responding.
facts.

Prepared by Vaibhav Yendole 111

 Challenge with Traditional LLMs How RAG Fixes It

Augments the model with organization-specific

Generic responses lacking product context.
data.

For Product Managers:

 Builds customer trust through accurate responses.

 Reduces retraining costs.

 Speeds up feature development that relies on factual AI output.

Mini Recap: RAG = Reliable + Adaptive + Grounded responses.

Concept Deep Dive — How RAG Works

Overview:
RAG follows a systematic 5-step pipeline:

Step Process PM Example

1. Data Gather all relevant internal and external data Product specs, FAQs, policies,
Collection sources. manuals.

2. Data Break text into smaller, topic-specific sections Split FAQs into individual
Chunking (chunks). question-answer pairs.

3. Embedding Convert text chunks into vector Enables semantic similarity

Creation representations capturing meaning. search.

User asks: “How to reset my

Find the most relevant text chunks for a given
4. Retrieval device?” → retrieves reset
query.
section.

LLM combines retrieved data with its internal

Generates step-by-step reset
5. Generation knowledge to produce a precise, contextual
instructions.
response.

Visual Placeholder:
A diagram showing: User Query → Retrieval → Augmentation → Generation → Output

Concept Focus — External Data Retrieval

Definition: External Data Retrieval means fetching relevant, verified information from sources outside
the model’s native training data.
These can include APIs, databases, CRMs, or documentation libraries.

Prepared by Vaibhav Yendole 112

 Example:
A healthcare app uses RAG to fetch dosage information directly from the hospital’s verified medical
database before generating a doctor’s response.
This ensures factual correctness and compliance.

Why It Matters for PMs:

PMs can ensure their AI products remain reliable without constant retraining or manual content updates.

Mini Recap: Retrieval = Real-time truth checking for AI.

Concept Focus — Prompt Augmentation

Definition: Prompt Augmentation is the process of enriching the LLM’s prompt with the retrieved data
before generating an answer.

How It Works:
When a user asks, “What are the warranty terms for Model X?”, RAG retrieves that specific product’s
warranty document, and the model’s input becomes:
“User asked about Model X. Warranty details: [retrieved paragraph]. Generate a concise answer.”

Benefit:
Ensures responses are both contextually rich and factually correct.

PM Lens:
Prompt augmentation allows fine-grained control over how data is integrated into responses — critical
for regulatory, compliance, and UX-sensitive domains.

Mini Recap: Augmentation = Enriching prompts for precision.

Example — How RAG Transforms Product Support

Scenario:
You manage customer experience for an electronics brand. Customers frequently ask about product
setup, troubleshooting, and warranty coverage.

Traditional LLM: Gives general answers, sometimes inaccurate.

RAG-Powered Bot: Retrieves exact instructions from internal manuals and provides verified solutions
instantly.

Impact:

 45% faster customer resolution time.

 30% higher satisfaction scores.

 Significant reduction in repetitive support queries.

Tools & Frameworks for RAG Implementation

Category Tool Examples Why It Matters for PMs

Prepared by Vaibhav Yendole 113

 Category Tool Examples Why It Matters for PMs

Simplify retrieval + generation

RAG Frameworks LangChain, LlamaIndex
integration.

Vector Databases Milvus, Pinecone, FAISS Enable fast semantic search.

LLMs GPT-4, Claude, Mistral Generate language responses.

Embedding OpenAI Embeddings, Sentence- Convert text to vector space for

Models BERT retrieval.

PM Tip: When scoping an AI feature, align the RAG architecture with data security, latency, and scaling
requirements.

Mini Recap: Frameworks simplify build; vector DBs power retrieval.

Limitations and Best Practices

Challenge Description PM Best Practice

Integration Multiple data formats and APIs can make Start small — one clean
Complexity retrieval messy. source first.

Maintain a regular data refresh

Data Quality Issues Poor or outdated data reduces accuracy.
process.

Use caching and optimize

Latency Retrieval can slow down responses.
embeddings.

Include human-in-the-loop
Hallucination Risk Model may still misinterpret data context.
QA.

Mini Recap: High-quality data and structured retrieval ensure reliable AI outcomes.

Reflection Prompts

1. What internal data sources could strengthen your product’s AI capabilities?

2. How might you measure the factual accuracy of your AI product?

3. What user journey in your product could benefit from real-time retrieval?

Think About: How can RAG help balance creativity and factual reliability in your domain?

Key Takeaways

Prepared by Vaibhav Yendole 114

  RAG = Retrieval + Augmentation + Generation.

 Combines factual accuracy with natural language fluency.

 Reduces hallucinations and enhances product reliability.

 Enables PMs to deliver grounded, data-informed AI experiences.

 Implementation success depends on clean data, thoughtful architecture, and continuous

evaluation.

Come prepared to discuss a real-world example of how RAG could enhance your product’s AI-driven
features.

Limitations of Large Language Models (LLMs)

Large Language Models (LLMs), despite their capabilities, have several limitations that affect their
practical use, especially in enterprises.

 Knowledge Cutoff: LLMs are trained on data up to a certain date (knowledge cutoff). They cannot
access information beyond this date unless fine-tuned or retrained.

 Static Knowledge: Once trained, LLM's knowledge remains static and does not update
dynamically.

 Hallucination: LLMs may generate plausible-sounding but incorrect or fabricated information

(hallucinations).

 Generalization: LLMs provide generalized information and do not consider personalized or

proprietary data unless explicitly included in the prompt.

 Token Limitations: LLMs have a limited context window, meaning they can process only a fixed
number of tokens at a time.

 Data Privacy and Security: Enterprises hesitate to upload proprietary data to public LLMs due to
concerns about privacy and uncontrolled model training.

Introduction to Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is a technique that augments the generation ability of LLMs with
relevant documents retrieved from a proprietary or dynamic knowledge base, addressing many LLM
limitations.

 Retrieval: At query time, relevant information is retrieved from a private or dynamic data source.

 Augmentation: The retrieved information is added (augmented) to the prompt given to the LLM.

 Generation: The LLM generates a response based on both its internal knowledge and the provided
external context.

This ensures the response is grounded on the latest or proprietary data without exposing the data for
model training.
Prepared by Vaibhav Yendole 115
 Advantages of RAG:

 Dynamic knowledge updated independently from the model.

 Provides traceability by citing data sources.

 Reduces hallucinations by grounding in real data.

 Efficient token usage by limiting input to relevant chunks.

 Data privacy as proprietary data is not used for general model training.

Data Control and Privacy in RAG

Data control is crucial in enterprise use cases where privacy and proprietary data must be protected.

 In a RAG system, enterprises maintain ownership and control over their knowledge base.

 Data is never uploaded to the LLM service; only retrieved relevant chunks are augmented with the
prompt.

 Role-Based Access Control (RBAC) can restrict who can access what data.

 Hashing and encryption techniques secure embeddings and stored data.

 Enterprise RAG implementations often include governance layers to ensure compliance.

Vector Databases and Embeddings in RAG

Vector databases are specialized databases that store vector embeddings representing text chunks.

 Embeddings: Each document or text chunk is converted to a numerical vector capturing semantic
meaning.

 Vector Database: Stores these embeddings and supports similarity search to find relevant chunks
efficiently.

 Advantages: Suitable for unstructured data from multiple sources - PDFs, websites, databases.

 Popular Vector Databases: Pinecone, Supabase, etc.

Chunking, Indexing, and Overlapping in RAG

Large documents cannot fit entirely into an LLM prompt due to token limits; chunking is the technique of
breaking documents into manageable pieces.

 Chunking Strategies: Fixed size, recursive, or based on sentence/paragraph boundaries.

 Overlapping: Chunks can overlap (e.g., last 50 tokens of one chunk repeat at the start of the next)
to preserve context across chunks.

 Indexing: Each chunk is indexed with its embedding for retrieval.

 Tradeoffs: Smaller chunks may increase precision but increase latency and token usage.

Prepared by Vaibhav Yendole 116

Retriever, Generator, and Knowledge Base Components in RAG

RAG combines three main components:

1. Retriever: Finds relevant chunks from the knowledge base based on the user query.

 Controlled by system instructions which guide what constitutes 'relevant.'

2. Generator: LLM that generates the response based on the augmented prompt.

 Balances weight between LLM’s internal knowledge and retrieved context.

3. Knowledge Base: Source documents that are chunked, embedded, and indexed.

System Instructions in RAG

System instructions are prompts or rules given to the retriever or generator to control behavior.

 They instruct retriever on how to prioritize or filter documents.

 They instruct the generator on the style or constraints of response generation.
 Precise system instructions improve retrieval relevance and reduce hallucinations.

Handling Latency, Cost, and Security in RAG

RAG systems must balance efficiency, cost, and security.

 Latency: More data and smaller chunks increase retrieval time.

 Cost: More tokens processed means higher API or computation costs.

 Security: Role-based access control (RBAC) ensures users see only authorized data.

Strategies to optimize:

 Label and tag data during ingestion to filter efficiently.

 Avoid storing redundant or irrelevant data.

 Use secure vector storage with encryption.

Practical Implementation and Visualization of RAG

RAG implementations involve:

 Uploading documents.
 Chunking and embedding texts.
 Storing vectors in a vector database.
 Query embedding matching.
 Prompt augmentation and generation.
Visualization tools help understand chunk embeddings by projecting high-dimensional embeddings into
2D or 3D spaces for inspection.

Practical example: Using a playground to upload match commentary, chunk it, embed chunks, retrieve
relevant ones for queries like "Who scored a century?", and generate responses.

Prepared by Vaibhav Yendole 117

 W20L2 Type of LLM Models, LLM Benchmarks, GAN and other Image
Models (AN)
1. Why Learn This?

AI is built from many types of models — some understand language, others generate text, and some
create images or videos.
As a Product Manager, Designer, or Analyst, knowing what these models do helps you choose the right
tool for your product.

By the end of this reading, you’ll know:

 The main types of Large Language Models (LLMs) and their differences.

 What GANs and other image models are and when to use them.

 How to interpret benchmarks like MMLU and GSM8K.

 Which tools help compare models (Artificial Analysis, LM Arena).

2. Types of Large Language Models (LLMs)

LLMs don’t all work the same way. Think of them like types of engines — each designed for different
terrains.

2.1 Decoder-Only Models – The Storytellers

Decoder-only models are great at writing, generating, or completing text.

They read your prompt and continue it naturally — like finishing your sentence.

Examples: GPT, Mistral, LLaMA.

Use Cases: Chatbots, article generation, summarization.
Key Trait: They think forward, predicting one word at a time to create natural language.

Analogy: Imagine a storyteller who keeps adding sentences to make a full story.

2.2 Encoder-Only Models – The Analyzers

Encoder-only models don’t generate text — they understand and classify it.
They are like editors who read and analyze text to find meaning, patterns, or categories.

Examples: BERT, RoBERTa.

Use Cases: Search ranking, sentiment analysis, content moderation.
Key Trait: They process language by reading everything at once and capturing relationships between
words.

Analogy: Think of an editor who highlights errors and tags topics instead of writing new content.

2.3 Encoder-Decoder Models – The Translators

Prepared by Vaibhav Yendole 118

Encoder-decoder models combine both powers — they read input and produce new structured output.
They’re used for tasks like translation or summarization, where the output depends on the full input.

Examples: T5, FLAN-T5.

Use Cases: Machine translation, summarization, Q&A systems.
Key Trait: They encode meaning from input and decode it into a rephrased version.

Analogy: A translator listens to you in one language and speaks fluently in another.

2.4 Multimodal Models – The All-Rounders

These models can process text, images, and even audio or video together.
They understand context across different media — for example, describing an image or answering
questions about a chart.

Examples: GPT-4o, Gemini, LLaVA.

Use Cases: Vision-language chat, data dashboards, accessibility tools.
Key Trait: They merge multiple types of input into one reasoning engine.

Analogy: Imagine a smart assistant that looks at an image, reads a caption, and tells you what’s
happening.

3. What to Measure in LLMs

Choosing a model isn’t about picking the “biggest” one — it’s about the right balance between speed,
cost, and accuracy.

Key Metrics to Understand

 Accuracy / Quality: How correct or relevant the output is (measured by benchmarks like MMLU).

 Latency: How fast the model responds — critical for live applications.

 Tokens per second: The throughput of text generation (useful for measuring performance).

 Context window: How much text the model can “remember” at once.

 Cost: How much GPU power or money it consumes during inference.

Tool Tip: Explore [Link] to visually compare models across these parameters — accuracy vs
cost, latency vs throughput, etc.

4. GANs and Other Image Models

4.1 What Are GANs?

Generative Adversarial Networks (GANs) are AI models that create new images, videos, or sounds.
They have two parts:

 Generator: Makes new data (e.g., a realistic photo of a face).

 Discriminator: Judges whether the data looks real or fake.

Prepared by Vaibhav Yendole 119

Both improve together — like two artists competing and learning from each other.
Over time, the generator produces hyper-realistic results.

Applications: Face generation, art, image enhancement, data augmentation.

Analogy: One person forges artwork, another tries to spot fakes — both get better with practice.

4.2 Diffusion Models – The Artists

Unlike GANs, Diffusion Models (like DALL·E or Stable Diffusion) work by gradually refining noise into an
image.
They’re slower but produce more detailed and creative visuals.

Use Cases: Concept art, design mockups, high-resolution generation.

Key Trait: Great for fine detail and style control.

Analogy: Like a sculptor slowly carving a statue from a block of marble — step by step, the image
becomes clear.

4.3 Variational Autoencoders (VAEs) – The Compressors

VAEs learn to compress and reconstruct data.

They’re used for representation learning, anomaly detection, or creating new variants of existing data.

Key Trait: They understand the underlying structure of images and can recreate them with small changes.

5. LLM Benchmarks and Evaluations

Benchmarks tell you how well models perform on standard tests. Think of them as exams for AI.

Common Benchmarks

 MMLU: Tests general reasoning and world knowledge across 50+ subjects.

 GSM8K: Measures logical and mathematical reasoning.

 BIG-Bench: Evaluates creativity, analytical thinking, and open-domain tasks.

 HELM: A holistic benchmark checking accuracy, fairness, and safety.

You can explore these results live on [Link] — a public leaderboard comparing popular models
(GPT-4o, Claude 3.5, Gemini 2.0, etc.) on accuracy, latency, and reasoning.

6. Case Study Preview – Building with Streamlit and Gradio

In the next class, you’ll create a simple web app using Streamlit and Gradio to compare two models.

You’ll:

 Enter prompts and get outputs from both models.

 Measure latency and quality side by side.

Prepared by Vaibhav Yendole 120

  Discuss how interface and feedback loops affect user experience.

These tools make AI experiments accessible, even if you don’t code every day.

7. Key Takeaways

 LLMs vary in structure — some write, some understand, and others combine modalities.

 GANs are fast and controllable; Diffusion Models are detailed and creative.

 Benchmarks like MMLU and GSM8K guide model comparison.

 Tools like Artificial Analysis and LM Arena make evaluation easy and visual.

 The right model depends on your task, cost, and latency constraints.

8. Reflection Questions

1. How does understanding model type help you choose the right one for your product?

2. Why might a smaller, faster model sometimes be better than the most accurate one?

3. How could GANs and LLMs work together in a multimodal app?

4. Which metric — accuracy, latency, or cost — do you think matters most in real products?

Lecture Summary

Overview

The lecture focused on practical usage and theory behind various AI productivity tools, Large Language
Models (LLM), AI product management, and image models. It covered hands-on demonstrations with
tools like Notion AI, Miro, Figma, Canva AI, and Chisel Labs, followed by an extensive theoretical
discussion on AI product management roles, stakeholder management, AI model training (pre-training and
fine-tuning), deployment, and monitoring.

Hands-On with AI Productivity Tools

 Notion AI: Demonstrated creating product roadmaps, task assignments by teams, and generating
SQL queries for project data. The tool is contextualized with GPT-5 and can integrate with Jira and
Kanban for task tracking. Emphasis on manually writing prompts during interviews rather than
copying templates.

 Miro AI: Used for creating sticky notes summarizing user stories, pain points, and performing
sentiment analysis and clustering. Miro’s AI can generate PRD documents, meeting notes,
prototypes, and offers feedback from a “product leader” AI agent to improve work quality.

 Figma & FigJam: Used for collaborative design, brainstorming, mind maps, and flowcharts, where
precise prompting affects the output quality.

Prepared by Vaibhav Yendole 121

  Canva AI: For quick creation of marketing collateral like flyers and social media posts.
Demonstrated customization based on color schemes and branding for consistent marketing
collateral.

 Chisel Labs & Chat PRD: Useful for creating product requirement documents (PRDs), user stories,
acceptance criteria, and ongoing brainstorming and feedback gathering with AI support.

 General Tips:

 Creating accounts for these productivity tools is essential for hands-on practice.

 During interviews or product development, use AI tools to optimize work but avoid over-
reliance on copy-pasting prompts.

 Choose AI tools relevant to specific domains and invest in paid versions if productivity gain
is significant.

Roles and Responsibilities of AI Product Managers (AIPMs)

 Stakeholders: Include traditional teams (design, marketing, engineering) plus ML researchers,

data scientists, prompt engineers, and AI evaluation teams.

 Managing AI teams: Set clear goals for data science and responsible AI teams, focusing on model
accuracy, fairness, ethical AI principles, and compliance.

 Deliverables:

 Define problem statements and product outlines.

 Manage data acquisition plans specifying data sources (structured/unstructured) and

licensing.

 Specify training requirements, evaluation metrics, and compliance measures.

 Monitor model health, performance, and ethical guardrails during deployment.

 Responsibility: Oversee the entire AI product lifecycle from data procurement to model
deployment and continuous monitoring.

Understanding Pre-Training and Fine-Tuning

 Pre-Training:

 Large-scale initial training on vast datasets (generally done by organizations like OpenAI,
Google).

 Produces a base or foundational model trained on general data.

 Fine-Tuning:

 Tailoring the pre-trained model to a specific domain or use case by training on domain-
specific or task-specific data, improving relevance and performance for that use case.

 Includes writing system prompts, filtering quality data, and setting evaluation criteria.

Prepared by Vaibhav Yendole 122

  PMs focus more on fine-tuning rather than pre-training unless working in a model-building
company.

 Difference: Pre-training builds the general understanding, while fine-tuning adapts the model to
specialized tasks without changing the base model fundamentally.

AI Product Lifecycle and Metrics

 Phases: Pre-training (for model creators), fine-tuning (for domain-specific adaptation),

deployment, and post-deployment monitoring.

 Monitoring aspects:

 Performance (latency, model reliability, hallucinations, truthfulness).

 User feedback (engagement, satisfaction).

 Compliance (bias detection, ethical standards).

 Model versioning and updates.

 Data and Model Drift:

 Data drift: Changes or obsolescence in training data causing model degradation.

 Model drift: Performance degradation over time requiring retraining.

 Testing and Evaluation: Define “golden sets” (test cases) for validation, including unit tests and
reference sets. Continually update evaluation criteria and monitor real-world feedback to ensure
accuracy and fairness.

AI Product Roadmaps and Strategy

 Modern AI Roadmap Framework:

i. Capabilities: Defining AI functions (e.g., text generation, agentic abilities).

ii. User Value & Features: Focus on productivity and user experience improvements.

iii. Ecosystem & Integration: Connecting with third-party services (Slack, Jira, Notion).

iv. Governance & Compliance: Responsible AI practices, privacy, and security.

 Example of Gmail roadmap shows incremental AI feature delivery from smart compose to agentic
capabilities.

 Emphasis on overlapping traditional product management roles with AI-specific layers for
integrations and governance.

Case Studies and Real-World Applications

 Tesla:

Prepared by Vaibhav Yendole 123

  Utilizes over-the-air (OTA) deployment to continuously improve its models based on live
fleet data.

 Shadow testing deploys new models selectively to test routes to validate safety before full
rollout.

 Grammarly: Transitioned from rule-based correction tools to generative AI-powered writing

assistance, improving content generation and tone adjustment.

 Mahindra: Building domain-specific AI models (e.g., automotive) with custom data acquisition and
licensing strategies.

AI Interview Preparation and Best Practices

 Interview Strategies:

 Prepare to write your own AI prompts rather than copy-paste.

 Understand the AI tools allowed and be prepared to use them effectively with manual
prompt writing.

 Highlight understanding of PM roles in AI context—data acquisition, team collaboration,

metrics, and AI ethics.

 Tool Selection: Choose AI tools based on domain relevance; explore multiple tools in a trial to find
what fits best.

 Prompt Engineering: Use prompt optimizers and practice crafting concise prompts for efficiency.

 Ethical Use: Comply with organizational policies regarding data confidentiality and AI tool usage.

Additional Insights

 Sampling and labeling is used to assess and ensure data quality before fine-tuning.

 AI features can include automation, agentic capabilities (memory, reasoning, independent actions,
tool usage).

 Continuous learning on AI advancements is essential – follow product launches and conferences

from major AI providers.

 Data governance, licensing, and bias detection are crucial responsibilities of AI product teams.

This lecture offered both practical demonstrations of AI tools for product management tasks and a
comprehensive theoretical understanding of building, managing, and evaluating AI products, particularly
focusing on the role of a product manager in the AI space.

Introduction to Productivity Tools with AI

Prepared by Vaibhav Yendole 124

The lecture starts with a practical approach to using various AI productivity tools such as Notion AI, Miro,
Figma, FigJam, Chisel Labs, and Canva AI. Students are encouraged to create accounts on these
platforms to engage in hands-on exercises that complement theoretical studies on stakeholder
management and AI product strategies.

The session emphasizes the importance of deep-diving into these tools rather than just surface-level
usage, enabling product managers to be more productive and efficient while managing AI product
lifecycles.

Example: Creating and managing a product roadmap using Notion AI for a driver-rider mismatch problem
in an Uber-like scenario.

Notion AI for Product Management

Notion AI integrates GPT-based language models to assist product managers in creating roadmaps, task
tracking, and SQL queries contextual to product data.

 Use Notion AI to create an end-to-end six-week product roadmap addressing specific problems like
supply-demand mismatch in ride-sharing apps.

 Automatically generate task assignments by engineering streams (front-end, back-end, design,

marketing).

 Generate SQL queries to extract specific task data for team leads.

Example:

--- Prompt in Notion AI ---

I am a product manager at Uber facing driver-rider mismatch in Jaipur. Create a roadmap and product
outline for this.

--- Response ---

Six-week detailed roadmap including back-end, front-end, design, marketing tasks with timelines.

The tool can also save conversations and output as pages and supports iterative refinement through chat-
like interaction.

Comparison of Notion AI and ChatGPT

Notion AI is powered by GPT models but offers contextualization for your specific product documents,
unlike generic ChatGPT:

 Notion AI contextualizes its outputs based on your documents, tasks, and workflows.

 Enables you to add meeting recordings and generate trackers or project outlines automatically
from these inputs.

 Allows SQL query generation aimed specifically at your project data.

In contrast, ChatGPT is a general LLM without built-in contextual awareness of your workspace or product
documents unless explicitly provided within the same chat.

Prepared by Vaibhav Yendole 125

Using Miro AI for Collaboration and Product Discovery

Miro AI helps in user story creation, diagramming workflows, and obtaining feedback from AI in
collaborative workspaces.

 Create sticky notes reflecting user and driver pain points with AI assistance.

 Generate workflow diagrams or flowcharts to communicate complex processes visually.

 Use AI sidekick feedback for evaluating user discovery sessions and improving product
documentation.

Example: Creating a driver journey flowchart using a prompt based on a supply-demand mismatch
problem:

Using Miro AI for Collaboration and Product Discovery

Miro AI helps in user story creation, diagramming workflows, and obtaining feedback from AI in
collaborative workspaces.

 Create sticky notes reflecting user and driver pain points with AI assistance.

 Generate workflow diagrams or flowcharts to communicate complex processes visually.

 Use AI sidekick feedback for evaluating user discovery sessions and improving product
documentation.

Example: Creating a driver journey flowchart using a prompt based on a supply-demand mismatch
problem:

Prompt: Create an end-to-end driver journey workflow for Uber in Jaipur addressing trip cancellations.

--- Output ---

Miro AI generates shapes and connectors reflecting steps in driver journey formed as a flowchart.

Additional features include clustering notes by sentiment or keywords and prototyping directly from
generated diagrams.

FigJam for Ideation and Sprint Planning

FigJam facilitates brainstorming, sprint planning, and ideation with AI support.

 Allows creating mind maps and flow charts based on detailed prompts.

 Enables structured team collaboration by organizing brainstorming sessions for different teams
(engineering, design, marketing).

 Generate timelines and task breakdowns for sprint planning.

Example:

Prompt: Brainstorm ideas for Uber ride supply-demand mismatch improvements with separate sections
for engineering, marketing, and design teams.

--- AI-generated output ---

Mind map organized by team-specific ideas and KPIs for a weekly sync meeting.
Prepared by Vaibhav Yendole 126
FigJam requires specific prompts to avoid default brainstorming outputs and maximize tool usefulness.

Canva AI for Marketing Collateral Creation

Canva AI empowers product teams to quickly generate marketing materials like flyers and social media
posts aligned with branding guidelines.

 Users select design types (e.g., flyer, Instagram post) and provide content prompts.

 AI proposes multiple style options editable within Canva's editor.

 Supports consistency by uploading design themes or samples for brand adherence.

Example:

Prompt: Design a driver recruitment flyer with headline 'Drive Smart, Earn More during Peak Hours' for
Jaipur city.

--- AI-generated output ---

Four flyer options with editable text and images, suitable for print or digital marketing.

This significantly reduces time required to create quality marketing content for product launches.

Chisel Labs and Chat PRD for Product Requirement Documents

Chisel Labs offers AI-powered tools specialized in writing and optimizing Product Requirement
Documents (PRDs) integrating with popular formats like Lovable.

 AI can auto-generate PRDs, acceptance criteria, user stories, and deliverables based on prompts.

 Supports enterprise policy compliance checks and predefined company templates.

 Enables advanced prompt optimization for rapid and professional documentation.

Example:

Template selection: Lovable PRD

Prompt: Create user stories and acceptance criteria for Uber driver rider mismatch feature.

--- Output ---

Well-structured PRD with clearly defined user stories, criteria, and milestones.

Helps maintain quality documentation even if the enterprise uses strict formats or policies.

Interview Readiness with AI Productivity Tools

During product manager interviews, reliance on AI productivity tools is limited; candidates should:

 Write own prompts rather than copying templates.

 Demonstrate prompt engineering skills live.

 Understand that tools may or may not be allowed; be prepared for manual work.
Prepared by Vaibhav Yendole 127
  Use prompt optimization tools off-interview to prepare efficient prompts.

This helps in showcasing 100x productivity through better use of AI even without external aids during
interviews.

Stakeholders in AI Product Management

AI Product Management involves traditional and new stakeholders:

 Traditional: Designers, Marketing, Engineering teams.

 New AI-centric: ML Researchers, Data Scientists, Prompt Engineers, AI Evaluation teams.

 DevOps evolves to ML Ops focusing on AI pipeline deployment.

Role Differences: Training vs. Fine-tuning in AI Products

Understanding AI Model development stages is critical for AI PMs:

 Pre-training: Training large foundational models on massive general data (e.g., internet
text/images). This is resource-intensive and done by AI companies like OpenAI.

 Fine-tuning: Adapting/preparing pre-trained models for specific domains or applications with

targeted data (e.g., healthcare, finance).

As an AI PM:

 Most work focuses on fine-tuning, data acquisition for domain-specific data, and overseeing model
evaluation.

 Pre-training involves data procurement, licensing, compliance, and is mainly the data scientists'
domain.

Example: A healthcare app fine-tunes GPT model using medical data, ensuring it provides accurate, safe
responses tailored to health use cases.

Data Acquisition and Governance for AI Models

Critical for AI PMs to ensure data quality and legal compliance:

 Identify structured and unstructured data sources (e.g., internal customer data, public datasets).

 Establish data licensing agreements, compliance with privacy and security policies.

 Hunt for biases, cultural or domain-specific mismatches (e.g., left-hand vs right-hand drive traffic
data).

 Collaborate with legal, compliance teams to enforce responsible AI policies.

Example: Tesla's AI team sources driving data from vehicles globally but must exclude countries with
incompatible driving conventions or adjust accordingly.

Evaluation Metrics and Monitoring for AI Products

Prepared by Vaibhav Yendole 128

Measuring AI product success demands continuous monitoring of specific metrics:

 Usefulness/User Feedback: Ratings, thumbs up/down, user adoption.

 Latency: Time for the first token of response and entire completion time.

 Reliability: Hallucination rate, factual accuracy, safety compliance.

 Model Versioning: Tracking upgrades from GPT-3 to GPT-4, and so forth.

 Post-Launch Monitoring: Detect data drift (outdated training data) and model drift (loss of
accuracy over time).

Example Metrics acronym: SMS — Safety, Model reliability, Speed.

Deploy real-time dashboards to track these for prompt interventions and retraining.

Data Sampling and Quality Control in AI Training

To ensure datasets used for training/fine-tuning are of high quality:

 Start with random sampling of data subsets to estimate overall data quality.

 Apply automated classifiers or scoring models to filter out poor-quality or irrelevant data.

 Set quality thresholds to exclude data below desired standards, mitigating training on noisy or
biased samples.

Data labeling and classification tasks are generally performed by data science teams under PM guidance.

Example: Using an image recognition classifier model to score image datasets from 0-100%, selecting
only images >70% for model fine-tuning.

AI Product Roadmap Components

AI product roadmaps incorporate unique layers beyond traditional product management:

 Capability Roadmap: Defines AI model capabilities to be developed (e.g., smart compose, agent
functionalities).

 User Value & Productivity Features: Defines UX features that deliver direct value (e.g., auto-reply,
action item extraction).

 Ecosystem Integrations: Specifies third-party tools integrations (Jira, Slack, Notion) enabling
richer user experiences.

 Governance & Responsible AI: Includes privacy, compliance, fairness, bias mitigation, and model
trustworthiness.

Example: For Gmail AI roadmap:

 Capability: Smart compose.

 Feature: Quick actions toolbar, email template library.
 Integrations: Calendar, Slack.
 Governance: Two-factor authentication, citation standards.
Prepared by Vaibhav Yendole 129
This multi-dimensional approach reflects AI product complexities.

Difference Between Agentic AI and Non-Agentic AI

Agentic AI systems exhibit advanced autonomous capabilities including:

 Memory: Remembers past interactions and context.

 Reasoning: Can infer and make decisions beyond simple commands.

 Actions: Performs tasks autonomously (e.g., booking flights, ordering food without explicit user
clicks).

 Tool Usage: Can interact with external tools/systems via APIs.

Non-agentic AI systems respond to requests but lack independent action capability or persistent memory.

Example: An AI chat that just redirects user clicks to webpages is non-agentic. An Agentic AI can navigate
a booking website, filling forms and confirming reservations autonomously.

AI Model Update Practices: Shadow Testing and OTA Deployment

AI model updates incorporate strategies to ensure reliability and safety:

 Shadow Testing: Deploying new models to a subset of controlled environments (e.g., specific
routes or users in Tesla cars) without exposing risks to all users, monitoring performance.

 Over-The-Air (OTA) Updates: Continuous training and deployment where model updates are
pushed remotely and autonomously to devices (e.g., Tesla cars) without user intervention.

Shadow testing often uses historical user data or less risky segments to validate model improvements
before full rollout.

Example: Tesla deploys a new lane-following model to test cars on low-risk routes while monitoring for
regressions before mass deployment.

Fine-Tuning and Model Differentiation

Fine-tuning layers AI models for domain-specific tasks without altering base weights of pre-trained
models:

 Fine-tuned models behave as wrappers adapting responses to custom behavior or knowledge.

 Different fine-tuned versions of a base model cater to various industries (healthcare, finance,
entertainment).

 Fine-tuning uses high-quality, domain-specific datasets and instructions (system prompts) stored
persistently.

Example: GPT base model, fine-tuned for medical advice, refuses harmful self-harm suggestions; same
base model fine-tuned for finance focuses on market regulations.

Importantly, user-level prompts differ from fine-tuning — prompts adapt model behavior per session; fine-
tuning adapts model permanently for domain.

Prepared by Vaibhav Yendole 130

Prepared by Vaibhav Yendole 131
 W20L3 Github Copilot for Product Managers - understanding and
contributing to Code (AN)
Pre-class resources: GitHub Copilot for Product Managers — understanding and contributing to
code

1. What is GitHub Copilot?

Picture this: you open a code file and start typing, and an AI “pair programmer” looks at the file, the
project, and your question and says:

“Here’s the code that probably belongs here.”

That assistant is GitHub Copilot — GitHub’s AI coding assistant (originally built with OpenAI models, now
powered by multiple top models). It helps you read, explain, write, and improve code — even if you’re
not a full-time developer.

Copilot works inside your developer tools — VS Code, Visual Studio, JetBrains IDEs, even [Link] —
but in this course we’ll use Visual Studio Code (VS Code) because it’s free and PM-friendly.

Think of Copilot as:

 Your explainer — “What does this function do?”

 Your drafter — “Write a Python script that…”

 Your helper — “Add unit tests / fix this error.”

A quick analogy

If a developer is the chef, Copilot is the sous-chef: it chops, suggests, and preps, but you still decide the
recipe.

2. How Copilot shows up in your workflow

GitHub now talks about Copilot in two main ways.

Mode /
What it does Why it matters for PMs
Feature

A chat panel in VS Code / GitHub where you can ask

Lets you understand
Copilot Chat in natural language: “Explain this file”, “Summarize
codebases and changes
(Ask Copilot) this PR”, “Write acceptance criteria for this function.”
without pinging devs.
It uses your open files as context.

Agent mode Copilot can plan a multi-step coding task, decide You can hand over small, well-
(when which files to touch, make edits, run or suggest described tasks (prototypes,

Prepared by Vaibhav Yendole 132

Mode /
What it does Why it matters for PMs
Feature

available) terminal commands, and iterate to fix errors — logging, refactors) instead of
directly in your IDE. This feature is rolling out and doing them manually.
may depend on your plan/org.

Important nuance: to use Copilot in VS Code you must sign in with a GitHub account that has Copilot
enabled (Free/Pro/Enterprise). A plain GitHub login is not always enough.

3. The three tools in this lesson

Tool What it is Why it matters to you

Visual Studio Free code editor we’ll use in class.

This is your “workspace”.
Code (VS Code) Copilot has first-class support here.

Version control system that tracks every Lets you see who changed what,
Git
change. when — like “track changes” for code.

Online home for your repositories, PRs, Think “Google Drive for code” +
GitHub
issues, and Copilot integration. collaboration + AI.

So, when your engineering team says:

“I pushed to feature/copilot-poc — review the PR.”

…you can open the same repo in VS Code, and ask Copilot:

“Summarize the changes in this file” “Explain the difference between main and this branch” “Write a short
release note for PMs.” This is fully supported because Copilot Chat can explain selected code and files.

4. Setting up your first project

Step 1 — Install VS Code

1. Go to the official VS Code site and download for your OS.

2. Install and open it.

Step 2 — Install and connect GitHub Copilot

1. In VS Code, open the Extensions view.

2. Search “GitHub Copilot” and install it.

3. VS Code will ask you to sign in with GitHub.

Prepared by Vaibhav Yendole 133

 4. Make sure the account you sign in with has a Copilot plan or access (Free / Pro / Enterprise). If you
see “You don’t have access,” that’s a licensing issue, not a VS Code issue.

Step 3 — Create a tiny Python project

1. File → New Folder → name it my-first-copilot-project.

2. Open that folder in VS Code.

3. Make a file: [Link].

4. Type:

5. # say hello

6. print("Hello, world!")

7. Pause — Copilot may suggest something like:

8. print("Welcome to GitHub Copilot!")

9. Press Tab to accept.

You just accepted your first AI completion.

5. Git basics for PMs

Think of Git as a time machine + safety net.

Term What it means Analogy

Version
Saved snapshot of the project at a moment in time. “Report_v1”, “Report_v2”.
(commit)

A safe side-track to try an idea without breaking the Test a new recipe without
Branch
main app. serving it to guests.

Pull Request A request to merge your branch into the main “Here’s my draft, can someone
(PR) branch, with review/comments. review?”

This is exactly how Copilot’s agent mode often ends its work — it proposes changes that you can review or
merge.

6. Hands-on mini challenge: build a tiny “Snake” game

Goal: see Copilot build something interactive from a natural-language prompt.

1. Create a file snake_game.py.

2. Ask Copilot in the editor or in Chat:

Prepared by Vaibhav Yendole 134

“Create a simple Snake game in Python that runs in the terminal, uses arrow keys, has walls, and ends on
collision.”

3. In order to run the game, you need to install Python interpreter. Then run the game:

4. python snake_game.py

This mirrors how GitHub describes Copilot Chat — you start with a prompt, then you refine until it works.

7. Why this matters for PMs

You are not becoming a full-time engineer. You are learning to enter the technical room with confidence.

With Copilot + Git you can:

 Read code and get an English explanation.

 Summarize a PR before standup.

 Prototype small utilities (“convert this CSV”, “call this API”) without blocking an engineer.

 Review branches and understand why a feature is delayed (“tests failing”, “migration pending”).

This makes your conversations with engineering faster and more informed.

8. Reflection questions

1. In your own words, what gap does Copilot close for non-developers on a product team?

2. Which parts of your current product work (release notes, PRD snippets, acceptance criteria) could
you ask Copilot Chat to draft for you?

3. How could branches → PR → merge map to product iteration → stakeholder review → release in
your project?

9. Key takeaways

 Copilot is available in more than just VS Code, but we’ll use VS Code for simplicity.

 Use Copilot Chat to understand code; use agent mode (if you have it) to do small multi-step tasks.

 You must sign in with a GitHub account that has Copilot enabled. If you can’t, it’s a
licensing/access issue.

 Small prototypes (like the Snake game) are a safe way for PMs to practice without touching
production code.

 Knowing Git terms (commit, branch, PR) lets you join engineering conversations without slowing
anyone down.

GitHub Copilot for Product Managers — Read, Navigate, and Prototype Code

Objective By the end, you’ll feel comfortable using GitHub Copilot to:
Prepared by Vaibhav Yendole 135
  Understand what a piece of code is doing in simple language.

 Use two ways of working with Copilot: Ask Mode and Agent Mode.

 Know the basics of Git: repository, branch, commit, pull request.

 Suggest or prototype small changes without blocking on an engineer.

Why this matters for PMs You don’t need to be a full-time coder. But when you can read code, ask
sharper questions, and try tiny prototypes, you:

 Communicate better with engineers.

 Write clearer tickets and PRDs.

 Validate ideas faster.

What is GitHub Copilot?

Copilot is an AI assistant inside VS Code. You can ask it to explain code, generate small snippets, and
suggest improvements.

Simple mental model If a developer is the architect, Copilot is the assistant who drafts, explains, and
suggests.

Two ways to use Copilot

Ask Mode

 You ask questions about code or request small snippets.

 Great for: “What does this loop do?”, “Summarize this file,” “Suggest test cases.”

Agent Mode

 You delegate small code tasks and the agent can make and test changes.

 Great for: “Add basic logging,” “Handle invalid input,” “Create a simple README.”

Tip Start with Ask Mode when you’re trying to understand. Switch to Agent Mode when you’re ready to
make a safe, small change.

Why Copilot is useful for PMs

1. Speeds up understanding of unfamiliar code.

2. Lets you try tiny experiments without blocking a developer.

3. Improves backlog items with clearer acceptance criteria and edge cases.

4. Encourages low-risk prototyping.

Getting set up

Step 1 — Install tools

1. Install VS Code.

Prepared by Vaibhav Yendole 136

 2. In VS Code, open the Extensions panel and install GitHub Copilot.

3. Sign in with your GitHub account.

Think of VS Code as your workbench. GitHub is your filing cabinet. Copilot is your assistant sitting next to
you.

Step 2 — Create a practice repo

 On GitHub: create a New Repository named pm-copilot-demo and initialize it with a README.

Step 3 — Talk to Copilot

 Open the repo in VS Code.

 Open the Command Palette: press Cmd+Shift+P on Mac or Ctrl+Shift+P on Windows/Linux, then
choose Copilot: Ask.

 Try this prompt: “Write a function in Python that prints: Hello, Product Managers!”

Git basics for PMs

Let’s keep it simple:

Term Plain meaning PM analogy

Your whole project folder tracked on

Repository A shared project folder in Drive.
GitHub.

Branch A safe workspace copy for changes. A duplicate file to try ideas.

Save with a short note on what

Commit A snapshot with a message.
changed.

Pull Request Ask teammates to review and merge your Send your doc for approval and
(PR) changes. comments.

Example flow you’ll use today

1. Create a branch named feature-snake-game.

2. Make changes with Copilot.

3. Commit with a clear message like “Add score tracking.”

4. Open a PR describing what changed and why.

Mini guided demo: Simple Snake game

You don’t need to love Python—this is about the workflow.

What we’ll do

Prepared by Vaibhav Yendole 137

  Create a file named snake_game.py.

 Ask Copilot to draft a simple snake game.

 Use Ask Mode to understand the code.

 Use Agent Mode to make a small upgrade.

Prompts to try

1. Create the basic game:

2. Create a simple snake game in Python with arrow key movement, fixed boundaries, and scoring.

3. Understand the code:

 “Explain how the movement loop works.”

 “Where does scoring increase? Show me.”

4. Upgrade it:

 “Increase snake speed slightly after each fruit.”

 “Add a short summary at the top of the file explaining how to run the game.”

What to look for

 Movement loop

 Collision detection

 Score updates

Run it

 If you have Python set up, run: python snake_game.py

 If not, the goal is still to see how you’d ask, read, and iterate.

If something breaks

 Ask: “Why is this error happening? Suggest a fix.”

 Then: “Apply the smallest change to fix it.”

Pitfalls and best practices

Common pitfall Why it happens What to do instead

Treating Copilot like Accepting code without

Ask it to explain before you accept.
magic understanding

Commit often with clear messages. Keep

Skipping Git hygiene Overwrites and confusion in PRs
PRs small.

Prepared by Vaibhav Yendole 138

Common pitfall Why it happens What to do instead

Replacing judgment with Use AI for speed, apply PM judgment for

Over-automating
automation context.

Be specific about goal, constraints, and

Vague prompts Confusing outputs
examples.

Safety reminder Copilot suggests patterns learned from public code. Always validate logic and data
handling before merging.

Quick self-check

 Can you explain Ask Mode and Agent Mode in one sentence each?

 If someone creates a new branch, do you know why?

 If Copilot suggests code, how will you verify it before approving?

Key takeaways

 Copilot helps PMs understand and iterate, not just observe.

 Light Git skills—branch, commit, PR—enable confident collaboration.

 Use Ask Mode to understand; use Agent Mode for small, safe changes.

 Prototype early to validate ideas faster.

Lecture Summary

Overview

The lecture focused on introducing GitHub Copilot to product managers, emphasizing how it can bridge
the gap between non-coding PMs and developers by helping them understand, contribute, and interact
with code using natural language. The session included theoretical explanations, practical
demonstrations using VS Code, and discussed foundational Git concepts like repositories, branching,
committing, and pull requests. There was also discussion on how GitHub Copilot can assist in writing
documentation, test cases, and simplifying collaboration in technology projects.

Understanding GitHub Copilot and its Role for Product Managers

 GitHub Copilot acts as a "magical wizard" or assistant that helps product managers understand
codebases without deep coding knowledge.

 It enables PMs to execute tasks like writing test cases, product requirement documents (PRDs),
and even running or modifying code without manually coding.

 Copilot integrates with editors like VS Code and IntelliJ, making code writing and reading accessible
and collaborative.

Prepared by Vaibhav Yendole 139

  It supports three modes:

 Ask Mode: Provides explanations or code-related answers without modifying code.

 Edit Mode: Makes changes to specific files as requested.

 Agent Mode: Performs actions including running servers, making commits, and raising pull
requests with minimal user intervention.

Practical Demonstration with VS Code and GitHub Copilot

 The instructor demonstrated cloning a repository and interacting with GitHub Copilot within VS
Code to understand a memory card game codebase.

 Asked Copilot to generate test cases, run the app locally, create PRDs in markdown format, and
make code improvements.

 Showed how Copilot can automate repetitive tasks like creating new games (e.g., Flappy Bird),
adjusting game sensitivity, and committing changes.

 Highlighted the advantage of using markdown files for all documentation to make it developer-
friendly and reduce misunderstandings.

 Emphasized that GitHub Copilot helps PMs become "builders" who can contribute directly to the
codebase, thereby improving collaboration with engineering teams.

Key Git Concepts for PMs: Repository, Cloning, Branching, Committing, Pull Requests

 Repository: Central code storage where all product-related code files are maintained; a product
may have multiple repositories.

 Cloning: Copying a repository from GitHub to a local machine to work on the code.

 Branching: Creating independent branches to work on features/bug fixes without affecting the
main codebase.

 Committing: Saving changes to one’s branch with descriptive messages about the changes.

 Pull Requests (PRs): Proposals to merge changes from a branch back into the main branch; must
be reviewed and approved by a gatekeeper (senior engineer or manager).

 Explained common practical workflows such as multiple developers working on separate branches
to avoid merge conflicts.

 Highlighted the importance of small atomic commits for easier code review and conflict resolution.

Usage of Copilot for Enhancing Development and PM Workflows

 GitHub Copilot can generate functional code snippets from natural language prompts, speeding up
development.

 It helps in drafting detailed PRDs, test cases, and code documentation in a maintainable format
(markdown).

 Encouraged iterative fine-tuning of generated code by querying Copilot about specific changes
needed.

Prepared by Vaibhav Yendole 140

  Demonstrated how Copilot automates version control tasks (creating branches, commits, PRs)
significantly reducing manual effort.

 Proposed hands-on practices: creating small games, building test cases, updating code
documentation, and deploying on GitHub as portfolio work.

Licensing and Tool Integration

 Availability depends on GitHub plans; free version supports basic models like GPT-5 Mini, premium
plans unlock advanced capabilities.

 Copilot can be integrated with multiple IDEs such as VS Code, IntelliJ, or others depending on the
tech stack.

 Extensions from tools like Notion or Figma can integrate with VS Code, streamlining workflows
further.

Git Best Practices and Organizational Usage

 Discussed how large organizations break down codebases into multiple smaller repositories
managed by dedicated teams.

 Outlined the typical workflow for feature development from cloning repo to merging PRs.

 Stress on collaboration discipline: avoiding large bulky commits, using descriptive commit
messages, and following review protocols.

 Shared real-world anecdotes about code quality, production crashes, and the responsibility of
gatekeepers in maintaining stable code.

 Emphasized the importance of transparency and traceability using git commits for code lineage.

Learning and Upskilling with GitHub Copilot

 Recommended using Copilot as a learning assistant to grasp unfamiliar programming languages or

concepts.

 Suggested setting exercises such as creating simple applications and progressively increasing
complexity with Copilot’s help.

 Highlighted YouTube demo and shared exercises to promote hands-on exploration.

 Encouraged product managers to use markdown-based PRDs and documentation for seamless
alignment with developers.

Next Steps and Resources

 Students are encouraged to install VS Code, GitHub Copilot, clone the provided repositories, and
practice coding and documentation tasks using given prompts.

 Future sessions will focus more on hands-on usage, including working on large repos and
contributing code directly.

 Shared instructional documents with step-by-step setup, exercises, and prompts.

 Plans to extend learning to large-scale, LLM-driven applications.

 Encouraged patience and incremental learning to harness AI-assisted development fully.

Prepared by Vaibhav Yendole 141
This summary captures the detailed foundational knowledge, practical demonstrations, and strategic
advice presented in the lecture to equip product managers with essential GitHub Copilot skills.

Introduction to GitHub Copilot

GitHub Copilot is an AI-powered code completion and assistance tool that integrates with code editors
like VS Code. It serves as a virtual coding assistant, helping users to understand, write, and improve code
using natural language prompts. It is particularly useful for product managers and developers to bridge the
gap between coding and product management by generating code snippets, tests, documentation, and
even running applications.

The instructor emphasized the importance of understanding GitHub Copilot to reduce friction between
product managers and engineers and to enhance collaboration.

Example

In VS Code, after installing GitHub Copilot, a friendly frog icon appears. Clicking on it or invoking it allows
you to ask:

// Prompt in the VS Code GitHub Copilot window

"What is this code about?"

GitHub Copilot replies with a short summary of the existing code.

Modes of Interaction in GitHub Copilot

There are three primary modes of interaction with GitHub Copilot:

 Ask Mode: You can ask questions or request explanations about the code but Copilot will not make
changes directly.

 Edit Mode: Copilot suggests edits or additions to existing code, but requires user approval before
applying.

 Agent Mode: Copilot can act autonomously to make changes, run commands, start servers,
commit code, and raise pull requests, with minimal user intervention.

Example

In Agent Mode, you can ask Copilot:

"Run this app for me"

Copilot will start the local server, open the application in a browser, and perform operations automatically.

Repository and Version Control Concepts

Repository: The central place where the entire code base of a project is stored, usually on platforms like
GitHub.

Prepared by Vaibhav Yendole 142

Clone: Creating a local copy of a remote repository on your system to work with it.

Branch: A separate line of development in a repository to isolate changes.

Commit: Saving changes to your branch with a descriptive message.

Push: Uploading local commits to the remote repository.

Pull Request (PR): A request to merge changes from one branch into another, typically the main branch.

Merge: The acceptance and integration of changes from a pull request into the main branch.

Gatekeeper: The person responsible for reviewing and approving pull requests to maintain code quality.

These concepts enable multiple developers to work concurrently without overwriting each other's work
and help maintain code history and traceability.

Example

Typical Git workflow:

# Clone repository
$ git clone [Link]

# Create a new branch

$ git checkout -b feature/new-game-mode

# Make changes and commit

$ git add .
$ git commit -m "Add new game mode feature"

# Push branch
$ git push origin feature/new-game-mode

# Create a Pull Request on GitHub

# After review, the gatekeeper merges the PR into main
Using GitHub Copilot for Product Management Tasks

GitHub Copilot can assist product managers (PMs) in tasks such as:

 Understanding existing codebases

 Creating Product Requirement Documents (PRDs) in markdown

 Writing test cases

 Generating documentation

 Running and debugging applications

PMs can interact with codebases without deep coding expertise by leveraging natural language queries.

Example

Ask GitHub Copilot to generate test cases:

"Write five test cases for the memory card game to validate functionality"

Prepared by Vaibhav Yendole 143

Or create a PRD markdown file:

# Memory Card Matching Game PRD

## Target Audience
Players who enjoy casual puzzle games

## Functional Requirements
- Responsive 4x4 and 6x6 grids
- Timer and move counter
...
GitHub Copilot can generate such content based on the natural language inputs.

Building and Extending Applications Using GitHub Copilot

Product managers and developers can use GitHub Copilot to write new features, create games, or
applications by providing natural language prompts, which Copilot translates into code.

For example, creating new games such as Snake and Ladder, Flappy Bird, or Mario by describing
functionality and UI.

If errors occur, you can ask Copilot to explain the errors and suggest fixes.

Example

Natural language prompt:

"Create a simple Flappy Bird game in HTML and JavaScript"

GitHub Copilot generates the necessary code, and you can further refine it by asking:

"Explain the part of the code that controls the bird's flap strength"

"Modify the flap strength to reduce sensitivity"

This iterative process helps build and customize applications efficiently.

Integrating GitHub Copilot with IDEs

GitHub Copilot integrates primarily with VS Code but also supports other IDEs like IntelliJ, JetBrains suite,
and more.

VS Code is widely used for various languages and is favored in Azure and many enterprises.

Depending on your tech stack, you can choose the IDE that best supports your development needs and
still use GitHub Copilot.

Example

Installing GitHub Copilot on VS Code:

1. Open VS Code.

2. Go to the Extensions Marketplace.

Prepared by Vaibhav Yendole 144

 3. Search for "GitHub Copilot".

4. Install and sign in with your GitHub account.

Once installed, you start seeing code completions and can use the Copilot side panel to type prompts.

Branching Strategies and Best Practices

Best practices in branching that were shared include:

 Creating branches for each atomic change or feature.

 Keeping branches small and focused to simplify reviews and merges.

 Developers often create multiple branches if working on concurrent features.

 Branches enable parallel development and reduce merge conflicts.

PMs and developers should collaborate by raising pull requests with clear commit messages to describe
the purpose of changes.

Example

Branch creation for a new feature:

$ git checkout -b feature/add-easy-medium-hard-levels

After completing the feature:

$ git add .

$ git commit -m "Add difficulty levels to the memory game: easy, medium, hard"

$ git push origin feature/add-easy-medium-hard-levels

Then raise a pull request for review.

Learning and Upskilling Using GitHub Copilot

GitHub Copilot can be a personal learning assistant by:

 Explaining unfamiliar code.

 Generating exercise problems at varying difficulty levels.

 Providing step-by-step guidance to build projects.

For example, learning programming languages like Java or Flutter by asking Copilot for exercises and
explanations.

Example

A prompt to learn Java:

"I want to learn Java. Give me four programming exercises from beginner to intermediate level"

GitHub Copilot then provides problem statements and code snippets.

You can build confidence by practicing with these generated exercises.

Prepared by Vaibhav Yendole 145

Documentation and PRD Markdown Files for Collaboration

Creating documentation and PRDs in markdown format is a best practice.

Markdown files can be directly consumed by developers and stored within the code repository alongside
the code.

This approach enhances transparency and traceability of feature requirements.

GitHub Copilot can assist in generating detailed markdown documents for product requirements, test
cases, and usage instructions.

Example

Sample PRD partial markdown file generated by GitHub Copilot:

# Feature: Memory Card Game

## Functional Requirements

- 4x4 and 6x6 grid sizes

- Timer and move counter displays

## Performance Metrics

- Load time under 2 seconds

- Responsive on desktop and mobile

## Testing Requirements

- Verify grid size switching functionality

- Validate timer accuracy

Storing these files in the repository ensures everyone stays in sync.

Prepared by Vaibhav Yendole 146