VS

AI Python

Who is the Real

Data Analyst?
BTech CSE (Data Science) · Semester 2 · Week 14
Let’s give a prompt - Use AI tools such as chatgpt/[Link]/others

"Analyze this dataset into dashboards"

8-Step Data Wrangling with AI

01 02 03 04

Invalid / Outlier
Data Understanding Missing Value Analysis Data Type Correction
Detection

05 06 07 08

Duplicate Detection Cleaning Pipeline Validation Ready for Analysis

Classroom Activity

Group A (Observe Group B) Group B (Observe Group A)

"Analyse this dataset and give insights" Use the 8 structured prompts step-by-step

• Likely creates charts immediately • Identifies all 5 data issues

• Does not clean data first VS • Cleans before analysing

• May include invalid Rating = 6 • Handles NaN, abc, Rating = 6

• Results look polished but wrong • Generates verified insights

• Fast — but unreliable • Slower — but trustworthy

Compare outputs. Which group do YOU trust?

Steps 1–4: Understand → Clean → Fix → Detect

01 Data Understanding

Prompt:
You are a data analyst. Analyze the structure of this shared dataset.
Give:
1. Column names and data types
2. Summary of each column
3. Identify possible data issues (missing values, wrong types, anomalies)
Do not create charts. Focus only on understanding the dataset.

✅ Forces AI to slow down, mimics [Link]() + [Link]()

⚠ May misclassify types or miss subtle anomalies
Steps 1–4: Understand → Clean → Fix → Detect

02 Missing Value Analysis

Prompt:
Identify missing values in the dataset.
For each column:
- Count missing values
- Percentage of missing values
- Suggest how to handle them (drop, fill, or ignore) with reasoning

✅ Adds decision-making layer and data strategy thinking

⚠ Suggestions may be generic (e.g. "fill with mean" blindly)
Steps 1–4: Understand → Clean → Fix → Detect

03 Data Type Correction

Prompt:
Check for incorrect data types or inconsistent values.
Examples:
- Numbers stored as text
- Mixed data types
- Invalid entries
List issues and suggest how to fix them.
Also provide Python (pandas) code to correct them.

✅ Bridges AI + coding; helps students see real fix workflows

⚠ May miss edge cases like "abc" in numeric column
Steps 1–4: Understand → Clean → Fix → Detect

04 Invalid / Outlier Detection

Prompt:
Detect invalid values and outliers in the dataset.
Examples:
- Ratings beyond expected range
- Extremely high/low values
Explain:
1. Why they are problematic
2. How to handle them
3. Provide Python code for filtering/removal
✅ Introduces data validation; moves beyond surface EDA
⚠ AI may assume ranges incorrectly without business context
Steps 5–8: Deduplicate → Clean → Validate → Analyse

05 Duplicate Detection

Prompt:
Check for duplicate records in the dataset.
Explain:
- How duplicates affect analysis
- How to detect them
- Provide Python code to remove duplicates

✅ Teaches data integrity; duplicates inflate averages silently

📌 pandas [Link]() + df.drop_duplicates()
Steps 5–8: Deduplicate → Clean → Validate → Analyse

06 Cleaning Pipeline

Prompt:
Based on all previous steps, create a complete data cleaning pipeline.
Include:
- Handling missing values
- Fixing data types
- Removing invalid values
- Removing duplicates
Provide clean, step-by-step pandas code.

✅ Combines everything into real industry-grade workflow

📌 Most impactful prompt — generates runnable code
Steps 5–8: Deduplicate → Clean → Validate → Analyse

07 Validation

Prompt:
After cleaning the dataset, how do we validate that the data is now reliable?
List checks and provide Python code to verify:
- No missing values
- Correct data types
- Valid ranges

✅ Where most beginners fail — teaches quality assurance

📌 Check dtypes, null counts, value ranges
Steps 5–8: Deduplicate → Clean → Validate → Analyse

08 Ready for Analysis

Prompt:
Now that the dataset is cleaned, suggest:
1. 5 meaningful business questions
2. Best visualizations for each
3. Python code for plotting

✅ Moves from wrangling → analytics. AI as thinking partner

📌 Only trustworthy once Steps 1–7 are complete
Key Takeaways
AI is fast but not smart about data quality
It will plot dirty data without warning. Never trust AI output on unseen, uncleaned data.

Python gives you control and transparency

pandas, [Link](), [Link]() — these tools show you what's actually in the data.

Prompting AI is a learnable skill

Break every task into structured steps. Vague prompts → garbage output.

Domain knowledge is irreplaceable

AI doesn't know your business rules. You must define what Rating = 6 means in
context.
Validate before you visualise
Clean → Validate → Analyse. In that order. Always. No exceptions.
 Using AI is a skill.
Prompting AI like a Data Scientist
is a powerful skill to master.

BTech CSE (DS) · Semester 2 · Week 14 | AI vs Python: Who is the Real Data Analyst?
How much data is called as big data?
How much data is called as big data?
This is a conceptual question — no dataset needed here.
What Counts as "Big Data"?
• There is no single number that defines big data.
• It is not a file size threshold.
• The industry definition is based on characteristics, not gigabytes.

The 3 Vs Framework (Original Definition — Doug Laney, 2001)

V Meaning Example
Petabytes of transaction
Volume How much data
logs
Velocity How fast it arrives Real-time stock tick data
Text + images + sensor data
Variety How many formats
together
How much data is called as big data?

Data qualifies as "big" when it becomes too large, too fast, or too complex for
traditional tools like Excel or a single SQL database to handle effectively.

Extended Vs (modern additions)

• Veracity — How trustworthy is the data? Noisy, inconsistent data at scale is
harder to clean than your 10-row [Link].
• Value — Does processing it produce actionable insight? Raw volume
without value is just storage cost.

Some frameworks extend this to 7 or even 10 Vs, but Volume, Velocity, and
Variety remain the core three.
How much data is called as big data?

Rough Size Benchmarks (practical, not official)

Scale Size Tools typically used

Small data KB to MB Excel, pandas
pandas, SQL
Medium data MB to a few GB
databases
SQL + optimised
Large data Tens of GB
queries
Spark, Hadoop, cloud
Big data 100 GB → Petabytes+
platforms
How much data is called as big data?

The honest answer:

when your laptop can no longer load, process, or

analyse it within reasonable time and memory, you
are dealing with big data in a practical sense.
How much data is called as big data?

A Useful Rule of Thumb

• If your data fits in RAM on a single machine → not big data.

• If it requires distributed computing across multiple machines to

process → big data.

• A typical laptop has 8–16 GB RAM.

• So in practice, once your dataset exceeds that, you start needing

big data tools.
How much data is called as big data?

Real-World Context
Company Data generated
Facebook ~4 petabytes per day
Google Search ~8.5 billion queries per day
NYSE (stock exchange) ~1 TB of trade data per day
Your [Link] 10 rows, ~500 bytes
How much data is called as big data?

Key Takeaway for Data Science Enthusiast:

• Big data is less about a specific size and more about whether your
current tools can handle it.
• The moment pandas crashes your kernel, SQL queries time out,
• or a single machine cannot store the data — you have crossed
into big data territory and need distributed systems like
• Apache Spark, Hadoop, or cloud platforms (AWS, GCP, Azure).

That is the practical line. The 3 Vs framework is the theoretical one.

Thank you 