📊 Data Wrangling, Analysis&Visualization(Very
Simple&DetailedNotes)
1⃣What is Data Wrangling?
Data Wranglingmeans converting intoa
form so that we can analyze it easily.
👉 Inreal life, datais rarelyperfect. It maycontain: - Missing values - Errors - Duplicates -
Wrong formats - Unnecessarycolumns
🔁 StepsinDataWrangling
1. Data Cleaning– Fix errorsandmissing values
2. Data Transformation – Change data format or values
3. Data Merging– Combine datasets
4. Data Reshaping – Change data structure
5. Data Aggregation&Grouping – Summarize data
2⃣DataCleaning
Data Cleaning is the processof removingorcorrecting incorrect data.
🔹 CommonDataCleaningTasks
[Link]
Missingvalues are oftenshownas: - NaN - NULL - Empty cells
Ways to handle missingdata: - Remove rows or columns- Fill withmean, median, or
mode - Fill with afixed value (like 0 or “Unknown” )
Example: If age is missing → replace withaverage age
[Link]
Duplicate rows cangive wrong analysis.
Example: Same student recordentered twice
👉 Solution: Keep only one copy
[Link]
Sometimes numbersare stored as text.
Example: “25” (string)insteadof 25 (integer)
👉 Convert to correct data type
[Link]
Outliers are extremelyhigh or lowvalues.
Example: Average salary= ₹ 30,000 but one entry= ₹ 10,00,000
👉 Options: - Remove outlier- Cap the value - Investigate the cause
3⃣Data Transformation
Data Transformationmeans changing dataintoa suitableformat for analysis.
🔹 CommonTransformations
[Link]
Used whenvalueshave different ranges.
Example: - Age: 18– 60 - Salary: 10,000– 1,00,000
👉 Scale valuesto same range
[Link]
Convert text categories intonumbers.
Example: - Gender→ Male = 1, Female = 0 - City→ Delhi, Mumbai, Chennai
[Link]
Newcolumns canbe derivedfromexisting ones.
Example: - Full Name = First Name + Last Name - Total Marks= Subject1+ Subject2
[Link]&TimeTransformation
Extract useful informationfrom date.
Example: - Date → Year, Month, Day
4⃣Data Merging
Data Merging means combining two ormoredatasets.
🔹 Why MergeData?
Data isoftenstored inmultiple files
We needcomplete information
TypesofMerging(Joins)
[Link]
Keeps onlycommonrecords.
Example: Studentstable ∩ Markstable
[Link]
Keeps all recordsfromlefttable.
[Link]
Keeps all recordsfromright table.
[Link]
Keeps all recordsfrombothtables.
5⃣Data Reshaping
Data Reshapingmeans changing thestructureof data.
🔹 Why ReshapeData?
Forbetteranalysis
Forvisualization
Formachine learning
CommonReshapingOperations
[Link] toLongFormat
WideFormat: | Name | Math| Science | |——|——|———| | A | 80 | 85 |
Long Format: | Name | Subject | Marks |
[Link] Format
Reverse of above
6⃣DataAggregation&Group Operations
Aggregationmeans summarizing data.
🔹 CommonAggregationFunctions
Sum
Mean(Average)
Count
Minimum
Maximum
GroupByOperation
Used to performaggregationgroup-wise.
Example: Average marks of eachclass
Class| Avg Marks |
Multiple Aggregations
Example: - Total sales per region- Average salaryperdepartment
7⃣Summarizing&Describing aDataset
This step helps us understand data quickly.
🔹 Numerical Summary
Mean
Median
Mode
Standard Deviation
Variance
🔹 DescriptiveStatistics
Measure Meaning
Mean Average value
Median Middle value
Mode Most frequent value
StdDev Spreadof data
🔹 DatasetOverview
Number of rows
Number of columns
Columnnames
Data types
8⃣Creating Customized Visualizations
Visualizationhelpsus understand patterns andtrends visually.
CommonTypesofCharts
[Link]
Used forcomparing categories.
Example: Sales by product
[Link]
Used to showtrendsover time.
Example: Monthlysales
[Link]
Used to showproportions.
Example: Market share
[Link]
Used to showdistribution.
Example: Age distribution
CustomizationOptions
Title
Labels
Colors
Legends
Grid
👉 Customization makescharts clear and attractive
9⃣Hypothesis Testing
Hypothesis Testing is astatistical methodto make decisions using data.
🔹 WhatisaHypothesis?
A hypothesis is anassumptionabout data.
TypesofHypotheses
[Link] Hypothesis (H₀ )
No effect
No difference
Example: Average marks of Class A = Class B
[Link] (H₁ )
There isaneffect
There isa difference
Example: Average marks of Class A ≠ Class B
🔹 StepsinHypothesis Testing
6. Define hypotheses
7. Choose significance level (α = 0.05)
8. Select test
9. Calculate test statistic&p-value
10. Make decision
🔹 CommonHypothesisTests
Test UsedFor
t-test Compare means
z-test Large sample mean
Chi-square test Categorical data
ANOVA Compare more than2means
🔹 p-value
If p≤ 0.05→ Reject H₀
If p>0.05 → AcceptH₀
🔟 Final Summary
✔ Data Wrangling preparesrawdata ✔ Cleaning removes errors ✔ Transformation
makes datausable ✔ Mergingcombines datasets ✔ Reshapingchangesstructure ✔
Aggregationsummarizes data ✔ Visualizationshows insights ✔ Hypothesis testing
supportsdecisions