MODULE 3
1. Working with Real-Life Data Using Statistical Software
1.1 Introduction
Statistical software like R and R Commander are powerful tools used for
analysing real-world data efficiently.
They simplify the process of:
• Importing and cleaning data
• Running descriptive and inferential statistical analyses
• Creating data visualizations
1.2 Introduction to R
What is R?
• R is a programming language and environment developed
specifically for statistical computing, data analysis, and
visualization.
It was created by Ross Ihaka and Robert Gentleman at the University
of Auckland, New Zealand, and is now maintained by the R Core Team
and a large open-source community.
• R is distributed under the GNU General Public License (GPL), which
means it is free to download, modify, and share.
R is one of the most popular tools in statistics, data science, and research
because it:
1. Handles large and complex datasets efficiently.
2. Offers thousands of built-in statistical functions.
3. Provides advanced visualization tools for clear, customizable
graphs.
4. Integrates well with Excel, CSV, SQL, Python, and web data
sources.
1.3 Introduction to R Commander
R Commander is a graphical user interface (GUI) for R — suitable for
beginners who prefer menus instead of code.
You can:
• Enter and import data
• Perform summary statistics and graphs
• Conduct t-tests, regression, correlation, etc.
2. Defining and Managing Variables
2.1 Types of Variables
• Numeric Variables: Represent quantitative data (e.g., height, weight,
age).
• String Variables: Represent qualitative or categorical data (e.g.,
gender, city, department).
2.2 Assigning Names and Labels
• Each variable in R must have a unique name (no spaces or special
symbols).
Rules for Naming Variables in R
1. Variable Name Must Begin with a Letter
2. Variable Names Can Contain Letters, Numbers, Dots (.) or
Underscores (_). (No special symbols like @,!..)
3. R is Case-Sensitive
4. No Spaces in Variable Names
5. Variable Name Cannot Be a Reserved Word
Labels or factor levels can be assigned to categorical data to make
interpretation easier.
gender <- factor(c("M", "F", "M"), levels=c("M", "F"), labels=c("Male",
"Female"))
2.3 Entering Data
Data can be:
• Entered manually using vectors or data frames:
data <- [Link](Name=c("A", "B", "C"), Marks=c(80, 75, 90))
Imported from files (CSV, Excel, SPSS) using:
[Link]("[Link]")
3. Summary Statistics
3.1 Frequencies
Frequency shows how often each value occurs.
table(data$Gender)
3.2 Descriptive Statistics
Provide simple summaries of data.
Mean (Average):
mean(data$Marks)
Median: Middle value
median(data$Marks)
Mode: Most frequent value (can use package modeest)
Standard Deviation:
sd(data$Marks)
3.3 Cross Tabulation (Crosstab)
Used to summarize categorical data.
table(data$Gender, data$Department)
4. Graphical Representations
4.1 Graphs and Charts
R provides multiple functions for visual analysis:
Histogram: For frequency distribution
hist(data$Marks)
Bar Chart: For categorical data
barplot(table(data$Gender))
Scatter Diagram: Shows relationship between two numeric variables.
plot(data$Height, data$Weight)
Pie Chart: Displays proportions
pie(table(data$Department))
5. Bivariate Analysis
5.1 Correlation
Measures strength and direction of relationship between two continuous
variables.
• Karl Pearson’s Correlation Coefficient:
cor(data$X, data$Y)
Values range from -1 to +1.
5.2 Linear Regression
Shows how one variable (dependent) changes with another (independent).
• Simple Linear Regression:
model <- lm(Y ~ X, data=data)
summary(model)
• Equation:
𝑌 = 𝑎 + 𝑏𝑋
6. Inferential Statistics (Statistical Tests)
6.1 Testing a Mean
Used when population standard deviation is known.
• Z-test:
[Link](x, mu=population_mean, sigma=known_sd)
6.2 One-Sample t-test
Tests if sample mean differs from population mean.
[Link](x, mu=50)
6.3 Two-Sample Z-Test for Means
Used when comparing means from two populations with known variances.
6.4 Two-Sample t-test
Compares means of two independent samples.
[Link](x1, x2, [Link]=TRUE)
6.5 Paired t-test
Used for dependent samples (before–after data).
[Link](before, after, paired=TRUE)
6.6 Chi-Square Test for Independence
Used for categorical data to check if two attributes are independent.
[Link](table(data$Gender, data$Preference))
7. Ethical Theories and Principles in Data Science
7.1 Importance of Ethics in Data Science
In today’s data-driven world, vast amounts of personal, social, and
financial information are collected, analyzed, and used for decision-
making.
Ethics in data science ensures that this process is responsible, fair, and
transparent.
7.2 Core Ethical Principles
1. Integrity
• Present results honestly without altering or selecting data to fit a
desired outcome.
• Avoid “data dredging” or “p-hacking.”
• Always mention limitations or assumptions of analysis.
Example:
If an analysis shows that a marketing campaign was ineffective, the analyst
must report it truthfully instead of adjusting data to show success.
2. Transparency
• Clearly document:
o Data sources
o Methods of collection
o Analytical procedures
o Model assumptions
3. Privacy
• Protect personal or sensitive data from unauthorized access or
exposure.
• Use anonymization or data masking where possible.
• Store and transmit data securely.
4. Accountability
• Data analysts, scientists, and organizations must accept
responsibility for the ethical and social consequences of their work.
• Regular audits and ethical reviews should be conducted.
7.3 Ethical Frameworks in Data Analysis
Ethical frameworks provide structured ways to evaluate and guide
decisions in data analysis.
1. Fairness
• Ensure models and sampling methods are unbiased.
• Data should represent all relevant groups fairly.
• Be cautious of algorithmic bias, which can lead to discrimination.
Example:
In healthcare data, ensure equal representation of all genders and age
groups to avoid biased diagnostic predictions.
2. Consent
• Individuals should know what data is collected, why it’s collected,
and how it will be used.
• Obtain informed consent before using personal or survey data.
Example:
Before conducting a student survey, participants must be informed that
their responses will be used for academic research only.
3. Data Minimization
• Collect only data that is necessary for the research objective.
• Avoid collecting extra personal information “just in case.”
Example:
If a study analyzes exam scores, it may not require collecting students’
home addresses.
4. Confidentiality
• Ensure collected data is accessible only to authorized individuals.
• Use encryption, password protection, and secure storage.
Example:
Confidential employee data should be stored in protected files accessible
only to the HR research team.
5. Objectivity
• Avoid personal bias during data selection, analysis, or interpretation.
• Use statistical evidence rather than assumptions to draw
conclusions.
Example:
Do not ignore outliers simply because they contradict your hypothesis.
6. Social Responsibility
• Consider how results will impact society.
• Avoid creating models or analyses that could harm individuals or
communities.
Example:
Predictive policing algorithms must be evaluated carefully to prevent
discrimination against minority groups.
7.4 Ethical Challenges in Data Science
Challenge Description Example
Protecting personal Leakage of users’ health
Data Privacy
information data
Recruitment AI
Bias and Algorithm trained on
preferring certain
Discrimination biased data
genders
Misinterpretation of Wrong conclusions due to Correlation mistaken for
Results misuse of statistics causation
Not sharing code or data Hidden proprietary
Lack of Transparency
sources algorithms
Ownership and Shared datasets without
Who owns collected data
Copyright permission