Module V

Data Analytics, Ethics, and Applications

Part 1: Data Analytics Methodologies

Data analytics methodologies are standardized, step-by-step frameworks that guide data
science projects from conception to deployment. They ensure projects are structured,
repeatable, and aligned with business goals.

1. CRISP-DM Methodology (Cross-Industry Standard Process for Data Mining)

CRISP-DM is the most widely used and proven methodology for data mining and analytics
projects. It is technology-neutral and consists of six iterative phases.

 Phases of CRISP-DM:

1. Business Understanding: The most critical first step. It focuses on understanding the project
objectives and requirements from a business perspective, then converting this knowledge into
a data mining problem definition and a preliminary plan to achieve the objectives.

 Example Question: "How can we reduce customer churn?" becomes "Can we build a model
to predict which customers are likely to cancel their subscriptions in the next 3 months?"

2. Data Understanding: Starts with initial data collection and proceeds to activities to get
familiar with the data, identify data quality problems, discover first insights, or detect
interesting subsets to form hypotheses.

 Activities: Data exploration, descriptive statistics, visualizing data distributions.

3. Data Preparation: Covers all activities to construct the final dataset from the initial raw
data. This is typically the most time-consuming phase (often 70-80% of the project time) and
includes tasks like table, record, and attribute selection, as well as cleaning and
transformation of data for modeling tools.

 Activities: Handling missing values, creating new features (feature engineering), merging
datasets.

4. Modeling: Various modeling techniques are selected and applied, and their parameters are
calibrated to optimal values. Often, there is a loop back to Data Preparation to bring the data
into a format suited for a specific modeling technique.

 Activities: Applying algorithms like decision trees, neural networks, or regression models.

5. Evaluation: Thoroughly evaluate the model(s) to ensure they achieve the business objectives
established in Phase 1. A key step is to see if any business issues have not been sufficiently
considered. At the end of this phase, a decision on the use of the data mining results should
be reached.
 Activities: Testing model accuracy on unseen data, checking if the model's results make
business sense.

6. Deployment: The knowledge gained will need to be organized and presented in a way that
the customer can use it. The deployment phase can be as simple as generating a report or as
complex as implementing a repeatable data mining process across the enterprise.

 Activities: Creating a dashboard, writing a final report with actionable insights, integrating
the model into a production application.

2. SEMMA Methodology (Sample, Explore, Modify, Model, Assess)

Developed by the SAS Institute, SEMMA is another prominent data mining methodology. It
focuses on the technical aspects of model building, assuming the business understanding has
already been addressed.

 Phases of SEMMA:

1. Sample: Extract a representative sample of the data. Working with a manageable subset
makes analysis faster and more efficient. The sample should be large enough to contain
significant information, yet small enough to process quickly.

2. Explore: Visually and statistically explore the data for underlying structures, trends, outliers,
and anomalies. This phase uses techniques like clustering, factor analysis, and visualization
to form initial hypotheses.

3. Modify: Transform, select, and create new variables (features) to prepare the data for
modeling. This includes handling missing values, grouping values, and applying
mathematical transformations.

4. Model: Apply various modeling techniques (neural networks, decision trees, logistic
regression) to the prepared data to search for a combination of variables that reliably predicts
the desired outcome.

5. Assess: Evaluate the usefulness and reliability of the models by testing them on new data.
This phase checks how well the model performs against real-world scenarios and compares
different models to choose the best one.

3. Big Data Life Cycle

This framework is adapted for the unique challenges of big data: volume, velocity, and
variety. It acknowledges that not all data can be easily loaded into a traditional database.

 Phases of the Big Data Life Cycle:

1. Data Ingestion: The process of collecting and importing raw data from various sources (e.g.,
IoT devices, social media feeds, weblogs) into a storage system. Tools like Apache Kafka or
Flume are often used.

2. Data Storage: Storing the ingested data in a distributed, scalable file system like HDFS
(Hadoop Distributed File System) or cloud storage (e.g., AWS S3).
3. Data Processing: Cleaning, transforming, and analyzing the data using distributed
processing frameworks like Apache Spark or Hadoop MapReduce.

4. Data Analysis: Performing complex queries, statistical analysis, and machine learning on the
processed data.

5. Data Visualization & Interpretation: Presenting the results in a consumable format, often
through dashboards or reports that can handle large-scale data.

4. SMAM (Stanford Data Mining and Management)

While not a formal "methodology" like CRISP-DM, the term often refers to the holistic
approach to data science emphasized by Stanford's programs. It integrates data mining, data
management, statistical analysis, and machine learning into a cohesive workflow, stressing
the importance of robust data infrastructure and scalable algorithms.

5. ASUM-DM (Analytics Solutions Unified Method for Data Mining/Predictive

Analytics)
Developed by IBM, ASUM-DM is an extension and enhancement of CRISP-DM. It provides
a more detailed, step-by-step guide with specific tasks, roles, and deliverables for each phase.
It incorporates modern agile principles and addresses the entire analytics life cycle, including
the often-challenging deployment and maintenance phases.

Part 2: Ethics for Data Scientists

As data scientists wield immense power to analyze and influence human behavior, ethical
guidelines are not optional—they are a fundamental responsibility.

Ethical Guidelines for Data Scientist:

1. Beneficence ("Do Good"): Strive to use data for beneficial purposes that improve individual
and societal well-being.

2. Non-maleficence ("Do No Harm"): Actively work to avoid causing harm. This includes
considering potential misuse of your models or analyses and mitigating negative
consequences.

3. Autonomy (Respect for Persons): Respect individuals' rights to make their own decisions.
This involves obtaining informed consent for data collection and use.

4. Justice (Fairness): Ensure that the benefits and burdens of data science are distributed fairly.
Actively test for and mitigate bias in algorithms that could discriminate against protected
groups (e.g., based on race, gender, age).

5. Transparency & Explainability: Be open about the methods used, the data sources, and the
limitations of the models. When possible, use models that can be explained to those affected
by their decisions.
6. Accountability: Take responsibility for the outcomes of your work. Establish clear lines of
responsibility for the models and decisions they inform.

7. Privacy: Treat all personal data with the utmost care, adhering to privacy laws and
organizational policies. Anonymize data where possible.

Data Science Concerns:

 Bias and Fairness: Models can learn and amplify societal biases present in the training data,
leading to unfair or discriminatory outcomes (e.g., biased hiring algorithms, biased loan
approval systems).

 Opacity ("Black Box" Problem): Some complex models (like deep neural networks) are so
intricate that it's impossible to understand how they arrived at a specific decision, making it
difficult to trust or debug them.

 Surveillance: The use of data science for mass surveillance can infringe on civil liberties and
create a chilling effect on free expression.

 Security: Data breaches can expose sensitive personal information, leading to identity theft,
financial loss, and reputational damage.

 Misinformation: Data science techniques can be used to create and spread highly targeted
and convincing misinformation and "deepfakes."

Data Privacy and Legal Aspects:

 Data Privacy: Refers to the proper handling, processing, and storage of personal
information. It's about giving individuals control over their data.

 Key Legal Frameworks:

o GDPR (General Data Protection Regulation): A comprehensive EU law that sets a global
standard for data protection. It grants individuals rights over their data (e.g., right to access,
right to be forgotten) and imposes heavy fines for non-compliance.

o CCPA (California Consumer Privacy Act): A state-level law in the U.S. that gives
California residents similar rights over their personal information.

o HIPAA (Health Insurance Portability and Accountability Act): A U.S. law that sets strict
standards for the protection and confidential handling of protected health information (PHI).

Rightful Use of Data Science:

Data science should be used to empower individuals and benefit society, not to deceive,
manipulate, or discriminate. Rightful use involves:

 Being transparent about data collection and use.

 Ensuring fairness and accountability in automated decisions.

 Protecting individual privacy and data security.

 Using data to solve genuine problems and create positive value.

Part 3: Applications of Data Science

Data science is a transversal discipline, meaning its methods can be applied to almost any
field. Here are some key application areas:

 Bioinformatics:

o Genomics: Analyzing DNA sequences to identify genes linked to diseases, understand

genetic variations, and enable personalized medicine (tailoring treatments to an individual's
genetic makeup).

o Drug Discovery: Using machine learning to screen millions of chemical compounds to

predict which ones might be effective as new drugs, significantly speeding up the discovery
process.

o Protein Structure Prediction: Predicting the 3D structure of proteins from their amino acid
sequences, which is fundamental to understanding their function (e.g., DeepMind's
AlphaFold).

 Education:

o Personalized Learning: Analyzing student performance data to create customized learning

paths, recommend relevant materials, and adapt the pace of instruction to individual needs.

o Predicting Student Success: Identifying students at risk of dropping out or failing early,
allowing educators to intervene with targeted support.

o Automated Grading & Feedback: Using NLP to grade essays and provide instant feedback
on assignments, freeing up teachers' time.

 Engineering:

o Predictive Maintenance: Analyzing sensor data from machinery (e.g., jet engines, wind
turbines) to predict when a component is likely to fail. This allows for proactive, just-in-time
maintenance, reducing downtime and costs.

o Design Optimization: Using simulations and machine learning to optimize engineering

designs for performance, cost, and material usage (e.g., designing lighter, stronger airplane
wings).
o Quality Control: Using computer vision to automatically inspect products on an assembly
line for defects with greater accuracy and speed than humans.

 Finance and Economy:

o Algorithmic Trading: Using complex models to make high-speed, automated trading

decisions in financial markets.

o Fraud Detection: Analyzing transaction data in real-time to identify and block fraudulent
credit card usage or insurance claims. This is a classic anomaly detection problem.

o Risk Management: Assessing the credit risk of loan applicants (credit scoring) and modeling
market risks for investment portfolios.

o Customer Analytics: Segmenting customers for targeted marketing, predicting customer

lifetime value, and developing personalized financial product recommendations.

 Gaming:

o Game Design & Balancing: Analyzing player behavior data to understand how levels are
played, identify bottlenecks, and balance game difficulty to keep players engaged.

o Player Experience Modeling: Predicting player churn and offering incentives to retain them,
or dynamically adjusting game difficulty in real-time to match a player's skill level.

o Cheat Detection: Using anomaly detection to identify players using unauthorized software
or exploiting bugs to gain an unfair advantage.

o Procedural Content Generation: Using algorithms to automatically create new game levels,
maps, or characters, increasing the game's replayability.