Programming for Data Science
Course Introduction
Phạm Trọng Nghĩa
ptnghia@[Link]
We do data science
activities in daily life
2
Introduction
• Should I register for this
course?
• Collect data
• Syllabus
• Reviews of students who
havelearned the course
• … • Is data correct
• Run some analysis algorithm in and enough?
the head
• Make decision
3
Introduction
• Should I buy this product on
Tiki?
• Collect data: reviews
• Run some analysis algorithm in
thehead
• Make decision
• Is data correct
and enough?
4
What data science is
5
Data science lifecycle
• The "data science lifecycle" you will see in the wild may be
slightly different than the one we teach you, but the core
ideas are all the same.
6
Data science lifecycle
• Ask a meaningful question
• Collect data
• Explore data: look at data to understand more aboutdata, to
identify problems in data, …
• Preprocess data
• Analyze data (simple analysis: do computation
andvisualization to answer questions about things in
ourobserved data; complex analysis: model data
usingmachine learning to answer questions about things
outof our observed data)→ the answer
• Communicate results / make decision
7
Data science lifecycle
• The data science
lifecycle is a high-level
Ask a Obtain
description of the data Question Data
science workflow.
• Note the two distinct
entry points!
Understand Understand
the World the Data
Reports, Decisions,
and Solutions
8
1. Question/Problem Formulation
● What problems should
we solve?
● What hypotheses should Ask a Obtain
Question Data
we test?
● What are our metrics for
success?
Understand
the Data
Reports, Decisions,
and Solutions
9
2. Data Acquisition and Cleaning
● What data do we have
and what data do we
Ask a Obtain
need? Question Data
● How can we sample
more data?
● Is our data Understand Understand
the World the Data
representative of the
population we want to
Reports, Decisions,
study? and Solutions
10
3. Exploratory Data Analysis &
Visualization
● How is our data
organized and what does Ask a Obtain
it contain? Question Data
● Do we have relevant
data?
● What are the biases, Understand Understand
the World the Data
anomalies, or other
issues with the data?
Reports, Decisions,
and Solutions
● How do we transform the
data to enable effective
analysis?
11
4. Prediction and Inference
● What does the data say
about the world?
Ask a Obtain
● Does it answer our Question Data
questions or accurately
solve the problem?
● How robust are our Understand Understand
conclusions and can we the World the Data
trust the predictions?
Reports, Decisions,
and Solutions
12
Data science lifecycle
• This data science process is not linear step-by-step
• It can start from a question
• But it can also start from data and we will identify questions after
exploring data
• From one step, we may need to go back to a step before to adjust
things, and we may need to go backand forth many times
• To do this process well, data scientists need to:
• Stay calm, objective, honest
• Master software computer tools for data science
13
Data science: definitions
• Data Science is the application of data-centric,
computational, and inferential thinking to:
• Understand the world (science)
• Solve problems (engineering)
14
What Do Industry Data Scientists Do?
• The tasks that data
scientists say they
work on regularly.
• Hmm, can't a large
language model
(LLM) do a lot of
these tasks? Will
LLMs replace data
scientists?
15
Source: Based on the Berkeley Data Science Salary Survey
Are LLMs going to replace data
scientists?
• Short answer: No one knows for sure, but ...
Step 1 Step 2 Step 3
Gather context and Plan analyses that
Run your chosen
ask insightful address your
analyses.
questions. question(s).
LLMs cannot deeply LLMs can provide advice as With proper context and
understand the nuanced you weigh options. instructions, LLMs can
desires, abilities, and help a lot here!
personalities of complex But, the choice of "best"
organizations. analysis is often subjective. But, you must be able to
Experience matters. verify outputs. Don't do
At least for now analysis by "vibes".
16
Are LLMs going to replace data
scientists?
• Short answer: No one knows for sure, but ...
Step 1 Step 2 Step 3
Gather context and Plan analyses that
Run your chosen
ask insightful address your
analyses.
questions. question(s).
• Many Data homework problems can be answered by LLMs.
• If LLMs write your answers, you are fully outsourcing Steps 1
and 2, and you never learn to verify the outputs of Step 3.
• In other words, you are setting yourself up for future
replacement by LLMs, regardless of your
degree/honors/grades.
17
LLM is this course
• We encourage you to use LLMs appropriately in this course!
• Use of AI-assisted methods and large language models
(LLMs), such as ChatGPT, to generate written or code
solutions to assignments is prohibited.
• Copying or closely paraphrasing any text directly a homework or lab
into LLM is considered cheating
• You should break problems into steps and ask for help with each
step in your own words
18
Using LLM in appropriate ways
• “What does this error message mean? <error message> What
could I do to fix it?”
• “Pandas is confusing me; what does the following pandas
expression do? <weird expression>”
• “How would I convert columns into separate rows in my
dataframe?”
• “Can you explain why the regular expression r’[a-z]’ only matches
one character? Can you give me a few practice regular
expression exercises?”
• If you use an LLM to assist with your exercises, please provide a
clear citation. Specify where you used it, what prompt you gave,
the response you received, and how you modified or adjusted that
response.
19
Important reminder
• Remember: the main goal here is to learn, truly learn
• You can discuss ideas with others as well as consult Internet
sources, but your writing and code must be your own,based
on your own understanding
• If you violate this rule, you will get 0 score for the course
20
Data science is hot
nowadays
22
Data science is hot nowadays
• Many Vietnamese companies need data scientists:
tiki,lazada, shopee, zalo, grab, fpt, …
• 2 main reasons make data science hot:
• Data gets bigger and bigger (beware: incorrect data and
correct data mix together, we often don’t knowwhich is
which)
• Computers get faster and faster, allowing process big
data in endurable time
23
Data science is hot nowadays
• Let’s make a tour of its applications in
different fields
24
Recommender system
• Question: what products will the user X like?
• Data: history data of the user X (have liked/bought what
products) and other users
25
Web design for online shop
• Question: how should the website be designed to attract
customers?
• Data: interaction data of customers with different design
versions of the website
Image source: [Link]
26
Sentiment analysis
• Question: Does user community respond positively or
negatively to the new product X?
• Data: users’ reviews in social networks
27
Epidemic control
• Question: what is the current state of the epidemic X in
different locations?
• Data: data about the epidemic X in different locations (which
can be collected via newspapers, social networks, …)
[Link] 28
Precision agriculture
• Question: where in the farm field do we need supply
water/fertilizer/…?
• Data: data about the field collected via drone, …
Image source: [Link]
29
farming/
Education improvement
• Question: what aspects of the course X do we need to
improve?
• Data: feedback data of students about the course
30
Criminal investigation
• Question: what crimes were committed by the same
offender?
• Data: data about different crimes
31
Course contents
• In this course, we will learn about tools for doing data
science process:
• Linux commands
• Git & Github
• Conda
• Jupyter notebook
• Markdown
• Python
• Matplotlib
• Numpy
• Pandas
32
Data science activities
• Ask a meaningful question
• Collect data
• Explore data: look at data to understand more aboutdata, to
identify problems in data, …
• Preprocess data
• Analyze data (simple analysis: do computation
andvisualization to answer questions about things in
ourobserved data; complex analysis: model data
usingmachine learning to answer questions about things
outof our observed data)→ the answer
• Communicate results / make decision
33
Course contents
• After successfully completing this course, you will be able to
use learned tools to do learned data science process with
new data
34
Course contents
• Question: What is the relationship between this course and
the“Introduction to Data Science” course?
• Answer (my view):
• The “Introduction to Data Science” course focuses more on
data science process than tools (try to cover all steps in data
science process, understand tools at a basic level)
• This course focuses more on tools than data science
process (not try to cover all steps in data science process;
instead, using time to go deep into important tools)
35
Course assessment
• Homeworks throughout the course: 20% of the grade
• Lab exercises throughout the course: 40% of the grade
• Group final project: 40% of the grade
• 2-3 students / group
• Use Git & Github to control versions and collaborate
• Find a public dataset about things your group is interested in, use
tools to: explore data, identify meaningful questions which can be
answered with thisdata, preprocess and analyze data to answer
each question
• Present results to Teacher
36
References
• The “Stopping to Sharpen Your Tools” talk of Brandon Rhodes
• The “The Missing Semester” course of MIT
• The “Computational and Inferential Thinking” book of Berkeley
• Tool documents:
• Git
• Conda
• Jupyter notebook
• Python
• Numpy
• Matplotlib
• Pandas
37
THE END
38