14/09/24

Data Mining
S1

NOVA-IMS 2024/2025
Fernando Lucas Bação
bacao@[Link]
[Link]

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

Agenda

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

1
 14/09/24

Intro to the Data Mining

Course

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

Syllabus

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

2
 14/09/24

Course Success

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

Course Success

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

3
 14/09/24

Project

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

Grading

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

4
 14/09/24

References

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

References

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

10

5
 14/09/24

References

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

11

The Context

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

12

6
 14/09/24

Agenda

• The Growth of the Digital Universe

• Buzz Words and Definitions
• Data Science
• Different Roles in Data Science

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

13

The Growth of the Digital

Universe

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

14

7
 14/09/24

The Digital Universe

[Link]

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

15

Sensors Everywhere

Apps
Sensors

Frequent customers
cards

Online shop
Credit cards

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

16

8
 14/09/24

Sensors Everywhere

Location

Email

Blogs

Social networks Video and pictures

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

17

Sensors Everywhere

• The current A350 model has a total

of close to 6,000 sensors across the
entire plane and generates 2.5 Tb of
data per day, while the newer model
– expected to take to the skies in
2020 – will capture more than triple
that amount.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

18

9
 14/09/24

Sensors Everywhere

• A host of electronic devices, including

ECU (Engine Control Unit or as some
people call it Electronic Control Unit)
which transmits specific data, for
example measurements, but not only, to
a remote site, in F1 case, to pit wall and
pit garage. It electronically records
performance of engine, status of
suspensions, gearbox data, fuel status,
all temperature readings including tires
temperature, g-forces and actuation of
controls by the driver. The data is then
used as a foundation for determining car
setup and all problems.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

19

The Data Gap

“Data have swept into every industry and

business function and are now an
important factor of production,
alongside labor and capital.”

McKinsey Global Institute

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

20

10
 14/09/24

The Data Gap

“Although the volume of data created is expected to

increase fivefold over the next five years, best-guess
estimates suggest that less than 10 percent of the
information created is meaningfully organized or deployed.
That number will only shrink as the rate of information
production goes up.”

“The productivity im perative”, Bisson, P., Stephenson, E., Viguerie, S., M cKinsey Quarterly June 2010)

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

21

The Data Gap

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

22

11
 14/09/24

The Economist: The data deluge

[Link]

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

23

Buzz Words and Definitions

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

24

12
 14/09/24

Buzzwords

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

25

Buzzwords

Reality Media

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

26

13
 14/09/24

Big Data

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

27

Big Data

‘‘Anything that Won't Fit in Excel”

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

28

14
 14/09/24

Big Data

‘we will simply take big data to mean

datasets that are too large for
traditional data-processing systems
and that therefore require new
technologies.”

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

29

Big Data

30
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

30

15
 14/09/24

Big Data

31
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

31

Big Data

32
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

32

16
 14/09/24

Big Data

Key Spotify Statistics

551 monthly active Spotify users;
220 million Spotify Premium subscribers;
25 hours spent listening per month;
Average revenue per user $4.72
82 million tracks available on Spotify;

[Link]
33
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

33

Big Data

Didi Chuxing
60 million rides each day;
450 million users;
400 chinese cities;
7.43 billion rides in 2017;
15 million registered drivers;

[Link]
34
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

34

17
 14/09/24

Artificial Intelligence

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

35

Artificial Intelligence

• Artificial Intelligence is the science of making things

smart, and can be defined as:

“human intelligence exhibited by machines”

• A broad term for getting computers to perform human

tasks

• The scope of AI is disputed and constantly changing

over time

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

36

18
 14/09/24

Artificial Intelligence

• The systems implemented today are a form of

narrow AI – a system that can do just one (or a
few) defined things as well or better than
humans
• Like recognizing objects
• Play chess
• Jeopardy

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

37

Machine Learning

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

38

19
 14/09/24

Machine Learning

• Machine learning can be defined generally as:

• An approach to achieve Artificial Intelligence
through systems that can learn from experience to
find patterns on a set of data
• ML involves teaching a computer to recognize
patterns by example, rather than programming it
with specific rules.
• These patterns can be found within data

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

39

Machine Learning

• Machine learning can be defined generally as:

ML is about creating algorithms (or a set of

rules) that learn complex functions (or patterns)
from data and make predictions on it

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

40

20
 14/09/24

Machine Learning

• Machine learning can be defined generally as:

• ML is essentially about predicting stuff

1. It takes some data, to train the system

2. Learns patterns from this data

3. Classifies new data it has not seen before (makes a

guess on what probably is based on the knowledge
gained in the previous step)

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

41

Machine Learning

• What is extremelly cool about ML is that it learns by itself

from the data

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

42

21
 14/09/24

Machine Learning

• What is extremelly cool about ML is that it learns by itself

from the data

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

43

Machine Learning

Exponential
increases in
computing power
and storage

Algorithmic
Explosion of data
Advances

New A convergence of algorithmic

Opportunities advances, data proliferation, and
to Solve tremendous increases in computing
Problems power and storage has propelled AI
from hype to reality.

44
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

44

22
 14/09/24

Artificial Intelligence and

Machine Learning

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

45

Artificial Intelligence and

Machine Learning

Engineering of making intelligent machines and programs

Ability to learn without being explicitly programmed

46
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

46

23
 14/09/24

Artificial Intelligence and

Machine Learning

47
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

47

Artificial Intelligence and

Machine Learning

Artificial Intelligence

Machine Learning
Deep Learning

Foundation Models
LLM Vis. Aud.

Generative AI

LG CT NN Ct SVM

48
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

48

24
 14/09/24

Artificial Intelligence and

Machine Learning

Artificial Intelligence

Machine Learning

Big Data

Source: AI Lum inary Series: Pedro Dom ingos Explains M achine Learning [Link] w w .[Link]/content/w w w /us/en/analytics/ai-lum [Link] l
49
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

49

Data Science

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

50

25
 14/09/24

Data Science

• Data science is the study of where information comes

from, what it represents and how it can be turned into a
valuable resource in the creation of business and IT
strategies.

• “Data scientists are involved with gathering data,

massaging it into a tractable form, making it tell its story,
and presenting that story to others,” — Mike Loukides

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

51

Data Science vs Data Mining

• Data Science • Data mining

• Data science is a set of • The actual extraction of
fundamental principles that
support and guide the knowledge from data via
principled extraction of technologies that incorporate
information and knowledge from the data science principles.
data.
• There are hundreds of
• Data science involves more different data-mining
than just data-mining algorithms, and a great deal
algorithms. Successful data of detail to the methods of the
scientists must be able to view field.
business problems from a data
perspective.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

52

26
 14/09/24

Data Science

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

53

Data Science

• Data Mining Scientist, Apple - Austin - Texas –US

• Key Qualifications

• Strong working knowledge of data mining algorithms including decision

trees, probability networks, association rules, clustering, regression, and
neural networks.
• Familiarity with database modeling and data warehousing principles with a
working knowledge of SQL.
• Creativity to go beyond current tools to deliver best solution to the problem
• Strong programming skills in Python, Java, C , or similar language
• Excellent interpersonal, written, and verbal communication skills
• Ability and comfort working independently and making key decisions on
projects

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

54

27
 14/09/24

Data Science

• iPhone Operations Data Scientist, Apple - Santa Clara Valley - California -

US

• Key Qualifications

• The position requires a software programming skill set, utilization of statistical

techniques, experience managing data integrity, and implementing automated
solutions. The Data Scientist will need to have an understanding of relational database
management systems, design, and structured query language.
• Excellent analytical skills, high level of statistics with the ability to identify and predict
trends and anomalies.
• Experience in data mining extremely large data sets, high proficiency in SQL (Teradata,
Oracle, or MySQL), relational database management systems and design.
• Strong programming, with excellent object-oriented and dynamic scripting language
skills. Python, Java/C#/Objective-C, HTML5, CSS3, JavaScript, and Unix shell
scripting are strongly desired.
• Experience with statistical tools like JMP, Minitab, and Stata
• Strong ability to manage multiple tasks concurrently and in a timely manner, including
large, complex projects.
• Effective presentation skills and be able to explain complex data and charts in a clear
manner to large audiences. Outstanding communication skills, both verbal and written.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

55

Data Science

• Big Data Scientist Healthcare, Philips - US-Massachusetts-

Cambridge

• The successful candidate in this position brings/has:

• PhD in a quantitative field such as Computer Science, Engineering, Statistics or Data
Science.
• Strong knowledge of machine learning algorithms (clustering, classification and
regression), probability theory, and statistical analysis.
• Predictive modelling experience.
• Experience using one or more data analytics tools: R, Python, or MATLAB
• Strong communications skills and ability to work on a global team.
• Skills and experience in some of the following areas would be a
plus:
• Experience working with Big Data and strong knowledge of big data infrastructure (e.g., Hadoop,
Hive, Spark, Elastic Search, AWS)
• Experience in using data exploration and visualization tools (e.g., Tableau, Splunk)
• Programming experience with C/C++/C# or Java
• Expert skills in SQL
• Experience in using Healthcare databases

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

56

28
 14/09/24

Data Science

• Data Scientist, KPMG US - New York, NY

• Qualifications:
• Experience with command-line scripting, data structures and algorithms and ability to
work in a Linux environment, processing large amounts of data in a cloud environment
• Masters degree or PhD from an accredited college/university in Computer Science,
Statistics, Mathematics, Engineering, Bioinformatics, Physics, Operations Research, or
related fields, with a minimum of two years of relevant experience (strong mathematical
background with ability to understand algorithms and methods from a mathematical
viewpoint and an intuitive viewpoint)
• Strong knowledge in at least one of the following fields: machine learning, data
visualization, statistical modeling, data mining, or information retrieval
• Strong data extraction and processing, using MapReduce, Pig, and/or Hive preferred
• Proficiency in analysis (e.g. R, SAS, Matlab) packages, programming languages (e.g.
Java, Python, Ruby) as well as the ability to implement, maintain, and troubleshoot big
data infrastructure, such as distributed processing paradigms, stream processing and
databases such as Hadoop, Storm, SQL and Solr

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

57

Data Science

• Data Scientist, Intel Security - NA-USA-CA-Santa Clara

• Qualifications
• PhD or MS degree in Computer Science, Electrical Engineering, Statistics,
Applied Math, Econometrics, Operations Research, or other related fields
• Deep understanding of statistical modeling, machine learning, or data mining
concepts, and a track record of solving problems with these methods
• Familiar with one or more machine learning or statistical modeling tools such as
R, Matlab and scikit learn
• Experience with big data techniques (such as Hadoop, MapReduce, Hive, Pig,
Spark, Storm)
• Proficient in one or more programming languages such as Python, Java and C
• Knowledge and experience of working with SQL and NoSQL databases
• Strong analytical and quantitative problem solving ability
• Excellent communication, relationship skills and a strong team player

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

58

29
 14/09/24

Data Science

• Data & Applied Scientist, Microsoft - Redmond, WA, US

• Job requirements:
• Programming skills related to data using technologies like Python, PERL, C#, etc.
• Stats and data analysis experience working with advanced tools like R, SAS and advanced
Excel
• Experience using data to build ML and statistical models that impact a product or business
• Ability to access, reduce, and join large data sets programmatically
• Natural curiosity that extends beyond your daily activities to help stitch together different
areas of the company, market and technology in new ways
• Ability to think algorithmically about product and business issues
• Willingness to work in a start-up organization with evolving responsibilities and a wide variety
of work
• Understanding that getting value from imperfect data and systems is a core virtue for a data
scientist
• Creativity to find pragmatic paths through ambiguity
• Attention to detail and accuracy
• Ability to collaborate with people representing diverse points of view
• Solid writing, presentation and data visualization skills
• Other desirable skills and experiences:
• Engineering experience building large data systems based on SQL, Hadoop, etc.

• Experience with product and service telemetry systems

• Experience with Feedback/Text analysis

• Experience conducting multivariate experiments such as A/B testing

• Ability to bring broad product, customer, and market context into data analyses
• Ability to interact with senior leaders to drive product and business impact

• 2 years' experience in data exploration, analysis, programming, and modeling

• Degree in computer science, machine learning, statistics, math, economics, business or other scientific or quant-focused field, MS or PHD preferred.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

59

Data Science

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

60

30
 14/09/24

Different Roles

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

61

Data Science – Different Roles

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

62

31
 14/09/24

Data Science – Different Roles

• Business Analyst:
• Business analysts’ strengths
lie in their business acumen.

• They can communicate well

with both the data scientist
and C-suite to help drive
data-driven decisions faster.

• The best business analysts

also have skills in statistics to
be able to glean interesting
insights from past behavior.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

63

Data Science – Different Roles

• Data Scientist:
• Data science is largely rooted
in statistics, data modeling,
analytics and algorithms.

• They focus on conducting

research, optimizing data to
help companies get better at
what they do.

• The minds behind

recommended products on
Amazon.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

64

32
 14/09/24

Data Science – Different Roles

• Data Engineer:
• While data scientists dig into
the research and visualization
of data, data engineers
ensure the data is
powered and flows correctly
through the pipeline.

• They’re typically software

engineers who can engineer
a strong foundation for data
scientists or analysts to think
critically about the data.

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa

65

Data Science – Different Roles

Fonte: [Link]
Instituto Superior de Estatística e Gestão de Informação
Universidade Nova de Lisboa

66

33
 14/09/24

Questions?

Instituto Superior de Estatística e Gestão de Informação

Universidade Nova de Lisboa 67

67

34