+ Big Data Challenge
+ How big is “Big Data”?
Big Data and You
+ How big is “Big Data”?
Big Data and You
It is the amount of data that is big enough to
challenge our computational resources!
+ Big Data Defined (3V)
n Velocity
n Moves at very high rates (think sensor-driven systems)
n Valuable in its temporal state
n Volume
n Fast-moving data creates massive historical archives
n Valuable for mining patterns, trends and relationships
n Variety
n From structured to unstructured
n Valuable for data enrichment/fusion
n 4V?
+ Big Data – Storage and Analytics
+ Where does Big Data come from?
Storage Analytics
Online NewSQL Analytic
gaming
§ Structured data Datastore
Ad § ACID guarantees
serving § Relational/SQL
Sensor
data
Financial
trade
§ Unstructured data
Internet § Eventual consistency
commerce
§ Schemaless
SaaS, § KV, document
Web 2.0
Mobile
platforms
NoSQL
+ Big Data Management Infrastructure
Storage Analytics
Online NewSQL Analytic
gaming
§ Structured data Datastore
Ad § ACID guarantees
serving § Relational/SQL
Sensor
data
Financial
trade
§ Unstructured data
Internet § Eventual consistency
commerce
§ Schemaless
SaaS, § KV, document
Web 2.0
Mobile
platforms
NoSQL
+ Big Data Management Infrastructure
Storage Analytics
Online NewSQL Analytic
gaming Datastore
Ad
serving
Sensor
data
Financial
trade
Internet
commerce
SaaS,
Web 2.0
Mobile
platforms
NoSQL
+ Big Data Management Infrastructure
Storage Analytics
Online NewSQL Analytic
gaming Datastore
Ad
serving
Sensor
data
Financial
trade
Internet
commerce
SaaS,
Web 2.0
Mobile
platforms
NoSQL
+ The Data LifeCycle
Exploratory
analysis Insights &
Data Data Data
& Decision
collection processing Analytics
Data Making
visualization
+ The Two Steps of Working with Data
(1) Collect data
Via computers, sensors, people, events, …
(2) Do something with it
Make decisions, gain insights, predict future, …
+ The Data LifeCycle
Exploratory
analysis Insights &
Data Data Data
& Decision
collection processing Analytics
Data Making
visualization
+
+ Netflix Prize I
n Recommender systems: predict a user’s rating of an item
n Twilight Wall-E Twilight II Furious 7
User 1 +1 -1 +1 ?
User 2 +1 -1 ? ?
User 3 -1 +1 -1 +1
Netflix Prize: $1MM to the first team that beats our in-
house engine by 10%
• Happened after about three years
• Model was never used by Netflix for a variety of reasons
• Out of date (DVDs vs streaming)
• Too complicated / not interpretable
+ Flu Prediction
n “DetectingInfluenza Epidemics using Search Engine
Query Data”, Nature 2009
n 50M common search terms
n 2003-2008 CDC seasonal flu spreading trends
n Training data: 2003-2006; Test data: 2007–2008
n 450M combinations è 45 search terms with strong
association
n Validation：successful predication of 2009 bird flu
outbreak
n Deployed in 29 countries by WHO
+
Sports
(1) Collect data
(2) Do something with it
Overview CS102
+ The Data LifeCycle
Exploratory
analysis Insights &
Data Data Data
& Decision
collection processing Analytics
Data Making
visualization
+ Data Collection and Preparation
n Data Preparation
n Filling in missing values
n Removing suspicious data
n Making formats, and units consistent
n Data Collection
n Storing big data
n Collecting (big) continuous stream data
+ Data Processing
Performing well-defined computations or asking
well-defined questions (“queries”)
n Average January low temperature for each country
over last 20 years
n Numberof items over $100 bought by females
between ages 20 and 30
n Theten stocks whose price varied the most over the
past year
+ What is DBMS?
n Database
n Data
n DBMS Application Interactive
Programs Queries
n Applications/Users
n What is DBMS？
n The relational model DBMS
n SQL/Query processing
n ACID/Transaction
Management
n Whatdoes RDBMS do Database
and don’t do?
+ The Database
n Data models
n The relational model
n Non-relational; object oriented, object relational, network,
hierarchical
n Now semi-structured and unstructured
n Database design
n Entity-Relationship diagrams, functional dependencies,
normalisation
n Physical storage
n Organisation, hashing, indexing
+ The Database
n Data models
n The relational model
n Non-relational; object oriented, object relational, network,
hierarchical
n Now semi-structured and unstructured
n Database design
n Entity-Relationship diagrams, functional dependencies,
normalisation
n Physical storage
n Organisation, hashing, indexing
+ The Data LifeCycle
Exploratory
analysis Insights &
Data Data Data
& Decision
collection processing Analytics
Data Making
visualization
+ Data Visualization
Basic Data Visualization
n “A picture is worth a thousand words”
Don’t underestimate the power of basic visualizations
§ Bar charts
§ Pie charts
§ Scatterplots
§ Maps
Overview CS102