Introduction to Big Data Analytics

(BCSE0157)

Credit Unit: 03

Dr. Robin Singh Bhadoria,

Asst. Professor, Dept. of CEA
 Jaccard Similarity
• The Jaccard index, also known as the Jaccard
similarity coefficient, is a statistic used for
gauging the similarity and diversity of sample
sets. It was developed by Paul Jaccard,
originally giving from the French name.
How to Calculate the Jaccard Index
The formula to find the Index is:
• Jaccard Index = (the number in both sets) /
(the number in either set) * 100

The same formula in notation is:

 The Jaccard Distance
• The Jaccard distance measures the
dissimilarity between two datasets and is
calculated as:

Jaccard distance = 1 – Jaccard Similarity

 Example
• A simple example using set notation: How
similar are these two sets?
A = {0,1,2,5,6}
B = {0,2,3,4,5,7,9}
• Solution: J(A,B) = |A∩B| / |A∪B| =
=|{0,2,5}| /
|{0,1,2,3,4,5,6,7,9}|
= 3/9 = 0.33.

Jaccard Index= 0.33 * 100 = 33

 Another Example
• E = ['cat', 'dog', 'hippo', 'monkey']
• F = ['monkey', 'rhino', 'ostrich', 'salmon']

• Number of observations in both, i.e. |E ∩ F|:

= {‘monkey’} = 1
• Number of observations in either, i.e. |E ∪ F| :
={‘cat’, ‘dog’, hippo’, ‘monkey’, ‘rhino’, ‘ostrich’,
‘salmon’} = 7
• Jaccard Similarity: 1 / 7= 0.142857
 Big Data Stack (Layers)
• Organizations support storing, ingesting, processing
and analyzing huge quantities of data
 Data Layer (Physical)
• Store masses of raw data, which comes from
traditional sources.
– OLTP databases,
– less structured sources like log files, sensors, web
analytics, document and media archives.
• It is the backend of the entire system
 Data Storage Systems
• Hadoop HDFS—the classic big data file system. It
became popular due to its robustness and limitless
scale.
• Amazon S3— It is Simple Storage Service (S3) create
buckets and load the data using a variety of
integrations, with 99.999999999% guaranteed of
durability. S3 is simple, secure, and provides a quick
and cheap solution for storing limitless amounts of big
data.
• MongoDB— a mature open source document-based
database, built to handle data at scale with proven
performance.
Data Ingestion & Integration Layer
• To create a big data store, you’ll need to
import data from its original sources into the
data layer.
• Data Ingestion is the process of transportation
of data from assorted sources to a storage
medium where it can be accessed, used, and
analyzed by an organization.
You can leverage a rich ecosystem of big data
integration tools, including powerful open source
integration tools, to pull data from sources,
transform it, and load it to a target system of
your choice
 Big Data Integration Tools
• Stitch — lightweight ETL (Extract, Transform,
Load) tool which pulls data from multiple pre-
integrated data.
• Blendo— A Cloud data integration tool that lets
you connect data sources with a few clicks, and
pipe all data to Amazon Redshift, PostgreSQL, or
MS SQL Server.
• Apache Kafka — an open source streaming
messaging bus that can creates a feed from your
data sources, partitions the data, and streams it
to a passive listener.
 Data Processing Layer
• It is most important layer in the end to end
Big Data technology stack.
• A analyst process large volume of data into
relevant data marts which finally goes to the
presentation layer.
• It is a layer where data is needed for analysis is
selected, cleaned, formatted for further
analysis and modelling.
 Data Processing Tools
• Apache Spark—like the old Map/Reduce but
over 100X faster. Runs parallelized queries on
unstructured, distributed data in Hadoop,
Mesos, & Kubernetes.
• PostgreSQL—many organizations pipe their
data to good old Postgres to facilitate queries.
• Amazon Redshift—a cloud-based data
warehouse offering blazing query speeds that
can be used as a relational database.
 Analytics & BI Layer
(Presentation)

It is the topmost layer

in the technology
stack which is where
the actual analysis &
insight generation
happens
 BI Tools
• Tableau—powerful BI and data visualization
tool, which connects to your data and allows
you to drill down, perform complex analysis,
and build charts and dashboards.
• Chartio—cloud BI service allowing you to
connect data sources, explore data and create
live auto-refreshing dashboards.
• Looker —cloud-based BI platform that lets
you query and analyze large data sets.
Big Data 1.0, Big Data 2.0 &
Big Data 3.0
 Traditional BI vs. Big Data
Environment
• Business Intelligence (BI) is the collection of
systems, software, and products, which can
import large data streams and use them to
generate meaningful information.
• Big Data is the most buzzing word in the
business. It is not a just massive amount of
data, it is also about processing the data with
the purpose of delivering added value to the
organization.
 NoSQL Databases
• A originally referring to "non-SQL" or "non-
relational” database
• NoSQL relies upon a softer model known as the
BASE model. BASE (Basically Available, Soft state,
Eventual consistency).
• Such databases have existed since the late 1960s,
but the name "NoSQL" was only coined in the
early 21st century.
• NoSQL databases are increasingly used in big data
and real-time web applications.
• NoSQL systems are also sometimes called "Not
only SQL"
 Characteristics of NoSQL
• Schema free
• Eventually consistent (as in the BASE property)
• Replication of data stores to avoid Single Point
of Failure.
• Can handle Data variety and huge amounts of
data.
 NoSQL databases fall into four main
categories
• Key value Stores — Riak, Voldemort, and
Redis
• Wide Column Stores — Cassandra and HBase.
• Document databases — MongoDB
• Graph databases — Neo4J and
HyperGraphDB.
Type of NoSQL Database
Column Family Databases
 Document Databases
• Document stores uses JSON, XML, or BSON
(binary encoding of JSON) documents to store
data.
• It is like a key-value database, but a document
store consists of semi-structured data.
• A single document is to store records and its
data
 Graph databases
• Graph database requires only storing data
once (nodes). The different types of
relationships (edges) are specified to the
stored data.
• The relationships between the nodes are
predetermined.
• Traversing persisted relationships are faster.
 Advantages of NoSQL
• Cost-effectiveness: it allow quickly scale
horizontally, better allocating resources to
minimize costs.
• Flexibility: can address large volumes of rapidly
changing data, making them great for agile
development.
• Replication: provides data reliability, ensuring
access during down time and protecting against
data loss if servers go offline.
• Speed: enables faster, more agile storage and
processing for all users,
NoSQL Vs. RDBMS
SQL & NoSQL
• It is a class of RDBMS
that seek to provide the
scalability of NoSQL
systems for online
transaction processing
(OLTP) workloads
• Also guarantees the
ACID properties of a
traditional database
system.
NewSQL compare with NoSQL & SQL
 Data Science
• Data science is an interdisciplinary field that
uses scientific methods, processes, algorithms
and systems to extract knowledge and insights
from noisy, structured and unstructured data.
• Apply knowledge and actionable insights from
data across a broad range of application
domains.
Components of Data Science
Responsibilities of a Data Scientist
• To first discover the actual problem and then analyze it
in such a way that it can benefit the organization to the
fullest.
• To gather a large amount of raw data from discrete
sources.
• To clean the large datasets and validate it to ensure
greater accuracy of the results.
• Finding different patterns, values, and relationships in
the data.
• Finally presenting the findings and results of the
complete process through data visualization and
various other means.
Data Science Life Cycle