Chapter Two
Activity 2
üWhat is Data Science?
üWhat are Data and Information?
üWhat is Big Data?
vWhat is Data Science?
o Multi disciplinary field that uses scientific methods,
processes, algorithms and systems to extract knowledge
and insights from structured, semi-structured and
unstructured data.
vWhat are Data and Information?
o Data: representation of facts, concepts, or instructions
in a formalized manner
o Information: processed data that decisions are based .
Data Processing Cycle
INPUT PROCESSING OUTPUT
Figure1: Data Processing Cycle
Examples:
Input: input data is storage on hard disk, flash disk.
Processing: Interest can be calculated on deposit to a bank.
Output: payroll for employees.
Ø Data types from Computer programming perspective:
oIntegers(int)- is used to store whole numbers, mathematically
known as integers
o Booleans(bool)- is used to represent restricted to one of two
values: true or false
oCharacters(char)- is used to store a single character
oFloating-point numbers(float)- is used to store real numbers
o Alphanumeric strings(string)- used to store a combination
of characters and numbers
---continued
vA data type makes the values that expression, such as a
variable or a function, might take.
üVariable: a memory location that can hold values of a
specific type.
üFunction: a group of statements that together performs
a task and method or subroutine or procedure.
vIt defines the operations that can be done on the data, the
meaning of the data, and the way values of that type can be
stored.
vData types from Data Analytics perspective:
§ Structured Data: adheres to a pre-defined data model
and is straightforward to analyze. Eg. Excel files and
SQL databases.
§ Semi-Structured Data: contains tags or other markers
to separate semantic elements . Eg. JSON and XML .
§ Unstructured Data: it is information typically text
heavy may contains data such as dates, numbers and
facts . Eg. Audio, Video files or Non SQL database.
---continued
Figure2 : Data types from a data analytics perspective
Ø Metadata is data about data.
Ø It provides additional information about a specific set of
data.
Ø In a set of photographs, for example, metadata could
describe when and where the photos were taken.
Ø It then provides fields for dates and locations which, by
themselves, can be considered structured data.
Ø Because of this reason, metadata is frequently used by Big
Data solutions for initial analysis.
Fig3: Examples of Metadata
Ø The is the information flow within a
big data system as a series of steps needed to generate
value and useful insights from data.
üData Acquisition : the process of gathering, filtering,
and cleaning data before it is put in a data warehouse or
any other storage solution .
üData Analysis: involves exploring, transforming, and
modeling data with the goal of highlighting relevant
data, synthesizing and extracting useful hidden
information with high potential from a business point of
view. Eg. DM,AI-ML
Ø Data Curation: categorized in to creation, selection,
classification, transformation, validation, and preservation.
- It is performed by expert curators that are responsible for
improving the accessibility and quality of data.
Ø Data Storage: It is the persistence and management of data
in a scalable way that satisfies the needs of applications that
require fast access to the data. Eg. RDBMS.
Ø Data Usage: It covers the data-driven business activities
that need access to data, its analysis, and the tools needed
to integrate the data analysis within the business activity.
Ø What is Big Data ?
ü is the term for a collection of data sets so large and
complex that it becomes difficult to process using on-
hand database management tools or traditional data
processing applications.
ü It is a blanket term for the non-traditional strategies
and technologies needed to gather, organize, process,
and gather insights from large datasets.
• Volume: large amounts of data Zeta bytes/Massive
datasets
• Velocity: Data is live streaming or in motion
• Variety: data comes in many different forms from diverse
sources
• Veracity: can we trust the data? How accurate is it? etc.
Fig4: Characteristics of big data
Ø Clustered Computing
• Because of the qualities of big data, individual
computers are often inadequate for handling the data at
most stages.
• To better address the high storage and computational
needs of big data, computer clusters are a better fit.
• Big data clustering software combines the resources of
many smaller machines, seeking to provide a number of
benefits:
• Resource Pooling: Combining the available storage space
to hold data is a clear benefit, but CPU and memory
pooling are also extremely important. Processing large
datasets requires large amounts of all three of these
resources.
• High Availability: Clusters can provide varying levels of
fault tolerance and availability guarantees to prevent
hardware or software failures from affecting access to data
and processing. This becomes increasingly important as we
continue to emphasize the importance of real-time analytics.
• Easy Scalability: Clusters make it easy to scale
horizontally by adding additional machines to the group. --
- This means the system can react to changes in resource
requirements without expanding the physical resources on
a machine.
ü Cluster membership and resource allocation can be
handled by software like Hadoop’s YARN (which stands
for Yet Another Resource Negotiator).
Ø Hadoop and Its Ecosystem
• Hadoop is an open-source framework intended to make
interaction with big data easier.
• It is a framework that allows for the distributed processing
of large datasets across clusters of computers using simple
programming models.
• It is inspired by a technical document published by
Google.
• Economical: Its systems are highly economical as ordinary
computers can be used for data processing.
• Reliable: It is reliable as it stores copies of the data on
different machines and is resistant to hardware failure.
• Scalable: It is easily scalable both, horizontally and
vertically. A few extra nodes help in scaling up the
framework.
• Flexible: It is flexible and you can store as much structured
and unstructured data as you need to and decide to use
them later.
vHadoop has an ecosystem that has evolved from its four
core components: data management, access, processing,
and storage.
vIt is continuously growing to meet the needs of Big Data.
v It comprises the following components and many others:
o HDFS: Hadoop Distributed File System
o YARN: Yet Another Resource Negotiator
o MapReduce: Programming based Data Processing
o Spark: In-Memory data processing
o PIG, HIVE: Query-based processing of data services
o HBase: NoSQL Database
o Mahout, Spark MLLib: Machine Learning algorithm
libraries
o Solar, Lucene: Searching and Indexing
o Zookeeper: Managing cluster
o Oozie: Job Scheduling
Fig6: Hadoop Ecosystem
Ø Ingesting data into the system
• The first stage of Big Data processing is Ingest.
• The data is ingested or transferred to Hadoop from
various sources such as relational databases, systems, or
local files.
• Sqoop transfers data from RDBMS to HDFS, whereas
Flume transfers event data.
Ø Processing the data in storage
• The second stage is Processing. In this stage, the data is
stored and processed. The data is stored in the
distributed file system, HDFS, and the NoSQL
distributed data, HBase. Spark and MapReduce perform
Ø Computing and analyzing data
• The third stage is to Analyze. Here, the data is analyzed by
processing frameworks such as Pig, Hive, and Impala.
• Pig converts the data using a map and reduce and then
analyzes it.
• Hive is also based on the map and reduce programming and
is most suitable for structured data.
Ø Visualizing the results
• The fourth stage is Access, which is performed by tools
such as Hue and Cloudera Search.
• In this stage, the analyzed data can be accessed by users.
Chapter Two Review Questions
1. Define data science; what are the roles of a data scientist?
2. Discuss data and its types from computer programming and
data analytics perspectives?
3. Discuss a series of steps needed to generate value and useful
insights from data?
4. What is the principal goal of data science?
5. List out and discuss the characteristics of Big Data?
6. How we ingest streaming data into Hadoop Cluster?
End
Thank You !!!