JKU

Course Name
Introduction to Emerging Technologies
Course code: EMTE1011/1012
To: Social Science (Sec. 6 & 7), Regular

By: Mamaru E. (MSc. in IT)

@ 2017 E.C
Jinka, Ethiopia.
1
 Chapter 2
[Link] Science
 Data science, data vs. information, data types and representation, data value chain,
and basic concepts of big data.
 After completing this chapter, the students will be able to:

l i n
➢Data science and its role
u t
➢Data Vs. information. O e
➢Data processing life cycle
➢Data types from diverse perspectives
➢Data value chain in emerging era of big data.
➢ Big Data.
➢Hadoop ecosystem. 2
 2.1. Overview of Data Science
 Data science is a multi-disciplinary field that uses scientific methods, processes,
algorithms, and systems to extract knowledge.
 It offers a range of roles and requires a range of skills.
 Today, successful data professionals understand that they must advance past the
traditional skills of analyzing large amounts of data, data mining, and programming skills.
 In order to uncover useful intelligence for their organizations, data scientists must master
the full spectrum of the data science life cycle and possess a level of flexibility and
understanding to maximize returns at each phase of the process.

3
 Data scientists need to be curious (unique) and result-oriented, with exceptional industry-
specific knowledge & communication skills that allow them to explain highly technical results.
 They possess a strong quantitative background in statistics and linear algebra as well as
programming with a focus on data warehousing, mining, and modeling to build and analyze
algorithms.

2.2. What are Data and Information

A. Data
 It is a representation of unprocessed facts, figures, concepts, or instructions in a formalized manner
 Suitable for communication, interpretation, or processing by human or electronic machines
 It is represented with the help of characters like alphabets (A-Z, a-z), digits (0-9), special
characters (+, -, /, *, =)
4
B. Information
 It is data that has been processed into a form that is meaningful to the recipient.
 It is real or perceived value in the current or the prospective action or decision of the recipient.
 Created from organized, structured, and processed data in a particular context.
 Information is interpreted data;
C. Data Processing Cycle
 Data processing is the restructuring or reordering of data by people or machines to increase its
usefulness and add value.
 The data processing cycle consists of 3 steps: -

Data Processing Cycle 5

1. Input: prepared in some convenient form for processing.
 The form will depend on the type of processing machine.
 E.g., when electronic computers are used, the input data can be recorded on any one of the
several types of storage media, such as hard disk, CD, flash disk, and so on.
2. Processing: The input data is changed to produce data in a more useful form.
 For example, interest can be calculated on a deposit to a bank, or a summary of sales for the
month can be calculated from the sales orders.
3. Output: The result of the preceding processing step is collected.
 The particular form of the output data depends on the use of the data.
 E.g., output data may be payroll for employees.

6
 2.3. Data types and their representation
1. Data type in computer science and computer programming perspectives:
 Data type: attribute of data that tells the compiler/interpreter how the programmer intends to
use data.
 Common data types include:
 Integers(int): used to store whole numbers, mathematically known as integers
 Booleans(bool): are used to represent restricted to one of two values: true or false
 Characters(char): is used to store a single character
 Floating-point numbers(float) are used to store real numbers
 Alphanumeric strings(string): used to store a combination of characters and numbers
 A data type makes the values that an expression, such as a variable or a function, might take.
 Data type defines the operations that can be done on the data, the meaning of the data, and
the way values of that type can be stored. 7
 2. Data types from a Data Analytics perspective
 Common types of data types or structures: Structured, Semi-structured, and Unstructured data.

Data types from a data analytics perspective

A. Structured: data that adheres/attach to a pre-defined data model, and is straightforward to analyze.
 Structured data conforms to a tabular format with a relationship b\n different rows and columns.
 eg. Excel files or SQL databases. Each of these has structured rows and columns that can be sorted.
8
B. Unstructured Data: is information that either does not have a predefined data model or is not
organized in a predefined manner.
 It is typically text-heavy but may contain data such as dates, numbers, and facts.
 This results in irregularities and ambiguities that make it difficult to understand using traditional
programs as compared to data stored in structured databases.
 eg. Audio, Video files, or NoSQL databases.
C. Semi-structured(self-describing structure): is a form of structured data that does not conform
to the formal structure of data models associated with relational databases or other forms of data
tables, contains tags or other markers to separate semantic elements, and enforces hierarchies of
records and fields within the data.
 eg. JSON and XML are forms of semi-structured data.

9
D. Metadata(Data about Data): From a technical point of view, this is not a separate data
structure, but it is considered structured data,
 most important elements for Big Data initial analysis and big data solutions.
 Metadata provides additional information about a specific set of data.
 In a set of photographs, e.g., metadata could describe when and where photos were taken.
 The metadata then provides fields for dates and locations which, by themselves.

10
 2.4. Data Value Chain
 It describes the information flow within a big data system.
 a series of steps needed to generate value and useful insights from data.
 The Big Data Value Chain identifies the following key high-level activities:

Data Value Chain 11

1. Data Acquisition (new prosperity): is a process of gathering, filtering, and cleaning data before
it is put in a data warehouse or storage solution.
 Infrastructure required to support the acquisition of big data must deliver low, predictable latency
in both capturing data and executing queries;
 able to handle very high transaction volumes, often in a distributed environment, and support
flexible and dynamic data structures.
2. Data Analysis: making raw data acquired amenable/question for use in decision-making as well
as domain-specific usage
 It involves exploring, transforming, & modeling data to highlight relevant data; synthesize
and extract useful hidden information with high potential from a business point of view.
 Related areas include data mining, business intelligence, and machine learning.
12
3. Data Curation (Scientific curators or data annotators): active management of data over its
life cycle to ensure it meets the necessary Data Quality requirements for its effective usage.
 It can be categorized into different activities such as content creation, selection, classification,
transformation, validation, and preservation.
 Performed by expert curators who are responsible for improving accessibility and quality of data.
 It holds the responsibility of ensuring data is trustworthy, discoverable, accessible, reusable, and
fits its purpose.
 A key trend for the duration of big data utilizes community and crowdsourcing approaches.
4. Data Storage: persistence and management of data in a scalable way that satisfies the needs of
applications that require fast access to the data.
 Relational Database Management Systems (RDBMS) have been the main, and almost unique,
solution to the storage paradigm for nearly 40 years.
13
 However, ACID (Atomicity, Consistency, Isolation, and Durability) properties that guarantee
database transactions lack flexibility about schema changes and the performance and fault
tolerance when data volumes and complexity grow, making them unsuitable for big data
scenarios
 NoSQL technologies have been designed with the scalability goal in mind and present a wide
range of solutions based on alternative data models.
5. Data Usage: covers the data-driven business activities that need access to data, its analysis,
and the tools needed to integrate the data analysis within the business activity.
 Data usage in business decision making can enhance competitiveness through the reduction of
costs, increased added value, or any other parameter that can be measured against existing
performance criteria.

14
 2.5. Basic concepts of big data
 Big data is a blanket term for the non-traditional strategies and technologies needed to gather,
organize, process, and gather insights from large datasets.
 Big data is the collection of datasets. So large and complex that it becomes difficult to process
using on-hand database management tools or traditional data processing applications.
 “Large dataset” means a dataset too large to reasonably process or store with traditional tooling or
on a single computer.
 This means that the common scale of big datasets is constantly shifting and may vary significantly
from organization to organization.
 Big data is characterized by V and more:
 Volume: large amounts of data, Zeta bytes/Massive datasets
 Velocity: Data is live streaming or in motion
 Variety: data comes in many different forms from diverse sources
 Veracity: Can we trust the data? How accurate is it? etc.
15
Characteristics of big data

16
2.5.1. Clustered Computing and Hadoop Ecosystem

1. Clustered Computing
 Qualities of big data, individual computers are often inadequate for handling the data at
most stages.
 To better address the high storage and computational needs of big data, computer clusters are
a better fit.
 Big data clustering software combines the resources of many smaller machines, seeking to
provide several benefits:
A. Resource Pooling: Combining the available storage space to hold data is a clear benefit, but
CPU & memory pooling are also extremely important.
B. High Availability: Clusters can provide varying levels of fault tolerance and availability
guarantees to prevent hardware or software failures from affecting access to data and processing.
 emphasize the importance of real-time analytics.
17
C. Easy Scalability: easy to scale horizontally by adding additional machines to the
group.
 This means the system can react to changes in resource requirements without expanding
the physical resources on a machine.
 Using clusters requires a solution for managing cluster membership, coordinating
resource sharing, and scheduling actual work on individual nodes.
 Cluster membership and resource allocation can be handled by software like Hadoop’s
YARN (Yet Another Resource Negotiator).
 The assembled computing cluster often acts as a foundation that other software
interfaces with to process the data.
 The machines involved in the computing cluster are also typically involved with the
18
2. Hadoop and its Ecosystem
 Hadoop is an open-source framework intended to make interaction with big data easier.
 It is a framework that allows for the distributed processing of large datasets across clusters of
computers using simple programming models.
 It is inspired by a technical document published by Google.
 The four key characteristics of Hadoop are:
 Economical: systems are highly economical as ordinary computers can be used for data processing.
 Reliable: is reliable as it stores copies of data on different machines & resistant to hardware failure.
 Scalable: easily scalable both horizontally and vertically. A few extra nodes help in scaling up the
framework
 Flexible: you can store as much structured and unstructured data as you want to and decide to use it
later.
19
 Hadoop ecosystem: evolved from four core components:
 Data management, access, processing, and storage.
 It is continuously growing to meet the needs of Big Data.
 It comprises the following components/tools/processing framework and many others:
 HDFS: Hadoop Distributed File System
 YARN: Yet Another Resource Negotiator
 MapReduce: Programming based Data Processing
 Spark: In-Memory data processing
 PIG, HIVE: Query-based processing of data services
 HBase: NoSQL Database
 Mahout, Spark MLLib: Machine Learning algorithm libraries
 Solar, Lucene: Searching and Indexing
 Zookeeper: Managing cluster
 Oozie: Job Scheduling
20
Hadoop Ecosystem
21
2.5.2. Big Data Life Cycle with Hadoop
Stage(step) of Big Data processing
1. Ingesting data into the system: data is ingested or transferred to Hadoop from various sources
(relational databases, systems, or local files).
 Sqoop transfers data from RDBMS to HDFS.
2. Processing the data in storage: data is processed and stored in the distributed file system, HDFS,
and the NoSQL distributed data, HBase, Spark, and MapReduce perform data processing.
3. Computing and analyzing data: Data is analyzed by processing frameworks like Pig, Hive, Impala
 Pig converts the data using a map and reduces it, then analyzes it.
 Hive is also based on the map and reduced programming and is most suitable for structured data.
4. Visualizing the results: Access, which is performed by tools such as Hue and Cloudera Search.
 In this stage, the analyzed data can be accessed by users.
22
 u f o r
n k y o
Th a ti on !
atte n
you r

???
ti on
e s
Qu ?

23
 Self Quiz
1. Define data science; what are the roles of a data scientist?
2. Discuss data and its types from computer programming and data analytics perspectives?
3. Discuss a series of steps needed to generate value and useful insights from data?
4. What is the principal goal of data science?
5. List out and discuss the characteristics of Big Data?
6. How we ingest streaming data into Hadoop Cluster?

24