MapReduce, Hive and Pig

Manjunath G.S.
Asst. Professor
Dept. of ISE, BNMIT
Learning Objectives
After studying this chapter, you will be able to:
 Understanding of MapReduce, map tasks using the key-value stores
 Knowledge of composing MapReduce programs for calculations
 Understanding of Hive, comparison of Hive data store with traditional DBMS
 Apply HiveQL for querying, storing, aggregating, querying scripts,
MapReduce joins and sub-queries
 Knowledge of pig architecture, developing scripts and extensibility using
UDFs
Designing Data Architecture (C1)
Introduction
 The data processing layer is the application support layer.

 Big data processing layer includes the APIs of program such as MapReduce
and Spark.
 This layer includes HBase which creates column-family data store.
 This layer includes Hive which creates a SQL-like tables.
 This layer includes Pig is a data-flow language and an execution framework.
Contd,…
Hadoop Ecosystem Components (C2)
Hadoop ecosystem refers to a combination of technologies.
MapReduce
 MapReduce is a Hadoop framework used for writing applications that can
process vast amounts of data on large clusters.
 It can also be called a programming model in which we can process large
datasets across computer clusters.
 This application allows data to be stored in a distributed form.
 It simplifies enormous volumes of data and large-scale computing.
 There are two primary tasks in MapReduce:
 map
 reduce
MapReduce
MapReduce programming model has broadly classified into three types

 Map-Tasks
 Reduce Tasks
 MapReduce Execution
MapReduce Contd,…
Example-1
 How can a car company quickly compute an aggregation function using the
number of cars of a specific car-model sold at the company showrooms as
input?

Example-2
 How does MapReduce enable query processing quickly in Big Data Platforms?
MapReduce process on client
submitting a job
MapReduce Contd,…
 MapReduce architecture consists of various components. A brief description of
these components can improve our understanding on how MapReduce works.
 Job
 Task
 Job Tracker
 Task Tracker
 Input Data
 Output Data
 Client
 Hadoop MapReduce Master
 Job-parts
MapReduce Contd,…
Phases of MapReduce
 Mapping Phase
 Shuffling Phase
 Reducer Phase
 Combiner Phase

Applications of Hadoop MapReduce

 E-commerce
 Social networks
 Entertainment
Map-Tasks
 Map task means a task that implements a map ( ), which runs user
application codes for each key-value pair (k1, v1).
 The output of the map ( ) would be zero or intermediate key-value pairs (k2,
v2).
 Reduce task refers to a task which takes the output v2 from the map as an
input and combines those data pieces into a smaller set of data using
combiner.
Map-Tasks Contd,…
The sample code for Mapper Class
public class SampleMapper extends Mapper<k1, v1, k2, v2>
{
void map(k1 key, v1 value, Context context) throws IoException,
InterruptedException
……………….
}
 Individual mapper do not communicate with each other
Key-Value Pair
 Each phase (Map phase and Reduce phase) of MapReduce has key-value pairs
as input and output.
Grouping by Key, Partitioning
 When a map task completes, shuffling and sorting will takes place.
 The partitions are semi-mappers in MapReduce.
 A partition process the output of map tasks before submitting it to the
reducer tasks.
 Partitioner is an optimization in MapReduce that allows local partitioning
before reduce-task phase.
 The main function of a Partitioner is to split the map output records with the
same key
Combiners
 The combiners are semi-reducers in MapReduce.
 Combiners use grouping by key for carrying out this function.
 The combiner works as follows:
 It does not have its own interface and it must implement the interface at
reduce ( ).
 It operates on each map output key. It must have the same input and output
key-value types as the Reducer class.
 It can produce the summary information from a large dataset.
MapReduce Processing Steps
 Execution involves the formatting (transforming) of data at each step.
Coping with Node Failures
 The primary way using which Hadoop achieves fault tolerance is through
restarting the tasks.
 If the job is in the reducing phase, then another TaskTracker will re-execute
all reduce tasks that were in progress on the failed TaskTracker.
 MapReduce does not use any task identities to communicate between nodes
or which reestablishes the communication with other task node.
 The failure and restart process are clean and reliable.
 The coping mechanism with distinct Node Failures:

a) Map TaskTracker failure b) Reduce TaskTracker failure c) Master

JobTracker failure
Composing MapReduce for Calculations
and Algorithms
Composing Map-Reduce for Calculation
The calculations for various operations compose are:
 Counting and Summing
 Sorting
 Finding Distinct Values (Unique)
 Collating
 Filtering or Parsing
 Distributed Tasks Execution
 Graph Processing using Iterative Message Passing
 Cross Correlation
Matrix-Vector Multiplication by
MapReduce
 Number of applications need multiplication of n*n matrix A with vector B of
dimension n.
Relational-Algebra Operations
 Selection: selection(WHERE clause in SQL) lets you apply a condition over
the data you have and only get the rows that satisfy the condition.
 Projection: In order to select some columns only we use the projection
operator. It’s analogous to SELECT in SQL.
 Union: We concatenate two tables vertically. Like UNION in SQL, but
the duplicate rows are removed implicitly.
 Intersection: Same as INTERSECT in SQL. It intersects two tables and selects
only the common rows.
Relational-Algebra Operations
 Difference: The rows that are in the first table but not in second are selected
for output.
 Natural Join: Merge two tables based on some common column. It represents
the INNER JOIN in SQL. But the condition is implicit on the column that is
common in both tables. The output will only contain rows for which the
values in the common column matches.
 Grouping and Aggregation: Group rows based on some set of columns and
apply some aggregation (sum, count, max, min, etc.) on some column of the
small groups that are formed.
Hive
Hive
Characteristics
 Has the capability to translate queries into MapReduce jobs.
 Supports web interfaces as well.
 Provides an SQL dialect (Hive Query Language), Results of HQL and the data
load in the tables which store at the Hadoop Cluster in HDFS.
Limitations
 Hive does not provide update, alter and deletion of records in database.
 Not developed for unstructured data.
 Not designed for real-time queries.
 Performs the partition always from the last column.
Hive Architecture
Hive Architecture Contd,…
Components of Hive Architecture are:
 Hive Server (Thrift)
 Hive CLI (Command Line Interface)
 Web Interface
 Metastore
 Hive Driver
Comparison with RDBMS
Hive is a DB system which defines databases and tables. Hive also analyzes
structured data in DB.
Characteristics Hive RDBMS
Record level queries No update and delete Insert, update, and delete
Transaction support No Yes
Latency Minutes or more In fractions of a second
Data size Petabytes Terabytes
Data per query Petabytes Gigabytes
Query language HiveQL SQL
Support JDBC/ODBC Limited Full
Hive Data Types and File Formats
Hive defines various primitive, complex string, date/time, collection data types
and file formats for handling and storing different data formats.
• TINYINT, SMALLINT, INT, BIGINT, FLOAT, DOUBLE, BOOLEAN, TIMESTAMP,
DATE, VARCHAR, CHAR, DECIMAL, UNION, NULL

Hive three collection data types

• STRUCT, MAP, ARRAY

File Formats
• Test file, Sequential file, RCFile, ORCFILE
Hive Integration and Workflow Steps
Hive integrates with the MapReduce and HDFS. The below figure shows the
dataflow sequences and workflow steps between Hive and Hadoop.
Contd,…
Steps:
 Execute query
 Get plan
 Get metadata
 Send metadata
 Send plan
 Execute plan
 Execute job
 Metadata operations
 Fetch result
 Send results (10 and 11)
Hive built-in Functions
Return Type Syntax
BIGINT round (double a)
floor (double a)
ceil (double a)
double rand ( ), rand (int seed)
string concate (string str1, string str2,……)
substr (string str, int start)
substr (string str, int start, int length)
upper (string str), ucase (string str)
lower (string str), lcase (string str)
trim (string str), ltrim (string str), rtrim (string str)
int year (string date), month (string date), day (string date)
HiveQL
 Hive Query Language (HiveQL or HQL) is used for querying large datasets
which resides in HDFS environment.
 HiveQL script commands enable data definition, data manipulation and
query processing.

 HiveQL Process Engine

 Execution Engine
HiveQL DDL
 HiveQL database commands for data definition for DBs and tables are
CREATE DATABASE, SHOW DATABASE, CREATE SCHEMA, CREATE TABLE.

CREATE [TEMPORARY] [EXTERNAL] TABLE [IF NOT EXISTS] [<database

name>] <table name> [(<column name> <data type> [COMMENT <column
comment>], …)]
[COMMENT <table comment>]
[ROW FORMAT <row format>]
[STORED AS <file format>]
HiveQL DML
 HiveQL database commands for data manipulation are USE <database
name>, DROP DATABASE, DROP SCHEMA, ALTER TABLE, DROP TABLE, and
LOAD DATA.

LOAD DATA [LOCAL] INPATH ‘<file path>’ [OVERWRITE] INTO TABLE <table
name> [PARTITION (partcol1 = val1, partcol2 = val2 . . . )].

 LOCAL is an identifier to specify the local path. It is optional. OVERWRITE is

optional to overwrite the data in the table.
HiveQL for Querying the Data
1. Partitioning
 Hive organizes tables into partitions.
 Here tables are divided based on the values of set of columns.

Advantages
 Distributes execution load horizontally
 Response time becomes faster

Disadvantages
 Overhead to NameNode due to large partitions
 Less responsive queries
 Large number of tasks leads to overhead on JVM
HiveQL for Querying the Data
2. Bucketing
 Tables or partitions can be sub-divided into buckets.
 Division is based on the hash of a column in the table.
 Buckets provide an extra structure to the data that can lead to more efficient
query processing.
 Buckets store as a file in the partition directory.
 Records kept in each bucket provide sorting ease and enable Map task joins.
 A bucket can also be used as a sample dataset.
HiveQL for Querying the Data
3. Views
 A program uses functions or objects.
 Views provide ease of programming.
 A view provisions the following:
 Saves the query and reduces the query complexity
 View does not store data like a table
 Hive executes the view and then the planner combines the information.
 Hides the complexity by dividing the query into smaller, more manageable pieces.
Aggregation
 Data aggregation is the process where raw data is gathered and presented in
a summarized format for statistical analysis.
 Aggregation is the actual functional part of the more widely used terms, data
analytics and business intelligence (BI).

Who uses data Aggregation?

 Anyone doing any form of analytics uses aggregation since the two go
together.
 Different industries have different interests and outcomes, but they all must
process, present and analyze data.
Joins
 Join clause combines columns of two or more tables, based on the relation
between them.
 Join clause combines and retrieves the records from multiple tables.
 Types of Joins in Hive
 Inner join in Hive
 Left Outer Join in Hive
 Right Outer Join in Hive
 Full Outer Join in Hive
Pig
Apache developed Pig, which:
 Is an abstraction over MapReduce.
 Is an execution framework for parallel processing.
 Reduces the complexities of writing a MapReduce program.
 Is a high-level dataflow language.
 Is mostly used in HDFS environment.
 Performs data manipulation operations at files at data nodes in Hadoop.
Applications of Apache Pig
Applications of Pig, are:
 Analyzing large data sets
 Executing tasks involving adhoc processing
 Processing large data sources such as web logs and streaming online data
 Data processing for search platforms.
 Processing time sensitive data loads, data extracts and analyzes quickly.
Features of Apache Pig
 Apache PIG helps programmers write complex data transformations using
scripts.
 Creates user defined functions to write custom functions which are not
available in Pig.
 Process any kind of data.
 Reduces the length of the codes using multi-query approach.
 Handles inconsistent schema in case of unstructured data as well.
 Performs ETL data in the required format to HDFS.
 Automatic optimization of tasks before execution.
 Programmers and developers can concentrate on whole operation.
Differences b/w Pig and MapReduce
Pig MapReduce
A dataflow language A data processing paradigm
High level language and flexible Low level language and rigid
Performing Join, filter, sorting or ordering Relatively difficult to perform Join, filter,
operations are quite simple sorting or ordering operations between
datasets
Programmer with a basic knowledge of SQL Complex Java implementations require
can work conveniently exposure to Java language
Uses multi-query approach, thereby reducing Require almost 20 times more the number of
the length of the codes significantly lines to perform the same task
No need for compilation for execution Long compilation process of jobs
Provides nested data types like tuples, bags No such data types
and maps
Differences b/w Pig and SQL
Pig SQL
Pig latin is a procedural language A declarative language
Schema is optional, stored data Schema is mandatory
without assigning a schema
Nested relational data model Flat relational data model
Provides limited opportunity for More opportunity for query
Query optimization optimization
Pig Architecture
Pig Latin Data Model
 Pig Latin supports primitive data types which are atomic or scalar data types.
 Atomic data types: int, float, long, double, char[], and byte[].
 Complex data types: tuple, bag, and map.

 A simple atomic value is known as a field.

 Tuple: Tuple is a record of an ordered set of fields.
 Bag: A bag is an unordered set of tuples.
 Map: A map is a set of key-value pairs.
Pig Latin and Developing Pig Latin
Scripts
 Pig Latin enables developing the scripts for data analysis.
 Several operators in Pig Latin help to develop their own functions for reading,
writing, and processing data.
 Statements in Pig Latin:
 Basic constructs to process the data
 Include schemas and expressions
 End with a semicolon
 LOAD statement reads the data from file system
 Keywords (LOAD, STORE, DUMP)
 Function names, relations and paths are case-sensitive
Order of processing Pig statements
Operators in Pig Latin
Arithmetic + - * / %
Operators
Used for Addition Subtraction Multiplication Division Remainder

Comparison == != < > <= >=

operators
Used for Equality Not equal Less than Greater than Less than or Greater than
equal or equal

Boolean AND OR NOT

operators
Used for Logical AND Logical OR Logical NOT
Apache Pig Execution
pig –x local
pig –x mapreduce or pig
pig –help
pig –version

Piggy bank
UDFs