MODULE 4
Apache Pig
Introduction to Pig
Apache Pig is a high-level data processing tool built on top of Hadoop.
It uses a scripting language called Pig Latin to process and analyse large datasets.
Pig scripts are converted into MapReduce jobs, which run on Hadoop clusters.
Main Features:
Handles both structured and semi-structured data.
Simplifies complex MapReduce programming.
Supports User Defined Functions (UDFs) for custom logic.
2. Execution Modes of Pig
Mode Description Command
Runs on a single machine (for small
Local Mode Pig -x local
data/testing).
MapReduce Mode (Hadoop pig -x
Runs on a Hadoop cluster using HDFS.
Mode) mapreduce
Example:
pig -x mapreduce [Link]
Comparison of Pig with Databases
Feature Pig RDBMS
Language Pig Latin SQL
Processing Data flow (procedural) Query-based (declarative)
Schema Optional / dynamic Strict and predefined
Execution Converts into MapReduce Runs on a database engine
Use Case ETL((Extract, Transform, Load) Transactional operations
4. Grunt (Pig Shell)
Grunt is Pig’s interactive shell for running Pig Latin commands.
Used for testing, debugging, and executing scripts interactively.
Common Commands:
grunt> ls /data
grunt> cat /data/[Link]
grunt> describe relation_name
grunt> dump relation_name
5. Pig Latin
Pig Latin is a data flow language for defining data transformations.
Each step creates a relation (similar to a table).
Example Pig Script:
data = LOAD '/data/[Link]' USING PigStorage(',') AS (id:int, name:chararray,
marks:int);
high = FILTER data BY marks > 75;
grouped = GROUP high BY name;
result = FOREACH grouped GENERATE group, COUNT(high);
DUMP result;
Key Steps: Load → Transform → Group → Filter → Store
6. User Defined Functions (UDFs)
Custom functions written in Java, Python, or other languages.
Used for tasks not covered by built-in functions.
Example:
public class ToUpper extends EvalFunc<String> {
public String exec(Tuple input) {
return [Link](0).toString().toUpperCase();
}
}
Register and use:
REGISTER [Link];
upper_data = FOREACH data GENERATE ToUpper(name);
7. Data Processing Operators in Pig
Operator Function
LOAD Load data from file system
STORE Save output to file
FILTER Filter records based on condition
FOREACH...GENERATE Transform each record
GROUP Group data by key
JOIN Combine two or more datasets
Operator Function
ORDER BY Sort data
DISTINCT Remove duplicates
DUMP Display output to screen
Apache Hive
Introduction
Apache Hive is a data warehouse tool built on top of Hadoop.
It allows users to query and analyze large datasets stored in HDFS using HiveQL
(similar to SQL).
Originally developed by Facebook.
1. Hive Shell
Command-line interface for interacting with Hive.
Run using:
hive
Supports SQL-like commands for table creation, data loading, and querying.
2. Hive Services
Service Description
CLI (Command Line Interface) Interactive shell for Hive commands
HiveServer2 Provides JDBC/ODBC connectivity for external apps
WebHCat Web-based interface for Hive
Driver Manages query compilation and execution
Metastore Stores metadata (tables, schemas, locations)
3. Hive Metastore
Stores metadata about tables, partitions, schemas, and data locations.
Can be local (embedded Derby) or remote (MySQL/PostgreSQL).
Essential for query optimization and management.
4. Comparison with Traditional Databases
Feature Hive RDBMS
Query Language HiveQL (SQL-like) SQL
Data Storage HDFS Local disk
Feature Hive RDBMS
Schema Schema on Read Schema on Write
Use Case Analytical queries OLTP transactions
Execution MapReduce jobs Relational engine
Speed Batch-oriented (slower) Real-time (faster)
OLTP :- OnLine Transaction Preocessing
5. HiveQL (Hive Query Language)
Example:
CREATE TABLE students (id INT, name STRING, marks INT)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
LOAD DATA INPATH '/data/[Link]' INTO TABLE students;
SELECT name, AVG(marks) FROM students GROUP BY name;
Operations Supported:
DDL (CREATE, DROP, ALTER)
DML (LOAD, INSERT, SELECT)
Aggregations and Joins
6. Tables in Hive
Type Description
Managed Table Hive controls both data and metadata. Dropping table deletes data.
External Table Data resides externally; Hive manages only metadata.
Partitioned Table Data split by column values (e.g., date, region).
Bucketed Table Data divided into buckets (hash-based).
7. Querying Data in Hive
Example Queries:
SELECT * FROM sales WHERE region='Asia';
SELECT region, SUM(revenue) FROM sales GROUP BY region;
Joins:
SELECT [Link], [Link] FROM employees a JOIN salaries b ON [Link]=[Link];
8. User Defined Functions (UDFs) in Hive
Used for custom logic not supported by HiveQL.
Types:
o UDF: One input → one output.
o UDAF: Aggregate functions.
o UDTF: Returns multiple rows.
Example (Register):
ADD JAR [Link];
CREATE TEMPORARY FUNCTION to_upper AS '[Link]';
SELECT to_upper(name) FROM employees;
3. Apache HBase
Introduction
Apache HBase is a NoSQL database that runs on top of HDFS.
It provides random, real-time read/write access to large datasets.
Inspired by Google’s BigTable.
1. HBase Concepts
Concept Description
Table Collection of rows, similar to an RDBMS table.
Row Key Unique identifier for each row (sorted lexicographically).
Column Family Logical grouping of columns; defined at table creation.
Column Qualifier Specific column inside a family (dynamic schema).
Timestamp Versioning mechanism; multiple versions per cell.
Structure Example:
RowKey | cf1:name | cf1:age | cf2:marks
2. HBase Architecture
HMaster: Manages schema changes and load balancing.
RegionServer: Stores subsets of tables (regions).
Zookeeper: Coordinates distributed operations.
Data Flow:
Client → Zookeeper → HMaster → RegionServer → HDFS
3. HBase Clients
Java API
HBase Shell (interactive command-line tool)
REST and Thrift APIs
Common Shell Commands:
create 'student', 'info'
put 'student', '1', 'info:name', 'Alice'
get 'student', '1'
scan 'student'
4. Example
hbase(main):001:0> create 'employee', 'personal', 'professional'
hbase(main):002:0> put 'employee', '1', 'personal:name', 'John'
hbase(main):003:0> put 'employee', '1', 'professional:dept', 'HR'
hbase(main):004:0> scan 'employee'
Output:
ROW COLUMN+CELL
1 column=personal:name, value=John
1 column=professional:dept, value=HR
5. HBase vs RDBMS
Feature HBase RDBMS
Data Model Column-oriented Row-oriented
Schema Flexible (dynamic) Fixed
Query Language NoSQL (API-based) SQL
Transactions Limited Full ACID
Use Case Real-time analytics, random access OLTP
Scalability Horizontally scalable Vertically scalable
Summary
Component Main Purpose Type of System
Pig Data transformation using Pig Latin Data Flow Language
Hive SQL-like querying on HDFS Data Warehouse
HBase Real-time read/write access NoSQL Column Store