BIG DATA ANALYTICS ( BCS714D )
Module 4
All Answer are from the prescribed VTU textbook for the above course
1 What is Hive? List the features of Hive.
HIVE:
Hive is a Data Warehousing tool that sits on top of Hadoop. Hive is used to process structured data in
Hadoop.
The three main tasks performed by Apache Hive are:
1. Summarization
2. Querying
3. Analysis
Facebook initially created the Hive component to manage their ever-growing volumes of log data. Later, the
Apache Software Foundation developed it as open-source, and it came to be known as Apache Hive.
Hive makes use of the following:
1. HDFS for storage
2. MapReduce for execution
3. RDBMS to store metadata and schemas
Hive provides HQL (Hive Query Language) or HiveQL, which is similar to SQL. Hive compiles SQL queries into
MapReduce jobs and then runs these jobs on the Hadoop cluster. It is designed to support OLAP (Online
Analytical Processing). Hive provides extensive data types, functions, and formats for data summarization
and analysis.
Hive Features
1. It is similar to SQL.
2. HQL is easy to code.
3. Hive supports rich data types such as structs, lists, and maps.
4. Hive supports SQL filters, GROUP BY, and ORDER BY clauses.
5. Custom types and custom functions can be defined.
2 Explain Hive File Formats
Hive File Formats
Hive file formats define how data (records) are stored and encoded in Hive tables. Choosing the right file
format improves performance, storage efficiency, and query speed.
1. Text File
• Default file format in Hive.
• Each record is stored as a single line in the file.
• Uses control characters as delimiters:
o ^A (octal 001) → separates fields
o ^B (octal 002) → separates elements in arrays or structs
o ^C (octal 003) → separates key–value pairs
o \n → separates records
• Commonly supported formats:
o CSV (Comma Separated Values)
o TSV (Tab Separated Values)
• JSON and XML documents can also be stored as text files.
• Easy to read but not very efficient for large-scale analytics.
2. Sequence File
• A binary file format that stores data as key–value pairs.
• Supports compression, which:
o Reduces storage size
o Lowers CPU and I/O overhead
• Faster than text files for processing large datasets.
• Suitable for MapReduce-based processing.
3. RCFile (Record Columnar File)
• A column-oriented file format designed for efficient querying and aggregation.
• Ensures that Aggregation operation is not an expensive operation.
• Combines advantages of row-store and column-store approaches.
• Data storage process:
1. Table is partitioned horizontally into multiple row groups.
2. Each row group is then partitioned vertically by columns.
• This structure allows Hive to:
o Read only required columns
o Improve aggregation performance
• Ideal for OLAP workloads where queries involve fewer columns.
3 Explain bucketing with an example.
Bucketing in Hive
Bucketing is a technique in Hive used to divide table data into fixed number of files (buckets) based on the
hash value of a column. It helps in efficient querying, sampling, and joins.
How Bucketing Works
• Data is distributed into buckets using:
hash(column) % number_of_buckets
• Each bucket is stored as a separate file.
• Unlike partitioning, number of buckets is fixed
Example: Create a bucketed table on student_id with 4 buckets:
CREATE TABLE student (
student_id INT,
name STRING,
marks INT
)
CLUSTERED BY (student_id) INTO 4 BUCKETS;
In this example, Hive calculates the hash value of student_id and places each record into one of the 4
buckets. Records with the same student_id will always be stored in the same bucket.
4 What is Pig? explain the key features of Pig.
WHAT IS PIG?
Apache Pig is a platform for data analysis. It is an alternative to MapReduce programming. Pig was
developed as a research project at Yahoo.
Key Features of Pig
1. It provides an engine for executing data flows (defines how data should flow). Pig processes data in
parallel on the Hadoop cluster.
2. It provides a language called Pig Latin to express data flows.
3. Pig Latin contains operators for traditional data operations such as JOIN, FILTER, SORT, etc.
4. It allows users to develop their own User Defined Functions (UDFs) for reading, processing, and
writing data.
5 With a neat diagram explain the anotomy of pig
THE ANATOMY OF PIG
The anatomy of Apache Pig explains how Pig Latin scripts are converted into MapReduce jobs and executed
on the Hadoop cluster.
Main Components of Pig
1. Data Flow Language (Pig Latin)
2. Interactive Shell (Grunt)
3. Pig Interpreter and Execution Engine
1. Pig Latin Script
The user writes data processing logic using Pig Latin, which is a high-level data flow language. It contains
operations like LOAD, FILTER, FOREACH, GENERATE, STORE, etc.
2. Pig Interpreter / Execution Engine
This is the core of Pig. It:
• Parses and validates Pig Latin statements
• Checks data types
• Optimizes the execution plan
• Converts Pig Latin into one or more MapReduce jobs
• Submits jobs to Hadoop and monitors progress
3. MapReduce Jobs on Hadoop
The generated MapReduce jobs are executed on the Hadoop cluster using HDFS for storage and
MapReduce for processing.
6 Explain any five relational operators of pig with an example for each.
RELATIONAL OPERATORS IN PIG
Relational operators in Apache Pig are used to process and transform data stored in relations.
1. FILTER
The FILTER operator is used to select tuples from a relation based on specified conditions.
Objective:
Find the tuples of those students where the GPA is greater than 4.0.
Input:
Student (rollno:int, name:chararray, gpa:float)
Act:
A = load 'pigdemo/[Link]' as (rollno:int, name:chararray, gpa:float);
B = filter A by gpa > 4.0;
DUMP B;
Output:
(1003,Smith,4.5)
(1004,Scott,4.2)
2. FOREACH
The FOREACH operator is used to perform data transformation based on columns.
Objective:
Display the name of all students in uppercase.
Input:
Student (rollno:int, name:chararray, gpa:float)
Act:
A = load 'pigdemo/[Link]' as (rollno:int, name:chararray, gpa:float);
B = foreach A generate UPPER(name);
DUMP B;
Output:
(JOHN)
(JACK)
(SMITH)
(SCOTT)
(JOSHI)
3. GROUP
The GROUP operator is used to group tuples based on a common field.
Objective:
Group students based on their GPA.
Input:
Student (rollno:int, name:chararray, gpa:float)
Act:
A = load 'pigdemo/[Link]' as (rollno:int, name:chararray, gpa:float);
B = GROUP A BY gpa;
DUMP B;
Output:
(4.0,{(1008,James,4.0),(1002,Jack,4.0)})
(4.5,{(1006,Alex,4.5),(1003,Smith,4.5)})
4. DISTINCT
The DISTINCT operator is used to remove duplicate tuples.
It works on the entire tuple, not on individual fields.
Objective:
Remove duplicate student records.
Input:
Student (rollno:int, name:chararray, gpa:float)
Act:
A = load 'pigdemo/[Link]' as (rollno:int, name:chararray, gpa:float);
B = DISTINCT A;
DUMP B;
Output:
(1001,John,3.0)
(1002,Jack,4.0)
(1003,Smith,4.5)
5. LIMIT
The LIMIT operator is used to restrict the number of output tuples.
Objective:
Display the first 3 tuples from the student relation.
Input:
Student (rollno:int, name:chararray, gpa:float)
Act:
A = load 'pigdemo/[Link]' as (rollno:int, name:chararray, gpa:float);
B = LIMIT A 3;
DUMP B;
Output:
(1001,John,3.0)
(1002,Jack,4.0)
(1003,Smith,4.5)
Extra Imp Ques
E1 Explain Hive Architecture And Hive data units
Hive Architecture:
Main Components of Hive Architecture
1. User Interface (UI)
Hive provides different interfaces for users to submit queries:
• Command Line Interface (CLI)
• Hive Web Interface
• Hive Server (Thrift) for remote client access
Users write queries using HQL (Hive Query Language) through these interfaces.
2. Driver (Query Compiler and Executor)
The Driver manages the entire lifecycle of a Hive query.
• Receives queries from the user
• Compiles and parses HQL
• Creates an execution plan
• Coordinates execution and returns results
3. Metastore
The Metastore stores metadata information about Hive tables.
It contains:
• Table names
• Column names and data types
• Table location in HDFS
• Partition and bucket information
Metadata is stored in an RDBMS such as MySQL or Derby.
4. Execution Engine
The execution engine converts the optimized query plan into:
• MapReduce jobs
These jobs are submitted to Hadoop for execution.
5. Hadoop Components
JobTracker
• Manages and schedules MapReduce jobs
TaskTracker
• Executes tasks assigned by JobTracker
HDFS (Hadoop Distributed File System)
• Stores actual data of Hive tables
HIVE DATA UNITS:
2 Explain Pig Philosophy
3 Partitioning V/s Bucketing
PARTITIONING VS BUCKETING IN HIVE
Partitioning and Bucketing are techniques used in Apache Hive to improve query performance and data
management.
PARTITIONING
Partitioning divides a table into sub-directories based on the values of a column.
• Data is stored in separate folders in HDFS
• Number of partitions is dynamic
• Best for filtering queries (WHERE clause)
• Reduces the amount of data scanned
Example:
Partitioning a table by year creates folders like:
year=2023, year=2024, year=2025
BUCKETING
Bucketing divides a table into a fixed number of files (buckets) using a hash function on a column.
• Data is stored in bucket files, not folders
• Number of buckets is fixed
• Best for joins and sampling
• Uses: hash(column) % number_of_buckets
DIFFERENCE BETWEEN PARTITIONING AND BUCKETING
Partitioning Bucketing
Divides data into directories Divides data into files
Based on column values Based on hash function
Number of partitions is dynamic Number of buckets is fixed
Improves query filtering Improves joins and sampling
Creates many HDFS folders Creates fixed number of bucket files