Bismilla 786
Data is the new oil
90 % of the data today we have generated last two years.
2.5 billion of gb data generated every day.
Reasons
[Link] Technology
[Link] Internet usage(Facebook,twitter,gmail etc)
[Link] in storage cost
[Link] online shopping
[Link] online banking
Ex:
Data Can be categorized into 3 types
[Link] Data: This data comes from Rdbms systems
[Link] Data: This data comes from
audio,video,pictures etc
[Link] structured Data: This data comes from json data,xml
data,txt data
What is Big Data: Huge amount of data (structured or un-
structured and semi structured data) that cannot be resolved
using Rdbms is called Big Data.
Diff Rdbms tools used for data storage
[Link]
[Link] sql
3.DB2 etc
Drawbacks of Rdbms:
[Link] is one the main drawback.
[Link] structured data can be processed.
[Link] storing and processing TB's of data speed is low
4. IF UPDATES ARE MADE IN RDBMS LAST UPDATED VALUE WILL BE
STORED. AND PREVIOUS VALUES ARE NOT STORED.
Data Storage: To store the data in hdfs we need name node and
data node.
Data Process: To process the data we need job tracker and task
tracker.
Meta data: Data about data.
File system: to store files is called file system.
Deoman: back ground services which not have physical
apperance.
Namenode and Job tracker: single point of fail over.
streaming access pattern:
Block: HDFS store User data in to files. Internally, a file is split
into one or more blocks and these blocks are stored in a set of
DataNodes
Name Node: It is one of service running in Hadoop Cluster
It holds metadata details of files storing in Hadoop
The metadata details of namenode is known as fsimage
Secondary name node
It is one of service running in Hadoop Cluster
It will hold backup of namenode data
the metadata details in secondary name node is known as Editlog
Job Tracker
It is one of service running in Hadoop
It receives tasks it has to be performed from NameNode
It schedules Job
It will assign task to task tracker
TaskTracker
It is one of Service running in Hadoop
It receives task which has to be performed by it from job tracker
each datanode has one task tracker
DataNode
It is one of service running in Hadoop
It holds data inform of blocks
by default block size is maintained as 64 mb
Note: Write once and read any no of times but not try to change
the contents of file in hdfs.
Note: Namenode,secondary node,Jobtracker are called master
services or master nodes or deoman
Note: Datanode,task tracker are called slave nodes or deomans
Confifuration files in Hadoop
[Link]
Settings:
block size : by default block size in data node is maintained as 64
mb
replica 3
Trash
HeartBeat Interval
[Link]: It holds the name node ipaddress
Masters: It holds secondary name node ip address
Slaves: It holds ip address of all data nodes
Replica: Back of [Link] are three types of Replica
By DEfault REplica is 3
Over Replica: if no of replica is greater than 3
UNder Replica: if no of replica is less than 3
Missing Replica:
Scaling : It is mainly used to improve performance of system
Scaling is classified into two types
Horizontal scaling : Adding new data node to cluster is known as
Horizontal Scaling
Vertical Scaling: Increasing ram and hardisk o existing data node we say
it as vertical scaling
Different operations we have in Data Storage
HDFS WRITE : writing data into Hadoop Cluster
HDFS READ: Reading data from Hadoop Cluster
HDFS DELETE: deleting data from Hadoop cluster.
Heart Beat Signal : DataNodes send a heartbeat to the NameNode
every three seconds by default.
If the NameNode doesn’t receive any heartbeats for a specified time,
which is ten minutes by default, it assumes the DataNode is lost
Block Level Report: NameNode creates an internal structure of Block to
file level mapping, block to datanode mapping and datanode to RAC
mapping.
As part of the block report, DN's send what blocks are there, how many
are corrupted, how many are over/under replicated.
Based on this block report information, NameNode will build the above
mapping in fsimage. When clients request for a specificl file,
NN will look into the fsimage (mapping) and give the path/DN to the
client
Safemode: is a maintenance state of NameNode and its in read only
mode.
Hadoop dfsadmin –safemode leave
Hadoop dfsadmin –safemode enter
Hadoop dfsadmin –safemode get
ifconfig
Linux Commands:
Local file system commnads(linux commands)
~ : home directory
1. Present working directory: pwd
2. To know list of files in the current directory: ls
3. Long lasting files and directories: ls -l
if its start with - its a file
if its start with d its a directory
if its start with . its a hidden file
[Link] display hidden files and direcotiries ls -a
[Link] create a directory : mkdir directoryname
[Link] change the directory: cd shaik
7. to create more than one directory : mkdir -p durga/krish/natrag
[Link] change directory : cd durga/krish/natrag
[Link] go to home directory : cd ~
[Link] go to root directory : cd /
11. To go to previous directory : cd..
Vi editor:
:wq >> to save and exit
:q! >>exit with out save
Jar file is pending check once
Switch User:
Absoulte path and Relative path:
Absulute path: Giving Entire path(from root onwards)
Relative path: Adjust the location where you are.
Permission:
chmode check onece
HDFS COMMANDS : (Hadoop Distributed File system Commands)
[training@localhost ~]$ [Link]
Warning: $HADOOP_HOME is deprecated.
[training@localhost ~]$ [Link]
Warning: $HADOOP_HOME is deprecated.
Start with : hadoop fs
Note: To move one location to another: cp,mv(hdfs to hdfs)
copyFromlocal,moveFromLocal,put from one local file systems to hdfs.
Disck Usage:
3. copyToLocal(Copy a file or directory from HDFS to Local)
hadoop fs –copyToLocal /user/training/hdfs/file1 /home/training/Local
4. moveToLocal(Not yet implemented)
5. touchz ( can create n no: of empty files in HDFS)
hadoop fs –touchz /home/training/hdfs/file1
6. rm (Remove a file)
hadoop fs –rm /home/training/hdfs/file1
7. rmr (Can be used for removing a file or Directory recursively)
hadoop fs –rmr /home/training/hdfs/file
Hive Architecture:
Apache Derby: Store meta details about tables.
Probs: one user tables not visible to other user.
Default db: MySQL
Create a db: Create database <db-name>
To know dbs: show databases;
Delete db: DROP DATABASE database_name CASCADE;
Note: Whenever we create db an entry will be made to
user/warehouse
Hive: For processing/Analysis structured data.
Hive uses HQL which is subset of sql
Hive can store only structured data
Hive can process huge amount of data at high speed
Hive uses udf concept to create user defined functions
DisAdvantages:
hive will occupy space for null values
Updations are not posible(Delete or update rows)
Hive data types:
and string
Map: Key and value pairs
Array: Collection of Same type elements
Struct: Collection of Diff types of elements.
Tables in Hive are classified into two types
Managed Tables: In Managed Tables if we drop table table will
be deleted from Hive as well as from user warehouse
(/user/hive/warehouse)
Load data: Loading data can be from local file system or HDFS
hive> create database sample
>;
Creating table:
hive> create table employee(eno int,ename string,job string,salary dou
ble,comm float,deptno int)
> row format delimited
> fields terminated by ',';
Load data:
hive> load data local inpath '/home/training/Desktop/emp' into table e
mployee;
hdfs mode:
[training@localhost Desktop]$ hadoop fs -ls
/user/hive/warehouse/[Link]
Warning: $HADOOP_HOME is deprecated.
Found 1 items
drwxr-xr-x - training supergroup 0 2018-01-22 17:47 /user/hive/w
arehouse/[Link]/employee
[training@localhost Desktop]$ hadoop fs -cat
/user/hive/warehouse/[Link] /employee/emp
External Tables: In External tables if we drop table table will be
deleted from hive prompt only and it will be still available in
/user/hive/warehouse
External Tables are classified into two types
With Location
Without Location :
Here we create a directroy and create a table and point to that
directory
i.e Data loading and table creation is done in single step
hive > create external table ext_emp(eno int,ename string,job
string,salary double,comm float,deptno int)
> row format delimited
> fields terminated by ','
> location '/user/training/inpt_hive';
Partitions: Used for data segregation and to prevent the entire
table scan
Steps:
Create a temporary table
Load data into temporary table
create partition table
set [Link]=nonstrict
insert data into partition table by select data from temporary table
View:
Creating view: create view view-name as select stmt
sub view: u can create sub view from an existing view.
create view view_name as select column list from view_name
Bucketing: Optimization Technique
Used for join optimization and table sampling.
Here we used clustered by
Steps:
[Link] a temporary table
[Link] data into temp table
[Link] buckting table.
[Link] [Link]=true
[Link] data into buckting table from temp table.
hive> create table bemp(eno int,ename string,job string,salary
double,comm float,deptno int)
> clustered by (deptno)
> into 3 buckets
> row format delimited
> fields terminated by ',';
select * from bemp tablesample(70 percent);
select * from bemp tablesample (bucket 1 out of 3);
Index: Like a pointer to a particular column( store column address)
Why used: Peformance will be improved.
Index 2 types
[Link] map
[Link]
create index emp_index on table t1(eno) as 'COMPACT' with deferred
REBUILD;
Array ex:
Ramesh,1,maths$physics$chemistry,a
Suresh,2,bilogy$maths$physices,b
create table array_table(sname string,sid int,sub array<string>,grade
string)
> row format delimited
> fields terminated by ','
> collection items terminated by '$';
select sname,sid,sub[0],sub[1],sub[2],grade from array_table;
Map: [training@localhost Desktop]$ cat [Link]
ravi,123,salary#50000$comm#3000,10
suresh,456,salary#4000$comm#2000,20
hive> create table map_table(ename string,eid int,paydetails
map<string,double>,deptno int)
> row format delimited
> fields terminated by ','
> collection items terminated by '$'
> map keys terminated by '#';
Struct: [training@localhost Desktop]$ cat >[Link]
ravi,123,456$juhu$mumbai
suresh,345,789$electroniccity$banglore
hive> create table struct_table(ename string,eid int,address
struct<hno:int,street:string,city:string>,deptno int)
> row format delimited
> fields terminated by ','
> collection items terminated by '$';
PIG: It is second generation data analysis/Processing tool
It can process structured,unstructured,semistructured data
Pig is session oriented(Pig doesn't have warehouse or meta
store) like Gajini
Built on top of Map Reduce.
PigEngine it converts pig scripts to mapreduce jobs
Pig can work in two modes
install linux then install pig
Pig -x Mapreduce
install [Link],pig
local modes: Here we have input file and output file in linux file
system here we can install pig without hadoop mapreduce
input and output files are in HDFS filesystem we need hadoop
and then only pig
Pig Data types:
int,float,long,double
bytearray,chararray,bag,tuple and atom
Note: Default datatype bytearray.
OR define with out headers(Default data type will be taken)
Note: To see the out put or select stmt is DUMP
Filtering the bag(Row):
Note: For every command you have to give bag name.