Apache Spark
“In “God we trust, all others must bring data.”
-W. Edwards Deming
[Link]
Dataset :
WHAT IS IN IT ? LET'S SEE !!
VENUS R. PATEL
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("FriendsByAge")
#sc = SparkContext(conf = conf)
def parseLine(line):
fields = [Link](',')
age = int(fields[2])
numFriends = int(fields[3])
Code : return (age, numFriends)
lines = [Link]("/FileStore/tables/[Link]")
rdd = [Link](parseLine)
totalsByAge = [Link](lambda x: (x, 1)).reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
averagesByAge = [Link](lambda x: x[0] / x[1])
results = [Link]()
for result in results:
print(result)
VENUS R. PATEL
Dataset :
VENUS R. PATEL
01 02 03
Spark redeceBykey(): GroupBykey(): SortBykey():
Functions: combine values
with same keys
group values
with the same
keys
sort RDD by key
values.
VENUS R. PATEL
Setting up the spark configurations:
from pyspark import SparkConf, SparkContext
conf = SparkConf().setMaster("local").setAppName("FriendsByAge")
sc = SparkContext(conf = conf)
VENUS R. PATEL
Code:
• lines = [Link]("[Link]
• rdd = [Link](parseLine)
VENUS R. PATEL
Code:
def parseLine(line):
fields = [Link](',')
age = int(fields[2])
numFriends = int(fields[3])
return (age, numFriends)
VENUS R. PATEL
Output:
• 33,385
33,2
55,221
40,465
……….
VENUS R. PATEL
Code:
• totalsByAge = [Link](lambda x: (x, 1)).reduceByKey(lambda x,
y: (x[0] + y[0], x[1] + y[1]))
Line-1:
[Link](lambda x: (x, 1)).
VENUS R. PATEL
[Link](lambda x: (x, 1))
• (33,385) => (33,(385,1))
• (33,2) => (33,(2,1))
• (55,221) => (55,(221,1))
VENUS R. PATEL
reduceByKey(lambda x, y: (x[0] + y[0], x[1] + y[1]))
• Adds up all values for each unique key!
• (33,(387,2))
VENUS R. PATEL
Code:
• averagesByAge = [Link](lambda x: x[0] / x[1])
• (33,(387,2))=> (33,193.55)
VENUS R. PATEL
Code:
results = [Link]()
for result in results:
print(result)
VENUS R. PATEL
Apache Spark :
• Nothing happens until we call action “reducebykey” in above
program.
• Collect() is another action.
VENUS R. PATEL
Computing
an Average:
(scala)
VENUS R. PATEL
var rdd = [Link](array(1.0,2,3,4,5,6,7),3);
Computing var rdd_count = [Link]((_,1))
an Average: var(sum,count)=
(scala) rdd_count.reduce((x,y)=>x._1+y._1,x._2+y._2)
var avg = sum/count
VENUS R. PATEL
Q-1
• The metadata is stored on
A. Datanode
B. Namenode
VENUS R. PATEL
Q-2
• Which of the following is not a metadata of file?
A. Name of the file
B. Folder name of the file
C. Permission attributes of the file
D. The contents of the file
VENUS R. PATEL
Q-3
• Does datanode know the name and the parent folder name of the
file?
A. Yes, of course
B. No, because name and foldername is the metadata
VENUS R. PATEL
Q-4
• Which of the following is not true about
replication in HDFS:
A. The default replication factor of is 3
B. The default replication factor can be specified in settings
C. We can change the default replication factor per file
D. The HDFS automatically decides the replication factor based on the demand of the file
VENUS R. PATEL
Q-5
• What is not true about Apache spark?
a) It is fast large scale engine for data processing
b) Spark is faster than Apache Hadoop
c) It provides fast in-memory NoSQL datastore
d) It also has map-reduce paradigm
e) None
VENUS R. PATEL
Q-6
• If you need to process data continuously, which library you would use:
A. MLlib
B. GraphX
C. Spark Streaming
D. SparkR
VENUS R. PATEL