Scribd
Upload
20 views3 pages

GroupByKey vs ReduceByKey in Spark

reduceByKey works faster than groupByKey on larger datasets because reduceByKey can combine output with common keys within each partition before shuffling data between partitions, while groupByKey shuffles all key-value pairs across the network. It is recommended to use reduceByKey instead of groupByKey whenever possible to avoid unnecessary network shuffling of data.

Uploaded by

surendra yandra
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
20 views3 pages

GroupByKey vs ReduceByKey in Spark

reduceByKey works faster than groupByKey on larger datasets because reduceByKey can combine output with common keys within each partition before shuffling data between partitions, while groupByKey shuffles all key-value pairs across the network. It is recommended to use reduceByKey instead of groupByKey whenever possible to avoid unnecessary network shuffling of data.

Uploaded by

surendra yandra
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

groupByKey VS reduceByKey

scala> var data =


List("spark","scala","spark","spark","spark","scala","java","scala")

data: List[String] = List(spark, scala, spark, spark, spark, scala, java, scala)

scala> val mapData = [Link](data).map(x => (x,1))

mapData: [Link][(String, Int)] = MapPartitionsRDD[1] at map at


<console>:26

reduceByKey
scala> [Link](_+_).[Link](println)

(spark,4)

(scala,3)

(java,1)

groupByKey
scala> [Link]().map(x => (x._1 , x._2.sum) ).[Link](println)

(spark,4)

(scala,3)

(java,1)
In the above two transformations (reduceByKey , groupByKey) we are getting the same
Output...however

Avoid “groupByKey” where ever possible....the reason being

 reduceByKey works faster on larger datasets...i.e because Spark knows it


can combine output with a common key on each partition before shuffling
the data.
 On the other hand, when calling groupByKey - all the key-value pairs are
shuffled around. This is a lot of unnessary data to being transferred over
the network.
Partitions on RDD
scala> var data =
[Link](List(1,2,3,4,5,6,7,7,8.9,12,34,5,4,76,90,87,87,65,36),4)
data: [Link][Double] = ParallelCollectionRDD[6] at
parallelize at <console>:24

scala> [Link]
res3: Int = 4

scala> [Link]().collect
res4: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0, 4.0), Array(5.0,
6.0, 7.0, 7.0, 8.9), Array(12.0, 34.0, 5.0, 4.0, 76.0), Array(90.0, 87.0, 87.0,
65.0, 36.0))

scala> var data =


[Link](List(1,2,3,4,5,6,7,7,8.9,12,34,5,4,76,90,87,87,65,36),3)
data: [Link][Double] = ParallelCollectionRDD[8] at
parallelize at <console>:24
scala> [Link]
res5: Int = 3

scala> [Link]().collect
res6: Array[Array[Double]] = Array(Array(1.0, 2.0, 3.0, 4.0, 5.0, 6.0),
Array(7.0, 7.0, 8.9, 12.0, 34.0, 5.0), Array(4.0, 76.0, 90.0, 87.0, 87.0, 65.0,
36.0))

scala>

You might also like