Scribd
Upload
44 views2 pages

Spark DataFrame and Dataset Examples

The document discusses Spark DataFrames and Datasets using Scala. It shows how to create DataFrames from data, rename columns, sort data and describe datasets. It also demonstrates creating datasets from JSON and Parquet files and reading a CSV file.

Uploaded by

Karthikeyan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
44 views2 pages

Spark DataFrame and Dataset Examples

The document discusses Spark DataFrames and Datasets using Scala. It shows how to create DataFrames from data, rename columns, sort data and describe datasets. It also demonstrates creating datasets from JSON and Parquet files and reading a CSV file.

Uploaded by

Karthikeyan
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd

1) DATAFRAME:

import [Link]
val sparkSession = [Link]("local").appName("Spark session in
Fresco").getOrCreate()
val langPercentDF = [Link](List(("Scala", 35), ("Python", 30), ("R",
15), ("Java", 20)))
[Link]()
val lpDF = [Link]("_1",
"language").withColumnRenamed("_2", "percent")
[Link](desc("percent")).show(false)

2) DATASET:
import [Link]
val spark = [Link]("local").appName("Spark session in
Fresco").getOrCreate()
val numDS = [Link](5, 100, 5)
[Link]()
[Link](desc("id")).show(5)
[Link]().show()

3) CREATE DATSET by JSON


{"name":"Rahul","age":"35"}
{"name":"Sachin","age":"46"}
import [Link]
val spark = [Link]("local").appName("Spark session in
Fresco").getOrCreate()
val peopleDS = [Link]("/projects/[Link]")
[Link]()
case class Person (name: String, age: String)
val personDS = [Link]("/projects/[Link]").as[Person]
[Link]()

{"name":"Rahul","age":"35"}
{"name":"Sachin","age":"46"}
import [Link]
val spark = [Link]("local").appName("Spark session in
Fresco").getOrCreate()
val peopleDS = [Link]("/projects/[Link]")
[Link]()
case class Person(name:String,age:String)
object Main
{
def main(args: Array[String])
{
var Person1 = Person("35", "Rahul")
var Person2 = Person("46", "Sachin")
println("Age of the Person1 is " + [Link]);
println("Name of the Person1 is " + [Link]);
println("Age of the Person2 is " + [Link]);
println("Name of the Person2 is " + [Link]);
}
}

4) PARQUET
{"name":"Rahul","age":"35"}
{"name":"Sachin","age":"46"}
import [Link]
val spark = [Link]("local").appName("Spark session in
Fresco").getOrCreate()
val peopleDS = [Link]("/projects/[Link]")
[Link]()
val peoplePAR = [Link]("/projects/challenge/[Link]")
val sqlContext = new [Link](sc)
val data = [Link]("/projects/challenge/[Link]")
[Link]()

5) CSV Files
git clone [Link]
import [Link]
val spark = [Link]().master("local[1]").appName("Spark Session in
Frescoplay").getOrCreate()
val dfs = [Link]("csv").option("header",
"true").option("Inferschema","true").option("mode",
"DROPMALFORMED").load("/projects/challenge/Census/[Link]")
val joined = [Link](TotalPopulation, "Total Population")

You might also like