Scribd
Upload
4 views9 pages

Spark Data Processing with Filters

The document outlines a Spark application workflow involving reading data from various sources such as employee, orders, payments, and customers. It describes filtering operations on these datasets, including filtering employees by ID and orders by status. Additionally, it mentions joining filtered datasets and writing the results back to storage.

Uploaded by

wobopim502
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
4 views9 pages

Spark Data Processing with Filters

The document outlines a Spark application workflow involving reading data from various sources such as employee, orders, payments, and customers. It describes filtering operations on these datasets, including filtering employees by ID and orders by status. Additionally, it mentions joining filtered datasets and writing the results back to storage.

Uploaded by

wobopim502
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Spark B - AWSS3

--
T -
⑭ yee-spark. read . (path of file
u esu
-

[Cole [
employee with Column (X)
-
.

↑ code
46) ⑱ C
Y↓
⑳Employee
filter an
a
I
partition
spark rat

&
⑭data-spark read Ext("content Ext's
. .

#RDD -X Not Dataframe

O
coudater spark Read
= . . CSU
("employee CSU'l
.

-Dataframe
Rip-tatiame
-Y -

employeepf

employee-filtered
employee-filtered
filter (id 2)
Player
=

-
&
-
Lazy Evaluation Python
Get

inta
Y i
C = a +b

coutC ;
Spark Application
-

E
spark-SparkSession builder getorherte
. .

employee-of-spark Read CU("employee cuis


E
.

T
.

employee-filtered employee &[Link] (id 2) =

-employee-fitored.
Count
<3 A

Giiiye
3
DAG
/

Trigger ↓

·
Employee fitered
Tob
, Stage
Task
orders
= Spark. read .
CV ("orders Cv"I
.

payments= 2park read


Cv("payments Csul)
.
.
.

customers
sparkread
=
.
CV ("Custome . u)
orders-filter-orders Litter (status "Delivered's
.
=

Join Payments-fitter-paymentsfilter (status "Succeed


Eorders-with-company-order-filter With Colum
-

Company name"! "Swiggy


customer-fite-astomers Filter/"Country =
USA
join[ pay-with-order-orders-with
.

company Join (
.

pay-with-order Write() Payments-filter


.

pay-with-astomo-pay-with-order Join .

Customer-filta)
pay-with-customer write()
-
#
odsfilter
2
orders
with Company

JOBO
Stag1

writer
JOBI

#bo

You might also like