Scribd
Upload
15 views3 pages

Exiting PySpark Shell Guide

Uploaded by

black hello
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
15 views3 pages

Exiting PySpark Shell Guide

Uploaded by

black hello
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

BMCS2013 DATA ENGINEERING 1 of 3

PRACTICAL 3.1 PySpark Shell

~~~ As the user hduser ~~~


💡 Start up HDFS and YARN.

~~~ As the user student ~~~


1. Add Spark-related variables to student’s profile
1.1. Open ~/.profile
student@PC25:~$ nano ~/.profile
and add the following environment variables to the end of the file:
export SPARK_HOME=/home/hduser/spark
export PATH=$PATH:$SPARK_HOME/bin

1.2. Source the ~/.profile file.


student@PC25:~$ source ~/.profile

2. Using the PySpark Interactive Shell: Word count example

2.1. Launch the PySpark interactive shell


$ pyspark
BMCS2013 DATA ENGINEERING 2 of 3

2.2. Create an RDD with data from a text file:


Note: The [Link] was already downloaded
in Practical 2.
>>> text = [Link]("[Link]")
>>> print(text)

2.3. Transform the RDD to implement the word count application using
Spark:
>>> from operator import add
>>> def tokenize(text):
... return [Link]()
...
>>> words = [Link](tokenize)

2.4. Apply the map:


>>> wc = [Link](lambda x: (x, 1))
>>> print([Link]())

2.5. Apply the reduceByKey action to obtain the word counts and save the
results in a text file:
>>> counts = [Link](add)
>>> [Link]("wc")
Note: To explicitly indicate the HDFS path, use
"hdfs://localhost:9000/user/student/wc" as the
output file name:

2.6. Exit the PySpark interactive shell


>>> exit()

2.7. Check the HDFS current working directory for the newly created
directory’s contents:
student@PC25:~$ hdfs dfs -ls wc
BMCS2013 DATA ENGINEERING 3 of 3

2.8. Use the head command to view one of the part files:
$ hdfs dfs -head wc/part-00000

〰〰〰〰〰〰 End of Practical 〰〰〰〰〰〰

💡 If no more activity using Spark or Hadoop, remember to stop the YARN service
followed by the HDFS service.

Common questions

Powered by AI

The structure of RDD transformations and actions contributes to data processing efficiency by enabling parallel computation and minimizing data movement. Transformations like `flatMap` and `map` are lazy, allowing Spark to perform optimizations and compute a logical plan before execution. Actions like `reduceByKey` trigger the actual computation, leveraging distributed processing power to aggregate results efficiently. This architecture maximizes in-memory processing and reduces I/O operations, significantly enhancing performance for large-scale data operations .

The environment setup for the 'hduser' involves starting up HDFS and YARN services, which are essential for enabling Hadoop and Spark operations. In contrast, the 'student' user focuses on configuring the local environment for PySpark usage by adding Spark-related variables to their profile. This includes defining `SPARK_HOME` and updating the `PATH` to include the PySpark bin directory, then sourcing the profile to activate these changes .

The process utilizes PySpark by first launching the PySpark interactive shell and creating an RDD from a text file using `sc.textFile()`. It then splits the text into words using the `flatMap()` transformation with a tokenizing function. After that, it maps each word to a tuple of (word, 1) using the `map()` function. Finally, it applies the `reduceByKey()` action to sum the counts of each word and saves the result to a text file in HDFS .

Sourcing the profile is important because it applies the changes made to the environment variables immediately. Without sourcing the profile, the new variables defining the PySpark setup (`SPARK_HOME` and updated `PATH`) would not be recognized in the current session, which is necessary for executing PySpark commands correctly .

Apache Spark offers several advantages for executing the word count task in a distributed environment. It allows for efficient data processing through in-memory computation, which minimizes disk I/O. Spark's RDD transformations and actions, such as `flatMap`, `map`, and `reduceByKey`, provide a straightforward way to process large-scale data in parallel across a cluster. Additionally, Spark's integration with HDFS facilitates distributed data storage and retrieval, enhancing scalability and fault tolerance .

The `reduceByKey` action in the PySpark example aggregates the word counts by reducing the data for each key using a specified associative function, `add`, which sums the values. It is significant because it efficiently groups the data by key (i.e., words) and processes this aggregation in parallel across partitions, allowing for scalable and efficient computation of word frequencies .

Data transformation using RDD operations in PySpark begins by creating an RDD from the text file using `sc.textFile()`. The `flatMap()` operation is then applied to split the text into individual words through a tokenizing function that splits lines into words. Each word is mapped to a key-value pair (word, 1) using the `map()` function. Finally, `reduceByKey()` sums the values of identical keys, effectively counting the occurrences of each word. The results are saved to a text file using `saveAsTextFile()` .

Before executing a PySpark session, you need to add Spark-related environment variables to the user's profile. This includes setting the SPARK_HOME variable and updating the PATH. Specifically, you open the ~/.profile file and add: `export SPARK_HOME=/home/hduser/spark` and `export PATH=$PATH:$SPARK_HOME/bin`. After editing, source the profile to apply the changes .

To verify the successful creation of processed output files, one can check the contents of the HDFS directory where the output is stored using the command `hdfs dfs -ls wc`. Additionally, viewing the contents of the output files using a command like `hdfs dfs -head wc/part-00000` allows for confirmation that the word count task was successfully executed and the outputs were correctly saved .

Properly shutting down services like YARN and HDFS is crucial to ensure resource allocation and system stability in a distributed computing environment. Leaving these services running can lead to unnecessary resource consumption, potential data corruption, and increased load on the system. It prevents potential conflicts during subsequent startups and contributes to resource optimization, ensuring the Hadoop ecosystem remains efficient and reliable .

You might also like