Scribd
Upload
5 views61 pages

4 Spark Databricks Installation Tutorial

The document is an installation tutorial for Databricks, an online platform for data engineers that utilizes Spark for big data processing. It outlines the steps to sign up for the free Community Edition, create a cluster, upload datasets, and run Spark applications. Additionally, it mentions the Enterprise Edition for more advanced configurations and pricing details based on usage.

Uploaded by

Akshay Panchal
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
5 views61 pages

4 Spark Databricks Installation Tutorial

The document is an installation tutorial for Databricks, an online platform for data engineers that utilizes Spark for big data processing. It outlines the steps to sign up for the free Community Edition, create a cluster, upload datasets, and run Spark applications. Additionally, it mentions the Enterprise Edition for more advanced configurations and pricing details based on usage.

Uploaded by

Akshay Panchal
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Distributed Data

Management
Databricks:
Installation Tutorial

Dr. Ignacio Castineiras


Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● Databricks is a spin-off company leveraging Spark as a
cloud-based big data processing tool [Link]
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● The Databricks Community Edition is free to use.
It will be the approach we will follow.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● The Databricks Community Edition is free to use.
It will be the approach we will follow.

● The Community Edition provides us with a local mode-based


Spark configuration (similar to using Spark in our own local machine).
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● The Databricks Community Edition is free to use.
It will be the approach we will follow.

● The Community Edition provides us with a local mode-based


Spark configuration (similar to using Spark in our own local machine).

● Specifically, it provides us with A Cluster of 1 Databricks Unit (DBU):


“A unit of processing capability per hour”

○ 1 physical machine with


○ 2 cores and
○ 15.3GB of memory storage.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● On the other hand, the Databricks Enterprise Edition allows us to rent a
cluster in a price-per-usage basis, and configure it to target the Business
Unit specific needs.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● On the other hand, the Databricks Enterprise Edition allows us to rent a
cluster in a price-per-usage basis, and configure it to target the Business
Unit specific needs.

● The cluster is indeed hosted by Amazon Web Services, Microsoft Azure or


Google Cloud, with Databricks providing the interface to automate the
cluster setup.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


● On the other hand, the Databricks Enterprise Edition allows us to rent a
cluster in a price-per-usage basis, and configure it to target the Business
Unit specific needs.

● The cluster is indeed hosted by Microsoft Azure or Amazon Web Services,


with Databricks providing the interface to automate the cluster setup.

● The range of prices depends on the functionality provided by Databricks,


and can be consulted at: [Link]
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers

How to...
Sign up / sign in.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


1. We register/sign up to Databricks Community Edition at:
[Link]
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Very important!

Do not click on any


of the above cloud
providers.

Click here instead


to use the
Databricks
Community
Edition for free.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


1. We receive an email to validate the account:
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


1. We can create the password of the account:
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


2. Once registered, we sign in at:
[Link]
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers

How to...
Create a new cluster.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


3. Our first step is to set up a new Community Edition cluster by clicking in
Compute on the panel on the left hand side of the page.
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

The cluster might take 1-5


minutes to be available
Dr. Ignacio Castineiras Department of Computer Science

The cluster might take 1-5


minutes to be available
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers

How to...
Upload a dataset to Databricks.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


4. Let’s download the Spark material from our module in Canvas
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


4. Let’s download the Spark material from our module in Canvas
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


4. Next, we can upload all our datasets to the Databricks Distributed File System
DBFS by clicking on Create => Table on the panel on the left hand side of the
page.
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

We type this by ourselves


Dr. Ignacio Castineiras Department of Computer Science

We type this by ourselves


Dr. Ignacio Castineiras Department of Computer Science

We drag and drop


the files here
Dr. Ignacio Castineiras Department of Computer Science

The files are


uploaded to DBFS
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers

How to...
Create a new program / Spark Application.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


4. Our next step is to create a new Spark application as a new notebook in our
Workspace.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


5. We can now create a Spark program, for it to be executed in the Databricks
cluster. For it, we click on Workspace on the panel on the left hand side of
the page.
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

Notebook

Right click at this


area
Dr. Ignacio Castineiras Department of Computer Science

We click here to rename the file


Dr. Ignacio Castineiras Department of Computer Science

We open a code example file with PyCharm, and copy its


content to the Databricks notebook just created (using
CTRL+A to select all the text, CTRL+C to copy it and CTRL+V
to paste it in Databricks.
Dr. Ignacio Castineiras Department of Computer Science

We open a code example file with PyCharm, and copy its


content to the Databricks notebook just created (using
CTRL+A to select all the text, CTRL+C to copy it and CTRL+V
to paste it in Databricks.
Dr. Ignacio Castineiras Department of Computer Science

We open a code example file with PyCharm, and copy its


content to the Databricks notebook just created (using
CTRL+A to select all the text, CTRL+C to copy it and CTRL+V
to paste it in Databricks.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers

How to...
Run a Spark Application.
Dr. Ignacio Castineiras Department of Computer Science

Databricks: An Online Platform for Data Engineers


6. We can finally run the Spark program.
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

If you cannot see this DBFS tab we need to enable it


Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science
Dr. Ignacio Castineiras Department of Computer Science

The dataset we uploaded is stored in


the folder my_dataset.
Dr. Ignacio Castineiras Department of Computer Science

The data analysis produced by the


Spark program is stored in the folder
my_result.
Thank you for your attention!

You might also like