Console
Start free

Configure Dataproc Hub

Dataproc Hub is a customized JupyterHubserver. Admins configure and create Dataproc Hub instances that can spawn single-user Managed Service for Apache Spark clusters to host Jupyter and JupyterLab notebook environments (see Use Dataproc Hub).

Launch Notebooks for multiple users. You can create a Managed Service for Apache Spark-enabled Vertex AI Workbench instance or install the Managed Service for Apache Spark JupyterLab plugin on a VM to to serve notebooks to multiple users.

Objectives

  1. Define a Managed Service for Apache Spark cluster configuration (or use one of the predefined config files).

  2. Set Dataproc Hub instance environment variables.

  3. Create a Dataproc Hub instance.

Before you begin

If you haven't already done so, create a Google Cloud project and a Cloud Storage bucket.

  1. Setting up your project

    1. Sign in to your Google Cloud account. If you're new to Google Cloud, create an account to evaluate how our products perform in real-world scenarios. New customers also get $300 in free credits to run, test, and deploy workloads.

    2. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

      Go to project selector

    3. Verify that billing is enabled for your Google Cloud project.

    4. Enable the Dataproc, Compute Engine, and Cloud Storage APIs.

      Roles required to enable APIs

      To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

      Enable the APIs

    5. Install the Google Cloud CLI.

    6. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    7. To initialize the gcloud CLI, run the following command: 

      gcloud init

    8. In the Google Cloud console, on the project selector page, select or create a Google Cloud project.

      Roles required to select or create a project

      • Select a project: Selecting a project doesn't require a specific IAM role—you can select any project that you've been granted a role on.
      • Create a project: To create a project, you need the Project Creator role (roles/resourcemanager.projectCreator), which contains the resourcemanager.projects.create permission. Learn how to grant roles.

      Go to project selector

    9. Verify that billing is enabled for your Google Cloud project.

    10. Enable the Dataproc, Compute Engine, and Cloud Storage APIs.

      Roles required to enable APIs

      To enable APIs, you need the serviceusage.services.enable permission. If you created the project, then you likely already have this permission through the Owner role (roles/owner). Otherwise, you can get this permission through the Service Usage Admin role (roles/serviceusage.serviceUsageAdmin). Learn how to grant roles.

      Enable the APIs

    11. Install the Google Cloud CLI.

    12. If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

    13. To initialize the gcloud CLI, run the following command: 

      gcloud init

  2. Creating a Cloud Storage bucket in your project to hold the data used in this tutorial.

    1. In the Google Cloud console, go to the Cloud Storage Buckets page.

      Go to Buckets

    2. Click Create.
    3. On the Create a bucket page, enter your bucket information. To go to the next step, click Continue.
      1. In the Get started section, do the following:
        • Enter a globally unique name that meets the bucket naming requirements.
        • To add a bucket label, expand the Labels section (), click Add label, and specify a key and a value for your label.
      2. In the Choose where to store your data section, do the following:
        1. Select a Location type.
        2. Choose a location where your bucket's data is permanently stored from the Location type drop-down menu.
        3. To set up cross-bucket replication, select Add cross-bucket replication via Storage Transfer Service and follow these steps:

          Set up cross-bucket replication

          1. In the Bucket menu, select a bucket.

          2. In the Replication settings section, click Configure to configure settings for the replication job.

            The Configure cross-bucket replication pane appears.

            • To filter objects to replicate by object name prefix, enter a prefix that you want to include or exclude objects from, then click Add a prefix.
            • To set a storage class for the replicated objects, select a storage class from the Storage class menu. If you skip this step, the replicated objects will use the destination bucket's storage class by default.
            • Click Done.
      3. In the Choose how to store your data section, do the following:
        1. Select a default storage class for the bucket or Autoclass for automatic storage class management of your bucket's data.
        2. To enable hierarchical namespace, in the Optimize storage for data-intensive workloads section, select Enable hierarchical namespace on this bucket.
      4. In the Choose how to control access to objects section, select whether or not your bucket enforces public access prevention, and select an access control method for your bucket's objects.
      5. In the Choose how to protect object data section, do the following:
        • Select any of the options under Data protection that you want to set for your bucket.
          • To enable soft delete, click the Soft delete policy (For data recovery) checkbox, and specify the number of days you want to retain objects after deletion.
          • To set Object Versioning, click the Object versioning (For version control) checkbox, and specify the maximum number of versions per object and the number of days after which the noncurrent versions expire.
          • To enable the retention policy on objects and buckets, click the Retention (For compliance) checkbox, and then do the following:
            • To enable Object Retention Lock, click the Enable object retention checkbox.
            • To enable Bucket Lock, click the Set bucket retention policy checkbox, and choose a unit of time and a length of time for your retention period.
        • To choose how your object data will be encrypted, expand the Data encryption section (), and select a Data encryption method.
    4. Click Create.

Define a cluster configuration

A Dataproc Hub instance creates a cluster from configuration values contained in a YAML cluster configuration file.

Your cluster configuration can specify any feature or component available to Managed Service for Apache Spark clusters (such as machine type, initialization actions, and optional components). The cluster image version must be 1.4.13 or higher. Attempting to spawn a cluster with an image version lower than 1.4.13 will cause an error and fail.

Sample YAML cluster configuration file

clusterName: cluster-name
config:
  softwareConfig:
    imageVersion: 2.2-ubuntu22
    optionalComponents:
    - JUPYTER

Each configuration must be saved in Cloud Storage. You can create and save multiple configuration files to give users a choice when they use Dataproc Hub to create their Managed Service for Apache Spark cluster notebook environment.

There are two ways to create a YAML cluster configuration file:

  1. Create YAML cluster configuration file from the console

  2. Export a YAML cluster configuration file from an existing cluster

Create YAML cluster configuration file from the console

  1. Open the Create a cluster page in the Google Cloud console, then select and fill in the fields to specify the type of cluster the Dataproc Hub will spawn for users.
    1. At the bottom of the left panel, select "Equivalent REST".
    2. Copy the generated JSON block, excluding the leading POST request line, then paste the JSON block into an online JSON-to-YAML converter (search online for "Convert JSON to YAML").
    3. Copy the converted YAML into a local cluster-config-filename.yaml file.

Export a YAML cluster configuration file from an existing cluster

  1. Create a cluster that matches your requirements.
  2. Export the cluster configuration to a local cluster-config-filename.yaml file. 
    gcloud dataproc clusters export cluster-name \
    --destination cluster-config-filename.yaml  \
    --region region

Save the YAML configuration file in Cloud Storage

Copy your local YAML cluster configuration file to your Cloud Storage bucket.

gcloud storage cp cluster-config-filename.yaml gs://bucket-name/

Set Dataproc Hub instance environment variables

The administrator can set the hub environment variables listed in the table, below, to set attributes of the Managed Service for Apache Spark clusters that will be spawned by hub users.

Variable Description Example
NOTEBOOKS_LOCATION Cloud Storage bucket or bucket folder that contains user notebooks. The `gs://` prefix is optional. Default: The Managed Service for Apache Spark staging bucket. gs://bucket-name/
DATAPROC_CONFIGS Comma delimited list of strings of the Cloud Storage paths to YAML cluster config files. The `gs://` prefix is optional. Default: gs://dataproc-spawner-dist/example-configs/. which contains predefined example-cluster.yaml and example-single-node.yaml. gs://cluster-config-filename.yaml
DATAPROC_LOCATIONS_LIST Zone suffixes in region where the Dataproc Hub instance is located. Users can select one of these zones as the zone where their Managed Service for Apache Spark cluster will be spawned. Default: "b". b,c,d
DATAPROC_DEFAULT_SUBNET Subnet on which Dataproc Hub instance will spawn Managed Service for Apache Spark clusters. Default: the Dataproc Hub instance subnet. https://www.googleapis.com/compute/v1/projects/project-id/regions/region/subnetworks/subnet-name
DATAPROC_SERVICE_ACCOUNT Service account that Managed Service for Apache Spark VMs will run as. Default: If not set, the default Managed Service for Apache Spark service account is used. service-account@project-id.iam.gserviceaccount.com
SPAWNER_DEFAULT_URL Whether to show the Jupyter or JupyterLab UI on spawned Managed Service for Apache Spark clusters by default. Default: "/lab". `/` or `/lab`, for Jupyter or JupyterLab, respectively.
DATAPROC_ALLOW_CUSTOM_CLUSTERS Whether to allow users to customize their Managed Service for Apache Spark clusters. Default: false. "true" or "false"
DATAPROC_MACHINE_TYPES_LIST List of machine types that users are allowed to choose for their spawned Managed Service for Apache Spark clusters, if cluster customization (DATAPROC_ALLOW_CUSTOM_CLUSTERS) is enabled. Default: empty (all machine types are allowed). n1-standard-4,n1-standard-8,e2-standard-4,n1-highcpu-4
NOTEBOOKS_EXAMPLES_LOCATION Cloud Storage path to notebooks bucket or bucket folder to be downloaded to the spawned Managed Service for Apache Spark cluster when the cluster starts. Default: empty. gs://bucket-name/

Setting hub environment variables

There are two ways to set hub environment variables:

  1. Set hub environment variables from the console

  2. Set hub environment variables in a text file

Set hub environment variables from the console

When you create a Dataproc Hub instance from the User-Managed Notebooks tab on the Dataproc→Workbench page in the Google Cloud console, you can click the Populate button to open a Populate Dataproc Hub form that allows you to set each environment variable.

Set hub environment variables in a text file

  • Create the file. You can use a text editor to set Dataproc Hub instance environment variables in a local file. Alternatively, you can create the file by running the following command after filling in placeholder values and changing or adding variables and their values. 

    cat <<EOF > environment-variables-file
DATAPROC_CONFIGS=gs://bucket/cluster-config-filename.yaml
NOTEBOOKS_LOCATION=gs://bucket/notebooks
DATAPROC_LOCATIONS_LIST=b,c
EOF

    • Save the file in Cloud Storage. Copy your local Dataproc Hub instance environment variables file to your Cloud Storage bucket. 

    gcloud storage cp environment-variable-filename gs://bucket-name/folder-name/

    Set Identity and Access Management (IAM) roles

    Dataproc Hub includes the following identities with the following abilities:

    Each identity requires specific roles or permissions to perform their associated tasks. The table below summarizes IAM roles and permissions required by each identity.

    Identity Type Role or permission
    Dataproc Hub administrator User or Service account roles/notebooks.admin
    Dataproc Hub user User notebooks.instances.use, dataproc.clusters.use
    Dataproc Hub Service account roles/dataproc.hubAgent
    Dataproc Service account roles/dataproc.worker

    Create a Dataproc Hub instance

    1. Before you begin: To create a Managed Service for Apache Spark Hub instance from the Google Cloud console, your user account must have compute.instances.create permission. Also, the service account of the instance—the Compute Engine default service account or your user-specified service account listed in IAM & admin > Service Accounts (see Managed Service for Apache Spark VM service account)— must have iam.serviceAccounts.actAs permission.

    2. Go to the Dataproc→Workbench page in the Google Cloud console, then select the User-Managed Notebooks tab.

    3. If not pre-selected as a filter, click in the Filter box, then select **Environment:Dataproc Hub"".

    4. Click New Notebook→Dataproc Hub.

    5. On the Create a user-managed notebook page, provide the following information:

      1. Notebook name: Dataproc Hub instance name.
      2. Region: Select a region for the Dataproc Hub instance. Managed Service for Apache Spark clusters spawned by this Dataproc Hub instance will also be created in this region.
      3. Zone: Select a zone within the selected region.
      4. Environment:
        1. Environment: Select Dataproc Hub.
        2. Select a script to run after creation (optional): You can insert or browse and select an initialization action script or executable to run on the spawned Managed Service for Apache Spark cluster.
        3. Populate Dataproc Hub (optional): Click Populate to open a form that allows you to set each of the hub environment variables (see Set Dataproc Hub instance environment variables for a description of each variable). Managed Service for Apache Spark uses default values for any unset environment variables. As an alternative, you can set Metadata key:value pairs to set environment variables (see next item).
        4. Metadata:
          1. If you created a text file that contains your hub environment variable settings (see Setting hub environment variables), provide the name of the file as the key and the gs://bucket-name/folder-name/environment-variable-filename Cloud Storage location of the file as the value. Managed Service for Apache Spark uses default values for any unset environment variables.
      5. Machine configuration:
        1. Machine Type: Select the Compute Engine machine type.
        2. Set other machine configuration options.
      6. Other Options:
        1. You can expand and set or replace default values in the Disks, Networking, Permission, Security, and Environment upgrade and system health sections.
      7. Click Create to launch the Dataproc Hub instance.

    6. The Open JupyterLab link for the Dataproc Hub instance becomes active after the instance is created. Users click this link to open the JupyterHub server page to configure and create a Managed Service for Apache Spark JupyterLab cluster (see Use Dataproc Hub).

    Clean up

    Delete the Dataproc Hub instance

    Delete the bucket

    What's next