These release notes include announcements of new and updated Managed Service for Apache Spark features, bug fixes, known issues, and deprecated functionality.
Release schedule: The release of the latest Managed Service for Apache Spark images can take up to one week to roll out to all regions. Until the rollout is complete, the latest Managed Service for Apache Spark images may not be available in your region.
You can see the latest product updates for all of Google Cloud on the Google Cloud page, browse and filter all release notes in the Google Cloud console, or programmatically access release notes in BigQuery.
To get the latest product updates delivered to you, add the URL of this page to your feed reader, or add the feed URL directly.
Announcing the General Availability (GA) release of Managed Service for Apache Spark cluster
image version 3.0,
as follows:
3.0.0-debian13, 3.0.0-ml-ubuntu24, 3.0.0-rocky9, and 3.0.0-ubuntu24.
3.0.0-ml-ubuntu24 image extends the 3.0 base image with ML-specific libraries.Image version 3.0 is a lightweight image that contains the following
pre-installed core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs):
You can add
listed optional components
when you create a 3.0 image version cluster.
Recommendation: Use the 3.0 (and later) image versions to meet security
compliance requirements.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
2.1, 2.2 and 2.3 cluster image versions now support
Confidential Compute
for the
g4-standard-48
GPU machine type.Managed Service for Apache Spark (formerly Google Cloud Serverless for Apache Spark):
The 3.0 runtime now uses fewer executors, as follows:
spark.dynamicAllocation.minExecutors propertyspark.executor.instances and spark.dynamicAllocation.initialExecutors propertiesAll runtimes now configure spark.scheduler.listenerbus.exitTimeout to 30 seconds.
Key updates in these image versions include:
2.1.117 and 2.2.85 subminor image versions don't have preconfigured Conda channels, and are not mapped to default aliases (such as 2.1-debian11 and 2.2-debian12) until August 25, 2026.
2.1.117-debian11 or 2.2.85-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.2.1.116-debian11 or 2.2.84-debian12).2.1 and 2.2 will point to image versions without preconfigured Conda channels (the latest subminor versions). All workloads must transition to these new images after August 25, 2026 since the use of prior subminor versions with preconfigured Conda channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The following new subminor image versions, 1.3.96, 1.4.81, 1.5.92, 2.0.161, and 2.3.32,
don't have preconfigured Conda channels, and are not mapped to default aliases
(such as 2.3-debian12 and 2.3-ubuntu22) until August 25, 2026.
The new 2.1 and 2.2 subminor images continue to have preconfigured Conda channels,
but note the following Default change schedule:, which will impact all supported
image versions.
2.3.32-debian12). Packages cannot be installed using Conda unless channels are manually configured during cluster initialization.1.3.96, 1.4.81, 1.5.92, and 2.0.161 will become default after August 25, 2026.
Additionally, newer subminor versions for 2.1, 2.2, and 2.3 released after August 25, 2026 will not have preconfigured Conda channels and will be mapped to default aliases.
All workloads must use new images after August 25, 2026
since the use of prior subminor versions with preconfigured Conda
channels will be disallowed.You may need to delete and replace existing clusters After August 25, 2026, existing clusters created with images that have preconfigured Conda channels (even if cluster jobs don't use Conda to install packages) need to be deleted and replaced with new clusters created or recreated with images that don't have preconfigured Conda channels.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Key updates in these image versions include:
ICEBERG optional component. Users can opt-in to Iceberg 1.10 by setting the dataproc:dataproc.iceberg.version cluster property to 1.10 during cluster creation.spark.scheduler.listenerbus.exitTimeout to 30s for Compute Engine deployments.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Rollout of the new sub-minor versions without pre-configured channels will begin on June 22, 2026, delayed from the previously planned date of June 15, 2026 ETA.
Key updates in these runtime versions include:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
Added support for selecting specific Confidential Computing technologies (AMD SEV, AMD SEV-SNP, Intel TDX) when creating clusters using the new --confidential-compute-type flag in gcloud and the confidentialInstanceType field in the API. The boolean --enable-confidential-compute flag is now deprecated but will continue to function, defaulting to AMD SEV for backward compatibility.
confidentialInstanceType enum in the API.--enable-confidential-compute flag and enableConfidentialCompute field are deprecated in favor of the new type-specific flag/field.SEV.SEV, SEV-SNP, and TDX.Managed Service for Apache Spark (formerly Dataproc on Compute Engine): The following subminor image versions announced on May 19, 2026 have been rolled back:
Rollback Notice: The 2.2.82 image versions were rolled back on May 25, 2026.
Managed Service for Apache Spark (formerly Dataproc on Compute Engine):
The configuration for Spark shuffle partitions (spark.sql.shuffle.partitions) has changed from an integer to a string type.
This change impacts image versions 2.3.30 and later in version 2.3, and 2.2.82 and later in version 2.2.
spark.conf.set() with an integer literal.
spark.conf.set("spark.sql.shuffle.partitions", 100)spark.conf.set("spark.sql.shuffle.partitions", "100")spark-submit --conf spark.sql.shuffle.partitions=100), properties files, or Spark SQL commands (spark.sql("SET spark.sql.shuffle.partitions=100")) remains unaffected, as these methods naturally parse the input as strings.New Managed Service for Apache Spark (formerly Dataproc on Compute Engine) subminor cluster image versions for 3.0 (Preview):
Key updates in this release:
Key updates in this release:
v0.0.79.26.04.0 version in the Managed Service
for Apache Spark version 3.0 serverless runtime.Key updates in this release:
Managed Service for Apache Spark (formerly Dataproc on Compute Engine): Announcing the following cluster features:
Dataproc and Google Cloud Serverless for Apache Spark are now unified in the Google Cloud console under the Managed Service for Apache Spark brand. This change consolidates our managed Spark deployment options into a single umbrella brand that includes the full breadth of our Spark capabilities. No existing functionality is being removed as part of this change, and there will be no impact to the Dataproc API, client library, gcloud CLI, or Dataproc IAM role and permission names.
3.9.5 in image version 2.3.New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
2.1, 2.2 and 2.3 images.New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following subminor image versions announced on March 08, 2026 have been rolled back:
New Dataproc on Compute Engine subminor image versions:
Rollback Notice: These image versions were rolled back on March 11, 2026.
New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark 3.0 runtime version:
New Serverless for Apache Spark runtime versions:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Fixed an issue that could cause gsutil failures on clusters using custom images built from new base images. This problem occurred when the gcloud core/universe_domain property was not set in the build environment, resulting in a malformed /etc/boto.cfg. The image build process now ensures a correct default configuration, restoring proper Google Cloud Storage connectivity.
New Serverless for Apache Spark runtime versions:
Hyperdisks can now be added to a Dataproc cluster.
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark: Added support for removing conscrypt
from Serverless for Apache Spark 2.3+ runtimes using the
dataproc.artifacts.remove
property .
Dataproc on Compute Engine: Sharing snapshot diagnostic data: Setting the --tarball-access=GOOGLE_DATAPROC_DIAGNOSE flag with the gcloud dataproc clusters diagnose command shares all of the output Cloud Storage bucket contents with Google Cloud support if uniform bucket-level access is enabled on the output Cloud Storage bucket. If object-level access control is enabled on the output Cloud Storage bucket, only the generated diagnostic tar file is shared.
New Dataproc on Compute Engine subminor image versions:
Zookeeper version upgraded to 3.9.4 in Dataproc image 2.3.22.
Parquet CLI version upgraded to 1.15.2 in 2.1 and 2.2 images.
Delta subminor version upgraded to 3.2.1 in Dataproc on Compute Engine image 2.2 and 2.3.
Apache Pig is now available in ARM images.
Added a new dataproc:pypi.repository property to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.
Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.
Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.
Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.
Upcoming Spark data lineage changes See the upcoming May, 2026 Dataproc and Serverless for Apache Spark release notes for an announcement of a change that will automatically enable Dataproc Spark data lineage and Serverless for Apache Spark data lineage when you enable the Data Lineage API (see Control lineage ingestion for a service) without requiring additional project, cluster, batch workload, or interactive session settings.
New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following subminor image versions announced on January 24, 2026 have been rolled back:
New Dataproc on Compute Engine subminor image versions:
Rollback Notice: These image versions were rolled back on January 30, 2026.
Upgraded the Delta subminor version to 3.2.1 in images 2.2 and 2.3.
Apache Pig is now available in ARM images.
Added a new dataproc:pypi.repository property to customize the PyPI repository that pip uses.
The value can be a internet-accessible URL, or you can specify google to use a Google-hosted cache of PyPI, which
is accessible without public internet connectivity. Starting with Managed Service for Apache Spark image version
3.1, google will be the default (you can specify the pypi property value to use public PyPI instead of the default google 3.1 value).
Removed the use of deprecated Hadoop configuration properties fs.default.name and
yarn.resourcemanager.system-metrics-publisher.enabled.
Fixed a bug in the ARM image that prevented connecting to a Dataproc Metastore instance with a gRPC protocol endpoint.
Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.
New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following subminor image versions announced on January 06, 2026 have been rolled back:
New Serverless for Apache Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Rollback Notice: These image versions were rolled back on January 13, 2026.
Added a new property dataproc:pypi.repository to customize the PyPI repository used for pip. The value can be a URL, or google to use a Google-hosted cache of PyPI, accessible without public internet connectivity. Starting in image version 3.1, google will be the default; to opt out and return to public PyPI, use the value pypi.
Removed use of deprecated Hadoop configuration properties fs.default.name and yarn.resourcemanager.system-metrics-publisher.enabled.
Fixed the spark.driver.extraClassPath delimiter for the Jupyter SparkMonitor Listener.
New Serverless for Apache Spark runtime versions:
The following Dataproc on Compute Engine subminor image versions, released on December 05, 2025, are now blocklisted:
New Dataproc on Compute Engine subminor image versions:
dataproc-ml is available by default in 2.3 -ml images.
Serverless for Apache Spark: Runtime version 3.0 is now generally available. This version simplifies onboarding, improves reliability, reduces startup latency, and adds support for Spark 4.0.
Features and improvements:
bigquery Spark catalog, pre-configured for out-of-the-box BigQuery
native table interactionsdataproc-rm.googleapis.com API enablement is requiredNew Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Updated Cloud Storage connector:
3.1.10 in image version 2.3.18.3.0.15 in image version 2.2.71.hadoop3-2.2.30 in image versions 2.1.103 and 2.0.154.New Serverless for Apache Spark runtime versions:
Serverless for Apache Spark: Added the
dataproc.artifacts.remove property,
which lets users remove default artifacts, such as spark-bigquery-connector,
iceberg, and delta-lake from a Serverless for Apache Spark runtime.
Serverless for Apache Spark: Apache Spark upgrade to version 3.5.3 for the 1.2 and 2.2 Serverless for Apache Spark runtime versions announced on October 13, 2025 has been rolled back.
Dataproc on Compute Engine subminor image version
2.3.16, announced on
October 20, 2025
has been blocklisted and cannot be used when creating a new cluster.
New Dataproc on Compute Engine subminor image versions:
Fixed a Jupyter Kernel Gateway bug that caused failures while restarting kernels.
Announcing the General Availability (GA) of Lightning Engine for Google Cloud Serverless for Apache Spark. Lightning Engine is a high-performance query accelerator that delivers up to 4.3x faster performance for Spark workloads compared to open-source Spark, as measured on TPC-H-like benchmarks.
Lightning Engine and Native Query Execution are not available in 3.0 runtimes. For more details on enabling Lightning Engine and its advanced features like Native Query Execution (NQE), see the official documentation.
Serverless for Apache Spark: With the Lightning Engine GA release, the property to enable Native Query Execution (NQE) feature has been updated.
In order to use Lightning Engine, submit your jobs in the Premium tier. Under Lightning Engine, if you would like to use the NQE feature, set the new flag: spark.dataproc.lightningEngine.runtime=native. Users are encouraged to try this feature to explore the full potential of Lightning Engine.
For backward compatibility, the legacy property that was used to enable NQE spark.dataproc.runtimeEngine=native will continue to be honored in the existing runtimes 1.2, 2.2 and 2.3, but it's not supported in future releases (3.0+ runtimes).
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The default image version of premium tier clusters is now 2.3.
New Dataproc on Compute Engine subminor image versions:
Fixed startup race condition in multi-tenant clusters using the dataproc:pip.packages property that could cause authentication failures while starting Jupyter notebook kernels.
New Serverless for Apache Spark runtime versions:
Dataproc on Compute Engine: The following diagnostic properties are now enabled by default for new Dataproc clusters created with 2.0+ image versions:
dataproc:diagnostic.capture.enabled: Collects checkpoint diagnostic data in the cluster temp bucket.dataproc:dataproc.logging.extended.enabled: Collects logs for the Knox, Zeppelin, Ranger-usersync, Jupyter_notebook, Jupyter_kernel_gateway components and the Spark History-Server in Cloud Logging.dataproc:dataproc.logging.syslog.enabled: Collects VM syslogs in Cloud Logging.Note: To disable any of these features, set the corresponding property to false during cluster creation.
To continue using the Ops Agent initialization action opsagent.sh to ingest syslogs from Dataproc cluster nodes, do one of the following:
opsagent_nosyslog.sh since
VM syslogs are emitted by default from Dataproc clusters.dataproc:dataproc.logging.syslog.enabled=false and continue using opsagent.sh
to ingest syslogs.New Dataproc on Compute Engine subminor image versions:
New Dataproc on Compute Engine subminor image versions:
New Serverless for Apache Spark runtime versions:
Announcing the Preview release of Dataproc on Compute Engine image version 3.0.0-RC1:
Announcing the Preview release of Serverless for Apache Spark 3.0.0-RC3 runtime:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Multi-tenant clusters are now available in Preview. Many data engineers and scientists can share a multi-tenant cluster to execute their workloads in isolation from each other.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
Serverless for Apache Spark: Fixed a bug in Dataproc Batches that occasionally caused higher latency before an application was started.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Sharing checkpoint diagnostic data: Setting the dataproc:diagnostic.capture.access=GOOGLE_DATAPROC_DIAGNOSE property during cluster creation shares all of the temp bucket contents with Google Cloud support if uniform bucket-level access is enabled on temp bucket. If object-level access control is in effect on the temp bucket, only the checkpoint diagnostic data folder corresponding to the cluster in Cloud Storage is shared.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Image versions 2.2 and 2.3: The Iceberg optional component supports the BigLake Iceberg REST catalog.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Subminor version 1.1.111 is the last release of runtime version 1.1, which will no longer be supported and will not receive new releases.
New Dataproc on Compute Engine subminor image versions:
2.3.7-debian12, 2.3.7-ubuntu22, 2.3.7-ubuntu22-arm, 2.3.7-ml-ubuntu22, and 2.3.7-rocky9
The 2.3.7-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.
New Dataproc on Compute Engine subminor image versions:
2.3.6-debian12, 2.3.6-ubuntu22, 2.3.6-ml-ubuntu22, and 2.3.6-rocky9
The 2.3.6-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.
Dataproc on Compute Engine: Starting August 18, 2025, the following diagnostic properties will be enabled by default for newly created Dataproc clusters:
dataproc:diagnostic.capture.enabled: Enables the collection of checkpoint data in the cluster temp bucket.dataproc:dataproc.logging.extended.enabled: Enables the collection of logs for the Knox, Zeppelin, Ranger-usersync, Jupyter_notebook, Jupyter_kernel_gateway components and the Spark History-Server in Cloud Logging.dataproc:dataproc.logging.syslog.enabled: Enables the collection of VM syslogs in Cloud Logging.
To continue using the Ops Agent initialization action opsagent.sh to ingest syslogs from Dataproc cluster nodes, do one of the following:
opsagent_nosyslog.sh since
VM syslogs will now be emitted by default from Dataproc clusters.dataproc:dataproc.logging.syslog.enabled=false and continue using opsagent.sh
to ingest syslogs.Note: To disable any of these features, set the corresponding property to false during cluster creation.
Dataproc now allows Dynamic update of multi-tenancy clusters.
The Cluster Scheduled Stop feature is available in preview. You can use this feature to stop clusters after a specified idle period, at a specified future time, or after a specified period from the cluster creation or update request.
New Dataproc on Compute Engine subminor image versions:
2.3.5-debian12, 2.3.5-ubuntu22, 2.3.5-ml-ubuntu22, and 2.3.5-rocky9
The 2.3.5-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.
Serverless for Apache Spark (formerly known as Dataproc Serverless for Spark) now supports OS Login organization policy. Organizations, folders, and projects that enforce the OS Login policy can now use Serverless for Apache Spark.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: The built-in Iceberg now supports the BigLake Iceberg REST catalog on the 2.2 runtime.
Dataproc Serverless for Spark: Upgraded the Cloud Storage connector version to 2.2.28 in the 1.1 runtime.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Upgraded the Cloud Storage connector version to 2.2.28 in the latest 2.0 and 2.1 images.
New Dataproc on Compute Engine subminor image versions:
2.3.4-ml-ubuntu22.The 2.3.4-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.
Dataproc on Compute Engine: Dataproc now automatically configures Knox Gateway configuration properties gateway.dispatch.whitelist.services and gateway.dispatch.whitelist for component web UIs within the cluster.
Dataproc on Compute Engine: Fixed a bug in trino-jvm cluster properties. To configure Trino JVM options prefixed with trino-jvm, follow these guidelines:
-XX:, without :. For JVM flags without a value, add = at the end. For example, add trino-jvm:-XX+HeapDumpOnOutOfMemoryError= as -XX:+HeapDumpOnOutOfMemoryError in the jvm.config.-D prefix the same way. For example, trino-jvm:-Dsystem.property.name=value.: cannot be provided as a cluster property.Dataproc on Compute Engine & Dataproc Serverless: Backported GH-3198 in Parquet addressing CVE-2025-46762.
New Dataproc Serverless for Spark runtime versions:
Announcing the GA release of Dataproc on Compute Engine image version 2.3:
Image Version 2.3 is a lightweight image that contains only core components, reducing exposure to Common Vulnerabilities and Exposures (CVEs). For higher security compliance requirements, use the image version 2.3 or later when creating a Dataproc cluster. Optional components can still be deployed on-demand.
The following images are the latest available 2.3 subminor image versions:
2.3.3-debian12, 2.3.3-rocky9, 2.3.3-ubuntu22, and 2.3.3-ml-ubuntu22.The 2.3.3-ml-ubuntu22 image extends the 2.3 base image with ML-specific libraries.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Fixed a bug that prevented the spark.executorEnv property from correctly setting specific executor environment variables across all runtimes.
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Fixed the ordering of log entries generated from clusters created with 2.2+ image versions by assigning timestamps closer to the log generation time.
The support dates for Dataproc on Compute Engine image versions 2.0, 2.1, and 2.2 have been extended, as follows:
2.2: Supported until 03/31/20272.1: Supported until 03/31/20262.0 Supported until 09/30/2025New Dataproc Serverless for Spark runtime versions:
Announcing the General Availability release of Spark on BigQuery, which lets you create a serverless Spark session in a BigQuery Studio notebook. Use this feature to create, run, and test Spark jobs quickly and easily. For more information, see Run PySpark code in BigQuery Studio notebooks.
Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime versions 2.3, which include the following components:
New Dataproc Serverless for Spark runtime versions:
Dataproc now supports the creation of zero-scale clusters, available in preview. This feature provides a cost-effective way to use Dataproc clusters, as they utilize only secondary workers that can be scaled down to zero when not in use.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Spark UI for Dataproc Serverless batches and interactive sessions, which lets you monitor and debug your serverless Spark workloads, now features Event Timeline and Task Quantile views for enhanced troubleshooting.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: The default enabling of the following cluster properties previously announced to occur on May 10, 2025 (see the February 10, 2025 release note) has been postponed to a future date. The future date will be announced in a release note at least one month in advance of the change. Until then, these diagnostic properties will continue to be set to false by default unless set to true by the user.
dataproc:diagnostic.capture.enableddataproc:dataproc.logging.extended.enableddataproc:dataproc.logging.syslog.enabledDataproc on Compute Engine: Upgraded NodeProblemDetector to 0.8.20 based version for 2.2 image.
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Upgraded oauth2l to v1.3.3 to address CVEs.
Dataproc on Compute Engine: Fixed an issue with Apache Hudi that caused failure in Hudi CLI.
Dataproc Serverless: Backported GH-3168 in Parquet addressing CVE-2025-30065.
New Dataproc Serverless for Spark runtime versions:
Native Query Execution now supports reading Apache ORC complex types.
Dataproc on Compute Engine: Fixed Job ID retrieval in Dataproc job logs for clusters created with 2.0, 2.1 image versions, by ignoring timestamp prefix.
New Dataproc on Compute Engine subminor image versions:
2.0.138-debian10, 2.0.138-rocky8, 2.0.138-ubuntu18
2.1.86-debian11, 2.1.86-rocky8, 2.1.86-ubuntu20, 2.1.86-ubuntu20-arm
2.2.54-debian12, 2.2.54-rocky9, 2.2.54-ubuntu22
Dataproc on Compute Engine: Backported GH-3168 in Parquet addressing CVE-2025-30065.
Dataproc on Compute Engine: Added an temporary object hold on the spark-job-history folder in Cloud Stroage to prevent deletion by Cloud Storage life cycling.
New Dataproc Serverless for Spark runtime versions:
Fixed a bug in which Jupyter fails to restart upon cluster restart on Personal Authentication clusters.
Dataproc on Compute Engine: The Spark BigQuery connector has been upgraded to version 0.34.1 in the latest 2.2 image version.
New Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: Gemini Cloud Assist Investigations is available in Preview for the following runtimes:
Dataproc on Compute Engine: Fixed an issue with the retrieval of an Access token when using the ranger-gcs-plugin with 2.2 images.
New Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: Installed CUDA, cuDNN and NCCL NVIDIA libraries in 1.2 and 2.2 runtimes.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Hyperdisk-Balanced is now the default primary disk type when creating a cluster from the console.
Dataproc on Compute Engine: Fixed incorrectly attributed Dataproc job logs in Cloud Logging for clusters created with 2.2+ image versions. This happened when multiple Dataproc jobs were running concurrently on the same cluster.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Hadoop Native libraries are installed by default in all runtimes.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Added support for Enhanced Flexibility Mode (EFM) with primary worker shuffle mode on Spark for image version 2.2.50 and above.
Dataproc on Compute Engine: The latest Dataproc 2.2 image version now supports Spark data lineage.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc is now available in the europe-north2 region (Stockholm, Sweden).
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Explicitly disabled sha1, md5 algorithms for use with kex and kex-gss sshd features.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Data Lineage for Dataproc Hive is now in Public Preview, which can be enabled using the Hive Lineage initialization action.
Dataproc on Compute Engine: To help diagnose Dataproc clusters, you can set the following cluster properties to true when you create a cluster:
dataproc:diagnostic.capture.enabled: When set to true, enables the collection of checkpoint data in the cluster temp bucket.dataproc:dataproc.logging.extended.enabled: When set to true, enables the collection of logs for the Knox, Zeppelin, Solr, Trino, Presto, Ranger-usersync, Jupyter_notebook components and the Spark History-Server in Cloud Logging.dataproc:dataproc.logging.syslog.enabled: When set to true, enables the collection of VM syslogs in Cloud Logging.Note: starting May 10, 2025, these properties will be set to true by default.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Spark UI for Dataproc Serverless Batches and Interactive sessions, which lets you to monitor and debug your serverless Spark workloads, is now available for CMEK (Customer-Managed Encryption Keys) and Assured Workloads. The Spark UI is available by default and free of cost.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Hyperdisk Balanced primary disk type available on Dataproc clusters.New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Private Google Access is now automatically enabled in the configured subnetwork when creating clusters with internal IP addresses.
Dataproc Serverless for Spark: Private Google Access is now automatically enabled in the configured subnetwork when running batch workloads and interactive sessions.
Dataproc cluster caching now supports ARM images.
Zeppelin component added to 2.1-Ubuntu20-arm images.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: On March 10, 2025, the Dataproc Resource Manager API will be enabled as part of General Availability (GA) for Dataproc Serverless 3.0+ versions.
User action will not be required in response to this API enablement change.
The Dataproc Resource Manager will be implemented as a stand-alone Google Cloud API, dataprocrm.googleapis.com. It will allow Dataproc distributions of open source software, ,particularly Apache Spark, to directly communicate resource requirements.
New Dataproc on Compute Engine subminor image versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Updated Dataproc Metastore (DPMS) gRPC proxy image version to v. 0.0.70
Dataproc Serverless for Spark: Spark Lineage is available for all supported Dataproc Serverless for Spark runtime versions.
Dataproc is now available in the northamerica-south1 region (Queretaro, Mexico).
Announcing the General Availability (GA) of Spot and non-preemptible VM mixing for Dataproc secondary workers which allows you to mix spot and non-preemptible secondary workers when you create a Dataproc cluster.
Announcing the General Availability (GA) of Flexible shapes for Dataproc secondary workers which allows you to provide a ranked selection of machine types to use for the creation of VMs.
New Dataproc Serverless for Spark runtime versions:
Disabled HiveServer2 Ranger policy synchronization in non-HA clusters for latest image version 2.1 and later. Policy synchronization is causing instability of the HiveServer2 process while trying to connect to ZooKeeper, which is not active by default in non-HA clusters.
New Dataproc on Compute Engine subminor image versions:
Note: When using Dataproc version 2.0.125 with the ranger-gcs-plugin, please create a customer support request for your project to use the enhanced version of the plugin prior to its GA release. This note does not apply Dataproc on Compute Engine image versions 2.1 and 2.2.
Dataproc Serverless for Spark: The Hadoop Google Secret Manager Credential Provider feature is now available in the Dataproc Serverless for Spark 1.2 and 2.2 runtimes.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Announcing the General Availability (GA) release of Spark UI for Dataproc Serverless Batches and Interactive sessions which allows you to monitor and debug your serverless Spark workloads. Spark UI is available by default and free of cost for all Dataproc Serverless workloads.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Clusters created with image versions 2.0.57+, 2.1.5+, or 2.2+: Secondary workers' control plane operations are made by the Dataproc Service Agent service account (service-<project-number>@dataproc-accounts.iam.gserviceaccount.com). They will no longer use the Google APIs Service Agent service account (<project-number>@cloudservices.gserviceaccount.com).
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
Blocklisted the following Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: Added the google-cloud-dlp Python package by default to the Dataproc Serverless for Spark runtimes.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: In runtime versions 1.2 and 2.2, minimized the dynamic memory footprint of the Spark application by setting XX:MaxHeapFreeRatio to 30% and XX:MinHeapFreeRatio to 10%.
Dataproc Serverless for Spark: Fixed an issue that would cause some batches and sessions to fail to start when using the premium compute tier.
Blocklisted the following Dataproc on Compute Engine subminor image versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Fixed a bug that caused some batches and sessions to fail to start when using the premium compute tier.
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: The latest 2.2 image versions support Hudi Trino integration natively. If both components are selected when you create a Dataproc cluster, Trino will be configured to support Hudi automatically.
Dataproc on Compute Engine: The latest 2.2 image versions now support Hudi 0.15.0.
Dataproc on Compute Engine: Dataproc image version 2.2 will become the default Dataproc on Compute Engine image version on September 6, 2024.
New Dataproc on Compute Engine subminor image versions:
Dataproc on GKE runtime versions 2.0 (Spark 3.1) is deprecated.
Dataproc on Compute Engine: Apache Spark upgraded to version 3.5.1 in image version 2.2 starting with image version 2.2.30.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Subminor version 2.0.85 is the last release of runtime version 2.0, which will no longer be supported and will not receive new releases.
New Dataproc on Compute Engine subminor image versions:
syslog is now available for Dataproc cluster nodes in Cloud Logging. See Dataproc logs for cluster and job log information.
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Enabled user sync by default for clusters using Ranger.
Replaced Spark external packages with connector folder on Dataproc 2.2 clusters.
Fixed a bug that caused intermittent delays and failures in clusters with 3 HDFS.
Added support for N4 and C4 machine types for Dataproc image versions 2.1 and above. The following default configurations are now applied to clusters created with N4 or C4 machine types:
bootdisktype = "hyperdisk-balanced"nictype = "gvnic"Hyperdisks for Dataproc clusters are now created with default throughput and IOPS. When this behavior becomes configurable, it will be announced in a future release note.
When a Cluster, Job, AutoscalingPolicy, or WorkflowTemplate API resource does not exist and the requestor does not have access to the project, a 403 error code is now issued instead of a 404 error code.
New Dataproc Serverless for Spark runtime versions:
Note: Dataproc Serverless for Spark runtime versions 1.1.71, 1.2.15, 2.0.79, and 2.2.15 were not released.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Added Cloud Profiler support in Dataproc Serverless for Spark. Enable profiling via the dataproc.profiling.enabled=true property and configure it via dataproc.profiling.name=<PROFILE_NAME>
Dataproc on Compute Engine: Apache Hadoop upgraded to version 3.2.4 in image version 2.0 starting with image version 2.0.109.
New Dataproc on Compute Engine subminor image versions:
Backported fixes for HIVE-25958 and HIVE-20220 (new configuration hive.groupby.enable.deterministic.distribution=false/true).
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: To fix compatibility with open table formats (Apache Iceberg, Apache Hudi and Delta Lake), the ANTLR version downgraded from 4.13.1 to 4.9.3 in Dataproc Serverless for Spark runtime versions 1.2 and 2.2.
The Dataproc Component Gateway is now activated by default when you create a Dataproc on Compute Engine cluster using the Google Cloud console.
New Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: To fix compatibility with open table formats (Apache Iceberg, Apache Hudi and Delta Lake), the ANTLR version will be downgraded from 4.13.1 to 4.9.3 in Dataproc Serverless for Spark runtime versions 1.2 and 2.2 on June 26, 2024.
Dataproc Serverless for Spark: Spark runtime version 2.2 will become the default Dataproc Serverless for Spark runtime version on September 6, 2024.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Support configuration to prevent HiveMetaStore metrics expensive database queries. To prevent expensive queries during HiveMetaStore startup, set Hive property metastore.initial.metadata.count.enabled to false.
The Apache Spark in BigQuery feature is available in Private Preview. This feature lets you create a Spark session in a BigQuery notebook that you can use to develop and submit PySpark code from BigQuery. To access this feature, fill in and submit the Dataproc Preview access request form.
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: When creating a cluster with the latest Dataproc on Compute Engine image versions, the secondary worker boot disk type now defaults to the primary worker boot disk type, which is pd-standard if the primary worker boot disk type is not specified.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Update restartable job error messages to include job IDs.
Dataproc Serverless for Spark: Automatically apply goog-dataproc-session-id, goog-dataproc-session-uuid and goog-dataproc-location labels for a session resource.
New Dataproc on Compute Engine subminor image versions:
Dataproc Serverless for Spark: Removed Spark data lineage support for runtime version 1.2.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Subminor version 2.1.50 is the last release of runtime version 2.1, which will no longer be supported and will not receive new releases.
Blocklisted the following Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
2.0.102-debian10, 2.0.102-rocky8, 2.0.102-ubuntu18
2.1.50-debian11, 2.1.50-rocky8, 2.1.50-ubuntu20, 2.1.50-ubuntu20-arm
2.2.16-debian12, 2.2.16-rocky9, 2.2.16-ubuntu22
Anaconda's default channel is disabled for package installations on Dataproc on Compute Engine.
New Dataproc on Compute Engine subminor image versions:
2.0.101-debian10, 2.0.101-rocky8, 2.0.101-ubuntu18
2.1.49-debian11, 2.1.49-rocky8, 2.1.49-ubuntu20, 2.1.49-ubuntu20-arm
2.2.15-debian12, 2.2.15-rocky9, 2.2.15-ubuntu22
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark:
When you submit a Dataproc Serverless Batch with a CMEK key:
serviceusage.services.use permission to the custom role attached to the Dataproc Service Agent service account.unreachable field that lists any batches with job arguments that couldn't be decrypted. You can issue a batches.get request to obtain more information on an unreachable batch.us-central1 region
must also be located in the us-central1 region.New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Announcing Dataproc Workflow Templates supports the CMEK organization policy.
New Dataproc Serverless for Spark runtime versions:
Updated the default autoscaling V2 cool-down time from 2m to 1m to reduce scaling latency.
Set the soft delete policy of newly created Dataproc staging and temp Cloud Storage buckets to 0 days.
Fixed a bug where Dataproc Serverless sessions that live longer than 48 hours are underbilled.
Dataproc Serverless for Spark: The preview release of Advanced troubleshooting, including Gemini-assisted troubleshooting, is now available for Spark workloads submitted with the following or later-released runtime versions:
Dataproc Serverless for Spark: Announcing the preview release of Autotuning Spark workloads.
New Dataproc Serverless for Spark runtime versions:
Added bigframes Python package by default in the Dataproc Serverless for Spark runtime versions 1.2 and 2.2
The following previously released sub-minor versions of Dataproc on Compute Engine images have been rolled back and can only be used when updating existing clusters that already use them:
Dataproc Serverless for Spark: runtime version 2.2 will become the default Dataproc Serverless for Spark runtime version on May 3, 2024.
Note: This announcement was updated in the April 19, 2024 release note.
New Dataproc on Compute Engine subminor image versions:
Note: the above subminor image versions were rolled back on April 2, 2024
Dataproc on Compute Engine: New Hadoop Google Secret Manager Credential Provider feature introduced in latest Dataproc on Compute Engine 2.0 image versions.
New Dataproc Serverless for Spark runtime versions:
Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime versions 1.2 and 2.2, which include the following components:
New Dataproc on Compute Engine subminor image versions:
Announcing the Preview release of Dataproc Serverless for Spark 1.2 runtime:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Added the bigframes (BigQuery DataFrames) Python package in the Dataproc Serverless for Spark 2.1 runtime.
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.20 version in the latest 1.1, 2.0, and 2.1 runtimes.
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Upgraded Cloud Storage connector version to 2.2.20 for 2.0 and 2.1 images.
Dataproc on Compute Engine: Mounted Java cacerts into containers by default when the Docker-on-YARN feature is enabled.
Dataproc Serverless for Spark: Extended Spark metrics collected for a batch now include executor:resultSize, executor:shuffleBytesWritten, and executor:shuffleTotalBytesRead.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: The new Secret Manager credential provider feature is available in the latest 2.1 image versions.
Dataproc on Compute Engine:
Dataproc on Compute Engine: Fixed ZooKeeper startup failures in image 2.2 HA (High Availability) clusters that use fully qualified hostnames.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: The internalIpOnly cluster configuration setting now defaults to true for clusters created with 2.2 image versions. Also see Create a Dataproc cluster with internal IP addresses only.
Dataproc Serverless for Spark: Spark Lineage is available for Dataproc Serverless for Spark 1.1 runtime.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: The new Secret Manager credential provider feature is available in the latest 2.2 image versions.
Dataproc on Compute Engine Ranger Cloud Storage enhancement:
Both settings are configurable and can be enabled by customers: see Use Ranger with caching and downscoping .
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Backported patch for HADOOP-18652.
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark: Backported patch for HADOOP-18652.
Dataproc on Compute Engine: Bucket ttl validation now also runs for buckets created by Dataproc.
Dataproc Serverless for Spark: When dynamic allocation is enabled, the initial executor number is determined by max of spark.dynamicAllocation.initialExecutors and spark.executor.instances.
Dataproc on Compute Engine: Added a warning during cluster creation if the cluster Cloud Storage staging bucket is using the legacy fine-grained/ACL IAM configuration instead of the recommended Uniform bucket-level access controls.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Backported patches for HIVE-21214, HIVE-23154, HIVE-23354 and HIVE-23614.
Dataproc is now available in the africa-south1 region (Johannesburg, South Africa).
The GitHub Ops Agent initialization action installs the Ops Agent on a Dataproc cluster, and provides metrics similar to the metrics that were enabled with the --metric-sources=monitoring-agent-defaults setting available for use with Dataproc images versions prior to version 2.2.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Backport HIVE-19568: Active/Passive HiveServer2 HA: Disallow direct connection to passive instance.
Backport HIVE-27715: Remove ThreadPoolExecutorWithOomHook.
Dataproc on Compute Engine: The default yarn.nm.liveness-monitor.expiry-interval-ms Hadoop YARN setting has been changed in the latest image versions from 15000 (15 seconds) to 120000 (2 minutes).
New Dataproc on Compute Engine subminor image versions:
Dataproc on Compute Engine: Upgraded Miniconda to 23.11, Python to 3.11, and curl to 8.5 to fix CVE-2023-38545 in the latest 2.2 images.
Dataproc on Compute Engine: Fixed the gsutil: command not found error in the latest Ubuntu images.
Dataproc on Compute Engine: Fixed Trino startup issue in the latest 2.2 images.
Dataproc on Compute Engine: Upgraded Cloud Storage connector version to 2.2.19 in the latest 2.0 and 2.1 images.
Dataproc Serverless for Spark: Upgraded Cloud Storage connector to 2.2.19 version in the latest 1.1, 2.0, and 2.1 runtimes.
New Dataproc Serverless for Spark runtime versions:
Beginning March 31, 2024, when you submit a Dataproc Serverless Batch with a CMEK key:
unreachable field that lists any batches with job arguments that couldn't be decrypted. You can issue a batches.get request to obtain more information on an unreachable batch.us-central1 region
must also be located in the us-central1 region.Dataproc Serverless for Spark:
Upgraded the following components to the following versions in the latest 2.2 runtime:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
The following previously released sub-minor versions of Dataproc images have been rolled back and can only be used when updating existing clusters that already use them:
New Dataproc on Compute Engine subminor image versions:
2.2.2-debian12, 2.2.2-rocky9, 2.2.2-ubuntu22
Rollback Notice: See the January 4, 2024 release note rollback notice.
Dataproc on Compute Engine: Backported the patch for YARN-10975 in the latest 2.0 images.
Dataproc on Compute Engine: Changed the Hive Server2 and MetaStore maximum default JVM heap size to 32GiB. Previously, the limit was set to 1/4 of total node memory, which could be too large on large-memory machines.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Added the google-cloud-secret-manager Python package in the latest Dataproc Serverless for Spark runtimes.
Announcing the GA release of Dataproc on Compute Engine image version 2.2 :
The 2.2.0 release includes the following components:
Monitoring-agent-defaults metrics are not available in Dataproc on Compute Engine image version 2.2 clusters unless the Ops Agent is installed. Other metrics for Dataproc provided components will continue to work.
Blocklisted the following Dataproc on Compute Engine Images due to issue with increase in startup time:
New Dataproc Serverless for Spark runtime versions:
Announcing the Preview release of Dataproc Serverless for Spark 2.2 runtime:
New Dataproc on Compute Engine subminor image versions:
Added the Confidential Computing option on the "Manage Security" panel on the "Create a Dataproc cluster on Compute Engine" page in the Google Cloud console.
Updated the Zookeeper component version from 3.8.0 to 3.8.3 in the latest Dataproc on Compute Engine 2.1 image version.
Fixed Dataproc Hub issue in latest Dataproc on Compute Engine 2.1 image.
Backported HIVE-21698 in Hive 3.1.3 component in latest Dataproc on Compute Engine image versions.
New Dataproc Serverless for Spark runtime versions:
The Cloud Storage connector has been upgraded to version 2.2.18 in all Dataproc Serverless for Spark runtimes.
In the Flink component in the latest Dataproc on Compute Engine 2.1 image version, added the following java-storage client properties:
gs.retry.max.attempts property to set the max number of retry attemptsgs.retry.total.timeout property to set the total retry timeoutUpgraded the Cloud Storage connector version to 2.2.18 in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.
New Dataproc on Compute Engine subminor image versions:
Fixed a regression in the Zeppelin websocket rules that caused a websocket error in Zeppelin notebooks.
The Python kernel does not work in Zeppelin on the Dataproc on Compute Engine 2.1 image version. Other kernels are not impacted.
You can use CMEK (Customer Managed Encrytion Keys) with encrypted Dataproc cluster data, incuding persistent disk data, job arguments and queries submitted with Dataproc jobs, and cluster data saved in the cluster Dataproc staging bucket. See Use CMEK with cluster data for more information.
Announcing the General Availability (GA) release of Dataproc Jupyter Plugin and its availability in Vertex AI Workbench instance notebooks.
New Dataproc on Compute Engine subminor image versions:
Announcing the release of Workflow Template CMEK (Customer Managed Encryption Key) encryption. Use this feature to apply CMEK encryption to workflow template job arguments. For example, when this feature is enabled, the query string of a workflow template SparkSQL job is encrypted using CMEK.
You can now use Dataproc Serverless autoscaling V2 to help you manage Dataproc Serverless workloads, improve workload performance, and save costs.
Set spark.shuffle.mapOutput.minSizeForBroadcast=128m to fix SPARK-38101 when Dataproc Serverless Spark dynamic allocation is enabled.
Announcing the Preview release of Dataproc Flexible VMs. This feature lets you specify prioritized lists of secondary worker VM types that Dataproc will select from when creating your cluster. Dataproc will select the VM type with sufficient available capacity while taking quotas and reservations into account.
New Dataproc on Compute Engine subminor image versions:
Added spark.dataproc.scaling.version=2 config to let customers control the Dataproc Serverless for Spark autoscaling version.
Increased the TTL for Dataproc on Compute Engine custom images from 60 days to 365 days.
Fixed Knox rewrite rules for Zeppelin URLs in some cases in the latest 2.0 and 2.1 Dataproc on Compute Engine image versions.
New Dataproc Serverless for Spark runtime versions:
Announcing the General Availability (GA) release of Dataproc Serverless GPU accelerators.
New Dataproc Serverless for Spark runtime versions:
Dataproc on Compute Engine: Dataproc now collects the dataproc.googleapis.com/job/yarn/vcore_seconds and dataproc.googleapis.com/job/yarn/memory_seconds job-level resource attribution metrics to track YARN application vcore and memory usage during job execution. These metrics are collected by default and are not chargeable to customers.
Dataproc on Compute Engine: Properties dataproc:agent.ha.enabled and dataproc:componentgateway.ha.enabled now default to true to provide high availability for the Dataproc Agent and Component Gateway.
Dataproc on Compute Engine: Dataproc now collects a dataproc.googleapis.com/node/yarn/nodemanager/health health metric to track the health of individual YARN node managers running on VMs. This metric is written against the gce_instance monitored resource to help you find suspect nodes. It is collected by default and is not chargeable to customers.
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Announcing the General Availability (GA) release of Dataproc Serverless for Spark Interactive sessions.
New Dataproc on Compute Engine image version 2.2 is available for preview with upgraded components.
Upgraded Hadoop version from 3.3.3 to 3.3.6 in the latest Dataproc on Compute Engine 2.1 image version.
New Dataproc on Compute Engine subminor image versions:
Upgraded the Cloud Storage connector version to 2.2.17 in the latest Dataproc Serverless for Spark runtimes.
New Dataproc Serverless for Spark runtime versions:
Added the gs.filesink.entropy.enabled property in Flink to enable entropy
injection in filesink Cloud Storage path in the latest Dataproc on Compute Engine 2.1 image version.
Added the gs.http.connect-timeout and gs.http.read-timeout properties in Flink to
set the connection timeout and read timeout for java-storage client
in the latest Dataproc on Compute Engine 2.1 image version.
New Dataproc on Compute Engine subminor image versions:
Upgraded Hive version from 3.1.2 to 3.1.3 in the latest Dataproc on Compute Engine 2.0 image version.
New Dataproc on Compute Engine subminor image versions:
In the latest Dataproc on Compute Engine 2.0 and 2.1 image versions, unset the CLOUDSDK_PYTHON variable to allow the gcloud command-line tool to use its bundled Python interpreter.
Fixed Jupyter notebooks bug that made Scala compilation errors invisible with the Toree kernel in Dataproc on Compute Engine 2.1 images.
New Dataproc Serverless for Spark runtime versions:
Dataproc is now available in the me-central2 region (Dammam, Saudi Arabia).
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Scala has been upgraded to version 2.12.18 and Apache Tez has been upgraded to version 0.10.2 in Dataproc on Compute Engine 2.1 images.
Announcing the Private Preview release of the Dataproc on Compute Engine Flink Jobs resource. During Private Preview, you can contact your Google Cloud Sales representative to have your project(s) added to an allowlist to allow you to submit Flink jobs to the Dataproc on Compute Engine service.
The dataproc.diagnostics.enabled property is now avaiable to enable running diagnostics on Dataproc Serverless for Spark. The existing spark.dataproc.diagnostics.enabled property will be deprecated for use with newer runtimes.
Dataproc Auto zone placement for clusters is now available in the Google Cloud console by selecting the "Any" option for the cluster zone.
New Dataproc Serverless for Spark runtime versions:
The Apache Spark version has been upgraded from 3.3.0 to 3.3.2 in Dataproc on Compute Engine 2.1 images.
New Dataproc on Compute Engine subminor image versions:
Announcing the General Availability (GA) release of Data Lineage for Dataproc, which captures data transformations (lineage events) in Dataproc Spark jobs, and publishes them to Dataplex Lineage.
Dataproc Serverless Interactive sessions detail and list pages are now available in the Google Cloud console.
Announcing the Preview release of Dataproc Serverless for Spark Interactive sessions and the Dataproc Jupyter Plugin.
New Dataproc Serverless for Spark runtime versions:
Fixed a Dataproc Serverless issue where Spark batches failed with unhelpful error messages.
Dataproc is now available in the europe-west10 region (Berlin).
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Backported the patches for HIVE-20618 in the new Dataproc on Compute Engine 2.0 and 2.1 images.
New Dataproc on Compute Engine subminor image versions:
New Dataproc Serverless for Spark runtime versions:
Added new Dataproc Serverless Templates for batch workload creation:
Improved the reliability of Dataproc Serverless compute node initialization with a Premium disk tier option.
Added a dataproc:dataproc.cluster.caching.enabled flag to enable and disable Dataproc on Compute Engine cluster caching. The flag is false by default. Use this feature with the latest Dataproc on Compute Engine images.
The following previously released sub-minor versions of Dataproc on Compute Engine images unintentionally reverted several dependency library versions. This caused a risk of backward-incompatibility for some workloads.
These sub-minor versions have been rolled back, and can only be used when updating existing clusters that already use them:
Backported ZEPPELIN-5434 to image 2.1 to fix CVE-2022-2048.
Upgraded Hudi to 0.12.3 and added the BigQuery Sync tool as part of the Hudi optional component.
New Dataproc on Compute Engine image versions:
Downgraded Cloud Storage connector version to 2.2.15 in all Dataproc on Compute Engine image versions to prevent potential performance regression.
Backported the patches for HIVE-22170 and HIVE-22331.
New Dataproc Serverless for Spark runtime versions:
Downgraded Cloud Storage connector to 2.2.15 version in all Dataproc Serverless for Spark runtimes to prevent potential performance regression.
New Dataproc on Compute Engine image versions:
Note: The above image versions were rolled back. See the August 6, 2023 release note
The Maximum total memory per core for Dataproc Serverless Premium compute tiers has increased to 24576m (7424m for Standard compute tiers unchanged). See Dataproc Serverless Resource allocation properties.
New Dataproc Serverless for Spark runtime versions:
Clusters cannot be created with a driver node group if the cluster image version is older than 2.0.57 or 2.1.5, or if the permissions for the staging bucket are missing.
Added recommendation details in Autoscaler Stackdriver logs for the CANCEL and DO_NOT_CANCEL recommendations.
New Dataproc Serverless for Spark runtime versions:
Fixed a race condition in Spark startup that could lead to nodes failing to initialize when using premium disk tier.
New Dataproc on Compute Engine image versions, which includes a 2.1.18-ubuntu20-arm image that supports ARM machine types:
New Dataproc Serverless for Spark runtime versions:
Upgraded the Cloud Storage connector version to 2.2.16 in Dataproc Serverless for Spark runtimes.
Clusters that use a driver node group now configure YARN queues with user-limit-factor set to 2, allowing for a single user to burst to 2x utilization of capacity, which is set to 50. This achieves better resource utilization for workloads submitted by a single user.
New Dataproc on Compute Engine image versions:
Upgraded the Cloud Storage connector version to 2.2.16 for Dataproc on Compute Engine 2.0 and 2.1 images.
Dataproc Serverless Spark 1.1 and 2.0 runtime subminor versions can now be used 365 days after their release (instead of 90 days).
The goog-dataproc-batch-id, goog-dataproc-batch-uuid and goog-dataproc-location labels are now automatically applied to Dataproc Serverless batch resources.
Dataproc Serverless for Spark now supports updating the BigQuery connector using the dataproc.sparkBqConnector.version and dataproc.sparkBqConnector.uri properties
see Use the BigQuery connector with Dataproc Serverless for Spark.
New Dataproc Serverless for Spark runtime versions:
Added support for Premium compute and storage pricing tiers for Dataproc Serverless Spark workloads. Premium compute offers higher performance per core, and Premium storage offers higher throughput and IOPs. To use Premium compute and storage, set the following Spark runtime environment properties:
spark.dataproc.(driver|executor).compute.tier=premiumspark.dataproc.(driver|executor).storage.tier=premium.New Dataproc Serverless for Spark runtime versions:
Backported ZEPPELIN-5755 to Zeppelin 0.10 in 2.1 images for Spark 3.3 support.
New Dataproc on Compute Engine subminor image versions:
Added Dataproc Serverless Templates for batch creation:
New Dataproc Serverless for Spark runtime versions:
New Dataproc on Compute Engine subminor image versions:
Fixed a bug that caused cluster creation to fail when ATSv2 is enabled for tables that have a garbage collection policy setup other than maxversions.
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
Upgrade Cloud Storage connector to 2.2.14 version in Dataproc Serverless for Spark runtimes.
New Dataproc Serverless for Spark runtime versions:
Upgrade Cloud Storage connector version to 2.2.14 for 2.0 and 2.1 images
New sub-minor versions of Dataproc images:
Backport HIVE-22891, HIVE-21660, HIVE-21915 to 2.0 images.
Backport HIVE-22891, HIVE-21660, HIVE-25520, HIVE-25521 to 2.1 images.
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions:
Unauthorized callers attempting to get, delete, or terminate non-existent Sessions will now receive a 403 response code instead of a 404 response code. This does not impact authorized callers.
Fixed Serverless history server endpoint URL when Persistent History Server (PHS) was setup without using a wildcard.
New Dataproc Serverless for Spark runtime versions:
Upgraded the Cloud Storage connector to 2.2.13 version in Dataproc Serverless for Spark runtimes.
Backported HIVE-22891 to 2.0 images.
Fixed the NoClassDefFoundError for log4j class in Zeppelin BigQuery interpreter in 2.0 images.
New sub-minor versions of Dataproc images:
You can now use --properties=dataproc:componentgateway.ha.enabled=true to enable the Dataproc Component Gateway and Knox along with the Spark History Server (SHS) UI in HA mode.
New Dataproc Serverless for Spark runtime versions:
Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime version 2.1, which includes the following components:
Upgraded Conda to 23.3 in Dataproc Serverless for Spark runtime 2.1.
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
hive principal will be used for Hive catalog queries via presto in kerberos cluster.
New sub-minor versions of Dataproc images:
Upgrade Spark to 3.4.0 and its dependencies in Dataproc Serverless for Spark 2.1 runtime:
Dataproc now supports the usage of cross-project service account.
Autoscaler recommendation reasoning details are available now in Cloud Logging logs.
Default batch TTL is set to 4 hours for Dataproc Serverless for Spark runtime version 2.1.
New sub-minor versions of Dataproc images:
Running Spark jobs with the DataprocFileOutoutputCommitter is now supported. Enable the committer for Spark applications that write to a Cloud Storage destination concurrently.
Add Autoscaler recommendation reasoning details in Cloud Logging.
Dataproc on GKE SLM force delete timeout exception converted to DataprocIoException.
Announcing Dataproc General Availability (GA) support for CMEK organization policy.
New Dataproc Serverless for Spark runtime versions:
Make spark user an owner for all items in the driver working directory for Dataproc Serverless for Spark workloads to fix permissions issues after Hadoop upgrade to 3.3.5.
New Dataproc Serverless for Spark runtime versions:
Upgrade Hadoop to 3.3.5 in Dataproc Serverless for Spark runtimes
Announcing the General Availability (GA) release of Key Access Justifications for Dataproc.
Dataproc is now available in the me-central1 region (Doha).
New sub-minor versions of Dataproc images:
Dataproc cluster creation now supports the pd-extreme disk type.
Dataproc on GKE now disallows update operations.
Dataproc on GKE diagnose operation now verifies that the master agent is running.
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
Upgrade Python to 3.11 and Conda to 23.1 in Dataproc Serverless for Spark runtime 2.1
Dataproc is now available in the europe-west12 region (Turin).
New Dataproc Serverless for Spark runtime versions:
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions:
Added stronger validations to disallow upper-case characters in template IDs per Resource Names guidance, which allows Workflow template creation to fail fast instead of failing at workflow template instantiation.
Added decision metric field in Stackdriver autoscaler logs.
New Dataproc Serverless for Spark runtime versions:
Release Dataproc Serverless for Spark runtime 2.1 preview:
--properties=dataproc:agent.ha.enabled=true can now be used to enable the Dataproc Agent in high availability mode. This property is supported by Dataproc Image versions 2.0 and above.
New Dataproc Serverless for Spark runtime versions:
New Dataproc Serverless for Spark runtime versions:
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions:
Add support for filters when listing batches. Batches may be filtered on one or more of batch_id, batch_uuid, state, or create_time (for example, state = RUNNING AND create_time < "2023-01-01T00:00:0Z"). See Filter expressions for more information.
Dataproc Serverless for Spark now supports unconditional TTL to batches. The workload will be terminated after the TTL without waiting for work to complete.
Dataproc Serverless for Spark now supports statically-sized Dataproc Serverless for Spark batch workloads with more than 500 executors.
Generate a warning when initialization actions are used in a cluster created with a driver node group.
The default Dataproc Serverless for Spark runtime version has changed to 2.0.
New Dataproc Serverless for Spark runtime versions:
1.0.29 is the last release of the Dataproc Serverless for Spark runtime version 1.0, it will no longer be supported and will not receive new releases.
Upgrade Cloud Storage connector to 2.2.11 version in Dataproc Serverless for Spark runtimes.
Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime version 1.1, which includes the following components:
New Dataproc Serverless for Spark runtime versions:
Dataproc Serverless for Spark runtime version 1.0 changed to non-LTS because of the release of backward-compatible Dataproc Serverless for Spark runtime version 1.1 LTS.
Announcing the General Availability (GA) release of the Dataproc Hudi Optional Component.
Announcing the General Availability (GA) release of the Dataproc driver node groups.
New Dataproc Serverless for Spark runtime versions:
Added support for enabling Hive Metastore OSS metrics by passing hivemetastore to --metric-sources property during cluster creation.
Added support for Dataproc Metastore integration with Trino.
New sub-minor versions of Dataproc images:
Upgraded Parquet to 1.12.2 for 2.1 images.
The value of hive.server2.builtin.udf.blacklist is now set by default to reflect,reflect2 in hive-site.xml to prevent arbitrary code execution.
New Dataproc Serverless for Spark runtime versions:
Backported Spark patch in Dataproc Serverless for Spark runtime 1.0 and 2.0:
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions:
General Availability (GA) release of Dataproc 2.1 images.
Upgrade Cloud Storage connector version to 2.1.9 for 1.5 images.
Upgrade Cloud Storage connector version to 2.2.9 for 2.1 images.
New sub-minor versions of Dataproc images:
New Dataproc Serverless for Spark runtime versions: 1.0.24 and 2.0.4
Dataproc Serverless for Spark runtime 1.0:
Dataproc Serverless for Spark runtime 2.0:
Use jemalloc as a default OS memory allocator in Dataproc Serverless for Spark runtime.
Backport Spark patches in Dataproc Serverless for Spark runtime 1.0 and 2.0:
Added the dataproc.googleapis.com/job/state metric to track the status of Dataproc Jobs states (such as, RUNNING or PENDING). This metric is collected by default and is not chargeable to customers.
Dataproc job IDs are now queryable and viewable from MQL(Monitoring Query Language), and the metric can be used for long-running job monitoring and alerting.
Dataproc Serverless for Spark runtime version 2.0 will become the default Dataproc Serverless for Spark runtime version on January 24, 2023 (instead of December 13, 2022, as previously announced).
Dataproc Serverless for Spark supports Spark and System metrics. These metrics are enabled by default. Spark driver and executor metrics can be customised using overrides.
Nodemanagers in DECOMMISSIONING, NEW, and SHUTDOWN state are now included in the /cluster/yarn/nodemanagers metric.
Secure Boot, Virtual trusted platform module (vTPM), and Integrity monitoring Shielded VM features are enabled by default for Dataproc on Compute Engine clusters that use 2.1 preview images.
Added support for Dataproc to attach to a gRPC Dataproc Metastore in any region.
Dataproc Serverless for Spark now shows the subminor runtime version used in the runtimeConfig.version field,
Fixed a bug that caused a Dataproc cluster with a Dataproc Metastore service to fail the creation process, if the cluster was in the same network but different subnetworks.
New sub-minor versions of Dataproc images:
1.5.77-debian10, 1.5.77-rocky8, 1.5.77-ubuntu18,
2.0.51-debian10, 2.0.51-rocky8, 2.0.51-ubuntu18,
preview 2.1.0-RC4-debian11, preview 2.1.0-RC4-rocky8, preview 2.1.0-RC4-ubuntu20.
Dataproc Serverless for Spark now now uses runtime version 1.0.23 and 2.0.3.
Backported HIVE-17317 in the latest 2.0 and 2.1 images.
Downgraded google-auth-oauthlib Python package to fix gcsfs Python package for 2.0 and 2.1 images.
Dataproc Serverless for Spark runtime version 1.0.23 and 2.0.3 downgrades google-auth-oauthlib Python package to fix gcsfs Python package.
Upgraded Apache Commons Text to 1.10.0 for Knox in 1.5 images, and for Spark, Pig, Knox in 2.0 images, addressing CVE-2022-42889.
Dataproc Serverless for Spark runtime version 1.0.23 and 2.0.3 adds PyMongo Python library.
Dataproc images 2.0.50 and preview 2.1.0-RC3 are deprecated and cluster creations based on these images will fail starting 11/11/2022. This is due to an update to the google auth library which breaks running Pyspark batch workloads having dependency on gcsfs. Upcoming image versions will have a fix to address this issue.
Dataproc Serverless for Spark runtime versions 1.0.22 and 2.0.2 will be deprecated on 11/11/2022. New batch submissions that use these runtime versions will fail starting 11/11/2022. This is due to an update to the google auth library which breaks running Pyspark batch workloads having dependency on gcsfs. Upcoming runtime versions will address this issue.
New sub-minor versions of Dataproc images:
1.5.76-debian10, 1.5.76-rocky8, 1.5.76-ubuntu18
2.0.50-debian10, 2.0.50-rocky8, 2.0.50-ubuntu18
preview 2.1.0-RC3-debian11, preview 2.1.0-RC3-rocky8, preview 2.1.0-RC3-ubuntu20,
Dataproc Serverless for Spark now now uses runtime version 1.0.22 and 2.0.2.
If a Dataproc Metastore service uses the gRPC endpoint protocol, a Dataproc or self-managed cluster located in any region can attach to the service.
Dataproc Serverless for Spark now allows the customization of driver and executor memory using the following properties:
spark.driver.memoryspark.driver.memoryOverheadspark.executor.memoryspark.executor.memoryOverheadDataproc Serverless for Spark now outputs approximate_usage after a workload finishes that shows the approximate DCU and shuffle storage resource consumption by the workload.
Removed the Auto Zone placement check for supported machine types.
The following preview Dataproc image versions are available:
The following component versions are available for use with the 2.1.0-RC2 images (the HBase and Druid components are not supported in 2.1 image versions):
Apache Atlas 2.2.0
Apache Flink 1.15.0
Apache Hadoop 3.3.3
Apache Hive 3.1.3
Apache Hive WebHCat 3.1.3
Apache Kafka 3.1.0
Apache Pig 0.18.0-SNAPSHOT
Apache Spark 3.3.0
Apache Sqoop v1 1.5.0-SNAPSHOT
Apache Sqoop v2 1.99.6
Apache Tez 0.10.1
Cloud Storage Connector hadoop3-2.2.8
Conscrypt 2.5.2
Docker 20.10
Hue 4.10.0
Java temurin-11-jdk
JupyterLab Notebook 3.4
Oozie 5.2.1
Presto 376
Python 3.10
R 4.1
Ranger 2.2.0
Scala 2.12.14
Solr 9.0.0
Zeppelin Notebook 0.10.1
Zookeeper 3.8.0
Dataproc Serverless for Spark runtime version 2.0.1 upgrades Apache Commons Text to 1.10.0, addressing CVE-2022-42889
Dataproc Serverless for Spark runtime version 2.0.1 upgrades the following components:
Dataproc Serverless for Spark now now uses runtime version 1.0.21 and 2.0.1.
All Dataproc Serverless for Spark runtime versions prior to 1.0.21 and 2.0.1 will be deprecated on November 2, 2022.
Dataproc Serverless for Spark runtime version 2.0 will become the default Dataproc Serverless for Spark runtime version on December 13, 2022.
Dataproc Serverless for Spark now supports spark.dataproc.diagnostics.enabled property that enables auto diagnostics on Batch failure. Note that enabling auto diagnostics will hold compute and storage quota after Batch is complete and until diagnostics is finished.
Announcing the General Availability (GA) release of Dataproc Serverless for Spark runtime 2.0.
New sub-minor versions of Dataproc images:
1.5.75-debian10, 1.5.75-rocky8, 1.5.75-ubuntu18
2.0.49-debian10, 2.0.49-rocky8, 2.0.49-ubuntu18
Dataproc Serverless for Spark runtime version 2.0.0 upgrades the following components:
Upgraded Cloud Storage connector version to 2.2.8 in the latest 2.0 images.
Dataproc Serverless for Spark now uses runtime version 1.0.20 and 2.0.0.
Upgraded the Conscrypt library to 2.5.2 in the latest 1.5 and 2.0 images.
Disabled auto deletion of files under /tmp in the latest Rocky images. Previous Rocky images have files in the /tmp folder deleted every 10 days due to default OS system setting in /usr/lib/tmpfiles.d/tmp.conf.
Set yarn:spark.yarn.shuffle.stop_on_failure to true by default in the latest 1.5 and 2.0 images. This change causes YARN node manager startup to fail if the Spark external shuffle service startup fails. On VM boot, Dataproc will continuously restart the YARN node manager until it is able to start. This change reduces Spark executor errors, such as: org.apache.spark.SparkException: Unable to register with external shuffle server due to : Failed to connect to <worker host>:7337, particularly when starting a stopped cluster. See Spark external shuffle service documentation for details.
Changed Hive TokenStoreDelegationTokenSecretManager in the latest 1.5 and 2.0 images so that it updates the base class's current key ID after generating a new master key. This is important for users of DBTokenStore, which generates key IDs based on a monotonically increasing sequence from the database. Prior to this fix, there was a race condition during master key rollover that could cause it to attempt updating the prior master key using an incorrect ID value. This would fail and then quickly retry, sometimes multiple times, causing too many rows in the database.
Addressing CVE-2022-23305, CVE-2022-23302, CVE-2021-4104, CVE-2019-17571, migrated log4j 1.2 to reload4j for Hadoop, Spark in the latest 1.5 images and Hadoop, Spark, ZooKeeper, Oozie, Knox in the latest 2.0 images.
Enabled Spark authentication and encryption for Kerberos enabled clusters created with the latest 1.5 and 2.0 images.
Backported the patch for HADOOP-18316 in the latest 2.0 images, addressing CVE-2022-25168.
Set HDFS /user/<name> directory permission with owner=<name> and mode=700 for Kerberos enabled clusters created with the latest 1.5 and 2.0 images.
Backported the patch for HIVE-25468 in the latest 1.5 and 2.0 images, addressing CVE-2021-34538.
Backported the patch for HIVE-20607 in the latest 2.0 images.
Backported the patch for HIVE-19310 in the latest 1.5 images.
Fixed an issue in which Presto queries might fail when submitted to HA clusters in the latest 1.5 and 2.0 images.
Backported the patch for HIVE-26447 in the latest 2.0 images.
Backported the patch for SPARK-36383 in the latest 2.0 images.
Fixed a "gsutil not found" issue in the latest 1.5 and 2.0 Ubuntu images.
Fixed a bug where metrics created based on the yarn:yarn.resourcemanager.metrics.runtime.buckets property were not exported to Cloud Monitoring, even though listed in --metric-overrides during cluster creation.
Backported the patch for HIVE-20004 in the latest 2.0 images.
Dataproc is now available in the me-west1 region (Tel Aviv, Israel).
Preemptible SPOT VMs can be used as secondary workers in a Dataproc cluster. Unlike legacy preemptible VMs with a 24-hour maximum lifetime, Spot VMs have no maximum lifetime.
Dataproc Serverless for Spark now supports Artifact Registry with image streaming.
Dataproc Metastore: Fixed an endpoint resolution issue that caused 500 type errors for valid setups. The service was overly aggressive in describing networks and subnetworks attached to the service via the NetworkConfig field.
Dataproc Auto Zone Placement now takes ANY reservation into account by default.
Dataproc Serverless for Spark now uses runtime version 1.0.19 and 2.0.0-RC4, which also upgrades both runtimes to Cloud Storage Connector to 2.2.8.
Dataproc Serverless for Spark:
You can now use the spark.dynamicAllocation.executorAllocationRatio property to configure how aggressively to scale up Serverless workloads. A value of 1.0 provides maximum scale up.
Dataproc Serverless for Spark: Increased initial and maximum Spark executor limits to 500 and 2,000, respectively.
Dataproc Serverless for Spark: Sets a maximum limit of 500 workers per scale up or scale down operation.
Dataproc Serverless for Spark: Reduced the latency between batch workload completion and when a batch is marked SUCCEEDED.
Dataproc on Compute Engine: Stop all master and worker VMs when starting a cluster fails due to stockout or insufficient quota.
Dataproc Serverless for Spark now uses runtime version 1.0.18 and 2.0.0-RC3.
Dataproc Serverless for Spark now uses runtime version 1.0.17 and 2.0.0-RC2.
Avoid using the following image versions when creating new clusters:
2.0.31-debian10, 2.0.31-ubuntu18, 2.0.31-rocky82.0.32-debian10, 2.0.32-ubuntu18, 2.0.32-rocky82.0.33-debian10, 2.0.33-ubuntu18, 2.0.33-rocky81.5.57-debian10, 1.5.57-ubuntu18, 1.5.57-rocky81.5.58-debian10, 1.5.58-ubuntu18, 1.5.58-rocky81.5.59-debian10, 1.5.59-ubuntu18, 1.5.59-rocky8If your cluster uses one of these image versions, there is a small chance that the cluster might enter an ERROR_DUE_TO_UPDATE state while being updated, either manually or as a result of autoscaling. If that happens, contact support. You can avoid future occurrences by creating new clusters with a newer image version.
Fixed issue where gcloud dataproc batches list hangs when a large
number of batches is present.
Announcing the Preview release of Dataproc custom constraints, which can be used to allow or deny specific operations on Dataproc clusters.
Announcing Dataproc Serverless for Spark preview runtime version 2.0.0-RC1, which includes the following components:
Dataproc Serverless for Spark now uses runtime version 1.0.16, which upgrades the following components to the following versions:
New sub-minor versions of Dataproc images:
1.5.73-debian10, 1.5.73-rocky8, 1.5.73-ubuntu18
2.0.47-debian10, 2.0.47-rocky8, 2.0.47-ubuntu18
Dataproc on Compute Engine images now have master VM memory protection enabled by default. Jobs may be terminated to prevent the master VM running out of memory.
Enabled Spark authentication and encryption for Kerberos clusters in 1.5 and 2.0 images.
Dataproc Serverless for Spark now uses runtime version 1.0.15, which upgrades the following Spark dependencies to the following versions:
FallbackHiveAuthorizerFactory is now set by default on newly created 1.5 and 2.0 image clusters that have the any of the following features enabled:
If you encounter a Cannot modify <PARAM> or similar runtime error when running a SET statement in a Hive query, this means the parameter is not in list of allowable runtime parameters. You can allow the parameter using
hive.security.authorization.sqlstd.confwhitelist.append as a cluster property when you create a cluster.
Example:
--properties="hive:hive.security.authorization.sqlstd.confwhitelist.append=tez.application.tags,<ADDITIONAL-1>,<ADDITIONAL-2>"
Default MySQL instance root password changed to a random value in 1.5 and 2.0 images. New password is now stored in MySQL configuration file accessible only by the OS level root user.
New sub-minor versions of Dataproc images:
1.5.72-debian10, 1.5.72-rocky8, 1.5.72-ubuntu18
2.0.46-debian10, 2.0.46-rocky8, 2.0.46-ubuntu18
Upgraded Hadoop to version 2.10.2 version 2.10.2 in 1.5 images.
Upgraded Hadoop to version 3.2.3 in 2.0 images.
Backported the patches for HIVE-19047 and HIVE-19048 in 1.5 images.
Backported the patch for HIVE-19048 in 2.0 images.
Backported the patch for KNOX-1997 in 2.0 images.
Dataproc support for the following images has been extended to the following dates:
New sub-minor versions of Dataproc images:
1.5.71-debian10, 1.5.71-rocky8, 1.5.71-ubuntu18
2.0.45-debian10, 2.0.45-rocky8, 2.0.45-ubuntu18
For 1.5 images and the 2.0.45-ubuntu18 image, backported the upstream fix for KNOX-1997.
New sub-minor versions of Dataproc images:
1.5.70-debian10, 1.5.70-rocky8, 1.5.70-ubuntu18
2.0.44-debian10, 2.0.44-rocky8, 2.0.44-ubuntu18
Dataproc Metastore: For 1.5 images, added a spark.hadoop.hive.eager.fetch.functions.enabled Spark Hive client property to control whether the client fetches all functions from Hive Metastore during initialization. The default setting is true, which preserves the existing behavior of fetching all functions. If set to false, the client will not fetch all functions during initialization, which can help reduce high latency during initialization, particularly when there are many functions and the Metastore is not located in the client's region.
For 1.5 and 2.0 images, backported YARN-9608 to fix the issue in graceful decommissioning.
New sub-minor versions of Dataproc images:
1.5.69-debian10, 1.5.69-rocky8, 1.5.69-ubuntu18
2.0.43-debian10, 2.0.43-rocky8, 2.0.43-ubuntu18
Backported the patch for HBASE-23287 to HBase 1.5.0 in 1.5 image
Announcing the General Availability (GA) release of Dataproc Custom OSS Metrics GA, which collects then integrates Dataproc cluster OSS component metrics into Cloud Monitoring.
Announcing the General Availability (GA) release of Ranger Cloud Storage plugin. This plugin activates an authorization service on each Dataproc cluster VM, which evaluates requests from the Cloud Storage connector against Ranger policies and, if the request is allowed, returns an access token for the cluster VM service account
Dataproc is now available in the us-south1 region (Dallas, Texas).
Announcing the General Availability (GA) release of Dataproc Persistent History Server, which provides web interfaces to view job history for jobs run on active or deleted Dataproc clusters.
Dataproc Serverless for Spark now uses runtime version 1.0.13.
New sub-minor versions of Dataproc images:
1.5.68-debian10, 1.5.68-rocky8, 1.5.68-ubuntu18
2.0.42-debian10, 2.0.42-rocky8, 2.0.42-ubuntu18
Dataproc Serverless for Spark runtime versions 1.0.2, 1.0.3 and 1.0.4 are unavailable for new batch submissions.
Upgrade Cloud Storage connector version 2.1.8 for 1.5 images only.
Dataproc on GKE Spark 3.1 images upgraded to Spark version 3.1.3.
Fixed a bug where HDFS directories initialization could fail when user names in a project contain special characters.
Fix a Dataproc on GKE bug that caused upload of driver logs to Cloud Storage to fail.
Dataproc is now available in the us-east5 region (Columbus, Ohio).
Dataproc is now available in the europe-west9 region (Paris, France).
Dataproc is now available in the europe-southwest1 region (Madrid, Spain).
New sub-minor versions of Dataproc images:
1.5.67-debian10, 1.5.67-ubuntu18, 1.5.67-rocky8
2.0.41-debian10, 2.0.41-ubuntu18, 2.0.41-rocky8
Dataproc on GKE error messages now provide additional information.
Backported fixes for HIVE-22098, HIVE-23809, HIVE-20462, HIVE-21675 to Hive 3.1 in Dataproc 2.0 images.
Fixed a bug where older-image (for example, 1.3.95) cluster create operations failed with the error message : "does not support specifying local SSD interface other than 'SCSI'".
Fix a bug where properties related to Kerberos cross realm trust were not properly set.
New sub-minor versions of Dataproc images:
1.5.66-debian10, 1.5.66-ubuntu18, 1.5.66-rocky8
2.0.40-debian10, 2.0.40-ubuntu18, 2.0.40-rocky8
Upgraded Spark to 3.1.3 in Dataproc image version 2.0.
Fixed a bug where job was not being marked as terminated after master node reboot.
Backported the fix for HIVE-20514 to Hive 2.3 in Dataproc image version 1.5.
Fixed a bug with HDFS directories initialization when core:fs.defaultFS is set to an external HDFS.
Fixed a bug where Flink was not able to run on HA clusters.
Dataproc Serverless for Spark now uses runtime version 1.0.12.
New sub-minor versions of Dataproc images:
1.5.65-debian10, 1.5.65-ubuntu18, 1.5.65-rocky8
2.0.39-debian10, 2.0.39-ubuntu18, 2.0.39-rocky8
Fixed an issue where chronyd systemd service failed to start due to a race condition between systemd-timesyncd and chronyd.
Dataproc Serverless for Spark runtime version 1.0.1 is unavailable for new batch submissions.
Dataproc Serverless for Spark now uses runtime version 1.0.11.
New sub-minor versions of Dataproc images:
1.5.64-debian10, 1.5.64-ubuntu18, 1.5.64-rocky8
2.0.38-debian10, 2.0.38-ubuntu18, 2.0.38-rocky8
If you request to cancel a job in one of the following states, Dataproc will return the job, but not initiate cancellation, since it is already in progress: CANCEL_PENDING, CANCEL_STARTED, or CANCELLED.
Added Dataproc Serverless support for updating the Cloud Storage connector using the dataproc.gcsConnector.version and dataproc.gcsConnector.uri properties.
When submitting a Dataproc job or workflow that selects a cluster that matches the specified labels, Dataproc will avoid choosing clusters that are in a state that disallows running jobs. Specifically, Dataproc will only choose among clusters in one of the following states: RUNNING, UPDATING, CREATING, or ERROR_DUE_TO_UPDATE.
Dataproc correctly defaults NodePool locations when the GKE cluster is in us-east1 and europe-west1.
Dataproc Serverless for Spark runtime version 1.0.0 is unavailable for new batch submissions.
Cloud Storage connector version upgraded to 2.2.6 in image version 2.0.
New sub-minor versions of Dataproc images:
1.5.63-debian10, 1.5.63-ubuntu18, 1.5.63-rocky8
2.0.37-debian10, 2.0.37-ubuntu18, 2.0.37-rocky8
Dataproc Serverless for Spark now uses runtime version 1.0.10.
Hive: Bundle threeten classes in hive-exec.jar in image version 2.0. ORC now requires date handling classes in the org.threeten package, which are not present in hive-exec.jar at query time.
HIVE-22589 fixed this bug upstream, but it was part of a large new feature. Instead, this change applies a small targeted fix to address the bug.
Dataproc is now available in the europe-west8 region (Milan, Italy).
Announcing the General Availability (GA) release of Dataproc on GKE, which allows you to execute Big Data applications using the Dataproc jobs API on GKE clusters.
The dataproc:dataproc.performance.metrics.listener.enabled cluster property, which is enabled by default, listens on port 8791 on all master nodes to extract performance-related telemetry Spark metrics. The metrics are published to the Dataproc service for it to use to set better defaults and improve the service. To opt-out of this feature, set dataproc:dataproc.performance.metrics.listener.enabled=false when creating a Dataproc cluster.
New sub-minor versions of Dataproc images:
1.5.62-debian10, 1.5.62-ubuntu18, and 1.5.62-rocky8
2.0.36-debian10, 2.0.36-ubuntu18, and 2.0.36-rocky8
Fixed and issue in which the Dataproc autoscaler would sometimes try to scale down a cluster by more than one thousand secondary worker nodes at one time. Now, the autoscaler will scale down at most one thousand nodes at one time. In cases where the autoscaler previously would have scaled down more than one thousand nodes at one time, it will scale down the nodes by at most one thousand nodes, and a log will be written to the autoscaler log noting this occurrence.
Changed the owner of /usr/lib/knox/conf/gateway-site.xml from root:root to knox:knox.
Fixed bugs that could cause Dataproc to delay marking a job cancelled.
New sub-minor versions of Dataproc images:
1.5.61-debian10, 1.5.61-ubuntu18, and 1.5.61-rocky8
2.0.35-debian10, 2.0.35-ubuntu18, and 2.0.35-rocky8
Changed the owner of /var/lib/zookeeper/myid from root to zookeeper.
New sub-minor versions of Dataproc images:
1.5.60-debian10, 1.5.60-ubuntu18, and 1.5.60-rocky8
2.0.34-debian10, 2.0.34-ubuntu18, and 2.0.34-rocky8
New sub-minor versions of Dataproc images:
1.5.59-debian10, 1.5.59-ubuntu18, and 1.5.59-rocky8
2.0.33-debian10, 2.0.33-ubuntu18, and 2.0.33-rocky8
New sub-minor versions of Dataproc images:
1.5.58-debian10, 1.5.58-ubuntu18, and 1.5.58-rocky8
2.0.32-debian10, 2.0.32-ubuntu18, and 2.0.32-rocky8
Fixed bug where clusters created with Kerberos failed with SSL
Certificate string is too long error.
Fixed bug where clusters created via Dataproc Hub failed with Unit
file jupyter.service does not exist error.
Added support for Enhanced Flexibility Mode (EFM) with primary worker shuffle mode on Spark for image version 2.0.
General Availability (GA) release of new Rocky Linux based images: 1.5.57-rocky8 and 2.0.31-rocky8. These images are replacing CentOS images which are EOL.
Upgraded Cloud Storage connector version to 2.1.7 in image version 1.5.
CentOS images are EOL. 1.5.56-centos8 and 2.0.30-centos8 are the final CentOS based images. CentOS images are no longer supported and will not receive new releases.
Dataproc Serverless for Spark now uses runtime version 1.0.4, which updates GCS connector to 2.2.5 version.
New sub-minor versions of Dataproc images:
1.5.57-debian10, 1.5.57-ubuntu18, and 1.5.57-rocky8
2.0.31-debian10, 2.0.31-ubuntu18, and 2.0.31-rocky8
Upgraded Cloud Storage connector version to 2.2.5 in image version 2.0.
A script that checks if a project or organization is using an unsupported Dataproc image is available for downloading (see Unsupported Dataproc versions).
Dataproc images prior to 1.3.95, 1.4.77, 1.5.53, and 2.0.27 are deprecated and cluster creations based on these images will fail starting 2/28/2022.
Added cluster_type field to job and operation metrics in Cloud Monitoring.
Enabled the Resource Manager UI and HA capable UIs in HA cluster mode.
1.4.80-debian10 and 1.4.80-ubuntu18 are the last releases for the 1.4 images. Dataproc 1.4 images will no longer be supported and will not receive new releases.
Configured Zeppelin Spark interpreter to run in YARN client mode by default for image version 2.0.
New sub-minor versions of Dataproc images:
1.4.80-debian10 and 1.4.80-ubuntu18
1.5.56-debian10, 1.5.56-ubuntu18, and 1.5.56-centos8
2.0.30-debian10, 2.0.30-ubuntu18, and 2.0.30-centos8
Dataproc Serverless for Spark now uses runtime version 1.0.2, which updates Spark to 3.2.1 version.
Dataproc Serverless for Spark now uses runtime version 1.0.1, which includes improved error messaging for network connectivity issues.
Announcing the General Availability (GA) release of Dataproc Serverless for Spark, which allows you to run your Spark jobs on Dataproc without having to spin up and manage your own cluster.
Added support for Dataproc Metastore's beta NetworkConfig field. Beta services using this field can now be used in conjunction with v1 Dataproc clusters.
Dataproc extracts the warehouse directory from the Dataproc Metastore service for the cluster-local warehouse directory.
New sub-minor versions of Dataproc images:
1.4.79-debian10 and 1.4.79-ubuntu18
1.5.55-debian10, 1.5.55-ubuntu18, and 1.5.55-centos8
2.0.29-debian10, 2.0.29-ubuntu18, and 2.0.29-centos8
The Cloud Storage connector jar is installed on the Solr server (even if dataproc:solr.gcs.path property is not set). Applies to image versions 1.4, 1.5, and 2.0.
Dataproc images 1.4.79, 1.5.55, and 2.0.29, listed above, were updated with log4j version 2.17.1. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.
Fixed a bug where cluster restart disabled Solr and Ranger services even if the components are selected. Applies to image versions 1.4, 1.5, and 2.0.
Migrated to Eclipse Temurin JDK in image versions 1.4, 1.5, and 2.0.
RANGER-3324: Make optimized db schema script idempotent for MySQL DB. Applies to 2.0 images.
YARN-8865: RMStateStore contains large number of expired RMDelegationToken. Applies to 1.5 images.
New sub-minor versions of Dataproc images:
1.4.78-debian10, and 1.4.78-ubuntu18
1.5.54-centos8, 1.5.54-debian10, and 1.5.54-ubuntu18
2.0.28-centos8, 2.0.28-debian10, and 2.0.28-ubuntu18
Dataproc images 1.4.78, 1.5.54, and 2.0.28, listed above, were updated with log4j version 2.17.0. It is strongly recommended that your clusters use previously released images 1.4.77, 1.5.53, or 2.0.27, or higher (see Supported Dataproc versions). While not urgent nor strongly recommended, Dataproc advises you to create or recreate Dataproc clusters with the latest sub-minor image versions when possible.
Fixed the problem that jars added with the --jars flag in gcloud dataproc jobs submit spark-sql are missing at runtime.
Upgraded Cloud Storage connector version to 2.2.4 in image version 2.0.
Dataproc has released 1.3.95-debian10/-ubuntu18 images with a one-time patch that addresses the Apache Log4j 2 CVE-2021-44228 and CVE-2021-45046 vulnerabilities, but note that that all 1.3 images remain unsupported, and Dataproc will not provide upgrades to 1.3 images .
Removed the Geode interpreter from Zeppelin notebook, which is affected by https://nvd.nist.gov/vuln/detail/CVE-2021-45046.
Dataproc has released the following sub-minor image versions to address an Apache Log4j 2 vulnerability (also see Create a cluster and Recreate and update a cluster for more information). Note: These images supersede the 1.5 and 2.0 images listed in the December 16, 2021 release note:
1.5.53-centos8, 1.5.53-debian10, 1.5.53-ubuntu18,
2.0.27-centos8, 2.0.27-debian10, 2.0.27-ubuntu18
Upgraded log4j version to 2.16.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-44228.
Dataproc has released the following [sub-minor image versions](https://cloud.google.com/dataproc/docs/concepts/versioning/dataproc-versions:
1.4.77-debian10, 1.4.77-ubuntu18,
1.5.52-centos8, 1.5.52-debian10, 1.5.52-ubuntu18,
2.0.26-centos8, 2.0.26-debian10, 2.0.26-ubuntu18
Dataproc has added new images, listed in this release note, to address an Apache Log4j 2 vulnerability.
Note: these images have been superseded. by the 12/16/21 images (see the December 16, 2021 release note). Also see Create a cluster and Recreate and update a cluster for more information.
Upgraded log4j version to 2.15.0, which fixes https://nvd.nist.gov/vuln/detail/CVE-2021-45046.
New sub-minor versions of Dataproc images:
1.4.76-debian10, 1.4.76-ubuntu18,
1.5.51-centos8, 1.5.51-debian10, 1.5.51-ubuntu18,
2.0.25-centos8, 2.0.25-debian10, 2.0.25-ubuntu18
HIVE-21040: msck does unnecessary file listing at last level of directory tree. Applies to 1.5 and 2.0 images.
Fixed a bug where driver log links on PHS Web UI stop working once the job cluster is deleted. Applies to 1.4 and 1.5 images.
Fixed executor log links on Spark History Server Web UI for running and completed applications. Applies to 1.4 and 1.5 images.
SPARK-35817: Queries against wide Avro tables can be slow. Applies to 2.0 images.
SPARK-7768: Make user-defined type (UDT) API public. Applies to 2.0 images.
YARN-8990: Fixed a Fairscheduler race condition. Applies to 2.0 images.
Dataproc is now available in the southamerica-west1 region (Santiago, Chile).
Added the following new Apache Spark properties to control Cloud Storage flush behavior for event logs for 1.4 and later images:
spark.history.fs.gs.outputstream.type (default: BASIC) spark.history.fs.gs.outputstream.sync.min.interval.ms (default: 5000ms). Note: The default configuration of these properties enables the display of running jobs in the Spark History Server UI for clusters using Cloud Storage to store spark event logs.
Added support in 1.5 and 2.0 images to filter Spark Applications on the Spark History Server Web UI based on Cloud Storage path. Filtering is accomplished using the eventLogDirFilter parameter, which accepts any Cloud Storage path substring and will return applications that match the Cloud Storage path.
New sub-minor versions of Dataproc images:
1.4.75-debian10, 1.4.75-ubuntu18,
1.5.50-centos8, 1.5.50-debian10, 1.5.50-ubuntu18,
2.0.24-centos8, 2.0.24-debian10, 2.0.24-ubuntu18
The previous Dataproc on GKE beta documentation has been replaced with a Dataproc on GKE private preview sign up form. Existing beta customers can continue using the beta release, but note that the beta release is planned to be deprecated and removed.
Removed Apache Iceberg and Delta Lake libraries in 2.0 images because they are not compatible with Spark 3.1.
Upgraded Cloud Storage connector to version 2.2.3 on 2.0 Images.
Backported SPARK-23182 in 1.4 and 1.5 images. This prevents long-running Spark shuffle servers from leaking connections when they are not cleanly terminated.
Backported SPARK-31946 in 2.0 images.
Patched Hive in 2.0 images with HIVE-20187, which fixes a bug where Hive returned incorrect query results when hive.convert.join.bucket.mapjoin.tez is set to true.
Fixed stdout and stderr links in the Spark History Server Web UI in 2.0 images.
The dataproc:dataproc.cluster-ttl.consider-yarn-activity cluster property is now set to true by default for image versions 1.4.64+, 1.5.39+, and 2.0.13+. With this change, with clusters created with these image versions, Dataproc Cluster Scheduled Deletion by default will consider YARN activity, in addition to Dataproc Jobs API activity, when determining cluster idle time . This change does not affect clusters with images with lower version numbers: cluster idle time for those clusters will continue to be computed based on Dataproc Jobs API activity only. When using image versions 1.4.64+, 1.5.39+, and 2.0.13+, you can opt out of this changed behavior by setting this property to false when you create the cluster.
In a future announcement (on approximately October 22, 2021), Dataproc will announce that Cluster Scheduled Deletion by default will consider YARN activity, in addition to Dataproc Jobs API activity, when determining cluster idle time. This change will affect image versions 1.4.64+, 1.5.39+, and 2.0.13+. To test this feature now, create a cluster with a recent image, setting the dataproc:dataproc.cluster-ttl.consider-yarn-activity cluster property to true. Note: After this behavior becomes the default, you can opt out when you create a cluster by setting the property to false.
Fixed an issue where complete YARN container logs were not visible in 1.5 and 2.0 Images.
New sub-minor versions of Dataproc images:
1.4.73-debian10, 1.4.73-ubuntu18,
1.5.48-centos8, 1.5.48-debian10, 1.5.48-ubuntu18,
2.0.22-centos8, 2.0.22-debian10, 2.0.22-ubuntu18
HADOOP-15129: Fixed in 2.0 Images: Datanode cached namenode DNS lookup failure and could not startup on.
Updated August 19, 2021 release notes with cluster creation Failure Action feature.
New sub-minor versions of Dataproc images: 1.4.71-debian10, 1.4.71-ubuntu18, 1.5.46-centos8, 1.5.46-debian10, 1.5.46-ubuntu18, 2.0.20-centos8, 2.0.20-debian10, 2.0.20-ubuntu18
Added support for enabling/disabling Ubuntu Snap daemon with cluster property dataproc:dataproc.snap.enabled. The default value is "true". If set to "false", pre-installed Snap packages in the image won't be affected, but auto refresh will be disabled. Applies to all Ubuntu images.
HIVE-21018: Grouping/distinct on more than 64 columns should be possible. Applies to 2.0 images.
The following previously released sub-minor versions of Dataproc images included a bug where the dataproc user account was broken. This prevented some Dataproc services from functioning properly, which resulted in features being unavailable. In particular, this prevented Jupyter from running in clusters with Personal Cluster Authentication enabled.
These sub-minor versions have been rolled back, and can only be used when updating existing clusters that already use them:
Added additional messages to the error messages for networking and IAM errors when creating a new cluster.
Image 2.0:
OOZIE-3599: Upgraded Jetty version to 9.4.
New sub-minor versions of Dataproc images: 1.4.70-debian10, 1.4.70-ubuntu18, 1.5.45-centos8, 1.5.45-debian10, 1.5.45-ubuntu18, 2.0.19-centos8, 2.0.19-debian10, 2.0.19-ubuntu18
Backported SPARK-34295: Added a new spark.yarn.kerberos.renewal.excludeHadoopFileSystemsconfiguration option.
New sub-minor versions of Dataproc images: 1.4.69-debian10, 1.4.69-ubuntu18, 1.5.44-centos8, 1.5.44-debian10, 1.5.44-ubuntu18, 2.0.18-centos8, 2.0.18-debian10, and 2.0.18-ubuntu18.
Backported the following Trino (PrestoSQL) BigQuery connector patches in image 2.0:
DestinationTableBuilder to BigQueryClientinformation_schema query failures in BigQuery connectorCREATE and DROP SCHEMA in BigQueryBigQuerySqlExecutor and refactor tests to use itview_definition system table for BigQuery viewImmutableMap to MapTestBigQueryIntegrationSmokeTest.testShowCreateTablepresto-tests from Druid, BigQuerypresto-bigquerybigquery.credentials-key configConfigured YARN ResourceManager to use port 8554 and Druid to use port 17071 for JMX Remote RMI port.
Users can now help assure the successful creation of a cluster by automatically deleting any failed primary workers (the master(s) and at least two primary workers must be successfully provisioned for cluster creation to succeed). To delete any failed primary workers when you create a cluster:
Using gcloud: Set the gcloud dataproc clusters create --action-on-failed-primary-workers flag to "DELETE".
Using the Dataproc clusters.create API: Set the actionOnFailedPrimaryWorkers field to "DELETE".
Added support for Dataproc Metastore in three recently turned up regions: .europe-west1, northamerica-northeast1, and asia-southeast1.
Dataproc issues a warning message if the staging or test bucket name contains an underscore.
New sub-minor versions of Dataproc images: 1.4.68-debian10, 1.4.68-ubuntu18, 1.5.43-centos8, 1.5.43-debian10, 1.5.43-ubuntu18, 2.0.17-centos8, 2.0.17-debian10, and 2.0.17-ubuntu18.
Upgrade Flink to version 1.12.5 in image 2.0.
HIVE-2527: Fixed slow Hive partition deletion for Cloud Object Stores with expensive ListFiles.
Fixed Jupyter startup on Personal Auth clusters on all images.
New sub-minor versions of Dataproc images: 1.4.67-debian10, 1.4.67-ubuntu18, 1.5.42-centos8, 1.5.42-debian10, 1.5.42-ubuntu18, 2.0.16-centos8, 2.0.16-debian10, and 2.0.16-ubuntu18.
SPARK-28290: Fixed an issue where Spark History Server failed to serve because of a wild card certificate in the 1.4 and 1.5 images.
Dataproc is now available in the northamerica-northeast2 region (Toronto).
1.3 images are no longer supported and will not receive new releases.
New sub-minor versions of Dataproc images: 1.4.66-debian10, 1.4.66-ubuntu18, 1.5.41-centos8, 1.5.41-debian10, 1.5.41-ubuntu18, 2.0.15-centos8, 2.0.15-debian10, and 2.0.15-ubuntu18.
In image 2.0, set the mapreduce.fileoutputcommitter.algorithm.version=2 property in Spark. This makes Spark commit algorithm version consistent with prior Dataproc image versions.
New sub-minor versions of Dataproc images: 1.3.94-debian10, 1.3.94-ubuntu18, 1.4.65-debian10, 1.4.65-ubuntu18, 1.5.40-centos8, 1.5.40-debian10, 1.5.40-ubuntu18, 2.0.14-centos8, 2.0.14-debian10, and 2.0.14-ubuntu18.
The following component versions were updated in image 2.0:
Fixed a rare bug that sometimes happened when scaling down the number of secondary workers in a cluster in which the update operation would fail with error 'Resource is not a member of' or 'Cannot delete instance that was already deleted'.
Announcing the General Availability (GA) release of Dataproc Enhanced Flexibility Mode.
New sub-minor versions of Dataproc images: 1.3.93-debian10, 1.3.93-ubuntu18, 1.4.64-debian10, 1.4.64-ubuntu18, 1.5.39-centos8, 1.5.39-debian10, 1.5.39-ubuntu18, 2.0.13-centos8, 2.0.13-debian10, and 2.0.13-ubuntu18.
Upgraded Cloud Storage connector to version 2.2.2 on 2.0 images.
Fixed an issue on 1.4 and 1.5 images where temporary shuffle data could be leaked when running Enhanced Flexibility Mode (EFM) with Spark.
Fixed Hue installation on Ubuntu 2.0 images.
For 2.0+ image clusters, the dataproc:dataproc.master.custom.init.actions.mode cluster property can be set to RUN_AFTER_SERVICES to run initialization actions on the master after HDFS and any services that depend on HDFS are initialized. Examples of HDFS-dependent services include: HBase, Hive Server2, Ranger, Solr, and the Spark and MapReduce history servers. Default: RUN_BEFORE_SERVICES.
Custom image limitation: New images announced in the Dataproc release notes are not available for use as the base for custom images until one week from their announcement date.
The Dataproc v1beta2 APIs are deprecated. Please use the Dataproc v1 APIs.
The end date of support for Dataproc image version 1.4 has been extended from August, 2021 to November, 2021.
Upgraded Spark version to 2.4.8 in the following images:
Upgrade HBase to version 2.2.7 in image 2.0.
New sub-minor versions of Dataproc images: 1.3.92-debian10, 1.3.92-ubuntu18, 1.4.63-debian10, 1.4.63-ubuntu18, 1.5.38-centos8, 1.5.38-debian10, 1.5.38-ubuntu18, 2.0.12-centos8, 2.0.12-debian10, and 2.0.12-ubuntu18.
Minimum boot disk sizes for Dataproc images:
Fixed stdout/stderr links on Spark History Server Web UI of the Persistent History Server in the following images:
Fixed a bug where personal auth credentials would not propagate to every VM in the cluster if VPC service controls were enabled.
Dataproc is now available in the asia-south2 region (Delhi).
The following previously released sub-minor versions of Dataproc images have been rolled back and can only be used when updating existing clusters that already use them:
Introduced a new ERROR_DUE_TO_UPDATE state, which indicates a cluster has encountered an irrecoverable error while scaling. Clusters in this state cannot be scaled, but can accept jobs.
Added support for Dataproc Metastore in three new recently turned up regions: europe-west3, us-west1, and us-east1.
Fixed an issue where a spurious unrecognized property warning was generated when the dataproc:jupyter.listen.all.interfaces cluster property is set.
Dataproc is now available in the australia-southeast2 region (Melbourne).
Dataproc Component Gateway URLs for any two new clusters that have the same project ID, region, and name will be identical unless Dataproc Personal Cluster Authentication is enabled.
Custom image limitation: Currently, the following Dataproc image versions are the latest images that can be used as the base for custom images:
New sub-minor versions of Dataproc images: 1.3.91-debian10, 1.3.91-ubuntu18, 1.4.62-debian10, 1.4.62-ubuntu18, 1.5.37-centos8, 1.5.37-debian10, 1.5.37-ubuntu18, 2.0.11-centos8, 2.0.11-debian10, and 2.0.11-ubuntu18.
Image 1.3 - 2.0
All jobs now share a single JobthreadPool.
The number of Job threads in the Agent is configurable with the dataproc:agent.process.threads.job.min and dataproc:agent.process.threads.job.max cluster properties, defaulting to 10 and 100, respectively. The previous behavior was to always use 10 Job threads.
Image 2.0
nbdime 2.1 -> 3.0, pyarrow 2.0 -> 3.0, spyder 4.2 -> 5.0, spyder-kernels 1.10 -> 2.0, regex 2020.11 -> 2021.4.Image 1.3 - 2.0
Image 1.5 and 2.0
/has_run_before sentinel file. If you use a fork of connectors initialization-actions, then sync from head.Image 2.0
Fixed the problem that the environment variable PATH was not set in YARN containers.
SPARK-34731: ConcurrentModificationException in EventLoggingListener when redacting properties.
You can customize the Conda environment during cluster creation using new Conda-related cluster properties. See Using Conda-related cluster properties.
Added validation for clusters created with Dataproc Metastore services to determine compatibility between the Dataproc image's Hive version and the DPMS service's hive version
Announcing Dataproc Confidential Compute: Dataproc clusters now support Compute Engine Confidential VMs.
Image 1.4
Image 1.5
adoptopenjdk is set as the default Java environment.New sub-minor versions of Dataproc images: 1.3.89-debian10, 1.3.89-ubuntu18, 1.4.60-debian10, 1.4.60-ubuntu18, 1.5.35-centos8, 1.5.35-debian10, 1.5.35-ubuntu18, 2.0.9-centos8, 2.0.9-debian10, and 2.0.9-ubuntu18.
Image 1.5 and 2.0
Added the ability to stop and start high-availability clusters.
Fixed a bug where scale-down update cluster requests failed due to quota validation if the user project was over a quota limit.
Image 2.0:
New sub-minor versions of Dataproc images: 2.0.8-centos8, 2.0.8-debian10, and 2.0.8-ubuntu18.
Added HBase Spark Connector.
Dataproc support of Dataproc Metastore services is now available in GA.
Image 2.0:
OUTBOUND to INHERIT_FROM_SUBNETWORK.Dataproc is now available in the europe-central2 region (Warsaw).
The default Dataproc image is now image version 2.0.
Announcing the General Availability (GA) release of Dataproc cluster Stop/Start.
New sub-minor versions of Dataproc images: 1.3.88-debian10, 1.3.88-ubuntu18, 1.4.59-debian10, 1.4.59-ubuntu18, 1.5.34-centos8, 1.5.34-debian10, 1.5.34-ubuntu18, 2.0.7-centos8, 2.0.7-debian10, and 2.0.7-ubuntu18.
Image 2.0:
Fixed a bug that caused Hive jobs to fail on Ranger-enabled clusters.
The end date of support for Dataproc image version 1.3 has been extended from March, 2021 to July, 2021.
Fixed a bug where Presto property value with ';' could not be set correctly in the config file.
CVE-2020-13957: SOLR-14663: ConfigSets CREATE does not set trusted flag.
Fixed a bug where Spark event logs directory and history server directory could not be set to Cloud Storage correctly.
CVE-2020-1926: HIVE-22708: Test fix for http transport.
Announcing the General availability (GA) release of CentOS 8 based 2.0 images.
New sub-minor versions of Dataproc images: 1.3.87-debian10, 1.3.87-ubuntu18, 1.4.58-debian10, 1.4.58-ubuntu18, 1.5.33-centos8, 1.5.33-debian10, 1.5.33-ubuntu18, 2.0.6-centos8, 2.0.6-debian10, and 2.0.6-ubuntu18.
Image 2.0: Upgraded Spark to version 3.1.1
Dataproc 2.0 image version will become a default Dataproc image version in 1 week on March 15, 2021.
New sub-minor versions of Dataproc images: 1.3.86-debian10, 1.3.86-ubuntu18, 1.4.57-debian10, 1.4.57-ubuntu18, 1.5.32-centos8, 1.5.32-debian10, 1.5.32-ubuntu18, 2.0.5-debian10, and 2.0.5-ubuntu18
Image 2.0:
Fixed the permission for mounted SSD Hadoop directories.
Fixed a bug where YARN applications launched by Hive jobs were not correctly tagged, leading to missing YARN application status from job state.
Added the --cluster-labels flag to gcloud dataproc jobs submit to allow submitting jobs to a cluster that matches specified cluster labels. Also see Submitting a Dataproc job.
Dataproc 2.0 image version will become a default Dataproc image version in 2 weeks on March 15, 2021.
New sub-minor versions of Dataproc images: 1.3.85-debian10, 1.3.85-ubuntu18, 1.4.56-debian10, 1.4.56-ubuntu18, 1.5.31-centos8, 1.5.31-debian10, 1.5.31-ubuntu18, 2.0.4-debian10, and 2.0.4-ubuntu18
Image 2.0: Upgraded Spark to 3.1.1 RC2 version
Allow stopping clusters that have autoscaling enabled, and allow enabling autoscaling on clusters that are STOPPED, STOPPING, or STARTING. If you stop a cluster that has autoscaling enabled, the Dataproc autoscaler will stop scaling the cluster. It will resume scaling the cluster once it has been started again. If you enable autoscaling on a stopped cluster, the autoscaling policy will only take effect once the cluster has been started (see Starting and stopping clusters).
Deactivated mysql and hive-metastore components for clusters created with a Dataproc Metastore service on an image that has the DISABLE_COMPONENT_HIVE_METASTORE and DISABLE_COMPONENT_MYSQL capabilities.
Image 1.3 - 1.5: HIVE-18871: hive on Tez execution error due to set hive.aux.jars.path to hdfs://
Dataproc 2.0 image version will become a default Dataproc image version in 3 weeks on March 15, 2021.
New sub-minor versions of Dataproc images: 1.3.84-debian10, 1.3.84-ubuntu18, 1.4.55-debian10, 1.4.55-ubuntu18, 1.5.30-centos8, 1.5.30-debian10, 1.5.30-ubuntu18, 2.0.3-debian10, and 2.0.3-ubuntu18
Fixed a bug that prevented Dataproc on GKE cluster creation.
Dataproc 2.0 image version will become a default Dataproc image version in 4 weeks on March 15, 2021.
Image 2.0:
New sub-minor versions of Dataproc images: 2.0.2-debian10, and 2.0.2-ubuntu18.
Fixed a bug for beta clusters using a Dataproc Metastore Service where using a subnetwork for the cluster resulted in an error.
Dataproc 2.0 image version will become a default Dataproc image version in 5 weeks on March 15, 2021.
Dataproc 2.0 image version will become a default Dataproc image version in 3 weeks on February 22, 2021.
Image 2.0:
Upgraded Hadoop to version 3.2.2
Upgraded Scala to version 2.12.13
Upgraded Flink to version 1.12.1
Upgraded Spark to version 3.1.1 RC1
New sub-minor versions of Dataproc images: 1.3.83-debian10, 1.3.83-ubuntu18, 1.4.54-debian10, 1.4.54-ubuntu18, 1.5.29-centos8, 1.5.29-debian10, 1.5.29-ubuntu18, 2.0.1-debian10, and 2.0.1-ubuntu18.
Dataproc 2.0 image version will become a default Dataproc image version in 4 weeks on February 22, 2021.
Announcing the General Availability (GA) release of Dataproc 2.0 images. This image will become the default Dataproc image version on February 22, 2021.
2.0 image clusters:
In 2.0 clusters, yarn.nm.liveness-monitor.expiry-interval-ms is set to 15000 (15 seconds). If the resource manager does not receive a heartbeat from a NodeManager during this period, it marks the NodeManager as LOST. This setting is important for clusters that use preemptible VMs. Usually, NodeManagers unregister with the resource manager when their VMs shut down, but in rare cases when they are be shut down ungracefully, it is important for the resource manager to notice this quickly.
New sub-minor versions of Dataproc images: 1.3.82-debian10, 1.3.82-ubuntu18, 1.4.53-debian10, 1.4.53-ubuntu18, 1.5.28-centos8, 1.5.28-debian10, 1.5.28-ubuntu18, 2.0.0-debian10, and 2.0.0-ubuntu18.
2.0 image clusters:
You can no longer pass the dataproc:dataproc.worker.custom.init.actions.mode property when creating a 2.0 image cluster. For 2.0+ image clusters, dataproc:dataproc.worker.custom.init.actions.mode is set to RUN_BEFORE_SERVICES. For more information, see
Important considerations and guidelines—Initialization processing.
Fixed bug affecting cluster scale-down: If Dataproc was unable to verify whether a master node exists, for example when hitting Compute Engine read quota limits, it would erroneously put the cluster into an ERROR state.
The gcloud_dataproc_personal_cluster.py tool for the personal auth beta is no longer supported for new images. It will be replaced by an equivalent set of commands in an upcoming gcloud release.
Announcing the Beta release of Dataproc Service Account Based Secure Multi-tenancy, which allows you to share a cluster with multiple users. With secure multi-tenancy, users can submit interactive workloads to the cluster with isolated user identities.
New sub-minor versions of Dataproc images: 1.3.81-debian10, 1.3.81-ubuntu18, 1.4.52-debian10, 1.4.52-ubuntu18, 1.5.27-centos8, 1.5.27-debian10, 1.5.27-ubuntu18, 2.0.0-RC23-debian10, and 2.0.0-RC23-ubuntu18.
Added support for user configuration of Compute Engine Shielded VMs in a Dataproc Cluster.
New sub-minor versions of Dataproc images: 1.3.80-debian10, 1.3.80-ubuntu18, 1.4.51-debian10, 1.4.51-ubuntu18, 1.5.26-centos8, 1.5.26-debian10, 1.5.26-ubuntu18, 2.0.0-RC22-debian10, and 2.0.0-RC22-ubuntu18.
Image 2.0 preview:
HIVE-21646: Tez: Prevent TezTasks from escaping thread logging context
Added support for new persistent disk type, pd-balanced.
Announcing the General Availability (GA) release of CentOS 8 based 1.5 images.
Image 2.0 preview:
Changed the default value of Spark SQL property spark.sql.autoBroadcastJoinThreshold to 0.75% of executor memory.
Fixed SPARK-32436: Initialize numNonEmptyBlocks in HighlyCompressedMapStatus.readExternal
New sub-minor versions of Dataproc images: 1.3.79-debian10, 1.3.79-ubuntu18, 1.4.50-debian10, 1.4.50-ubuntu18, 1.5.25-centos8, 1.5.25-debian10, 1.5.25-ubuntu18, 2.0.0-RC21-debian10, and 2.0.0-RC21-ubuntu18.
Images 1.5-2.0 preview:
Fixed ZOOKEEPER-1936: Server exits when unable to create data directory due to race condition.
Image 1.4-1.5:
Fixed a NullPointerException in a primary worker shuffle when the BypassMergeSortShuffleWriter is used when some output partitions are empty.
Fixed a bug where Dataproc agent logs had separate entries for exception stack trace in StackDriver.
Announcing the Beta release of the Dataproc cluster Stop/Start.
Announcing the General Availability (GA) release of the Dataproc Workflow Timeout feature, which allows users to set a timeout on their graph of jobs and automatically cancel their workflow after a specified period.
Restartable jobs: Added the ability for users to specify the maximum number of total failures when a job is submitted.
Image 2.0 preview
Using the n1-standard-1 machine type is no longer supported.
Changed default values of Spark SQL properties:
spark.sql.adaptive.enabled=truespark.sql.autoBroadcastJoinThreshold =< 2% of executor memory.Image 1.5:
The Dataproc Metastore Service is now available in the us-east4, europe-west2, asia-northeast1, and australia-southeast1 regions in addition to the existing us-central1 region.
New sub-minor versions of Dataproc images: 1.3.78-debian10, 1.3.78-ubuntu18, 1.4.49-debian10, 1.4.49-ubuntu18, 1.5.24-debian10, 1.5.24-ubuntu18, 2.0.0-RC20-debian10, and 2.0.0-RC20-ubuntu18.
Image 2.0 preview
New sub-minor versions of Dataproc images: 1.3.77-debian10, 1.3.77-ubuntu18, 1.4.48-debian10, 1.4.48-ubuntu18, 1.5.23-debian10, 1.5.23-ubuntu18, 2.0.0-RC19-debian10, and 2.0.0-RC19-ubuntu18.
New sub-minor versions of Dataproc images: 1.3.76-debian10, 1.3.76-ubuntu18, 1.4.47-debian10, 1.4.47-ubuntu18, 1.5.22-debian10, 1.5.22-ubuntu18, 2.0.0-RC18-debian10, and 2.0.0-RC18-ubuntu18.
Clusters that use Dataproc Metastore must be created in the same region as the Dataproc Metastore service that they will use.
Fixed a bug where the Jupyter optional component depended on the availability of GitHub at cluster creation time.
Image 2.0 preview
Added a dataproc:dataproc.cooperative.multi-tenancy.user.mapping cluster property which takes a list of comma-separated user-to-service account mappings. If a cluster is created with this property set, when a user submits a job, the cluster will attempt to impersonate the corresponding service account when accessing Cloud Storage through the Cloud Storage connector. This feature requires Cloud Storage connector version 2.1.4 or higher.
New sub-minor versions of Dataproc images: 1.3.75-debian10, 1.3.75-ubuntu18, 1.4.46-debian10, 1.4.46-ubuntu18, 1.5.21-debian10, 1.5.21-ubuntu18, 2.0.0-RC17-debian10, and 2.0.0-RC17-ubuntu18.
Image 2.0 preview:
Fixed HADOOP-15124: Slow FileSystem.Statistics counters implementation.
Image 1.3 to 2.0 preview:
Fixed HIVE-19202: CBO failed due to NullPointerException in HiveAggregate.isBucketedInput().
Fixed a bug in HBASE optional component on HA clusters in which hbase.rootdir was always configured to be hdfs://${CLUSTER_NAME}-m-0:8020/hbase, which assumes that master 0 is the active namenode. Now it is configured to be hdfs://${CLUSTER_NAME}:8020/hbase, so that the active master is always chosen.
2.0 preview image versions:
Pin MySQL Java connector version to prevent breakage of the /usr/share/java/mysql-connector-java.jar symlink on long-running and old clusters caused by auto-upgrade to a new MySQL Java connector.
Decreased the minimum allowed value of Dataproc Scheduled Deletion LifecycleConfig.idleDeleteTtl (Dataproc API) and --max-idle flag (gcloud command-line tool) from 10 minutes to 5 minutes.
New sub-minor versions of Dataproc images: 1.3.74-debian10, 1.3.74-ubuntu18, 1.4.45-debian10, 1.4.45-ubuntu18, 1.5.20-debian10, 1.5.20-ubuntu18, 2.0.0-RC16-debian10, and 2.0.0-RC16-ubuntu18.
All image versions:
1.4 and 1.5 image versions:
Sole-tenant node cluster create or update requests to use preemptible secondary workers or attach autoscaling policies that create preemptible secondary workers are now correctly rejected.
Announcing the Alpha release of the Dataproc Persistent History Server, which provides a UI to view job history for jobs run on active and deleted Dataproc clusters.
Announcing the GA (General Availability) release of the Dataproc Ranger Optional Component and the Dataproc Solr Optional Component.
Announcing the GA (General Availability) release of the Dataproc - Docker Optional Component and the Dataproc - Flink Optional Component.
New sub-minor versions of Dataproc images: 1.3.72-debian10, 1.3.72-ubuntu18, 1.4.43-debian10, 1.4.43-ubuntu18, 1.5.18-debian10, 1.5.18-ubuntu18, 2.0.0-RC14-debian10, and 2.0.0-RC14-ubuntu18.
Image 2.0
google-cloud-bigquery-storage in default conda environment.zeppelin.notebook.storage in zeppelin-site.xml to "org.apache.zeppelin.notebook.repo.GCSNotebookRepo".Image 1.4
Image 1.5
google-cloud-bigquery-storage package by default in the Anaconda component.zeppelin.notebook.storage in zeppelin-site.xml to "org.apache.zeppelin.notebook.repo.GCSNotebookRepo".New sub-minor versions of Dataproc images: 1.3.71-debian10, 1.3.71-ubuntu18, 1.4.42-debian10, 1.4.42-ubuntu18, 1.5.17-debian10, 1.5.17-ubuntu18, 2.0.0-RC13-debian10, and 2.0.0-RC13-ubuntu18.
Launched Dataproc integration with Compute Engine sole-tenant nodes, which allows users to create a cluster in a sole-tenant node group.
Creating clusters and instantiating workflow requests that succeed even when the requester did not have ActAs permission on the service account now generate a warning field in the audit log request.
New sub-minor versions of Dataproc images: 1.3.70-debian10, 1.3.70-ubuntu18, 1.4.41-debian10, 1.4.41-ubuntu18, 1.5.16-debian10, 1.5.16-ubuntu18, 2.0.0-RC12-debian10, and 2.0.0-RC12-ubuntu18.
All supported images
Upgraded Conscrypt to the 2.5.1 version.
Image 1.5 and Image 2.0 Preview
Image 1.5
Image 2.0 preview
The Anaconda optional component is no longer available or required when using the Jupyter optional component (Miniconda is installed and activated by default).
Updated R to the 4.0.0 version.
Configured YARN aggregated logs to use the IFile format.
Upgrade Flink to the 1.11.2 version.
New sub-minor versions of Dataproc images: 1.3.69-debian10, 1.3.69-ubuntu18, 1.4.40-debian10, 1.4.40-ubuntu18, 1.5.15-debian10, 1.5.15-ubuntu18, 2.0.0-RC11-debian10, and 2.0.0-RC11-ubuntu18.
All image versions
Image 2.0 preview
INSERT OVERWRITE TABLE doesn't clean the table directory before overwriting.Added the PrivateIpv6GoogleAccess API field to allow configuring IPv6 access to Dataproc cluster.
New sub-minor versions of Dataproc images: 1.3.68-debian10, 1.3.68-ubuntu18, 1.4.39-debian10, 1.4.39-ubuntu18, 1.5.14-debian10, 1.5.14-ubuntu18, 2.0.0-RC10-debian10, and 2.0.0-RC10-ubuntu18.
1.3-1.5 Images:
HIVE-18323:
Vectorization: add the support of timestamp in
VectorizedPrimitiveColumnReader for parquet
1.5 and 2.0 preview images:
Upgraded the jupyter-core and jupyter-client packages in the 1.5 and 2.0 images to be compatible with the installed notebook package version.
2.0 preview image:
Fixed a regression that could cause clusters to fail to start if user-supplied keystore/truststore are provided when enabling Kerberos.
Switched 1.3 and 1.3-debian image version aliases to point to 1.3 Debian 10 images.
Improved node memory utilization in clusters created with 2.0 preview images.
Changed secondary workers default boot disk size to 1000 GB in clusters created with 2.0 preview images.
Support more than 8 local SSDs on VMs. Compute Engine supports 16 and 24 SSDs for larger machine types.
When Enhanced Flexibility Mode is enabled, increased app master, task, and Spark stage retries to 10 to improve resiliency of applications to downscaling and preemption of preemptible VMs.
Launched Dataproc Workflow Timeout feature, which allows users to set a timeout on their graph of jobs and automatically cancel their workflow after a specified period.
Dataproc Metastore integration, which allows users to create a cluster using a Dataproc Metastore service as an external metastore, is now available for Alpha release testing.
Announcing the Beta release of Dataproc Enhanced Flexibility Mode (EFM), which manages shuffle data to minimize job progress delays caused by the removal of nodes from a running cluster.
Image 1.3 and 1.4: upgraded the Cloud Storage connector to version to 1.9.18.
Changed 1.4 and 1.4-debian image version aliases to point to 1.4-debian10. The version name 1.4-debian9 will continue to be available, but it won't get updates in future releases.
New sub-minor versions of Dataproc images: 1.3.67-debian10, 1.3.67-ubuntu18, 1.4.38-debian10, 1.4.38-ubuntu18, 1.5.13-debian10, 1.5.13-ubuntu18, 2.0.0-RC9-debian10, and 2.0.0-RC9-ubuntu18.
Launched new Personal Cluster Authentication feature, which allows the creation of single-user clusters that can access Cloud Storage using the user's own credentials instead of a VM service account.
Dataproc quotas are now regional: each region now has its own quota, which can be separately adjusted. All existing quota overrides have been migrated; customer action is not required.
Enabled Spark SQL parquet metadata cache (removed spark.sql.parquet.cacheMetadata=false from Spark default configuration).
New sub minor versions of Dataproc images: 1.3.66-debian10, 1.3.66-ubuntu18, 1.4.37-debian10, 1.4.37-ubuntu18, 1.5.12-debian10, 1.5.12-ubuntu18, 2.0.0-RC8-debian10, and 2.0.0-RC8-ubuntu18.
Image 2.0 preview:
Image 1.4:
Image 1.5:
Dataproc users are required to have service account ActAs permission to deploy Dataproc resources, for example, to create clusters and submit jobs. See Managing service account impersonation for more information.
Opt-in for existing Dataproc customers: This change does not automatically apply to current Dataproc customers without ActAs permission. To opt in, see Securing Dataproc, Dataflow, and Cloud Data Fusion.
Enabled Kerberos automatic-configuration feature. When creating a cluster, users can enable Kerberos by setting the dataproc:kerberos.beta.automatic-config.enable cluster property to true. When using this feature, users do not need to specify the Kerberos root principal password with the --kerberos-root-principal-password and --kerberos-kms-key-uri flags.
New sub-minor versions of Dataproc images: 1.3.65-debian10, 1.3.65-ubuntu18, 1.4.36-debian10, 1.4.36-ubuntu18, 1.5.11-debian10, 1.5.11-ubuntu18, 2.0.0-RC7-debian10, and 2.0.0-RC7-ubuntu18.
1.3+ images (includes Preview image):
HADOOP-16984: Added support to read history files only from the done directory.
MAPREDUCE-7279: Display the Resource Manager name on the HistoryServer web page.
SPARK-32135: Show the Spark driver name on the Spark history web page.
SPARK-32097: Allow reading Spark history log files via the Spark history server from multiple directories.
Images 1.3 - 1.5:
Images 1.5 - 2.0 preview:
Fixed an issue where optional components that depend on HDFS failed on single node clusters.
Fixed an issue that caused workflows to be stuck in the RUNNING state when managed clusters (created by the workflow) were deleted while the workflow was running.
Terminals started in Jupyter and JupyterLab now use login shells. The terminals behave as if you SSH'd into the cluster as root.
Upgraded the jupyter-gcs-contents-manager package to the latest version. This upgrade includes a bug fix to a 404 (NOT FOUND) error message that was issued in response to an attempt to create a file in the virtual top-level directory instead of the expected 403 (PERMISSION DENIED) error message.
New sub-minor versions of Dataproc images: 1.3.64-debian10, 1.3.64-ubuntu18, 1.4.35-debian10, 1.4.35-ubuntu18, 1.5.10-debian10, 1.5.10-ubuntu18, 2.0.0-RC6-debian10, and 2.0.0-RC6-ubuntu18.
Fixed a bug in which the HDFS DataNode daemon was enabled on secondary workers but not started (except on VM reboot if started automatically by systemd).
Fixed a bug in which StartLimitIntervalSec=0 appeared in the Service section instead of the Unit section for systemd services, which disabled rate limiting for retries when systemd restarted a service.
Dataproc now uses Shielded VMs for Debian 10 and Ubuntu 18.04 clusters by default.
Added support for Zeppelin Spark and shell interpreters in Kerberized clusters by default.
If a project's regional Dataproc staging bucket is manually deleted, it will be recreated automatically when a cluster is subsequently created in that region.
New sub-minor versions of Dataproc images: 1.3.63-debian10, 1.3.63-ubuntu18, 1.4.34-debian10, 1.4.34-ubuntu18, 1.5.9-debian10, 1.5.9-ubuntu18, 2.0.0-RC5-debian10, and 2.0.0-RC5-ubuntu18.
Image 2.0 preview:
0.18.0-SNAPHOT to 0b2066a revision to fix Hive 3.1 compatibility (PIG-4764).The Proxy-Authorization header is accepted in place of Authorization to authenticate to Component Gateway to the backend for programmatic API calls. If Proxy-Authorization is set to a bearer token, Component Gateway will forward the Authorization header if it does not contain a bearer token.
For example, this allows setting both Proxy-Authorization: Bearer <google-access-token> as well as setting Authorization: Basic ... to authenticate to HiveServer2 with HTTP basic auth.
Added --temp-bucket flag to gcloud dataproc clusters create and gcloud dataproc workflow-templates set-managed-cluster to allow users to configure a Cloud Storage bucket that stores ephemeral cluster and jobs data, such as Spark and MapReduce history files.
Dataproc images now include the oauth2l command line tool. The tool is installed in /usr/local/bin, which is available to all users in the VM.
Extended Jupyter to support notebooks stored on VM persistent disk. This change modifies the Jupyter contents manager to create two virtual top-level directories, named GCS, and Local Disk. The GCS directory points to the Cloud Storage location used by previous versions, and the Local Disk directory points to the persistent disk of the VM running Jupyter.
New sub-minor versions of Dataproc images: 1.2.102-debian9, 1.3.62-debian9, 1.4.33-debian9, 1.3.62-debian10, 1.4.33-debian10, 1.5.8-debian10, 1.3.62-ubuntu18, 1.4.33-ubuntu18, 1.5.8-ubuntu18, 2.0.0-RC4-debian10, 2.0.0-RC4-ubuntu18
Images 1.3 - 1.5:
Images 1.3 - 2.0 preview:
Fixed a race condition that could nondeterministically cause Hive-WebHCat to fail at startup when HBase is not enabled.
Announcing the General Availability (GA) release of Dataproc Component Gateway, which provides secure access to web endpoints for Dataproc default and optional components.
Image 2.0 preview:
spark.yarn.unmanagedAM.enabled property to true on clusters where
Kerberos is not enabled to run Spark Application Master in driver (not
managed in YARN) to improve job execution time.Updated R version to 3.6
Updated Spark to 3.0.0 version
Image 1.5
New subminor image versions: 1.2.100-debian9, 1.3.60-debian9, 1.4.31-debian9, 1.3.60-debian10, 1.4.31-debian10, 1.5.6-debian10, 1.3.60-ubuntu18, 1.4.31-ubuntu18, 1.5.6-ubuntu18, preview 2.0.0-RC2-debian10, and preview 2.0.0-RC2-ubuntu18.
Fixed a quota validation bug where accelerator counts were squared before validation -- for example, previously if you requested 8 GPUs, Dataproc validated whether your project had quota for 8^2=64 GPUs.
Users can now configure a tempBucket in API calls. The temp bucket is a Cloud Storage bucket used to store ephemeral cluster and jobs data, such as Spark and MapReduce history files. If you do not specify a temp bucket, Dataproc will determine a Cloud Storage location (US, ASIA, or EU) for your cluster's temp bucket according to the Compute Engine zone where your cluster is deployed, and then create and manage this project-level, per-location bucket.
New subminor image versions: 1.2.99-debian9, 1.3.59-debian9, 1.4.30-debian9, 1.3.59-debian10, 1.4.30-debian10, 1.5.5-debian10, 1.3.59-ubuntu18, 1.4.30-ubuntu18, and 1.5.5-ubuntu18.
New preview image 2.0.0-RC1-debian10, 2.0.0-RC1-ubuntu18, with the following components:
Image 1.3+
Dataproc Job container logging now supports Dataproc Kerberized clusters.
Druid's Historical's and Broker's JVM and runtime properties are now calculated using server resources. Previously, only the Historical's and MiddleManager's MaxHeapSize property was calculated using server resources. This change modifies how new values for MaxHeapSize and MaxDirectMemorySize properties are calculated for Broker and Historical processes. Also, new runtime properties druid.processing.numThreads and druid.processing.numMergeBuffers are calculated using server resources.
If the project-level staging bucket is manually deleted, it will be recreated when a cluster is created.
Image 1.5:
Dataproc is now available in the asia-southeast2 region (Jakarta).
Dataproc now provides beta support for Dataproc Hub.
You can now set core:fs.defaultFS to a location in Cloud Storage (for example, gs://bucket) when creating a cluster to set Cloud Storage as the default filesystem. This also sets core:fs.gs.reported.permissions, the reported permission returned by the Cloud Storage connector for all files, to 777. If Cloud Storage is not set as the default filesystem, this property will continue to return 700, the default value.
Image 1.4 and 1.5
HADOOP-16984: Enable persistent history server to read from done directory.
Image 1.5
Upgrade Druid to version 0.17.1.
Upgrade Cloud Storage Connector to version 2.1.3.
Upgrade Delta Lake to version 0.6.0.
New sub-minor versions of Dataproc images: 1.2.98-debian9, 1.3.58-debian9, 1.4.29-debian9, 1.3.58-debian10, 1.4.29-debian10, 1.5.4-debian10, 1.3.58-ubuntu18, 1.4.29-ubuntu18, 1.5.4-ubuntu18.
Image 1.3, 1.4, and 1.5
Restrict Jupyter, Zeppelin, and Knox to only accept connections from localhost when Component Gateway is enabled. This restriction reduces the risk of remote code execution over unsecured notebook server APIs. To override this change, when you create the cluster, set the Jupyter, Zeppelin, and Knox cluster properties, respectively, as follows: dataproc:jupyter.listen.all.interfaces=true, zeppelin:zeppelin.server.addr=0.0.0.0, and knox:gateway.host=0.0.0.0.
Upgrade Hive to version 2.3.7.
Image 1.4 and 1.5
SPARK-29367: Add ARROW_PRE_0_15_IPC_FORMAT=1 in yarn-env.sh to fix the Pandas UDF issue with pyarrow 0.15.
Set the hive.localize.resource.num.wait.attempts property to 25 to improve reliability of Hive queries.
Hide the "Quit" button from Jupyter notebook (c.NotebookApp.quit_button = False) when using the Jupyter optional component. The Jupyter environment is shut down when the cluster is deleted.
Image 1.5
Fix a race condition in which hbase-master would try to write /hbase/.tmp/hbase.version to HDFS before HDFS was initialized. This can increase cluster creation time for clusters created with HBase.
Fix a race condition in which, when the am.primary_only property is provided, the "non-preemptible" node label was not added to the resource manager's node label store before node managers started registering with the resource manager.
Store resource manager node labels in Cloud Storage when am.primary_only property is provided.
The dataproc:alpha.state.shuffle.hcfs.enabled cluster property has been deprecated. To enable Enhanced Flexibility Mode (EFM) for Spark, set dataproc:efm.spark.shuffle=hcfs. To enable EFM for MapReduce, set dataproc:efm.mapreduce.shuffle=hcfs.
Clusters can now be created with non-preemptible secondary workers.
Announcing the Beta release of Dataproc Component Gateway, which provides secure access to web endpoints for Dataproc default and optional components.
Dataproc on GKE version 1.4.27-beta is available with minor fixes.
Image 1.5
Delta Lake version is upgraded to 0.5.0 release. Delta Lake Hive Connector 0.1.0 is also added to the 1.5 image.
Customers can now adjust the amount of time the Dataproc startup script will wait for Presto Coordinator service to start before deciding that their startup has succeeded. This is set via dataproc:startup.component.service-binding-timeout.presto-coordinator property and takes a value in seconds. The maximum respected value is 1800 (30 minutes).
New sub-minor image versions: 1.2.96-debian9, 1.3.56-debian9, 1.4.27-debian9, 1.3.56-debian10, 1.4.27-debian10, 1.5.2-debian10, 1.3.56-ubuntu18, 1.4.27-ubuntu18, 1.5.2-ubuntu18
Image 1.5
Cloud Storage connector upgraded to version 2.1.2 (for more information, review the change notes in the GitHub repository)
Image 1.5
Notebook bug fixes: fixed a bug in Zeppelin and Jupyter that resulted in a failure to render images when using Component Gateway. Also fixed a Jupyter Notebooks bug that caused notebook downloads to fail.
Dataproc is now available in the us-west4 region (Las Vegas).
Announcing the Beta release of Dataproc on Google Kubernetes Engine. Customers can now create Dataproc on GKE clusters to run Spark jobs on Kubernetes via the Dataproc jobs API.
Image 1.5
Presto now includes two default catalogs:
bigquery pointing to the datasets of the cluster's project
bigquery_public_data pointing to the public datasets
Image 1.5
Jupyter on Dataproc now supports exporting notebooks as PDFs.
Image 1.3, 1.4 and 1.5
Added Component Gateway support for Dataproc clusters secured with Kerberos.
Image 1.5
Updated Presto to version 331.
Image 1.3 and 1.4
Debian 10 images will become default images for 1.3 and 1.4 image tracks and Debian 9 images will not be released for these tracks anymore after June 30, 2020.
New sub-minor versions of Dataproc images: 1.2.95-debian9, 1.3.55-debian9, 1.4.26-debian9, 1.3.55-debian10, 1.4.26-debian10, 1.5.1-debian10, 1.3.55-ubuntu18, 1.4.26-ubuntu18, 1.5.1-ubuntu18.
Created cloud-sql-proxy log file for the Cloud SQL Proxy initialization action and for Dataproc clusters with Ranger that use Cloud SQL Proxy.
Image 1.3, 1.4 and 1.5
Fixed a bug where Jupyter was unable to read and write notebooks stored in Cloud Storage buckets with CMEK enabled.
Images 1.4 and 1.5
SPARK-29080: Support R file extension case-insensitively when submitting Spark R jobs.
Image 1.3, 1.4 and 1.5
HIVE-17275: Auto-merge fails on writes of UNION ALL output to ORC file with dynamic partitioning.
Added Presto and SparkR job type support to Dataproc Workflows.
Fixed an Auto Zone Placement bug that incorrectly returned INVALID_ARGUMENT errors as INTERNAL errors, and didn't propagate these error messages to the user.
Announcing the General Availability (GA) release of Dataproc Presto job type, which can be submitted to a cluster using the gcloud dataproc jobs submit presto command. Note: The Dataproc Presto Optional Component must be enabled when the cluster is created to submit a Presto job to the cluster.
Dataproc clusters can now be created on the GKE platform by setting the GkeClusterConfig instead of the GceClusterConfig via the Beta API. This feature allows jobs to be submitted that will run on the Kubernetes cluster.
Announcing the Beta release of Dataproc - Ranger Top-Level Component and Dataproc - Solr Top-Level Component.
Added alphabetical sort order to Workflow Templates List methods.
Added pagination support to Clusters List methods to provide functionality to the pageSize parameter, which is a part of the API. This feature allows users to specify a page size to receive paginated data in the response. The default page size is 200 and the max page size is 1000.
Announcing the General Availability (GA) release of Dataproc - Presto Top-Level Component.
Announcing the General Availability (GA) release of Dataproc 1.5 images.
Images 1.2 and 1.4
Dataproc 1.4 will be the default image version after April 31, 2020.
Dataproc 1.2 will have no further releases after June 30, 2020.
New sub-minor versions of Dataproc images: 1.2.94-debian9, 1.3.54-debian9, 1.4.25-debian9, 1.5.0-debian10, 1.3.54-ubuntu18, 1.4.25-ubuntu18, and 1.5.0-ubuntu18
Image 1.5
Upgraded the Cloud Storage connector to version 2.1.1.
Images 1.3, 1.4, and 1.5
Fixed HDFS UI in the Component Gateway on HA clusters
Fixed issue where Jupyter hangs when loading a directory containing many large files. This also improves responsiveness when listing directories.
Added the following flag to the gcloud dataproc clusters update command:
--num-secondary-workersThe following flag to gcloud dataproc clusters update has been deprecated:
--num-preemptible-workers See the related change, above, for the new flag to use in place of this deprecated flag.
Added a mapreduce.jobhistory.always-scan-user-dir cluster property that enables the MapReduce job history server to read the history files (recommended when writing history files to Cloud Storage). The default is true.
Added support for n2-, c2-, e2-, n2d-, and m2- machine types when using
Auto Zone Placement.
Previously, users could only specify n1- or custom- machine types when using
Auto Zone Placement.
Added a dataproc:job.history.to-gcs.enabled cluster property that allows persisting MapReduce and Spark history files to the Dataproc temp bucket (default: true for image versions 1.5+). bucket. This property defaults to true for image versions 1.5 and up. Users can overwrite the locations of job history file persistence through the following properties:
mapreduce.jobhistory.done-dirmapreduce.jobhistory.intermediate-done-dir *spark.eventLog.dirspark.history.fs.logDirectoryCustomers can now enable the Cloud Profiler when submitting a Dataproc job by setting the
cloud.profiler.enable property. To use profiling, customers must enable the Cloud Profiler API for their
project and create the cluster with --scopes=cloud-platform. The following profiling properties can also be set
when submitting a Dataproc job:
cloud.profiler.name: to collect profiler data under the
specified name. If not specified, it defaults to the job UUID.
cloud.profiler.service.version: to compare profiler information from different job runs. If not specified, defaults to
the job UUID.
Image 1.4, 1.5 preview
Added ARROW_PRE_0_15_IPC_FORMAT=1 for spark-env for pyarrow upgrade from 0.13 to 0.15.
New sub-minor versions of Dataproc images: 1.2.93-debian9, 1.3.53-debian9, 1.4.24-debian9, 1.5.0-RC9-debian10, 1.3.53-ubuntu18, 1.4.24-ubuntu18, and 1.5.0-RC9-ubuntu18
Image 1.5 preview
Preinstalled additional Python packages and Jupyter[Lab] extensions to align Jupyter notebook environment with AI Platforms Notebooks when Jupyter optional component is enabled.
Upgraded Druid version to 0.17.0
Image 1.2, 1.3, 1.4
Upgraded HBase to 1.3.6
Cluster list methods now return results in lexical order.
Normalized custom image URLs to a full URL, for example: https://www.googleapis.com/compute/v1/projects/foo-project/global/images/foo-image. The cluster creation API will continue to accept relative names, such as
projects/foo-project/global/images/foo-image or foo-image (see Dataproc API doesn't return imageUri in a consistent format).
Image 1.3, 1.4, 1.5 preview
Fixed YARN container log links in Component Gateway
Announcing the General Availability (GA) release of Dataproc minimum CPU platform.
Added the following flags to gcloud dataproc clusters create and gcloud dataproc workflow-templates set-managed-cluster commands:
--num-secondary-workers --num-secondary-worker-local-ssds--secondary-worker-boot-disk-size--secondary-worker-boot-disk-type--secondary-worker-acceleratorThe following flags to gcloud dataproc clusters create and gcloud dataproc workflow-templates set-managed-cluster commands have been deprecated:
--num-preemptible-workers --num-preemptible-worker-local-ssds--preemptible-worker-boot-disk-size--preemptible-worker-boot-disk-type--preemptible-worker-acceleratorSee the related change, above, for the new flags to use in place of these deprecated flags.
Added a dataproc:yarn.log-aggregation.enabled cluster property that allows turning on YARN log aggregation to a Dataproc temporary bucket (default: true for image versions 1.5+). Users can also set the location of aggregated YARN logs by overwriting the yarn.nodemanager.remote-app-log-dir YARN property.
1.5 preview image:
Upgraded the Cloud Storage connector to version 2.0.1.
New sub-minor versions of Dataproc images: 1.2.92-debian9, 1.3.52-debian9, 1.4.23-debian9, 1.5.0-RC8-debian10, 1.3.52-ubuntu18, 1.4.23-ubuntu18, 1.5.0-RC8-ubuntu18
Changed the staging bucket naming scheme for new buckets from dataproc-<UUID>-<REGION> to dataproc-staging-<REGION>-<PROJECT_NUMBER>-<RANDOM_STRING>.
In addition to the staging bucket, Dataproc now creates a temporary bucket for storing feature-related data with a 90-day retention period per project per region. The bucket name will be of the following form: dataproc-temp-<REGION>-<PROJECT_NUMBER>-<RANDOM_STRING>.
1.3, 1.4, and 1.5 preview images:
Fixed a bug where Component Gateway pages added an additional banner on each page load in some browsers.
Added support for attaching GPUs to preemptible workers.
Component Gateway now supports VPC Service Controls.
Add OS-family image version aliases: 1.0-debian, 1.1-debian, 1.2-debian,
1.3-debian, 1.3-ubuntu, 1.4-debian, 1.4-ubuntu, preview-debian,
preview-ubuntu. Users can now reference these aliases: Example:
gcloud dataproc clusters create $CLUSTER_NAME --image-version=1.4-debian
Increase Dataproc agent buffer size for streaming driver output to Cloud Storage from 60KB to 8MB. This significantly increases streaming throughput: from ~200KBps to ~7MBps. However, it also increases the memory footprint for each job in the Dataproc agent from ~100KB to ~10MB.
New sub-minor versions of Dataproc images: 1.2.91-debian9, 1.3.51-debian9, 1.4.22-debian9, 1.5.0-RC7-debian10, 1.3.51-ubuntu18, 1.4.22-ubuntu18, 1.5.0-RC7-ubuntu18
The Compute Engine API call from the Dataproc backend to check the API-specific quota limit is now enforced. The default Compute Engine API quota limit should be sufficient for most projects. If your project is experiencing Compute Engine API throttling, you can request a higher quota. If your project requires QPS that is higher than the upper quota limit, consider splitting your project into smaller projects.
Image 1.2, 1.3, & 1.4:
Upgrade Oozie to 4.3.1 and backport OOZIE-3112.
Image 1.3:
Added BigQuery magics for Python2 and PySpark kernels to Dataproc
Jupyter notebooks. You can now use the BigQuery Python client library and Pandas in a Jupyter notebook to visualize data in the BigQuery table.The BigQuery client library provides a %%bigquery cell magic, which runs a SQL query and returns the results as a Pandas DataFrame (see Visualizing BigQuery data in a Jupyter notebook
https://cloud.google.com/bigquery/docs/visualize-jupyter ). Example:
%%bigquer SELECT * FROM bigquery-public-data.samples.natality LIMIT 1
Upgrade Zeppelin to 0.8.1 version
Images 1.3, 1.4, & 1.5:
Backport HIVE-16958.
Added support for parameterizing numInstances field in Workflow Templates.
Images 1.4 & 1.5 :
Upgrade Spark to 2.4.5 version
Image 1.4 & 1.5:
Added BigQuery magics for Python3 and PySpark kernels to Dataproc
Jupyter notebooks. You can now use the BigQuery Python client library and Pandas in a Jupyter notebook to visualize data in the BigQuery table.The BigQuery client library provides a %%bigquery cell magic, which runs a SQL query and returns the results as a Pandas DataFrame (see Visualizing BigQuery data in a Jupyter notebook
https://cloud.google.com/bigquery/docs/visualize-jupyter ). Example:
%%bigquer SELECT * FROM `bigquery-public-data.samples.natality` LIMIT 1
Fixed the bug that causing update cluster label operations failed on clusters using CMEK on PD.
Backport YARN-9011 to all 1.2, 1.3, 1.4, and 1.5 images. This fixes a race condition in the resource manager that would sometimes cause containers to be killed while performing a graceful decommission.
Image 1.3 & 1.4:
ZEPPELIN-4168: fixed downloading of Maven dependencies in Zeppelin
Fix "YARN available memory" metric when using YARN's FairScheduler. Importantly, this ensures autoscaling works with the FairScheduler.
Dataproc is now available in the us-west3 region (Salt Lake City).
You can adjust the amount of time the Dataproc startup script will wait for services (currently hive-metastore, hive-server2, hadoop-hdfs-namenode) to bind to ports before deciding that their startup has succeeded. These delays are set via the dataproc:startup.component.service-binding-timeout.hive-metastore, dataproc:startup.component.service-binding-timeout.hive-server2, and dataproc:startup.component.service-binding-timeout.hadoop-hdfs-namenode cluster properties, which take a value in seconds. The maximum recognized value is 1800 seconds (30 minutes).
1.5 preview image:
Add BigQuery connector to Presto SQL
New sub-minor versions of Dataproc images: 1.2.90-debian9, 1.3.50-debian9, 1.4.21-debian9, 1.5.0-RC6-debian10, 1.3.50-ubuntu18, 1.4.21-ubuntu18, 1.5.0-RC6-ubuntu18.
Announcing the General Availability (GA) release of Dataproc Scheduled Deletion. This feature provides options to delete a cluster:
Dataproc is now available in the asia-northeast3 region (Seoul).
New sub-minor versions of Dataproc images: 1.2.89-debian9, 1.3.49-debian9, 1.4.20-debian9,1.5.0-RC5-debian10, 1.3.49-ubuntu18, 1.4.20-ubuntu18, 1.5.0-RC5-ubuntu18.
Promoted Zonal Reservations to General Availability (GA).
Breaking Change: Reduced default spark.executor.memory by 1MB from 9310 to 9309 on n1-highmem-4 workers to improve YARN bin packing.
1.5 preview image updates:
spark.ui.port property to 0 to run Spark UI on random port -
removes limit on max number of concurrent Spark jobs (16 by default).
Previous behavior can be restored by setting spark.ui.port=4040
propertyyarn.nodemanager.address property to 0.0.0.0:8026 to run YARN Node Manager on the same port - if a Node Manager process restarts (e.g. when a VM is preempted), it shows up as the same node in YARN. Otherwise, you would see both a LOST node for the old port, as well as a new RUNNING node with the new porthive.fetch.task.conversion property to none to disable execution of the SELECT * WHERE conditions LIMIT x queries in the job drivermapreduce.jobhistory.always-scan-user-dir property to true to enable better support of history files stored on Cloud Storage.mapreduce.jobhistory.recovery.store.class property to org.apache.hadoop.mapreduce.v2.hs.HistoryServerLeveldbStateStoreService to reduce HDFS usage/etc/spark/conf/fairscheduler.xml configuration fileBreaking Change: Modified gcloud beta clusters create command, replacing --reservation-label with --reservation, which accepts the name of the reservation when --reservation-affinity is specific, matching gcloud compute instances create.
1.2 – 1.4 image updates: * Upgraded Apache Solr to 6.6.6 version
Breaking Change: gcloud command-line tool dataproc commands now require the --region flag on each invocation or dataproc/region config property to be set. Set your default Dataproc region via the gcloud config set dataproc/region us-central1
command.
New sub-minor versions of Dataproc images: 1.2.88-debian9, 1.3.48-debian9, 1.4.19-debian9, 1.5.0-RC4-debian10, 1.3.48-ubuntu18, 1.4.19-ubuntu18, 1.5.0-RC4-ubuntu18.
Component Gateway UIs now include a banner showing the project and cluster name being accessed and a button to sign out of Component Gateway for the cluster.
Fixed a bug in UpdateCluster quota validation logic impacting the use of zonal reservations where the resource requirements of new instances being added in an update failed to consider the capacity of unused zonal reservations, and thus would incorrectly return an "insufficient quota" error if the new instances could not fit in the unused quota outside of zonal reservations.
Added additional information into Autoscaler recommendations in Stackdriver logging. In particular each recommendation now includes 1) the minimum and maximum worker counts configured in the autoscaling policy, 2) the graceful decommission timeout (for SCALE_DOWN operations), and 3) additional context into why the autoscaler couldn't add or remove more nodes. For example, it will indicate if it could not scale up further due to available regional or global quotas in the project (SCALING_CAPPED_DUE_TO_LACK_OF_QUOTA).
New sub-minor versions of Cloud Dataproc images:: 1.2.87-debian9, 1.3.47-debian9, 1.4.18-debian9, 1.5.0-RC3-debian10, 1.3.47-ubuntu18, 1.4.18-ubuntu18, 1.5.0-RC3-ubuntu18
1.5 preview image updates
conf object using sc=SparkContext.getOrCreate(conf=conf). To create SparkSession, user can import SparkSession and use SparkSession.builder.appName("Any Name").getOrCreate()New images now install the latest Linux kernel version available in Debian backports repository through linux-image-amd64 meta package.
Disabled noisy Jersey logging in the YARN Timeline Server.
Fixed a bug in using the JupyterLab UI through the Component Gateway, where an error dialog would pop up after 5 minutes saying "Failed to fetch".
Expose Presto UI in under component gateway when Presto Dataproc Optional Component is enabled.
New sub-minor versions of Cloud Dataproc images: 1.2.86-debian9, 1.3.46-debian9, 1.4.17-debian9, 1.5.0-RC2-debian9, 1.3.46-ubuntu18, 1.4.17-ubuntu18, 1.5.0-RC2-ubuntu18
Added a warning when clusters are created with component gateway and kerberos as they are not currently supported together
1.3 image update - Upgraded Hive to version 2.3.6
1.4 image update - Fixed a bug in the Jupyter component that prevented creating text files using the Jupyter and JupyterLab UIs
1.5 preview image update:
java.net.preferIPv4Stack=true property in Hadoop1.3 and 1.4 images:
Cloud Dataproc images in the 1.3 and 1.4 minor version tracks are affected by an issue that causes the YARN Timeline Server to excessively log without proper rotation. This can specifically impact longer running Clusters and Clusters with high traffic of YARN applications (Dataproc Jobs).
There is a script that can be run as an initialization action or on live Master Nodes to mitigate this issue on new and running Clusters.
Allow creation of Dataproc master and worker VMs with Intel Optane DC Persistent memory using dataproc:dataproc.alpha.master.nvdimm.size.gb and dataproc:dataproc.alpha.master.nvdimm.size.gb properties. Note: Optane VMs can only be created in zones with only n1-highmem-96-aep machine type (such as us-central1-f).
Support updating Cloud Storage connector version by setting the GCS_CONNECTOR_VERSION metadata key. For example, --metadata GCS_CONNECTOR_VERSION=2.0.0.
Breaking Changes:
Dataproc will no longer accept requests to create new clusters using Debian 8 images. Existing clusters that are using Debian 8 can still be used and scaled.
The alpha version of the Kerberos optional component option has been removed from the v1beta2 API. To use Kerberos, please configure it via the SecurityConfig message instead.
Disallow setting or overriding goog-dataproc-* labels when creating or updating clusters. All attempts to set goog-dataproc-* labels on creation will be ignored. Any existing custom goog-dataproc-* labels on running clusters will be removed if cluster labels are updated, and attempts to add new custom goog-dataproc-* labels to existing clusters will be ignored.
Reset autoscaling if the same or a new autoscaling policy is attached to a cluster. E.g. through gcloud dataproc clusters update --autoscaling-policy=<policy-name>. Note that resetting autoscaling does not change the cluster's status. Autoscaling will only run while the cluster's status is RUNNING or UPDATING, and does not work if the cluster is in ERROR. In addition, if the cluster was previously resizing (UPDATING), the autoscaler will not be able to scale the cluster until the previous update operation has completed.
gcloud dataproc clusters update --disable-autoscaling and then gcloud dataproc clusters update --autoscaling-policy=<same-policy> should now correctly reset autoscaling regardless of how long autoscaling was disabled.New Dataproc image versions: 1.2.85-debian9, 1.3.45-debian9, 1.4.16-debian9, 1.3.45-ubuntu18, 1.4.16-ubuntu18, and Preview.
Update Conscrypt library to 1.4.2
Failed Jobs error messages now include a link to the logs in the Cloud Console rather than to the Google Cloud Storage directory containing them.
Fixed race condition where deleting a cluster shortly after updating it would not delete all of the cluster's VMs
Allow compute resource names with an Alpha API version. For example: https://compute.googleapis.com/compute/alpha/projects/my-project/global/networks/my-network.
gcloud alpha dataproc clusters create <cluster-name> --network=my-network to succeed, since gcloud sets the compute API version to the same as the gcloud command track.gcloud command-line tool, the Google Cloud Platform Console, or the Cloud Dataproc API.New sub-minor versions of Cloud Dataproc images: 1.1.121-debian9, 1.2.84-debian9, 1.3.44-debian9, 1.4.15-debian9, 1.3.44-ubuntu18, 1.4.15-ubuntu18.
Image 1.4
Added support for specifying images by image family URI when creating Dataproc clusters.
1.1.120-debian9, 1.2.83-debian9, 1.3.43-debian9, 1.4.14-debian9, 1.3.43-ubuntu18, 1.4.14-ubuntu18.druid-broker, druid-overlord, druid-coordinator, druid-historical, and druid-middlemanager property prefixes.1.1.119-debian9, 1.2.82-debian9, 1.3.42-debian9, 1.4.13-debian9, 1.3.42-ubuntu18, 1.4.13-ubuntu18.dataproc:jupyter.hub.enabled can now be used to enable jupyterhub-singleuser (version 1.0.0) when using the JUPYTER optional component. This is intended to be used with DataprocSpawner. Properties dataproc:jupyter.hub.env and dataproc:jupyter.hub.args are populated by the spawner.1.1.118-debian9, 1.2.81-debian9, 1.3.41-debian9, 1.4.12-debian9, 1.3.41-ubuntu18, 1.4.12-ubuntu18.1.1.117-debian9, 1.2.80-debian9, 1.3.40-debian9, 1.4.11-debian9, 1.3.40-ubuntu18, 1.4.11-ubuntu18.1.1.116-debian9, 1.2.79-debian9, 1.3.39-debian9, 1.4.10-debian9, 1.3.39-ubuntu18, 1.4.10-ubuntu18.fs.defaultFS was passed at job submission and the directory /user/spark/eventlog did not exist in the specified FileSystem. The spark event log directory now defaults to hdfs:///user/spark/eventlog without regard to the default FileSystem. Users can override this behavior by setting the properties spark.eventLog.dir and spark.history.fs.logDirectory when creating a cluster or submitting a job.fsck fails on Kerberized clusters, which can defeat cluster downscaling.yarn:yarn.node-labels.enabled). Non-exclusive node labels are not compatible with autoscaling since they cause YARN to incorrectly report cluster utilization metrics (including pending memory). See YARN-90881.1.115-debian9, 1.2.78-debian9, 1.3.38-debian9, 1.4.9-debian9, 1.3.38-ubuntu18, 1.4.9-ubuntu181.1.114-debian9, 1.2.77-debian9, 1.3.37-debian9, 1.4.8-debian9, 1.3.37-ubuntu18, 1.4.8-ubuntu181.1.113-debian9, 1.2.76-debian9, 1.3.36-debian9, 1.4.7-debian9, 1.3.36-ubuntu18, 1.4.7-ubuntu18June 2019 Release Note Bulletin
This bulletin applies if you override the defaultFS at job-submission time for Spark jobs.
An inadvertent change in behavior has been identified for determining the location of the Spark eventlog directory when setting spark.hadoop.fs.defaultFS to any setting other than HDFS at Spark job-submission time in the following Cloud Dataproc June 2019 image versions:
1.1.113-debian9, 1.2.76-debian9, 1.3.36-debian9, 1.4.7-debian9, 1.3.36-ubuntu18, 1.4.7-ubuntu18 1.1.114-debian9, 1.2.77-debian9, 1.3.37-debian9, 1.4.8-debian9, 1.3.37-ubuntu18, 1.4.8-ubuntu18
Although the /user/spark/eventlog directory normally stays in HDFS, it began referring to the specified filesystem from spark.hadoop.fs.defaultFS in these image versions, and can result in the failure of Spark jobs with an error similar to "java.io.FileNotFoundException: File not found : /user/spark/eventlog".
Use any of the following strategies as a workaround:
Run the following command manually before submitting jobs:
hdfs dfs -mkdir -p <filesystem base/bucket>/user/spark/eventlog
Add the following Spark properties to Spark jobs to override defaultFS:
spark.eventLog.dir=hdfs:///user/spark/eventlog,spark.history.fs.logDirectory=hdfs:///user/spark/eventlog
Temporarily pin to the immediately prior image version numbers until image versions newer than those listed above are available.
1.1.111-debian9, 1.2.74-debian9, 1.3.34-debian9, 1.4.5-debian9, 1.3.34-ubuntu18, 1.4.5-ubuntu18Hive.copyFiles method in Spark's fork of Hive.1.0.119-debian9, 1.1.110-debian9, 1.2.73-debian9, 1.3.33-debian9, 1.4.4-debian9, 1.3.33-ubuntu18, 1.4.4-ubuntu18dataproc:{jupyter,zeppelin}.port properties1.0.118-debian9, 1.1.109-debian9, 1.2.72-debian9, 1.3.32-debian9, 1.4.3-debian9, 1.3.32-ubuntu18, 1.4.3-ubuntu18dataproc.googleapis.com/cluster/job/running_count metric was not consistent with the active running jobs count on the Cloud Dataproc Job page in Google Cloud Platform console.1.0.117-debian9, 1.1.108-debian9, 1.2.71-debian9, 1.3.31-debian9, 1.4.2-debian9, 1.3.31-ubuntu18, 1.4.2-ubuntu18Hive.copyFiles method in Spark's fork of Hive.www.googleapis.com to https://compute.googleapis.com due to a change in compute client.dataproc:am.primary_only cluster property with autoscaling since this property, which restricts the scheduling of app masters to primary workers, uses YARN Node Labels. Non-exclusive node labels are not compatible with autoscaling since they cause YARN to incorrectly report cluster utilization metrics (including pending memory). See YARN-9088 Note: Hadoop MapReduce and Spark jobs in client mode, which is the Cloud Dataproc default, are resilient to app master failures (scheduling app masters on primary workers is not necessary in client mode).dataproc:dataproc.logging.stackdriver.job.driver.enable=true service property can now be used to enable sending Job driver logs to Stackdriver. Logs will be sent to the Stackdriver "Cloud Dataproc Job" resource under the dataproc.job.driver log.dataproc:dataproc.logging.stackdriver.job.yarn.container.enable service property can now be used to enable sending YARN container logs to Stackdriver. Logs will be sent to the Stackdriver "Cloud Dataproc Job" resource under the dataproc.job.yarn.container log (instead of the "Cloud Dataproc Cluster" resource under the yarn-userlogs log).java.lang.IllegalArgumentException: occurrences cannot be negative.yarn.nodemanager.local-dirs property from yarn-site.xml in a Kerberized cluster failed cluster creation.1.0.116-debian9, 1.1.107-debian9, 1.2.70-debian9, 1.3.30-debian9, 1.4.1-debian9, 1.3.30-ubuntu18, 1.4.1-ubuntu18dataproc:jobs.file-backed-output.enable service property now configures Cloud Dataproc jobs to pipe their output to temporary files in the /var/log/google-dataproc-job directory.Hive.copyFiles method in the Spark fork of Hive.asia-northeast2 region (Osaka).Announcing the Alpha release of Cloud Dataproc Component Gateway, which provides secure access to web endpoints for Cloud Dataproc core and optional components without relying on SSH tunnels or modifying firewall rules to allow inbound traffic (applies to clusters created with image version 1.3.29 and later).
1.0.115-debian9, 1.1.106-debian9, 1.2.69-debian9, 1.3.29-debian9, 1.4.0-debian9, 1.3.29-ubuntu18, 1.4.0-ubuntu18europe-west6 region (Zürich, Switzerland).1.0.114-deb9, 1.1.105-deb9, 1.2.68-deb9, 1.3.28-deb9, 1.4.0-RC12-deb9dataproc:alpha.autoscaling.secondary.max_workers had to be greater than 0.1.0.113-deb9, 1.1.104-deb9, 1.2.67-deb9, 1.3.27-deb9, 1.4.0-RC11-deb9replaceFiles method.Hive.moveFile method.FileUtils.trashFilesUnderDir method./usr/local/share/google/dataproc/lib directory.1.0.111-deb9, 1.1.102-deb9, 1.2.65-deb9, 1.3.25-deb9, 1.4.0-RC9-deb9) and the changes and fixes associated with those image versions are currently not available. The latest images versions available for cluster creation, and the changes and fixes associated with those image versions, are those announced on February 11, 2019 (1.0.110-deb9, 1.1.101-deb9, 1.2.64-deb9, 1.3.24-deb9, 1.4.0-RC8-deb9).1.0.111-deb9, 1.1.102-deb9, 1.2.65-deb9, 1.3.25-deb9, 1.4.0-RC9-deb9yarn.resourcemanager.webapp.methods-allowed property now defaults to "GET,HEAD". This change restricts the HTTP methods that can be called on the YARN Resource Manager web UI (default port 8088) and REST APIs to GET and HEAD only, and disables job submission and modifications via the YARN REST API. You can override the default values and enable specific HTTP methods on port 8088 by setting the yarn.resourcemanager.webapp.methods-allowed property to one or more comma-separated HTTP method names when you create a cluster. An ALL value will allow all HTTP methods on the port.gcloud dataproc clusters create --properties='yarn:yarn.resourcemanager.webapp.methods allowed=GET,POST,DELETE'replaceFiles method1.0.110-deb9, 1.1.101-deb9, 1.2.64-deb9, 1.3.24-deb9, 1.4.0-RC8-deb9.yarn.resourcemanager.webapp.methods-allowed) which is set with a comma- separated list of the names of HTTP methods that the YARN Resource Manager will allow to be called in its web UI and REST API (default port 8088). The current default value is "ALL", which allows all HTTP methods (GET,POST,PUT,PATCH,DELETE,HEAD,CONNECT,OPTIONS, TRACE). It is recommended that this property be set to "GET" to disable remote code submission or other modifications via the REST API. This property will default to "GET" for all Cloud Dataproc clusters in a future release (see Important cross-update notes, above).yarn.resourcemanager.webapp.methods-allowed) which is set with a comma- separated list of the names of HTTP methods that the YARN Resource Manager will allow to be called in its web UI and REST API (default port 8088). The current default value is "ALL", which allows all HTTP methods (GET,POST,PUT,PATCH,DELETE,HEAD,CONNECT,OPTIONS, TRACE). It is recommended that this property be set to "GET" to disable remote code submission or other modifications via the REST API. This property will default to "GET" for all Cloud Dataproc clusters in a future release (see Important cross-update notes, above).1.0.109-deb9, 1.1.100-deb9, 1.2.63-deb9, 1.3.23-deb9, 1.4.0-RC7-deb9.1.0.108-deb9, 1.1.99-deb9, 1.2.62-deb9, 1.3.22-deb9, 1.4.0-RC6-deb9.1.0.107-deb9, 1.1.98-deb9, 1.2.61-deb9, 1.3.21-deb9, 1.4.0-RC5-deb9.yarn.resourcemanager.nodemanager-graceful-decommission-timeout-secs to 86400 (1 day) on Cloud Dataproc 1.3+ clusters. This fixes the bug where trying to graceful decommission nodes with a timeout on Cloud Dataproc 1.3+ would still terminate at 1 hour.hadoop.ssl.enabled.protocols to "TLSv1,TLSv1.1,TLSv1.2". This change will affect any newly created clusters using a Dataproc image 1.2 and up. This is to help avoid issues caused by incompatibility between SSLv2Hello (enabled by default in Hadoop 2.8.4 and up) and the Conscrypt security provider, the default security provider used by the Cloud Storage connector.hadoop.ssl.enabled.protocols. It will be set by the Dataproc service.1.0.105-deb9, 1.1.96-deb9, 1.2.59-deb9, 1.3.19-deb9, 1.4.0-RC3-deb9.1.0.104-deb9, 1.1.95-deb9, 1.2.58-deb9, 1.3.18-deb9, 1.4.0-RC2-deb9.2.3.1 to 2.3.2 on image version 1.3./etc/hadoop/conf.empty/yarn-site.xml under yarn.application.classpath prevents running Hadoop jobs.1.2 only:
featureImportances results on computed vs. saved models. (SPARK-25959).1.3 and later only:
gs://staging-bucket/notebooks to gs://staging-bucket/notebooks/jupyter. This can be changed by setting dataproc:jupyter.notebook.gcs.dir cluster property.gs://staging-bucket/notebooks/zeppelin. This can be changed by setting zeppelin:zeppelin.notebook.gcs.dir cluster property.zeppelin and zeppelin-env property prefixes.gcloud beta dataproc clusters list and gcloud beta dataproc clusters describe commands.gcloud command-line tool, the Google Cloud Platform Console, or the Cloud Dataproc API.1.0.102-deb9, 1.1.93-deb9, 1.2.56-deb9, 1.3.16-deb9.PATH in custom images broke Cloud Dataproc initialization. For example, changing the default Python to Python 3 previously broke initialization. * Fixed a bug where POST ant PUT requests to the YARN REST API were blocked by anonymous users on Cloud Dataproc 1.3. This was fixed by adding org.apache.hadoop.http.lib.StaticUserWebFilter back to hadoop.http.filter.initializers in core-site.xml * Fixed logging warnings in Hive 2 in Cloud Dataproc 1.1, 1.2, and 1.3.1.0.101-deb9, 1.1.92-deb9, 1.2.55-deb9, 1.3.15-deb9.fadvise to SEQUENTIAL for Hadoop DistCp jobs. This mode is optimized for streaming reads, which are most efficient for these workloads.netty-tcnative, for example, io.grpc:grpc-netty-shaded. This already applies to 1.2 and 1.3.dfs.namenode.datanode.registration.retry-hostname-dns-lookup is now set to true.listManagedInstances on the Managed Instance Group that manages secondary workers, filters out instances whose current action is DELETING or ABANDONING, and then chooses instances to delete from the resulting set. Cloud Dataproc prefers to delete VMs that are being created, rather than running VMs.1.0.100-deb8, 1.1.91-deb8, 1.2.54-deb8, 1.3.14-deb8, 1.0.100-deb9, 1.1.91-deb9, 1.2.54-deb9, 1.3.14-deb9.asia-east2 region (Hong Kong).1.0.99-deb8, 1.1.90-deb8, 1.2.53-deb8, 1.3.13-deb8, 1.0.99-deb9, 1.1.90-deb9, 1.2.53-deb9, 1.3.13-deb9.1.0.98-deb8, 1.1.89-deb8, 1.2.52-deb8, 1.3.12-deb8, 1.0.98-deb9, 1.1.89-deb9, 1.2.52-deb9, 1.3.12-deb9.debconf: DbDriver "config": /var/cache/debconf/config.dat is locked by another process: Resource temporarily unavailable error.1.0.97-deb8, 1.1.88-deb8, 1.2.51-deb8, 1.3.11-deb8, 1.0.97-deb9, 1.1.88-deb9, 1.2.51-deb9, 1.3.11-deb9.1.0.97-deb8, 1.1.88-deb8, 1.2.51-deb8, 1.3.11-deb8, 1.0.97-deb9, 1.1.88-deb9, 1.2.51-deb9, 1.3.11-deb9.dataproc:am.primary_only cluster property to prevent application master from running on preemptible workers. This feature is only enabled for Dataproc 1.2+ clusters. To use the cluster property, set --properties dataproc:am.primary_only=true when creating a cluster.1.0.95-deb8, 1.1.86-deb8, 1.2.49-deb8, 1.3.9-deb8, 1.0.95-deb9, 1.1.86-deb9, 1.2.49-deb9, 1.3.9-deb9.1.3-deb9 images.1.0.94-deb8, 1.1.85-deb8, 1.2.48-deb8, 1.3.8-deb8, 1.0.94-deb9, 1.1.85-deb9, 1.2.48-deb9, 1.3.8-deb9.Flink 1.5.0 and HBase 1.3.2 to 1.3-deb8 images.E: Could not get lock /var/lib/dpkg/lock error.1.0.93-deb8, 1.1.84-deb8, 1.2.47-deb8, 1.3.7-deb8, 1.0.93-deb9, 1.1.84-deb9, 1.2.47-deb9, 1.3.7-deb9.1.0.92-deb8, 1.1.83-deb8, 1.2.46-deb8, 1.3.6-deb8, 1.0.92-deb9, 1.1.83-deb9, 1.2.46-deb9, 1.3.6-deb9.1.0.91-deb8, 1.0.91-deb9, 1.1.82-deb8, 1.1.82-deb9, 1.2.45-deb8, 1.2.45-deb9, 1.3.5-deb8, 1.3.5-deb9.1.0.90, 1.1.81, 1.2.45, 1.3.5mapreduce.jobhistory.recovery.store.fs.uri by setting it back to ${hadoop.tmp.dir}/mapred/history/recoverystore. It was set to hdfs:///mapred/history/recoverystore in error as part of the July 6 release.1.0.89, 1.1.80, 1.2.44, 1.3.4hadoop.security.token.service.use_ip is now set to false.1.3.3tez.am.node-blacklisting.enabled=false). This affects all Hive jobs, which run on Tez by default.gcloud command-line tool.1.0.88, 1.1.79, 1.2.43, 1.3.2us-west2 region (Los Angeles).1.0.87, 1.1.78, 1.2.42, 1.3.11.0.86, 1.1.77, 1.2.41, 1.3.01.0.85, 1.1.76, 1.2.401.0.84, 1.1.75, 1.2.39google.dataproc.startup log.europe-north1 region (Finland).gcloud beta dataproc workflow-templates instantiate-from-file to enable instantiation of workflow templates directly from a YAML file.gcloud beta dataproc clusters create-from-file to enable creation of clusters directly from a YAML file.1.0.83, 1.1.74, 1.2.381.0.82, 1.1.73, 1.2.37SPARK_HOME environment variable is now globally set.1.0.81, 1.1.72, 1.2.361.0.80, 1.1.71, 1.2.351.0.79, 1.1.70, 1.2.341.0.78, 1.1.69, 1.2.331.0.77, 1.1.68, 1.2.32http://clustername-m-0:9870/webhdfs/v1/. This is consistent with standard and single node clusters in Dataproc 1.2+. Dataproc 1.0 and 1.1 clusters will continue to use port 50070 for all cluster modes.ERROR state due to error when resizing a managed instance group.NullPointerException in Pig jobs.1.0.76, 1.1.67, 1.2.311.0.75, 1.1.66, 1.2.301.0.74, 1.1.65, 1.2.291.0.73, 1.1.64, 1.2.28gcs-connector-1.6.4 and the BigQuery connector has been upgraded to bigquery-connector-0.10.5. For more information, review the GitHub repository 2018-03-19 (Google Cloud Storage 1.6.4, BigQuery 0.10.5) changelog.1.0.73, 1.1.64, 1.2.28 .1.0.71, 1.1.62, 1.2.26.1.0.70, 1.1.61, 1.2.25.requestId field has been added to CreateCluster, UpdateCluster, DeleteCluster, and SubmitJob. The requestId field can be used to prevent the processing of duplicate requests (subsequent requests with a requestId that is the same as a previous requestId are ignored).1.0.69, 1.1.60, 1.2.24.1.0.68, 1.1.59, 1.2.23.1.0.67, 1.1.58, 1.2.22.The High Availability Mode feature is now in public release (was Beta). Cloud Dataproc clusters can be created with high availability mode enabled. This is an optional feature when creating a cluster. In high availability mode, Cloud Dataproc clusters have three master nodes instead of one. This enables both HDFS High Availability and YARN High Availability to allow uninterrupted YARN and HDFS operations despite any single-node failures or reboots.
This feature is available when creating clusters with the [gcloud](/sdk/gcloud/reference/dataproc/) command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See High Availability Mode for more information.
An Update Cluster operation now returns a DONE operation if the update request has no work to perform.
commitJob on Cloud Storage for jobs with many last-stage (reduce) tasks.1.0.66, 1.1.57, 1.2.21.gcs-connector-1.6.3 and the Google BigQuery connector has been upgraded to bigquery-connector-0.10.4. For more information, review the GitHub repository Cloud Storage change notes and BigQuery change notes.northamerica-northeast1 region (Montréal, Canada).europe-west4 region (Netherlands).1.0.63, 1.1.54, 1.2.18.mapreduce.fileoutputcommitter.failures.attempt to 1 to revert to the old behavior.The Single Node Clusters feature is now in public release (was Beta). Single node clusters are Cloud Dataproc clusters with only one node that acts as the master and worker. Single node clusters are useful for a number of activities, including development, education, and lightweight data science.
This feature is available when creating clusters with the gcloud command-line tool, the Cloud Dataproc REST API, and Google Cloud Platform Console. See Single node clustersfor more information.
1.0.58, 1.1.49, 1.2.13.1.0.57, 1.1.48, 1.2.12.2.8.2 in the Cloud Dataproc 1.2 image./locations/ in resource names.1.0.56, 1.1.47, 1.2.11.404 and 409 errors will now show the full resource name in order to provide more useful error messages.Cloud Dataproc is now available in the asia-south1 region (Mumbai, India).
WorkflowTemplate operations after October 27, 2017 will be logged in Cloud Audit Logging.1.0.55, 1.1.46, 1.2.10.java.lang.NullPointerException: ssl == null errors when accessing Cloud Storage.1.0.54, 1.1.45, 1.2.9.fluentd on Cloud Dataproc clusters has been reconfigured to concatenate multi-line error messages. This should make error messages easier to locate.1.0.53, 1.1.44, 1.2.8.1.0.51, 1.1.42, 1.2.6, respectively.Cloud Dataproc is now available in the southamerica-east1 region (São Paulo, Brazil).
1.0.49, 1.1.40, 1.2.4, respectively.goog-dataproc-cluster-name label that is propagated to underlying Google Compute Engine resources and can be used to determine combined Cloud Dataproc related costs in exported billing data.Beginning today, Cloud Dataproc 1.2 will be the default version for new clusters. To use older versions of Cloud Dataproc, you will need to manually select the version on cluster creation.
Apache Hadoop on Cloud Dataproc 1.2 has been updated to version 2.8.1
Graceful decomissioning – Cloud Dataproc clusters running Cloud Dataproc 1.2 or later now support graceful YARN decommissioning. Graceful decomissioning enables the removal of nodes from the cluster without interrupting jobs in progress. A user-specified timeout specifies how long to wait for jobs in progress to finish before forcefully removing nodes. The Cloud Dataproc scaling documentation contains details how enable graceful decomissioning.
Cloud Dataproc is now available in the europe-west3 region (Frankfurt, Germany).
1.2. It will become the default image version for new clusters starting in 2 weeks. See the Cloud Dataproc version list for more information. Some important changes included in this new image version:
Starting with the Cloud Dataproc 1.2 release, the ALPN boot jars are no longer provided on the Cloud Dataproc image. To avoid Spark job breakage, upgrade Cloud Bigtable client versions, and bundle boringssl-static with Cloud Dataproc jobs. Our initialization action repository contains initialization actions to revert to the previous (deprecated) behavior of including the jetty-alpn boot jar. This change should only impact you if you use Cloud Bigtable or other Java gRPC clients from Cloud Dataproc.
The v1alpha1 and v1beta1 Cloud Dataproc APIs are now deprecated and cannot be used. Instead, you should use the current v1 API.
Cloud Dataproc is now available in the australia-southeast1 region (Sydney).
Cloud Dataproc is now available in the europe-west2 region (London).
Cloud Storage and BigQuery connector upgrades – The Cloud Storage connector has been upgraded to gcs-connector-1.6.1 and the BigQuery connector has been upgraded to bigquery-connector-0.10.2. For more information, review the Cloud Storage or BigQuery change notes in the GitHub repository.
The v1alpha1 and v1beta1 Cloud Dataproc APIs are now deprecated and cannot be used. Instead, you should use the current v1 API.
Dataproc/Dataproc Worker IAM role.Apache Hive on Cloud Dataproc 1.1 has been updated to version 2.1.1.
The Conscrypt security provider has been temporarily changed from the default to an optional security provider. This change was made due to incompatibilities with some workloads. The Conscrypt provider will be re-enabled as the default with the release of Cloud Dataproc 1.2 in the future. In the meantime, you can re-enable the Conscrypt provider when creating a cluster by specifying this Cloud Dataproc property:
--properties dataproc:dataproc.conscrypt.provider.enable=true
Cloud Dataproc worker IAM role – A new Cloud Dataproc IAM role called Dataproc/Dataproc Worker has been added. This role is intended specifically for use with service accounts.
Updates to user labels applied to Cloud Dataproc clusters will now be applied to managed instance group templates. Since preemptible virtual machines are included in a managed instance group, label updates are now applied to preemptible VMs.
Conscrypt for Cloud Storage connector – The default security (SSL) provider used by the Cloud Storage connector has been changed to one based on Conscrypt. This change should more efficiently utilize the CPU for SSL operations. In many cases, this change should result in better read and write performance between Cloud Dataproc and Cloud Storage.
As mentioned in the February 9th release notes, Cloud Audit Logs for Cloud Dataproc are no longer emitted to the dataproc_cluster resource type. Starting with this release, Cloud Audit Logs are emitted to the new cloud_dataproc_cluster resource type.
The gcloud command now requires a double dash (--) between gcloud-specific arguments and arguments to those commands. For example, if you used this command in the past:
gcloud dataproc jobs submit spark --cluster example-cluster \
--class sample_class --jars jar_file 1000
The new format for the command requires a double-dash with a space before and after the double dash:
gcloud dataproc jobs submit spark --cluster example-cluster \
--class sample_class --jars jar_file -- 1000
.
cloud_dataproc_cluster (was previously dataproc_cluster).cloud_dataproc_cluster and dataproc_cluster (deprecated) until the March 9th release.1.2 preview – The preview image has been updated to reflect the planned Cloud Dataproc 1.2 release. This image includes Apache Spark 2.1 and Apache Hadoop 2.8-SNAPSHOT. This preview image is provided so we can provide access to Hadoop 2.8 in Dataproc 1.2 once Hadoop 2.8 is formally released and access to release candidates.gcs-connector-1.6.0 and the BigQuery connector has been upgraded to bigquery-connector-0.10.1. For more information, review the Cloud Storage or BigQuery change notes in the GitHub repository.systemd on clusters in the event of unexpected or unhealthy behavior.--jars argument was missing from gcloud dataproc jobs submit pyspark command.bigquery-connector-0.10.1-SNAPSHOT. This version introduces the new IndirectBigQueryOutputFormat that uses Hadoop output formats that write directly to a temporary Cloud Storage bucket, and issues a single BigQuery load job per Hadoop/Spark job at job-commit time. For more information, review the BigQuerychange notes in the GitHub repository.key=value labels to Cloud Dataproc clusters and jobs. This allows you to group resources and related operations for later filtering and listing. As an example, you can use labels with clusters to break out Cloud Dataproc usage by groups or individuals. For more information see the user labels documentation.Running state.gcs-connector-1.5.4 and the BigQuery connector has been upgraded to bigquery-connector-0.8.0. For more information, review the Cloud Storage or BigQuerychange notes in the GitHub repository.dataproc.localssd.mount.enable that can be set at cluster deployment time to make Cloud Dataproc ignore local SSDs. If set, Cloud Dataproc will use the main persistent disks for HDFS and temporary Hadoop directories so the local SSDs can be used separately for user-defined purposes. This property can be set by using the argument --properties dataproc:dataproc.localssd.mount.enable=false when creating a Cloud Dataproc cluster.Hadoop High Availability Mode [BETA] – Cloud Dataproc clusters can be created with high availability mode enabled. This is an optional feature when creating a cluster. In high availability mode, Cloud Dataproc clusters have three master nodes instead of one. This enables both HDFS High Availability and YARN High Availability to allow uninterrupted YARN and HDFS operations despite any single-node failures or reboots.
Presently, this feature is available when creating clusters with the [gcloud](/dataproc/docs/gcloud-installation) command-line tool or the Cloud Dataproc REST API. A future release will enable support for creating clusters with high availability in the Google Cloud Platform Console.
See the high availability mode documentation for more information.
gcs-connector-1.5.3 and the BigQuery connector has been upgraded to bigquery-connector-0.7.9. For more information, review the change notes in the GitHub repository.--preemptible-worker-boot-disk-size command no longer requires that you specify 0 preemptible workers if you do not want to add preemptible machines when you create a cluster.gcloud command-line tool at cluster creation, even when preemptibles are not added to a cluster using the commend --preemptible-worker-boot-disk-size.1.0.11 instead of 1.0). This makes it easier to temporarily revert to an older sub-minor version. See the Cloud Dataproc versioning for more information.--properties dataproc:dataproc.logging.stackdriver.enable=false when creating a cluster with the gcloud command-line tool.Cloud Dataproc 1.1 default – Cloud Dataproc 1.1 is now the default image version for new clusters.
gcs-connector-1.5.2 and the BigQuery connector has been upgraded to bigquery-connector-0.7.8, which bring performance improvements. See the release notes for the gcs-connector and bigquery-connector for more information.0.6.1. This new version of Zeppelin brings support for Google BigQuery.Image versions released after Cloud Dataproc became generally available, starting with 1.0, will be subject to the Cloud Dataproc versioning policy.
The two Cloud Dataproc image versions released during the Cloud Dataproc beta, 0.1 and 0.2, will no longer receive updates. You can continue to use the beta images; however, no new updates, such as bug fixes and connector updates, will be applied to these two deprecated image versions.
Cloud SDK release 121.0.0 – Updated several gcloud dataproc arguments.
--preemptible-worker-boot-disk-size argument has been promoted to general availability and can be used to adjust the persistent disk size (in GB) of preemptible workers.--master-boot-disk-size-gb and --worker-boot-disk-size-gb arguments have been removed. Use --master-boot-disk-size and --worker-boot-disk-size instead.Cloud Dataproc 1.1 – A new image version, Cloud Dataproc 1.1, has been released. Several components have been updated for this image version including:
2.0.02.1.00.16.0To create a cluster with the 1.1 image, you can use the gcloud command-line tool with the --image-version argument, such as gcloud dataproc clusters create --image-version 1.1.
Fixed an issue in which the NFS-based Cloud Storage consistency cache would not get cleaned on long-running clusters with sustained high file creation rates (> ~1,000,000 files per hour for a sustained period of time).
Updated preview image – Several preview image components have been updated, including:
2.0.02.1.00.16.0Cloud Storage and BigQuery connector upgrades – The Cloud Storage connector has been upgraded to gcs-connector-1.5.1 and the BigQuery connector has been upgraded to bigquery-connector-0.7.7. See the release notes for the gcs-connector and bigquery-connector for more information.
us-west1 region - Cloud Dataproc is available from day
one in the newly announced west-1
region.
As mentioned in the announcement, some West Coast users may see reductions
in latency.1.0 Cloud
Dataproc image version has been
upgraded from 1.6.1 to 1.6.2.gcs-connector-1.5.0 and the BigQuery
connector has been upgraded to bigquery-connector-0.7.6. These new
versions bring a number of new features and fixes.
fs.gs.outputstream.type property
to SYNCABLE_COMPOSITE.v1 instead of the
v1beta1 API. Clicking on the equivalent REST link will show the
appropriate v1 API paths and resource names.gcloud command-line tool
--preemptible-worker-boot-disk-size which can be used
to adjust the boot disk size of preemptible workers. This was added in
the gcloud beta track.--*-boot-disk-size-gb flag is now deprecated in all tracks and has
been replaced by the --*-boot-disk-size commands.FAILED state instead of remaining in a DELETING state.--properties
command from changing
MapReduce properties.1.0 has been updated to
include the Spark
1.6.1 maintenance
release instead of Spark 1.6.0.--num-preemptible-workers 0 through the
gcloud command-line tool. Previously this would fail.500 HTTP errors when the response
should have been 400 bad input or 200 OK.fs.gs.implicit.dir.infer.enable).Too many
pending operations on a resource error.https://<authority>/compute/<version>/projects/...compute/<version>/projects/...projects/...gcloud command-line
tool. You can now use the --subnet SUBNET command to specify a subnetwork
when you create a Cloud Dataproc cluster.Cloud Dataproc is now generally available! For more information, see our announcement blog post.
Custom compute engine machine types - Cloud Dataproc clusters now support custom Compute Engine machine types for both master and worker nodes. This means you can create clusters with customized amounts of virtual CPUs and memory. For more information, please read the Dataproc documentation for custom machine types.
OSS upgrades - We have released Cloud Dataproc version
1.0.
This release includes an upgrade to Apache Spark 1.6.0 and Apache Hadoop
2.7.2. This release also includes new versions of the Cloud Storage
and Google BigQuery
connectors.
v1 API - The v1 API for Cloud Dataproc is now live. This API includes
support for regionality along with minor fixes and adjustments. The API is
available in the APIs
Explorer and
also has a Maven
artifact
in Maven Central. For more information, review the REST API
documentation.
Support for --jars for PySpark - Added support for using the --jars
option for PySpark jobs.
API auto-enable - Enabling the Cloud Dataproc API now automatically enables required dependent APIs, such as Cloud Storage and Google Compute Engine
DeleteJob command.RUNNING state after the job
completed successfully.--image-version option has been added: preview
0.1 and 0.2, the preview
version will contain newer Hadoop/Spark/Pig/Hive components targeted for
potential release in the next stable Cloud Dataproc distribution
version, and will change over time.preview version contains Spark 1.6.0, with
the same Hadoop/Pig/Hive versions as Cloud Dataproc 0.2.preview option is being rolled out to the GCP Console
gradually, so may not be visible in your account for another week or so.
For all users, the preview option is accessible by deploying clusters
with the gcloud command-line tool.maxExecutors setting is now set to 10000 to avoid the
AppMaster failing on jobs with many tasksgcloud command-line
tool for adding tags and metadata to virtual machines used in Cloud Dataproc
clusters. These tags and metadata will apply to both regular and preemptible
instances.
--tags option will add tags to the Google Compute Engine instances
in a cluster. For example, using the argument --tags foo,bar,baz will
add three tags to the virtual machine instances in the cluster.--metadata option will add metadata to the compute engine
instances. For example, using --metadata 'meta1=value1,key1=value2'
will add two key-value pairs of metadata.Line wrapping option to
make it easier to view job output containing very long lines--driver-logging-levels option
no longer interferes with Java driver options0.1 that caused zero-byte _SUCCESS and
_FAILURE files for each job to be continually re-written to Cloud Storage.--properties
option for adding or updating properties in some cluster configuration
files, such as core-site.xml. Properties are mapped to configuration files
by specifying a prefix, such as core:io.serializations. This command makes
it possible to modify multiple properties and files when creating a cluster.
For more information, see the Cloud Dataproc documentation for the
--properties
command.Preemptible workers, bucket,
network, version, initialization, & access options section at the
bottom of the form.vim, git, and bash-completion
installed by defaultClone button has been added that allows you to easily copy all
information about a job to a new job submission formImage version field has been added to the bottom of the create
cluster form that allows you to select a specific Cloud Dataproc image
version
when creating a clusterVM Instances tab has been added on the cluster detail page, which
you can use to display a list of all VMs in a cluster and easily SSH
into the master nodeInitialization Actions field has been added to the bottom of the
create cluster form, which allows you to specify initialization
actions when creating a clusterdistcp settings to match mapred-site.xml settings to provide
additional fixes for the distcp command (see this related
JIRA)Failed to load message with details about the cause
of an error when an error occurs that is not related to Cloud DataprocMain class or jar field on the Submit a Job form, and
no longer required it to also be listed in the Jar files fieldCore service improvements
Optimizations
Version selection - With the release of Cloud Dataproc version 0.2, you can now select among different versions of Cloud Dataproc (see Cloud Dataproc Versioning for information on support for previous versions and Cloud Dataproc Version List a list of the software components supported in each version). You can select a Cloud Dataproc version when creating a cluster through the Cloud Dataproc API, Cloud SDK (using the gcloud beta dataproc clusters create --image-version command) or through the https://console.cloud.google.com/. Note that within four days of the release of a new version in a region, the new version will become the default version used to create new clusters in the region.
OSS upgrades - We have released Cloud Dataproc version 0.2. The new Spark component includes a number of bug fixes. The new Hive component enables use of the hive command, contains performance improvements, and has a new metastore.
Connector updates - We released updates to our BigQuery and Google Cloud Storage connectors (0.7.3 and 1.4.3, respectively.) These connectors fix a number of bugs and the new versions are now included in Cloud Dataproc version 0.2.
Hive Metastore - We introduced a MySQL-based per-cluster persistent metastore, which is shared between Hive and SparkSQL. This also fixes the hive command.
More Native Libraries - Cloud Dataproc now includes native Snappy libraries. It also includes native BLAS, LAPACK and ARPACK libraries for Spark's MLlib.
Clusters --diagnose command - The Cloud SDK now includes a --diagnose command for gathering logging and diagnostic information about your cluster. More details about this command are available in the Cloud Dataproc support documentation.
distcp
commandHive command has been fixedRate Limit Exceeded errors occurredCreated column, and the cluster
configuration tab now includes a Created field, telling the creation
time of the clusterAdditional Python Files and Jar Files) have been addedGCP Console: Added support for adding, editing, and removing preemptible instances in clusters.
Error state.500 errors when deploying Cloud Dataproc clusters.distcp out-of-memory errors with better cluster configuration.Deleting state.Core service improvements
500 errors instead of showing 4xx
errors.Resource already exists
errors.Optimizations
yarn.scheduler.capacity.maximum-am-resource-percent to
enable better utilization and concurrent job support.Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2026-07-27 UTC.