/

Console

• English
• Deutsch
• Español
• Español – América Latina
• Français
• Indonesia
• Italiano
• Português
• Português – Brasil
• עברית
• 中文 – 简体
• 中文 – 繁體
• 日本語
• 한국어

Sign in

• Cloud Service Mesh

Start free

Provision a managed Cloud Service Mesh control plane on GKE

Cloud Service Mesh is a Google-managed service mesh that you simply enable. Google handles reliability, upgrades, scaling, and security for you.

This page shows you how to use the fleet API to set up managed Cloud Service Mesh using Istio APIs.

Prerequisites

As a starting point, this guide assumes that you have:

• A Cloud project
• A Cloud Billing account
• Obtained the required permissions to provision Cloud Service Mesh
• Enabled Workload Identity on your clusters and node pools.

Requirements

• One or more clusters with a supported version of GKE, in one of the supported regions.
• Note that managed Cloud Service Mesh uses GKE release channels to balance between stability and upgrade speed. New changes to Cloud Service Mesh in-cluster components (including CNI, MDPC, proxies, and Istio CRDs) will rollout to clusters that subscribe GKE rapid channel first. They will then be promoted to GKE regular channel, and finally the GKE stable channel, once they demonstrate enough stability.
  • Best practice is to provision a subset of your fleet clusters in an earlier GKE release channel to ensure they receive updates to the Cloud Service Mesh components first.
  • Managed Cloud Service Mesh doesn't support changing GKE release channels.
  • If you do change the GKE release channel, Cloud Service Mesh won't prevent the operation. Cloud Service Mesh automatically updates the in-cluster components (CNI, MDPC, default injected proxy version and Istio CRDs) to align with the current GKE release channel.
• Ensure that your cluster has enough capacity for the required components that managed Cloud Service Mesh installs in the cluster.
  • The mdp-controller deployment in kube-system namespace requests cpu: 50m, memory: 128Mi.
  • The istio-cni-node daemonset in kube-system namespace requests cpu: 100m, memory: 100Mi on each node.
• Ensure that the organizational policy constraints/compute.disableInternetNetworkEndpointGroup is disabled. If the policy is enabled, ServiceEntry may not work.
• Ensure that the client machine that you provision managed Cloud Service Mesh from has network connectivity to the API server.
• Your clusters must be registered to a fleet. This is included in the instructions if it's not already registered prior to the provision.
• Your project must have the Service Mesh fleet feature enabled. This is included in the instructions if you haven't enabled it yet.

• GKE Autopilot is only supported with GKE version 1.21.3+.

• Cloud Service Mesh can use multiple GKE clusters in a single-project single-network environment or a multi-project single-network environment.

  • If you join clusters that are not in the same project, they must be registered to the same fleet host project, and the clusters must be in a shared VPC configuration together on the same network.
  • For a single-project multi-cluster environment, the fleet project can be the same as the cluster project. For more information about fleets, see Fleets Overview.
  • For a multi-project environment, we recommend that you host the fleet in a separate project from the cluster projects. If your organizational policies and existing configuration allow it, we recommend that you use the shared VPC project as the fleet host project. For more information, see Setting up clusters with Shared VPC.

Roles required to install Cloud Service Mesh

The following table describes the roles that are required to install managed Cloud Service Mesh.

Roles required to run Cloud Service Mesh

The following table describes the roles that are required by the service account to run managed Cloud Service Mesh. If your network or cluster projects differ from the fleet host project, you need to give the service account in the fleet project access to these roles in the other projects.

Limitations

We recommend that you review the list of Cloud Service Mesh supported features and limitations. In particular, note the following:

• Workload pods running with hostNetwork: true are not supported.

• The IstioOperator API isn't supported since its main purpose is to control in-cluster components.

• For GKE Autopilot clusters, cross-project setup is only supported with GKE 1.23 or later.

• For GKE Autopilot clusters, in order to adapt to the GKE Autopilot resource limit, the default proxy resource requests and limits are set to 500m CPU and 512 Mb memory. You can override the default values using custom injection.

• For GKE Autopilot clusters, there may be warnings for Cloud Service Mesh components "DaemonSet has no nodes selected" until the clusters' NodePool scales.

• During the provisioning process for a managed control plane, Istio CRDs are provisioned in the specified cluster. If there are existing Istio CRDs in the cluster, they will be overwritten.

• Istio CNI and Traffic Director are not compatible with GKE Sandbox. Therefore, managed Cloud Service Mesh with the TRAFFIC_DIRECTOR implementation does not support clusters with GKE Sandbox enabled.

Before you begin

       gcloud auth login --project FLEET_PROJECT_ID

       gcloud components update

  gcloud services enable mesh.googleapis.com \
      --project=FLEET_PROJECT_ID

Configure managed Cloud Service Mesh

The steps required to provision managed Cloud Service Mesh using the fleet API depend on whether you prefer to enable it per cluster or enable by default for new fleet clusters.

Configure per cluster

Use the following steps to configure managed Cloud Service Mesh for each cluster in your mesh individually.

Enable the Cloud Service Mesh fleet feature

Enable the mesh feature on the fleet project. Note that if you plan to register multiple clusters, enabling the Cloud Service Mesh fleet feature happens at the fleet-level so you only have to run this command once.

gcloud container fleet mesh enable --project FLEET_PROJECT_ID

Register clusters to a fleet

1. Register a GKE cluster using fleet workload identity. The --location flag is the compute zone or region (such as us-central1-a or us-central1) for the cluster.

   gcloud container clusters update CLUSTER_NAME \
     --location CLUSTER_LOCATION \
     --fleet-project FLEET_PROJECT_ID
   

2. Verify your cluster is registered:

   gcloud container fleet memberships list --project FLEET_PROJECT_ID
   

   Example output:

   NAME                 EXTERNAL_ID                           LOCATION
   cluster-1            1d8e255d-2b55-4df9-8793-0435461a2cbc  us-central1
   

   Make note of the MEMBERSHIP_NAME, as you will need it when you enable automatic management.

3. If your cluster's network's project differs from your fleet host project (for example you are using a shared VPC), you must allow Cloud Service Mesh service accounts in the fleet project to access the network project. You only need to do this once for the network project.

   Grant service accounts in the fleet project permission to access the network project:

    gcloud projects add-iam-policy-binding "NETWORK_PROJECT_ID"  \
         --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
         --role roles/anthosservicemesh.serviceAgent
   

   For further details on the firewall rules created and how to configure narrower permissions, see Understand health checks.

4. If your cluster's project differs from your fleet host project, you must allow Cloud Service Mesh service accounts in the fleet project to access the cluster project, and enable required APIs on the cluster project.

   You only need to do this once for each cluster project. If you previously configured managed Cloud Service Mesh for this combination of cluster and fleet projects, then these changes have already been applied and you don't have to run the following commands.

   Grant service accounts in the fleet project permission to access the cluster project:

    gcloud projects add-iam-policy-binding "CLUSTER_PROJECT_ID" \
        --member "serviceAccount:service-FLEET_PROJECT_NUMBER@gcp-sa-servicemesh.iam.gserviceaccount.com" \
        --role roles/anthosservicemesh.serviceAgent
   

   Enable the Mesh API on the cluster's project:

    gcloud services enable mesh.googleapis.com \
        --project=CLUSTER_PROJECT_ID
   

Configure Certificate Authority Service (Optional)

If your service mesh deployment requires Certificate Authority Service (CA Service), then follow Configure Certificate Authority Service for managed Cloud Service Mesh to enable it for your fleet. Make sure to complete all steps before proceeding to the next section.

Enable automatic management

Run the following command to enable automatic management:

  gcloud container fleet mesh update \
     --management automatic \
     --memberships MEMBERSHIP_NAME \
     --project FLEET_PROJECT_ID \
     --location MEMBERSHIP_LOCATION

where:

• MEMBERSHIP_NAME is the membership name listed when you verified that your cluster was registered to the fleet.

• MEMBERSHIP_LOCATION is the location of your membership (either a region, or global).

  If you recently created the membership using the command in this guide, this should be the region of your cluster. If you have a zonal cluster, use the region corresponding to the cluster's zone. For example, if you have a zonal cluster in us-central1-c, then use the value us-central1.

  This value may be global if you registered prior to May 2023, or if you specified the global location when registering the membership. You can check your membership's location with gcloud container fleet memberships list --project FLEET_PROJECT_ID.

Proceed to Verify the control plane has been provisioned.

Configure for your fleet

You must have Google Kubernetes Engine (GKE) Enterprise edition enabled to enable managed Cloud Service Mesh as a default configuration for your fleet. This means that every new GKE on Google Cloud cluster registered during cluster creation will have managed Cloud Service Mesh enabled on the cluster. You can find out more about fleet default configuration in Manage fleet-level features.

Note that fleet-level defaults only apply automatically to new clusters created after enabling fleet defaults. You must sync existing clusters to apply new default settings.

Enabling managed Cloud Service Mesh as a default configuration for your fleet and registering clusters to the fleet during cluster creation only supports Mesh CA. If you want to use Certificate Authority Service, we recommend that you enable it per cluster.

To enable fleet-level defaults for managed Cloud Service Mesh, complete the following steps:

Proceed to Verify the control plane has been provisioned.

Terraform support

Cloud Service Mesh supports provisioning via Terraform through the GKEHub feature membership module.

Verify the control plane has been provisioned

After a few minutes, verify that the control plane status is ACTIVE:

gcloud container fleet mesh describe --project FLEET_PROJECT_ID

The output is similar to:

...
membershipSpecs:
  projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
    mesh:
      management: MANAGEMENT_AUTOMATIC
membershipStates:
  projects/746296320118/locations/us-central1/memberships/demo-cluster-1:
    servicemesh:
      controlPlaneManagement:
        details:
        - code: REVISION_READY
          details: 'Ready: asm-managed'
        state: ACTIVE
        implementation: ISTIOD | TRAFFIC_DIRECTOR
      dataPlaneManagement:
        details:
        - code: OK
          details: Service is running.
        state: ACTIVE
    state:
      code: OK
      description: 'Revision(s) ready for use: asm-managed.'
...

Take note of the control plane displayed in the implementation field, either ISTIOD or TRAFFIC_DIRECTOR. See Cloud Service Mesh supported features for control plane differences and supported configurations, and for how the control plane implementation is selected.

Configure kubectl to point to the cluster

The following sections involve running kubectl commands against each one of your clusters. Before proceeding through the following sections, run the following command for each of your clusters to configure kubectl to point to the cluster.

gcloud container clusters get-credentials CLUSTER_NAME \
      --location CLUSTER_LOCATION \
      --project CLUSTER_PROJECT_ID

Note that an ingress gateway isn't automatically deployed with the control plane. Decoupling the deployment of the ingress gateway and control plane lets you manage your gateways in a production environment. If you want to use an Istio ingress gateway or an egress gateway, see Deploy gateways. If you want to use the Kubernetes Gateway API, see Prepare Gateway for Mesh. To enable other optional features, see Enabling optional features on Cloud Service Mesh.

Managed data plane

If you use managed Cloud Service Mesh, Google fully manages upgrades of your proxies.

With the managed data plane feature enabled, the sidecar proxies and injected gateways are actively and automatically updated in conjunction with the managed control plane by restarting workloads to re-inject new versions of the proxy. This starts after the control plane has been upgraded and normally completes within 2 weeks after starting.

If disabled, proxy management is done passively - driven by the natural lifecycle of the pods in the cluster and must be manually triggered by the user to control the update rate.

The managed data plane upgrades proxies by evicting pods that are running earlier versions of the proxy. The evictions are done gradually, honoring the pod disruption budget and controlling the rate of change.

For more information, see Managed data plane.

Configure endpoint discovery (only for multi-cluster installations)

If your mesh has only one cluster, skip these multi-cluster steps and proceed to Deploy applications or Migrate applications.

Before you continue, ensure that Cloud Service Mesh is configured on each cluster.

Enabling Cloud Service Mesh with the fleet API will enable endpoint discovery for this cluster. However, you must open firewall ports. To disable endpoint discovery for one or more clusters, see the instructions to disable it in Endpoint discovery between clusters with declarative API.

For an example application with two clusters, see HelloWorld service example.

Network Endpoint Group support

Cloud Service Mesh with the TRAFFIC_DIRECTOR control plane implementation requires Network Endpoint Groups (NEGs) to represent service endpoints in GKE.

Automatic NEG creation

When you enable Cloud Service Mesh with the Traffic Director control plane, the mesh automatically annotates your Kubernetes services with the cloud.google.com/neg annotation. This allows the GKE NEG controller to create standalone NEGs for your service's endpoints.

These NEGs are essential for the control plane to discover your service endpoints and program your clients to discover them.

What to expect

• Automatic Annotations: All services (including ClusterIP, NodePort, and LoadBalancer types) in a mesh-enabled namespace are automatically annotated with the cloud.google.com/neg annotation.
• NEG Status: A cloud.google.com/neg-status annotation is also added to your services, providing information about the created NEGs.
• Required Behavior: This automation is essential for services to function within the modern control plane and cannot be disabled for mesh services.

Interaction with CI/CD tools

If you use CI/CD tools to manage your services, these tools might detect the automatic annotations as configuration drift and attempt to remove them.

To prevent issues, configure your CI/CD tools to ignore the cloud.google.com/neg and cloud.google.com/neg-status annotations. For more information and examples, see Troubleshooting NEGs with CI/CD Tools.

Deploy applications

Enable the namespace for injection. The steps depend on your control plane implementation.

kubectl label namespace NAMESPACE \
    istio.io/rev- istio-injection=enabled --overwrite

  kubectl label namespace NAMESPACE \
      istio.io/rev- istio-injection=enabled --overwrite

Validate that the namespace label is applied correctly using the following command.

  kubectl get namespace -L istio-injection

Example output:

  NAME                 STATUS   AGE     ISTIO-INJECTION
  default              Active   5m9s    enabled

At this point, you have successfully configured managed Cloud Service Mesh. If you have any existing workloads in labeled namespaces, then restart them so they get proxies injected.

If you deploy an application in a multi-cluster setup, replicate the Kubernetes and control plane configuration in all clusters, unless you plan to limit that particular config to a subset of clusters. The configuration applied to a particular cluster is the source of truth for that cluster.

Customize injection (optional)

You can override default values and customize injection settings but this can lead to unforeseen configuration errors and resulting issues with sidecar containers. Before you customize injection, read the information after the sample for notes on particular settings and recommendations.

Per-pod configuration is available to override these options on individual pods. This is done by adding an istio-proxy container to your pod. The sidecar injection will treat any configuration defined here as an override to the default injection template.

For example, the following configuration customizes a variety of settings, including lowering the CPU requests, adding a volume mount, and adding a preStop hook:

apiVersion: v1
kind: Pod
metadata:
  name: example
spec:
  containers:
  - name: hello
    image: alpine
  - name: istio-proxy
    image: auto
    resources:
      requests:
        cpu: "200m"
        memory: "256Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
    volumeMounts:
    - mountPath: /etc/certs
      name: certs
    lifecycle:
      preStop:
        exec:
          command: ["sleep", "10"]
  volumes:
  - name: certs
    secret:
      secretName: istio-certs

In general, any field in a pod can be set. However, care must be taken for certain fields:

• Kubernetes requires the image field to be set before the injection has run. While you can set a specific image to override the default one, we recommend that you set the image to auto, which will cause the sidecar injector to automatically select the image to use.
• Some fields in containers are dependent on related settings. For example, must be less than or equal to the CPU limit. If both fields are not properly configured, the pod may fail to start.
• Kubernetes lets you set both requests and limits for resources in your Pod spec. GKE Autopilot only considers requests. For more information, see Setting resource limits in Autopilot.

Additionally, certain fields are configurable by annotations on the Pod, although it is recommended to use the above approach to customizing settings. Take additional care for the following annotations:

• For GKE Standard, if sidecar.istio.io/proxyCPU is set, make sure to explicitly set sidecar.istio.io/proxyCPULimit. Otherwise the sidecar's CPU limit will be set as unlimited.
• For GKE Standard, if sidecar.istio.io/proxyMemory is set, make sure to explicitly set sidecar.istio.io/proxyMemoryLimit. Otherwise the sidecar's memory limit will be set as unlimited.
• For GKE Autopilot, configuring resource requests and limits using annotations might overprovision resources. Use the image template approach to avoid. See Resource modification examples in Autopilot.

For example, see the below resources annotation:

spec:
  template:
    metadata:
      annotations:
        sidecar.istio.io/proxyCPU: "200m"
        sidecar.istio.io/proxyCPULimit: "200m"
        sidecar.istio.io/proxyMemory: "256Mi"
        sidecar.istio.io/proxyMemoryLimit: "256Mi"

Migrate applications to managed Cloud Service Mesh

To migrate applications from in-cluster Cloud Service Mesh to managed Cloud Service Mesh, perform the following steps:

1. Replace the current namespace label. The steps depend on your control plane implementation.

kubectl label namespace NAMESPACE \
    istio.io/rev- istio-injection=enabled --overwrite

  kubectl label namespace NAMESPACE \
      istio.io/rev- istio-injection=enabled --overwrite

1. Perform a rolling upgrade of deployments in the namespace:

   kubectl rollout restart deployment -n NAMESPACE
   

2. Test your application to verify that the workloads function correctly.

3. If you have workloads in other namespaces, repeat the previous steps for each namespace.

4. If you deployed the application in a multi-cluster setup, replicate the Kubernetes and Istio configuration in all clusters, unless there is a desire to limit that configuration to a subset of clusters only. The configuration applied to a particular cluster is the source of truth for that cluster.

If you are satisfied that your application works as expected, you can remove the in-cluster istiod after you switch all namespaces to the managed control plane, or keep them as a backup - istiod will automatically scale down to use fewer resources. To remove, skip to Delete old control plane.

If you encounter problems, you can identify and resolve them by using the information in Resolving managed control plane issues and if necessary, roll back to the previous version.

kubectl delete Service,Deployment,HorizontalPodAutoscaler,PodDisruptionBudget istiod -n istio-system --ignore-not-found=true

kubectl -n istio-system rollout undo deploy istio-ingressgateway

deployment.apps/istio-ingressgateway rolled back

Uninstall Cloud Service Mesh

Managed control plane auto-scales to zero when no namespaces are using it. For detailed steps, see Uninstall Cloud Service Mesh.