Multi-region deployment

GKE

Configure the multi-region seed host

This section describes how to expand the existing Cassandra cluster to a new region. This setup allows the new region to bootstrap the cluster and join the existing data center. Without this configuration, the multi-region Kubernetes clusters would not know about each other.

Follow the procedures for the tool you are using to manage and install Apigee hybrid:

Helm charts

For the first region created, get the pods in your apigee namespace:
```
kubectl get pods -o wide -n apigee
```
Identify the multi-region seed host address for Cassandra in this region, for example 10.0.0.11.
Prepare the overrides.yaml file for the second region and add in the seed host IP address as follows:
```
cassandra:
  multiRegionSeedHost: "SEED_HOST_IP_ADDRESS"
  datacenter: "DATACENTER_NAME"
  rack: "RACK_NAME"
  hostNetwork: false
  clusterName: CLUSTER_NAME
```
Replace the following:
- SEED_HOST_IP_ADDRESS with the seed host IP address, for example 10.0.0.11.
- DATACENTER_NAME with the datacenter name, for example dc-2.
- RACK_NAME with the rack name, for example ra-1.
- CLUSTER_NAME with the name of your Cassandra cluster. By default the value is apigeecluster. If you use a different cluster name, you must specify a value for cassandra.clusterName. You are free to choose your own value but it must be the same in all regions.

Configure the second region

To set up the new region:

Install cert-manager in region 2:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context ORIGINAL_CLUSTER_NAME
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace apigee -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context NEW_CLUSTER_NAME
```
5. Import the namespace configuration to the new cluster. Be sure to update the namespace in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Follow the steps to Install the Apigee hybrid CRDs into the new region.

Now use Helm charts to install Apigee hybrid in the new region with the following Helm Chart commands (as done in region 1):

Note: You can use the same CRDs for consistency by using the same folder where you have the charts downloaded, you can also keep your overrides file unique between region 1 and 2 by naming your overrides something unique such as overrides-region_name.yaml, etc.

Important: Omit the ‑‑atomic flag for migrated clusters.

helm upgrade operator apigee-operator \
  --install \
  --create-namespace \
  --namespace apigee-system \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade datastore apigee-datastore \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade telemetry apigee-telemetry \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade redis apigee-redis \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ingress-manager apigee-ingress-manager \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ORG_NAME apigee-org \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env mentioned on the overrides
helm upgrade ENV_NAME apigee-env/ \
  --install \
  --namespace apigee \
  --atomic \
  --set env=ENV_NAME \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env group mentioned on the overrides
helm upgrade apigee-virtualhost-ENV_GROUP_NAME apigee-virtualhost/ \
  --install \
  --namespace apigee \
  --atomic \
  --set envgroup=ENV_GROUP_NAME \
  -f overrides-DATACENTER_NAME.yaml

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
-- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.
1. Get apigeeorg from the cluster with the following command:
```
kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
```
  For example:
```
Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"
```
2. Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
  The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
  Where:
  - REGION_EXPANSION is the name you are giving this metadata. You can use any name.
  - NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
  - APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
  - SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
  For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
3. Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"state": "complete",
"updated": 1623105760
},
"apigee-cassandra-default-1": {
"state": "complete",
"updated": 1623105765
},
"apigee-cassandra-default-2": {
"state": "complete",
"updated": 1623105770
}
},
"state": "complete",
"updated": 1623105770
}

Once the data replication is complete and verified, update the seed hosts:
1. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DATACENTER_NAME.yaml.
2. Reapply the change to update the apigee datastore CR:
  
  Important: Omit the ‑‑atomic flag for migrated clusters.
```
helm upgrade datastore apigee-datastore/ \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
```

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Tip: APIGEE_JMX_USER and APIGEE_JMX_PASSWORD are usually defined in the environment variables $APIGEE_JMX_USER and $APIGEE_JMX_PASSWORD.

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

`apigeectl`

Set the kubectl context to the original cluster before retrieving the seed name:
```
kubectl config use-context original-cluster-name
```

Run the following kubectl command to identify a seed host address for Cassandra in the current region.

A seed host address allows a new regional instance to find the original cluster on the very first startup to learn the topology of the cluster. The seed host address is designated as the contact point in the cluster.

kubectl get pods -o wide -n apigee -l app=apigee-cassandra --field-selector status.phase=Running

NAME                      READY   STATUS      RESTARTS   AGE   IP          NODE                                          NOMINATED NODE
apigee-cassandra-default-0        1/1     Running     0          5d    10.0.0.11   gke-k8s-dc-2-default-pool-a2206492-p55d
apigee-cassandra-default-1        1/1     Running     0          5d    10.0.2.4    gke-k8s-dc-2-default-pool-e9daaab3-tjmz
apigee-cassandra-default-2        1/1     Running     0          5d    10.0.3.5    gke-k8s-dc-2-default-pool-e589awq3-kjch

Decide which of the IPs returned from the previous command will be the multi-region seed host.

In data center 2, configure cassandra.multiRegionSeedHost and cassandra.datacenter in Manage runtime plane components, where multiRegionSeedHost is one of the IPs returned by the previous command:

cassandra:
  multiRegionSeedHost: seed_host_IP
  datacenter: data_center_name
  rack: rack_name
  hostNetwork: false
  clusterName: cluster_name # must be the same for all regions

For example:

cassandra:
  multiRegionSeedHost: 10.0.0.11
  datacenter: "dc-2"
  rack: "ra-1"
  hostNetwork: false
  clusterName: my-apigee-cluster

In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides.yaml as you set in the first region.
Note: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in the overrides file in the second data center. For details see Configuring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context original-cluster-name
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace namespace -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context new-cluster-name
```
5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Install hybrid in the new region. Be sure that the overrides-DC_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.
Execute the following two commands to install hybrid in the new region:

Important: When expanding Apigee hybrid to a new regional datacenter, be sure to install the same version of hybrid as you have in your existing datacenter. Avoid running different versions of hybrid in multiple datacenters.
```
apigeectl init -f overrides/overrides-DC_name.yaml
```
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```
Verify the hybrid installation is successful by running the following command:
```
apigeectl check-ready -f overrides_DC_name.yaml
```

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.

Get apigeeorg from the cluster with the following command:

kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"

For example:

Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"

Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
Where:
- REGION_EXPANSION is the name you are giving this metadata. You can use any name.
- NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
- APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
- SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
  "rebuildDetails": {
    "apigee-cassandra-default-0": {
      "state": "complete",
      "updated": 1623105760
    },
    "apigee-cassandra-default-1": {
      "state": "complete",
      "updated": 1623105765
    },
    "apigee-cassandra-default-2": {
      "state": "complete",
      "updated": 1623105770
    }
  },
  "state": "complete",
  "updated": 1623105770
}

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
Note: Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in overrides.yaml and then reapply the configuration.
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```

Check the Cassandra cluster status

The following command is useful to see if the cluster setup is successful in two data centers. The command checks the nodetool status for the two regions.

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status


Datacenter: dc-1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          100.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          100.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          100.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1

GKE on-prem

Configure the multi-region seed host

Follow the procedures for the tool you are using to manage and install Apigee hybrid:

Helm charts

For the first region created, get the pods in your apigee namespace:
```
kubectl get pods -o wide -n apigee
```
Identify the multi-region seed host address for Cassandra in this region, for example 10.0.0.11.
Prepare the overrides.yaml file for the second region and add in the seed host IP address as follows:
```
cassandra:
  multiRegionSeedHost: "SEED_HOST_IP_ADDRESS"
  datacenter: "DATACENTER_NAME"
  rack: "RACK_NAME"
  hostNetwork: false
  clusterName: CLUSTER_NAME
```
Replace the following:
- SEED_HOST_IP_ADDRESS with the seed host IP address, for example 10.0.0.11.
- DATACENTER_NAME with the datacenter name, for example dc-2.
- RACK_NAME with the rack name, for example ra-1.
- CLUSTER_NAME with the name of your Cassandra cluster. By default the value is apigeecluster. If you use a different cluster name, you must specify a value for cassandra.clusterName. You are free to choose your own value but it must be the same in all regions.

Configure the second region

To set up the new region:

Install cert-manager in region 2:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context ORIGINAL_CLUSTER_NAME
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace apigee -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context NEW_CLUSTER_NAME
```
5. Import the namespace configuration to the new cluster. Be sure to update the namespace in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Follow the steps to Install the Apigee hybrid CRDs into the new region.

Now use Helm charts to install Apigee hybrid in the new region with the following Helm Chart commands (as done in region 1):

Important: Omit the ‑‑atomic flag for migrated clusters.

helm upgrade operator apigee-operator \
  --install \
  --create-namespace \
  --namespace apigee-system \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade datastore apigee-datastore \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade telemetry apigee-telemetry \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade redis apigee-redis \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ingress-manager apigee-ingress-manager \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ORG_NAME apigee-org \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env mentioned on the overrides
helm upgrade ENV_NAME apigee-env/ \
  --install \
  --namespace apigee \
  --atomic \
  --set env=ENV_NAME \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env group mentioned on the overrides
helm upgrade apigee-virtualhost-ENV_GROUP_NAME apigee-virtualhost/ \
  --install \
  --namespace apigee \
  --atomic \
  --set envgroup=ENV_GROUP_NAME \
  -f overrides-DATACENTER_NAME.yaml

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
-- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.
1. Get apigeeorg from the cluster with the following command:
```
kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
```
  For example:
```
Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"
```
2. Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
  The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
  Where:
  - REGION_EXPANSION is the name you are giving this metadata. You can use any name.
  - NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
  - APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
  - SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
  For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
3. Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"state": "complete",
"updated": 1623105760
},
"apigee-cassandra-default-1": {
"state": "complete",
"updated": 1623105765
},
"apigee-cassandra-default-2": {
"state": "complete",
"updated": 1623105770
}
},
"state": "complete",
"updated": 1623105770
}

Once the data replication is complete and verified, update the seed hosts:
1. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DATACENTER_NAME.yaml.
2. Reapply the change to update the apigee datastore CR:
  
  Important: Omit the ‑‑atomic flag for migrated clusters.
```
helm upgrade datastore apigee-datastore/ \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
```

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Tip: APIGEE_JMX_USER and APIGEE_JMX_PASSWORD are usually defined in the environment variables $APIGEE_JMX_USER and $APIGEE_JMX_PASSWORD.

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

`apigeectl`

In the overrides.yaml file for your original cluster, make sure cassandra:hostNetwork is set to true. For example:
```
cassandra:
  hostNetwork: true
```
See Prerequisites for more information about when to set hostNetwork: true.
If the cassandra:hostNetwork is not set to true, do the following:
1. Change cassandra.hostNetwork to true.
2. Apply the overrides.yaml config file with the command:
```
  apigeectl apply -f overrides.yaml --datastore
  
```
3. Wait for the Cassandra pods to complete a rolling restart.
4. Verify the Cassandra cluster is healthy with the following commands:
```
  kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
  
```
```
  nodetool -u  -pw APIGEE_JMX_PASSWORD status
  
```
  Make sure all Cassandra nodes in the output are in UN (Up/Normal) status:
```
  nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD describecluster
  
```
  Make sure there are no unreachable nodes listed in the output.
Set the kubectl context to the original cluster before retrieving the seed name:
```
kubectl config use-context original-cluster-name
```

Run the following kubectl command to identify a seed host address for Cassandra in the current region.

kubectl get pods -o wide -n apigee -l app=apigee-cassandra --field-selector status.phase=Running

NAME                      READY   STATUS      RESTARTS   AGE   IP          NODE                                          NOMINATED NODE
apigee-cassandra-default-0        1/1     Running     0          5d    10.0.0.11   gke-k8s-dc-2-default-pool-a2206492-p55d
apigee-cassandra-default-1        1/1     Running     0          5d    10.0.2.4    gke-k8s-dc-2-default-pool-e9daaab3-tjmz
apigee-cassandra-default-2        1/1     Running     0          5d    10.0.3.5    gke-k8s-dc-2-default-pool-e589awq3-kjch

Decide which of the IPs returned from the previous command will be the multi-region seed host.

In data center 2, configure cassandra.multiRegionSeedHost in your overrides file, where multiRegionSeedHost is one of the IPs returned by the previous command:

cassandra:
  hostNetwork: true
  multiRegionSeedHost: seed_host_IP
  datacenter: data_center_name
  rack: rack_name
  clusterName: cluster_name # must be the same for all regions

For example:

cassandra:
  hostNetwork: true
  multiRegionSeedHost: 10.0.0.11
  datacenter: "dc-2"
  rack: "ra-1"
  clusterName: my-apigee-cluster

In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides.yaml as you set in the first region.
Note: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in the overrides file in the second data center. For details see Configuring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context original-cluster-name
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace namespace -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context new-cluster-name
```
5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Install hybrid in the new region. Be sure that the overrides-DC_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.
Execute the following two commands to install hybrid in the new region:

Important: When expanding Apigee hybrid to a new regional datacenter, be sure to install the same version of hybrid as you have in your existing datacenter. Avoid running different versions of hybrid in multiple datacenters.
```
apigeectl init -f overrides/overrides-DC_name.yaml
```
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```
Verify the hybrid installation is successful by running the following command:
```
apigeectl check-ready -f overrides_DC_name.yaml
```

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.

Get apigeeorg from the cluster with the following command:

kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"

For example:

Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"

Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
Where:
- REGION_EXPANSION is the name you are giving this metadata. You can use any name.
- NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
- APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
- SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
  "rebuildDetails": {
    "apigee-cassandra-default-0": {
      "state": "complete",
      "updated": 1623105760
    },
    "apigee-cassandra-default-1": {
      "state": "complete",
      "updated": 1623105765
    },
    "apigee-cassandra-default-2": {
      "state": "complete",
      "updated": 1623105770
    }
  },
  "state": "complete",
  "updated": 1623105770
}

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
Note: Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in overrides.yaml and then reapply the configuration.
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```

Check the Cassandra cluster status

The following command is useful to see if the cluster setup is successful in two data centers. The command checks the nodetool status for the two regions.

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status


Datacenter: dc-1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          100.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          100.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          100.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1

AKS

Create a virtual network in each region

Follow the Azure recommendations for establishing cross-region communication here: VNet-to-VNet: Connecting Virtual Networks in Azure across Different Regions.

Create multi-regional clusters

Set up Kubernetes clusters in multiple regions with different CIDR blocks. See also Step 1: Create a cluster. Use the locations and virtual network names you created previously.

Open Cassandra ports between Kubernetes clusters across all regions to enable worker nodes across regions and datacenters to communicate. See Configure ports for the Cassandra port numbers.

Configure the multi-region seed host

Follow the procedures for the tool you are using to manage and install Apigee hybrid:

Helm charts

For the first region created, get the pods in your apigee namespace:
```
kubectl get pods -o wide -n apigee
```
Identify the multi-region seed host address for Cassandra in this region, for example 10.0.0.11.
Prepare the overrides.yaml file for the second region and add in the seed host IP address as follows:
```
cassandra:
  multiRegionSeedHost: "SEED_HOST_IP_ADDRESS"
  datacenter: "DATACENTER_NAME"
  rack: "RACK_NAME"
  hostNetwork: false
  clusterName: CLUSTER_NAME
```
Replace the following:
- SEED_HOST_IP_ADDRESS with the seed host IP address, for example 10.0.0.11.
- DATACENTER_NAME with the datacenter name, for example dc-2.
- RACK_NAME with the rack name, for example ra-1.
- CLUSTER_NAME with the name of your Cassandra cluster. By default the value is apigeecluster. If you use a different cluster name, you must specify a value for cassandra.clusterName. You are free to choose your own value but it must be the same in all regions.

Configure the second region

To set up the new region:

Install cert-manager in region 2:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context ORIGINAL_CLUSTER_NAME
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace apigee -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context NEW_CLUSTER_NAME
```
5. Import the namespace configuration to the new cluster. Be sure to update the namespace in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Follow the steps to Install the Apigee hybrid CRDs into the new region.

Now use Helm charts to install Apigee hybrid in the new region with the following Helm Chart commands (as done in region 1):

Important: Omit the ‑‑atomic flag for migrated clusters.

helm upgrade operator apigee-operator \
  --install \
  --create-namespace \
  --namespace apigee-system \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade datastore apigee-datastore \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade telemetry apigee-telemetry \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade redis apigee-redis \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ingress-manager apigee-ingress-manager \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ORG_NAME apigee-org \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env mentioned on the overrides
helm upgrade ENV_NAME apigee-env/ \
  --install \
  --namespace apigee \
  --atomic \
  --set env=ENV_NAME \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env group mentioned on the overrides
helm upgrade apigee-virtualhost-ENV_GROUP_NAME apigee-virtualhost/ \
  --install \
  --namespace apigee \
  --atomic \
  --set envgroup=ENV_GROUP_NAME \
  -f overrides-DATACENTER_NAME.yaml

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
-- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.
1. Get apigeeorg from the cluster with the following command:
```
kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
```
  For example:
```
Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"
```
2. Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
  The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
  Where:
  - REGION_EXPANSION is the name you are giving this metadata. You can use any name.
  - NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
  - APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
  - SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
  For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
3. Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"state": "complete",
"updated": 1623105760
},
"apigee-cassandra-default-1": {
"state": "complete",
"updated": 1623105765
},
"apigee-cassandra-default-2": {
"state": "complete",
"updated": 1623105770
}
},
"state": "complete",
"updated": 1623105770
}

Once the data replication is complete and verified, update the seed hosts:
1. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DATACENTER_NAME.yaml.
2. Reapply the change to update the apigee datastore CR:
  
  Important: Omit the ‑‑atomic flag for migrated clusters.
```
helm upgrade datastore apigee-datastore/ \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
```

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Tip: APIGEE_JMX_USER and APIGEE_JMX_PASSWORD are usually defined in the environment variables $APIGEE_JMX_USER and $APIGEE_JMX_PASSWORD.

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

`apigeectl`

In the overrides.yaml file for your original cluster, make sure cassandra:hostNetwork is set to true. For example:
```
cassandra:
  hostNetwork: true
```
See Prerequisites for more information about when to set hostNetwork: true.
If the cassandra:hostNetwork is not set to true, do the following:
1. Change cassandra.hostNetwork to true.
2. Apply the overrides.yaml config file with the command:
```
  apigeectl apply -f overrides.yaml --datastore
  
```
3. Wait for the Cassandra pods to complete a rolling restart.
4. Verify the Cassandra cluster is healthy with the following commands:
```
  kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
  
```
```
  nodetool -u  -pw APIGEE_JMX_PASSWORD status
  
```
  Make sure all Cassandra nodes in the output are in UN (Up/Normal) status:
```
  nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD describecluster
  
```
  Make sure there are no unreachable nodes listed in the output.
Set the kubectl context to the original cluster before retrieving the seed name:
```
kubectl config use-context original-cluster-name
```
Run the following kubectl command to identify a seed host address for Cassandra in the current region.

A seed host address allows a new regional instance to find the original cluster on the very first startup to learn the topology of the cluster. The seed host address is designated as the contact point in the cluster.
```
kubectl get pods -o wide -n apigee -l app=apigee-cassandra --field-selector status.phase=Running
```
```
apigee-cassandra-default-0  1/1   Running   0   4d17h   120.38.1.9  aks-agentpool-21207753-vmss000000
```
Decide which of the IPs returned from the previous command will be the multi-region seed host. In this example, where only a single node cassandra cluster is running, the seed host is 120.38.1.9.
In data center 2, copy your overrides file to a new file whose name includes the cluster name. For example, overrides_your_cluster_name.yaml.

In data center 2, configure cassandra.multiRegionSeedHost and cassandra.datacenter in overrides_your_cluster_name.yaml, where multiRegionSeedHost is one of the IPs returned by the previous command:

cassandra:
     multiRegionSeedHost: seed_host_IP
     datacenter: data_center_name
     rack: rack_name
     hostNetwork: true

clusterName: cluster_name # must be the same for all regions

For example:

cassandra:
  multiRegionSeedHost: 120.38.1.9
  datacenter: "centralus"
  rack: "ra-1"
  hostNetwork: true
  clusterName: my-apigee-cluster

In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides_your_cluster_name.yaml as you set in the first region.
Note: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in the overrides file in the second data center. For details see Configuring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context original-cluster-name
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace namespace -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context new-cluster-name
```
5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Install hybrid in the new region. Be sure that the overrides_your_cluster_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.
Execute the following two commands to install hybrid in the new region:

Important: When expanding Apigee hybrid to a new regional datacenter, be sure to install the same version of hybrid as you have in your existing datacenter. Avoid running different versions of hybrid in multiple datacenters.
```
apigeectl init -f overrides_your_cluster_name.yaml
```
```
apigeectl apply -f overrides_your_cluster_name.yaml
```
Verify the hybrid installation is successful by running the following command:
```
apigeectl check-ready -f overrides_your_cluster_name.yaml
```

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.

Get apigeeorg from the cluster with the following command:

kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"

For example:

Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"

Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
Where:
- REGION_EXPANSION is the name you are giving this metadata. You can use any name.
- NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
- APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
- SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
  "rebuildDetails": {
    "apigee-cassandra-default-0": {
      "state": "complete",
      "updated": 1623105760
    },
    "apigee-cassandra-default-1": {
      "state": "complete",
      "updated": 1623105765
    },
    "apigee-cassandra-default-2": {
      "state": "complete",
      "updated": 1623105770
    }
  },
  "state": "complete",
  "updated": 1623105770
}

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
Note: Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in overrides.yaml and then reapply the configuration.
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```

Check the Cassandra cluster status

The following command is useful to see if the cluster setup is successful in two data centers. The command checks the nodetool status for the two regions.

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status


Datacenter: dc-1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          100.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          100.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          100.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1

EKS

Create a virtual network in each region

Follow the AWS recommendations for establishing cross-region communication as described in: What is VPC peering?. The AWS term for using different regions is inter-Region VPC peering.

Create multi-regional clusters

Set up Kubernetes clusters in multiple regions with different CIDR blocks. See also Step 1: Create a cluster. Use the locations and virtual network names you created previously.

Open Cassandra ports between Kubernetes clusters across all regions to enable worker nodes across regions and datacenters to communicate. See Configure ports for the Cassandra port numbers.

Configure the multi-region seed host

Follow the procedures for the tool you are using to manage and install Apigee hybrid:

Helm charts

For the first region created, get the pods in your apigee namespace:
```
kubectl get pods -o wide -n apigee
```
Identify the multi-region seed host address for Cassandra in this region, for example 10.0.0.11.
Prepare the overrides.yaml file for the second region and add in the seed host IP address as follows:
```
cassandra:
  multiRegionSeedHost: "SEED_HOST_IP_ADDRESS"
  datacenter: "DATACENTER_NAME"
  rack: "RACK_NAME"
  hostNetwork: false
  clusterName: CLUSTER_NAME
```
Replace the following:
- SEED_HOST_IP_ADDRESS with the seed host IP address, for example 10.0.0.11.
- DATACENTER_NAME with the datacenter name, for example dc-2.
- RACK_NAME with the rack name, for example ra-1.
- CLUSTER_NAME with the name of your Cassandra cluster. By default the value is apigeecluster. If you use a different cluster name, you must specify a value for cassandra.clusterName. You are free to choose your own value but it must be the same in all regions.

Configure the second region

To set up the new region:

Install cert-manager in region 2:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context ORIGINAL_CLUSTER_NAME
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace apigee -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context NEW_CLUSTER_NAME
```
5. Import the namespace configuration to the new cluster. Be sure to update the namespace in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Follow the steps to Install the Apigee hybrid CRDs into the new region.

Now use Helm charts to install Apigee hybrid in the new region with the following Helm Chart commands (as done in region 1):

Important: Omit the ‑‑atomic flag for migrated clusters.

helm upgrade operator apigee-operator \
  --install \
  --create-namespace \
  --namespace apigee-system \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade datastore apigee-datastore \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade telemetry apigee-telemetry \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade redis apigee-redis \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ingress-manager apigee-ingress-manager \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ORG_NAME apigee-org \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env mentioned on the overrides
helm upgrade ENV_NAME apigee-env/ \
  --install \
  --namespace apigee \
  --atomic \
  --set env=ENV_NAME \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env group mentioned on the overrides
helm upgrade apigee-virtualhost-ENV_GROUP_NAME apigee-virtualhost/ \
  --install \
  --namespace apigee \
  --atomic \
  --set envgroup=ENV_GROUP_NAME \
  -f overrides-DATACENTER_NAME.yaml

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
-- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.
1. Get apigeeorg from the cluster with the following command:
```
kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
```
  For example:
```
Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"
```
2. Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
  The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
  Where:
  - REGION_EXPANSION is the name you are giving this metadata. You can use any name.
  - NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
  - APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
  - SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
  For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
3. Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"state": "complete",
"updated": 1623105760
},
"apigee-cassandra-default-1": {
"state": "complete",
"updated": 1623105765
},
"apigee-cassandra-default-2": {
"state": "complete",
"updated": 1623105770
}
},
"state": "complete",
"updated": 1623105770
}

Once the data replication is complete and verified, update the seed hosts:
1. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DATACENTER_NAME.yaml.
2. Reapply the change to update the apigee datastore CR:
  
  Important: Omit the ‑‑atomic flag for migrated clusters.
```
helm upgrade datastore apigee-datastore/ \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
```

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Tip: APIGEE_JMX_USER and APIGEE_JMX_PASSWORD are usually defined in the environment variables $APIGEE_JMX_USER and $APIGEE_JMX_PASSWORD.

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

`apigeectl`

In the overrides.yaml file for your original cluster, make sure cassandra:hostNetwork is set to true. For example:
```
cassandra:
  hostNetwork: true
```
See Prerequisites for more information about when to set hostNetwork: true.
If the cassandra:hostNetwork is not set to true, do the following:
1. Change cassandra.hostNetwork to true.
2. Apply the overrides.yaml config file with the command:
```
  apigeectl apply -f overrides.yaml --datastore
  
```
3. Wait for the Cassandra pods to complete a rolling restart.
4. Verify the Cassandra cluster is healthy with the following commands:
```
  kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
  
```
```
  nodetool -u  -pw APIGEE_JMX_PASSWORD status
  
```
  Make sure all Cassandra nodes in the output are in UN (Up/Normal) status:
```
  nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD describecluster
  
```
  Make sure there are no unreachable nodes listed in the output.
Set the kubectl context to the original cluster before retrieving the seed name:
```
kubectl config use-context original-cluster-name
```
Run the following kubectl command to identify a seed host address for Cassandra in the current region.

A seed host address allows a new regional instance to find the original cluster on the very first startup to learn the topology of the cluster. The seed host address is designated as the contact point in the cluster.
```
kubectl get pods -o wide -n apigee -l app=apigee-cassandra --field-selector status.phase=Running
```
```
apigee-cassandra-default-0  1/1   Running   0   4d17h   120.38.1.9  aks-agentpool-21207753-vmss000000
```
Decide which of the IPs returned from the previous command will be the multi-region seed host. In this example, where only a single node cassandra cluster is running, the seed host is 120.38.1.9.
In data center 2, copy your overrides file to a new file whose name includes the cluster name. For example, overrides_your_cluster_name.yaml.

cassandra:
     multiRegionSeedHost: seed_host_IP
     datacenter: data_center_name
     rack: rack_name
     hostNetwork: true

clusterName: cluster_name # must be the same for all regions

For example:

cassandra:
  multiRegionSeedHost: 120.38.1.9
  datacenter: "centralus"
  rack: "ra-1"
  hostNetwork: true
  clusterName: my-apigee-cluster

In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides_your_cluster_name.yaml as you set in the first region.
Note: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in the overrides file in the second data center. For details see Configuring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context original-cluster-name
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace namespace -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context new-cluster-name
```
5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Install hybrid in the new region. Be sure that the overrides_your_cluster_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.
Execute the following two commands to install hybrid in the new region:

Important: When expanding Apigee hybrid to a new regional datacenter, be sure to install the same version of hybrid as you have in your existing datacenter. Avoid running different versions of hybrid in multiple datacenters.
```
apigeectl init -f overrides_your_cluster_name.yaml
```
```
apigeectl apply -f overrides_your_cluster_name.yaml
```
Verify the hybrid installation is successful by running the following command:
```
apigeectl check-ready -f overrides_your_cluster_name.yaml
```

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.

Get apigeeorg from the cluster with the following command:

kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"

For example:

Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"

Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
Where:
- REGION_EXPANSION is the name you are giving this metadata. You can use any name.
- NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
- APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
- SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
  "rebuildDetails": {
    "apigee-cassandra-default-0": {
      "state": "complete",
      "updated": 1623105760
    },
    "apigee-cassandra-default-1": {
      "state": "complete",
      "updated": 1623105765
    },
    "apigee-cassandra-default-2": {
      "state": "complete",
      "updated": 1623105770
    }
  },
  "state": "complete",
  "updated": 1623105770
}

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
Note: Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in overrides.yaml and then reapply the configuration.
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```

Check the Cassandra cluster status

The following command is useful to see if the cluster setup is successful in two data centers. The command checks the nodetool status for the two regions.

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status


Datacenter: dc-1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          100.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          100.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          100.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1

OpenShift

Configure the multi-region seed host

Follow the procedures for the tool you are using to manage and install Apigee hybrid:

Helm charts

For the first region created, get the pods in your apigee namespace:
```
kubectl get pods -o wide -n apigee
```
Identify the multi-region seed host address for Cassandra in this region, for example 10.0.0.11.
Prepare the overrides.yaml file for the second region and add in the seed host IP address as follows:
```
cassandra:
  multiRegionSeedHost: "SEED_HOST_IP_ADDRESS"
  datacenter: "DATACENTER_NAME"
  rack: "RACK_NAME"
  hostNetwork: false
  clusterName: CLUSTER_NAME
```
Replace the following:
- SEED_HOST_IP_ADDRESS with the seed host IP address, for example 10.0.0.11.
- DATACENTER_NAME with the datacenter name, for example dc-2.
- RACK_NAME with the rack name, for example ra-1.
- CLUSTER_NAME with the name of your Cassandra cluster. By default the value is apigeecluster. If you use a different cluster name, you must specify a value for cassandra.clusterName. You are free to choose your own value but it must be the same in all regions.

Configure the second region

To set up the new region:

Install cert-manager in region 2:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.1/cert-manager.yaml

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context ORIGINAL_CLUSTER_NAME
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace apigee -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context NEW_CLUSTER_NAME
```
5. Import the namespace configuration to the new cluster. Be sure to update the namespace in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Follow the steps to Install the Apigee hybrid CRDs into the new region.

Now use Helm charts to install Apigee hybrid in the new region with the following Helm Chart commands (as done in region 1):

Important: Omit the ‑‑atomic flag for migrated clusters.

helm upgrade operator apigee-operator \
  --install \
  --create-namespace \
  --namespace apigee-system \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade datastore apigee-datastore \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade telemetry apigee-telemetry \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade redis apigee-redis \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ingress-manager apigee-ingress-manager \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
helm upgrade ORG_NAME apigee-org \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env mentioned on the overrides
helm upgrade ENV_NAME apigee-env/ \
  --install \
  --namespace apigee \
  --atomic \
  --set env=ENV_NAME \
  -f overrides-DATACENTER_NAME.yaml
# repeat the below command for each env group mentioned on the overrides
helm upgrade apigee-virtualhost-ENV_GROUP_NAME apigee-virtualhost/ \
  --install \
  --namespace apigee \
  --atomic \
  --set envgroup=ENV_GROUP_NAME \
  -f overrides-DATACENTER_NAME.yaml

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
-- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.
1. Get apigeeorg from the cluster with the following command:
```
kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
```
  For example:
```
Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"
```
2. Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
  The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
  Where:
  - REGION_EXPANSION is the name you are giving this metadata. You can use any name.
  - NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
  - APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
  - SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
  For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
3. Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
"rebuildDetails": {
"apigee-cassandra-default-0": {
"state": "complete",
"updated": 1623105760
},
"apigee-cassandra-default-1": {
"state": "complete",
"updated": 1623105765
},
"apigee-cassandra-default-2": {
"state": "complete",
"updated": 1623105770
}
},
"state": "complete",
"updated": 1623105770
}

Once the data replication is complete and verified, update the seed hosts:
1. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DATACENTER_NAME.yaml.
2. Reapply the change to update the apigee datastore CR:
  
  Important: Omit the ‑‑atomic flag for migrated clusters.
```
helm upgrade datastore apigee-datastore/ \
  --install \
  --namespace apigee \
  --atomic \
  -f overrides-DATACENTER_NAME.yaml
```

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Tip: APIGEE_JMX_USER and APIGEE_JMX_PASSWORD are usually defined in the environment variables $APIGEE_JMX_USER and $APIGEE_JMX_PASSWORD.

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

`apigeectl`

In the overrides.yaml file for your original cluster, make sure cassandra:hostNetwork is set to true. For example:
```
cassandra:
  hostNetwork: true
```
See Prerequisites for more information about when to set hostNetwork: true.
If the cassandra:hostNetwork is not set to true, do the following:
1. Change cassandra.hostNetwork to true.
2. Apply the overrides.yaml config file with the command:
```
  apigeectl apply -f overrides.yaml --datastore
  
```
3. Wait for the Cassandra pods to complete a rolling restart.
4. Verify the Cassandra cluster is healthy with the following commands:
```
  kubectl exec -it -n apigee apigee-cassandra-default-0 -- /bin/bash
  
```
```
  nodetool -u  -pw APIGEE_JMX_PASSWORD status
  
```
  Make sure all Cassandra nodes in the output are in UN (Up/Normal) status:
```
  nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD describecluster
  
```
  Make sure there are no unreachable nodes listed in the output.
Set the kubectl context to the original cluster before retrieving the seed name:
```
kubectl config use-context original-cluster-name
```

Run the following kubectl command to identify a seed host address for Cassandra in the current region.

kubectl get pods -o wide -n apigee -l app=apigee-cassandra --field-selector status.phase=Running

NAME                      READY   STATUS      RESTARTS   AGE   IP          NODE                                          NOMINATED NODE
apigee-cassandra-default-0        1/1     Running     0          5d    10.0.0.11   gke-k8s-dc-2-default-pool-a2206492-p55d
apigee-cassandra-default-1        1/1     Running     0          5d    10.0.2.4    gke-k8s-dc-2-default-pool-e9daaab3-tjmz
apigee-cassandra-default-2        1/1     Running     0          5d    10.0.3.5    gke-k8s-dc-2-default-pool-e589awq3-kjch

Select the IP address of your source Cassandra host to use as the multi-region seed host. In this example, this is the apigee-cassandra-default-0cluster is running, the seed host is 10.0.0.11.
In data center 2, copy your overrides file to a new file whose name includes the cluster name. For example, overrides_your_cluster_name.yaml.

cassandra:
     hostNetwork: true
     multiRegionSeedHost: seed_host_IP # Cassandra pod IP address from the source region.
     datacenter: data_center_name
     rack: rack_name
     clusterName: cluster_name # must be the same for all regions

For example:

cassandra:
  hostNetwork: true
  multiRegionSeedHost: 10.0.0.11
  datacenter: "dc-2"
  rack: "ra-1"
  clusterName: my-apigee-cluster

In the new data center/region, before you install hybrid, set the same TLS certificates and credentials in overrides_your_cluster_name.yaml as you set in the first region.
Note: Be sure to use the same Cassandra TLS certificates and credentials in the second data center as you provided in the original data center. The credentials you set in the overrides file in the first data center must match the ones you specify in the overrides file in the second data center. For details see Configuring TLS for Cassandra.

Set up the new region

After you configure the seed host, you can set up the new region.

To set up the new region:

Copy your certificate from the existing cluster to the new cluster. The new CA root is used by Cassandra and other hybrid components for mTLS. Therefore, it is essential to have consistent certificates across the cluster.
1. Set the context to the original namespace:
```
kubectl config use-context original-cluster-name
```
2. Export the current namespace configuration to a file:
```
kubectl get namespace namespace -o yaml > apigee-namespace.yaml
```
3. Export the apigee-ca secret to a file:
```
kubectl -n cert-manager get secret apigee-ca -o yaml > apigee-ca.yaml
```
4. Set the context to the new region's cluster name:
```
kubectl config use-context new-cluster-name
```
5. Import the namespace configuration to the new cluster. Be sure to update the "namespace" in the file if you're using a different namespace in the new region:
```
kubectl apply -f apigee-namespace.yaml
```
6. Import the secret to the new cluster:
```
kubectl -n cert-manager apply -f apigee-ca.yaml
```
Install hybrid in the new region. Be sure that the overrides_your_cluster_name.yaml file includes the same TLS certificates that are configured in the first region, as explained in the previous section.
Execute the following two commands to install hybrid in the new region:

Important: When expanding Apigee hybrid to a new regional datacenter, be sure to install the same version of hybrid as you have in your existing datacenter. Avoid running different versions of hybrid in multiple datacenters.
```
apigeectl init -f overrides_your_cluster_name.yaml
```
```
apigeectl apply -f overrides_your_cluster_name.yaml
```
Verify the hybrid installation is successful by running the following command:
```
apigeectl check-ready -f overrides_your_cluster_name.yaml
```

Verify the Cassandra cluster setup by running the following command. The output should show both the existing and new data centers.

kubectl exec apigee-cassandra-default-0 -n apigee  \
  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

Example showing a successful setup:

Datacenter: dc-1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.87.93   68.07 GiB  256     ?     fb51465c-167a-42f7-98c9-b6eba1de34de  c
UN  10.132.84.94   69.9 GiB   256     ?     f621a5ac-e7ee-48a9-9a14-73d69477c642  b
UN  10.132.84.105  76.95 GiB  256     ?     0561086f-e95b-4232-ba6c-ad519ff30336  d

Datacenter: dc-2
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns  Host ID                               Rack
UN  10.132.0.8     71.61 GiB  256     ?     8894a98b-8406-45de-99e2-f404ab10b5d6  c
UN  10.132.9.204   75.1 GiB   256     ?     afa0ffa3-630b-4f1e-b46f-fc3df988092e  a
UN  10.132.3.133   68.08 GiB  256     ?     25ae39ab-b39e-4d4f-9cb7-de095ab873db  b

Set up Cassandra on all the pods in the new data centers.

Get apigeeorg from the cluster with the following command:

kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"

For example:

Ex: kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name"
"rg-hybrid-b7d3b9c"

Create a cassandra data replication custom resource (YAML) file. The file can have any name. In the following examples the file will have the name datareplication.yaml.
The file must contain the following:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: REGION_EXPANSION
  namespace: NAMESPACE
spec:
  organizationRef: APIGEEORG_VALUE
  force: false
  source:
    region: SOURCE_REGION
```
Where:
- REGION_EXPANSION is the name you are giving this metadata. You can use any name.
- NAMESPACE is the same namespace that is provided in overrides.yaml. This is usually "apigee".
- APIGEEORG_VALUE is the value output from the kubectl get apigeeorg -n apigee -o json | jq ".items[].metadata.name" command in the previous step. For example, rg-hybrid-b7d3b9c
- SOURCE_REGION is the source region, datacenter value under cassandra section from source region overrides.yaml
For example:
```
apiVersion: apigee.cloud.google.com/v1alpha1
kind: CassandraDataReplication
metadata:
  name: region-expansion
  namespace: apigee
spec:
  organizationRef: rg-hybrid-b7d3b9c
  force: false
  source:
    region: "dc-1"
```
Apply the CassandraDataReplication with the following command:
```
kubectl apply -f datareplication.yaml
```

Verify the rebuild status using the following command.

kubectl -n apigee get apigeeds -o json | jq ".items[].status.cassandraDataReplication"

The results should look something like:

{
  "rebuildDetails": {
    "apigee-cassandra-default-0": {
      "state": "complete",
      "updated": 1623105760
    },
    "apigee-cassandra-default-1": {
      "state": "complete",
      "updated": 1623105765
    },
    "apigee-cassandra-default-2": {
      "state": "complete",
      "updated": 1623105770
    }
  },
  "state": "complete",
  "updated": 1623105770
}

Verify the rebuild processes from the logs. Also, verify the data size using the nodetool status command.

kubectl logs apigee-cassandra-default-0 -f -n apigee

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status

The following example shows example log entries:

INFO  01:42:24 rebuild from dc: dc-1, (All keyspaces), (All tokens)
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Executing streaming plan for Rebuild
INFO  01:42:24 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.1.45
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.432KiB), sending 0 files(0.000KiB)
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.1.45 is complete
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.4.36
INFO  01:42:25 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Starting streaming to /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 1 files(0.693KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.4.36 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889, ID#0] Beginning stream session with /10.12.5.22
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889 ID#0] Prepare completed. Receiving 3 files(0.720KiB), sending 0 files(0.000KiB)
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] Session with /10.12.5.22 is complete
INFO  01:42:26 [Stream #3a04e810-580d-11e9-a5aa-67071bf82889] All sessions completed

Update the seed hosts. Remove multiRegionSeedHost: 10.0.0.11 from overrides-DC_name.yaml and reapply.
Note: Seed hosts are local cluster members. To boot up a new region an external seed host is required. Once a region boots up you need to change the seed hosts back to their local clusters in overrides.yaml and then reapply the configuration.
```
apigeectl apply -f overrides/overrides-DC_name.yaml
```

Check the Cassandra cluster status

The following command is useful to see if the cluster setup is successful in two data centers. The command checks the nodetool status for the two regions.

kubectl exec apigee-cassandra-default-0 -n apigee  -- nodetool -u APIGEE_JMX_USER -pw APIGEE_JMX_PASSWORD status


Datacenter: dc-1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.12.1.45  112.09 KiB  256          100.0%            3c98c816-3f4d-48f0-9717-03d0c998637f  ra-1
UN  10.12.4.36  95.27 KiB  256          100.0%            0a36383d-1d9e-41e2-924c-7b62be12d6cc  ra-1
UN  10.12.5.22  88.7 KiB   256          100.0%            3561f4fa-af3d-4ea4-93b2-79ac7e938201  ra-1
Datacenter: us-west1
====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  10.0.4.33   78.69 KiB  256          100.0%              a200217d-260b-45cd-b83c-182b27ff4c99  ra-1
UN  10.0.0.21   78.68 KiB  256          100.0%              9f3364b9-a7a1-409c-9356-b7d1d312e52b  ra-1
UN  10.0.1.26   15.46 KiB  256          100.0%              1666df0f-702e-4c5b-8b6e-086d0f2e47fa  ra-1

Multi-region deployment Stay organized with collections Save and categorize content based on your preferences.

Prerequisites

GKE

GKE on-prem

AKS

EKS

OpenShift

Island mode clusters and hostNetwork

Cassandra hostname resolution

Configure Apigee hybrid for multi-region

GKE

Configure the multi-region seed host

Helm charts

Configure the second region

apigeectl

Set up the new region

Check the Cassandra cluster status

GKE on-prem

Configure the multi-region seed host

Helm charts

Configure the second region

apigeectl

Set up the new region

Check the Cassandra cluster status

AKS

Create a virtual network in each region

Create multi-regional clusters

Configure the multi-region seed host

Helm charts

Configure the second region

apigeectl

Set up the new region

Check the Cassandra cluster status

EKS

Create a virtual network in each region

Create multi-regional clusters

Configure the multi-region seed host

Helm charts

Configure the second region

apigeectl

Set up the new region

Check the Cassandra cluster status

OpenShift

Configure the multi-region seed host

Helm charts

Configure the second region

apigeectl

Set up the new region

Check the Cassandra cluster status

Troubleshooting

`apigeectl`

`apigeectl`

`apigeectl`

`apigeectl`

`apigeectl`