Scribd
Upload
4 views13 pages

MySQL Clusterset Setup

The document outlines the steps to deploy MySQL InnoDB ClusterSet across Kubernetes clusters using OpenShift and Cilium. It details prerequisites, DNS configuration, LoadBalancer setup for MySQL pods, and the installation of the MySQL Operator, along with necessary network policies. Additionally, it compares Cilium's approach to OpenShift-native methods for cross-cluster connectivity and DNS resolution.

Uploaded by

devops.mvc
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
4 views13 pages

MySQL Clusterset Setup

The document outlines the steps to deploy MySQL InnoDB ClusterSet across Kubernetes clusters using OpenShift and Cilium. It details prerequisites, DNS configuration, LoadBalancer setup for MySQL pods, and the installation of the MySQL Operator, along with necessary network policies. Additionally, it compares Cilium's approach to OpenShift-native methods for cross-cluster connectivity and DNS resolution.

Uploaded by

devops.mvc
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

[Link]

com/mysql/deploying-mysql-innodb-clusterset-across-
kubernetes-clusters-using-cilium

Phase 1 — Prerequisites
Network: Both OpenShift clusters must have L3 connectivity (VPN, transit VPC, or direct
peering). Confirm that worker node IPs are mutually reachable. Test:

bash
# From a debug pod on DC cluster, reach a DR worker node
oc debug node/<dc-worker> -- curl -v telnet://<dr-worker-ip>:30053

Firewall rules to open (Must require Openshift Cluster admin Access):

 TCP/UDP 30053 — cross-cluster CoreDNS NodePort


 TCP 3306 — MySQL client connections
 TCP 33061 — MySQL Group Replication (InnoDB ClusterSet uses this)
 TCP 6446/6447 — MySQL Router read-write / read-only

Expose CoreDNS as NodePort on both clusters (Must require Openshift Cluster admin
Access):

# [Link] — apply to BOTH clusters


apiVersion: v1
kind: Service
metadata:
name: coredns-nodeport
namespace: openshift-dns # OpenShift DNS namespace (not kube-
system)
labels:
app: coredns-nodeport
spec:
selector:
[Link]/daemonset-dns: default # OpenShift
CoreDNS label
type: NodePort
ports:
- name: dns-udp
port: 53
protocol: UDP
targetPort: 5353 # OpenShift CoreDNS listens on 5353,
not 53
nodePort: 30053
- name: dns-tcp
port: 53
protocol: TCP
targetPort: 5353
nodePort: 30053
# Apply on DC cluster
oc apply -f [Link] --context dc-cluster

# Apply on DR cluster
oc apply -f [Link] --context dr-cluster

OpenShift difference: CoreDNS in OpenShift runs under the openshift-dns namespace and
the dns-default DaemonSet, listening on port 5353 (not 53). The selector label is different
from upstream Kubernetes.

Collect worker node IPs for each cluster — you'll need these for the stub zone config:

# DC cluster worker IPs


oc get nodes --context dc-cluster \
-l [Link]/worker \
-o jsonpath='{.items[*].[Link][?
(@.type=="InternalIP")].address}'

# DR cluster worker IPs


oc get nodes --context dr-cluster \
-l [Link]/worker \
-o jsonpath='{.items[*].[Link][?
(@.type=="InternalIP")].address}'

Configure cross-cluster DNS stub zones


This is the direct replacement for Cilium's DNS mesh. Each cluster's CoreDNS learns to forward
the other cluster's namespace DNS zone to the remote CoreDNS NodePort.

On DC cluster — forward [Link] to DR


In OpenShift, CoreDNS config is managed via the [Link] CR, not by
editing a ConfigMap directly. Add a stub zone using a DNSForwardingRules object via the
cluster DNS operator:

oc edit [Link]/default --context dc-cluster (Must require Openshift


Cluster admin Access):

spec:
servers:
- name: dr-stub
zones:
- [Link]
forwardPlugin:
upstreams:
- <DR_WORKER_1_IP>:30053
- <DR_WORKER_2_IP>:30053
- <DR_WORKER_3_IP>:30053

On DR cluster — forward [Link] to DC (Must require


Openshift Cluster admin Access):

oc edit [Link]/default --context dr-cluster


spec:
servers:
- name: dc-stub
zones:
- [Link]
forwardPlugin:
upstreams:
- <DC_WORKER_1_IP>:30053
- <DC_WORKER_2_IP>:30053
- <DC_WORKER_3_IP>:30053

The DNS operator automatically rebuilds the CoreDNS ConfigMap and restarts the DaemonSet
pods — you don't need to touch them manually.

Validate DNS resolution:

# From a pod on DC cluster, resolve a DR pod FQDN


oc run dnstest --image=busybox --rm -it --restart=Never \
-n mysql-prod --context dc-cluster -- \
nslookup [Link]
# Should return the dr-0 pod IP or an external LB IP

Expose each MySQL pod externally via LoadBalancer


Cilium made pods reachable cross-cluster using its mesh. Without it, you create a dedicated
LoadBalancer Service per pod. InnoDB ClusterSet requires FQDN resolution to reach specific
pod instances (not just a VIP), so each pod needs its own external endpoint.

# [Link]
apiVersion: v1
kind: Service
metadata:
name: prd-0-ext
namespace: mysql-prod
spec:
selector:
[Link]/pod-name: prd-0
type: LoadBalancer
ports:
- name: mysql
port: 3306
targetPort: 3306
- name: gr
port: 33061
targetPort: 33061
---
apiVersion: v1
kind: Service
metadata:
name: prd-1-ext
namespace: mysql-prod
spec:
selector:
[Link]/pod-name: prd-1
type: LoadBalancer
ports:
- name: mysql
port: 3306
targetPort: 3306
- name: gr
port: 33061
targetPort: 33061
---
apiVersion: v1
kind: Service
metadata:
name: prd-2-ext
namespace: mysql-prod
spec:
selector:
[Link]/pod-name: prd-2
type: LoadBalancer
ports:
- name: mysql
port: 3306
targetPort: 3306
- name: gr
port: 33061
targetPort: 33061

oc apply -f [Link] --context dc-cluster

Repeat the same pattern for DR (dr-0-ext, dr-1-ext, dr-2-ext) in mysql-dr namespace on
the DR cluster.

Collect the external IPs once they're assigned:

oc get svc -n mysql-prod --context dc-cluster | grep ext


# prd-0-ext LoadBalancer ... <EXTERNAL-IP>
3306:xxx/TCP,33061:xxx/TCP

Add ExternalName Services to bridge DNS to LB IPs


This is a key step that has no Cilium equivalent but is required for OpenShift. InnoDB ClusterSet
expects [Link] to resolve to something reachable
from the DC cluster. You create ExternalName Services on the DC cluster that map each DR
pod's FQDN to the DR LoadBalancer IP.

# On DC cluster — make DR pods resolvable via their FQDN


apiVersion: v1
kind: Service
metadata:
name: dr-0-ext-alias
namespace: mysql-prod # must be in the same namespace for FQDN
tricks
spec:
type: ExternalName
externalName: <DR_POD_0_LB_EXTERNAL_IP>
ports:
- port: 3306

Alternatively, you can skip ExternalName and simply use the LoadBalancer IPs
directly in CHANGE REPLICATION SOURCE TO and createReplicaCluster mysqlsh
commands — InnoDB ClusterSet accepts IP addresses, not just FQDNs. Confirm pod
hostnames are set correctly with report_host in MySQL config.

Set report_host in each pod's MySQL config so Group Replication advertises the
correct external address:

# In InnoDBCluster spec, add mycnf overrides


spec:
mycnfConfigMap:
name: mysql-extra-config

# mysql-extra-config ConfigMap
data:
[Link]: |
[mysqld]
report_host = <THIS_POD_LB_EXTERNAL_IP>
report_port = 3306

Set up NetworkPolicy and EgressFirewall (Must require Openshift Cluster admin


Access):

OpenShift enforces network policy strictly. Without these, cross-cluster traffic will be silently
dropped.

# Allow egress from mysql-prod pods to DR LB IPs (ports 3306, 33061)


apiVersion: [Link]/v1
kind: NetworkPolicy
metadata:
name: allow-dr-replication
namespace: mysql-prod
spec:
podSelector:
matchLabels:
app: prd
policyTypes:
- Egress
egress:
- to:
- ipBlock:
cidr: <DR_LB_CIDR>/32 # repeat for each DR LB IP
ports:
- protocol: TCP
port: 3306
- protocol: TCP
port: 33061

Also create an EgressNetworkPolicy (OpenShift-specific, for legacy SDN) or


AdminNetworkPolicy if using OVN-K: (Must require Openshift Cluster admin Access):

apiVersion: [Link]/v1
kind: EgressNetworkPolicy
metadata:
name: allow-dr
namespace: mysql-prod
spec:
egress:
- type: Allow
to:
cidrSelector: <DR_LB_CIDR>/32

Install MySQL Operator on both clusters

# Apply CRDs on both clusters


kubectl apply -f [Link]
operator/trunk/deploy/[Link] \
--context dc-cluster

kubectl apply -f [Link]


operator/trunk/deploy/[Link] \
--context dr-cluster
# Download operator manifest
wget
[Link]
[Link]

MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN variable stays [Link]:

env:
- name: MYSQL_OPERATOR_K8S_CLUSTER_DOMAIN
value: [Link]

kubectl apply -f [Link] --context dc-cluster


kubectl apply -f [Link] --context dr-cluster

OpenShift-specific: The MySQL Operator runs as a non-root container. In


OpenShift you may need to grant it the anyuid SCC:

oc adm policy add-scc-to-user anyuid \


system:serviceaccount:mysql-operator:mysql-operator-sa \
--context dc-cluster

# Create namespaces and secrets


oc new-project mysql-prod --context dc-cluster
oc -n mysql-prod create secret generic mypwds \
--from-literal=rootUser=root \
--from-literal=rootHost=% \
--from-literal=rootPassword="StrongPass123" \
--context dc-cluster

oc new-project mysql-dr --context dr-cluster


oc -n mysql-dr create secret generic mypwds \
--from-literal=rootUser=root \
--from-literal=rootHost=% \
--from-literal=rootPassword="StrongPass123" \
--context dr-cluster

# PRD_cluster.yaml
apiVersion: [Link]/v2
kind: InnoDBCluster
metadata:
name: prd
namespace: mysql-prod
spec:
secretName: mypwds
baseServerId: 100
tlsUseSelfSigned: true
instances: 3
router:
instances: 1

# DR_cluster.yaml
apiVersion: [Link]/v2
kind: InnoDBCluster
metadata:
name: dr
namespace: mysql-dr
spec:
secretName: mypwds
baseServerId: 200
tlsUseSelfSigned: true
instances: 3
router:
instances: 1

kubectl apply -f PRD_cluster.yaml --context dc-cluster


kubectl apply -f DR_cluster.yaml --context dr-cluster

# Wait and verify


oc -n mysql-prod get ic --context dc-cluster
oc -n mysql-dr get ic --context dr-cluster

Validate cross-cluster connectivity


use mysqlsh from one pod to reach the other via FQDN directly over the Cilium mesh. Without
Cilium, connectivity goes through the LoadBalancer IPs:

# From DC prd-0 pod, connect to DR pod via its external LB IP


oc -n mysql-prod --context dc-cluster exec -it prd-0 -- \
mysqlsh root:StrongPass123@<DR_POD_0_LB_IP>:3306

# Confirm the report_host seen by Group Replication is the external IP


oc -n mysql-prod --context dc-cluster exec -it prd-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- \
sql "SELECT member_host, member_port FROM
performance_schema.replication_group_members;"

Create ClusterSet and add Replica Cluster

# Step 1: Create ClusterSet on PROD


oc --context dc-cluster -n mysql-prod exec -it prd-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- cluster createClusterSet mycs

# Step 2: Dissolve DR's standalone cluster


oc --context dr-cluster -n mysql-dr exec -it dr-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- cluster dissolve

# Step 3: Create replica cluster — use DR pod's LB IP (or FQDN if DNS works)
oc --context dc-cluster -n mysql-prod exec -it prd-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- \
clusterset createReplicaCluster \
root:StrongPass123@<DR_POD_0_LB_IP>:3306 \
myreplica --recoveryMethod=clone

# Step 4: Export and sync secrets (identical to blog)


export routerPassword=$(oc --context dc-cluster -n mysql-prod \
get secret prd-router -o json | jq -r '.data["routerPassword"]')
export clusterAdminPassword=$(oc --context dc-cluster -n mysql-prod \
get secret prd-privsecrets -o json | jq -r '.data["clusterAdminPassword"]')
export backupPassword=$(oc --context dc-cluster -n mysql-prod \
get secret prd-backup -o json | jq -r '.data["backupPassword"]')

oc --context dr-cluster -n mysql-dr get secret dr-backup -o json | \


jq --arg x "$backupPassword" '.data["backupPassword"]=$x' | \
kubectl --context dr-cluster apply -f -

oc --context dr-cluster -n mysql-dr get secret dr-privsecrets -o json | \


jq --arg x "$clusterAdminPassword" '.data["clusterAdminPassword"]=$x' | \
kubectl --context dr-cluster apply -f -

# Step 5: Add secondary DR pods to replica cluster


oc --context dr-cluster -n mysql-dr exec -it dr-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- \
cluster add-instance root:StrongPass123@<DR_POD_1_LB_IP>:3306 \
--recoveryMethod=clone

oc --context dr-cluster -n mysql-dr exec -it dr-0 -- \


mysqlsh root:StrongPass123@localhost:3306 -- \
cluster add-instance root:StrongPass123@<DR_POD_2_LB_IP>:3306 \
--recoveryMethod=clone

# Step 6: Restart routers


oc --context dc-cluster -n mysql-prod rollout restart deployment prd-router
oc --context dr-cluster -n mysql-dr rollout restart deployment dr-router

Validate ClusterSet status


# Full ClusterSet topology — should show both clusters
oc --context dc-cluster -n mysql-prod exec -it prd-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- clusterset status
# Check replication health from DR side
oc --context dr-cluster -n mysql-dr exec -it dr-0 -- \
mysqlsh root:StrongPass123@localhost:3306 -- cluster status

Expected ClusterSet output:


{
"domainName": "mycs",
"primaryCluster": "prd",
"status": "HEALTHY",
"statusText": "All Clusters available. Topology active.",
"clusters": {
"prd": {"status": "OK", "clusterRole": "PRIMARY"},
"myreplica": {"status": "OK", "clusterRole": "REPLICA", "replicationLag":
"00:00:00"}
}
}

Key differences summary — Cilium vs OpenShift-native


Concern Cilium approach OpenShift-native approach

Cross-cluster pod
Cluster mesh (pod IPs routed) Per-pod LoadBalancer Services
reachability

Cross-cluster Mesh makes [Link] CoreDNS stub zones via DNS


DNS global Operator CR

CoreDNS NodePort 30053


DNS exposure CoreDNS NodePort 30053 (kube-
(openshift-dns, port 5353
port system)
internally)

Service
[Link]/global=true Not needed — routing is explicit
annotations

OVN-K NetworkPolicy +
Network policy Cilium Network Policy
EgressNetworkPolicy

Pod FQDN (mesh makes it


report_host External LB IP of each pod
routable)
Concern Cilium approach OpenShift-native approach

ClusterSet Identical — mysqlsh doesn't


Identical
commands care about CNI

You might also like