Deploying WeScale Cluster on Amazon EKS

This document describes how to deploy a WeScale cluster in Kubernetes on Amazon EKS.

Deploying WeScale in Kubernetes is similar to deploying a WeSQL-Server cluster. If you have already deployed a WeSQL-Server cluster, you can proceed directly to Step 4: Deploy the WeScale Cluster.

Prerequisites

Before we get started, let’s get a few prerequisites out of the way:

Prepare an AWS S3 bucket for WeSQL backups and logging.
Install kubectl and ensure it is in your PATH.
Install the MySQL client locally.
Prepare a Amazon EKS Kubernetes cluster.

Deploying the Cluster

1. Create the Configuration Files

Use kubectl to create a ConfigMap with your WeScale configuration.

You need to replace the objectstore_provider, objectstore_region, and objectstore_bucket with your AWS S3 bucket info.

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: wesql-server-config
data:
  MYSQL_CUSTOM_CONFIG: |
    [mysqld]
    objectstore_provider=aws
    objectstore_region=us-east-1
    objectstore_bucket=wesql-cluster-test
    serverless=on
    datadir=/data/mysql/data
    log-error=/data/mysql/log/mysqld-error.log
    log-bin=binlog
    gtid_mode=ON
    enforce_gtid_consistency=ON
    log_slave_updates=ON
    binlog_format=ROW
EOF

2. Create the Secret

Use kubectl to create a secret with your AWS S3 credentials. Replace YOUR_S3_ACCESS_KEY, YOUR_S3_SECRET_KEY, and YOUR_MYSQL_ROOT_PASSWORD with your actual AWS S3 credentials and desired MySQL root password.

kubectl create secret generic wesql-server-secret \
  --namespace default \
  --type Opaque \
  --from-literal=WESQL_OBJECTSTORE_ACCESS_KEY=${YOUR_S3_ACCESS_KEY} \
  --from-literal=WESQL_OBJECTSTORE_SECRET_KEY=${YOUR_S3_SECRET_KEY} \
  --from-literal=MYSQL_ROOT_PASSWORD=${YOUR_MYSQL_ROOT_PASSWORD}

3. Prepare a Kubernetes Cluster

Make sure you have a Kubernetes cluster running. You can check the status of the cluster by running the following command:

kubectl cluster-info

Example Output (Amazon EKS):

Kubernetes control plane is running at https://<eks-endpoint>.eks.amazonaws.com
CoreDNS is running at https://<eks-endpoint>.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

4. Deploy the WeScale Cluster

Deploy the WeScale cluster by running:

kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/artifact/wescale-gp3.yaml

Note: Different cloud providers have different storage classes. You can check the storage classes by running:
kubectl get storageclasses
Example Output:
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp3 (default)        kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  83d

Check the status of the pods to ensure the WeScale pods are running:

kubectl get pods

Example Output:

NAME                               READY   STATUS    RESTARTS   AGE
mycluster-wesql-0-0                2/2     Running   0          2m
mycluster-wesql-1-0                1/1     Running   0          2m
mycluster-wesql-2-0                1/1     Running   0          2m
wesql-vtcontroller                 2/2     Running   0          2m
wesql-vtgate-<pod-id>              1/1     Running   0          2m

5. Set Up Port Forwarding (Optional)

Forward the MySQL port to your local machine and connect to the WeScale cluster:

kubectl port-forward svc/wesql-vtgate-headless 15306:15306
mysql -h127.0.0.1 -P15306

Accessing the Cluster via the Public Network

To access your WeScale cluster from the internet, you need to expose the service externally by creating a public LoadBalancer. This allows you to connect to your database from outside the Kubernetes cluster.

1. Deploy the Public Service

Apply the public service configuration using kubectl:

kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtgate-public-service.yaml
kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtcontroller-public-service.yaml

Content of wescale-vtgate-public-service.yaml and wescale-vtcontroller-public-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: mycluster-wescale-vtgate-public-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: wesql-vtgate
  ports:
    - name: webport
      port: 15001
      protocol: TCP
      targetPort: webport
    - name: grpcport
      port: 15991
      protocol: TCP
      targetPort: grpcport
    - name: serverport
      port: 15306
      protocol: TCP
      targetPort: serverport
---
apiVersion: v1
kind: Service
metadata:
  name: mycluster-wescale-vtcontroller-public-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: wesql-vtcontroller
  ports:
    - name: vtctld-webport
      port: 15000
      protocol: TCP
      targetPort: vtctld-webport
    - name: vtctld-grpcport
      port: 15999
      protocol: TCP
      targetPort: vtctld-grpcport

2. Verify the LoadBalancer Creation

After applying the public service, Kubernetes will request your cloud provider (AWS) to provision a LoadBalancer. This process may take a few minutes.

Check the status of the service to get the external IP address or hostname:

kubectl get svc mycluster-wescale-vtgate-public-service
kubectl get svc mycluster-wescale-vtcontroller-public-service

Example Output:

NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                                         AGE
mycluster-wescale-vtgate-public-service    LoadBalancer   10.100.23.45     a1b2c3d4e5f6.cn-northwest-1.elb.amazonaws.com   15001:xxxxx/TCP,15991:xxxxx/TCP,15306:xxxxx/TCP   3m
mycluster-wescale-vtcontroller-public-service   LoadBalancer   10.100.67.89     b1c2d3e4f5g6.cn-northwest-1.elb.amazonaws.com   15000:xxxxx/TCP,15999:xxxxx/TCP                     3m

Note: The EXTERNAL-IP field shows the DNS name of the LoadBalancer. It may take a few minutes for the EXTERNAL-IP to be assigned.

3. Configure AWS Resources (If Necessary)

While Kubernetes automates the creation of the LoadBalancer, ensure the following AWS configurations are in place:

IAM Permissions: The AWS IAM role associated with your EKS worker nodes must have permissions to create and manage Elastic Load Balancers (ELBs). The necessary permissions include:
- elasticloadbalancing:*
- ec2:*
- iam:ListServerCertificates
VPC and Subnets:
- The EKS cluster should be deployed in a VPC with public subnets to allow internet access.
- Public subnets should be tagged appropriately:
```
kubernetes.io/role/elb: 1
```
Security Groups:
- Ensure the security group associated with the LoadBalancer allows inbound traffic on port 15305 from your client IP address or the desired IP range.

4. Wait for DNS Propagation

The DNS name provided in the EXTERNAL-IP may require some time to propagate. You can verify when it's ready by checking DNS resolution:

nslookup a1b2c3d4e5f6.cn-northwest-1.elb.amazonaws.com

Replace a1b2c3d4e5f6.cn-northwest-1.elb.amazonaws.com with the actual EXTERNAL-IP DNS name from your service.

Once the DNS name resolves to an IP address, you can proceed to connect.

5. Connect to the WeScale Cluster

Use the MySQL client to connect to your database using the LoadBalancer's DNS name:

mysql -h a1b2c3d4e5f6.cn-northwest-1.elb.amazonaws.com -P15306 -uroot -p${YOUR_MYSQL_ROOT_PASSWORD}

Important: Replace a1b2c3d4e5f6.cn-northwest-1.elb.amazonaws.com with the actual EXTERNAL-IP DNS name from your mycluster-wescale-vtgate-public-service.

Troubleshooting Tips

Connection Errors:
- If you receive an error like Host 'X.X.X.X' is not allowed to connect to this MySQL server, ensure your MySQL user is configured to allow connections from your client IP address.
Firewall and Security Groups:
- Double-check that your AWS security groups allow inbound traffic on port 15306 from your IP address.
DNS Resolution Issues:
- If the DNS name isn't resolving, try flushing your DNS cache or wait a few more minutes for propagation.
LoadBalancer Status:
- Verify the LoadBalancer is in the Active state in the AWS console under EC2 > Load Balancers.

Security Considerations

Exposing your database to the internet introduces security risks. Take the following precautions:

Restrict Access:
- Limit inbound traffic to specific IP addresses using security group rules.
Use Secure Passwords:
- Ensure your MySQL root password is strong and not easily guessable.
Enable SSL/TLS:
- Configure MySQL to use SSL/TLS encryption for connections.
Monitor Access Logs:
- Regularly check logs for unauthorized access attempts.
Consider VPN or Bastion Hosts:
- For enhanced security, access the database through a VPN connection or via a bastion host instead of exposing it directly to the internet.

Delete the Cluster

Delete the WeScale cluster by running:

kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/artifact/wescale-gp3.yaml

Delete the Load Balancer services (if created):

kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtgate-public-service.yaml
kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtcontroller-public-service.yaml

Delete the ConfigMap and Secret:

kubectl delete configmap wesql-server-config
kubectl delete secret wesql-server-secret

Additionally, you can wipe out the data by deleting PersistentVolumeClaims (PVCs) and S3 bucket objects:

kubectl get pvc
kubectl delete pvc data-mycluster-wesql-0-0 data-mycluster-wesql-1-0 data-mycluster-wesql-2-0

Deploying WeScale Cluster on Amazon EKS

Prerequisites​

Deploying the Cluster​

1. Create the Configuration Files​

2. Create the Secret​

3. Prepare a Kubernetes Cluster​

4. Deploy the WeScale Cluster​

5. Set Up Port Forwarding (Optional)​

Accessing the Cluster via the Public Network​

1. Deploy the Public Service​

2. Verify the LoadBalancer Creation​

3. Configure AWS Resources (If Necessary)​

4. Wait for DNS Propagation​

5. Connect to the WeScale Cluster​

Delete the Cluster​