Skip to main content

Run Both WeSQL-Server and WeScale in EKS

This document describes how to deploy a WeScale cluster (including both WeSQL-Server and WeScale) on Amazon EKS.

Deploying WeScale in Kubernetes is similar to deploying a WeSQL-Server cluster. If you have already deployed a WeSQL-Server cluster, you can proceed directly to Step 4: Deploy the WeScale Cluster.

Prerequisites​

Before we get started, let’s get a few prerequisites out of the way:

  1. Prepare an AWS S3 bucket for WeSQL backups and logging.
  2. Install kubectl and ensure it is in your PATH.
  3. Install the MySQL client locally.
  4. Prepare a Amazon EKS Kubernetes cluster.

Deploying the Cluster​

1. Create the Configuration Files​

Use kubectl to create a ConfigMap with your WeScale configuration.

You need to replace the objectstore_provider, objectstore_region, and objectstore_bucket with your AWS S3 bucket info.

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: wesql-server-config
data:
  MYSQL_CUSTOM_CONFIG: |
    [mysqld]
    objectstore_provider=aws
    objectstore_region=us-west-1
    objectstore_bucket=wesql-storage
    repo_objectstore_id=sysbench
    branch_objectstore_id=main
    log-bin=binlog
    gtid_mode=ON
    enforce_gtid_consistency=ON
    log_slave_updates=ON
    binlog_format=ROW
    skip_name_resolve=ON
EOF
note

Since we need to expose MySQL's service address through AWS Load Balancer (NLB), and the NLB will periodically perform health checks on the port, we need to enable skip_name_resolve to avoid errors like this:

ERROR 1129 (HY000): Host '192.168.47.47' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts'

2. Create the Secret​

Use kubectl to create a secret with your AWS S3 credentials. Replace YOUR_S3_ACCESS_KEY, YOUR_S3_SECRET_KEY, and YOUR_MYSQL_ROOT_PASSWORD with your actual AWS S3 credentials and desired MySQL root password.

kubectl create secret generic wesql-server-secret \
  --namespace default \
  --type Opaque \
  --from-literal=WESQL_OBJECTSTORE_ACCESS_KEY=${YOUR_S3_ACCESS_KEY} \
  --from-literal=WESQL_OBJECTSTORE_SECRET_KEY=${YOUR_S3_SECRET_KEY} \
  --from-literal=MYSQL_ROOT_PASSWORD=${YOUR_MYSQL_ROOT_PASSWORD}

3. Prepare a Kubernetes Cluster​

Make sure you have a Kubernetes cluster running. You can check the status of the cluster by running the following command:

kubectl cluster-info

Example Output (Amazon EKS):

Kubernetes control plane is running at https://<eks-endpoint>.eks.amazonaws.com
CoreDNS is running at https://<eks-endpoint>.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

4. Deploy the WeScale Cluster​

Deploy the WeScale cluster by running:

kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/artifact/wescale.yaml

Check the status of the pods to ensure the WeScale pods are running:

kubectl get pods

Example Output:

NAME                               READY   STATUS    RESTARTS   AGE
mycluster-wesql-0-0                2/2     Running   0          2m
mycluster-wesql-1-0                1/1     Running   0          2m
mycluster-wesql-2-0                1/1     Running   0          2m
wesql-vtcontroller                 2/2     Running   0          2m
wesql-vtgate-<pod-id>              1/1     Running   0          2m

5. Set Up Port Forwarding (Optional)​

Forward the MySQL port to your local machine and connect to the WeScale cluster:

kubectl port-forward svc/wesql-vtgate-headless 15306:15306
mysql -h127.0.0.1 -P15306

Accessing the Cluster via the Public Network​

To access your WeScale cluster from the internet, you need to expose the service externally by creating a public LoadBalancer. This allows you to connect to your database from outside the Kubernetes cluster.

1. Deploy the Public Service​

Apply the public service configuration using kubectl:

kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtgate-public-service.yaml
kubectl apply -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtcontroller-public-service.yaml
Content of wescale-vtgate-public-service.yaml and wescale-vtcontroller-public-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: mycluster-wescale-vtgate-public-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: wesql-vtgate
  ports:
    - name: webport
      port: 15001
      protocol: TCP
      targetPort: webport
    - name: grpcport
      port: 15991
      protocol: TCP
      targetPort: grpcport
    - name: serverport
      port: 15306
      protocol: TCP
      targetPort: serverport
---
apiVersion: v1
kind: Service
metadata:
  name: mycluster-wescale-vtcontroller-public-service
  namespace: default
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: wesql-vtcontroller
  ports:
    - name: vtctld-webport
      port: 15000
      protocol: TCP
      targetPort: vtctld-webport
    - name: vtctld-grpcport
      port: 15999
      protocol: TCP
      targetPort: vtctld-grpcport

2. Verify the LoadBalancer Creation​

After applying the public service, Kubernetes will request your cloud provider (AWS) to provision a LoadBalancer. This process may take a few minutes.

Check the status of the service to get the external IP address or hostname:

kubectl get svc mycluster-wescale-vtgate-public-service
kubectl get svc mycluster-wescale-vtcontroller-public-service

Example Output:

NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)                                         AGE
mycluster-wescale-vtgate-public-service    LoadBalancer   10.100.23.45     a1b2c3d4e5f6.us-west-1.elb.amazonaws.com   15001:xxxxx/TCP,15991:xxxxx/TCP,15306:xxxxx/TCP   3m
mycluster-wescale-vtcontroller-public-service   LoadBalancer   10.100.67.89     b1c2d3e4f5g6.us-west-1.elb.amazonaws.com   15000:xxxxx/TCP,15999:xxxxx/TCP                     3m

Note: The EXTERNAL-IP field shows the DNS name of the LoadBalancer. It may take a few minutes for the EXTERNAL-IP to be assigned.

3. Configure AWS Resources (If Necessary)​

While Kubernetes automates the creation of the LoadBalancer, ensure the following AWS configurations are in place:

  • IAM Permissions: The AWS IAM role associated with your EKS worker nodes must have permissions to create and manage Elastic Load Balancers (ELBs). The necessary permissions include:

    • elasticloadbalancing:*
    • ec2:*
    • iam:ListServerCertificates

  • VPC and Subnets:

    • The EKS cluster should be deployed in a VPC with public subnets to allow internet access.

    • Public subnets should be tagged appropriately:

      kubernetes.io/role/elb: 1

  • Security Groups:

    • Ensure the security group associated with the LoadBalancer allows inbound traffic on port 15305 from your client IP address or the desired IP range.

4. Wait for DNS Propagation​

The DNS name provided in the EXTERNAL-IP may require some time to propagate. You can verify when it's ready by checking DNS resolution:

nslookup a1b2c3d4e5f6.us-west-1.elb.amazonaws.com

Replace a1b2c3d4e5f6.us-west-1.elb.amazonaws.com with the actual EXTERNAL-IP DNS name from your service.

Once the DNS name resolves to an IP address, you can proceed to connect.

5. Connect to the WeScale Cluster​

Use the MySQL client to connect to your database using the LoadBalancer's DNS name:

mysql -h a1b2c3d4e5f6.us-west-1.elb.amazonaws.com -P15306 -uroot -p${YOUR_MYSQL_ROOT_PASSWORD}

Important: Replace a1b2c3d4e5f6.us-west-1.elb.amazonaws.com with the actual EXTERNAL-IP DNS name from your mycluster-wescale-vtgate-public-service.

Troubleshooting Tips

  • Connection Errors:

    • If you receive an error like Host 'X.X.X.X' is not allowed to connect to this MySQL server, ensure your MySQL user is configured to allow connections from your client IP address.

  • Firewall and Security Groups:

    • Double-check that your AWS security groups allow inbound traffic on port 15306 from your IP address.

  • DNS Resolution Issues:

    • If the DNS name isn't resolving, try flushing your DNS cache or wait a few more minutes for propagation.

  • LoadBalancer Status:

    • Verify the LoadBalancer is in the Active state in the AWS console under EC2 > Load Balancers.
Security Considerations

Exposing your database to the internet introduces security risks. Take the following precautions:

  • Restrict Access:

    • Limit inbound traffic to specific IP addresses using security group rules.

  • Use Secure Passwords:

    • Ensure your MySQL root password is strong and not easily guessable.

  • Enable SSL/TLS:

    • Configure MySQL to use SSL/TLS encryption for connections.

  • Monitor Access Logs:

    • Regularly check logs for unauthorized access attempts.

  • Consider VPN or Bastion Hosts:

    • For enhanced security, access the database through a VPN connection or via a bastion host instead of exposing it directly to the internet.

Delete the Cluster​

Delete the WeScale cluster by running:

kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/artifact/wescale.yaml

Delete the Load Balancer services (if created):

kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtgate-public-service.yaml
kubectl delete -f https://raw.githubusercontent.com/wesql/deploy/refs/heads/main/public-service/wescale-vtcontroller-public-service.yaml

Delete the ConfigMap and Secret:

kubectl delete configmap wesql-server-config
kubectl delete secret wesql-server-secret

Additionally, you can wipe out the data by deleting PersistentVolumeClaims (PVCs) and S3 bucket objects:

kubectl get pvc
kubectl delete pvc data-mycluster-wesql-0-0 data-mycluster-wesql-1-0 data-mycluster-wesql-2-0