Deploying WeSQL Serverless Cluster in Kubernetes

This document outlines the steps to deploy a WeSQL cluster using Kubernetes StatefulSets. The WeSQL cluster consists of three MySQL pods (mycluster-wesql-0, mycluster-wesql-1, and mycluster-wesql-2) with specific configurations for leader election, replication, and object storage. Follow these instructions to ensure a smooth deployment of the cluster.

Prerequisites

• Kubernetes Cluster: Ensure you have a working Kubernetes cluster with sufficient resources.
• Kubectl: Installed and configured to manage your Kubernetes cluster.
• Persistent Volume: Ensure you have the necessary storage class (e.g., gp2) configured for the persistent volume claims (PVCs).
• Object Storage: An AWS S3-compatible object storage for WeSQL backups and logging.

YAML Configuration

This configuration file deploys a WeSQL cluster using StatefulSet. Below, we will describe each component of the YAML configuration.

Pod Environment Variables

Each WeSQL container defines a set of environment variables to configure cluster behavior.

Common Variables

• WESQL_CLUSTER_MEMBERS: Specifies the endpoints of all WeSQL nodes in the cluster.
• WESQL_CLUSTER_MEMBER_INDEX: Specifies the endpoints of all WeSQL nodes in the cluster.
• WESQL_OBJECTSTORE_ACCESS_KEY: AWS credentials for accessing the object store.
• WESQL_OBJECTSTORE_SECRET_KEY: AWS credentials for accessing the object store.
• MYSQL_ROOT_PASSWORD: Credentials for the root MySQL user.
• MYSQL_CUSTOM_CONFIG: Custom WeSQL settings.

WeSQL Serverless Configuration

Each pod includes custom WeSQL settings. Below are some key settings for serverless:

Object Storage Configuration

WeSQL uses object storage as its persistent medium. Here are the key settings:

• objectstore_provider=aws: Defines AWS as the object storage provider.
• objectstore_bucket=wesql-cluster-1: Specifies the S3 bucket to be used for storage.

Binlog Raft replication Configuration

WeSQL use raft protocol to persist binlog in all pods, relation configuration please referred.

Persistent Storage Configuration

Each StatefulSet uses a Persistent Volume Claim (PVC) for data storage:

volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 50Gi
    storageClassName: gp2

• Access Mode: ReadWriteOnce ensures that only a single node can write to the volume.
• Storage Request: Each pod requests 50Gi of storage for its MySQL data.

Headless service

A headless service will enable communication between the Pods in a WeSQL cluster, leveraging WeSQL on port 3306 and Raft on port 13306.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: wesql
    app.kubernetes.io/instance: mycluster
  name: mycluster-wesql-headless
  namespace: default
spec:
  clusterIP: None
  ports:
  - name: wesql
    port: 3306
    protocol: TCP
    targetPort: wesql
  - name: raft
    port: 13306
    protocol: TCP
    targetPort: raft
  publishNotReadyAddresses: true
  selector:
    app.kubernetes.io/instance: mycluster
    app.kubernetes.io/component: wesql
  type: ClusterIP

StatefulSet Definitions

The cluster is composed of three StatefulSets for the WeSQL nodes. Each StatefulSet is responsible for managing its own instance in a WeSQL cluster. All StatefulSets use the service mycluster-wesql-headless for internal communication.

• mycluster-wesql-0

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mycluster-wesql-0
  namespace: default
  labels:
    app.kubernetes.io/name: mycluster-wesql-0
spec:
  serviceName: "mycluster-wesql-headless"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: mycluster-wesql-0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mycluster-wesql-0
        app.kubernetes.io/component: wesql
        app.kubernetes.io/instance: mycluster
    spec:
      containers:
      - name: mysql
        image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/wesql-server:8.0.35-6.alpha9.20240904.ge7e5076.23
        imagePullPolicy: IfNotPresent
        env
        - name: MYSQL_ROOT_PASSWORD
          value: "1234"
        - name: WESQL_CLUSTER_MEMBERS
          value: mycluster-wesql-0-0.mycluster-wesql-headless:13306;mycluster-wesql-1-0.mycluster-wesql-headless:13306;mycluster-wesql-2-0.mycluster-wesql-headless:13306
        - name: WESQL_CLUSTER_MEMBER_INDEX
          value: "0"
        - name: MYSQL_CUSTOM_CONFIG
          value: |
            [mysqld]
            serverless=on
            objectstore_provider=aws
            objectstore_region=cn-northwest-1
            objectstore_bucket=wesql-cluster-1
            datadir=/data/mysql/data
            log-error=/data/mysql/log/mysqld-error.log
            log-bin=binlog
        - name: WESQL_OBJECTSTORE_ACCESS_KEY
          value: '***********************************'
        - name: WESQL_OBJECTSTORE_SECRET_KEY
          value: '***********************************'
        ports:
        - containerPort: 3306
          name: mysql
          protocol: TCP
        - containerPort: 13306
          name: raft
          protocol: TCP
        resources:
          limits:
            cpu: "4"
            memory: 8Gi
          requests:
            cpu: "4"
            memory: 8Gi
        volumeMounts:
        - mountPath: /data/mysql
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi
      storageClassName: gp2

• mycluster-wesql-1

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mycluster-wesql-1
  namespace: default
  labels:
    app.kubernetes.io/name: mycluster-wesql-1
spec:
  serviceName: "mycluster-wesql-headless"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: mycluster-wesql-1
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mycluster-wesql-1
        app.kubernetes.io/component: wesql
        app.kubernetes.io/instance: mycluster
    spec:
      containers:
      - name: mysql
        image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/wesql-server:8.0.35-6.alpha9.20240904.ge7e5076.23
        imagePullPolicy: IfNotPresent
        env:
        - name: WESQL_CLUSTER_MEMBERS
          value: mycluster-wesql-0-0.mycluster-wesql-headless:13306;mycluster-wesql-1-0.mycluster-wesql-headless:13306;mycluster-wesql-2-0.mycluster-wesql-headless:13306
        - name: WESQL_CLUSTER_MEMBER_INDEX
          value: "1"
        - name: MYSQL_ROOT_PASSWORD
          value: "1234"
        - name: MYSQL_CUSTOM_CONFIG
          value: |
            [mysqld]
            serverless=on
            objectstore_provider=aws
            objectstore_region=cn-northwest-1
            objectstore_bucket=wesql-cluster-1
            datadir=/data/mysql/data
            log-error=/data/mysql/log/mysqld-error.log
            log-bin=binlog
        - name: WESQL_LOGGER_MODE
          value: "1"
        - name: WESQL_OBJECTSTORE_ACCESS_KEY
          value: '***********************************'
        - name: WESQL_OBJECTSTORE_SECRET_KEY
          value: '***********************************'
        ports:
        - containerPort: 3306
          name: mysql
          protocol: TCP
        - containerPort: 13306
          name: raft
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /data/mysql
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi
      storageClassName: gp2

• mycluster-wesql-2

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mycluster-wesql-2
  namespace: default
  labels:
    app.kubernetes.io/name: mycluster-wesql-2
spec:
  serviceName: "mycluster-wesql-headless"
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: mycluster-wesql-2
  template:
    metadata:
      labels:
        app.kubernetes.io/name: mycluster-wesql-2
        app.kubernetes.io/component: wesql
        app.kubernetes.io/instance: mycluster
    spec:
      containers:
      - name: mysql
        image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/wesql-server:8.0.35-6.alpha8.20240819.g6b3665a.20
        imagePullPolicy: IfNotPresent
        env:
        - name: WESQL_CLUSTER_MEMBERS
          value: mycluster-wesql-0-0.mycluster-wesql-headless:13306;mycluster-wesql-1-0.mycluster-wesql-headless:13306;mycluster-wesql-2-0.mycluster-wesql-headless:13306
        - name: WESQL_CLUSTER_MEMBER_INDEX
          value: "2"
        - name: MYSQL_ROOT_PASSWORD
          value: "1234"
        - name: MYSQL_CUSTOM_CONFIG
          value: |
            [mysqld]
            serverless=on
            objectstore_provider=aws
            objectstore_region=cn-northwest-1
            objectstore_bucket=wesql-cluster-1
            datadir=/data/mysql/data
            log-error=/data/mysql/log/mysqld-error.log
        - name: WESQL_LOGGER_MODE
          value: "1"
        - name: WESQL_OBJECTSTORE_ACCESS_KEY
          value: '***********************************'
        - name: WESQL_OBJECTSTORE_SECRET_KEY
          value: '***********************************'
        ports:
        - containerPort: 3306
          name: mysql
          protocol: TCP
        - containerPort: 13306
          name: paxos
          protocol: TCP
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
        volumeMounts:
        - mountPath: /data/mysql
          name: data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi
      storageClassName: gp2

Deploying the Cluster

1. Apply the YAML file: Use kubectl apply to deploy the StatefulSets.

kubectl apply -f wesql-cluster.yaml

2. Verify Pods: Check if all pods are running.

kubectl get pods

You should see mycluster-wesql-0-0, mycluster-wesql-1-0, and mycluster-wesql-2-0 running successfully.

3. Check Logs: Monitor the MySQL logs to ensure that the cluster is functioning properly.

kubectl logs mycluster-wesql-0

4. Access MySQL: Forward the MySQL port to your local machine to interact with the database.

kubectl port-forward mycluster-wesql-0 3306:3306
mysql -h 127.0.0.1 -P 3306 -u root -p1234