---
# System prepended metadata

title: TLS

---

# TLS 

> - **Objective:** Configure TLS in a TiDB Cluster
> - **Prerequisites:**
>   - background knowledge of TiDB components
>   - AWS account
> - **Optionality:** Optional
> - **Estimated time:** TBD

## TLS Support in TiDB

TiDB Operator 1.1.0 supports Transport Layer Security (TLS) between cluster components and between client and the cluster. This document describes how to enable TLS in TiDB including: 

- TLS between cluster components.
- TLS between MySQL client and TiDB cluster. 

This document is organized as the follows, we first create secrets in Kubenetes and if the certificates is not already in Kubernetes.

Note that this document assumes the following TiDB cluster name and namespace are used. Please make necessary changes to match the cluster name and namespace of the TiDB cluster.

```
$ export cluster_name=basic
$ export namespace=demo
```
## Issue Certificates

We need to generate certificates for inter and intra component communication. In summary, we need to generate the following certificates:

- PD Server certificate for communicating between PD instances.
- TiKV Server certificate for communicating between TiKV instances.
- TiDB Server certificate for communicating between MySQL client and TiDB Cluster. 
- Client certificate for communicating between components and between MySQL client and TiDB Cluster. 

TLS certificates can be issued by multiple methods. Please consult your organization on how you can obtain them. For testing purposes, you can follow the steps in [Issue TLS Certificates Using cfssl](https://hackmd.io/gL-13LYlTA2vPRzzMrA22w?both) to issue and verify certificates.

## TLS Configuration

There are two configurations to enable TLS. To enable TLS between clusters, set `spec.tlsCluster.enabled` to `true`. To enable TLS between MySQL client and TiDB cluster, set `spect.tidb.tlsClient.enabled` to `true`.

To enable mutual TLS, for each compoment, we need to config the `config.security.cert-allowed-cn`. 

**Note that as the configuration `cert-allowed-cn` is stored PD, we can only set one value. This means that when generating the certificates for cluster components, we need to make sure that the CN field is correctly set and is the same between different certificates. This can be verified by using the openssl tool.**

### Enable TLS 

#### Deploy a TiDB Cluster with TLS enabled

One example configuration for TiDB cluster with TLS enabled is as follows:

```
apiVersion: pingcap.com/v1alpha1
kind: TidbCluster
metadata:
  name: basic
spec:
  tlsCluster:
    enabled: true
  version: v4.0.0-rc
  timezone: UTC
  pvReclaimPolicy: Delete
  pd:
    baseImage: pingcap/pd
    replicas: 1
    requests:
      storage: "1Gi"
    config:
      security:
        cert-allowed-cn:
          - TiDB
  tikv:
    baseImage: pingcap/tikv
    replicas: 1
    requests:
      storage: "1Gi"
    config:
      security:
        cert-allowed-cn:
          - TiDB
  tidb:
    baseImage: pingcap/tidb
    replicas: 1
    service:
      type: ClusterIP
    tlsClient:
      enabled: true
    config:
      security:
        cert-allowed-cn:
          - TiDB
```
**Note that the value `spec.pd.config.security.cert-allowed-cn`,`spec.tikv.config.security.cert-allowed-cn` and `spec.tikv.config.security.cert-allowed-cn` needs to be the same and match the CN field in the corresponding certificates.**

Assuming that the yaml file is stored in the following path (TODO: put the configuration file in the tidb operator repo). Please also set the environment variable to point to the path of the configuration. 

Deploy a TiDB cluster with TLS enabled:

```
$ kubectl apply -f ${tidb_cluster_conf} -n ${namespace}
```

#### Enable TLS on an Existing TiDB Cluster

We can edit the TiDB cluster configuration to enable TLS configuraitons using the following command:

```
$ kubectl edit tc ${cluster_name} -n ${namespace} 
```

After the prompt, peforming the following changes:

- Set `spec.tlsCluster.enabled` to `true`.
- Set `spec.pd.config.security.cert-allowed-cn`, `spec.tikv.config.security.cert-allowed-cn` and `spec.tikv.config.security.cert-allowed-cn` to be the same and match the CN field in the corresponding certificates.
- Set `spect.tidb.tlsClient.enabled` to `true`.

After the saving the configuration, if there's no issues, TiDB Operator will perform a rolling restart of the TiDB cluster and apply the configuration changes. 

### Verify Cluster is Successfully Deployed
```
$ kubectl get po -n ${namespace}

NAME                               READY   STATUS    RESTARTS   AGE
basic-discovery-69879c5b85-jvz2f   1/1     Running   0          103s
basic-pd-0                         1/1     Running   0          103s
basic-tidb-0                       1/2     Running   0          22s
basic-tikv-0                       1/1     Running   0          82s
```

```
$ kubectl get svc -n ${namespace}

NAME              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)              AGE
basic-discovery   ClusterIP   10.101.136.222   <none>        10261/TCP            2m33s
basic-pd          ClusterIP   10.100.120.45    <none>        2379/TCP             2m33s
basic-pd-peer     ClusterIP   None             <none>        2380/TCP             2m33s
basic-tidb        ClusterIP   10.102.77.101    <none>        4000/TCP,10080/TCP   72s
basic-tidb-peer   ClusterIP   None             <none>        10080/TCP            72s
basic-tikv-peer   ClusterIP   None             <none>        20160/TCP            2m12s
```

```
$ kubectl logs ${cluster_name}-pd-0 -n ${namespace}

[2020/04/29 00:46:13.045 +00:00] [INFO] [etcd.go:576] ["serving peer traffic"] [address="[::]:2380"]
[2020/04/29 00:46:13.045 +00:00] [INFO] [etcd.go:241] ["now serving peer/client/metrics"] [local-member-id=5d5d47e6e68a829f] [initial-advertise-peer-urls="[https://basic-pd-0.basic-pd-peer.demo.svc:2380]"] [listen-peer-urls="[https://0.0.0.0:2380]"] [advertise-client-urls="[https://basic-pd-0.basic-pd-peer.demo.svc:2379]"] [listen-client-urls="[https://0.0.0.0:2379]"] [listen-metrics-urls="[]"]
```

```
$ kubectl logs ${cluster_name}-tikv-0 -n ${namespace}

[2020/04/29 00:46:42.230 +00:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=addr-resolver]
[2020/04/29 00:46:42.230 +00:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=region-collector-worker]
[2020/04/29 00:46:46.912 +00:00] [INFO] [future.rs:136] ["starting working thread"] [worker=gc-worker]
[2020/04/29 00:46:46.912 +00:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=lock-collector]
[2020/04/29 00:46:46.945 +00:00] [INFO] [mod.rs:181] ["Storage started."]
[2020/04/29 00:46:46.948 +00:00] [INFO] [mod.rs:335] ["starting working thread"] [worker=split-check]
[2020/04/29 00:46:46.989 +00:00] [INFO] [node.rs:145] ["trying to bootstrap cluster"] [region="id: 2 region_epoch { conf_ver: 1 version: 1 } peers { id: 3 store_id: 1 }"] [store_id=1]
[2020/04/29 00:46:46.994 +00:00] [INFO] [node.rs:281] ["bootstrap cluster ok"] [cluster_id=6820928503865373278]
[2020/04/29 00:46:46.994 +00:00] [INFO] [node.rs:348] ["start raft store thread"] [store_id=1]
[2020/04/29 00:46:46.994 +00:00] [INFO] [peer.rs:148] ["create peer"] [peer_id=3] [region_id=2]
```

```
$ kubectl logs ${cluster_name}-tidb-0 -c tidb -n ${namespace} 

[2020/04/29 00:47:31.029 +00:00] [INFO] [server.go:235] ["server is running MySQL protocol"] [addr=0.0.0.0:4000]
[2020/04/29 00:47:31.030 +00:00] [INFO] [http_status.go:80] ["for status and metrics report"] ["listening on addr"=0.0.0.0:10080]

```

### Verify Security Configuraiton
There are a couple of ways to verify security is correctly configured.

#### Using pd-ctl

```
$ pd-ctl --cacert=${certificate_path}/ca.pem --cert=${certificate_path}/client.pem --key=${certificate_path}/client-key.pem -u https://${cluster_name}-pd.${namespace}.svc:2379 member
```

**Note that you may need to do a port forwarding to access pd.**

The output is:

```
{
  "header": {
    "cluster_id": 6820928503865373278
  },
  "members": [
    {
      "name": "basic-pd-0",
      "member_id": 6727612475397276319,
      "peer_urls": [
        "https://basic-pd-0.basic-pd-peer.demo.svc:2380"
      ],
      "client_urls": [
        "https://basic-pd-0.basic-pd-peer.demo.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v4.0.0-rc"
    }
  ],
  "leader": {
    "name": "basic-pd-0",
    "member_id": 6727612475397276319,
    "peer_urls": [
      "https://basic-pd-0.basic-pd-peer.demo.svc:2380"
    ],
    "client_urls": [
      "https://basic-pd-0.basic-pd-peer.demo.svc:2379"
    ]
  },
  "etcd_leader": {
    "name": "basic-pd-0",
    "member_id": 6727612475397276319,
    "peer_urls": [
      "https://basic-pd-0.basic-pd-peer.demo.svc:2380"
    ],
    "client_urls": [
      "https://basic-pd-0.basic-pd-peer.demo.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v4.0.0-rc"
  }
}
```

Note that `https` is used for `client_urls` and `peer_urls`.

#### Using MySQL Client
```
$ mysql -u root -P 4000 -h ${tidb_host} --ssl-cert=${certificate_path}/client.pem --ssl-key=${certificate_path}/client-key.pem --ssl-ca=${certificate_path}/ca.pem --ssl-mode=VERIFY_IDENTITY
```

```
mysql> \s
--------------
mysql  Ver 8.0.19 for osx10.15 on x86_64 (Homebrew)

Connection id:		29
Current database:
Current user:		root@127.0.0.1
SSL:			Cipher in use is TLS_AES_256_GCM_SHA384
Current pager:		stdout
Using outfile:		''
Using delimiter:	;
Server version:		5.7.25-TiDB-v4.0.0-rc TiDB Server (Apache License 2.0), MySQL 5.7 compatible
Protocol version:	10
Connection:		127.0.0.1 via TCP/IP
Server characterset:	utf8mb4
Db     characterset:	utf8mb4
Client characterset:	utf8mb4
Conn.  characterset:	utf8mb4
TCP port:		4000
Binary data as:		Hexadecimal
--------------
```

```
mysql> SHOW STATUS LIKE "%Ssl%";
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Variable_name   | Value                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Ssl_cipher      | TLS_AES_256_GCM_SHA384                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
| Ssl_cipher_list | RC4-SHA:DES-CBC3-SHA:AES128-SHA:AES256-SHA:AES128-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-ECDSA-RC4-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-RC4-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-RSA-AES128-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-CHACHA20-POLY1305:TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256: |
| Ssl_verify_mode | 5                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
| Ssl_version     | TLSv1.3                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
+-----------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
4 rows in set (0.01 sec)
```