# [kubernetes] fix incorrect endpoint in cluster
## Check current endpoint

$ kubectl get endpoint | grep kubernetes
kubernetes         10.0.0.20:16443,10.0.0.21:16443   15d

The "10.0.0.20" is unreachable now, so we need to remove it manually.
## root cause
- We restore cluster from a backup file which contains old endpoint that we don't use now
- Query from dqlite(cluster metadata database, like etcd), we can find the old server still has **masterleases**

# login dqlite terminal
$ /snap/microk8s/current/bin/dqlite -s file:///var/snap/microk8s/current/var/kubernetes/backend/cluster.yaml -c /var/snap/microk8s/current/var/kubernetes/backend/cluster.crt -k /var/snap/microk8s/current/var/kubernetes/backend/cluster.key -f json k8s

dqlite> select * from kine where name is '/registry/masterleases/10.0.0.20'
111555|/registry/masterleases/10.0.0.20|1|0|0|0|0|[107 56 115 0 10 15 10 22 333 49 18 9 69 110 100 112 111 105 110 116 115 18 38 10 10 10 0 18 0 26 0 34 0 42 0 50 0 56 13 66 0 122 0 18 16 10 14 10 10 49 48 46 48 46 48 46 50 48 51 26 0 26 0 34 0]|[]

## dqlite table

dqlite> SELECT sql FROM sqlite_master;
CREATE TABLE kine
                        (
                                id INTEGER primary key autoincrement,
                                name INTEGER,
                                created INTEGER,
                                deleted INTEGER,
                                create_revision INTEGER,
                                prev_revision INTEGER,
                                lease INTEGER,
                                value BLOB,
                                old_value BLOB
                        )
CREATE TABLE sqlite_sequence(name,seq)
CREATE INDEX kine_name_index ON kine (name)
CREATE UNIQUE INDEX kine_name_prev_revision_uindex ON kine (name, prev_revision)

## solution
Manully update old server's masterlease to deleted

dqlite> UPDATE kine SET deleted=1 WHERE name='/registry/masterleases/10.0.0.20'

Check again endpoint in cluster, and find out it successfully removed.

$ kubectl get endpoint | grep kubernetes
kubernetes         10.0.0.21:16443   15d