# Setting up a gaia validator using tmkms + yubikey
----
High Level Todo

DEADLINE: Thursday 27th of October

✅ Figure out how to become a validator on mainnet (probobly the same as testnet)
✅ Get node running on testnet
✅ Get node using YubiHSM 2 on testnet
✅ Promote to Validator on Test Net
✅ Get validator proposing blocks
✅ Reset HSM

Reset HSM

Re-Provision Keys

Confirm blocks are being signed

✅ (Re) Deploy Production/Staging Env
✅ Re-Deploy Clean Test Net server
✅ Re-Deploy Clean main net server
✅ cloned existing node to gaia-test.stg.hypha.coop (51.222.32.220) kvmid 209
✅ deployed 2 new machines
gaia-1.prod.hypha.coop (51.222.32.210) kvmid 102
gaia-1.stg.hypha.coop (51.222.32.212) kvmid 202
✅ Install Gaia on STG and Prod @elon
✅ Configure data folder to NVME drive @elon
✅ Sync Theta and Mainnet @elon
Milestone - Above done by Tuesday Monday
✅ Move Validator role to new Theda Server
✅ Enable TMKMS on Theta
✅ Move HSM
✅ Have theda start validating blocks
✅ Generate production keys
✅ Generate keys on Air gapped computer
✅ Provision Self Delegating keys on a Ledger
✅ Create paper backups of
✅ Self Delegation Keys
✅ Signing Keys
✅ HSM Keys
✅ Register new credentials on testnet
✅ testnet validator working with newly provisioned HSM keys
Milestone - Above done by Wednsday Tuesday
Verify offline secrets are available
✅ Move Validator over to Mainnet
✅ Move HSM to other computer
✅ Spin up TMKMS
✅ Register Validator
✅ Start signing blocks

Milestone - Above done by Thursday Wednesday

⬜ Replace theda TMKMS with SoftHSM and TEST KEYS
⬜ Figure out how to filter out oneshot in prometheus
✅ Get panic setup (maybe in an ansible role? or a work log)

✅ Add to prometheus
✅ node exporter
✅ gaiad metrics
✅ promtail

Check offline backups

✅priv_validator
✅ Encrypted QR
✅ Encrypted Text
✅ Printed Clear Text

HSM

✅ 24 Words
✅ Ledger
✅ 24 Words

Work Log at : https://hackmd.io/rVHxcNMNTSKmVotcaQ-lZw

## Meeting 2022-22-10
Meeting room in case you get kicked out
Join Zoom Meeting
https://us02web.zoom.us/j/3101837052?pwd=WFlUOVk4QUNiYVR2bUJiQ0duTEo1QT09

Meeting ID: 310 183 7052
Passcode: c1WFzz
One tap mobile
[phone],,3101837052#,,,,*438272# Canada
[phone],,3101837052#,,,,*438272# Canada

----

References:
- https://docs.osmosis.zone/osmosis-core/keys/tmkms
- https://github.com/iqlusioninc/tmkms/blob/main/README.yubihsm.md
- https://hub.cosmos.network/main/validators/kms/kms_ledger.html
- https://github.com/hyphacoop/cosmos-ansible/blob/feature/163-kms-local/roles/kms/tasks/main.yml
- https://github.com/hyphacoop/cosmos-configurations-private/wiki/Create-Validator

### Install tmkms
* Use the [ansible role](https://github.com/hyphacoop/cosmos-ansible/blob/feature/163-kms-local/roles/kms/tasks/main.yml) that Yurko build for installing
* Creates kms user
* Generates tmks binary
* Creates secret file (this is used for communication with the YubiKey)
* Creates service
* Starts up service

### Setup YubiKey
1. Run setup
`tmkms yubihsm setup`

This generates 4 authentication keys:
- admin (authentication key 0x0001): full access to all HSM capabilities
- operator (authentication key 0x0002): ability to generate new signing keys, export/import encrypted backups of keys, and view the audit log
- auditor (authentication key 0x0003): ability to view and consume the audit log
- validator (authentication key 0x0004): ability to generate signatures using signing keys loaded within the device

**Note**
- none of these keys the actual validator signing keys.
- these keys are limited to access various other keys that can be stored on the YubiKey

## Generate signing key for Hypha validator
`tmkms yubihsm keys generate 1 -p cosmosvalconspub -b hypha-1.enc`

### Setup tmkms

# Tendermint KMS configuration file

## Chain Configuration

### Cosmos Hub Network

[[chain]]
id = "theta-testnet-001"
# ??? what are these values
key_format = { type = "bech32", account_key_prefix = "cosmospub", consensus_key_prefix = "cosmosvalconspub" }
state_file = "/home/kms/.tmkms/config/state/priv_validator_state.json"

## Signing Provider Configuration

### Software-based Signer Configuration

### KMS Signer Configuration
[[providers.yubihsm]]
adapter = { type = "usb" }
#auth = { key = 4, password = "kms-validator-password-1md4czuye5zq56cmz35rkee0dhsmgm602" }
auth = { key = 1, password = "password" }
keys = [

    { key = 1, type = "consensus", chain_ids = ["theta-testnet-001"] },
]
## Validator Configuration

[[validator]]
chain_id = "theta-testnet-001"
addr = "tcp://127.0.0.1:26659"
secret_key = "/home/kms/.tmkms/config/secrets/kms-identity.key"
protocol_version = "v0.34"
reconnect = true

### create new key
gaiad keys add new-valildator

request from testnet faucet to address `cosmos13mddt6wv2l5lsk9f7d83vtuv580lsqvdhhe5td`

cosmosvalconspub1zcjduepqm5680numd9pvz20vp0p6yjvx0nls82ltgx8f8vmd3tdpsylgqg6st70n5k

### parsing keys

gaiad keys parse cosmosvalconspub1zcjduepqm5680numd9pvz20vp0p6yjvx0nls82ltgx8f8vmd3tdpsylgqg6st70n5k
human: cosmosvalconspub
bytes: 1624DE6420DD3477CF9B6942C129EC0BC3A249867CFF03ABEB418E93B36D8ADA1813E80235

import base64
b= bytes.fromhex("1624DE6420DD3477CF9B6942C129EC0BC3A249867CFF03ABEB418E93B36D8ADA1813E80235")
base64EncodedStr = base64.b64encode(b)
print(base64EncodedStr)

`FiTeZCDdNHfPm2lCwSnsC8OiSYZ8/wOr60GOk7NtitoYE+gCNQ==`

`d1pQSuSH52WmU7gqiFgI0kuIgU8=`

Udit's local version

gaiad tendermint show-validator
{"@type":"/cosmos.crypto.ed25519.PubKey","key":"9QIo4gEbSEBoOapeO2fasRqCxu2TBmZ33upUqNd+1Ks="}

Hex Version of key: f50228e2011b48406839aa5e3b67dab11a82c6ed93066677deea54a8d77ed4ab

gaiad tendermint show-address
cosmosvalcons1wad9qjhyslnktfjnhq4gskqg6f9c3q20cq5e3k

gaiad keys parse cosmosvalcons1wad9qjhyslnktfjnhq4gskqg6f9c3q20cq5e3k
human: cosmosvalcons
bytes: 775A504AE487E765A653B82A885808D24B88814F

gaiad keys parse 775A504AE487E765A653B82A885808D24B88814F
formats:
- cosmos1wad9qjhyslnktfjnhq4gskqg6f9c3q20f8ns3y
- cosmospub1wad9qjhyslnktfjnhq4gskqg6f9c3q20kplrgp
- cosmosvaloper1wad9qjhyslnktfjnhq4gskqg6f9c3q20vn89ah
- cosmosvaloperpub1wad9qjhyslnktfjnhq4gskqg6f9c3q207s7xlj
- cosmosvalcons1wad9qjhyslnktfjnhq4gskqg6f9c3q20cq5e3k
- cosmosvalconspub1wad9qjhyslnktfjnhq4gskqg6f9c3q20059w4r

### Importing key to KMS
setup key
tmkms use OP key in config
copy validator_key.json to kms home
tmkms import json
tmkms key list to verify

tmkms switch to validator key in config
tmkms test 1 to see if signing works
use gaiad to save show validator and show address before deleting files

add tokens to account and verify

make validator with gaiad tx staking create-validator

## Re-Provisioning Work 2022-23-10

- Cloned testing gaia over to vmid 209.
- Deleted old one
- Provisioned `gaia-1.prod.hypha.coop` and `gaia-2.prod.hypha.coop`
- Updted DNS
- Ansible for deployment: https://github.com/hyphacoop/ansible-provision-vm/blob/main/inventory.yml
- Installed standard ansibles on both servers:

Inventory

all:
  vars:
    # Sensible defaults
    ansible_user: sysadmin
    ansible_python_interpreter: /usr/bin/python3

    # Set root and sysadmin passwords on all servers
    root_password: "{{ secret_root_password }}"
    sysadmin_password: "{{ secret_root_password }}"

    # Keys to add to all servers
    ssh_user_keys:
      - name: yurko
        key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtv4I2W8TlVsWKKHwASF78cGrn06BJi9dGQQQksW0Ke9ED/KJG2pvG4GXnh57/AOuJtUaS3gXBEj/knE3t/qtvO2cUg1EufnkItzNLdY1B0Hwm98I8O7JdhE>
      - name: "elon"
        key: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCt7N4cJCFFmetGyDo2giMdPlHOBOdEJt8DnidBTMxevlGi02YVLwdYwbPtdoSHJKRlY4boV80sPOhMI3Im2TXNXHQIhO2LWKdSUTdfFU5ul+ySeZlUm3eDpt>


    monitor_server: monitor.prod.hypha.coop
    loki_url: "{{monitor_server}}:443/loki/api/v1/push"
    loki_protocol: "https"
    loki_username: loki
    loki_password: "{{secret_loki_password}}"
#    cadvisor_exporter_port: 9110

  children:
    gaiad:
      vars:
        monitor_job: "production"
        firewall_ports:
          - "ssh":
            port: 22
          - "tendermint":
            port: 26656
#          - "nodeexporter":
#            port: 9100
#            source: "{{ monitor_server }}"
#            sourceIPv6: "2604:a880:cad:d0::f49:4001"
      hosts:
        gaia-1.prod.hypha.coop:
        gaia-1.stg.hypha.coop:

Inventory:

- hosts: gaiad
  become: true
  gather_facts: true
  roles:
    - hypha.common.provision_ssh_keys
    - hypha.common.harden_ssh
    - hypha.common.setup
    - hypha.common.unattended_upgrades
#    - geerlingguy.docker
    - hypha.common.firewall
#    - hypha.common.node_exporter
#    - hypha.common.prometheus_exporter_addnode
#    - hypha.common.prometheus_exporter_docker
#    - hypha.common.promtail

## Co-Work with elon 2022-23-10

### Factory Rest and Re-Provision HSM
- Factory Rest HSM
- Updated TMKMS to use default password
- Run Setup

kms@gaia-test:~$ tmkms yubihsm setup -c .tmkms/config/tmkms.toml
This process will *ERASE* the configured YubiHSM2 and reinitialize it:

- YubiHSM serial: 0020780003

Authentication keys with the following IDs and passwords will be created:

- key 0x0001: admin:

    mouse spider donkey media nest casino inflict shaft feed differ father away
    glass husband gap galaxy choose patrol romance inject rhythm useless dawn satoshi

- authkey 0x0002 [operator]:  kms-operator-password-1l5mzau8pnxeheydw27y7x86z0q6u3c6p
- authkey 0x0003 [auditor]:   kms-auditor-password-1r7ssmvdnkgeek4aa3fetn5jccsypukxr
- authkey 0x0004 [validator]: kms-validator-password-1jmu34jezw9fks0p27sk7vsnqvvstmuq0
- wrapkey 0x0001 [primary]:   f44a30b4be80a9a14df0a2bfc78e04cd19fa94cc074e2c6c332295a0a21bb00a

*** Are you SURE you want erase and reinitialize this HSM? (y/N): y
2022-10-23T16:55:56.258697Z  WARN yubihsm::client: factory resetting HSM device! all data will be lost!
2022-10-23T16:55:57.287690Z  INFO yubihsm::client: waiting for device reset to complete
2022-10-23T16:56:02.735374Z  INFO yubihsm::setup: installed temporary setup authentication key into slot 65534
2022-10-23T16:56:02.755526Z  WARN yubihsm::setup: deleting default authentication key from slot 1
2022-10-23T16:56:02.782396Z  INFO yubihsm::setup::profile: installing role: admin:2022-10-23T16:55:44Z
2022-10-23T16:56:02.806398Z  INFO yubihsm::setup::profile: installing role: operator:2022-10-23T16:55:44Z
2022-10-23T16:56:02.826939Z  INFO yubihsm::setup::profile: installing role: auditor:2022-10-23T16:55:44Z
2022-10-23T16:56:02.847402Z  INFO yubihsm::setup::profile: installing role: validator:2022-10-23T16:55:44Z
2022-10-23T16:56:02.867951Z  INFO yubihsm::setup::profile: installing wrap key: primary:2022-10-23T16:55:44Z
2022-10-23T16:56:02.891375Z  INFO yubihsm::setup::profile: storing provisioning report in opaque object 0xfffe
2022-10-23T16:56:02.917028Z  WARN yubihsm::setup: deleting temporary setup authentication key from slot 65534
     Success reinitialized YubiHSM (serial: 0020780003)
kms@gaia-test:~$

Imported `priv_validator_key.json`

`tmkms yubihsm keys import -t json -i 1 priv_validator_key.json -c .tmkms/config/tmkms.toml`

Checked Key
`tmkms yubihsm keys list -c .tmkms/config/tmkms.toml`

### Moved HSM to target server:
On target srever
edit tmkms.conf to update key
Update gaiad confg:

# Comment out these two lines
# Path to the JSON file containing the private key to use as a validator in the consensus protocol
# priv_validator_key_file = "config/priv_validator_key.json"

# Path to the JSON file containing the last sign state of a validator
# priv_validator_state_file = "data/priv_validator_state.json"

# Change this line (or add it if not exists)
priv_validator_laddr = "tcp://127.0.0.1:26659"

UPDATE ANSIBLE
add to tmkms template

keys = [

    { key = 1, type = "consensus", chain_ids = ["theta-testnet-001"] },
]

### Possible airgap HSM provisioning
Require `gaiad`,`tmkms` binaries and `tmkms.yml` file

./gaiad init Solidarity --home .
mv ~/.gaia/config/priv_validator_key.json .
./tmkms yubihsm setup 
# update tmkms with operator key
tmkms yubihsm keys import -t json -i 1 priv_validator_key.json

# Backup
- priv_validator_key.json
- yubikey output

## Produciton key Generatiosn

mkdir home
mount -t tmpfs tmpfs home
cd home
cp ../gaia .
cp ../tmkms .
cp ../tmkms.conf .

./gaiad init Solidarity --home .
mv config/priv_validator_key.json .
./tmkms yubihsm setup 
# Convert output to PDF then print
tmkms yubihsm keys import -t json -i 1 priv_validator_key.json

#Encrypt priv_validator_key
openssl aes-256-cbc -a -salt -pbkdf2 -pass pass:<REDACTED> -in priv_validator_key.json -out  priv_validator_key.enc    
# Decrypt test
openssl aes-256-cbc -d -a  -salt -pbkdf2 -in priv_validator_key.enc

# convert priv_validator_key.enc to pdf

# print both PDFs

## Production key on Theda test

./gaiad tx staking create-validator \
--amount 1000000uatom \
--pubkey '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"qRzITWr/nQ+J31ODy8fZDD/SvUorYZaH5C786x5VQ9k="}' \
--moniker "Validator5000" \
--chain-id "theta-testnet-001" \
--commission-rate "0.10" \
--commission-max-rate "0.20" \
--commission-max-change-rate "0.01" \
--min-self-delegation "1000000" \
--gas "auto" --fees 100uatom \
--from validator_hypha \
--sign-mode amino-json \
--node http://sentry-01.theta-testnet.polypore.xyz:26657 \
--home .

{
    "body": {
        "messages": [
            {
                "@type": "/cosmos.staking.v1beta1.MsgCreateValidator",
                "description": {
                    "moniker": "Validator5000",
                    "identity": "",
                    "website": "",
                    "security_contact": "",
                    "details": ""
                },
                "commission": {
                    "rate": "0.100000000000000000",
                    "max_rate": "0.200000000000000000",
                    "max_change_rate": "0.010000000000000000"
                },
                "min_self_delegation": "1000000",
                "delegator_address": "cosmos1mt6mf0ny6cstytjz00kz3h7cyyrh9m5sllef6r",
                "validator_address": "cosmosvaloper1mt6mf0ny6cstytjz00kz3h7cyyrh9m5s6tduks",
                "pubkey": {
                    "@type": "/cosmos.crypto.ed25519.PubKey",
                    "key": "qRzITWr/nQ+J31ODy8fZDD/SvUorYZaH5C786x5VQ9k="
                },
                "value": {
                    "denom": "uatom",
                    "amount": "1000000"
                }
            }
        ],
        "memo": "",
        "timeout_height": "0",
        "extension_options": [],
        "non_critical_extension_options": []
    },
    "auth_info": {
        "signer_infos": [],
        "fee": {
            "amount": [
                {
                    "denom": "uatom",
                    "amount": "100"
                }
            ],
            "gas_limit": "173697",
            "payer": "",
            "granter": ""
        }
    },
    "signatures": []
}

## Production key on mainnet

#Add hypha validator ledger to daiaga
./gaiad keys add hypha_validator --ledger --home .
#Create validator
./gaiad tx staking create-validator \
--amount 1000000uatom \
--pubkey '{"@type":"/cosmos.crypto.ed25519.PubKey","key":"qRzITWr/nQ+J31ODy8fZDD/SvUorYZaH5C786x5VQ9k="}' \
--moniker " hypha-coo �" \
--chain-id "cosmoshub-4" \
--commission-rate "0.08" \
--commission-max-rate "0.20" \
--commission-max-change-rate "0.01" \
--min-self-delegation "1000000" \
--gas "auto" --fees 100uatom \
--from validator_hypha \
--sign-mode amino-json \
--node http://one.cosmos-mainnet.polypore.xyz:26657 \
--home .