---
# System prepended metadata

title: Bulats SSL recovery

---

# Bulatsa SSL recovery

---
**QUESTION** tls disable on each node or one node "mode"?
**INFO**: We don't need a full cluster to operate

**NOTE**: Current cert is in `/opt/vault/tls/tsl.crt`

**QUESTION**: Do we issue certs manually or via `consul_recovery.yml`?

---

1. Restore Vault
    On one Node of Vault:
    - set `tls_disable = 1` to start Vault without cert
    - issue Vault certs
    - Copy new Vault certs on each node
    - remove `tls_disable = 1`
    - restart vault `systemct vault restart`
    - unseal the Vault
    - Now we have a running vault?
2. Run `consul_recovery.yml`

3. PRAY

---

## 27.06.23 Call with Technologica Simeon Manolov

```bash=
ansible-playbook -e "vault_token=$VAULT_TOKEN" -i \
inventory playbooks/consul_recover.yml \
-e "vault_addr=http://172.16.37.31:8200" \
-e "init_vault_addr=http://172.16.37.31:8200" \
-e "domain_name={{ tenant }}{{ env }}.consul"
```

Playbook tries to unseal the Vault, but we are running Vault in `http` mode - check if we can override it 

```yaml=
- name: unseal vault
  shell: |
    vault operator unseal {{ item.0.stdout }}
  environment:
    VAULT_ADDR: "https://127.0.0.1:8200"
    VAULT_SKIP_VERIFY: true
  #when: sealed_status.stdout|bool
  loop: "{{ unseal_keys.results|product()}}"
  loop_control:
    pause: 5
  register: result
  until: result is succeeded
  retries: 5
  delay: 3
```

### What we did:

1. Issue new cert
```bash=
vault write /issue/{{ cert_role }} common_name="{{ vault_domain_name }}" alt_names="{{ lookup('template', './vault_hosts_fqdn.j2') }}" ttl="8760h" -format=json
```
1.1 `dron-tst-ptfm01.node.bulatsatst.consul,dron-tst-ptfm02.node.bulatsatst.consul,dron-tst-ptfm03.node.bulatsatst.consul` - to DNS/Email Subject Alt names
1.2 Issue certificate in `nomad-cluster` role
1.3 fix the role 

```shell=
vault write bulatsatst_consul/roles/bultsatstconsulcerts allowed_domains="bulatsatst.consul,service.consul,node.consul" allow_subdomains=true max_ttl=8760h`
```

2. Copy certs to all machines (in /opt/vault/tls)
3. Start the vault without `tls_disable = 1`
4. Check the raft `vault operator raft list-peers`
5. Rerun the recovery playbook



### Things to check

> Double restart of vault


`NO_PROXY: "*"` bulatsa uses proxy and we get an error from Sophos so we changed (lines 9 & 19):

```yaml=
- name: create vault policy for certificate endpoints
  ansible.builtin.shell: > 
    vault policy write cluster-tls {{ consul_template_data_dir }}/vault_cluster_policy.hcl
  run_once: true
  environment:
    VAULT_ADDR: "{{ init_vault_addr }}"
    VAULT_TOKEN: "{{ vault_token }}"
    VAULT_SKIP_VERIFY: true
    NO_PROXY: "*"
  become: true

- name: create consul-template-token
  ansible.builtin.shell: >
    vault token create -policy="cluster-tls" -period=24h -orphan -format=json
  environment:
    VAULT_ADDR: "{{ init_vault_addr }}"
    VAULT_TOKEN: "{{ vault_token }}"
    VAULT_SKIP_VERIFY: true
    NO_PROXY: "*"
  become: true
  register: consul_token_creation
```
**TODO**: check if we can add this only for bulatsa env - or they need to disable the proxy before they run ansible 
**FIX**: They disable the proxy with modification of `/etc/environment`


**TODO**: Update manual and add steps for the update, so we avoid situations when the client runs old containers (Technologica, I'm looking at you)