# Lorenzo - Centrifuge
- nix & nixos
- k8s operators implementation
- implementers on k8s api
- gke

## Centrifuge roadmap
### Setup Dev VPN
- In Centrifuge Dev project
- Admin Access to vpn

### Monitoring + Logging
#### Monitoring
- Current grafana url: http://34.107.88.100/d/twYeaxRMz/my-polkadot-metrics?orgId=1&refresh=10s
- u: admin
- p: grafana
- Deploy Grafana + Prometheus cleanly
- https://github.com/BenWhiteJam/node_deployer
- Ensure all pods/nodes are indexed
- Use latest grafana dashboard for Polkadot
- tweaking for network name
- Substrate visualize metrics: https://substrate.dev/docs/en/tutorials/visualize-node-metrics/
- Extra prometheus config substrate jobname/rule:
- https://github.com/BenWhiteJam/node_deployer/blob/main/kube/monitoring/manifest/prometheus-configmap.yaml

    - job_name: "substrate_node"
      scrape_interval: 5s
      scheme: http
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - separator: ;
        regex: __meta_kubernetes_pod_label_(.+)
        replacement: $1
        action: labelmap
      - source_labels: [__address__]
        separator: ;
        regex: ([^:]+)(?::\d+)?
        target_label: __address__
        replacement: $1:9615
        action: replace
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_k8s_app]
        separator: ;
        regex: monitoring;prometheus
        replacement: $1
        action: keep
      - source_labels: [__meta_kubernetes_namespace]
        separator: ;
        regex: (.*)
        target_label: namespace
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        separator: ;
        regex: (.*)
        target_label: pod
        replacement: $1
        action: replace
      - source_labels: [__meta_kubernetes_pod_name]
        separator: ;
        regex: (.*)
        target_label: instance
        replacement: $1
        action: replace

#### Logging
- Configure loki promtail to index from systemd:
- Search journald level
- Add sample arbitrary label - Text: "Parachain"

#### Alerting
- Push alerts to some telegram channel
- Initial metrics and rules:
- host mem
- host cpu
- host disk
- validator (n_peers, block_number hasn't changed in n mintues)
- error rate increased by n%

#### Documentation
https://docs.google.com/document/d/1kj9zx_vbtzJaUQqC29JZH37beFkjMKzHF1Nnuixk6uk/edit#heading=h.hfgs75y3mg9x

## Next
- infra more reboust + security
- eng processes
- access management