**Proposer team:**
**Contact Info:**
**- E-mail address:** [e-mail]
**- Tw handle:** [@tugytur](https://twitter.com/TugyTur)
**- Kappa Sigma Mu Society Member address:** Gt8ferkwFEX9jLbxpuhLYqKdNzP9KtjZcGYVN2mMbuCwhAP
**- Element handle:** @tugytur:matrix.org

---

# Context of the proposal

My proposal is that we as a society run validators with Kappa Sigma Mu Identity to generate an additional source of funding for the pot and mark our presence.

## About the team

We are experienced in running validators since NPoS was enabled. Last year we founded our own company ([Amforc AG](https://amforc.com)) which provides dedicated & white label staking solutions. The company is domiciled and based in Switzerland.

I will go into detail only about the people relevant to staking.

Before getting into crypto I was working as a Security Engineer (blue team), then I wanted to get into blockchain space and started working for a crypto company. There, I was responsible for security, network, infrastructure and PoS systems until last year when I left for my own journey.

Additionally, we have one more employee who is hired on an hourly rate to mainly help to cover on-call and help with projects. He is a former Linux administrator who started studying again and currently is finishing his Bachelor in IT security.

## Our Setup

We run our validators in a hybrid-cloud setup as IaC. We use multiple VPS providers. This allows us not to be dependent on one provider or geographical location. We run our validators for the most time “immutable”, which means that we always provision a new machine instead of patching a validator if the reason is not security related or urgent.

With our setup we have the possibility to bootstrap a Kusama validator in under 5 minutes. For this we use our own snapshot solution, which we also made publicly available to anyone who wants to use it (https://parasnaps.io/).

Additionally, we have our own hypervisor which consists of 3 servers (192 cores, ~1TB RAM and 10TB NVME storage) which has a shared ceph storage that allows us to do VM live migrations inside the hypervisor nodes. This hypervisor is used for most of the internal infrastructure systems (monitoring/log management/security related systems), play-/testground, and sometimes for our “Thousand Validators Program” validators.

### Some security/monitoring related information

Session keys are always generated on the newly bootstrapped validator and are stored in a secrets management tool for backup purposes. Access to servers is role based and managed via a jumphost with time limited signed ssh certificates.

We monitor and log System-, Validator- and on-chain-metrics.

Alerts are handled according to urgency. Critical alerts are sent to the person on-call and if not acknowledged in the defined period it will be escalated to the backup person on-call.

We have our setup completely automated except for the session keys. That part is semi-automated and requires multiple checks, confirmations and following of internal procedures and manual execution to prevent double signing by automation or mistake. This only comes into play in a scenario where there are issues, and an offline event would be inevitable.

## Historical metrics
Since running validators for Dotsama ecosystem I have never had an offline or slashing event and our validators are performing above average.

I cannot prove this without breaking NDAs or disclosing customers. But we are members of the Kusama 1KV program for 8 months and have 2 validators that are in the active set for most of the time. We are earning ~20-40% above global average points per era and have no faults.

- [Amforc/1 (DVUNoinHdSNfismcrFaBwdJfysxc7A48QkNvTDnTSPXPw3q)](https://thousand-validators.kusama.network/#/leaderboard/DVUNoinHdSNfismcrFaBwdJfysxc7A48QkNvTDnTSPXPw3q)
- [Amforc/2 (DpLatoXXBiSAPooF17bzUZGo7huNB7USfRqd2SgL6RBy2zr)](https://thousand-validators.kusama.network/#/leaderboard/DpLatoXXBiSAPooF17bzUZGo7huNB7USfRqd2SgL6RBy2zr)

Two months ago, we joined the Polkadot 1KV program as well.
- Amforc/1 (1y6CPLgccsysCEii3M7jQF834GZsz9A3HMcZz3w7RjGPpBL)

We run our public validators on Kusama and Polkadot with 3% commission as for Kusama this was enforced with [Referendum 176](https://kusama.polkassembly.io/referendum/176).

We handle the reward claims/payouts for our customers daily. People delegating don’t have to worry about reward claims.

## Kappa Sigma Mu Setup
If we want to run one validator, I can run it free of charge on our hypervisor.

But if we want more then I would prefer if the society would cover the validator costs. We as a company are fine by providing our labour for free. Depending on how we decide to cover hardware costs there would be a scenario where we would be making a little surplus. Please see below. I will also list normal costs and work involved to put it into perspective for those not familiar with validators.

For the option with 2+ validators I would recommend a setup where we rent hardware at Hetzner, at the cost of centralization and only two locations but leverage Amforc’s existing setup with multiple VPS providers for fail-over and maintenance purposes to have the lowest possible expenses. There will be a small buffer for costs to cover expenses generated while being on a VPS for short time periods. In the case of hardware or hosting provider issues Amforc can bootstrap a validator in under 5 minutes at a new location.

## Budget & Labour

### General Cost of Running Validators example (labour not included)

| Description | Cost USD (month) |
| -------- | -------- |
| Base Infrastructure to support Validators (VMs, Hardware, Product Licences) | 3000-5000 |
| 1 Validator (VPS) Strongly varies depending on the network requirements, usage and our selected VPS | 400-1500 |

### Kappa Sigma Mu Validators

| Description | Cost USD (month) |
| -------- | -------- |
| Hardware per Validator | 68 |
| Buffer for VPS costs (maintenance & failover) | 37 |
| Setup costs | 0 |

### Labor in Hours (weekly) with no updates to maintain validators

| Description | Hours (weekly) |
| -------- | -------- |
| Daily checks into metric and log dashboards to make sure that everything is running smoothly, and no problem is building up | 7-9 |
| Improvements, adjustments and testing | 3-30 |

### Labour with updates and releases
It’s nearly impossible to put it into hours when there are either Linux related updates (critical or non-critical) or Polkadot releases because -> **Expect Chaos**.

Instead, we will walk you through a Polkadot release and how we make sure that everything goes smoothly.

When there is a new update, we spin up a new validator (will not become active) and make sure that it’s starting up smoothly with a snapshot from the previous release.

Next, we update our normal nodes and check again if everything goes to plan.

Next, we update one of our 1KV Kusama validators and if it produces blocks and works as a para validator, we do the second one.

Afterwards the 1KV Polkadot validator.
(It’s a 1KV requirement to update within 24h of a normal release, else we would do only one first).

Then we let it run for at least 24 hours and if we detected no issues, we would update Kappa Sigma Mu validators.

Meantime if there are any kind of changes to rocksdb or paritydb we start a sync from scratch with the latest release and check if there are any performance improvements or problems. If everything is well with the new sync, we make it available on parasnaps and use that to provision new validators.

## Coverage of costs
One way of covering the hardware costs would be that the 3% commission would stay with us, which means we would make ~1 KSM surplus a month per validator at the current USD prices and the nomination rewards would go to the society address.

Second option is that the validator rewards go to the society address as well and we handle the costs by creating a quarterly incquery and asking it to be covered by the treasury funds (See table [Kappa Sigma Mu Validators](#Kappa-Sigma-Mu-Validators)” above).

If there are any further questions, please reach out to me or if your tokens desire new validators we are here to support you as well.