# infrastructure manager / devops engineer

### best practice

our operations management utilises what we deem best practice. that is, we either have implemented or are striving to achieve the following in everything we implement and manage:

- **provisioning** of new (or re-purposing of existing) infrastructure is either **automated or documented** transparently.
- **bootstrapping** and **configuration** of infrastructure is **automated** transparently.
- **routine maintenance** of infrastructure is **automated** transparently.
- **extraordinary maintenance** of infrastructure is **automated or documented** transparently.
- **output/logs** from infrastructure are **aggregated** and available transparently for analysis.
- **issues** with infrastructure are transparently **visible**, obvious and trigger alerting.

### technologies and platforms

to realise these objectives, some of the technologies and tools we utilise include:
- **ansible** for instance configuration and maintenance.
- **cloud-config** for instance bootstrapping.
- custom orchestration: **[rubberneck](https://github.com/Manta-Network/rubberneck)** utilises *cloud-config like* maintenance configurations.
- **docker** for applications with complex dependencies that benefit from being shared and run by an expanded audience.
- **github actions** with self-hosted runners for ci build, test, release, deployment.
- **letsencrypt/certbot** for automated cert acqisition and renewal.
- **nginx** for cert serving, reverse proxying, load balancing.
- **mongodb**, **pgsql** for application state.
- **prometheus**, **promtool**. **loki**, **grafana** for log aggregation and dashboarding.
- **route53**, **cloudflare** for dns management, load balancing, health/uptime monitoring.
- **ssh**, **gpg**, **wireguard** and **ed25519/curve25519** are the underpinnings of our security infrastructure.
- **terraform** for instance provisioning and bootstrapping.

### candidate responsibilities

a successful candidate would likely have *experience*, *skills* and *aptitudes* that include:

- given our **mission**, a strong, personal motivation and conviction to contribute in the transaction **privacy** space is an essential aspect.
- written and spoken english proficiency or the ability to interact with written and spoken english effortlessly using technology.
- **championing** technology, tools and practices that facilitate the mission. understanding *when*, *why* and *how* to use and champion *what*.
- an ability to rapidly prototype and **demonstrate** the benefits and pitfalls of solution proposals. any aptitude for dashboarding tools like react, vue, grafana or others is useful.
- **networking**
a practical understanding of networks, dns, tls, firewalls, port forwarding, reverse proxying, traffic shaping and packet dropping
- **distributed source control**
git and github/gitlab and the use of their associated tooling and configuration should be second nature. a github/gitlab profile that demonstrates a history of best practice is always impressive. if privacy or other concerns prevent this, then some other way of conveying this experience is helpful.
- a bias toward **transparency** when communicating changes is essential.
our intention is to make everything we do discoverable, understandable and repeatable by others. ie by automating:
> *see the patch implementation in the deployment script at this diff url: ...*

or slightly less optimally, by documenting:
> *nginx `proxy_pass` port number, corrected from `x` to `y` on host `z` at path `p`.*

are both infinitely preferable to:
> *i fixed that problem and it's no longer broken.*

the first approach links to evidence that automation will prevent the problem from reoccuring. the second allows the audience to learn how to fix similar issues in the future. its usefulness is limited by the ability to recollect the location of the documentation and it doesn't mitigate against reoccurence. the final approach is only noise about things that no longer matter.
- an appreciation of **security** concerns including (an aptitude to learn quickly is fine):
- what aspects of an implementation or configuration should be deemed **secrets**?
- what aspects of an implementation or configuration can benefit from being **transparent**/public and why?
- a practical understanding of gpg, ssh, wireguard, rsa, ed25519 and curve25519 and when to use which is essential.
- being adept at spotting vulnerabilities and weaknesses.
- being able to read **bash** will make it easier to understand what's going on. we use a lot of it in both documentation and implementation.
- understanding when to containerise or componentise a thing and being able to articulate the justifications for either.
- understanding of checksums and hashing.
- understanding what type of updates should happen automatically and what type should be more considered or deliberate.
- **weekly status report** (more frequently when there are critical issues in the pipeline) including:
- a written summary (internal, notion) of what was achieved, what is planned and any blockers.
- two zoom meetings (immediate infrastructure team, wider engineering department), articulating the same, with webcam on.
- out-of-hours reachability and availability for mission-critical or emergency issue resolutions.
- being relaxed about admitting mistakes or problems with a view to sharing learnings and improving the overall sittuation.

### exemplary considerations

some skillsets and other considerations that will set you apart, include:

- being available during pst, cst or est hours is a bonus as eet and aest are currently covered.
- being willing and able to hold peers accountable for the responsibilities above.
- knowing how to use cryptocurrencies, create and manage accounts/keys/wallets, instigate and analyse transactions and other extrinsics or smart contracts.
- experience running a complete/full blockchain node (for any chain but, substrate or ethereum are especially useful, understanding tokenomics, validation or staking dynamics, more so).
- understanding distributed and decentralised systems, incentivisation, ipfs, tor, etc...
- rust, go, javascript and python (or any async capable language) skills.
- cryptography primitives and/or blockchain at the protocol or white-paper level.