---
# System prepended metadata

title: The journey to AWS EMR cloudformation based cluster

---

# The journey to AWS EMR cloudformation based cluster

## Challenges

- Being fast ...
- Reproducability
- How to test?
- Where to test?
- How to keep things clean?
- What to do with dependencies (like VPC)?
- New vs. migrating existing

## Being fast - improvement on speed

Avoid usage of AWS console. So put everything in code, which needs to be repeated.
Use `aws-sso-util` for AWS credentials handling. Removes the need of relogin every hour.
Automate the tests ...

## Reproducability

Make the work reproducable at any time (or at least the next day).
- Cover work with tests.
- Commit every day
- Keep things clean (see below)
- provide an entrypoint (👉🏻 CONTRIBUTION.md)


## Testing

- linting ✅
- validating ✅
- deploying ✅
- validating deployment 🚫
- integration testing ☕️

### Example workflow

Make your changes in `emr/cluster.yaml`.
Before committing, check locally:
```
make verify
make test
```

For more see CONTRIBUTION.md in `cfn/`.

### Challenges with testing

#### Destroy s3 buckets

As s3 buckets cannot be deleted while having objects in it ...

👉🏻 Trying lamdba based purge of s3 buckets using it with CFN custom ressource ... ✅ (working 🎉)

POC: https://github.com/h1f1x/aws-truth-teller/tree/s3-deletion-w-cloudformation


#### Uploading objects to s3

With terraform there is a nice method. With CF I don't know yet. With CDK it would be easy again.

👉🏻 Trying lamdba based initial upload ...

👇 Should try to setup this macro:
https://github.com/awslabs/aws-cloudformation-templates/tree/a11722d/aws/services/CloudFormation/MacrosExamples/S3Objects


## Cleanup

Keep things clean with:
- Everything can be deployed by CF
- Deploy to sandbox will cleanup due nuke every night

How to detect and cleanup old resources?
👉🏻 Finding cloudformation stacks ...

📚 Learn to use aws-cli query
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html
Cool tool:
https://github.com/jmespath/jmespath.terminal

## To migrate or not migrate

Trying to setup the base stack from existing solution in CF:
https://docs.aws.amazon.com/solutions/latest/real-time-analytics-spark-streaming/welcome.html

Bring it to a stable deploy which can be easiely repeated and adopted.
Later check and migrate bootstrap, parameters from existing.

## The components

### VPC

Need to setup as the new account don't have an existing VPC. 
New cloudformation based approach, because:
- there seems to be an overhead in deploy existing one with the tf stack.
- in general we want to use only one technology (cloudfromation) instead of mixing tf w/ cf
- improve VPC setup - not optimal
- use existing template (not do it by ourself)

Benefits:
- bastion host (increased security according to best practices)
- based on existing template (development effort is small, maturity high)
- testable (can setup multiple stacks at the same time)
- account agnostic (can be rolled out everywhere)

#### Challenges

1. How to use IPv6 only?
1. How to deploy one NAT GW for multiple subnets?

### How users should be able to connect to nodes

Dependend implementations:
- Bastion Hosts
- EC2 key creation


## TODOs

Mainly improvements which I don't want to do now:

1. Add Cloudformation MetaData like:
https://github.com/widdix/aws-cf-templates/blob/master/ecs/cluster.yaml

👉🏻 Look into the issues of DEP 

1. Test if SSM Session can be used to access EMR nodes
2. upload bootstrap files w/ CFN (Macros ?!)
3. #9 (metaData)
4. #10 (EMR security)

## Implementation considerations

See miro


## Things to share (learning friday)

📚 Learn to use aws-cli query
https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-filter.html
Cool tool:
[https://github.com/jmespath/jmespath.terminal
](https://github.com/jmespath/jmespath.terminal
)
📚 GitHub CoPilot 
Show with our real example

📚 VPC with bastion hosts


