# DevOps Engineering on AWS

Welcome to the DevOps Engineering on AWS class. This notes contain a lot of supporting material from public sources that I feel will improve your understanding of DevOps and the corresponding AWS tools.

## DevOps

https://www.youtube.com/watch?v=LdOe18KhtT4
https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/

### Organization

https://www.youtube.com/watch?v=4GK1NDTWbkY

### InnerSource

https://www.youtube.com/watch?v=r4QU1WJn9f8
https://www.youtube.com/watch?v=D3C12ojRcp0
Give back to the OpenSource community by being part of it!

## Banking & DevOps

https://www.youtube.com/watch?v=t7iVCIYQbgk
https://www.youtube.com/watch?v=xgSkva_Eq5s
https://www.youtube.com/watch?v=CTDx627FRVg
https://www.youtube.com/watch?v=i13v-m4nXpM
https://www.youtube.com/watch?v=EKEaMeaiZOw

### Incident & Problem Management

https://community.monzo.com/t/resolved-current-account-payments-may-fail-major-outage-27-10-2017/26296/95
https://monzo.statuspage.io

## Additional AWS Technology Content

### AWS CDK

https://www.youtube.com/watch?v=ZWCvNFUN-sU
https://www.youtube.com/watch?v=1ps0Wh19MHQ
https://cdkworkshop.com
https://www.youtube.com/watch?v=T-H4nJQyMig

Migrating to CDK v2
https://dev.to/aws-builders/how-to-migrate-cdk-v1-to-cdk-v2-in-10-minuets-6i6

Testing CDK Constructs
https://docs.aws.amazon.com/cdk/v2/guide/testing.html

### CloudFormation

https://aws.amazon.com/blogs/mt/remediate-drift-via-resource-import-with-aws-cloudformation/

### Code Guru

https://www.youtube.com/watch?v=4DXZQ9ZOdVw

### CodeArtifact

https://www.youtube.com/watch?v=pxV5E83S7Bw
https://www.youtube.com/watch?v=vEwv2cBS-MQ

### Cognito

https://www.youtube.com/watch?v=jLQjQpUYw6g
https://auth.serverlessworkshops.io

### API GW

https://www.youtube.com/watch?v=yfJZc3sJZ8E
https://www.youtube.com/watch?v=9ElpSPXk-g8

### AWS SAM

https://www.youtube.com/watch?v=CIdUU6rNdk4

### AWS Amplify

https://www.youtube.com/watch?v=uiTQL7Ne8rY
https://www.youtube.com/watch?v=SnqABG8e9Zk

### AppSync

https://www.youtube.com/watch?v=XVU4pYeNfNo
https://graphql.org

### Step Functions

https://aws.amazon.com/about-aws/whats-new/2020/09/aws-step-functions-adds-support-for-aws-x-ray/

### Lambda

https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
https://www.refinery.io/post/how-to-chain-serverless-functions-call-invoke-a-lambda-from-another-lambda
https://www.youtube.com/watch?v=Jkx6kVbDpL4

### Kinesis

https://www.youtube.com/watch?v=jKPlGznbfZ0
https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/

### MSK Managed Kafka

https://www.youtube.com/watch?v=HtU9pb18g5Q

### Microservices

https://samnewman.io/talks/principles-of-microservices/
https://martinfowler.com/articles/microservices.html
https://martinfowler.com/bliki/BoundedContext.html
https://martinfowler.com/bliki/CQRS.html
https://martinfowler.com/bliki/DomainDrivenDesign.html

### Resiliency

https://www.youtube.com/watch?v=Fup5vHEvU50
https://github.com/bbc/chaos-lambda

### DynamoDB

https://www.youtube.com/watch?v=HaEPXoXVf2k
https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

### Nitro System

https://www.youtube.com/watch?v=0qcUOKupt7Y

### Re:Invent 2020

https://reinvent.awsevents.com

## Using Docker for Development

### Dockerfile

FROM node:12

# install Python 3.8 taken from python/3.8/buster/slim/Dockerfile

# ensure local python is preferred over distribution python
ENV PATH /usr/local/bin:$PATH

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

# runtime dependencies
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		ca-certificates \
		netbase \
	; \
	rm -rf /var/lib/apt/lists/*

ENV GPG_KEY E3FF2839C048B25C084DEBE9B26995E310250568
ENV PYTHON_VERSION 3.8.11

RUN set -ex \
	\
	&& savedAptMark="$(apt-mark showmanual)" \
	&& apt-get update && apt-get install -y --no-install-recommends \
		dpkg-dev \
		gcc \
		libbluetooth-dev \
		libbz2-dev \
		libc6-dev \
		libexpat1-dev \
		libffi-dev \
		libgdbm-dev \
		liblzma-dev \
		libncursesw5-dev \
		libreadline-dev \
		libsqlite3-dev \
		libssl-dev \
		make \
		tk-dev \
		uuid-dev \
		wget \
		xz-utils \
		zlib1g-dev \
# as of Stretch, "gpg" is no longer included by default
		$(command -v gpg > /dev/null || echo 'gnupg dirmngr') \
	\
	&& wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
	&& wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
	&& export GNUPGHOME="$(mktemp -d)" \
	&& gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY" \
	&& gpg --batch --verify python.tar.xz.asc python.tar.xz \
	&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \
	&& rm -rf "$GNUPGHOME" python.tar.xz.asc \
	&& mkdir -p /usr/src/python \
	&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \
	&& rm python.tar.xz \
	\
	&& cd /usr/src/python \
	&& gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
	&& ./configure \
		--build="$gnuArch" \
		--enable-loadable-sqlite-extensions \
		--enable-optimizations \
		--enable-option-checking=fatal \
		--enable-shared \
		--with-system-expat \
		--with-system-ffi \
		--without-ensurepip \
	&& make -j "$(nproc)" \
		LDFLAGS="-Wl,--strip-all" \
	&& make install \
	&& rm -rf /usr/src/python \
	\
	&& find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name '*.a' \) \) \
			-o \( -type f -a -name 'wininst-*.exe' \) \
		\) -exec rm -rf '{}' + \
	\
	&& ldconfig \
	\
	&& apt-mark auto '.*' > /dev/null \
	&& apt-mark manual $savedAptMark \
	&& find /usr/local -type f -executable -not \( -name '*tkinter*' \) -exec ldd '{}' ';' \
		| awk '/=>/ { print $(NF-1) }' \
		| sort -u \
		| xargs -r dpkg-query --search \
		| cut -d: -f1 \
		| sort -u \
		| xargs -r apt-mark manual \
	&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
	&& rm -rf /var/lib/apt/lists/* \
	\
	&& python3 --version

# make some useful symlinks that are expected to exist
RUN cd /usr/local/bin \
	&& ln -s idle3 idle \
	&& ln -s pydoc3 pydoc \
	&& ln -s python3 python \
	&& ln -s python3-config python-config

# if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value '<VERSION>'"
ENV PYTHON_PIP_VERSION 21.2.1
# https://github.com/pypa/get-pip
ENV PYTHON_GET_PIP_URL https://github.com/pypa/get-pip/raw/a1675ab6c2bd898ed82b1f58c486097f763c74a9/public/get-pip.py
ENV PYTHON_GET_PIP_SHA256 6665659241292b2147b58922b9ffe11dda66b39d52d8a6f3aa310bc1d60ea6f7

RUN set -ex; \
	\
	savedAptMark="$(apt-mark showmanual)"; \
	apt-get update; \
	apt-get install -y --no-install-recommends wget; \
	\
	wget -O get-pip.py "$PYTHON_GET_PIP_URL"; \
	echo "$PYTHON_GET_PIP_SHA256 *get-pip.py" | sha256sum --check --strict -; \
	\
	apt-mark auto '.*' > /dev/null; \
	[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \
	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
	rm -rf /var/lib/apt/lists/*; \
	\
	python get-pip.py \
		--disable-pip-version-check \
		--no-cache-dir \
		"pip==$PYTHON_PIP_VERSION" \
	; \
	pip --version; \
	\
	find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \
			\( -type f -a \( -name '*.pyc' -o -name '*.pyo' \) \) \
		\) -exec rm -rf '{}' +; \
	rm -f get-pip.py

# end install Python

RUN apt-get update && apt-get install -y \
    apt-transport-https \
    bash-completion \
    ca-certificates \
    curl \
    git \
    groff \
    less \ 
    jq \
    nano \
    openssh-client \
    git \
    python3-passlib \
    python3-pip \
    python3-virtualenv \
    python3-venv \
    software-properties-common \
    sshpass \
    vim \
    unzip \
    apache2-utils \
	sudo \
    wget && \
    rm -rf /var/lib/apt/lists/*

RUN pip3 install awscli j2cli[yaml] boto boto3 yq
RUN npm install -g aws-cdk

RUN useradd --uid 1001 --shell /bin/bash --create-home cdk && usermod -aG sudo cdk
RUN echo cdk:cdk | chpasswd

USER cdk

RUN echo "source /etc/bash_completion" >> ~/.bashrc && \
    echo "complete -C '/usr/local/bin/aws_completer' aws" >> ~/.bashrc && \
    echo "export AWS_PROFILE=training" >> ~/.bashrc && \
    git config --global user.email "[e-mail]" && \
    git config --global user.name "Your Name"

WORKDIR /project

### build.sh

#!/bin/bash

if [[ $# -eq 0 ]] ; then
    echo 'Usage: build.sh <unique tag name>'
    exit 0
fi

# run docker build with those variables as arguments
docker build -t cdk:$1 .

### run.sh

#!/bin/bash

docker run -ti --user cdk -v $(pwd):/project -v ~/.ssh:/home/cdk/.ssh -v ~/.aws:/home/cdk/.aws cdk:1 bash