Try   HackMD

DevOps Engineering on AWS

Welcome to the DevOps Engineering on AWS class. This notes contain a lot of supporting material from public sources that I feel will improve your understanding of DevOps and the corresponding AWS tools.

DevOps

CALMS
https://www.youtube.com/watch?v=LdOe18KhtT4
https://www.kitchensoap.com/2010/11/07/mttr-mtbf-for-most-types-of-f/

Organization

https://www.youtube.com/watch?v=4GK1NDTWbkY

InnerSource

https://www.youtube.com/watch?v=r4QU1WJn9f8
https://www.youtube.com/watch?v=D3C12ojRcp0
Give back to the OpenSource community by being part of it!

Banking & DevOps

https://www.youtube.com/watch?v=t7iVCIYQbgk
https://www.youtube.com/watch?v=xgSkva_Eq5s
https://www.youtube.com/watch?v=CTDx627FRVg
https://www.youtube.com/watch?v=i13v-m4nXpM
https://www.youtube.com/watch?v=EKEaMeaiZOw

Incident & Problem Management

https://community.monzo.com/t/resolved-current-account-payments-may-fail-major-outage-27-10-2017/26296/95
https://monzo.statuspage.io

Additional AWS Technology Content

AWS CDK

https://www.youtube.com/watch?v=ZWCvNFUN-sU
https://www.youtube.com/watch?v=1ps0Wh19MHQ
https://cdkworkshop.com
https://www.youtube.com/watch?v=T-H4nJQyMig

Migrating to CDK v2
https://dev.to/aws-builders/how-to-migrate-cdk-v1-to-cdk-v2-in-10-minuets-6i6

Testing CDK Constructs
https://docs.aws.amazon.com/cdk/v2/guide/testing.html

CloudFormation

https://aws.amazon.com/blogs/mt/remediate-drift-via-resource-import-with-aws-cloudformation/

Code Guru

https://www.youtube.com/watch?v=4DXZQ9ZOdVw

CodeArtifact

https://www.youtube.com/watch?v=pxV5E83S7Bw
https://www.youtube.com/watch?v=vEwv2cBS-MQ

Cognito

https://www.youtube.com/watch?v=jLQjQpUYw6g
https://auth.serverlessworkshops.io

API GW

https://www.youtube.com/watch?v=yfJZc3sJZ8E
https://www.youtube.com/watch?v=9ElpSPXk-g8

AWS SAM

https://www.youtube.com/watch?v=CIdUU6rNdk4

AWS Amplify

https://www.youtube.com/watch?v=uiTQL7Ne8rY
https://www.youtube.com/watch?v=SnqABG8e9Zk

AppSync

https://www.youtube.com/watch?v=XVU4pYeNfNo
https://graphql.org

Step Functions

https://aws.amazon.com/about-aws/whats-new/2020/09/aws-step-functions-adds-support-for-aws-x-ray/

Lambda

https://aws.amazon.com/blogs/compute/introducing-aws-lambda-destinations/
https://www.refinery.io/post/how-to-chain-serverless-functions-call-invoke-a-lambda-from-another-lambda
https://www.youtube.com/watch?v=Jkx6kVbDpL4

Kinesis

https://www.youtube.com/watch?v=jKPlGznbfZ0
https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/

MSK Managed Kafka

https://www.youtube.com/watch?v=HtU9pb18g5Q

Microservices

https://samnewman.io/talks/principles-of-microservices/
https://martinfowler.com/articles/microservices.html
https://martinfowler.com/bliki/BoundedContext.html
https://martinfowler.com/bliki/CQRS.html
https://martinfowler.com/bliki/DomainDrivenDesign.html

Resiliency

https://www.youtube.com/watch?v=Fup5vHEvU50
https://github.com/bbc/chaos-lambda

DynamoDB

https://www.youtube.com/watch?v=HaEPXoXVf2k
https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf

Nitro System

https://www.youtube.com/watch?v=0qcUOKupt7Y

Re:Invent 2020

https://reinvent.awsevents.com

Using Docker for Development

Dockerfile

FROM node:12

# install Python 3.8 taken from python/3.8/buster/slim/Dockerfile

# ensure local python is preferred over distribution python
ENV PATH /usr/local/bin:$PATH

# http://bugs.python.org/issue19846
# > At the moment, setting "LANG=C" on a Linux system *fundamentally breaks Python 3*, and that's not OK.
ENV LANG C.UTF-8

# runtime dependencies
RUN set -eux; \
	apt-get update; \
	apt-get install -y --no-install-recommends \
		ca-certificates \
		netbase \
	; \
	rm -rf /var/lib/apt/lists/*

ENV GPG_KEY E3FF2839C048B25C084DEBE9B26995E310250568
ENV PYTHON_VERSION 3.8.11

RUN set -ex \
	\
	&& savedAptMark="$(apt-mark showmanual)" \
	&& apt-get update && apt-get install -y --no-install-recommends \
		dpkg-dev \
		gcc \
		libbluetooth-dev \
		libbz2-dev \
		libc6-dev \
		libexpat1-dev \
		libffi-dev \
		libgdbm-dev \
		liblzma-dev \
		libncursesw5-dev \
		libreadline-dev \
		libsqlite3-dev \
		libssl-dev \
		make \
		tk-dev \
		uuid-dev \
		wget \
		xz-utils \
		zlib1g-dev \
# as of Stretch, "gpg" is no longer included by default
		$(command -v gpg > /dev/null || echo 'gnupg dirmngr') \
	\
	&& wget -O python.tar.xz "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz" \
	&& wget -O python.tar.xz.asc "https://www.python.org/ftp/python/${PYTHON_VERSION%%[a-z]*}/Python-$PYTHON_VERSION.tar.xz.asc" \
	&& export GNUPGHOME="$(mktemp -d)" \
	&& gpg --batch --keyserver hkps://keys.openpgp.org --recv-keys "$GPG_KEY" \
	&& gpg --batch --verify python.tar.xz.asc python.tar.xz \
	&& { command -v gpgconf > /dev/null && gpgconf --kill all || :; } \
	&& rm -rf "$GNUPGHOME" python.tar.xz.asc \
	&& mkdir -p /usr/src/python \
	&& tar -xJC /usr/src/python --strip-components=1 -f python.tar.xz \
	&& rm python.tar.xz \
	\
	&& cd /usr/src/python \
	&& gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" \
	&& ./configure \
		--build="$gnuArch" \
		--enable-loadable-sqlite-extensions \
		--enable-optimizations \
		--enable-option-checking=fatal \
		--enable-shared \
		--with-system-expat \
		--with-system-ffi \
		--without-ensurepip \
	&& make -j "$(nproc)" \
		LDFLAGS="-Wl,--strip-all" \
	&& make install \
	&& rm -rf /usr/src/python \
	\
	&& find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \( -type f -a \( -name '*.pyc' -o -name '*.pyo' -o -name '*.a' \) \) \
			-o \( -type f -a -name 'wininst-*.exe' \) \
		\) -exec rm -rf '{}' + \
	\
	&& ldconfig \
	\
	&& apt-mark auto '.*' > /dev/null \
	&& apt-mark manual $savedAptMark \
	&& find /usr/local -type f -executable -not \( -name '*tkinter*' \) -exec ldd '{}' ';' \
		| awk '/=>/ { print $(NF-1) }' \
		| sort -u \
		| xargs -r dpkg-query --search \
		| cut -d: -f1 \
		| sort -u \
		| xargs -r apt-mark manual \
	&& apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false \
	&& rm -rf /var/lib/apt/lists/* \
	\
	&& python3 --version

# make some useful symlinks that are expected to exist
RUN cd /usr/local/bin \
	&& ln -s idle3 idle \
	&& ln -s pydoc3 pydoc \
	&& ln -s python3 python \
	&& ln -s python3-config python-config

# if this is called "PIP_VERSION", pip explodes with "ValueError: invalid truth value '<VERSION>'"
ENV PYTHON_PIP_VERSION 21.2.1
# https://github.com/pypa/get-pip
ENV PYTHON_GET_PIP_URL https://github.com/pypa/get-pip/raw/a1675ab6c2bd898ed82b1f58c486097f763c74a9/public/get-pip.py
ENV PYTHON_GET_PIP_SHA256 6665659241292b2147b58922b9ffe11dda66b39d52d8a6f3aa310bc1d60ea6f7

RUN set -ex; \
	\
	savedAptMark="$(apt-mark showmanual)"; \
	apt-get update; \
	apt-get install -y --no-install-recommends wget; \
	\
	wget -O get-pip.py "$PYTHON_GET_PIP_URL"; \
	echo "$PYTHON_GET_PIP_SHA256 *get-pip.py" | sha256sum --check --strict -; \
	\
	apt-mark auto '.*' > /dev/null; \
	[ -z "$savedAptMark" ] || apt-mark manual $savedAptMark; \
	apt-get purge -y --auto-remove -o APT::AutoRemove::RecommendsImportant=false; \
	rm -rf /var/lib/apt/lists/*; \
	\
	python get-pip.py \
		--disable-pip-version-check \
		--no-cache-dir \
		"pip==$PYTHON_PIP_VERSION" \
	; \
	pip --version; \
	\
	find /usr/local -depth \
		\( \
			\( -type d -a \( -name test -o -name tests -o -name idle_test \) \) \
			-o \
			\( -type f -a \( -name '*.pyc' -o -name '*.pyo' \) \) \
		\) -exec rm -rf '{}' +; \
	rm -f get-pip.py

# end install Python

RUN apt-get update && apt-get install -y \
    apt-transport-https \
    bash-completion \
    ca-certificates \
    curl \
    git \
    groff \
    less \ 
    jq \
    nano \
    openssh-client \
    git \
    python3-passlib \
    python3-pip \
    python3-virtualenv \
    python3-venv \
    software-properties-common \
    sshpass \
    vim \
    unzip \
    apache2-utils \
	sudo \
    wget && \
    rm -rf /var/lib/apt/lists/*

RUN pip3 install awscli j2cli[yaml] boto boto3 yq
RUN npm install -g aws-cdk

RUN useradd --uid 1001 --shell /bin/bash --create-home cdk && usermod -aG sudo cdk
RUN echo cdk:cdk | chpasswd

USER cdk

RUN echo "source /etc/bash_completion" >> ~/.bashrc && \
    echo "complete -C '/usr/local/bin/aws_completer' aws" >> ~/.bashrc && \
    echo "export AWS_PROFILE=training" >> ~/.bashrc && \
    git config --global user.email "you@example.com" && \
    git config --global user.name "Your Name"

WORKDIR /project

build.sh

#!/bin/bash

if [[ $# -eq 0 ]] ; then
    echo 'Usage: build.sh <unique tag name>'
    exit 0
fi

# run docker build with those variables as arguments
docker build -t cdk:$1 .

run.sh

#!/bin/bash

docker run -ti --user cdk -v $(pwd):/project -v ~/.ssh:/home/cdk/.ssh -v ~/.aws:/home/cdk/.aws cdk:1 bash
Last changed by 
 
gekart
0
637

Read more

MLOps Engineering on AWS

Welcome to the Generative AI Essentials on AWS. This document contains useful links to complement our April 2025 training.

May 23, 2025
Argo CD

ApplicationSets Repository|https://codefresh.io/blog/how-to-structure-your-argo-cd-repositories-using-application-sets/Repository for Promotion|https://codefresh.io/blog/how-to-model-your-gitops-environments-and-promote-releases-between-them/

May 13, 2025
Create a simple kubeadm managed k8s cluster

aws ec2 run-instances –image-id ami-04a5bacc58328233d –count 1 –instance-type m5.large –key-name arch-kube –tag-specifications &#39;ResourceType=instance,Tags=[{Key=Name,Value=KubeCPNode}]&#39; –security-group-ids sg-00602c535f193f185 –subnet-id subnet-0b698c29da2490c1e –associate-public-ip-address

May 12, 2025
Architecting on Kubernetes

Welcome to the Architecting on Kubernetes class. This notes contain a lot of supporting material from public sources that will further improve your understanding of Kubernetes and the corresponding tools.

May 3, 2025
Read more from gekart
Published on HackMD