# DevOps Training (AWS)
###### tags: `Devops` `AWS`
Starting Date: 25 October 2021
Prepared by: Sohail Anjum
email: sohail.eic06g@nctu.edu.tw
#### This Training Covers all the basic of Cloud and suggested topics by experienced cloud developers/devops engineers.
#### First we can learn about the basics of Cloud and then move to more deep to hands on some topics as mentioned below.
### Cloud Top Playes
```java=
AWS.
Microsoft Azure.
GCP.
Alibaba Cloud.
Oracle.
VMWARE.
Fujitsu.
IBM etc.
```
### What is cloud by defination
```java=
1. Cloud computing is the delivery of computing services—including servers,storage, databases, networking, software, analytics, and
intelligence—over the Internet (“the cloud”) to offer faster innovation, flexible resources, and economies of scale.
```
### Characteristics of Cloud
```java=
On Demand Self Service
Broad Network Access
Scalability
Resource Polling
```
### Service of Cloud
```java=
IaaS (Infrastructure as a Service)
PaaS (Platform as a Service)
SaaS (Software as a Service)
```
### Deployement Model of Cloud
```java=
Public Cloud e.g. AWS, Azure, GCP
Private Cloud e.g. Enterprize Cloud (Multi-national companies cloud managed by them)
Hybrid Cloud
```
# AWS
## 1. Elastic Compute Cloud (EC2)
```java=
1. Amazon EC2 Provides Scalable Computing Capacity in the AWS Cloud.
2. You can use Amazon EC2 to launch as many or as few virtual servers as you need and
configure security and networking and manage storage.
3. Amazon EC2 enables you to scale up or scale down the instance.
4. Amazon EC2 is having two storage options e.g. EBS & Instance Store
5. PreConfigured templates are available known as Amazon Machine Learning.
6. By, Default, when you create an EC2 account with Amazon, you account is limited to
a maximum of 20 instances per EC2 Region with two default high I/O instance.
```
### Types of EC2 instances
```java=
1. General Purpose Instance (Balance Memory and CPU)
2. Compute Optimized (More CPU than RAM)
3. Memory Optimized (More RAM)
4. Storage Optimized (Low Latency)
5. High Memory (High Memory,Nitro System(A specialized Hypervior help
to increase Performance))
6. Accelerated Computing/GPU (Graphics Optimized)
7. Previous Generation
```
### General Purpose
```java=
General Purpose Instance Provides Balance Compute, Memory and Networking Reosurces, and Can be used for varity of workloads.
1. A Series (Includes A1) (Available Medium and large instances)
A1 Instances are ideally suited for Scale-out Workloads that are supported by ARM Ecosystem.
These Instances are well suited for the following application.
1. Web Servers
2. Containerized Micro Services
3. Caching Fleets
4. Distributed Data stores
5. Application thats requires ARM Instructions.
```
```java=
2. M Series (Includes M4, M5, M5a, M5ad, M5d) (Only Large size instance available)
- M4 Instance: The New M4 instances features a custom Intel Xeon E5-2676 v3 Haswell
Processor optimized specifically for EC2 instance.
- VCPU (2-40 Max)
- RAM (8-160Gbps Max)
- Instance Storage (EBS Only)
-M5, M5a, M5ad, M5d Instances: These Instance provide an ideal cloud infrastructure,
Offer a balance compute, memory and networking resources for broad range of applications.
- Used in Gaming Servers, Web servers, Small and medium databases.
- VCPU (2-96 Max)
- RAM (8-384Gbps Max)
- Instance Storage (EBS & NVMe SSD)
```
```java=
3. T Series (Includes T2, T3, T3a) (All Sizes available Nano/Micro, Small, Medium and Large).
- These isntance provides a baseline level of CPU performance with the ability to
brust to a higher level when require by workloads.
- An Unlimited can sustain in high cpu performance for any period of time when it required
- Used in Websites and Web Apps, Code repositories, development, build and test, Microservices.
- VCPU (2-8 )
- RAM (0.5-32Gbps)
```
### Compute Optimized Instances.
```java=
Compute Optimized Instances are Ideal for Compute-bound applications that benefit
from high performace processors.
- Three Types are available C4, C5, C5n
- C4: are optimise for compute intensive workloads and deliver very cost
effective high performance at a low price per computer ratio.
- Used in Web servers, Batch Processing, MMO Gaming, Video encoding.
- VCPU (2-36 Max)
- RAM (3.75-60Gbps Max)
- Instance Storage (EBS only)
- Network Bandwidth upto 10Gbps
```
```java=
- C5: are optimized for computer intensive workloads and deliver high-performance
at a low price per computer ratio.
- Powered by AWS Nitro System.
- Used in High Performance Web servers, Gaming, Video encoding.
- VCPU (2-72 Max)
- RAM (4-192Gbps Max)
- Instance Storage (EBS & NVMe SSD)
- Network Bandwidth upto 25Gbps
- C5 support maximum 25 EBS Volumes
- C5 use Elastic network adapter
- C5 use new EC2 Hypervisor
```
### Memory Optimized Instances.
```java=
- Memory optimized instances are design to deliver fast performance for workloads that
process large data sets in memory.
- Three Types of series are available R,X and Z.
- R4, R5, R5a, R5ad, R5d instances.
- High performance, Relational (MySql) and noSql (MongoDB) databases.
- Distributed web scale cache stores that provide in-memory caching of key vaue type data.
- Used in Financial series e.g. Hadoop.
- VCPU (2-96 Max)
- RAM (16-768Gbps Max)
- Instance Storage (EBS & NVMe SSD)
```
```java=
- X1, X1e Instances.
- Well suited for high performance databases, Memory intensive enterprise
application, relational databse workloads e.g. SAP
- use case: Electronic design automation.
- VCPU (4-128 Max)
- RAM (122-3904Gbps Max)
- Instance Storage (SSD)
```
```java=
- Z1d Instance
- High frequency Zd1 delivers a sustained all core frequency of upto 40GHz, the
fastest of any cloud instance.
- AWS Nitro system, Xeon Processor, upto 1.8Tb of instance storage.
- VCPU (2-48 Max)
- RAM (16-384Gbps Max)
- Instance Storage (NVMe SSD)
- Use Cases: Electronic design automation and certain databases workloads with
high per-core licensing cost.
```
### Storage Optimized Instances.
```java=
- Storage optimized instaces are designed for workload that require high, sequential
read and write access to very large data sets on local storage.
- They are optimized to deliver tens of thousands of low latency, I/O operations
per second (IOPS) to application.
- D2 Instance
- Well suite for the followings:
- Massive Parallel processing warehouse data.
- MapReduce and Hadoop distributed computing.
- Log or Data processing app.
- VCPU (4-36 Max)
- RAM (30.5-244Gbps Max)
- Instance Storage (SSD)
```
```java=
- H1 Instances
- This family features upto 16TB of HDD based local stroage, High disk throughput
and balance of compute and memory.
- well suited for apps requiring sequential access to large amount of data on
direct attached instance storage.
- Applications that require high throughput access to larage quantities of data.
- VCPU (8-64 Max)
- RAM (32-256Gbps Max)
- Instance Storage (HDD)
```
```java=
- I3 and I3en Instances.
- well suited for:
- High frequency online transaction processing system (OLTP).
- Relational Databases.
- NoSQL Databases.
- Distributed File Systems.
- Data Warehouse Applications.
- VCPU (2-96 Max)
- RAM (16-768Gbps Max)
- Instance Storage (NVMe SSD)
- Networking Performance (25Gbps to 100Gbps)
- Sequential Throughput
- Read 16Gb/s
- Write 64Gb/s (I3)
Write 8Gb/s (I3en)
```
### Accelerated Computing Instances.
```java=
- Accelarated computing instance families use hardware accelarators, or co-pocessor
to perform some functions such as floating point number calculations, graphics
processing or data pattron matching more efficiently than is possible in Software
Running on CPUs.
- Accelerated computing have P,G and F Series.
```
```java=
- F1 Instance: Offers customizable acceleration hardware acceleration with Field
Programmable Gate array (FPGA).
- Each FPGA contains 2.5 million logic elements and 6800 DSP engins.
- Design to accelerate computationally intensive algorithms, such as data flow
or highly parallel operations.
- Used in: Genomics research, finanical analytics, Realtime video processing & Big
Data search.
- F1 provides local NVMe SSD Storage.
- VCPU (8-64 Max)
- FPGA (1-8)
- RAM (122-976Gbps Max)
- Storage NVMe SSD
```
```java=
- P2 & P3 Instances:
- Its used NVIDIA Telsa GPUs
- Provide High bandwidth networking.
- Upto 32 Gbps of Memory per GPU, which makes them ideal for deep learning and
computational fluid dynamics.
- Used in: Machine Learning, Databases, Seismic analysis, Genomics, Molecular modeling, AI, Deep Learning.
- P2 Instance - P3 Instance
- VCPU (4-64 Max) - VCPU (8-96 Max)
- GPU (1-16) - GPU (1-8)
- RAM (61-732Gbps Max) - RAM (61-768Gbps Max)
- GPU RAM (12-192Gbps) - Storage (SSD & EBS)
- Network Bandwidth upto 25Gbps.
```
```java=
- G2 and G3 Instances.
- Optimize for Graphics intensive applications.
- Well suited for apps like 3D visualization.
- G3 instance use NVIDIA Telsa M60 GPU provide a cost effective, high performance
platform for Graphics Applications.
- Used in: Video Creation Services, 3D Visualization, Streaming, Graphics Intensive Applications.
- VCPU (4-64 Max)
- GPU (1-4)
- RAM (30.5-488Gbps Max)
- GPU Memory (8-32Gbps)
- Network Performance upto 25Gbps.
- Storage NVMe SSD
```
### High Memory Instance.
```java=
- High Memory Instances are bere metal instances and do not run on hypervisors.
- Only Available under dedicated host purchasing catagory atleast for 3 years.
- OS Installed directly on Hardware.
- High Memory Instances are purpose built to run large-in-memory databases,
including production developments of SAP, HANA in the cloud.
Features:
- Latest Generation Intel Xeon Pentium 8176M Processor.
- 6,9,12 TB of instance memory, the largest of any EC2 Instance.
- powered by AWS Nitro systems, a combination of dedicated hardware & lightweight Hypervisor.
- Bare Metal performance with direct access to host hardware.
- EBS Optimized by default at no additional cost.
- Network Performance 25Gbps.
- Dedicated EBS Bandwidth 14Gbps
- Each Instance offer 448 logical processor.
```
### Previous Generation Instances.
```java=
- These instances are still available if you are using. However these instances are old
- AWS put them into Previous Generation instaces list. They provide more better new
instancess to use.
- Previous generation instances are: T1, M1, C1, CC2, M2, CR1, CG1, i2, HS1, M3, C3 & R3.
```
## 2. EC2 instance Purchasing Options:
```java=
- On-demand
- On demand instances are vitual servers that run in AWS or AWS Relational
Database service (RDS) and are purchased at fix rate per hour.
- AWS recommends using on-demand instance for applications with short term
irregular workloads that can not be interrupted.
- Also suitable for using during testing and development on EC2.
- With on-demand instance you only pay for EC2 instance you are use.
- The use of on-demand instances frees you from the cost and complexities of
planning, purchasing and maintaining hardware and transform what are
commonly large fixed costs into much smaller vairable cost.
- Pricing is per instance == hours consumed for each instance, from the time of
instance is launched unitl it is terminated or stopped.
- Each partial instance hour consumed will be billed per-second for linux
instances and as full hour for all other instance types.
- Dedicated Instances
- Dedicated Instance are run in a VPC on Hardware that is dedicated to the single
customer.
- Your dedicated instances are physically isolated at the host hardware level from
instances that belong to other AWS account.
- Dedicated instances may share Hardware with other instance from the same AWS
account that are not dedicated instance.
- Pay for dedicated instances on-demand save upto 70% by purchasing reserved
instances, or saving upto 90% by purchasing spot instances.
- Dedicated Host
- An amazon EC2 dedicated host is a Physical Server with EC2 Instance Capacity
fully dedicated to your Use.
- Dedicated Host can help you address compilance requirement and reduce costs by
allowing you to use your existing server bound software licenses.
- Pay for a Physical host that is fully dedicated to running your instances,
and Bring your existing per-socket, per-core & per-VM Software license to
reduce cost.
- Spot Instances
- Amazon EC2 spot instances let you take advantage of used EC2 capacity in the
AWS cloud. spot instances are available at upto 90% discount, compare to
on-demand prices.
- You can use spot-instances for various test and development workloads.
- You also have the option to hibernate, stop, or terminate your spot instances
when EC2 Reclaims the capcity back with two mintues notice.
- Schedule Instance
- Schedule reserve instances enable you to prchase capacity reservations that
recure on a daily, weekly or monthly basis, with a specified start time and
duration for a one-year term.
- You Reserve the capacity in advance so that you know it is available when you
need.
- You pay for the time that the instances are scheduled, even if you dont use them.
- Schedule instances are good choice for workloads that do not run continously,
but do not run on a regular schedule.
- Purchase isntances that are always available on the specified recurring schedule,
for a one-year term.
- Reserved Instance
- Amazon EC2 RI provide a significant discount upto 75% compared to on-demand
pricing and provide a capacity reservation when used in a specific availability
zone.
- Reserved instance give you the option to reserve a DB instance for a one or
three year term and in turn receive a significant discount compared to the
on-demand instance pricing for DB instance.
Types of Reserved Instances are Standard RI, Convertible RI and Schedule RI.
- Standard RI: These provides the most significant discount up 75% off on-demand
and are best suited for steady state usage.
- Convertible RI: provide upto 54% discount and copability to change the attributes,
of the RI as long as the exchange result in the creation of reserved instances
of greater or equal value.
- Schedule RI: are available to launch within the time window you reserve.
```
## 3. Basics and Important Things about EC2.
### EC2 Access
```java=
- To access instance you need a key and key pair name.
- You can download the primary key only once.
- The public key is saved by AWS to match it, to the key pair name, and private key,
when you try to login to the EC2 Instance.
- Without key pair you cannot access instance via RDP or SSH.
- There are 20 EC2 soft limit per region, you submit the request AWS to increas it.
```
### EC2 Status Check
```java=
- By default AWS EC2 Service performed automated status checks in every one mintue.
- This is done on every EC2 Instance to Identify any Hardware or Software issue.
- Status check is built into the EC2 Instace.
- Status cannot be configured, deleted or Disable.
- EC2 Service can send it matrics data to AWS cloud watch every 5 minutes by default.
- Enabled Detailed monitoring is chargeable and send matrics in every minute.
- You can't charged for EC2 instance in case of stopped, however need to pay for
attached EBS Volume.
```
### When you stopped an EBS backed EC2 Instance.
```java=
- Instance perform a shutdown.
- State change from Running -> Stop.
- EBS Volume remain attached with the instance.
- Any Data Cached in RAM or instance store volume is gone.
- Instance retain its private IPV4, or any IPV6 address.
- Instance release it public IPV4 Address to back AWS pool.
- Instance retain its Elastic IP (Means Public and Static) address.
```
### EC2 Termination:
```java=
- When you terminate a running EC2 instance the instance state change as follows
Running -> Shuting Down -> Terminated
- Durint the shuting down to terminated state you do not incur charges.
- By default the EBS root device volumes are deleted automatically when the EC2
Instances are terminated.
- Any Additional volume attach to the instance by default, persist after the instance
is terminated.
- You can modify both behaviours by modifying the 'Delete on Termination' attribute
of any EBS volumes during instance launch or while running.
- Enable 'EC2 Termination option' against accidental termination.
```
### EC2 Metadata
```java=
- This is instance data that you can use to configure and manage the instance.
- E.g. IPv4 Addresses, IPv6 Addresses, DNS hostname, AMI-id, Instance ID, Instance
Type, Local hostname, public keys, security groups.
- Metadata can be only viewed from within the instance itself.
- Metadata is not protected by encryption, anyone that has access to the instance
can view this data.
```
### Instances User Data.
```java=
- Data Supplied by the user at instance launch in the form of a script to be executed
during the instance boot.
- User data is limited to 16Kb.
- You can change user data, by stopping EC2 First.
- User data is not encrypted.
```
### EC2 bare metal instances.
```java=
- Non Virtualized Environment.
- Operating systems runs directly on Hardware.
- Suitable for Licensing Restricted tier-1 business critical application. e.g.
i3 metal, i5 metal, Z1d metal, U-6tb1 metal.
```
### EBS Elastic Block Storage:
```java=
- Most common replicate with A-Z
- EBS Volume attached at launch are deleted when instance terminate.
- EBS volume attached to a running instance are not deleted when instance is
Terminated but are detached with data interact.
```
### Instance Storage:
```java=
- Physically attach to the host server.
- Data not lost when OS is Rebooted
- Data lost when:
- Underlaying Drive fails.
- Instance is stop or terminated.
- You can't detach or attach to another instance.
- Do not rely on for valuable long term data.
```
## 4. How to Create Free Account on AWS (demo)
Step 1:
open the link https://aws.amazon.com/free/
Step 2:
Step 3:
Step 4:
Make sure write the correct country name.
Should add the debit/credit card detail they will charge some money.
Step 5:
after account creation.
## 5. Amazon EC2 Instance creation.
```java=
- EC2 is one of the most popular of AWS offering
- It mainly consists in the capability of:
- Renting virtual machines (EC2).
- Stroing data on virtual drives(EBS)
- Distributing load across machines(ELB)
- Scaling the services using an auto-scaling group (ASG).
--- Knowing EC2 is fundamental to understand how the cloud works.
```
**Steps of launching the EC2.
Step 1: Login and see the AWS Management Console:**
**Step 2: Search the EC2 in the search bar and click on EC2.**
**Step 3: Choose the nearest region. (Choose your nearest region as per your location).**
**step 4: click on the launch instance button:**
**step 5: choose the AMI (Amazon Machine Image) depends on your requirement. The free tier eligible sticker means its available for free.
Click on the select.**
**step 6: Choose the instance type t2.micro and click on the Next: Configure Instance Details button to access all the setting and parametes**
**step 7: This is the Next: Configure Instance Details for now we can go with default settings.**
**Note: All parameters needs to understand by yourself and the purpose of each parameter.**
**Next: add storage:**
**step 8: add storage settings**
**Note: All parameters needs to understand by yourself and the purpose of each parameter.**
**Next: Add Tags**
**Step 9: Add Tages Window:**
**Step 10: Click on the add tags**
**You can add the tages one is important which is name time and is show on the UI. you can add more tags if you want**
**Next: Configure Security Group. (A Firewall)**
**Step 11: Secuirty Group Configuration**
**You can chnage the settings as you need but for now going with default in order to make this simpler.**
**Next: Review and Launch**
**Step 12: Review Window
You can check all the settings and edit here before lauch the instance.**
**Next: Launch**
**Step 13: Launch**
**When you click on launch you can see this kind of window.
A key pair give you access to your AWS Instance using SSH from outside.**
**Step 14: Create a new key pair and download the key pair.
When you can download you can have a file with .pem extension.**
**Then click on launch instances:**
**Step 15: click on the view instances**
**Step 16: See the Instaces**
**Step 17: Click on the Radio Buttion and get more detailed window:**
## 6. How to access EC2 instance using windows powershell:
**Step 1: you should know the public IP address and you have .pem file you already downloaded.**
**Step 2: you can check you details in security group about port No and Source or protocol etc**
**Step 3: Open the windows powershell on the windows and run pwd to check the path**
**Step 4: Check the list of folders and go the that specific folder where you download your .pem file. In my case its Downloads folder so I can go to the Downloads location.**
**Step 5: This is My file under Downloads Folder:**
**Step 6: Run the command and verify you are in the ec2 instance**
## 5. Amazon EC2 AMI(Amazon Machine Image) Back up and restore.
**Step 1: right click on the Instance -> Menu -> Image and templates -> create image**
**Step 2: Write the Image name and discription and click -> Create image**
**Step 3: Click on the left side menu -> Images -> AMIs and you will see window like this and your images is in the pending state it will take some time.**
**Now Status is Available**
**Step 4: Terminate the existing Instace we already created AMI.**
**The Terminated Instance stay for a while and removed from the instances list automatically after some time**
**Step 5: Launch the new instance with an existing AMI**
**Click -> Launch instance.**
**Click -> AMIs -> Select your Image**
**Next: Follow the "Steps 6-17 of launching the EC2"**
**Next: Follow the "Steps 6-17 of launching the EC2"**
## 6. AMI Snapshot
## 7. AMI and Snapshot schedule policy
## 8. Attach and deattach elastic IP
## 9. Attach Multiple Network Interfaces
## 10. Generate and recove pem files for SSH.
## 11. Access EC2 if pem file is lost
## 12. Security Group
## 13. ACLs
## 14. Subnets- Public, Private
## 15. Secure way to access EC2
## 16. Baastian Host
## 17. Attach EFS with EC2
## Amazon RDS
## Amazon S3
### 1. User Access restrictions for data and folders
### 2. IAM restriction
### 3. Data Retentain Policy
## Amazon EFS
## Amazon Glacier
## Amazon SNS
## Amazon SQS
## Amazon EBS
## Amazon VPC
### VPC Peering
### Make web and DB Environment
### NAT Gateway
### Internet Gateway
### VPN Connectivity with Private Subnet resource
## Amazon Auto-scaling
## Amazon IAM (User, Groups, Polices, Roles)
## Amazon Elastic Beanstalk
## Dynamo DB
## Amazon Elastic Cache
## Amazon Cloudfront
## Amazon Cloudwatch
Sign in with Wallet
Connect another wallet