# AWS Cloud Learning (雲端程式設計 Lab7-9)
###### tags: `Master` `Cloud Programming`
## Lab7:Javascript, Node.js (PYTHON)
- Overview
Part 1: Introduction
(1) Javascripot and Node.js basic concept
(2) AWS JS SDK & Asynchronous Call
Part 2: Building client-side and server-side web app
(1) Build a client-side Web app with AWS SDK
(2) Build a Node.js Console Application
(3) Build a Node.js Web Application
Part 3: Express web framework and AWS deployment
(1) Node.js Express Web Framework
(2) Deploy Node.js APP on AWS
### Part 1: Introduction
- Client-side vs Server-side JS
**1. Javascript**
(1) A high-level, dynamic, interpreted programming language.
(2) The **default scripting language in HTML** running in **a web browser**.
**2. Node.js**
(1) A **cross-platform** runtime environment for running **server-side JavaScript applications**
(2) An **event-driven, non-blocking I/O model** that makes it lightweight and efficient
- NPM & Express Web Framework
**1. NPM**
(1) the default package manager for Node.js
(2) It is automatically included when node.js is installed
(3) common npm commands
-> npm install "packge name"
-> npm uninstall "packge name"
-> npm search "packge name"
**2. Node.js Express Framework**
A minimal Node.js **web application framework** that provides a robust set of features to **develop web and mobile applications**
- 要先install Node.js
- JS web programming
(1) JS online tutorial
https://www.w3schools.com/js/default.asp
(2) What js can do?
(3) Where to put js code?
-> 在html裡面，JS可以插入在<script> </script> tags之間，在用head和body包起來
-> 建議put the extermal file right before the end of body
- Run your First JS web page
Step 1: Install HTTP server
install http-server globally
`npm install -g http-server`
Step 2: create a project folder and create index.html
index.html
```htmlembedded=
<!DOCTYPE html>
<html><body>
<h1>My First JavaScript</h1>
<button type="button" onclick="document.getElementById('demo').innerHTML = Date()"> Click me to display Date and Time.</button>
<p id="demo"></p>
</body></html>
```
- AWS JS SDK
(1) Introduction
Using the SDK with **Node.js** differs from using it with web browsers in the way of **loading the SDK** and **obtaining the credentials** needed to access specific web services
(2) With Node.js (server side)
-> Installing: using the npm package
`npm install aws-sdk`
-> Loading: using require in the code
`var AWS = require('aws-sdk');`
-> Credential:
If you are using a localhost: setup your credential using AWS CLI
If you will deploy your code to AWS: pass role to the EC2 instance. Don’t put your credential in files
(3) With web browser (client side)
-> Installing: **don't need it**
-> Loading:
Load the hosted SDK package directly from AWS by **adding the following script tag to HTML pages**
`<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.20.min.js"></script>`
-> Credential:
Create **an Amazon Cognito identity pool** with access enabled for unauthenticated identities
Clients obtain the identity pool ID through the Cognito user identity service (sign-in) or reconfiguration (hard code)
- Authentication & Authorization for web client
- AWS JS API
S3 methods:
- Calling Service Asynchronously
(1) All request made through **the AWS JS SDK** are **asynchronous**
(2) Must manage the calls not to use data before it is available
(3) One way to handle it is using **anonymous callback functions**
Keypoint: user data may arrive after you need it
- Anonymous Callback Function
(1) Each service object method can accept an anonymous callback function as the last parameter.
-> If the method call succeeds, the contents of the response are available to the callback function in the data parameter.
-> If the call doesn't succeed, the details about the failure are provided in the error parameter.
-> callback function基本架構如下
-> Sample code
### Part 2: Building client-side and server-side web app
- Build a Client-side Web App with AWS SDK
A simple HTML page provides a **browser-based application** for creating and deleting photo albums in an **S3 bucket**
steps:
(1) Create a S3 bucket and configure it
(2) Create an Amazon Cognito identity pool
keep the identity pool ID
(3) Download js code to your local project folder
(4) Edit the app.js file
"albumBucketName"
"bucketRegion"
"IdentityPoolId"
(5) Start the webpage
app.js
```javascript=
var albumBucketName = 'my-album-107062511';
var bucketRegion = 'us-east-1';
var IdentityPoolId = 'us-east-1:ae238218-c208-4e9c-bc6b-6f369323ab20';
AWS.config.update({
region: bucketRegion,
credentials: new AWS.CognitoIdentityCredentials({
IdentityPoolId: IdentityPoolId
})
});
var s3 = new AWS.S3({
apiVersion: '2006-03-01',
params: {Bucket: albumBucketName}
});
function listAlbums() {
s3.listObjects({Delimiter: '/'}, function(err, data) {
if (err) {
return alert('There was an error listing your albums: ' + err.message);
} else {
var albums = data.CommonPrefixes.map(function(commonPrefix) {
var prefix = commonPrefix.Prefix;
var albumName = decodeURIComponent(prefix.replace('/', ''));
return getHtml([
'<li>',
'<span onclick="deleteAlbum(\'' + albumName + '\')">X</span>',
'<span onclick="viewAlbum(\'' + albumName + '\')">',
albumName,
'</span>',
'</li>'
]);
});
var message = albums.length ?
getHtml([
'<p>Click on an album name to view it.</p>',
'<p>Click on the X to delete the album.</p>'
]) :
'<p>You do not have any albums. Please Create album.';
var htmlTemplate = [
'<h2>Albums</h2>',
message,
'<ul>',
getHtml(albums),
'</ul>',
'<button onclick="createAlbum(prompt(\'Enter Album Name:\'))">',
'Create New Album',
'</button>'
]
document.getElementById('app').innerHTML = getHtml(htmlTemplate);
}
});
}
function createAlbum(albumName) {
albumName = albumName.trim();
if (!albumName) {
return alert('Album names must contain at least one non-space character.');
}
if (albumName.indexOf('/') !== -1) {
return alert('Album names cannot contain slashes.');
}
var albumKey = encodeURIComponent(albumName) + '/';
s3.headObject({Key: albumKey}, function(err, data) {
if (!err) {
return alert('Album already exists.');
}
if (err.code !== 'NotFound') {
return alert('There was an error creating your album: ' + err.message);
}
s3.putObject({Key: albumKey}, function(err, data) {
if (err) {
return alert('There was an error creating your album: ' + err.message);
}
alert('Successfully created album.');
viewAlbum(albumName);
});
});
}
function viewAlbum(albumName) {
var albumPhotosKey = encodeURIComponent(albumName) + '//';
s3.listObjects({Prefix: albumPhotosKey}, function(err, data) {
if (err) {
return alert('There was an error viewing your album: ' + err.message);
}
// `this` references the AWS.Response instance that represents the response
var href = this.request.httpRequest.endpoint.href;
var bucketUrl = href + albumBucketName + '/';
var photos = data.Contents.map(function(photo) {
var photoKey = photo.Key;
var photoUrl = bucketUrl + encodeURIComponent(photoKey);
return getHtml([
'<span>',
'<div>',
'<img style="width:128px;height:128px;" src="' + photoUrl + '"/>',
'</div>',
'<div>',
'<span onclick="deletePhoto(\'' + albumName + "','" + photoKey + '\')">',
'X',
'</span>',
'<span>',
photoKey.replace(albumPhotosKey, ''),
'</span>',
'</div>',
'<span>',
]);
});
var message = photos.length ?
'<p>Click on the X to delete the photo</p>' :
'<p>You do not have any photos in this album. Please add photos.</p>';
var htmlTemplate = [
'<h2>',
'Album: ' + albumName,
'</h2>',
message,
'<div>',
getHtml(photos),
'</div>',
'<input id="photoupload" type="file" accept="image/*">',
'<button id="addphoto" onclick="addPhoto(\'' + albumName +'\')">',
'Add Photo',
'</button>',
'<button onclick="listAlbums()">',
'Back To Albums',
'</button>',
]
document.getElementById('app').innerHTML = getHtml(htmlTemplate);
});
}
function addPhoto(albumName) {
var files = document.getElementById('photoupload').files;
if (!files.length) {
return alert('Please choose a file to upload first.');
}
var file = files[0];
var fileName = file.name;
var albumPhotosKey = encodeURIComponent(albumName) + '//';
var photoKey = albumPhotosKey + fileName;
s3.upload({
Key: photoKey,
Body: file,
ACL: 'public-read'
}, function(err, data) {
if (err) {
return alert('There was an error uploading your photo: ', err.message);
}
alert('Successfully uploaded photo.');
viewAlbum(albumName);
});
}
function deletePhoto(albumName, photoKey) {
s3.deleteObject({Key: photoKey}, function(err, data) {
if (err) {
return alert('There was an error deleting your photo: ', err.message);
}
alert('Successfully deleted photo.');
viewAlbum(albumName);
});
}
function deleteAlbum(albumName) {
var albumKey = encodeURIComponent(albumName) + '/';
s3.listObjects({Prefix: albumKey}, function(err, data) {
if (err) {
return alert('There was an error deleting your album: ', err.message);
}
var objects = data.Contents.map(function(object) {
return {Key: object.Key};
});
s3.deleteObjects({
Delete: {Objects: objects, Quiet: true}
}, function(err, data) {
if (err) {
return alert('There was an error deleting your album: ', err.message);
}
alert('Successfully deleted album.');
listAlbums();
});
});
}
```
**Code Details**
1. listAlbums
2. CreateAlbums
index.html
```htmlembedded=
<!DOCTYPE html>
<html>
<body>
<h1>My Photo Albums App</h1>
<div id="app"></div>
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.7.20.min.js"></script>
<script src="./app.js"></script>
<script>
function getHtml(template) {
return template.join('\n');
}
listAlbums();
</script>
</body>
</html>
```
**Code Details**
Reference:
[AWS 官方程式講解](https://docs.aws.amazon.com/zh_tw/sdk-for-javascript/v2/developer-guide/s3-example-photo-album.html)
- **Build a Node.js console application with AWS SDK**
Run a server-side js to list S3 buckets
Steps:
(1) Create project folder
(2) Setup credential
method 1: use AWS CLI: aws configure(recommended)
method 2: store the credential info in "./config.json"
(3) Install AWS SDK locally in your project folder
aws-sdk will be installed under the node_modules folders
(4) Create node.js program
```javascript=
var AWS = require('aws-sdk');
//AWS.config.loadFromPath('./config.json'); //uncomment it if needed
s3 = new AWS.S3({apiVersion: '2006-03-01'}); // Create S3 service object
s3.listBuckets(function(err, data) { // Call S3 to list current buckets
if (err) { console.log("Error", err); }
else { console.log("Bucket List", data.Buckets); }
});
```
(5) Execute the code
- **Build a Node.js web app without AWS SDK**
Steps:
(1) Create project folder
```javascript=
// Load the http module to create an http server.
var http = require('http');
// Configure our HTTP server to respond with Hello World to all requests.
var server = http.createServer(function (request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
response.end("Hello World\n");
});
// Listen on port 8000, IP defaults to 127.0.0.1
server.listen(8000);
// Put a friendly message on the terminal
console.log("Server running at http://127.0.0.1:8000/");
```
(2) Create package.json with the command: `npm init`
(3) Run it with the command: node hello-http.js
(4) try http://localhost:8000/
- **Build a Node.js web app with AWS SDK**
Build a simple node.js web app to list S3 buckets
Steps:
(1) Create project folder
(2) Create package.json
(3) install aws-sdk --save
save means to store the dependency in package.json file
(4) create file "bucketlist-http.js"
```javascript=
// Load the http module to create an http server.
var http = require('http');
var AWS = require('aws-sdk');
var bucketRegion = 'us-east-1';
function getHtml(template) {
return template.join('\n');
}
// Configure our HTTP server to list S3 buckets.
var server = http.createServer(function (request, response) {
response.writeHead(200, {"Content-Type": "text/plain"});
var s3 = new AWS.S3({apiVersion: '2006-03-01'});
s3.listBuckets(function(err, data) {
if (err) {
response.end(err.message);
} else {
var buckets = data.Buckets.map(function(item) {
return item.Name;
});
response.end(getHtml(buckets));
}
});
});
// Listen on port 8000, IP defaults to 127.0.0.1
server.listen(8000);
// Put a friendly message on the terminal
console.log("Server running at http://127.0.0.1:8000/");
```
### Part 3: Express web framework and AWS deployment
- Goal:
Use the express package to build **MVC web applications**
- Steps:
(1) Install Express (only needed for the first time
(2) Create project
(3) Setup local dependencies
(4) Run it
- Express Web Framework
- Route and its handler function (middleware system)
-->Routing refers to the definition of application end points(URIs) and how they respond to client requests
--> make it easy to configure **RESTful web services** for your web application
- Rendering HTML views
- **Deploy Node.js to AWS using EB-CLI**
Deploying your application to EB using EB CLI and git,
and then updating the application to use the Express Framework
Steps:
(1) Prerequisite Installation
--> install git
--> install ebcli
(2) Go to your project folder
(3) Init git
(4) Exclude files from being added to the repository
這一步不是很重要，是決定檔案或路徑要不要一起包起來
把它寫在.gitignore的話，就不會包進去
(5) Create application
(6) Create an environment with the default sample code
(7) Open the environment's URL in browser
(8) Add a config file
--> .ebextensions/nodecommand.config” to set the Node Command to "npm start“
(9) Commit the updated code into git repository
(10) Deploy the application with the updated version
app.js
```javascript=
var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');
var indexRouter = require('./routes/index');
var usersRouter = require('./routes/users');
var app = express();
// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');
app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));
app.use('/', indexRouter);
app.use('/users', usersRouter);
// catch 404 and forward to error handler
app.use(function(req, res, next) {
next(createError(404));
});
// error handler
app.use(function(err, req, res, next) {
// set locals, only providing error in development
res.locals.message = err.message;
res.locals.error = req.app.get('env') === 'development' ? err : {};
// render the error page
res.status(err.status || 500);
res.render('error');
});
module.exports = app;
```
package.json
```json=
{
"name": "node-express-hello",
"version": "0.0.0",
"private": true,
"scripts": {
"start": "node ./bin/www"
},
"dependencies": {
"cookie-parser": "~1.4.3",
"debug": "~2.6.9",
"express": "~4.16.0",
"http-errors": "~1.6.2",
"jade": "~1.11.0",
"morgan": "~1.9.0"
}
}
```
index.js
```javascript=
var express = require('express');
var router = express.Router();
/* GET home page. */
router.get('/', function(req, res, next) {
res.render('index', { title: 'Express' });
});
module.exports = router;
```
users.js
```javascript=
var express = require('express');
var router = express.Router();
/* GET users listing. */
router.get('/', function(req, res, next) {
res.send('respond with a resource');
});
module.exports = router;
```
error.jade
```
extends layout
block content
h1= message
h2= error.status
pre #{error.stack}
```
index.jade
```
extends layout
block content
h1= title
p Welcome to #{title}
```
layout.jade
```
doctype html
html
head
title= title
link(rel='stylesheet', href='/stylesheets/style.css')
body
block content
```
## Lab 8: Serverless Architecture --- Lambda
### Part 1: Introduction
- What is AWS Lamda
It is a managed compute service that lets you run code **without provisioning or managing servers**
--> It **executes your code only when needed** and **scales automatically**
--> all with zero administration
- Limitation
1. 不能客製化
--> use EC2 and EB if you want to customize your environment
2. Asynchronous computation 有timeout的限制
- Serverless architecture
### Part 2: Lambda: Event Source
- Purpose
(1) 在設定完成後，your Lambda function gets **invoked automatically** when these event sources detect event
(2) 不同的event source events，失敗會有不同的處理方式
- Supported event sources
(1) S3, DynamoDB(must in the same region), Kinesis, SNS, SES, Cognito, CloudWatch, CloudFormation, CodeCommit, API Gateway, AWS Echo, AWS Lex, AWS IoT, etc.
(2) Use cases (下面會示範S3)
https://docs.aws.amazon.com/en_us/lambda/latest/dg/lambda-services.html
- Streaming Event Use Case
(1) The custom application writes records to an Amazon Kinesis stream
(2) AWS Lambda continuously **polls** the stream, and **invokes the Lambda function when the service detects new records on the stream**
(3) A permission policy is attached to allow AWS Lambda to poll the stream, and then execute the Lambda function
- Non-streaming Event Use Case
(1) The user creates an object in a bucket.
(2) Amazon S3 detects the object created event.
(3) Amazon S3 **invokes(pushes data to) your Lambda** function using the permissions provided by the execution role
(4) AWS Lambda executes the Lambda function, and necessary permission must be granted to the lambda function for execution
- On-demand Use Case
(1) The mobile application **sends a request to Amazon Cognito** with an identity pool ID in the request
(2) Amazon Cognito **returns temporary security credentials** to the application.
(3) The mobile application invokes the Lambda function using the temporary credentials (Cognito Identity).
(4) **AWS Lambda assumes the execution role** to execute your Lambda function on your behalf.
(5) The Lambda function executes.
(6) AWS Lambda returns results to the mobile application
(if Lambda function is invoked using the RequestResponse)
- Comparison of Invocation Ttpe
### Part 3: Example
#### 1. Hello World
- Goal:
create and test the simplest lambda function
- Steps:
直接套用AWS提供的sample就能完成
#### 2. S3 Thumbnail
- Goal:
automatically create a thumbnail for each image uploaded to a bucket
- Steps:
(1) 新增兩個bucket(一個放原本圖片，一個放壓縮後的)
(2) Create lambda execution role
(3) Prepare deployment package
--> Create a Node.js project folder
--> Install Async utility module & GraphicsMagick
--> Download sample code
--> Zip the CreateThumbnail.js file and the node_modules folder as CreateThumbnail.zip
(4) Create lambda function
(5) Test it
CreateThumbnail.js 程式碼解釋
- GraphicsMagick
A image processing library for various image formats
- Async utility nodule
A utility module which provides straight-forward, powerful functions for working with asynchronous JS.
- aws-sdk
我們沒有安裝他，因為已經在lambda node.js的環境中
- Retrieve and parse the data from event msg
- The syntax of S3 upload msg
the syntax could vary between different events
- The method **waterfull** from the async utility module is used in the sample code to **run an array of tasks sequentially, and pass their results to the next task in the array**
- Simply return "message" at the end
以下為完整程式碼
```javascript=
// dependencies
var async = require('async');
var AWS = require('aws-sdk');
var gm = require('gm')
.subClass({ imageMagick: true }); // Enable ImageMagick integration.
var util = require('util');
// constants
var MAX_WIDTH = 100;
var MAX_HEIGHT = 100;
// get reference to S3 client
var s3 = new AWS.S3();
exports.handler = function(event, context, callback) {
// Read options from the event.
console.log("Reading options from event:\n", util.inspect(event, {depth: 5}));
var srcBucket = event.Records[0].s3.bucket.name;
// Object key may have spaces or unicode non-ASCII characters.
var srcKey =
decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, " "));
var dstBucket = srcBucket + "resized";
var dstKey = "resized-" + srcKey;
// Sanity check: validate that source and destination are different buckets.
if (srcBucket == dstBucket) {
callback("Source and destination buckets are the same.");
return;
}
// Infer the image type.
var typeMatch = srcKey.match(/\.([^.]*)$/);
if (!typeMatch) {
callback("Could not determine the image type.");
return;
}
var imageType = typeMatch[1];
if (imageType != "jpg" && imageType != "png") {
callback('Unsupported image type: ${imageType}');
return;
}
// Download the image from S3, transform, and upload to a different S3 bucket.
async.waterfall([
function download(next) {
// Download the image from S3 into a buffer.
s3.getObject({
Bucket: srcBucket,
Key: srcKey
},
next);
},
function transform(response, next) {
gm(response.Body).size(function(err, size) {
// Infer the scaling factor to avoid stretching the image unnaturally.
var scalingFactor = Math.min(
MAX_WIDTH / size.width,
MAX_HEIGHT / size.height
);
var width = scalingFactor * size.width;
var height = scalingFactor * size.height;
// Transform the image buffer in memory.
this.resize(width, height)
.toBuffer(imageType, function(err, buffer) {
if (err) {
next(err);
} else {
next(null, response.ContentType, buffer);
}
});
});
},
function upload(contentType, data, next) {
// Stream the transformed image to a different S3 bucket.
s3.putObject({
Bucket: dstBucket,
Key: dstKey,
Body: data,
ContentType: contentType
},
next);
}
], function (err) {
if (err) {
console.error(
'Unable to resize ' + srcBucket + '/' + srcKey +
' and upload to ' + dstBucket + '/' + dstKey +
' due to an error: ' + err
);
} else {
console.log(
'Successfully resized ' + srcBucket + '/' + srcKey +
' and uploaded to ' + dstBucket + '/' + dstKey
);
}
callback(null, "message");
}
);
};
```
### Part 4: Programming Model
1. Handler
- Handler is the function AWS Lambda calls to start execution of your Lambda function
- It always contains 2 input parameter:
(1) input object: often in a dictionary(key-value) data type
(2) context object
2. Context
- Your code can **interact with AWS Lambda** through the **method or attribute** of the context object
- Some important info. including the follows
(1) **Remaining execution time** until Lambda terminates the function
--> 因為有timeout限制，所以顯得特別重要
(2) Function name, function version, memory limit, request id, etc.
3. Logging
- the logging statement from AWS Lambda is written to **CloudWatch**
- The log can be found from CloudWatch logs, AWS Lamda console and HTTP response header
4. Exception
- If your Lambda function raises an exception, AWS Lambda recognizes the failure and **Serializes the exception info. into JSON** and returns it
- The return msg is reterned to client according to the type of invocation.(Synchronous or asynchronous)
- 在某些event source情況下，發生exception後AWS Lambda有可能重新執行
Sign in with Wallet
Connect another wallet