### More on Backend Utilities --- npm, bcrypt, Babel, RESTful, and Node.js/Express.js Again!  Spring 2019 。 Ric Huang --- ### Disclaimer Web Programming 的學習一但開始走到後端，就會逐漸脫離只是「學習程式語言」的範疇，而開始要學習許多網路服務相關的知識、架構、工具、甚至是生態、歷史等等。 Again, 這部分非常的多且雜，但網路上的學習資源也非常的多，所以重點是各位要有求知的企圖心，才能自我從找資料、動手實驗、挑戰專題... 這個循環中獲得真正的成長。 --- ### npm ([Node Package Management](https://docs.npmjs.com/)) * The world’s largest software registry * Download packages, create organizations, form virtual teams, find other developers... * To share your packages or use its advanced features, you need to register an npm account: http://www.npmjs.com/~yourusername * Command Line Interface ([CLI](https://docs.npmjs.com/cli-documentation/)) ---- ### Packages in npm * A package is: 1. a folder containing a program described by a package.json file 2. a gzipped tarball containing (1) 3. a url that resolves to (2) 4. a \<name>@\<version> that is published on the registry (see npm-registry) with (3) 5. a \<name>@\<tag> (see npm-dist-tag) that points to (4) 6. a \<name> that has a “latest” tag satisfying (5) 7. a \<git remote url> that resolves to (1) ---- ### $ npm help ```bash Usage: npm <command> where <command> is one of: access, adduser, audit, bin, bugs, c, cache, ci, cit, clean-install, clean-install-test, completion, config, create, ddp, dedupe, deprecate, dist-tag, docs, doctor, edit, explore, get, help, help-search, hook, i, init, install, install-ci-test, install-test, it, link, list, ln, login, logout, ls, org, outdated, owner, pack, ping, prefix, profile, prune, publish, rb, rebuild, repo, restart, root, run, run-script, s, se, search, set, shrinkwrap, star, stars, start, stop, t, team, test, token, tst, un, uninstall, unpublish, unstar, up, update, v, version, view, whoami ``` ---- ### $ npm init * create a package.json file * Generate a plain old package.json using legacy init: ```bash $ mkdir my-npm-pkg && cd my-npm-pkg $ git init $ npm init # If you want to skip questions, use "npm init -y" ``` * Create a React-based project ```bash $ npm init react-app ./my-react-app ``` ---- ### Do this... ```bash= mkdir webpack-test1 && cd webpack-test1 git init npm init -y ls -al ``` * What do you see? Take a look at 'package.json' ---- ### $ npm install * This command installs a package, and any packages that it depends on. If the package has a package-lock (representation of a dependency tree subsequent installs are able to generate identical trees, regardless of intermediate dependency updates) or shrinkwrap (publishable package-lock) file, the installation of dependencies will be driven by that, with an npm-shrinkwrap.json taking precedence if both files exist. ---- ### $ npm install ```bash # alias 'npm i' npm install (with no args, in package dir) npm install [<@scope>/]<name> npm install [<@scope>/]<name>@<tag> npm install [<@scope>/]<name>@<version> npm install [<@scope>/]<name>@<version range> npm install <git-host>:<git-user>/<repo-name> npm install <git repo url> npm install <tarball file> npm install <tarball url> npm install <folder> ``` ---- ### Do this... ```bash= npm install webpack webpack-cli --save-dev ls -al ``` * What are the new files? * Check 'package.json' again * See 'package-lock.json' ---- ### npm package dependencies * By default, **npm install** saves any specified packages into *dependencies*. Additionally, you can control where and how they get saved with some additional flags: ```bash -P, --save-prod: Package will appear in your dependencies. This is the default unless -D or -O are -D, --save-dev: Package will appear in your devDependencies. -O, --save-optional: Package will appear in your optionalDependencies. --no-save: Prevents saving to dependencies -S, --save(deprecated after npm 5.0): Saved to dependencies ``` * Why '--save-dev' for webpack in the previous page? ---- * Note, **npm install** installs the package as specified in *package.json* or *package-lock.json* * With the **--production** flag, npm will not install modules listed in devDependencies. * **npm install \<packageName>** installs \<packageName> in the local directory * 'package.json' and 'package-lock.json' will be updated * With **-g** or **--global** flag, it installs as a global package. ---- ### Other Options for npm install * The **--dry-run** argument will report in the usual way what the install would have done without actually installing anything. * The **--package-lock-only** argument will only update the package-lock.json, instead of checking node_modules and downloading dependencies. * The **-f** or **--force** argument will force npm to fetch remote resources even if a local copy exists on disk. ---- ### $ npm run-script <command> * alias 'npm run' * This command runs an arbitrary command from 'package.json: scripts' object. If no "command" is provided, it will list the available scripts. * **test**, **start**, **restart**, and **stop** can be run directly (e.g. 'npm start') ---- ### $ npm audit * Scan your project for vulnerabilities and automatically install any compatible updates to vulnerable dependencies: ```bash $ npm audit fix ``` ---- ### $ npm link * Create symbolic link from one package to another ```bash cd ~/projects/package-dir # go into the package directory npm link # creates global link cd ~/projects/linked-dir # go into some other package directory. npm link packageName # link-install the package # packageName is what specifed in 'package.json: name' ``` * The above is the same as: ```bash cd ~/projects/link-dir npm link ../packageName ``` ---- ### Other npm commands ```bash $ npm ls # list installed packages $ npm publish # Publishes a package to the registry so that it can be installed by name $ npm uninstall # uninstall package $ npm unpublish # Remove a package from the registry $ npm update # Update a package $ npm version # Bump a package version ``` ---- ### **nvm** vs **npm**? * nvm --- Simple bash script to manage multiple active node.js versions * Alternatively, there is an interactive node.js version manager: [**n**](https://github.com/tj/n) // created by 'tj' 大神 --- ### A Simple Node Project ```bash= mkdir simple-node-example && cd simple-node-example git init npm init -y ``` ```bash= # edit 'src/index.js'; put a console.log() node src/index.js # edit 'package.json'; add "start": "nodemon src/index.js" npm start # Try to modify 'src/index.js' to see what happens ``` --- ### Introducing Babel ---- ```bash= npm install @babel/core @babel/node @babel/preset-env --save-dev # # Modify 'package.json' with: # "start": "nodemon --exec babel-node src/index.js", # # edit a file '.babelrc'; add the following: # { # "presets": [ # "@babel/preset-env" # ] # } # npm start # Nothing changes, but now you run node.js with babel # ``` --- ### Node.js web applications * It's very common that your application (frontend) needs to read something from the server (e.g. username/password, portfolio, files/DB...), and thus your server (backend) needs to respond it through certain modules/functions or web APIs. * There are many distinct ways to do this. We will cover them with different examples. ---- ### Environment Variables in Node.js * An environment variable is a dynamic-named value that can affect the way running processes will behave on a computer. * Common environment variables for shell: $HOME, $PATH, $LD_LIBRARY_PATH * You can also define your own environment variables in certain configuration files * Conventionally, make it hidden (i.e. .xxx) ---- ### Node.js package 'dotenv' * Allow Node.js source code to access the environment variables defined in **'.env'** ```bash npm install dotenv # Create a file '.env'; add this line MY_SECRET=mysupersecretpassword ``` ```javascript // In 'src/index.js', change it to: import 'dotenv/config'; import saySomething from './my-other-file.js'; ``` ```javascript // Create a file 'src/my-other-file.js'. Add console.log("Hello Ric"); console.log(process.env.MY_SECRET); ``` * **npm start** to see what happens! ---- ### Use 'bcrypt' to encrypt password * Of course, password needs to be encrypted to be safe. **'bcrypt'** is a popular node.js package that has 200~300K downloads per week! ```bash= npm install bcrypt ``` ---- ### A simple bcrypt test * Change 'src/index.js' to: ```javascript const bcrypt = require('bcrypt'); const saltRounds = 10; const myPassword = 'password1'; const testPassword = 'password2'; const myHash ='$2a$10$fok18OT0R/cWoR0a.VsjjuuYZV.XrfdYd5CpDWrYkhi1F0i8ABp6e'; // for test purpose bcrypt.hash(myPassword, saltRounds).then(function (hash) { console.log(hash); }); // Verify password bcrypt.compare(myPassword, myHash).then(function (res) { console.log(res); }); bcrypt.compare(testPassword, myHash).then(function (res) { console.log(res); }); ``` ---- * Of course, in real-life application, we should: 1. In 'src/index.js' (view), submit 'username/password' ```jsx ...render() { return ...<Form onSubmit={handleSubmit}>... } ``` 2. In backend, receive 'username/password' and compare it with the user data in DB * If succeed, return a session token 3. In DB, store the username and encrypted password => We will cover this later ---- ### Recall: Express.js * Change 'src/index.js' to: ```javascript import express from 'express'; const app = express(); app.get('/', (req, res) => { res.send('Hello World!'); }); app.listen(3000, () => console.log('Example app listening on port 3000!'), ); ``` * **npm start** and open localhost:3000 ---- * Of course, you can combine 'dotenv' and 'express' in the same example... ```bash # Add this line to .env PORT=3000 ``` ```javascript // Modify index.js const port = process.env.PORT; app.listen(port, () => console.log('Example app listening on port ' + port + '!') ); ``` --- ### Towards a complete web service * 到目前為止，我們可以透過 npm 啟動一個 node.js server, 並且有 Babel 協助做到跨瀏覽器、跨平台的 js 支援，然後我們利用 express 來進行 web application, 來讓前端的頁面/程式，可以將資料/運算的需求，透過某種網路 protocol (e.g. HTTP), 傳到適當的接口 (i.e. API)，以完成前端與後端的溝通。 * 接下來，我們會在逐漸說明： * 如何建置這樣的 API (RESTful) * 如何使用各種 node.js 套件來完成後端的商業邏輯 * 如何導入資料庫，以及資料庫的查詢語言 (如 graphQL), 來進行更有效率的資料操作 --- ### Learning cURL * 要了解前端程式如何透過網路協定、API 來跟後端服務溝通，我們先介紹一個 Command line URL (cURL) 資料傳輸工具，讓你可以從 command line 下指令，來跟一個 server 透過 URL 以及某種 protocol (e.g. HTTP) 來存取資料。 ```bash # for Mac user, install cURL by: brew install curl ``` ---- ### cURL to express.js * **npm start** the express service in previous slides * **curl http://localhost:3000** to see what happens! * Try to modify **index.js** and play around! --- ### RESTful APIs * API (application programming interface), 顧名思議就是定義好一個介面的協定來讓溝通的兩端得以**獨立的開發**，而不會受到任何一方改進、除錯的影響 * 在網路服務前後端的溝通，最有名、廣為大家遵循的 APIs 就是 RESTful APIs * 不過嚴格來說，RESTful 並不是一個 API 的規範，而是一種 **style** ---- ### REST * 表徵性狀態傳輸 (Representational State Transfer) * 2000 年就由 Roy Fielding 在博士論文提出 * 但在 Rails 1.2 實作後才紅起來 * 是設計「風格」而不是標準 * 符合 REST 設計風格的 Web API 稱為 RESTful API ---- ### RESTful * 資源是由 URI 指定 * 對資源的操作包括獲取、創建、修改、刪除，剛好對應 GET、POST、PUT、DELETE * 依照不同需求給予不同格式的回應 * 利用 Cache 增加性能 * 正確的 HTTP status code ---- ### 資源是由 URI 指定 * Bad examples ```bash /getDevice /getDeviceList /getDevices /getMyDevice /getOtherDevice /updateDeviceList /addNewDevice ``` * Why bad? * 不統一，不好記憶，重寫會不一樣 * get / post 無法看出來 * 參數如何傳遞？ ---- ### RESTful APIs ```bash GET /devices POST /devices/ GET /devices/123 PUT /devices/123 DELETE /devices/123 ``` * Why better? * 不同人依舊會實作相同 API * 清楚表示資源關係 ---- ### Examples of REATful APIs ```bash GET http://stackoverflow.com/questions POST http://stackoverflow.com/questions/389169 GET http://stackoverflow.com/tags PUT http://stackoverflow.com/users DELETE http://stackoverflow.com/users/3012290 ``` ---- ### RESTful with parameters - Filter - Sorting - Searching ```bash GET /tickets?q=return&state=open&sort=-prority,created_at ``` ---- ### HTTP 動詞 ```bash 取得資源清單 -> GET /resources 新增資源 -> POST /resources 取得單一資源 -> GET /resources/:id 修改單一資源 -> PUT /resources/:id 刪除單一資源 -> DELETE /resources/:id 更新資源部份內容 -> PATCH /resources 只回傳HTTP header -> HEAD /resources/:id ``` ---- ### URI 名詞 * 相對於 HTTP 動詞，URI 就是名詞了。URI 由 prefix + API endpoint 組成。Prefix的部份可有可無，例如/api或/api/v1，API endpoint的設計，幾個重要原則如下([ref](https://tw.twincl.com/programming/*641y))： * 一般資源用複數名詞，例如 /books 或 /books/123 * 唯一資源（亦即對client而言只有一份的資源）用單數名詞。例如 /user/subscriptions，其中 user 是指目前驗證的使用者，所以用單數 * 資源的層級架構，可以適當反應在 API endpoint 設計上，例如 /books/123/chapters/2 * URI components 都用小寫 ---- ### 回傳狀態碼 * API回傳的結果，應使用適當的HTTP狀態碼 * 1xx -> 訊息 * 2xx -> 成功 * 3xx -> 重導向 * 4xx -> 用戶端錯誤 * 5xx -> 伺服器端錯誤 ---- ### 常見 Status Codes ```bash 200 OK -- 301 Moved Permanently 302 Found 304 Not Modified -- 400 Bad Request 401 Unauthorized 403 Forbidden 404 Not Found 405 Method Not Allowed 408 Request Timeout -- 500 Internal Server Error 502 Bad Gateway 503 Service Unavailable 504 Gateway Timeout ``` ---- ### HTTP Header * 用戶端送出 API 請求時，要帶一些 http header 讓 server 端了解目的 * Authorization: 認證資訊 * Accept: 能夠接受的回應內容類型 (Content-Type)，屬於內容協商的一環 ```bash # 用 Accept 跟 Server 溝通格式 接收 純文字 -> Accept: text/plain 接收 HTML -> Accept: text/html 接收 JSON -> Accept: application/json ``` * 至於API回傳結果的HTTP header，沒甚麼特別之處，按照一般原則處理即可（例如Content-Type、Content-Length、ETag、Cache-Control…）。 ---- ### HTTP Body: JSON, 純文字，或XML格式 * 善用一些 node.js 套件，例如：body-parser ---- ### 結論：RESTful 是風格，不是規範 * 可以適當放寬設計 (或稱之為：RESTish) * 避免[工程師的鄙視鏈](https://vinta.ws/blog/695) --- ## cURL, express, RESTful ---- ### Recall: Types of Middleware in Express * Application-level middleware * Router-level middleware * Error-handling middleware * Built-in middleware * Third-party middleware ---- ### 基本的 Express Middlewares ```javascript var express = require('express'); var app = express(); app.use([path,] callback [, callback...]); app.METHOD(path, callback [, callback ...]); // METHOD can be: // checkout copy delete get head lock // merge mkactivity mkcol move m-search // notify options patch post purge put // report search subscribe trace unlock // unsubscribe ``` ---- ### Add these to your index.js ```javascript app.get('/', (req, res) => { return res.send('Received a GET HTTP method'); }); app.post('/', (req, res) => { return res.send('Received a POST HTTP method'); }); app.put('/', (req, res) => { return res.send('Received a PUT HTTP method'); }); app.delete('/', (req, res) => { return res.send('Received a DELETE HTTP method'); }); ``` ---- ### Try these cURL commands ```bash curl http://localhost:3000 curl -X POST http://localhost:3000 curl -X PUT http://localhost:3000 curl -X DELETE http://localhost:3000 ``` * By default cURL will use a HTTP GET method. * However, you can specify the HTTP method with the -X flag (or --request flag) ---- ### Try with differnt routing and parameters * Change route from / to /users ```javascript app.post('/users', (req, res) => { return res.send('POST HTTP method on users resource'); }); app.put('/users/:userId', (req, res) => { return res.send( `PUT HTTP method on users/${req.params.userId} resource`, ); }); ``` * Try cURL ```bash curl -X POST http://localhost:3000/users curl -X PUT http://localhost:3000/users/12 ``` ---- ### Reading data through RESTul APIs * Let's put some fake data in index.js (we will link with real DB and perform query later) ```javascript let users = { 1: { id: '1', username: 'Robin Wieruch', }, 2: { id: '2', username: 'Dave Davids', }, }; let messages = { 1: { id: '1', text: 'Hello World', userId: '1', }, 2: { id: '2', text: 'By World', userId: '2', }, }; ``` ---- ### Reading data through RESTul APIs * Modify these express middleware ```javascript app.get('/users', (req, res) => { return res.send(Object.values(users)); }); app.get('/users/:userId', (req, res) => { return res.send(users[req.params.userId]); }); app.get('/messages', (req, res) => { return res.send(Object.values(messages)); }); app.get('/messages/:messageId', (req, res) => { return res.send(messages[req.params.messageId]); }); ``` * Use cURL to test! ---- ### Sending data through RESTul APIs * Different from reading, when sending data through APIs, in addition to the URL, we also need to extract the body of the payload, and specify the identity of the data. This identity can be user ID, message ID, etc. * The 'body-parser' module helps parse the body part out of the payload * To mimic the reality, here we use an ID generator module *uuid* ```bash npm install body-parser npm install uuid ``` ---- * Modify 'index.js' as: ```javascript import bodyParser from 'body-parser'; import uuidv4 from 'uuid/v4'; ... app.use(bodyParser.json()); app.use(bodyParser.urlencoded({ extended: true })); ... app.post('/messages', (req, res) => { const id = uuidv4(); const message = { id, text: req.body.text, }; messages[id] = message; return res.send(message); }); ``` <small> * bodyParser.json(): Parses the text as JSON and exposes the resulting object on req.body.<br> * bodyParser.urlencoded(): Parses the text as URL encoded data and exposes the resulting object on req.body. </small> ---- * Try it with cURL!!! ```bash curl -X POST -H "Content-Type:application/json" http://localhost:3000/messages -d '{"text":"Hi again, World"}' ``` - where -H flag – that’s how we are saying we want to transfer JSON – and data as payload with the -d flag. ```bash curl http://localhost:3000/messages ``` * to see whether this new message has been update to *messages* ---- ### Completing this example * Add this to 'index.js' ```javascript app.use((req, res, next) => { req.me = users[1]; next(); // Why "next()"? }); ``` * Modify the post method as: ```javascript app.post('/messages', (req, res) => { const id = uuidv4(); const message = { id, text: req.body.text, userId: req.me.id }; messages[id] = message; return res.send(message); }); ``` ---- ### How about DELETE? (try it yourself!) --- ### What's next? What you should study? * Connecting React app with Express * Make good use of *fetch* module * Use *promise* or *async/await* to handle asynchronizations * Handle DB queries * Redux * GraphQL --- ### That's it! Good luck on your midterm project!!