# GCP Examples
## Example EX. (Date)
#### Project-Name
#### Demo & Processing
#### Key Technology
-----------------------
## Example EX.1 (2024/10/01)
#### Project-Name
+ 手動創建Web-Server
#### Demo & Processing
<details>
<summary>創建web須擬機</summary>
+ step1. 透過UI創建VM
 
+ step2. SSH連線VM
- 直接連線([3種SHH額外方式](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?both#SSH-Connect-to-VM-GCP-in-3-way))
+ step3. 終端機指令
```bash=0
安裝apache2並檢測server啟動狀態
$ sudo apt update
$ sudo apt install apache2 -y
$ sudo systemctl status apache2
$ sudo apt install net-tools
$ sudo netstat -tulnp | grep 80
```
+ step4. 測試連線
- 執行指令
```bash
curl http://[apache2 server ip]/
```
- 結果
+ step5. 建立網頁
- 指令
```bash=0
第三行展示錯誤的寫入html方法 (原因 : 這是一個多指令組合的指令，因此sudo只會允許第一個指令使用root，這就是為甚麼第三行會出現permission denied)
$ cd /var/www/html
$ sudo echo "RRRRRRRRRR" > markerpen_yelling.htm
-bash: markerpen_yelling.htm: Permission denied
$ sudo bash -c 'echo "RRRRRRRRRR" > markerpen_yelling.htm'
```
- 指令解說 `bash -c`
`bash` 是一個 Unix Shell，而 `-c` 參數告訴 `bash` 去執行後面引號中的命令字符串，並把它作為完整的腳本來運行。
- 指令解說 `sudo -l`
```bash=0
作用 : 可以確認自己可以使用到sudo的哪些指令
$ sudo -l
Matching Defaults entries for mingchu_chou92 on webtesting1:
env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin, use_pty
User mingchu_chou92 may run the following commands on webtesting1:
(ALL : ALL) NOPASSWD: ALL (這代表有sudo的最高權限)
```
- hostname -I
```bash=0
可以利用這項技術實現多台Server之間的IP管理
$ sudo bash -c 'echo "$(hostname -I)" > hostip.htm'
$ curl http://34.81.100.236/hostip.htm
10.140.0.3 (VM會把自己的IP顯示出來)
```
</details>
#### Key Technology
+ [SSH](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#SSH-Connect-to-VM-GCP-in-3-way)
+ [Basic VM Building](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Vitual-Machine20240924)
## Example EX.2 (2024/10/01)
#### Project-Name
+ 自動(即剛開機時)創建Web-Server
#### Demo & Processing
<details>
<summary>創建web須擬機</summary>
+ step1. 透過UI創建VM (先不要點create，點選下方的Advanced Option當中的Management)
 
+ step2. 透過UI創建VM (寫入自動執行腳本 + create VM)
- 腳本
```bash=0
這個腳本會在VM開機的後執行腳本
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>
```
</details>
#### Key Technology
+ [Example EX.1 (2024/10/01).](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX1-20241001)
## Example EX.3 (2024/10/08)
#### Project-Name
+ 名稱
建立本地網頁，運行於GCP平台
[會有permission denied的問題，導致stroage放到GCE的時候出現無法複製的問題]
[permission會因為service account導致這個問題發生]
+ 架構圖
#### Demo & Processing
<details>
<summary>GCP運行本地網頁操作過程</summary>
+ step1. 在本地端創建一個網頁
+ step2. 創建 Cloud Storage 內部的 Bucket + 上傳資料
- Bucket上傳方法1
* 找到 `Cloud Storage`
* 創建 `Bucket`
* 上傳檔案
- Bucket上傳方法2
* 先創建 `Bucket`
大致操作都一樣，但是可以嘗試不要使用multui-region比免花太多錢
* 上傳資料到 `GCE Shell` 上
請記得這個shell是GCP提供給user的特殊須擬機，他有5GB的儲存空間，儘管我們把機器reboot資料仍然會在裡面不會消失，除非我們刪除
 
* 從 `GCE Shell` 上傳到 `Bucket`
```bash=
從 GCE Shell 上傳到 Bucket
$ gsutil ls gs:// (顯示目前有哪些bucket)
$ gsutil cp [args] [fileA] [Bucket-Location] (Bucket位置可以從上一個指令獲得)
```
* 檢查是否有上傳成功
+ step3. 把Bucket的資料移到GCE的VM內
- 先跟以前一樣創建VM(ubuntu的虛擬機)
但是這個要特別注意，如果有其他project存在的話，會因為這個選項會導致我們沒辦法把東西移動過去(不過這都是以後再說的了...)
- SSH連線
```bash=0
安裝伺服器 + 架設伺服器
$ sudo apt update
$ sudo apt install apache2 -y
$ gsutil cp -r gs://marktestingbucket2-markwatchcloud .
$ sudo cp -r marktestingbucket2-markwatchcloud/ /var/www/html/
```
- 檢查
</details>
#### Key Technology
+ [Build Cloud Stroages](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?both#GCP-Build-Cloud-Storage)
## Example EX.4 (2024/10/08)
#### Project-Name
+ Project-Name
Storage逆向回傳
+ 架構
Example3展是怎麼從本地->Storages->VM
現在展示 VM -> Storages -> 本地
#### Demo & Processing
+ 運行指令
```baah=1
$ echo "hahahahahaha" > markerpenlaughing.htm
$ gsutil ls gs://
gs://marktestingbucket1-markwatchcloud/
gs://marktestingbucket2-markwatchcloud
$ gsutil cp markerpenlaughing.htm gs://marktestingbucket1-markwatchcloud/
```
+ 報錯訊息 (Service Account權限不足)
+ 解決方法
<details>
<summary>解決方法</summary>
+ 解決方法1.
+ 解決方法2.
- 創建IAM帳戶 (根據林廣哲所說，這個步驟似乎並不需要)
- 調整GCE的VM設定 (需要回到GCE暫停VM才可以調整設定)
- 將VM的Service Acocunt修改
- 修改權限
也要把`Sotrage`也改成可讀可寫
- 進入ssh修改系統
刪除原先的permission設定快取，否則系統會優先讀取這個設定
指令 : `rm -rf .gsutil/`
- 檢查
</details>
#### Key Technology
+ [Example EX.3](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX3-20241008)
## Example Ex.5 (2024/10/15)
#### Project-Name
+ project-name
建立DB，讓使用者連線Server進一步訪問DB
+ 架構
#### Demo & Processing
+ step1. server 建置
<details>
<summary>創建虛擬機-圖形化操作</summary>
+ step 1-1
 
+ step 1-2
 
```html=0
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>
```
</details>
+ step2. DB 建置
<details>
<summary>DB 建置</summary>
+ step2-1. 建立一個ubuntu22.04 + 不需要防火牆規則的VM
基本配置都跟VM建置一樣，不再贅述
+ step2-2. 進入DB虛擬機的ssh下載mariadb
```bash=0
按照下面指令以及程式碼輸入
$ sudo apt-get install apt-transport-https curl
$ sudo curl -o /etc/apt/keyrings/mariadb-keyring.pgp 'https://mariadb.org/mariadb_release_signing_key.pgp'
$ sudo vim /etc/apt/sources.list.d/mariadb.sources
$ sudo cat /etc/apt/sources.list.d/mariadb.sources
# MariaDB 11.1 repository list - created 2023-11-08 06:16 UTC
# https://mariadb.org/download/
X-Repolib-Name: MariaDB
Types: deb
# deb.mariadb.org is a dynamic mirror if your preferred mirror goes offline. See https://mariadb.org/mirrorbits/ for details.
# URIs: https://deb.mariadb.org/11.1/ubuntu
URIs: https://ftp.ubuntu-tw.org/mirror/mariadb/repo/11.1/ubuntu
Suites: jammy
Components: main main/debug
Signed-By: /etc/apt/keyrings/mariadb-keyring.pgp
$ sudo apt-get update # 更新
$ sudo apt install mariadb-server # 安裝 MariaDB
$ sudo systemctl status mariadb # 安裝完服務就已經啟動了，可以再檢查一下
$ sudo mysql_secure_installation # 初始化 MariaDB
Enter current password for root (enter for none): 直接按下Enter
Switch to unix_socket authentication [Y/n] : 選擇n
Change the root password? [Y/n] : 看個人想換甚麼密碼
Remove anonymous users? [Y/n] : 考量安全選擇y
Disallow root login remotely? [Y/n] : y
Remove test database and access to it? [Y/n] : y
Reload privilege tables now? [Y/n] : y
```
+ step2-3. 檢查DB狀態
+ step2-4. 嘗試登入 + 建立資料庫
- 終端指令
`sudo mysql -u root -p`
- DB指令
```db=
create database markdb;
use markdb;
create table addrbook(name varchar(50) not null, action char(20));
insert into addrbook(name,action) values ("mark","yelling:RRRRRRRR");
select name,action from addrbook;
```
+ step2-5. 修改DB配置，使其允許其他IP登入
- 下載net-tool 並解查 mariadb的port號
```bash=0
檢查DB
sudo apt install net-tools (記得要先exit離開DB)
sudo netstat -tunlp | grep mariadb
結果 : 127.0.0.1:3306 -> 表示只能連接到localhost才能到3306port存取DB
```
為了讓Server可以連接到db，我們會希望設定是`0.0.0.0:[對應port號]`
- 修改db配置
```bash=0
$ sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf
找到bind-address並改成0.0.0.0
$ sudo systemctl restart mariadb
```
- 仍然無法存取的原因及結果
```bash=0
但是DB的root本身仍不允許除了localhost以外的人登入
因此儘管看到mariadb的服務變成`0.0.0.0:3306`
依樣無法用其他ip登入root
mingchu_chou92@mdb-server:~$ mysql -h 10.140.0.8 -u root -p
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Enter password:
ERROR 2002 (HY000): Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1130 - Host 'mdb-server.asia-east1-a.c.imposing-union-436602-v2.internal' is not allowed to connect to this MariaDB server
```
- 打開DB允許哪些IP存取的權限
```bash=0
終端畫面
mingchu_chou92@mdb-server:~$ mysql -u root -p
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 32
Server version: 11.1.6-MariaDB-ubu2204 mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.140.0.8' IDENTIFIED BY 'mark' WITH GRANT OPTION;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'mark' WITH GRANT OPTION;
Query OK, 0 rows affected (0.001 sec)
MariaDB [(none)]> FLUSH PRIVILEGES; (更新)
GRANT ALL PRIVILEGES ON *.* TO '[使用者]'@'[允許的IP位置]' IDENTIFIED BY 'mark' WITH GRANT OPTION; (其中 `%` 可以表示 `任意的IP位置都被允許`)
```
結果: 
</details>
+ step3. 建立server與db之間的連線
<details>
<summary>圖形化操作 + 程式碼</summary>
- step3-1. 打開server的ssh
- step3-2. 嘗試ping db檢查連線
- step3-3. 在server端安裝maria-db的client程式
`$ sudo apt install mysql-client`
- step3-4. 嘗試登入DB
- step3-5. 建立防火牆
請注意因為這台db跟server的VM都使採用default的網路，因此一定要填寫default network
限制網路只限制在台灣的IP
將防火牆設定加入db當中
建立一台不在台灣的VM，並嘗試連線看看
- step3-6. 建立php後端程式向db發送請求
檢查`sudo dpkg -l | grep apache2`
```bash=0
終端畫面
$ sudo apt install software-properties-common
$ sudo add-apt-repository ppa:ondrej/php
$ sudo apt update
$ sudo apt upgrade -y
$ sudo apt install php8.1 libapache2-mod-php8.1 php8.1-gd php8.1-mysql php8.1-curl php8.1-mbstring php8.1-intl -y
$ sudo apt install php8.1-gmp php8.1-bcmath php8.1-imagick php8.1-xml php8.1-zip -y
$ sudo a2enmod php8.1
$ sudo systemctl restart apache2
<?php
phpinfo();
?>
```
```bash=0
終端畫面
mingchu_chou92@www-server:~$ vim test.php
mingchu_chou92@www-server:~$ cat test.php
<?php
$servername="10.140.0.8";
$username="root";
$password="mark";
$dbname="markdb";
$conn = new mysqli($servername, $username, $password, $dbname);
if($conn->connect_error){
die("connection failed: " . $conn->connect_error);
}
else{
echo "connect OK!" . "<br>";
}
$sql="select name,action from addrbook";
$result=$conn->query($sql);
if($result->num_rows>0){
while($row=$result->fetch_assoc()){
echo "name: " . $row["name"] . "\taction: " . $row["action"] . "<br>";
}
} else {
echo "0 record";
}
?>
mingchu_chou92@www-server:~$ sudo cp test.php /var/www/html/
```
- 結果
</details>
#### Key Technology
+ [GCP Build SSH Connection Between VMs](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?both#GCP-Build-SSH-Connection-Between-VMs)
+ [Example EX.2](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX2-20241001)
## Example EX6. (2024/10/22)
#### Project-Name
+ 基於自創VPC網路下設立獨立防火牆規則
#### Demo & Processing
+ step1. Build `VPC Network` and `VMs`
<details>
<summary>Build VPC (subnet/firewall/route)</summary>
+ Create VPC Network
+ Set VPC Network's firewall
- XXXX-allow-custom
這條規則表示內網的機器無論任何服務在沒有匹配其他規則的前提下都可以通過
+ Set VPC Newtork's Route Rule
+ Set the costum VPC
+ Set Costum VPC's subnet/firewall/route rules
+ Change Costum VPC's firewall rules
- 可以自己設定是要allow或是deny又或是Priority數值
- 檢查
</details>
<details>
<summary>Build VM1 with VPC1 + VM2 with VPC2</summary>
+ 建立VM1，並安裝上自己建立的VPC
+ 建立VM2，大致跟VM1相同，但是選擇VPC2 (VPC根VM的region要相同哦...)
</details>
+ step2. 建立雙向(VPC1與VPC2的雙向) VPC Peering
<details>
<summary>Build VPC Peering</summary>
+ 進入VPC Peering介面
+ 建立第一條 VPC Peering設定
- 可以發現建立完處於inactive的狀態是因為只有建立單向
- 所以建立另一向的VPC Peering以避免inactive狀態
</details>
+ 確認兩台VM是否可以互相用ICMP/HTTP/SSH通訊 | [SSH Between VMs](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?both#GCP-Build-SSH-Connection-Between-VMs)
<details>
<summary>results</summary>
+ ICMP
+ HTTP
+ SSH
</details>
#### Key Technology
+ [VPC knowledge](https://hackmd.io/JBPiZvHGS26quqcKaiKtPg?both#GCP%E7%9A%84VPC%E7%B6%B2%E8%B7%AF-20241015)
+ [VPC peerinng](https://hackmd.io/JBPiZvHGS26quqcKaiKtPg?both#VPC-Peering-20241022)
+ [default network setting](https://hackmd.io/JBPiZvHGS26quqcKaiKtPg?both#How-Does-Default-Firewall-works-20241022)
## Example EX.7 (2024/10/29)
#### Project-Name
+ 創建SQL + 限制Instance可以存取
#### Demo & Processing
<details>
<summary>CMD操作</summary>
+ 創建sql並且是private ip (default VPC) + 打開sql的admin管理(安裝SQL可以參考[這裡](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-DataBase-20241029))
+ 創建一個Instance + Ubuntu20.04作業系統 + default VPC設置
+ ssh進入Instance並安裝mysql的client端軟體(要怎麼SSH可以參考[這邊](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-SSH-Connection-Between-VMs-20241015))
+ 在Client端(沒有安裝sql-server的那台VM)安裝SQL-Client軟體 + 嘗試登入建立好的sql-server
```
$ sudo apt install mysql-client -y
$ mysql -h 'sql public ip' -u root -p
```
+ 在DB內部建立檔案
```sql=
create database markdb;
use markdb;
create table addrbook(name varchar(50) not null, action char(20));
insert into addrbook(name,action) values ("mark","yelling:RRRRRRRR");
insert into addrbook(name,action) values ("markerpen","laughing:HAHAHAHA");
select name,action from addrbook;
```
+ 回到instance並下載apache2以及php
```bash=
$ sudo apt install apache2 php libapache2-mod-php php-mysql -y
$ sudo systemctl restart apache2
$ sudo vim /var/www/html/mark.php
$ sudo cat /var/www/html/mark.php
<?php
$servername="10.56.224.3";
$username="root";
$password="123456";
$dbname="markdb";
$conn = new mysqli($servername, $username, $password, $dbname);
if($conn->connect_error){
die("connection failed: " . $conn->connect_error);
}
else{
echo "connect OK!" . "<br>";
}
$sql="select name,action from addrbook";
$result=$conn->query($sql);
if($result->num_rows>0){
while($row=$result->fetch_assoc()){
echo "name: " . $row["name"] . "\taction: " . $row["action"] . "<br>";
}
} else {
echo "0 record";
}
?>
```
</details>
+ 結果
#### Key Technology
+ [Build SQL](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-DataBase-20241029)
## Example EX.8 (2024/10/29)
#### Project-Name
+ 建立 `unmanaged load balancer` (這些server一定要在同樣的 region 跟 zone)
#### Demo & Processing
<details>
<summary>GUI操作</summary>
+ step1. 創建相同 Region&Zone 的兩台VM + 開機建立網頁伺服器 + Ubuntu作業系統
- 詳細可以參考 [EX.2](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX2-20241001)
- 兩台VM分別叫做 `www-mark1` & `www-mark2` + 地區都是 `Taiwan` & `asia-east1-c`
+ step2. 將要管理的VM加入到同一個 `Group` 底下
- step2-1. 切入 Instance groups + 點 `CREATE INSTANCE GROUP`
- step2-2. 創建 `unmanaged load-balancer`
+ step3. 創建 `Load-Balancer`
- step3-1. 進入 `Load Balancer(LB)` 頁面 + 創建LB
- step3-2. 創建LB種類
- step3-3. 創建LB的名稱+位置+網路
* 快速解法
* 正統解法
- step3-4. 創建LB的 `Frontend configuration`
- step3-5. 創建LB的 `Backend configuration`
- step3-6. 創建LB的 `Health Check`
</details>
+ 結果
#### Key Technology
+ [Load Balancer Concept (umanaged load balancer)](https://hackmd.io/JBPiZvHGS26quqcKaiKtPg?both#%E8%B2%A0%E8%BC%89%E5%9D%87%E8%A1%A1%E5%99%A8-%E7%A8%AE%E9%A1%9E%E4%BB%8B%E7%B4%B9)
+ [Ex2.](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX2-20241001)
## Example EX.9 (2024/11/12)
#### Project-Name
刪除web server的公有網路
#### Demo & Processing
<details>
<summary>GUI操作</summary>
+ step1. 照ex8創建整個網路架構
+ step2. 創建Firewall允許http存取load-balancer + 允許health-check存取load-balancer
- 允許http存取load-balancer
- 允許health-checkk存取load-balancer
* 請注意health-check的ip-range，這是GCP提供專門讓vpc可以跟heal-check連線的網段
+ step3. 修改GCE兩台VMs的設定
- 兩台VM的允許Http流量關閉 + 新增剛才創建的兩個Firewall tags
- 關閉External IP
+ step4. 檢查load-balance
</details>
+ 結果
#### Key Technology
+ [EX.8](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX8-20241029)
## Example EX10. (2024/11/12)
#### Project-Name
+ Create Managed Load-Balancer
#### Demo & Processing
<details>
<summary>GUI操作</summary>
+ step1. 創建template
- step1-1. 創建模板名稱(mark-template)
- step1-2. 選擇硬體配置
- step1-3. 選擇作業系統
- step1-4. 設定防火牆
- step1-5. 選擇網卡
- step1-6. 開機腳本
[開機腳本程式碼](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX2-20241001)
+ step2. 在GCE/Instance Group/ 創建instance-group
- step2-1. 創建Managed-Group
- step2-2. 創建Group的型態
- step2-3. 創建Group動態增加VM的規則
請注意`Minimum number of instances`至少要建立的VM數量 ; `Maximum number of instances`則表示系統最多建立的VMs數量
- step2-4. 創建Heal-Check(請注意這裡的Health-Check跟Load-Balancer的不一樣，這是專門為VM設立的)
- step2-5. 其餘都用default-value就好，然後點選Create
+ step3. 創建load-balancer
- step3-1. 創建load-balancer
- step3-2. 增加frontend-config
- step3-3. 增加backend-config
+ step4. 壓力測試
- step4-1. 開啟兩個由ManagedGroup所創建VM的SSH畫面(左邊執行壓力測試，右邊監測壓力)
- step4-2. 執行指令
左邊執行`cat /dev/zero > /dev/null` + 右邊執行`top`
</details>
+ result
#### Key Technology
+ [EX.8](https://hackmd.io/Avf11J7nSaWsdyk_oqPZRg?both#Example-EX8-20241029)
## Example EX11. (2024/11/19)
#### Project-Name
+ 透過load balancer將http服務轉換成https
#### Demo & Processing
<details>
<summaary>操作過程</summaary>
+ step1-1. 創建unmanaged instance group
+ step1-2. 在[DNS](https://dynv6.com/)網站建立一個Domain-Name
+ step2. 創建Load-Balancer
+ step3. Frontend configuration
+ step4. Backend configuration + Health-Check
</details>
## Example EX12. (2024/11/26)
#### Project-Name
+ Build Load-Balance to contain to apache-servers template without public ip
#### Demo & Processing
<details>
<summary>GUI操作</summary>
+ step1. 在VPC當中加入允許health-check的防火牆規則，並確認是否有允許Http(網頁伺服器)以及SSH(debug用)的規則
+ step2. 因為 apache-server 沒有 public ip，為了讓 apache-server 可以安裝網頁軟體，所以我們需要NAT設置讓 server 不用 public ip 也可以到公網
+ step3. 創建一個不用 public ip 的 template
```bash=0
這個腳本會在VM開機的後執行腳本
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>
```
+ step4. 創建instance group
+ step5. 創建load balancer
</details>
+ 結果
#### Key Technology
+ [Instance-Template](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Instance-Template)
+ [NAT-Gateway](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#Build-NAT-for-private-VM-20241119)
## Example EX.13 (2024/11/26)
#### Project-Name
+ Build Load Balancer with different region
+ 架構圖
#### Demo & Processing
<details>
<summary>創建VPC</summary>
- 額外新增Firewall Rule (health-check)
- 額外新增Firewall Rule (http)
- 加入NAT(給第一個region)
- 新增一個給別的region的subnet
- 加入NAT(給第二個region)
</details>
<details>
<summary>創建Group</summary>
+ 在第一個region創建兩台vm
```bash=0
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>
```
+ 在第二個region創建兩台vm(基本操作跟上面依樣，只有下面幾個操作略微不同)
```bash=0
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) . In US</p></body></html>
```
- SSH進入 + 執行以下指令
```bash=0
這個資料夾名稱，跟創建LB的時候有關聯很重要，請記住
$ cd /var/www/html/
$ sudo bash -c 'mkdir us; mv index.html us/'
```
+ 創建兩個groups
</details>
<details>
<summary>創建Load-Balancer</summary>
+ Frontend 設置
+ Backend 設置 (給region1)
+ Backend 設置 (給region2)
Healtth-Check共用
+ 創建Domain name
+ 新增對應的path
+ 修改Domain name
</details>
+ 結果
## Example EX14. (2024/11/26)
#### Project-Name
+ Internal Load-Balancer
+ 架構圖
#### Demo & Processing
<details>
<summary>VPC+EC2創建</summary>
+ 增加allow-http的規則
+ 架設NAT
+ 創建apache-server vm
```bash=0
automation腳本
#! /bin/bash
apt update
apt -y install apache2
cat <<EOF > /var/www/html/index.html
<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>
```
+ 創建apache-client vm (大部分都跟apache-server的設置一樣，唯獨以下4個設置不同)
</details>
<details>
<summary>LB創建</summary>
+ 創建group
+ 創建LB
- frontend 設置
- backend 設置
</details>
<details>
<summary>DNS創建</summary>
+ 先enable
+ 創建
+ 新增Domain-name
</details>
+ 結果
#### Key Technology
+ [NAT-Gateway](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#Build-NAT-for-private-VM-20241119)
## Example EX.15 (2024/12/03)
#### Project-Name
+ Iris prediction based on serverless vm
#### Demo & Processing
<details>
<summary>CMD + GUI 操作</summary>
+ step1. 上傳Iris訓練模型到cloud storage
```bash=0
$ mkdir test-iris; cd test-iris
$ git clone https://github.com/saedhussain/gcp_serverless_ml.git
$ pip install pandas; pip install scikit-learn
$ cd Iris_model; python
$ python create_iris_model.py
$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
$ export BUCKET=[取一個cloud storage名稱]
$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${BUCKET}
$ gsutil cp ./[model名稱] gs://${BUCKET}
```
+ step2. 創建cloud function
 python版本請改成3.8
- [對應的main.py](https://github.com/saedhussain/gcp_serverless_ml/blob/main/Iris_http_cloud_func/main.py)
- [對應的requirement.txt](https://github.com/saedhussain/gcp_serverless_ml/blob/main/Iris_http_cloud_func/requirements.txt)
- main要修改的地方
- 測試指令
```bash=0
test cmd :
$ curl -m 70 -X POST [你的cloud storage連結 : 可以在cloud function的TESTING當中查看] \
-H "Authorization: bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d '{
"features": [2,3,4,5]
}'
```
</details>
+ 結果
#### Key Technology
+ [cloud run function](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Cloud-Run-Functions-HTTP%E7%82%BA%E4%BE%8B)
+ [cloud run function concept](https://hackmd.io/@GWZ3Y5G0SeyBDB3eXq0Byw/rJyB-462C#GCP-Create-Machine-Concept-20240924--2024101)
## Example EX.16 (2024/12/03)
#### Project-Name
+ Use Cloud Function to modify cloud storage
+ 架構圖
1. 使用者上傳檔案到一個storage
2. 系統判斷如果檔案小於1M就會執行Exit
3. 如果大於1M就會另外存到一個storage
#### Demo & Processing
<details>
<summary>CMD + GUI 操作</summary>
+ step1. 創建兩個cloud storage(一個用來接收使用者上傳的資料，另一個儲存大於1MB的檔案)
```bash=0
$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
$ export ORIGIN_BUCKET=[取一個cloud storage名稱]
$ export LARGE_BUCKET=[取一個cloud storage名稱]
$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${ORIGIN_BUCKET}
$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${LARGE_BUCKET}
```
+ step2. 創建cloud function
- Cloud Storage對應的四種觸發條件 [(ref)](https://cloud.google.com/functions/docs/calling/storage)
1. On (archiving) file in the selected bucket : (檔案被封存時觸發)
2. On (deleting) file in the selected bucket : (檔案被刪除時觸發)
3. On (finalizing/creating) file in the selected bucket : (檔案完成上傳或創建時觸發)
4. On (metadata update) of the file in the selected bucket : (檔案的元數據更新時觸發)
- main.py
```python=0
# cloud storage 對應的 cloud function
import functions_framework
from google.cloud import storage
from google.cloud.storage import Blob
# Triggered by a change in a storage bucket
@functions_framework.cloud_event
def hello_gcs(cloud_event):
data = cloud_event.data
event_id = cloud_event["id"]
event_type = cloud_event["type"]
bucket = data["bucket"]
name = data["name"]
metageneration = data["metageneration"]
timeCreated = data["timeCreated"]
updated = data["updated"]
print("="*30)
print(f"Event ID: {event_id}")
print(f"Event type: {event_type}")
print(f"Bucket: {bucket}")
print(f"File: {name}")
print(f"Metageneration: {metageneration}")
print(f"Created: {timeCreated}")
print(f"Updated: {updated}")
print(f"Processing file: {name}.")
storage_client = storage.Client(project='[你的projet-id]')
source_bucket=storage_client.get_bucket('[你用來接收使用者上傳資料的storage]')
destination_bucket=storage_client.get_bucket('[你用來接收大於1MB資料的storage]')
blobs=list(source_bucket.list_blobs(prefix=''))
print(blobs)
for blob in blobs:
if blob.size > 1000000 and blob.name == name:
source_blob = source_bucket.blob(blob.name)
new_blob = source_bucket.copy_blob(source_blob, destination_bucket, blob.name)
blob.delete(if_generation_match=None)
print(f'File moved from {source_blob} to {new_blob}')
else:
print("File size is below 1MB\n")
```
- requirement.txt
```text=0
functions-framework==3.*
google-cloud-storage
google-cloud
```
</details>
+ 結果
#### Key Technology
+ [cloud storage](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Cloud-SQL-only-by-CLI-20241203)
+ [cloud storage concept](https://hackmd.io/@GWZ3Y5G0SeyBDB3eXq0Byw/rJyB-462C#GCP-Create-Machine-Concept-20240924--2024101)
## Example EX.17 (2024/12/10)
#### Project-Name
+ Build cloud DB and cloud function by CLI
#### Demo & Processing
+ step1. 創建DB
```bash
$ gcloud sql instances create [你的DB instance 名稱] --database-version=MYSQL_5_7 --cpu=2 --memory=4GB --root-password=[DB的密碼] --assign-ip --zone=[你想要的region+zone] --availability-type=zonal --no-backup
$ gcloud sql databases create [DB名稱] --instance=[DB instance 名稱]
$ gcloud sql connect [DB-Instance名稱] --user=root
> use mark-db;
```
```sql
>(mark-db) CREATE TABLE info (
id INT NOT NULL AUTO_INCREMENT,
firstname VARCHAR(20),
lastname VARCHAR(20),
age VARCHAR(3),
collegename VARCHAR(150),
PRIMARY KEY (id)
);
>(mark-db) exit
```
+ step2. 在當前目錄下建立程式碼
- main.py
```python=1
import sqlalchemy
#connection name we noted earlier
connection_name = "[你的project id:DB-Instance名稱]"
#database name
db_name = "[你的db名稱]"
db_user = "root"
db_password = "admin1234"
driver_name = 'mysql+pymysql'
query_string = dict({"unix_socket": "/cloudsql/{}".format(connection_name)})
def writeToSql(request):
#You can change this to match your personal details
stmt = sqlalchemy.text("INSERT INTO info ( firstname, lastname, age, collegename) values ('Sagadevan', 'Kounder', '21', 'XYZ College')")
db = sqlalchemy.create_engine(
sqlalchemy.engine.url.URL(
drivername=driver_name,
username=db_user,
password=db_password,
database=db_name,
query=query_string,
),
pool_size=5,
max_overflow=2,
pool_timeout=30,
pool_recycle=1800
)
try:
with db.connect() as conn:
conn.execute(stmt)
print("Insert successful")
except Exception as e:
print ("Some exception occured" + e)
return 'Error: {}'.format(str(e))
return 'ok'
```
- requirement.txt
```text
SQLAlchemy==1.3.12
PyMySQL==0.9.3
```
+ step3. 建立cloud function
```bash
$ gcloud functions deploy writeToSql --entry-point writeToSql --runtime python310 --trigger-http --allow-unauthenticated --no-gen2 --source .
$ gcloud functions deploy [cloud function 名稱] --entry-point [function的entry point] --runtime [python版本] --trigger-http --allow-unauthenticated --no-gen2 --source [程式碼的位置]
```
+ 結果
#### Key Technology
+ [cloud storage](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Cloud-SQL-only-by-CLI-20241203)
+ [cloud storage concept](https://hackmd.io/@GWZ3Y5G0SeyBDB3eXq0Byw/rJyB-462C#GCP-Create-Machine-Concept-20240924--2024101)
## Example EX.18 (2024/12/10)
#### Project-Name
+ Build APP Engine to predict iris by CLI
#### Demo & Processing
+ step1. 根據之前的iris範例建立出對應的模型
```bash
$ mkdir test-iris; cd test-iris
$ git clone https://github.com/saedhussain/gcp_serverless_ml.git
$ pip install pandas; pip install scikit-learn
$ cd Iris_model; python
$ python create_iris_model.py
$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
$ export BUCKET=[取一個cloud storage名稱]
$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${BUCKET}
$ gsutil cp ./[model名稱] gs://${BUCKET}
```
+ step2. 寫對應的app egine 程式碼
- main.py
```python=1
import numpy as np
import os
import pickle
from flask import Flask, jsonify
app = Flask(__name__)
# 全局模型變數
model = None
# 下載模型檔案的函數
def download_model_file():
from google.cloud import storage
BUCKET_NAME = "iris-bk"
PROJECT_ID = "avian-casing-435202-m5"
GCS_MODEL_FILE = "iris_model_jan_2020_v1.pkl"
client = storage.Client(PROJECT_ID)
bucket = client.get_bucket(BUCKET_NAME)
blob = bucket.blob(GCS_MODEL_FILE)
folder = '/tmp/'
if not os.path.exists(folder):
os.makedirs(folder)
blob.download_to_filename(folder + "local_model.pkl")
# 載入模型
def load_model():
global model
if not model:
download_model_file()
model = pickle.load(open("/tmp/local_model.pkl", 'rb'))
# 處理路徑格式的功能
@app.route('/features/<path:features>', methods=['GET'])
def iris_predict(features):
global model
# 確保模型已加載
load_model()
# 分解輸入的特徵
try:
feature_list = [float(i) for i in features.split("_")]
if len(feature_list) != 4:
return jsonify({"error": "Invalid number of features. Expected 4 values."}), 400
except ValueError:
return jsonify({"error": "Features must be numeric and separated by underscores."}), 400
# 使用模型進行預測
prediction = model.predict(np.array([feature_list]))
return jsonify({"prediction": prediction[0]})
if __name__ == "__main__":
app.run(host='0.0.0.0', port=8080)
```
- requirements.txt
```text
google-resumable-media==0.6.0
google-cloud-storage==1.30.0
google-cloud-bigquery==1.26.1
flask
pandas
numpy
scikit-learn
```
- app.yaml
```yaml
runtime: python310
service: iris-test
```
+ 結果
輸入對應的指令就可以得到preediction的結果
#### Key Technology
+ [APP-Egine](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-APP-Egine-by-CLI)
## Example EX.19 (2024/12/17)
#### Project-Name
+ Build Iris-Prediction Model on Cloud Run by Artifact Registry
#### Demo & Processing
+ step1. 創建以下幾個檔案
- train_model.py (訓練iris-prediction model)
```python=1
# -*- coding: utf-8 -*-
import pickle
from sklearn import datasets
from sklearn.model_selection import train_test_split
from sklearn import tree
# simple demo for traing and saving model
iris=datasets.load_iris()
x=iris.data
y=iris.target
#labels for iris dataset
labels ={
0: "setosa",
1: "versicolor",
2: "virginica"
}
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=.25)
classifier=tree.DecisionTreeClassifier()
classifier.fit(x_train,y_train)
predictions=classifier.predict(x_test)
#export the model
model_name = 'model.pkl'
print("finished training and dump the model as {0}".format(model_name))
pickle.dump(classifier, open(model_name,'wb'))
```
- main.py
```python=1
import pickle
from flask import Flask, request, jsonify
app = Flask(__name__)
# Load the model
model = pickle.load(open('model.pkl', 'rb'))
labels = {
0: "versicolor",
1: "setosa",
2: "virginica"
}
@app.route("/", methods=["GET"])
def index():
"""Basic HTML response."""
body = (
"<html>"
"<body style='padding: 10px;'>"
"<h1>Welcome to my Flask API</h1>"
"</body>"
"</html>"
)
return body
@app.route('/api', methods=['POST'])
def predict():
# Get the data from the POST request.
data = request.get_json(force = True)
predict = model.predict(data['feature'])
return jsonify(predict[0].tolist())
if __name__ == '__main__':
app.run(debug = True, host = '0.0.0.0', port=8080)
```
- requirements.txt
```text=1
scikit-learn
flask
```
- Dockerfile
```Docker=1
FROM python:3.9
WORKDIR /app
ADD . /app
RUN pip install -r requirements.txt
CMD ["python", "main.py"]
EXPOSE 8080
```
+ step2. 創建Arifact Registry，將Dockerfile創建出來的鏡像打包上傳
```bash
$ docker build -t asia-east1-docker.pkg.dev/avian-casing-435202-m5/mark-iris-docker/mark-iris:1.0 .
$ docker push asia-east1-docker.pkg.dev/avian-casing-435202-m5/mark-iris-docker/mark-iris:1.0
```
+ step3. 點選完Deploy之後
+ step4. 撰寫`clent.py`並執行
```python=1
# -*- coding: utf-8 -*-
import requests
# Change the value of experience that you want to test
url = '[你的cloud run網址]/api'
feature = [[5.8, 4.0, 1.2, 0.2]]
labels ={
0: "setosa",
1: "versicolor",
2: "virginica"
}
r = requests.post(url,json={'feature': feature})
print(labels[r.json()])
```
+ 執行`client.py`結果
```
$ python client.py
setosa
```
#### Key Technology
+ [Artifacts Registry](https://hackmd.io/O-CEE43oR7OTPapWV1N21A?view#GCP-Build-Artifacts-Registry)
Sign in with Wallet
Connect another wallet