Try   HackMD

GCP　Examples

Example EX. (Date)

Project-Name

Demo & Processing

Key Technology

Example EX.1 (2024/10/01)

Project-Name

  • 手動創建Web-Server

Demo & Processing

創建web須擬機

  • step1. 透過UI創建VM
    image image

  • step2. SSH連線VM

  • step3. 終端機指令

    





    ​​​​安裝apache2並檢測server啟動狀態
​​​​$ sudo apt update 
​​​​$ sudo apt install apache2 -y
​​​​$ sudo systemctl status apache2
​​​​$ sudo apt install net-tools
​​​​$ sudo netstat -tulnp | grep 80

  • step4. 測試連線

    • 執行指令
      image

      ​​​​​​​​curl http://[apache2 server ip]/

    • 結果
      image

  • step5. 建立網頁

    • 指令

      




      ​​​​​​​​第三行展示錯誤的寫入html方法 (原因 : 這是一個多指令組合的指令，因此sudo只會允許第一個指令使用root，這就是為甚麼第三行會出現permission denied)
​​​​​​​​$ cd /var/www/html
​​​​​​​​$ sudo echo "RRRRRRRRRR" > markerpen_yelling.htm
​​​​​​​​-bash: markerpen_yelling.htm: Permission denied
​​​​​​​​$ sudo bash -c 'echo "RRRRRRRRRR" > markerpen_yelling.htm'

    • 指令解說 bash -c
      bash 是一個 Unix Shell，而 -c 參數告訴 bash 去執行後面引號中的命令字符串，並把它作為完整的腳本來運行。

    • 指令解說 sudo -l

      






      ​​​​​​​​作用 : 可以確認自己可以使用到sudo的哪些指令
​​​​​​​​$ sudo -l
​​​​​​​​Matching Defaults entries for mingchu_chou92 on webtesting1:
​​​​​​​​    env_reset, mail_badpass, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin, use_pty

​​​​​​​​User mingchu_chou92 may run the following commands on webtesting1:
​​​​​​​​    (ALL : ALL) NOPASSWD: ALL (這代表有sudo的最高權限)

    • hostname -I

      



      ​​​​​​​​可以利用這項技術實現多台Server之間的IP管理
​​​​​​​​$ sudo bash -c 'echo "$(hostname -I)" > hostip.htm'
​​​​​​​​$ curl http://34.81.100.236/hostip.htm
​​​​​​​​10.140.0.3 (VM會把自己的IP顯示出來)

Key Technology

Example EX.2 (2024/10/01)

Project-Name

  • 自動(即剛開機時)創建Web-Server

Demo & Processing

創建web須擬機

  • step1. 透過UI創建VM (先不要點create，點選下方的Advanced Option當中的Management)
    image image

  • step2. 透過UI創建VM (寫入自動執行腳本 + create VM)
    image

    • 腳本 
      





      ​​​​​​​​這個腳本會在VM開機的後執行腳本
​​​​​​​​#! /bin/bash
​​​​​​​​apt update
​​​​​​​​apt -y install apache2
​​​​​​​​cat <<EOF > /var/www/html/index.html
​​​​​​​​<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>

Key Technology

Example EX.3 (2024/10/08)

Project-Name

  • 名稱
    建立本地網頁，運行於GCP平台
    [會有permission denied的問題，導致stroage放到GCE的時候出現無法複製的問題]
    [permission會因為service account導致這個問題發生]

  • 架構圖

    Image Not Showing Possible Reasons
    • The image was uploaded to a note which you don't have access to
    • The note which the image was originally uploaded to has been deleted
    Learn More →

Demo & Processing

GCP運行本地網頁操作過程

  • step1. 在本地端創建一個網頁
    image

  • step2. 創建 Cloud Storage 內部的 Bucket + 上傳資料

    • Bucket上傳方法1

      • 找到 Cloud Storage
        image

      • 創建 Bucket
        image
        image
        image
        image
        image
        image

      • 上傳檔案
        image
        image

    • Bucket上傳方法2

      • 先創建 Bucket
        大致操作都一樣，但是可以嘗試不要使用multui-region比免花太多錢
        image

      • 上傳資料到 GCE Shell 上
        請記得這個shell是GCP提供給user的特殊須擬機，他有5GB的儲存空間，儘管我們把機器reboot資料仍然會在裡面不會消失，除非我們刪除

        image image

      • 從 GCE Shell 上傳到 Bucket

        



        ​​​​​​​​​​​​從 GCE Shell 上傳到 Bucket
​​​​​​​​​​​​$ gsutil ls gs://   (顯示目前有哪些bucket)
​​​​​​​​​​​​$ gsutil cp [args] [fileA] [Bucket-Location]     (Bucket位置可以從上一個指令獲得)
​​​​​​​​​​​​

        image

      • 檢查是否有上傳成功
        image

  • step3. 把Bucket的資料移到GCE的VM內

    • 先跟以前一樣創建VM(ubuntu的虛擬機)
      image
      但是這個要特別注意，如果有其他project存在的話，會因為這個選項會導致我們沒辦法把東西移動過去(不過這都是以後再說的了…)
      image

    • SSH連線
      image

      




      ​​​​​​​​安裝伺服器 + 架設伺服器
​​​​​​​​$ sudo apt update
​​​​​​​​$ sudo apt install apache2 -y
​​​​​​​​$ gsutil cp -r gs://marktestingbucket2-markwatchcloud .
​​​​​​​​$ sudo cp -r marktestingbucket2-markwatchcloud/ /var/www/html/

    • 檢查
      image
      image

Key Technology

Example EX.4 (2024/10/08)

Project-Name

  • Project-Name
    Storage逆向回傳

  • 架構
    Example3展是怎麼從本地->Storages->VM
    現在展示 VM -> Storages -> 本地

Demo & Processing

  • 運行指令

    






    ​​​​$ echo "hahahahahaha" > markerpenlaughing.htm

​​​​$ gsutil ls gs://
​​​​  gs://marktestingbucket1-markwatchcloud/
​​​​  gs://marktestingbucket2-markwatchcloud

​​​​$ gsutil cp markerpenlaughing.htm gs://marktestingbucket1-markwatchcloud/

  • 報錯訊息 (Service Account權限不足)

    Image Not Showing Possible Reasons
    • The image was uploaded to a note which you don't have access to
    • The note which the image was originally uploaded to has been deleted
    Learn More →

  • 解決方法

    解決方法

    • 解決方法1.
      image

    • 解決方法2.

      • 創建IAM帳戶 (根據林廣哲所說，這個步驟似乎並不需要)

        image
        image
        image
        image
        image

      • 調整GCE的VM設定 (需要回到GCE暫停VM才可以調整設定)
        image

      • 將VM的Service Acocunt修改

        image
        image

      • 修改權限

        也要把Sotrage也改成可讀可寫
        image

      • 進入ssh修改系統
        刪除原先的permission設定快取，否則系統會優先讀取這個設定
        image
        指令 : rm -rf .gsutil/
        image

      • 檢查

      image

Key Technology

Example Ex.5 (2024/10/15)

Project-Name

  • project-name
    建立DB，讓使用者連線Server進一步訪問DB

  • 架構

    Image Not Showing Possible Reasons
    • The image was uploaded to a note which you don't have access to
    • The note which the image was originally uploaded to has been deleted
    Learn More →

Demo & Processing

  • step1. server 建置

    創建虛擬機-圖形化操作

    • step 1-1
      image image

    • step 1-2
      image image

      




      ​​​​​​​​#! /bin/bash
​​​​​​​​apt update
​​​​​​​​apt -y install apache2
​​​​​​​​cat <<EOF > /var/www/html/index.html
​​​​​​​​<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>

  • step2. DB 建置

    DB 建置

    • step2-1. 建立一個ubuntu22.04 + 不需要防火牆規則的VM
      基本配置都跟VM建置一樣，不再贅述
      image

    • step2-2. 進入DB虛擬機的ssh下載mariadb

      

























      ​​​​​​​​按照下面指令以及程式碼輸入
​​​​​​​​$ sudo apt-get install apt-transport-https curl
​​​​​​​​$ sudo curl -o /etc/apt/keyrings/mariadb-keyring.pgp 'https://mariadb.org/mariadb_release_signing_key.pgp'
​​​​​​​​$ sudo vim /etc/apt/sources.list.d/mariadb.sources
​​​​​​​​$ sudo cat /etc/apt/sources.list.d/mariadb.sources
​​​​​​​​# MariaDB 11.1 repository list - created 2023-11-08 06:16 UTC
​​​​​​​​# https://mariadb.org/download/
​​​​​​​​X-Repolib-Name: MariaDB
​​​​​​​​Types: deb
​​​​​​​​# deb.mariadb.org is a dynamic mirror if your preferred mirror goes offline. See https://mariadb.org/mirrorbits/ for details.
​​​​​​​​# URIs: https://deb.mariadb.org/11.1/ubuntu
​​​​​​​​URIs: https://ftp.ubuntu-tw.org/mirror/mariadb/repo/11.1/ubuntu
​​​​​​​​Suites: jammy
​​​​​​​​Components: main main/debug
​​​​​​​​Signed-By: /etc/apt/keyrings/mariadb-keyring.pgp
​​​​​​​​$ sudo apt-get update                   # 更新
​​​​​​​​$ sudo apt install mariadb-server       # 安裝 MariaDB
​​​​​​​​$ sudo systemctl status mariadb         # 安裝完服務就已經啟動了，可以再檢查一下
​​​​​​​​$ sudo mysql_secure_installation        # 初始化 MariaDB
​​​​​​​​    Enter current password for root (enter for none): 直接按下Enter
​​​​​​​​    Switch to unix_socket authentication [Y/n] : 選擇n
​​​​​​​​    Change the root password? [Y/n] : 看個人想換甚麼密碼
​​​​​​​​    Remove anonymous users? [Y/n] : 考量安全選擇y
​​​​​​​​    Disallow root login remotely? [Y/n] : y
​​​​​​​​    Remove test database and access to it? [Y/n] : y
​​​​​​​​    Reload privilege tables now? [Y/n] : y

    • step2-3. 檢查DB狀態
      image

    • step2-4. 嘗試登入 + 建立資料庫

      • 終端指令
        sudo mysql -u root -p

      • DB指令

        




        ​​​​​​​​​​​​create database markdb;
​​​​​​​​​​​​use markdb;
​​​​​​​​​​​​create table addrbook(name varchar(50) not null, action char(20));
​​​​​​​​​​​​insert into addrbook(name,action) values ("mark","yelling:RRRRRRRR");
​​​​​​​​​​​​select name,action from addrbook;

    • step2-5. 修改DB配置，使其允許其他IP登入

      • 下載net-tool 並解查 mariadb的port號

        




        ​​​​​​​​​​​​檢查DB
​​​​​​​​​​​​sudo apt install net-tools (記得要先exit離開DB)
​​​​​​​​​​​​sudo netstat -tunlp | grep mariadb
​​​​
​​​​​​​​​​​​結果 : 127.0.0.1:3306 -> 表示只能連接到localhost才能到3306port存取DB

        為了讓Server可以連接到db，我們會希望設定是0.0.0.0:[對應port號]

      • 修改db配置

        


        ​​​​​​​​​​​​$ sudo vim /etc/mysql/mariadb.conf.d/50-server.cnf
​​​​​​​​​​​​    找到bind-address並改成0.0.0.0
​​​​​​​​​​​​$ sudo systemctl restart mariadb

        image

      • 仍然無法存取的原因及結果

        







        ​​​​​​​​​​​​但是DB的root本身仍不允許除了localhost以外的人登入
​​​​​​​​​​​​因此儘管看到mariadb的服務變成`0.0.0.0:3306`
​​​​​​​​​​​​依樣無法用其他ip登入root
​​​​
​​​​​​​​​​​​mingchu_chou92@mdb-server:~$ mysql -h 10.140.0.8 -u root -p
​​​​​​​​​​​​mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
​​​​​​​​​​​​Enter password: 
​​​​​​​​​​​​ERROR 2002 (HY000): Received error packet before completion of TLS handshake. The authenticity of the following error cannot be verified: 1130 - Host 'mdb-server.asia-east1-a.c.imposing-union-436602-v2.internal' is not allowed to connect to this MariaDB server

      • 打開DB允許哪些IP存取的權限

        






















        ​​​​​​​​​​​​終端畫面
​​​​​​​​​​​​mingchu_chou92@mdb-server:~$ mysql  -u root -p
​​​​​​​​​​​​mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
​​​​​​​​​​​​Enter password: 
​​​​​​​​​​​​Welcome to the MariaDB monitor.  Commands end with ; or \g.
​​​​​​​​​​​​Your MariaDB connection id is 32
​​​​​​​​​​​​Server version: 11.1.6-MariaDB-ubu2204 mariadb.org binary distribution

​​​​​​​​​​​​Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

​​​​​​​​​​​​Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

​​​​​​​​​​​​MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'10.140.0.8' IDENTIFIED BY 'mark' WITH GRANT OPTION;
​​​​​​​​​​​​Query OK, 0 rows affected (0.001 sec)

​​​​​​​​​​​​MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'mark' WITH GRANT OPTION;
​​​​​​​​​​​​Query OK, 0 rows affected (0.001 sec)


​​​​​​​​​​​​MariaDB [(none)]> FLUSH PRIVILEGES; (更新)
​​​​
​​​​
​​​​​​​​​​​​GRANT ALL PRIVILEGES ON *.* TO '[使用者]'@'[允許的IP位置]' IDENTIFIED BY 'mark' WITH GRANT OPTION;   (其中 `%` 可以表示 `任意的IP位置都被允許`)

        結果: image

  • step3. 建立server與db之間的連線

    圖形化操作 + 程式碼

    • step3-1. 打開server的ssh

    • step3-2. 嘗試ping db檢查連線

      image

    • step3-3. 在server端安裝maria-db的client程式

      ​​​​​​​​​​  `$ sudo apt install mysql-client`

    • step3-4. 嘗試登入DB

      image

    • step3-5. 建立防火牆

      image

      image

      請注意因為這台db跟server的VM都使採用default的網路，因此一定要填寫default network
      image

      限制網路只限制在台灣的IP
      image

      image

      將防火牆設定加入db當中
      image

      image

      建立一台不在台灣的VM，並嘗試連線看看

    • step3-6. 建立php後端程式向db發送請求

      檢查sudo dpkg -l | grep apache2

      











      ​​​​​​​​終端畫面
​​​​​​​​$ sudo apt install software-properties-common
​​​​​​​​$ sudo add-apt-repository ppa:ondrej/php
​​​​​​​​$ sudo apt update
​​​​​​​​$ sudo apt upgrade -y
​​​​​​​​$ sudo apt install php8.1 libapache2-mod-php8.1 php8.1-gd php8.1-mysql php8.1-curl php8.1-mbstring php8.1-intl -y
​​​​​​​​$ sudo apt install php8.1-gmp php8.1-bcmath php8.1-imagick php8.1-xml php8.1-zip -y
​​​​​​​​$ sudo a2enmod php8.1
​​​​​​​​$ sudo systemctl restart apache2
​​​​​​​​<?php
​​​​​​​​phpinfo();
​​​​​​​​?>

      





























      ​​​​​​​​終端畫面
​​​​​​​​mingchu_chou92@www-server:~$ vim test.php
​​​​​​​​mingchu_chou92@www-server:~$ cat test.php 
​​​​​​​​<?php
​​​​​​​​$servername="10.140.0.8";
​​​​​​​​$username="root";    
​​​​​​​​$password="mark";
​​​​​​​​$dbname="markdb";

​​​​​​​​$conn = new mysqli($servername, $username, $password, $dbname);

​​​​​​​​if($conn->connect_error){
​​​​​​​​    die("connection failed: " . $conn->connect_error);
​​​​​​​​}
​​​​​​​​else{
​​​​​​​​    echo "connect OK!" . "<br>";
​​​​​​​​}

​​​​​​​​$sql="select name,action from addrbook";
​​​​​​​​$result=$conn->query($sql);

​​​​​​​​if($result->num_rows>0){
​​​​​​​​    while($row=$result->fetch_assoc()){
​​​​​​​​        echo "name: " . $row["name"] . "\taction: " . $row["action"] . "<br>";
​​​​​​​​    }
​​​​​​​​} else {
​​​​​​​​    echo "0 record";
​​​​​​​​}
​​​​​​​​?>
​​​​​​​​mingchu_chou92@www-server:~$ sudo cp test.php /var/www/html/

    • 結果
      image

Key Technology

Example EX6. (2024/10/22)

Project-Name

  • 基於自創VPC網路下設立獨立防火牆規則

Demo & Processing

  • step1. Build VPC Network and VMs

    Build VPC (subnet/firewall/route)
    • Create VPC Network

    image

    image

    • Set VPC Network's firewall

      • XXXX-allow-custom
        這條規則表示內網的機器無論任何服務在沒有匹配其他規則的前提下都可以通過

      image

    • Set VPC Newtork's Route Rule
      image

    • Set the costum VPC
      image

    • Set Costum VPC's subnet/firewall/route rules
      image
      image
      image

    • Change Costum VPC's firewall rules

      • 可以自己設定是要allow或是deny又或是Priority數值
        image
        image
        image
        image

      • 檢查
        image
        image

    Build VM1 with VPC1 + VM2 with VPC2

    • 建立VM1，並安裝上自己建立的VPC
      image
      image
      image
      image
      image

    • 建立VM2，大致跟VM1相同，但是選擇VPC2 (VPC根VM的region要相同哦…)
      image

  • step2. 建立雙向(VPC1與VPC2的雙向) VPC Peering

    Build VPC Peering

    • 進入VPC Peering介面
      image

    • 建立第一條 VPC Peering設定
      image

    • 可以發現建立完處於inactive的狀態是因為只有建立單向
      image

    • 所以建立另一向的VPC Peering以避免inactive狀態
      image

  • 確認兩台VM是否可以互相用ICMP/HTTP/SSH通訊 | SSH Between VMs

    results

    • ICMP
      image

    • HTTP
      image

    • SSH
      image

Key Technology

Example EX.7 (2024/10/29)

Project-Name

  • 創建SQL + 限制Instance可以存取

Demo & Processing

CMD操作

  • 創建sql並且是private ip (default VPC) + 打開sql的admin管理(安裝SQL可以參考這裡)

  • 創建一個Instance + Ubuntu20.04作業系統 + default VPC設置

  • ssh進入Instance並安裝mysql的client端軟體(要怎麼SSH可以參考這邊)

  • 在Client端(沒有安裝sql-server的那台VM)安裝SQL-Client軟體 + 嘗試登入建立好的sql-server

    ​​​​$ sudo apt install mysql-client -y
​​​​$ mysql -h 'sql public ip' -u root -p

  • 在DB內部建立檔案







create database markdb;
use markdb;
create table addrbook(name varchar(50) not null, action char(20));
insert into addrbook(name,action) values ("mark","yelling:RRRRRRRR");
insert into addrbook(name,action) values ("markerpen","laughing:HAHAHAHA");
select name,action from addrbook;
  • 回到instance並下載apache2以及php































$ sudo apt install apache2 php libapache2-mod-php php-mysql -y
$ sudo systemctl restart apache2
$ sudo vim /var/www/html/mark.php
$ sudo cat /var/www/html/mark.php
  <?php
  $servername="10.56.224.3";
  $username="root";    
  $password="123456";
  $dbname="markdb";

  $conn = new mysqli($servername, $username, $password, $dbname);

  if($conn->connect_error){
      die("connection failed: " . $conn->connect_error);
  }
  else{
      echo "connect OK!" . "<br>";
  }

  $sql="select name,action from addrbook";
  $result=$conn->query($sql);

  if($result->num_rows>0){
      while($row=$result->fetch_assoc()){
          echo "name: " . $row["name"] . "\taction: " . $row["action"] . "<br>";
      }
  } else {
      echo "0 record";
  }
  ?>

  • 結果

    Image Not Showing Possible Reasons
    • The image was uploaded to a note which you don't have access to
    • The note which the image was originally uploaded to has been deleted
    Learn More →

Key Technology

Example EX.8 (2024/10/29)

Project-Name

  • 建立 unmanaged load balancer (這些server一定要在同樣的 region 跟 zone)

Demo & Processing

GUI操作

  • step1. 創建相同 Region&Zone 的兩台VM + 開機建立網頁伺服器 + Ubuntu作業系統

    • 詳細可以參考 EX.2
    • 兩台VM分別叫做 www-mark1 & www-mark2 + 地區都是 Taiwan & asia-east1-c

  • step2. 將要管理的VM加入到同一個 Group 底下

    • step2-1. 切入 Instance groups + 點 CREATE INSTANCE GROUP
      image

    • step2-2. 創建 unmanaged load-balancer
      image

  • step3. 創建 Load-Balancer

    • step3-1. 進入 Load Balancer(LB) 頁面 + 創建LB
      image

    • step3-2. 創建LB種類
      image
      image
      image
      image

    • step3-3. 創建LB的名稱+位置+網路
      image

      • 快速解法
        image
      • 正統解法
        image
        image
        image

    • step3-4. 創建LB的 Frontend configuration
      image

    • step3-5. 創建LB的 Backend configuration
      image
      image

    • step3-6. 創建LB的 Health Check
      image

  • 結果
    imageimage
    image

Key Technology

Example EX.9 (2024/11/12)

Project-Name

刪除web server的公有網路

Demo & Processing

GUI操作

  • step1. 照ex8創建整個網路架構

  • step2. 創建Firewall允許http存取load-balancer + 允許health-check存取load-balancer

    • 允許http存取load-balancer
      image

    • 允許health-checkk存取load-balancer

      • 請注意health-check的ip-range，這是GCP提供專門讓vpc可以跟heal-check連線的網段
        image

  • step3. 修改GCE兩台VMs的設定

    • 兩台VM的允許Http流量關閉 + 新增剛才創建的兩個Firewall tags
      image

    • 關閉External IP
      image

  • step4. 檢查load-balance
    image

  • 結果
    image

Key Technology

Example EX10. (2024/11/12)

Project-Name

  • Create Managed Load-Balancer

Demo & Processing

GUI操作

  • step1. 創建template

    • step1-1. 創建模板名稱(mark-template)
      image
    • step1-2. 選擇硬體配置
      image
    • step1-3. 選擇作業系統
      image
    • step1-4. 設定防火牆
      image
    • step1-5. 選擇網卡
      image
    • step1-6. 開機腳本
      開機腳本程式碼
      image

  • step2. 在GCE/Instance Group/ 創建instance-group

    • step2-1. 創建Managed-Group
      image
    • step2-2. 創建Group的型態
      image
    • step2-3. 創建Group動態增加VM的規則
      請注意Minimum number of instances至少要建立的VM數量 ; Maximum number of instances則表示系統最多建立的VMs數量
      image
    • step2-4. 創建Heal-Check(請注意這裡的Health-Check跟Load-Balancer的不一樣，這是專門為VM設立的)
      image
    • step2-5. 其餘都用default-value就好，然後點選Create

  • step3. 創建load-balancer

    • step3-1. 創建load-balancer
      image
      image
      image
      image

    • step3-2. 增加frontend-config
      image

    • step3-3. 增加backend-config
      image

  • step4. 壓力測試

    • step4-1. 開啟兩個由ManagedGroup所創建VM的SSH畫面(左邊執行壓力測試，右邊監測壓力)
      image
    • step4-2. 執行指令
      左邊執行cat /dev/zero > /dev/null + 右邊執行top
  • result

image

Key Technology

Example EX11. (2024/11/19)

Project-Name

  • 透過load balancer將http服務轉換成https

Demo & Processing

<summaary>操作過程</summaary>

  • step1-1. 創建unmanaged instance group

  • step1-2. 在DNS網站建立一個Domain-Name

image

  • step2. 創建Load-Balancer

    image
    image

  • step3. Frontend configuration
    image
    image

  • step4. Backend configuration + Health-Check
    image
    image

Example EX12. (2024/11/26)

Project-Name

  • Build Load-Balance to contain to apache-servers template without public ip

Demo & Processing

GUI操作

  • step1. 在VPC當中加入允許health-check的防火牆規則，並確認是否有允許Http(網頁伺服器)以及SSH(debug用)的規則

    image
    image

  • step2. 因為 apache-server 沒有 public ip，為了讓 apache-server 可以安裝網頁軟體，所以我們需要NAT設置讓 server 不用 public ip 也可以到公網
    image

    image

  • step3. 創建一個不用 public ip 的 template

    image
    image
    image
    image
    image
    image

    





    ​​​​這個腳本會在VM開機的後執行腳本
​​​​#! /bin/bash
​​​​apt update
​​​​apt -y install apache2
​​​​cat <<EOF > /var/www/html/index.html
​​​​<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>

  • step4. 創建instance group
    image
    image
    image

  • step5. 創建load balancer
    image
    image
    image
    image
    image
    image

  • 結果

image

Key Technology

Example EX.13 (2024/11/26)

Project-Name

  • Build Load Balancer with different region

  • 架構圖
    image

Demo & Processing

創建VPC

image
image
image
image

  • 額外新增Firewall Rule (health-check)
    image
    image

  • 額外新增Firewall Rule (http)
    image
    image

  • 加入NAT(給第一個region)
    image

  • 新增一個給別的region的subnet
    image

  • 加入NAT(給第二個region)
    image

創建Group

  • 在第一個region創建兩台vm
    image
    image
    image
    image
    image
    image

    




    ​​​​#! /bin/bash
​​​​apt update
​​​​apt -y install apache2
​​​​cat <<EOF > /var/www/html/index.html
​​​​<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>

  • 在第二個region創建兩台vm(基本操作跟上面依樣，只有下面幾個操作略微不同)
    image
    image

    




    ​​​​#! /bin/bash
​​​​apt update
​​​​apt -y install apache2
​​​​cat <<EOF > /var/www/html/index.html
​​​​<html><body><p>Linux startup script added directly. $(hostname -I) . In US</p></body></html>

    • SSH進入 + 執行以下指令

      


      ​​​​​​​​這個資料夾名稱，跟創建LB的時候有關聯很重要，請記住
​​​​​​​​$ cd /var/www/html/
​​​​​​​​$ sudo bash -c 'mkdir us; mv index.html us/'

  • 創建兩個groups
    image
    image

創建Load-Balancer

image
image
image
image
image

  • Frontend 設置
    image

  • Backend 設置 (給region1)
    image
    image

  • Backend 設置 (給region2)
    image
    Healtth-Check共用

  • 創建Domain name
    image

  • 新增對應的path
    image

  • 修改Domain name
    image

  • 結果
    image

Example EX14. (2024/11/26)

Project-Name

  • Internal Load-Balancer

  • 架構圖
    image

Demo & Processing

VPC+EC2創建

image
image
image
image

  • 增加allow-http的規則
    image
    image
    image

  • 架設NAT
    image

  • 創建apache-server vm
    image
    image
    image
    image
    image
    image

    





    ​​​​automation腳本
​​​​#! /bin/bash
​​​​apt update
​​​​apt -y install apache2
​​​​cat <<EOF > /var/www/html/index.html
​​​​<html><body><p>Linux startup script added directly. $(hostname -I) </p></body></html>

  • 創建apache-client vm (大部分都跟apache-server的設置一樣，唯獨以下4個設置不同)
    image
    image
    image
    image

LB創建

  • 創建group
    image

  • 創建LB
    image
    image
    image
    image

    • frontend 設置
      image

    • backend 設置
      image
      image
      image

DNS創建

  • 先enable
    image

  • 創建
    image
    image

  • 新增Domain-name
    image
    image

  • 結果
    image

Key Technology

Example EX.15 (2024/12/03)

Project-Name

  • Iris prediction based on serverless vm

Demo & Processing

CMD + GUI 操作

  • step1. 上傳Iris訓練模型到cloud storage
    image

    








    ​​​​$ mkdir test-iris; cd test-iris
​​​​$ git clone https://github.com/saedhussain/gcp_serverless_ml.git
​​​​$ pip install pandas; pip install scikit-learn
​​​​$ cd Iris_model; python 
​​​​$ python create_iris_model.py
​​​​$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
​​​​$ export BUCKET=[取一個cloud storage名稱]
​​​​$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${BUCKET} 
​​​​$ gsutil cp ./[model名稱] gs://${BUCKET}

  • step2. 創建cloud function
    image
    image
    image python版本請改成3.8

    image

    • main要修改的地方
      image

    • 測試指令

    







    ​​​​test cmd : 
​​​​
​​​​$ curl -m 70 -X POST [你的cloud storage連結 : 可以在cloud function的TESTING當中查看] \
​​​​-H "Authorization: bearer $(gcloud auth print-identity-token)" \
​​​​-H "Content-Type: application/json" \
​​​​-d '{
​​​​    "features": [2,3,4,5]
​​​​}'
  • 結果
    image

Key Technology

Example EX.16 (2024/12/03)

Project-Name

  • Use Cloud Function to modify cloud storage

  • 架構圖

image

  1. 使用者上傳檔案到一個storage
  2. 系統判斷如果檔案小於1M就會執行Exit
  3. 如果大於1M就會另外存到一個storage

Demo & Processing

CMD + GUI 操作

  • step1. 創建兩個cloud storage(一個用來接收使用者上傳的資料，另一個儲存大於1MB的檔案)

    image

    




    ​​​​$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
​​​​$ export ORIGIN_BUCKET=[取一個cloud storage名稱]
​​​​$ export LARGE_BUCKET=[取一個cloud storage名稱]
​​​​$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${ORIGIN_BUCKET} 
​​​​$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${LARGE_BUCKET}

  • step2. 創建cloud function
    image
    image

    • Cloud Storage對應的四種觸發條件 (ref)
      1. On (archiving) file in the selected bucket : (檔案被封存時觸發)
      2. On (deleting) file in the selected bucket : (檔案被刪除時觸發)
      3. On (finalizing/creating) file in the selected bucket : (檔案完成上傳或創建時觸發)
      4. On (metadata update) of the file in the selected bucket : (檔案的元數據更新時觸發)

    image
    image

    • main.py

      











































      ​​​​​​​​# cloud storage 對應的 cloud function
​​​​
​​​​​​​​import functions_framework
​​​​​​​​from google.cloud import storage
​​​​​​​​from google.cloud.storage import Blob


​​​​​​​​# Triggered by a change in a storage bucket
​​​​​​​​@functions_framework.cloud_event
​​​​​​​​def hello_gcs(cloud_event):

​​​​​​​​    data = cloud_event.data

​​​​​​​​    event_id = cloud_event["id"]
​​​​​​​​    event_type = cloud_event["type"]

​​​​​​​​    bucket = data["bucket"]
​​​​​​​​    name = data["name"]
​​​​​​​​    metageneration = data["metageneration"]
​​​​​​​​    timeCreated = data["timeCreated"]
​​​​​​​​    updated = data["updated"]

​​​​​​​​    print("="*30)
​​​​​​​​    print(f"Event ID: {event_id}")
​​​​​​​​    print(f"Event type: {event_type}")
​​​​​​​​    print(f"Bucket: {bucket}")
​​​​​​​​    print(f"File: {name}")
​​​​​​​​    print(f"Metageneration: {metageneration}")
​​​​​​​​    print(f"Created: {timeCreated}")
​​​​​​​​    print(f"Updated: {updated}")
​​​​​​​​    print(f"Processing file: {name}.")
​​​​​​​​    storage_client = storage.Client(project='[你的projet-id]')
​​​​​​​​    source_bucket=storage_client.get_bucket('[你用來接收使用者上傳資料的storage]')
​​​​​​​​    destination_bucket=storage_client.get_bucket('[你用來接收大於1MB資料的storage]') 
​​​​​​​​    blobs=list(source_bucket.list_blobs(prefix=''))
​​​​​​​​    print(blobs)
​​​​​​​​    for blob in blobs:
​​​​​​​​        if blob.size > 1000000 and blob.name == name:
​​​​​​​​            source_blob = source_bucket.blob(blob.name)
​​​​​​​​            new_blob = source_bucket.copy_blob(source_blob, destination_bucket, blob.name) 
​​​​​​​​            blob.delete(if_generation_match=None)
​​​​​​​​            print(f'File moved from {source_blob} to {new_blob}')
​​​​​​​​        else:
​​​​​​​​            print("File size is below 1MB\n")

    • requirement.txt

      ​​​​​​​​functions-framework==3.*
​​​​​​​​google-cloud-storage
​​​​​​​​google-cloud
  • 結果
    image

Key Technology

Example EX.17 (2024/12/10)

Project-Name

  • Build cloud DB and cloud function by CLI

Demo & Processing

  • step1. 創建DB

    ​​​​$ gcloud sql instances create [你的DB instance 名稱] --database-version=MYSQL_5_7 --cpu=2 --memory=4GB --root-password=[DB的密碼] --assign-ip --zone=[你想要的region+zone] --availability-type=zonal --no-backup
​​​​$ gcloud sql databases create [DB名稱] --instance=[DB instance 名稱]
​​​​$ gcloud sql connect [DB-Instance名稱] --user=root
​​​​> use mark-db;

    ​​​​>(mark-db) CREATE TABLE info (
​​​​id INT NOT NULL AUTO_INCREMENT,
​​​​firstname VARCHAR(20),
​​​​lastname VARCHAR(20),
​​​​age VARCHAR(3),
​​​​collegename VARCHAR(150),
​​​​PRIMARY KEY (id)
​​​​);
​​​​
​​​​>(mark-db) exit

  • step2. 在當前目錄下建立程式碼

    • main.py

      


































      ​​​​​​​​import sqlalchemy
​​​​​​​​#connection name we noted earlier
​​​​​​​​connection_name = "[你的project id:DB-Instance名稱]"
​​​​​​​​#database name
​​​​​​​​db_name = "[你的db名稱]"
​​​​​​​​db_user = "root"
​​​​​​​​db_password = "admin1234"
​​​​​​​​driver_name = 'mysql+pymysql'
​​​​​​​​query_string = dict({"unix_socket": "/cloudsql/{}".format(connection_name)})

​​​​​​​​def writeToSql(request):
​​​​​​​​   #You can change this to match your personal details
​​​​​​​​   stmt = sqlalchemy.text("INSERT INTO info ( firstname, lastname, age, collegename) values ('Sagadevan', 'Kounder', '21', 'XYZ College')")

​​​​​​​​   db = sqlalchemy.create_engine(
​​​​​​​​   sqlalchemy.engine.url.URL(
​​​​​​​​   drivername=driver_name,
​​​​​​​​   username=db_user,
​​​​​​​​   password=db_password,
​​​​​​​​   database=db_name,
​​​​​​​​   query=query_string,
​​​​​​​​   ),
​​​​​​​​   pool_size=5,
​​​​​​​​   max_overflow=2,
​​​​​​​​   pool_timeout=30,
​​​​​​​​   pool_recycle=1800
​​​​​​​​   )
​​​​​​​​   try:
​​​​​​​​      with db.connect() as conn:
​​​​​​​​        conn.execute(stmt)
​​​​​​​​        print("Insert successful")
​​​​​​​​   except Exception as e:
​​​​​​​​      print ("Some exception occured" + e)
​​​​​​​​      return 'Error: {}'.format(str(e))
​​​​​​​​   return 'ok'

    • requirement.txt

      ​​​​​​​​SQLAlchemy==1.3.12
​​​​​​​​PyMySQL==0.9.3

  • step3. 建立cloud function

    ​​​​$ gcloud functions deploy writeToSql --entry-point writeToSql --runtime python310 --trigger-http --allow-unauthenticated  --no-gen2 --source .
​​​​
​​​​$ gcloud functions deploy [cloud function 名稱] --entry-point [function的entry point] --runtime [python版本] --trigger-http --allow-unauthenticated  --no-gen2 --source [程式碼的位置]

  • 結果
    image

Key Technology

Example EX.18 (2024/12/10)

Project-Name

  • Build APP Engine to predict iris by CLI

Demo & Processing

  • step1. 根據之前的iris範例建立出對應的模型

    ​​​​$ mkdir test-iris; cd test-iris
​​​​$ git clone https://github.com/saedhussain/gcp_serverless_ml.git
​​​​$ pip install pandas; pip install scikit-learn
​​​​$ cd Iris_model; python 
​​​​$ python create_iris_model.py
​​​​$ export PROJECT_ID=$DEVSHELL_PROJECT_ID
​​​​$ export BUCKET=[取一個cloud storage名稱]
​​​​$ gsutil mb -p $PROJECT_ID -c standard -l [region名稱] gs://${BUCKET} 
​​​​$ gsutil cp ./[model名稱] gs://${BUCKET}

  • step2. 寫對應的app egine 程式碼

    • main.py

      























































      ​​​​​​​​import numpy as np
​​​​​​​​import os
​​​​​​​​import pickle
​​​​​​​​from flask import Flask, jsonify

​​​​​​​​app = Flask(__name__)

​​​​​​​​# 全局模型變數
​​​​​​​​model = None

​​​​​​​​# 下載模型檔案的函數
​​​​​​​​def download_model_file():
​​​​​​​​    from google.cloud import storage

​​​​​​​​    BUCKET_NAME = "iris-bk"
​​​​​​​​    PROJECT_ID = "avian-casing-435202-m5"
​​​​​​​​    GCS_MODEL_FILE = "iris_model_jan_2020_v1.pkl"

​​​​​​​​    client = storage.Client(PROJECT_ID)
​​​​​​​​    bucket = client.get_bucket(BUCKET_NAME)
​​​​​​​​    blob = bucket.blob(GCS_MODEL_FILE)

​​​​​​​​    folder = '/tmp/'
​​​​​​​​    if not os.path.exists(folder):
​​​​​​​​        os.makedirs(folder)
​​​​​​​​    blob.download_to_filename(folder + "local_model.pkl")

​​​​​​​​# 載入模型
​​​​​​​​def load_model():
​​​​​​​​    global model
​​​​​​​​    if not model:
​​​​​​​​        download_model_file()
​​​​​​​​        model = pickle.load(open("/tmp/local_model.pkl", 'rb'))

​​​​​​​​# 處理路徑格式的功能
​​​​​​​​@app.route('/features/<path:features>', methods=['GET'])
​​​​​​​​def iris_predict(features):
​​​​​​​​    global model

​​​​​​​​    # 確保模型已加載
​​​​​​​​    load_model()

​​​​​​​​    # 分解輸入的特徵
​​​​​​​​    try:
​​​​​​​​        feature_list = [float(i) for i in features.split("_")]
​​​​​​​​        if len(feature_list) != 4:
​​​​​​​​            return jsonify({"error": "Invalid number of features. Expected 4 values."}), 400
​​​​​​​​    except ValueError:
​​​​​​​​        return jsonify({"error": "Features must be numeric and separated by underscores."}), 400

​​​​​​​​    # 使用模型進行預測
​​​​​​​​    prediction = model.predict(np.array([feature_list]))
​​​​​​​​    return jsonify({"prediction": prediction[0]})

​​​​​​​​if __name__ == "__main__":
​​​​​​​​    app.run(host='0.0.0.0', port=8080)

    • requirements.txt

      ​​​​​​​​google-resumable-media==0.6.0
​​​​​​​​google-cloud-storage==1.30.0
​​​​​​​​google-cloud-bigquery==1.26.1
​​​​​​​​flask
​​​​​​​​pandas
​​​​​​​​numpy
​​​​​​​​scikit-learn

    • app.yaml

      ​​​​​​​​runtime: python310
​​​​​​​​service: iris-test

  • 結果
    輸入對應的指令就可以得到preediction的結果
    image

Key Technology

Example EX.19 (2024/12/17)

Project-Name

  • Build Iris-Prediction Model on Cloud Run by Artifact Registry

Demo & Processing

  • step1. 創建以下幾個檔案

    • train_model.py (訓練iris-prediction model)

      


























      ​​​​​​​​# -*- coding: utf-8 -*-
​​​​​​​​import pickle
​​​​​​​​from sklearn import datasets
​​​​​​​​from sklearn.model_selection import train_test_split
​​​​​​​​from sklearn import tree

​​​​​​​​# simple demo for traing and saving model
​​​​​​​​iris=datasets.load_iris()
​​​​​​​​x=iris.data
​​​​​​​​y=iris.target

​​​​​​​​#labels for iris dataset
​​​​​​​​labels ={
​​​​​​​​  0: "setosa",
​​​​​​​​  1: "versicolor",
​​​​​​​​  2: "virginica"
​​​​​​​​}

​​​​​​​​x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=.25)
​​​​​​​​classifier=tree.DecisionTreeClassifier()
​​​​​​​​classifier.fit(x_train,y_train)
​​​​​​​​predictions=classifier.predict(x_test)

​​​​​​​​#export the model
​​​​​​​​model_name = 'model.pkl'
​​​​​​​​print("finished training and dump the model as {0}".format(model_name))
​​​​​​​​pickle.dump(classifier, open(model_name,'wb'))

    • main.py

      


































      ​​​​​​​​import pickle

​​​​​​​​from flask import Flask, request, jsonify

​​​​​​​​app = Flask(__name__)

​​​​​​​​# Load the model
​​​​​​​​model = pickle.load(open('model.pkl', 'rb'))
​​​​​​​​labels = {
​​​​​​​​  0: "versicolor",
​​​​​​​​  1: "setosa",
​​​​​​​​  2: "virginica"
​​​​​​​​}

​​​​​​​​@app.route("/", methods=["GET"])
​​​​​​​​def index():
​​​​​​​​    """Basic HTML response."""
​​​​​​​​    body = (
​​​​​​​​        "<html>"
​​​​​​​​        "<body style='padding: 10px;'>"
​​​​​​​​        "<h1>Welcome to my Flask API</h1>"
​​​​​​​​        "</body>"
​​​​​​​​        "</html>"
​​​​​​​​    )
​​​​​​​​    return body

​​​​​​​​@app.route('/api', methods=['POST'])
​​​​​​​​def predict():
​​​​​​​​    # Get the data from the POST request.
​​​​​​​​    data = request.get_json(force = True)
​​​​​​​​    predict = model.predict(data['feature'])
​​​​​​​​    return jsonify(predict[0].tolist())

​​​​​​​​if __name__ == '__main__':
​​​​​​​​    app.run(debug = True, host = '0.0.0.0', port=8080)

    • requirements.txt

      ​​​​​​​​scikit-learn 
​​​​​​​​flask

    • Dockerfile

      










      ​​​​​​​​FROM python:3.9
​
​​​​​​​​WORKDIR /app

​​​​​​​​ADD . /app

​​​​​​​​RUN pip install -r requirements.txt

​​​​​​​​CMD ["python", "main.py"]

​​​​​​​​EXPOSE 8080

  • step2. 創建Arifact Registry，將Dockerfile創建出來的鏡像打包上傳
    image
    image

    ​​​​$ docker build -t asia-east1-docker.pkg.dev/avian-casing-435202-m5/mark-iris-docker/mark-iris:1.0 .
​​​​$ docker push asia-east1-docker.pkg.dev/avian-casing-435202-m5/mark-iris-docker/mark-iris:1.0

    image
    image

  • step3. 點選完Deploy之後
    image
    image
    image
    image

  • step4. 撰寫clent.py並執行

    












    ​​​​# -*- coding: utf-8 -*-
​​​​import requests
​​​​# Change the value of experience that you want to test
​​​​url = '[你的cloud run網址]/api'
​​​​feature = [[5.8, 4.0, 1.2, 0.2]]
​​​​labels ={
​​​​  0: "setosa",
​​​​  1: "versicolor",
​​​​  2: "virginica"
​​​​}

​​​​r = requests.post(url,json={'feature': feature})
​​​​print(labels[r.json()])

  • 執行client.py結果

$ python client.py 
setosa

Key Technology

Last changed by 
 
周名初
0
121
Published on HackMD