NASA 3! - HackMD

## 安裝及設定 OPNsense ### 共通網路配置 (正常情況都是連這個) * **WAN**: 140.112.187.49/27 * **LAN**: 172.16.127.254/16 * **OpenVPN**: 10.8.0.1/24 ### opnsense-master (vmid: 252) 有三張網卡 * **net0 - LAN**: * 沒有 VLAN tag * IP: 172.16.127.252/16 * Web GUI 在 8443 port * **net1 - WAN**: * VLAN tag: 187 * IP: 140.112.187.50/27 * **net2 - PFSYNC**: * VLAN tag: 254 * IP: 10.0.0.1/30 ### opnsense-backup (vmid: 253) 有三張網卡 * **net0 - LAN**: * 沒有 VLAN tag * IP: 172.16.127.253/16 * Web GUI 在 8443 port * **net1 - WAN**: * VLAN tag: 187 * IP: 140.112.187.51/27 * **net2 - PFSYNC**: * VLAN tag: 254 * IP: 10.0.0.2/30 ### 目前防火牆規則 * **WAN**: 除了 icmp, TCP port 80,443, UDP port 1194 以外的向內流量都 block。往外流量全部 pass。 * **LAN**: 所有連外的流量都 pass。 * **OpenVPN**: VPN client 可以戳 LAN 內的機器。 ### HAProxy 架構圖 ![image](https://hackmd.io/_uploads/ryXduNJ6Zl.png) ### 目前串好的 Application | 組別 | Domain Name | IP(s) | Port(s) | |:----------- |:-------------------- |:-------------- |:------- | | WiFi | apmap.csie.org | 172.16.127.110 | 80 | | NewPrinting | newprinting.csie.org | 172.16.127.103 | 80 | ### 串 Application Server 流程 (以 WiFi 組的為例) - Services -> HAProxy -> Settings - -> Real Servrers -> Clone Test_server ``` Name or Prefix: APMap_server FQDN or IP: 172.16.127.110 ``` - -> Virtual Services -> Backend Pools -> Clone Test_backend (advanced mode) ``` Name: APMap_backend Servers: APMap_server Health Monitor: health_monitor Unhealthy Threshold: 3 Healthy Threshold: 3 ``` > Health Monitor 暫時先關掉 - -> Rules & Checks -> Conditions -> Clone is_nasa3j ``` Name: is_apmap Host String: apmap.csie.org ``` - -> Rules & Checks -> Rules -> Clone redirect_to_nasa3j ``` Name: redirect_to_apmap Select conditions: is_apmap Use backend pool: APMap_backend ``` - -> Virtual Services -> Public Services -> Edit 1_HTTPS_frontend - Add corresponding certificate - Append to 'Select Rules': redirect_to_apmap ### 2026/03/26 - 上傳 OPNsense-26.1.2-dvd-amd64.iso 到 pve1 - Create VM - vmid: 254 - 進去 VM 裡面把系統裝好 - LAN: - 對應 pve1 上的 Network Device (net0) - OPNsense 內的 device: vtnet0 - ip: 172.16.127.254/12 (暫時12才能用csie vpn進去gui) - WAN: - 對應 pve1 上的 Network Device (net1) - OPNsense 內的 device: vtnet1 ### 2026/04/04 - 在 Proxmox GUI 把 Network Device (net1) 設 vlan tag 187 - 加入 gateway ![image](https://hackmd.io/_uploads/S14qPmCoZl.png) - 把 wan 啟用 (140.112.187.49/27)![image](https://hackmd.io/_uploads/HJ4aUwAjWe.png) - 在 System -> Settings -> General 設定 - Hostname: opnsense - Domain: nasa - 防火牆 Rules: - WAN: 加了一個允許連 csie vpn 後能 ping 防火牆 public ip 的 rule。 - LAN: 把預設允許所有從 LAN 來的流量的 rule 停用。等 VPN 好了之後，應該會只允許特定管理 ip 碰 opnsense GUI。 ### 2026/04/09 - VPN -> OpenVPN -> Instances -> Static Keys -> Add - Description: NASA_OpenVPN_key - Mode: crypt > 外部沒有 Static key 的 client 的流量會被 drop，增加 server 安全性 - VPN -> OpenVPN -> Instances -> Edit - 打開設定畫面左上角的 advanced mode - TLS static key 選剛剛加的 NASA_OpenVPN_key - Local Network 從 172.16.127.0/24 改成 172.16.0.0/16 - Keep alive interval 設為 10, Keep alive timeout 設為 120 > 解決2分鐘內沒流量自動斷線重連問題 (不確定是不是只有 Tunnelblick 會) - Firewall -> Rules -> OpenVPN -> Edit - Source: OpenVPN net - Destination: LAN net - Interfaces -> LAN - IPv4 Address: 172.16.127.254/16 > 原先是 /12，但現在有 VPN 所以改回正確的 /16 - Firewall -> Aliases -> Add - Name: pves - Type: Host(s) - Content: 172.16.127.16-172.16.127.18 - Firewall -> Rules \[new\] -> Add - Protocol: TCP - Source Address/Port: pves/Any - Destination Address/Port: This Firewall/80 - Clone 剛才的 rule 把 Destination Port 改 443 > 讓 VPN 掛掉的時候有辦法用 pve 當跳板，連到 OPNsense Web GUI - Firewall -> Settings -> Advanced - Disable anti-lockout 打勾 > 讓 pve1-3 外的 LAN 機器不能連 GUI - System -> Settings -> Administration - Web GUI -> Listen interfaces: LAN ### 2026/04/10 VPN with internal LDAP (172.16.127.105) - System -> Trust -> Authorities -> Add ``` Description: NASA-LDAP-CA Certificate data: <content of ca.crt on 172.16.127.105> ``` - System -> Servers -> Add ``` Descriptive name: NASA_LDAP Type: LDAP Hostname or IP address: 172.16.127.105 Port value: 636 Transport: SSL - Encrypted Protocol version: 3 Bind credentials: User DN: cn=admin,dc=csie,dc=ntu,dc=edu,dc=tw Password: <DN password> Search scope: Entire Subtree Base DN: dc=csie,dc=ntu,dc=edu,dc=tw Authentication containers: ou=people,dc=csie,dc=ntu,dc=edu,dc=tw User naming attribute: uid ``` - VPN -> OpenVPN -> Instances -> Edit ``` Authentication: Local Database, NASA_LDAP Username as CN: ✅ ``` ### 2026/04/11 #### Get .csie.org. domain name - Go to https://csie.org - Login and add A record ``` hostname: b13902061.csie.org. ip: 140.112.187.49 ``` - After the DNS takes effect, add CNAME record ``` alias name: nasa3j.csie.org FQDN: b13902061.csie.org. ``` #### Setup ACME and HAProxy - System -> Firmware -> Status -> Check for updates - update opnsense to latest version (26.1.6) - System -> Firmware -> Plugins - install os-acme-client and os-haproxy - Services -> ACME Client -> Settings ![image](https://hackmd.io/_uploads/ry8mvIwnWe.png) - Services -> ACME Client -> Accounts -> Add ![image](https://hackmd.io/_uploads/B1GfDIw3-l.png) - Services -> ACME Client -> Challenge Types -> Add ![image](https://hackmd.io/_uploads/HyODvIPh-g.png) - Services -> ACME Client -> Automations -> Add ![image](https://hackmd.io/_uploads/HJxoDLD3be.png) - Services -> ACME Client -> Certificates -> Add ![image](https://hackmd.io/_uploads/SJ34OUDhbg.png) - Services -> HAProxy -> Settings -> Settings -> Service ![image](https://hackmd.io/_uploads/H13PtLD3Ze.png) - Services -> HAProxy -> Settings -> Settings -> Global Parameters ![image](https://hackmd.io/_uploads/rJG0K8Dh-l.png) - Services -> HAProxy -> Settings -> Settings -> Default Parameters ![image](https://hackmd.io/_uploads/SyAZ5Ivnbg.png) - Interfaces -> Virtual IPs -> Settings -> Add ``` Mode: IP Alias Interface: Loopback Network / Address: 127.4.4.3/32 ``` - Firewall -> Aliases -> Add ``` Name: HAProxy_ports Type: Port(s) Content: 80 443 ``` - Firewall -> Rules \[new\] - Replace the rules that block access to OPNsense Web GUI except pves with Destination Port 8443 - Add ``` Interface: WAN Version IPv4 Portocol: TCP Source/Port: Any/Any Destination/Port: WAN address/HAProxy_ports ``` - System -> Settings -> Administration ``` TCP port: 8443 HTTP Redirect: ✅ Disable web GUI redirect rule ``` - Services -> HAProxy -> Settings -> Real Servers -> Add ![image](https://hackmd.io/_uploads/HJYX2Lwn-x.png) - Services -> HAProxy -> Settings -> Virtual Services -> Backend Pools -> Add (advanced mode) ``` Name: SSL_backend Mode: TCP (Layer 4) Proxy Protocol: Version 2 Servers: SSL_server ``` - Services -> HAProxy -> Settings -> Virtual Services -> Public Services -> Add ![image](https://hackmd.io/_uploads/BJlda8D2-x.png) - Services -> ACME Client -> Certificates - Click `Issue or renew certificate` on the certificate Commands - Services -> HAProxy -> Settings -> Rules & Checks -> Conditions -> Add ``` Name: NoSSL_condition Condition type: ssl_fc - Traffic is SSL (locally deciphered) Negate condition: ✅ ``` - Services -> HAProxy -> Settings -> Rules & Checks -> Rules -> Add ``` Name: HTTPtoHTTPS_rule Select conditions: NoSSL_condition Match Mode: IF [default] Type: http-request Action: redirect Options/Values: scheme https code 301 ``` - Services -> HAProxy -> Settings -> Virtual Services -> Public Services -> Add (advanced mod) ``` Name: 1_HTTPS_frontend Listen Addresses: 127.4.4.3:443 Bind option pass-through: accept-proxy Type: HTTP / HTTPS (SSLD offloading) [default] Default Backend Pool: None Enable SSL offloading: ✅ Certificates: b13902061.csie.org (ACME Client) Enable Advanced settings: ✅ ``` - Services -> HAProxy -> Settings -> Virtual Services -> Public Services -> Add (advanced mod) ``` Name: 1_HTTP_frontend Listen Addresses: 127.4.4.3:80 Bind option pass-through: accept-proxy Type: HTTP / HTTPS (SSLD offloading) [default] Default Backend Pool: None Enable SSL offloading: ▫️ Certificates: b13902061.csie.org (ACME Client) Enable Advanced settings: ✅ Select Rules: HTTPtoHTTPS_rule ``` #### Test HAProxy - Services -> HAProxy -> Settings -> Real Servers -> Add ``` Name: Test_server Type: static FQDN or IP: 172.16.127.105 Port: 8080 Verify SSL Certificate: ▫️ ``` - Services -> HAProxy -> Settings -> Virtual Services -> Backend Pools -> Add ``` Name: Test_backend Mode: HTTP (Layer 7) [default] Servers: Test_server ``` - Services -> HAProxy -> Settings -> Rules & Checks -> Conditions -> Add ``` Name: is_nasa3j Condition type: hdr - HTTP Host Header matches Host String: nasa3j.csie.org ``` - Services -> HAProxy -> Settings -> Rules & Checks -> Rules -> Add ``` Name: redirect_to_nasa3j Select conditions: is_nasa3j Match Mode: IF [default] Type: Use specified Backend Pool Use backend pool: Test_backend ``` ``` Name: set_x_forwarded_for Select conditions: Nothing seleted Type: http-request Action: set-header Options/Values: X-Forwarded-For %[src] ``` - Services -> HAProxy -> Settings -> Virtual Services -> Public Services -> Edit 1_HTTPS_frontend ``` Select Rules: set_x_forwarded_for redirect_to_nasa3j ``` - Run the following on 172.16.127.105 `sudo docker run -d -p 8080:80 traefik/whoami` - Open any browser and type nasa3j.csie.org > 參考: https://forum.opnsense.org/index.php?topic=23339.0 ### 2026/04/16 #### Fix new ACME Cert (2026.nasa.csie.org) - Services -> HAProxy -> Settings -> Virtual Services -> Public Services - Edit 0_SNI_frontend - Click `Clear All` under Select Rules - Edit 1_HTTPS_frontend ``` Select Rules: redirect_acme_challenges route_to_test_server set_x_forwarded_for ``` - Services -> ACME Client -> Certificates - Click `Issue or renew certificate` - System -> Trust -> Certificates -> Edit OpenVPN-Server-Cert ``` Common Name: 2026nasa.csie.org ``` - Lobby -> Dashboard - Restart OpenVPN server NASA_OpenVPN ### 2026/04/18 - Firewall -> Settings -> Advanced ``` Firewall Optimization: conservative ``` > 看這樣會不會 SSH 比較不會出現 broken pipe 和 VPN 比較不會有 LDAP bind error [; Can't contact LDAP server] 的問題 ### 2026/04/19 #### 加入 Health Monitor - Services -> HAProxy -> Settings -> Rules & Checks -> Health Monitors -> Add ``` Name: health_monitor Check type: HTTP [default] Check interval: 10s HTTP method: HEAD Request URI: / ``` #### 串 WiFi 組和 NewPrinting 組的 Web server > [目前串好的 Application](https://hackmd.io/PmDU33L_RcuQyhbWxzLlPw?both=#%E7%9B%AE%E5%89%8D%E4%B8%B2%E5%A5%BD%E7%9A%84-Application) > [設定流程](https://hackmd.io/PmDU33L_RcuQyhbWxzLlPw?both=#%E4%B8%B2-Application-Server-%E6%B5%81%E7%A8%8B-%E4%BB%A5-WiFi-%E7%B5%84%E7%9A%84%E7%82%BA%E4%BE%8B) ### 2026/04/29 #### OPNsense HA - Clone 原先的 OPNsense VM ``` vmid: 252 name: opnsense-master ``` - Console 連到 opnsense-master ``` LAN: 172.16.127.252/16 WAN: 140.112.187.50 ``` `Reload all services` - Clone 原先的 OPNsense VM ``` vmid: 253 name: opnsense-backup ``` - Console 連到 opnsense-backup ``` LAN: 172.16.127.253/16 WAN: 140.112.187.51/27 ``` `Reload all services` - opnsense-master 和 opnsense-backup Add network device ``` VLAN Tag: 254 ``` #### 接下來的設定在 master 和 backup 都要做 - Interfaces -> Assignments -> Assign a new interface ``` Device: vtnet2 Description: PFSYNC ``` - Interfaces -> PFSYNC ``` IPv4 Configuration Type: Static IPv4 IPv4 address (master): 10.0.0.1/30, (backup): 10.0.0.2/30 ``` - Firewall -> Rules[new] -> Add ``` Interface: LAN, WAN Action: Pass Direction: In Protocol: CARP Source: any Destination: any ``` ``` Interface: PFSYNC Action: Pass Direction: In Protocol: any Source: any Destination: any ``` - Interfaces -> Virtual IPs -> Settings -> Add (advanced mode) ``` Mode: CARP Interface: LAN Network / Address: 172.16.127.254/16 Password: <PASSWORD> VHID Group: 1 Advskew (master): 0, (backup): 100 Description: VIP LAN ``` ``` Mode: CARP Interface: WAN Network / Address: 140.112.187.49/27 Password: <PASSWORD> VHID Group: 2 Advskew (master): 0, (backup): 100 Description: VIP WAN ``` - System -> High Availability -> Settings (master) ``` Synchronize all states via: PFSYNC Synchronize Peer IP: 10.0.0.2 Synchronize Config: 10.0.0.2 Remote System Username: root Remote System Password: <PASSWORD> Services: `Select All` ``` (backup) ``` Synchronize all states via: PFSYNC Synchronize Peer IP: 10.0.0.1 ``` > Don't configure XMLRPC sync on opnsense-backup - System -> Settings -> Administration -> Web GUI ``` Listen Interfaces: LAN, PFSYNC ``` #### 接下來的設定只需在 master 做 - Firewall -> Aliases -> Add ``` Name: VIP_LAN Type: Host(s) Content: 172.16.127.254 ``` ``` Name: VIP_WAN Type: Host(s) Content: 140.112.187.49 ``` - Firewall -> NAT -> Outbound -> Add ``` Interface: WAN Source address: LAN net Translation / target: VIP WAN ``` - Firewall -> Rules[new] -> Edit ``` Interface: WAN Action: Pass Protocol: TCP Destination: WAN address -> VIP_WAN Port: HAProxy_ports ``` - System -> High Availability -> Status `Synchronize and reconfigure all` ### 2026/05/04 - Services -> Unbound DNS -> General ``` Network Interfaces: LAN ``` - Services -> Unbound DNS -> Overrides -> Add ``` Host: iden1 Domain: nasa IP address: 172.16.127.105 ``` ``` Host: iden2 Domain: nasa IP address: 172.16.127.109 ``` - VPN -> OpenVPN -> Instances -> Edit ``` DNS Domain list: nasa DNS Domain search list: nasa DNS Servers: 172.16.127.254 ``` #### On hosts in LAN: - ssh to 172.16.127.105 (iden1) and 172.16.127.109 (iden2) and execute the following ``` sudo nmcli connection modify "Wired connection 1" ipv4.dns-search "nasa" sudo nmcli connection modify "Wired connection 1" ipv4.dns "172.16.127.254" sudo nmcli connection up "Wired connection 1" ``` > We can now connect to 172.16.127.105 on iden1 and iden2 or on our PC with VPN connected using the name 'iden1' or 'iden1.nasa'. > For example, `ping iden1` ### 2026/05/14 #### Add static route to VPN on pve1-3 - Add `post-up ip route add 10.8.0.0/24 via 172.16.127.254 dev vmbr0` under `iface vmbr0 inet static` in /etc/network/interfaces - ``` ifreload -a ``` ``` /var/log/system/latest.log /var/log/openvpn/latest.log /var/log/haproxy/latest.log ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.