---
# System prepended metadata

title: Reverse SSH Tunneling between EC2 (Amazon Linux) and RPi4
tags: [AWS]

---

:::warning
It's easier to use FRP, please refer to [FRP+Nginx+Certbot with EC2](https://hackmd.io/CAk-fCCLSpi4Vih-FQ5hxA?view=)
:::

# Reverse SSH Tunneling between EC2 (Amazon Linux) and RPi4

[TOC]

## Introduction
Remote management of a Raspberry Pi 4 (RPi4) without a public IP address can be challenging.  

This guide outlines how to securely access your RPi4 from anywhere by setting up a reverse SSH tunnel via an Amazon EC2 instance.  

This method enhances security by avoiding direct exposure of the RPi4 to the internet while providing reliable remote access.  

By following these instructions, you will establish a secure and flexible connection, leveraging the AWS infrastructure to manage your device remotely.

---

## 1. Settings on EC2

### Security Group Example
- Allow Inbound SSH (port 22) from `0.0.0.0/0`.
- Allow Inbound TCP port `2222` from `0.0.0.0/0`.
- Allow Inbound TCP port `8080` from `0.0.0.0/0`.
- Allow Inbound TCP port `1194` from `0.0.0.0/0` (for OpenVPN).

![Screenshot 2025-01-07 at 4.58.48 AM](https://hackmd.io/_uploads/H1BzE6YIke.png)

### `sshd_config`
1. Edit the SSH configuration file:
   ```sh
   sudo vim /etc/ssh/sshd_config
   ```
2. Enable `GatewayPorts` to allow remote port forwarding:
    ```
    AllowTcpForwarding yes
    ClientAliveCountMax 5
    ClientAliveInterval 30
    GatewayPorts yes
    TCPKeepAlive yes
    # LogLevel DEBUG
    ```
   ![Screenshot 2025-01-09 at 11.45.17 PM](https://hackmd.io/_uploads/S17mydaI1l.png)
(https://hackmd.io/_uploads/ByAVSOFLJx.png)
3. Restart the SSH service:
   ```sh
   sudo systemctl restart sshd
   
   # If want to read log
   # sudo journalctl -u sshd | vim -
   ```

---

## 2. Settings on RPi4

### `.ssh/config`
1. Edit the SSH configuration file:
   ```sh
   vim ~/.ssh/config
   ```
2. Add the following configuration:
   ```
   Host ec2-for-tunneling
       HostName ec2-public-ip
       User ec2-user
       IdentityFile /home/pi/.ssh/ec2.pem
   ```
   - Replace `ec2-public-ip` with the public IP of your EC2 instance.

---

### Forward EC2:2222 to RPi4:22 (SSH RPi4 via EC2)
This allows you to connect to RPi4 via EC2 using:
```sh
ssh -p 2222 -i <rpi4_identity> pi@ec2-public-ip
```

1. Create a service file:
   ```sh
   sudo vim /etc/systemd/system/autossh-tunnel-2222-22.service
   ```
2. Add the following content:
   ```ini
   [Unit]
   Description=AutoSSH tunnel from EC2:2222 to localhost:22
   After=network.target

   [Service]
   User=pi
   ExecStartPre=/bin/bash -c 'STALE_PIDS=$(ss -tnp state CLOSE-WAIT sport = :22 | awk "{print $6}" | cut -d, -f2 | xargs -I{} echo {} | cut -d= -f2 | sort -u); if [ -n "$STALE_PIDS" ]; then sudo kill -9 $STALE_PIDS || true; fi'
   # ExecStart=/usr/bin/autossh -M 0 -N -R 2222:localhost:22 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
   ExecStart=/usr/bin/ssh -N -R 2222:localhost:22 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
   Restart=always
   RestartSec=5
   ExecStop=/usr/bin/pkill -f "ssh -N -R 2222:localhost:22"

   [Install]
   WantedBy=multi-user.target
   ```
   - **User**: ==Remember to modify user if it's not `pi`==
   - **ExecStartPre**: Cleans up stale SSH processes in `CLOSE-WAIT` state on EC2 before starting the tunnel. The `|| true` ensures the service starts even if this step fails.
   - **ExecStart**: Uses `autossh` with robust retry logic.
   - **Restart=always**: Ensures the service restarts after failure.
   - **ExecStop**: Kills stale SSH processes when the service stops.

3. Start and enable the service:
   ```sh
   sudo systemctl daemon-reload
   sudo systemctl restart autossh-tunnel-2222-22
   sudo systemctl enable autossh-tunnel-2222-22
   sudo systemctl status autossh-tunnel-2222-22 --no-pager
   
   # If want to read the log
   # sudo journalctl -u autossh-tunnel-2222-22 | vim -
   ```

---

### Forward EC2:{port} to RPi4:{port}
This allows you to access services running on RPi4:{port} via EC2:{port}.

1. Run the following script

    Simply edit the ports listed
    ```sh=
    #!/bin/bash
    ports=(1194 3306 8080 8081 8484 8585 8586 8587 8596)

    service_template='[Unit]
    Description=AutoSSH tunnel from EC2:PORT to localhost:PORT
    After=network.target

    [Service]
    User=pi
    ExecStartPre=/bin/bash -c '"'"'STALE_PIDS=$(ss -tnp state CLOSE-WAIT sport = :PORT | awk "{print $6}" | cut -d, -f2 | xargs -I{} echo {} | cut -d= -f2 | sort -u); if [ -n "$STALE_PIDS" ]; then sudo kill -9 $STALE_PIDS || true; fi'"'"'
    # ExecStart=/usr/bin/autossh -M 0 -N -R PORT:localhost:PORT -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
    ExecStart=/usr/bin/ssh -N -R PORT:localhost:PORT -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
    Restart=always
    RestartSec=5
    ExecStop=/usr/bin/pkill -f "ssh -N -R PORT:localhost:PORT"

    [Install]
    WantedBy=multi-user.target'

    for port in "${ports[@]}"; do
        service_content="${service_template//PORT/$port}"
        service_file="/etc/systemd/system/autossh-tunnel-${port}-${port}.service"
        echo "$service_content" | sudo tee "$service_file" > /dev/null
        sudo systemctl daemon-reload
        echo "Service file for port $port has been updated: $service_file"
    done
    ```

2. Start and enable the service:
   ```sh=
    #!/bin/bash

    # List of ports
    ports=(1194 3306 8080 8081 8484 8585 8586 8587 8596)

    # Loop through each port and enable/start/restart the service
    for port in "${ports[@]}"; do
        service_name="autossh-tunnel-${port}-${port}.service"

        # Enable the service (if not already enabled)
        sudo systemctl enable "$service_name"

        # Restart the service
        echo "Restarting service $service_name"
        sudo systemctl restart "$service_name"

        # Optionally, check the status of the service
        echo "Status of $service_name:"
        sudo systemctl status "$service_name" --no-pager

        echo "Service for port $port has been enabled and managed: $service_name"
    done
   ```

---

### Forward for OpenVPN
:::warning
**WireGuard** only supports UDP, while SSH Tunneling only support TCP.  
Therefore it's tricky to use **WireGuard** with SSH Tunneling.  

We chose to use **OpenVPN** here.
:::
This allows you to securely access your RPi4 and its local network via OpenVPN.

1. **Install PiVPN on RPi4**:
   - Update the system:
     ```sh
     sudo apt update && sudo apt upgrade -y
     ```
   - Install PiVPN:
     ```sh
     curl -L https://install.pivpn.io | bash
     ```
     - Choose **OpenVPN** as the VPN type.
     - Select **TCP** as the protocol.
     - Use the default port **1194**.
     - Set up a static IP for your RPi4 if not already configured.
     - Use a DNS name or the public IP of your EC2 instance for the VPN connection.
   - Reboot the RPi4:
     ```sh
     sudo reboot
     ```

2. **Generate a client profile**:
   After installation, generate a client profile for your device:
   ```sh
   pivpn add
   ```
   - Provide a name for the client (e.g., `mydevice`).
   - The profile will be saved in `/home/pi/ovpns/`.

3. **Transfer the client profile**:
   Use `scp` to securely transfer the `.ovpn` file to your local machine:
   ```sh
   scp pi@rpi4-ip:/home/pi/ovpns/mydevice.ovpn ~/
   ```
   Replace `rpi4-ip` with the IP address of your RPi4.

4. **Set up SSH tunneling as a service**:
   Create a systemd service to forward traffic from EC2:1194 to RPi4:1194:
   ```sh
   sudo vim /etc/systemd/system/autossh-tunnel-1194-1194.service
   ```
   Add the following content:
   ```ini
   [Unit]
   Description=AutoSSH tunnel from EC2:1194 to localhost:1194
   After=network.target

   [Service]
   User=pi
   ExecStartPre=/bin/bash -c 'STALE_PIDS=$(ss -tnp state CLOSE-WAIT sport = :1194 | awk "{print $6}" | cut -d, -f2 | xargs -I{} echo {} | cut -d= -f2 | sort -u); if [ -n "$STALE_PIDS" ]; then sudo kill -9 $STALE_PIDS || true; fi'
   # ExecStart=/usr/bin/autossh -M 0 -N -R 1194:localhost:1194 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
   ExecStart=/usr/bin/ssh -N -R 1194:localhost:1194 -o "ServerAliveInterval 30" -o "ServerAliveCountMax 5" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" -o "UserKnownHostsFile=/dev/null" -o "StrictHostKeyChecking=no" -v ec2-for-tunneling
   Restart=always
   RestartSec=5
   ExecStop=/usr/bin/pkill -f "ssh -N -R 1194:localhost:1194"

   [Install]
   WantedBy=multi-user.target
   ```
   - Replace `ec2-for-tunneling` with the hostname defined in your `~/.ssh/config`.
   - **ExecStartPre**: Cleans up stale SSH processes in `CLOSE-WAIT` state on EC2 before starting the tunnel. The `|| true` ensures the service starts even if this step fails.
   - **ExecStart**: Uses `autossh` with robust retry logic.
   - **Restart=always**: Ensures the service restarts after failure.
   - **ExecStop**: Kills stale SSH processes when the service stops.

5. **Reload systemd and start the service**:
   ```sh
   sudo systemctl daemon-reload
   sudo systemctl restart autossh-tunnel-1194-1194
   sudo systemctl enable autossh-tunnel-1194-1194
   sudo systemctl status autossh-tunnel-1194-1194 --no-pager
   ```

6. **Ensure OpenVPN Starts After SSH Tunnel**:
   To ensure the `openvpn` service starts **after** the `autossh-tunnel-1194-1194` service, create a custom OpenVPN service file with the appropriate dependencies.

   1. **Create a Custom OpenVPN Service File**:
      ```sh
      sudo vim /etc/systemd/system/openvpn-server.service
      ```

   2. **Add the Following Content**:
      ```ini
       [Unit]
       Description=AutoSSH tunnel from EC2:1194 to localhost:1194
       After=network.target

       [Service]
       User=pi
       ExecStartPre=/bin/bash -c 'STALE_PIDS=$(ss -tnp state CLOSE-WAIT sport = :1194 | awk "{print \$6}" | cut -d, -f2 | xargs -I{} echo {} | cut -d= -f2 | sort -u); if [ -n "$STALE_PIDS" ]; then sudo kill -9 $STALE_PIDS || true; fi'
       ExecStart=/usr/bin/autossh -M 0 -N -R 1194:localhost:1194 -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -o "ExitOnForwardFailure=yes" -o "TCPKeepAlive=yes" ec2-for-tunneling
       Restart=always
       RestartSec=5
       ExecStop=/usr/bin/pkill -f "ssh -N -R 1194:localhost:1194"

       [Install]
       WantedBy=multi-user.target
      ```
      - **`After=autossh-tunnel-1194-1194.service`**: Ensures `openvpn-server.service` starts after `autossh-tunnel-1194-1194.service`.
      - **`Requires=autossh-tunnel-1194-1194.service`**: Ensures `autossh-tunnel-1194-1194.service` is started before `openvpn-server.service`.

   3. **Reload Systemd and Restart Services**:
      Reload the systemd daemon to apply the changes:
      ```sh
      sudo systemctl daemon-reload
      ```

      Restart the services to ensure the dependency is respected:
      ```sh
      sudo systemctl restart autossh-tunnel-1194-1194
      sudo systemctl restart openvpn-server
      ```

   4. **Enable the Services**:
      Enable the services to start on boot:
      ```sh
      sudo systemctl enable autossh-tunnel-1194-1194
      sudo systemctl enable openvpn-server
      ```

   5. **Disable the Original OpenVPN Service**:
      Disable the original `openvpn.service` to avoid conflicts:
      ```sh
      sudo systemctl disable openvpn
      sudo systemctl stop openvpn
      ```

   6. **Verify the Dependency**:
      Check the status of the services to ensure they are running and the dependency is respected:
      ```sh
      sudo systemctl status autossh-tunnel-1194-1194
      sudo systemctl status openvpn-server
      ```

7. **Modify the `.ovpn` file**:
   Open the `.ovpn` file you transferred earlier and update the `remote` line to use `localhost` instead of the RPi4's IP:
   ```=
   client
   dev tun
   proto tcp
   remote ec2-public-ip 1194
   ```
   ![Screenshot 2025-01-07 at 5.00.48 AM](https://hackmd.io/_uploads/H1aY4atL1g.png)

8. **Connect to OpenVPN**:
   - Import the `.ovpn` file into your OpenVPN client.
   - Start the OpenVPN connection using the imported profile.
   - Once connected, you will have secure access to your RPi4 and its local network.

---

## 3. Testing the Setup

### Test SSH Access
From your PC, connect to RPi4 via EC2:
```sh
ssh -p 2222 -i <rpi4_identity> pi@ec2-public-ip
```

You can also configure `~/.ssh/config` on your PC for easier access:
```
Host rpi4-ec2-tunneling
    HostName ec2-public-ip
    Port 2222
    User pi
    PreferredAuthentications publickey
    IdentityFile <rpi4_identity>
```

### Test Port Forwarding
If you have a service running on RPi4:8080, you can access it via:
```
http://ec2-public-ip:8080
```

### Test OpenVPN Access
- Ensure the OpenVPN client is connected.
- Verify access to the RPi4 and its local network.

---

## 4. Troubleshooting

### Common Issues
1. **Connection Refused**:
   - Ensure the Security Group allows inbound traffic on your ports (e.g., `2222`, `8080`, `1194`, ...).
   - Verify the SSH tunnel is active using `sudo systemctl status autossh-tunnel-*`.

2. **Permission Denied**:
   - Ensure the SSH key (`ec2.pem`) has the correct permissions:
     ```sh
     chmod 600 /home/pi/.ssh/ec2.pem
     ```

3. **GatewayPorts Not Working**:
   - Double-check that `GatewayPorts yes` is set in `/etc/ssh/sshd_config` on the EC2 instance.

4. **Stale SSH Processes**:
   - If the tunnel fails to reconnect, manually clean up stale SSH processes on the EC2 instance:
     ```sh
     sudo kill -9 $(sudo lsof -t -i :2222)
     sudo kill -9 $(sudo lsof -t -i :8080)
     sudo kill -9 $(sudo lsof -t -i :1194)
     ```

5. **Network Unreachable**:
   - Ensure the RPi4 has a stable network connection.
   - Test the connection to the EC2 instance:
     ```sh
     ping ec2-public-ip
     ```

---

## 5. Additional Notes
- Use `autossh` to ensure the tunnel remains active even after disconnections.
- Replace `ec2-public-ip` with the actual public IP of your EC2 instance.
- For better security, restrict the source IP in the Security Group instead of using `0.0.0.0/0`.
- Monitor the tunnel status using:
  ```sh
  sudo journalctl -u autossh-tunnel-2222-22 | vim -
  sudo journalctl -u autossh-tunnel-8080-8080 | vim -
  sudo journalctl -u autossh-tunnel-1194-1194 | vim -
  ```

By following this guide, you can securely access your RPi4 and its local network from anywhere using SSH tunneling and OpenVPN via an EC2 instance. The SSH tunnels are set up as services to ensure they remain active and automatically reconnect after failures or reboots. The `ExecStartPre` commands now include `|| true` to ignore failures and ensure the services start reliably. The `openvpn-server.service` is configured to start after the `autossh-tunnel-1194-1194` service to ensure proper dependency handling, and the original `openvpn.service` is disabled to avoid conflicts.