# **meeting 11/16**

**Advisor: Prof. Wei-Ho Chung \
Presenter: Shao-Heng Chen \
Date: Nov 16, 2023**
<!-- Chih-Yu Wang -->
<!-- Wei-Ho Chung -->

## **System model**

- Downlink RIS-aided MU-MISO system
- $N_k$ single-antenna users ($N_r = 1$)
- one BS with $N_t$ antennas
- one RIS with $N_s$ elements
- The received signal at the $k$-th users
$$
\begin{align*}
y_k &= (\mathbf{h}_{k, 2} \mathbf{\Phi} \mathbf{H}_1 + \mathbf{h}_{k, 3}) \mathbf{F} \mathbf{x} + n_k,
\end{align*}
$$
- $n_k$ is the additive receiver noise at the $k$-th user, $n_k \sim \mathcal{CN}(0, \sigma_n^2)$ for all $k$
- $\mathbf{x} \in \mathbb{C}^{N_k \times 1}$ is the data streams for $N_k$ users
- $\mathbf{F} \in \mathbb{C}^{N_t \times N_k}$ is the transmission beamforming matrix
- Max Ration Transmission (MRT)
- SVD beamforming + equal power allocation
- $\mathbf{\Phi} \triangleq diag(\phi_1, ..., \phi_{N_s}) \in \mathbb{C}^{N_s \times N_s}$ is the diagonal reflection matrix of the RIS
- $\phi_i = \beta(\varphi_i) \cdot e^{j\varphi_{i}}, \; \forall i = 1, ..., N_s$
- $\beta(\varphi_i) = (1 - \beta_{min}) \cdot (\frac{\sin(\varphi_i - \mu) + 1}{2})^\kappa + \beta_{min}$
- $\varphi_i = \hat{\varphi_i} + \varphi_i'$
- actual phase shift = desired phase shift + phase error
- $\hat{\varphi_i} \in \mathcal{A} = \{ e^{(j\frac{ \; 2\pi n \;}{2^{bits \;}})} \}_{n = 0}^{ 2^{bits - 1}}, \; \forall i = 1, ..., N_s$
- $\varphi_i'$ folows the von Mises distribution with PDF $f(\varphi_i'(\mu, \kappa)) = \frac{\;e^{\kappa \cos(\varphi_i' - \mu)}\;\;\;}{\;2\pi I_0(\kappa)\;}$
- $I_0(\kappa)$ is the modified Bessel function of the first kind of order $0$
- The stacked received signal of all $N_k$ users
$$
\begin{align*}
\mathbf{y} &= (\mathbf{H}_{2} \mathbf{\Phi} \mathbf{H}_1 + \mathbf{H}_{3}) \mathbf{F} \mathbf{x} + \mathbf{n} = \tilde{\mathbf{H}}\mathbf{F} \mathbf{x} + \mathbf{n},
\end{align*}
$$
- $\mathbf{H}_1 \in \mathbb{C}^{N_s \times N_t}$ is the BS-RIS channel
- $\mathbf{H}_{2} \in \mathbb{C}^{N_k \times N_s}$ is the RIS-users channel
- $\mathbf{h}_{k, 2} \in \mathbb{C}^{1 \times N_s}, \forall k = 1, ..., N_k$ is the channel vector from RIS to the $k$-th user
$$
\begin{align*}
\mathbf{h}_{k, 2} =& \ \hat{\mathbf{h}}_{k, 2} + \Delta\mathbf{h}_{k, 2} \in \mathbb{C}^{1 \times N_s}, \\
\Delta\mathbf{h}_{k, 2} &= \psi\frac{\mathbf{h}_{k, 2}^{N_1}}{\| \mathbf{h}_{k, 2}^{N_1} \|_2}, \;\; \mathbf{h}_{k, 2}^{N_1} \sim \mathcal{CN}(0, 1),
\end{align*}
$$
- $\mathbf{H}_{3} \in \mathbb{C}^{N_k \times N_t}$ is the BS-users channel
- $\mathbf{h}_{k, 3} \in \mathbb{C}^{1 \times N_t}, \forall k = 1, ..., N_k$ is the channel vector from BS to the $k$-th user
$$
\begin{align*}
\mathbf{h}_{k, 3} =& \ \hat{\mathbf{h}}_{k, 3} + \Delta\mathbf{h}_{k, 3} \in \mathbb{C}^{1 \times N_t}, \\
\Delta\mathbf{h}_{k, 3} &= \psi\frac{\mathbf{h}_{k, 3}^{N_2}}{\| \mathbf{h}_{k, 3}^{N_2} \|_2}, \;\; \mathbf{h}_{k, 3}^{N_2} \sim \mathcal{CN}(0, 1),
\end{align*}
$$

## **Channel model**
### **mmWave Systems**
- Because of its small wavelength, mmWave encounters challenges in diffracting around obstacles. Consequently, mmWave channels display a sparse multipath structure, commonly described by the **Saleh-Valenzuela (SV)** channel model.
- P. Wang, J. Fang, L. Dai and H. Li, "[Joint Transceiver and Large Intelligent Surface Design for Massive MIMO mmWave Systems](https://ieeexplore.ieee.org/document/9234098)," in *IEEE Transactions on Wireless Communications*, vol. 20, no. 2, pp. 1052-1064, Feb. 2021. (Cited by 80)
- K. Ying, Z. Gao, S. Lyu, Y. Wu, H. Wang and M. -S. Alouini, "[GMD-Based Hybrid Beamforming for Large Reconfigurable Intelligent Surface Assisted Millimeter-Wave Massive MIMO](https://ieeexplore.ieee.org/abstract/document/8964330)," in *IEEE Access*, vol. 8, pp. 19530-19539, 2020. (Cited by 91)
- The $\mathbf{H}_1 \in \mathbb{C}^{N_s \times N_t}, \hat{\mathbf{h}}_{k, 2} \in \mathbb{C}^{1 \times N_s}$ and $\hat{\mathbf{h}}_{k, 3} \in \mathbb{C}^{1 \times N_t}$ are modeled as
$$
\begin{align*}
\mathbf{H}_1 &= \sqrt{\frac{N_t N_s}{L_1}} \sum_{i = 1}^{L_1} \kappa_i \ \textbf{a}_\mathrm{RIS}(\gamma_i^r, \eta_i^r)\textbf{a}_\mathrm{BS}(\theta_i^t)^H, \\
\hat{\mathbf{h}}_{k, 2} &= \sqrt{\frac{N_s N_r}{L_2}} \sum_{i = 1}^{L_2} \kappa_i \ \textbf{a}_\mathrm{UE}(\theta_i^r)\textbf{a}_\mathrm{RIS}(\gamma_i^t, \eta_i^t)^H, \\
\hat{\mathbf{h}}_{k, 3} &= \sqrt{\frac{N_t N_r}{L_3}} \sum_{i = 1}^{L_3} \kappa_i \ \textbf{a}_\mathrm{UE}(\theta_i^r)\textbf{a}_\mathrm{BS}(\theta_i^t)^H,
\end{align*}
$$
- where $L_1, L_2$ and $L_3$ are the total number of signal paths between BS-RIS, RIS-UE and BS-UE channel, respectively
- $\kappa_{LoS}$ and $\kappa_{NLoS}$ are the complex channel gains modeled by $\mathcal{CN}(0, 1)$ and $\mathcal{CN}(0, 10^{-0.1\mu})$
- with Ricain factor $\mu = 10$
- $\textbf{a}_{BS}, \textbf{a}_{RIS}$ and $\textbf{a}_{UE}$ are the steering vectors at the BS, RIS, and UE, respectively
- the values of $AoD$ and $AoA$ are randomly generated between $0$ and $2\pi$
- $\theta_i^t$ and $\theta_i^r$ denote the angles of departure $(AoD)$ at the BS and angles of arrival $(AoA)$ at the UE
- $\gamma_i^t$ and $\gamma_i^r$ are the azimuth $AoD$ and $AoA$ at the RIS
- $\eta_i^t$ and $\eta_i^r$ are the elevation $AoD$ and $AoA$ at the RIS

#### **Channel matrices**

### **General Communication Systems**
- The $\mathbf{H}_1 \in \mathbb{C}^{N_s \times N_t}, \hat{\mathbf{h}}_{k, 2} \in \mathbb{C}^{1 \times N_s}$ and $\hat{\mathbf{h}}_{k, 3} \in \mathbb{C}^{1 \times N_t}$ are modeled as
$$
\begin{align*}
{\mathbf {H}}_{1} &= \sqrt{\frac{K_{1}}{K_{1} + 1}} \kappa _\mathrm{LoS} \ \textbf{a}_\mathrm{RIS}(\gamma^r, \eta^r)\textbf{a}_\mathrm{BS}(\theta^t)^H + \sqrt{\frac{1}{K_{1} + 1}} \kappa_\mathrm{NLoS} \; \overline{\mathbf {H}}_\mathrm{NLoS}, \\
\hat{\mathbf{h}}_{k, 2} &= \sqrt{\frac{K_{1}}{K_{1} + 1}} \kappa _\mathrm{LoS} \ \textbf{a}_\mathrm{UE}(\theta^r)\textbf{a}_\mathrm{RIS}(\gamma^t, \eta^t)^H + \sqrt{\frac{1}{K_{1} + 1}} \kappa_\mathrm{NLoS} \; \overline{\mathbf {h}}_\mathrm{NLoS}, \\
\hat{\mathbf{h}}_{k, 3} &= \sqrt{\frac{K_{1}}{K_{1} + 1}} \kappa _\mathrm{LoS} \ \textbf{a}_\mathrm{UE}(\theta^r)\textbf{a}_\mathrm{BS}(\theta^t)^H+ \sqrt{\frac{1}{K_{1} + 1}} \kappa_\mathrm{NLoS} \; \overline{\mathbf {h}}_\mathrm{NLoS},
\end{align*}
$$
- where $K_1$ is the Rician factor, and the former and latter terms are line-of-sight (LoS) and non-line-of-sight (NLoS) components
- with $K_1 = 10 dB$
- $\kappa_\mathrm{LoS}$ and $\kappa_\mathrm{LoS}$ are the large-scale fading constant for the LoS channel and NLoS channel, respectively
- with $\kappa_\mathrm{LoS} = \kappa_\mathrm{LoS} = 10^{-0.5} \sim 0.3162$
- $\textbf{a}_{BS}, \textbf{a}_{RIS}$ and $\textbf{a}_{UE}$ are the steering vectors at the BS, RIS, and UE, respectively
- the values of $AoD$ and $AoA$ are randomly generated between $0$ and $2\pi$
- $\theta^t$ and $\theta^r$ denote the angles of departure $(AoD)$ at the BS and angles of arrival $(AoA)$ at the UE
- $\gamma^t$ and $\gamma^r$ are the azimuth $AoD$ and $AoA$ at the RIS
- $\eta^t$ and $\eta^r$ are the elevation $AoD$ and $AoA$ at the RIS
- $\overline{\mathbf{H}}_\mathrm{NLoS}$ and $\overline{\mathbf {h}}_\mathrm{NLoS}$ are the small-scale fading modeled by complex Gaussian with zero mean and unit variance $\sim \mathcal{CN}(0, 1)$

### **Steering vectors**
- Considering the uniform linear array $(ULA)$ structure at the BS and UE, and the uniform planar array $(UPA)$ structure at the RIS, the steering vectors are expressed as
$$
\begin{align*}
\textbf{a}_{BS}(\theta) =& \; \textbf{a}_{UE}(\theta) = \sqrt{\frac{1}{N}} \cdot [ 1, \ e^{j2\pi\frac{d_{a}}{\lambda}\sin(\theta)}, \, \ldots, \ e^{j2\pi\frac{d_{a}}{\lambda}(N - 1)\sin(\theta)} ]^T, \\
\textbf{a}_{RIS}(\gamma, \eta) =& \; \textbf{a}_{y}(\gamma, \eta) \ \otimes \ \textbf{a}_{z}(\eta), \\
=& \; \sqrt{\frac{1}{M}} \cdot [ 1, \ \ldots, \ e^{j2\pi\frac{d_{RIS}}{\lambda}((M_y - 1)\sin(\gamma)\cos(\eta) + (M_z - 1)\sin(\eta))} ]^T, \\
\textbf{a}_y(\gamma, \eta) =& \sqrt{\frac{1}{M_y}} \cdot [ 1, \ e^{j2\pi\frac{d_{RIS}}{\lambda}\sin(\gamma)\sin(\eta)}, \ldots, \ e^{j2\pi\frac{d_{RIS}}{\lambda}(M_y - 1)\sin(\gamma)\sin(\eta)} ]^T, \\
\textbf{a}_z(\eta) =& \sqrt{\frac{1}{M_z}} \cdot [ 1, \ e^{j2\pi\frac{d_{RIS}}{\lambda}\cos(\eta)}, \ldots, \ e^{j2\pi\frac{d_{RIS}}{\lambda}(M_z - 1)\cos(\eta)} ]^T,
\end{align*}
$$
- $d_a$ and $d_{RIS}$ represent the antenna and RIS element spacing, which are set to half wavelength $\frac{\lambda}{2}$
- J. Yuan, Y. -C. Liang, J. Joung, G. Feng and E. G. Larsson, "[Intelligent Reflecting Surface-Assisted Cognitive Radio System](https://ieeexplore.ieee.org/document/9235486)," in *IEEE Transactions on Communications*, vol. 69, no. 1, pp. 675-687, Jan. 2021. (Cited by 130)
- $N$ represents $N_t$ and $N_r$ for the case of BS and UE, respectively
- $M = M_y \times M_z$, where $M_y$ and $M_z$ represent the numbers of horizontal and vertical elements of the RIS

#### **Array response implementations**

## **Problem formulations**
### **Min-max MSE**
- The objective is to minimize the worst case Mean Squared Error (MSE)
$$
\begin{align*}
\min\limits_{\mathbf{\Phi}} \;\; &\max\limits_{\Delta\mathbf{h}_{k, 2}, \; \Delta\mathbf{h}_{k, 3}, \; \varphi_i', \; \beta(\varphi_i)} \;\;\;\;\; \alpha \\
\textrm {s.t.} \;\;
& \alpha \geq 0, \\
& E\left\{ (x_k - y_k)(x_k - y_k)^H \right\} \leq \alpha, \forall k = 1, \ldots, N_k, \\
& tr\{ \mathbf{F}\mathbf{F}^{H} \} \leq P_{t}, \\
& \| \phi_i \|_2^2 = \beta(\varphi_i) = (1 - \beta_{min}) \cdot (\frac{\sin(\varphi_i - \mu) + 1}{2})^\kappa + \beta_{min}, \; \beta_{min} \geq 0, \; \mu\geq 0, \;\; \forall i = 1, \ldots, N_s, \\
& \varphi_i = \hat{\varphi_i} + \varphi_i' \ , \; \varphi_i \in [0, 2\pi), \;\; \forall i = 1, \ldots, N_s, \\
& \hat{\varphi_i} \in \mathcal{A} = \{ e^{(j\frac{ \; 2\pi n \;}{2^{bits \;}})} \}_{n = 0}^{ 2^{bits - 1}}, \; f(\varphi_i'(\mu, \kappa)) = \frac{\;e^{\kappa \cos(\varphi_i' - \mu)}\;\;\;}{\;2\pi I_0(\kappa)\;}, \;\forall i = 1, \ldots, N_s, \\
& \| \Delta\mathbf{h}_{k, 2} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k, \\
& \| \Delta\mathbf{h}_{k, 3} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k.
\end{align*}
$$
- the variable $\alpha$ is the **upper bound** of our objective function
- the diverse mathematical characteristics give rise to the intricate mixed discrete and continuous programming in the optimization problem, rendering it a Non-deterministic Polynomial-time hard (**NP-hard**) problem
- The resulting MSE of our Min-Max MSE problem is derived as follows:
$$
\begin{align*}
\text{MSE} &= E\left\{ (\mathbf{x} - \mathbf{y})(\mathbf{x} - \mathbf{y})^H \right\} \\
&= \mathbf{I}_{N_k} - (\tilde{\mathbf{H}}\mathbf{F})^H - \tilde{\mathbf{H}}\mathbf{F} + (\tilde{\mathbf{H}}\mathbf{F})(\tilde{\mathbf{H}}\mathbf{F})^H + \sigma_n^2\mathbf{I}_{N_k}
\end{align*}
$$
- where $\tilde{\mathbf{H}} = \mathbf{H}_{2} \mathbf{\Phi} \mathbf{H}_1 + \mathbf{H}_{3}$

#### **MSE reward**

### **Max-min downlink rate**
- The objective is to maximize the worst case downlink achievable data rate
$$
\begin{align*}
\max\limits_{\mathbf{\Phi}} \;\; &\min\limits_{\Delta\mathbf{h}_{k, 2}, \; \Delta\mathbf{h}_{k, 3}, \; \varphi_i', \; \beta(\varphi_i)} \;\;\;\;\; R \\
\textrm {s.t.} \;\;
& R \geq 0, \\
& R_{k} \geq R_{min}, \forall k = 1, \ldots, N_k, \\
& tr\{ \mathbf{F}\mathbf{F}^{H} \} \leq P_{t}, \\
& \| \phi_i \|_2^2 = \beta(\varphi_i) = (1 - \beta_{min}) \cdot (\frac{\sin(\varphi_i - \mu) + 1}{2})^\kappa + \beta_{min}, \; \beta_{min} \geq 0, \; \mu\geq 0, \;\; \forall i = 1, \ldots, N_s, \\
& \varphi_i = \hat{\varphi_i} + \varphi_i' \ , \; \varphi_i \in [0, 2\pi), \;\; \forall i = 1, \ldots, N_s, \\
& \hat{\varphi_i} \in \mathcal{A} = \{ e^{(j\frac{ \; 2\pi n \;}{2^{bits \;}})} \}_{n = 0}^{ 2^{bits - 1}}, \; f(\varphi_i'(\mu, \kappa)) = \frac{\;e^{\kappa \cos(\varphi_i' - \mu)}\;\;\;}{\;2\pi I_0(\kappa)\;}, \;\forall i = 1, \ldots, N_s, \\
& \| \Delta\mathbf{h}_{k, 2} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k, \\
& \| \Delta\mathbf{h}_{k, 3} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k.
\end{align*}
$$
- the variable $R_{min}$ is the **lower bound** of our objective function
- $R_k$ represents the downlink achievable rate of the $k$-th user
- The above received signal model can be further written as
$$
\begin{align*}
y_{k} = (\mathbf{h}_{k,2} \mathbf{\Phi} \mathbf{H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{k}x_{k} + \sum\limits_{j = 1, \ j \neq k}^{N_k} (\mathbf{h}_{k,2} \mathbf{\Phi} \mathbf{H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{j}x_{j} + n_{k},
\end{align*}
$$
- without joint detection of data streams for all users, the second term is treated as cochannel interference
- $\mathbf{h}_{k, 2} \in \mathbb{C}^{1 \times N_s}$ and $\mathbf{h}_{k, 3} \in \mathbb{C}^{1 \times N_t}$ are the RIS-UE channel and BS-UE channel, respectively
- $\mathbf{H}_{1} \in \mathbb{C}^{N_s \times N_t}$ is the BS-RIS channel and $\mathbf{\Phi} \in \mathbb{C}^{N_s \times N_s}$ is the RIS matrix
- $\mathbf{f}_k \in \mathbb{C}^{N_t \times 1}$ is the $k$-th column vector of the beamforming matrix $\mathbf{F} \in \mathbb{C}^{N_t \times N_k}$
- The SINR at the $k$-th user is then given by
$$
\begin{align*}
\rho_{k} = \frac{| (\mathbf{h}_{k,2} \mathbf{\Phi H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{k} |^{2} }{ \sum\limits_{j = 1, \ j \neq k}^{N_k} | (\mathbf {h}_{k,2} \mathbf{\Phi H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{j} |^{2} + \sigma_{k}^{2} }.
\end{align*}
$$
- The resulted downlink rate for the $k$-users is written as
$$
\begin{align*}
R_k = \log_2(1 + \rho_{k}).
\end{align*}
$$

#### **downlink rate reward**

### **Sum-Rate Maximization**
- The goal is to maximize the sum downlink rate (spectral efficiency) in the system, which is given by
$$
\begin{align*}
\sum\limits_{k = 1}^{N_k} R_{k} = \sum\limits_{k = 1}^{N_k} \log_2(1 + \rho_{k}) = \sum\limits_{k = 1}^{N_k} \log_2(1 + \frac{| (\mathbf{h}_{k,2} \mathbf{\Phi H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{k} |^{2} }{ \sum\limits_{j = 1, \ j \neq k}^{N_k} | (\mathbf {h}_{k,2} \mathbf{\Phi H}_{1} + \mathbf{h}_{k, 3}) \mathbf{f}_{j} |^{2} + \sigma_{k}^{2} }).
\end{align*}
$$
- The optimization problem is expressed as:
$$
\begin{align*}
\max\limits_{\mathbf{\Phi}} \;\; & \; \sum\limits_{k = 1}^{N_k} R_{k} \\
\textrm {s.t.} \;\;
& tr\{ \mathbf{F}\mathbf{F}^{H} \} \leq P_{t}, \\
& \| \phi_i \|_2^2 = \beta(\varphi_i) = (1 - \beta_{min}) \cdot (\frac{\sin(\varphi_i - \mu) + 1}{2})^\kappa + \beta_{min}, \; \beta_{min} \geq 0, \; \mu\geq 0, \;\; \forall i = 1, \ldots, N_s, \\
& \varphi_i = \hat{\varphi_i} + \varphi_i' \ , \; \varphi_i \in [0, 2\pi), \;\; \forall i = 1, \ldots, N_s, \\
& \hat{\varphi_i} \in \mathcal{A} = \{ e^{(j\frac{ \; 2\pi n \;}{2^{bits \;}})} \}_{n = 0}^{ 2^{bits - 1}}, \; f(\varphi_i'(\mu, \kappa)) = \frac{\;e^{\kappa \cos(\varphi_i' - \mu)}\;\;\;}{\;2\pi I_0(\kappa)\;}, \;\forall i = 1, \ldots, N_s, \\
& \| \Delta\mathbf{h}_{k, 2} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k, \\
& \| \Delta\mathbf{h}_{k, 3} \|_2 \leq \psi, \; \forall k = 1, \ldots, N_k.
\end{align*}
$$

#### **Sum-Rate reward**

## **Code repository**
Github Private Repo [RIS-MISO-DRL](https://github.com/paulchen2713/RIS-MISO-DRL)

## **Project structure**

C:\Users\Paul\Downloads\RIS-MISO-DRL>tree \f
C:.
├─dependency
├─figures
│  └─DQN-2023-11-15
├─logs
│  ├─DQN-2023-11-20
│  ├─DQN-2023-11-21
│  └─DQN-2023-11-22
├─models
│  ├─DQN-2023-11-21-Ns-36
│  └─DQN-2023-11-22-Ns-36
├─notebooks
│      beamformers.ipynb
│      discrete_angle_set.ipynb
│      observation_shape.ipynb
│      steering_vectors.ipynb
│      von_mises.ipynb
├─PPOs
│      PPO.py
│      test.py
│      VMPO.py
├─QLearning
│  └─figures
├─references
│      custom_gym_env.py
│      environment_creation.py
│      env_ddpg.py
│      env_sac.py
│      visibility_cond.py
├─src
│      discrete_env.py
│      discrete_learn.py
│      discrete_load.py
│      discrete_torch_env.py
│      learn_and_save.py
│      load_and_predict.py
│      numpy_env.py
│      temp_env.py
│      torch_env.py
├─system_simulation
│      array_response.py
│      channel_uncert.py
│      discrete_angle_set.py
│      sparse_SV_channel_RIS_slices.py
│      steering_vectors.py
│      von_mises.py
└─__pycache__