The reduced mass of a phonon mode is implicitly contained in the normalization and metric of the phonon displacements.

The scalar product and with that the norm in the displacents' space is defined as:

$$\large \left(U,V \right) \equiv \sum_{k}{m_{k} \mathbf{U}_{k}\cdot\mathbf{V}_{k}}$$

where $k$ is the atomic index and $m_k$ are the mass and the displacement of the atom $k$.

Each normal mode $\nu$ is thus normalized according to the condition

$$\large \left(u^\nu,u^\nu\right)^\frac{1}{2} = \sqrt{\sum_k{m_k (\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k})} = 1 $$

With this metric one can then define the coordinate $Q_\nu$ of for a generic displacement pattern $\left\{ U_k \right\}$ along normal node $\{u^\nu_k\}$ as:

$$\large Q_\nu = \frac{\sum_k{m_k}(\mathbf{U}_k\cdot \mathbf{u}^\nu_k)}
{\sqrt{\sum_k{ m_k(\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k) }}}$$

This has units $[m]^\frac{1}{2}[l]$ but $Q_\nu$ is also the value of the adimensional number representing the component of the ${U_k}$ pattern along the ${u^\nu_k}$ mode. Indicating the dimensional value as $\bar{Q}_\nu$ and the numerical value as $\tilde{Q}_\nu$ and considering that the modes are orthonormal we have

$$\large \bar{Q}_\nu= \tilde{Q}_\nu \times \sqrt{\sum_k{m_k (\mathbf{u}^\nu_k \cdot \mathbf{u}^\nu_k})}$$

A way to define the reduced mass $\mu_\nu$ could then be :
$$\large \mu_\nu^\frac{1}{2} = \frac{\sqrt{\sum_k{ m_k(\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k) }}}{\sqrt{\sum_k{\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k}}} $$

or

$$\large \mu_\nu = \frac{\sum_k{ m_k(\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k) }}{\sum_k{\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k}} $$

and if we redefine the normal mode coordinate as

$$ \large \bar{Q}^\prime_\nu = L_\nu \tilde{Q}_\nu $$.
$L_\nu = \sqrt{\sum_k{\mathbf{u}^\nu_k\cdot\mathbf{u}^\nu_k}}$ is a length and $\tilde{Q}_\nu$ the adimensional numerical value representing the projection of our pattern on mode $\nu$; it is easy to see that
$$ \large \bar{Q}_\nu = \sqrt{\mu_\nu} \; \bar{Q}^\prime_\nu$$
In this way for example the potential energy term of the normal mode oscillator would become

$$\large \frac{1}{2} \omega^2_\nu \bar{Q}^2_\nu \rightarrow \frac{1}{2}\mu_\nu\omega^2_\nu{\bar{Q}^\prime_\nu}^2$$