---
# System prepended metadata

title: Untitled

---


**As I mentioned in my comment, your $p(\sigma_{1:n}|G_0)$ needs to be derived from your model -- you cannot say that you ''use'' the distribution of node orders from the diffusion process.

Similarly, you cannot say that you use your true posterior in a variational inference problem (e.g. VAE) as your variational distribution -- you don't have the freedom to do so.**



We want to clarify that (1) $p(\sigma_{1:n}|G_n)$ can be modeled independently --- instead of having to be derived by marginalizing $p(G_{0:n-1},\sigma_{1:n}|G_n)$ --- because of how $p(G_{0:n-1},\sigma_{1:n}|G_n)$ is factorized in our model; (2) $q(\sigma_{1:n}|G_0)$ and $p(\sigma_{1:n}|G_n)$ model two different things in autoregressive diffusion models; their relation is not as using a variational distribution to approximate the true posterior in VAE. One thus does have the freedom to set $p(\sigma_{1:n}|G_n)$, and the VLB is maximized when and only when $p(\sigma_{1:n}|G_n) = q(\sigma_{1:n}|G_0)$.


First, our factorization leads to $p(G_{0:n-1},\sigma_{1:n}|G_n)=p(G_{0:n-1}|\sigma_{1:n}, G_n)p(\sigma_{1:n}|G_n)$.
Given $G_n$, the ordering $\sigma_{1:n}$ is an independent variable, while $G_{0:n-1}$ is dependent on $\sigma_{1:n}$. Therefore, $p(\sigma_{1:n}|G_n)$ can be modeled independently --- this is akin to the freedom of parameterizing independent variables in classic graphical models.


Second,  in autoregressive diffusion [1], $q(\sigma_{1:n}|G_0)$ is not a variational distribution of $p(\sigma_{1:n}|G_n)$. They are modeling different things: the diffusion ordering $q(\sigma_{1:n}|G_0)$ is conditioned on the original graph $G_0$; the generation ordering $p(\sigma_{1:n}|G_n)$ is conditioned on the masked graph $G_n$. The forward diffusion process masks the graph governed by distribution $q(\sigma_{1:n}|G_0)$, while the backward generation process denoises the masked graph governed by another distribution $p(\sigma_{1:n}|G_n)$. During training, the VLB is maximized when and only when the KL-divergence term is 0, i.e., $p(\sigma_{1:n}|G_n)=q(\sigma_{1:n}|G_0)$. Therefore, when training the generation network $p(G_t|G_{t+1},\sigma_{1:n})$, we can directly set $p(\sigma_{1:n}|G_n)=q(\sigma_{1:n}|G_0)$ and ignore the KL-divergence term. A similar derivation can be found in the original autoregressive diffusion model [1] (page 21).


### References

[1] Autoregressive Diffusion Model, Emiel Hoogeboom et, al., ICLR 2022