In the previous post we saw that exact Bayesian inference is usually intractable, which leaves approximate inference as a practical route. One of the approaches to approximate inference that I like is the Free Energy functional. The idea of the Free Energy functional is to replace an intractable intregral with an optimisation over a chosen family of approximating distributions using the Kullbach-Leibler (KL) divergence.
To derive the free energy we start with Bayes’ rule and rearrange to focus on the intractable marginal \(p(y)\)
\[p(y) = \frac{p(y|x)p(x)}{p(x|y)}.\]
Let \(q_{\theta}\) be an arbitrary probability distribution parameterised by \(\theta\in\Theta\), with density \(q_{\theta}(x)\). We have the freedom to insert this density into the top and bottom of the fraction on the RHS
\[p(y) = \frac{p(y|x)p(x)q_{\theta}(x)}{p(x|y)q_{\theta}(x)}.\]
Next we take logs of both sides and negate, converting the LHS into surprise
\[-\log{p(y)} = -\log{\frac{p(y|x)p(x)q_{\theta}(x)}{p(x|y)q_{\theta}(x)}}.\]
Then taking the expectation of both sides over \(q_{\theta}\) we have
\[-\mathbb{E}_{q_{\theta}}[\log{p(y)}] = -\mathbb{E}_{q_{\theta}}\left[\log{\frac{p(y|x)p(x)q_{\theta}(x)}{p(x|y)q_{\theta}(x)}}\right].\]
The LHS is not a function of \(x\) so the integral simplifies
\[-\mathbb{E}_{q_{\theta}}[\log{p(y)}] = -\int q_{\theta}(x)\log{p(y)}dx = -\log{p(y)} \int q_{\theta}(x)dx = -\log{p(y)}.\]
On the RHS we can choose how to split the terms inside the logarithm
\[-\mathbb{E}_{q_{\theta}}\left[\log{\frac{p(y|x)p(x)q_{\theta}(x)}{p(x|y)q_{\theta}(x)}}\right] = -\mathbb{E}_{q_{\theta}}\left[\log{\frac{p(y|x)p(x)}{q_{\theta}(x)}}\right]-\mathbb{E}_{q_{\theta}}\left[\log{\frac{q_{\theta}(x)}{p(x|y)}}\right].\]
Recognising that in the second term we are subtracting the Kullback–Leibler divergence \(D_{KL}(q_{\theta}(x) \;||\; p(x|y)) = \mathbb{E}_{q_{\theta}}\left[\log\frac{q_{\theta}(x)}{p(x|y)}\right]\), which is always nonnegative, we have the inequality1
\[-\log{p(y)} \leq -\mathbb{E}_{q_{\theta}}\left[\log{\frac{p(y|x)p(x)}{q_{\theta}(x)}}\right]\]
with equality attained when \(q_{\theta}(x)=p(x | y)\). The right hand side is the free energy \(F[q_{\theta}]\); its negative is what the variational inference literature calls the evidence lower bound, or ELBO, so minimising the free energy and maximising the ELBO are the same operation. We can split terms out on the right hand side into an energy term minus an entropy term
\[-\log{p(y)} \leq -\mathbb{E}_{q_{\theta}}\left[\log{p(y|x)p(x)}\right] - H(q_{\theta}) \]
where \(H(q_{\theta}) = -\mathbb{E}_{q_{\theta}}\left[\log{q_\theta(x)}\right]\). In other words, the surprise of an observation is bounded above by the expected energy of the joint model under \(q_\theta\), minus the entropy of \(q_\theta\). The two terms pull in opposite directions. The energy term is happiest when all of \(q_\theta\)’s mass sits on the single state that best explains the observation; the entropy term resists exactly that, because a concentrated \(q_\theta\) has low entropy, and subtracting something small leaves the bound high. The balance is a \(q_\theta\) that spreads mass across states in proportion to how well each explains \(y\) under the model.
The right hand side involves only \(p(y|x)\), \(p(x)\) and \(q_\theta(x)\) — the densities of the likelihood and prior we specified, and of the approximating distribution we chose ourselves. The intractable \(p(x|y)\), and with it the integral we could not do, has dropped out entirely. And since the gap between the two sides is exactly \(D_{KL}(q_{\theta}(x) \,\|\, p(x|y))\), pushing the bound down simultaneously sharpens our estimate of the surprise and drags \(q_{\theta}\) toward the true posterior.
Alternatively we can rewrite the equation above in the form that will be familiar to people with an interest in Variational Autoencoders:
\[-\log{p(y)} \leq -\mathbb{E}_{q_{\theta}}\left[\log{p(y|x)}\right] + D_{KL}(q_{\theta}(x) \; || \; p(x)), \] which decomposes the free energy in terms of an accuracy term and a complexity term (divergence from the prior).
The natural next questions are how can we choose \(q_{\theta}(x)\) to be expressive enough to represent different states of the world yet make the optimisation problem of minimising the resulting functional computationally tractable? This question will be a subject for a subsequent post.
Footnotes
We also need \(q_{\theta}\) to be absolutely continuous with respect to the posterior, \(p(x|y)\): if \(q_{\theta}\) places mass where the posterior has none then the divergence is infinite and the bound, while still true, tells us nothing.↩︎