Skip to main content

Bayesian Linear Regression

BLR is used when Gaussian discriminative model p(y∣X)p(y|X) used for regression with a Bayesian analysis for the weights.

Recall if a random variable x∼N(μ,Σ)x \sim \mathcal{N}(\mu, \Sigma), then the log probability density function is given by log⁡p(x)=−12(x−μ)TΣ−1(x−μ)+const=−12xTΣ−1x+xTΣ−1μ+const\log p(x) = -\frac{1}{2} (x - \mu)^T \Sigma^{-1} (x - \mu)+\text{const} = -\frac{1}{2} x^T \Sigma^{-1} x + x^T \Sigma^{-1} \mu + \text{const}. That is, if we know some random variable ww follow a Gaussian distribution with log⁡p(w)=−12wTAw+wTb+const\log p(w) = -\frac{1}{2} w^T A w + w^T b + \text{const}, then w∼N(A−1b,A−1)w \sim \mathcal{N}(A^{-1}b, A^{-1}).

Consider y∣x∼N(w⊤ψ(x),σ2)y|x \sim \mathcal{N}(w^{\top}\psi(x), \sigma^2), then

the log likelihood for linear regression is the log⁡p(y∣x)=∑i=1Nlog⁡p(yi∣xi)=∑i=1Nlog⁡N(yi∣w⊤ψ(xi),σ2)=−N2log⁡2π−N2log⁡σ2−12σ2∑i=1N(yi−w⊤ψ(xi))2=const−12σ2∥y−Ψw∥2\log p(y|x) = \sum_{i=1}^N \log p(y_i|x_i) = \sum_{i=1}^N \log \mathcal{N}(y_i|w^{\top}\psi(x_i), \sigma^2) = -\frac{N}{2} \log 2\pi - \frac{N}{2} \log \sigma^2 - \frac{1}{2\sigma^2} \sum_{i=1}^N (y_i - w^{\top}\psi(x_i))^2 = \text{const} - \frac{1}{2\sigma^2} \|y - \Psi w\|^2.

the MAP estimation for regularized linear regression is arg⁡max⁡wlog⁡p(w∣D)=arg⁡max⁡w[log⁡p(w)+log⁡p(D∣w)]\arg \max_{w} \log p(w|D) = \arg \max_{w} [\log p(w) + \log p(D|w)] where:

  • log⁡p(D∣w)=const−12σ2∥y−Ψw∥2\log p(D|w) = \text{const} - \frac{1}{2\sigma^2} \|y - \Psi w\|^2
  • assume prior w∼N(m,S)w\sim \mathcal{N}(m, S), then log⁡p(w)=−12(w−m)⊤S−1(w−m)+const\log p(w) = -\frac{1}{2} (w - m)^{\top} S^{-1} (w - m) + \text{const}
    • Commonly, m=0m = 0 and S=ηIS = \eta I, then log⁡p(w)=−12η∥w∥2+const\log p(w) = -\frac{1}{2\eta} \|w\|^2 + \text{const}

That is, the posterior predictive distribution is: p(y∣x,D)=∫p(w∣D)p(y∣x,w)dwp(y|x, D) = \int p(w|D) p(y|x, w) dw

BLR assumptions and steps​

The BLR assumes:

  • Prior w∼N(0,S)w \sim \mathcal{N}(0, S)
  • Likelihood y∣x,w∼N(w⊤ψ(x),σ2)y|x, w \sim \mathcal{N}(w^{\top}\psi(x), \sigma^2)
  • S,σ2S,\sigma^2 are known/fixed

Continue on the posterior of regularized linear regression, we have: log⁡p(w∣D)=log⁡p(w)+log⁡p(D∣w)=−12w⊤S−1w−12σ2∥y−Ψw∥2+const=−12w⊤(σ−2Ψ⊤Ψ+S−1)w+σ−2y⊤Ψ⊤w+const\log p(w|D) = \log p(w) + \log p(D|w) = -\frac{1}{2} w^{\top} S^{-1} w - \frac{1}{2\sigma^2} \|y - \Psi w\|^2 + \text{const} = -\frac{1}{2}w^{\top}(\sigma^{-2}\Psi^{\top}\Psi + S^{-1})w + \sigma^{-2} y^{\top} \Psi^{\top} w + \text{const}

Gaussian prior leads to a Gaussian posterior, and so the Gaussian distribution is the conjugate prior for linear regression model. That is w∣D∼N((σ2S−1+Ψ⊤Ψ)−1Ψ⊤y,σ2(Ψ⊤Ψ+σ2S)−1)w|D \sim \mathcal{N}((\sigma^{2}S^{-1} + \Psi^{\top} \Psi)^{-1} \Psi^{\top}y, \sigma^{2}(\Psi^{\top} \Psi + \sigma^{2} S)^{-1}).

Recall the colsed-form direct solution w=(Ψ⊤Ψ+λI)−1Ψ⊤yw = (\Psi^{\top} \Psi + \lambda I)^{-1} \Psi^{\top} y, that is, S=σ2λIS = \frac{\sigma^2}{\lambda} I. From this, we can observe when λ→0\lambda \to 0 (i.e. no regularization), the posterior is converges to the mle solution for linear regression.

Since p(y∣x,D)=∫p(w∣D)p(y∣x,w)dwp(y|x,D) = \int p(w|D) p(y|x,w) dw, then y∼N(((σ2S−1+Ψ⊤Ψ)−1Ψ⊤y)⊤ψ(x),ψ(x)⊤(σ2(Ψ⊤Ψ+σ2S)−1)ψ(x)+σ2)y \sim \mathcal{N} (((\sigma^{2}S^{-1} + \Psi^{\top} \Psi)^{-1} \Psi^{\top}y)^{\top} \psi(x), \psi(x)^{\top}(\sigma^{2}(\Psi^{\top} \Psi + \sigma^{2} S)^{-1}) \psi(x) + \sigma^2).

  • Or we can use y=w⊤ψ(x)+ϵy = w^{\top}\psi(x) + \epsilon then plugin w∣Dw|D and ϵ\epsilon