Estimation

## Terminology - $\mu$: population mean (usually unknown) - $\bar x$ : sample mean - $\hat \mu$: estimator of mean - $\hat \sigma^2$: estimator of variance - $\theta_*$: true parameter (unknown) - Parameter bias - Parameter variance ## MAP estimation We think of $\theta$ as a random variable. $$ \begin{align*} \theta_{MAP} &= \arg\max_\theta p(\theta | d) \\ &= \arg\max_\theta p(d|\theta)p(\theta) \\ &= \arg\max_\theta \log p(d|\theta) + \log p(\theta) \end{align*} $$ Let's assume that our likelihood function is $$ p(x|\mu) = G_x[\mu,1], \quad p(\mu) = G_\mu[0,\alpha^2] $$ ## ML estimation We think of $\theta$ as a parameter, instead of a random variable. The log-likelihood of $p(x|\theta)=G(\mu,\sum)$ is given by $$ L = \frac 12 \sum_{i=1}^m \{ \log (\det [{\sum}^{-1}] ) - \frac 12(x^{(i)}-\mu)^T {\sum}^{-1} (x^{(i)}-\mu) \} $$ Derivative w.r.t. $\mu$: $$ \mu = \frac 1m \sum_{i=1}^m x^{(i)} $$ Derivative w.r.t. $\sum$: $$ \sum = \frac 1m \sum_{i=1}^m (x^{(i)}-\mu)(x^{(i)}-\mu)^T $$ These are called the sample mean and sample covaraince. ## Expected square error of estimation $$ \begin{align*} E[(\hat\theta - \theta_*)^2] &= E[\hat\theta^2] + \theta_*^2 - 2\theta_* E[\hat\theta] \\ &= E[\hat\theta^2] + (\theta_*-\bar\theta)^2 - \bar\theta^2 \\ &= (\theta_*-\bar\theta)^2 + E[(\bar\theta-\hat\theta)^2] \end{align*} $$ The first term is referred to as squared bias and the second term as the asymptotic variance. ## ML estimation ### If population mean $\mu$ is known, $$ \frac{\partial \log p(x;s)}{\partial s} = -\frac N2 + \frac{1}{2s^2}\sum_i (x^{(i)} - \mu)^2=0 \\ \hat s = \frac 1N \sum_{i=1}^N (x^{(i)} - \mu)^2 $$ Performance evaluation: We can show that $$ E[\hat s] = s $$ ### If sample mean $\hat \mu$ is considered, $$ \frac{\partial \log p(x;s)}{\partial s} = -\frac N2 + \frac{1}{2s^2}\sum_i (x^{(i)} - \frac 1N \sum_j x^{(j)})^2=0 \\ s = \frac{\sum x_i^2}{N} - \left( \frac{ \sum_i x_i }{N} \right)^2 $$ We can show that $$ E[\hat s] = s \left( \frac{N-1}{N} \right) $$ To make it unbiased, we adjust $$ s' = \frac{N}{N-1} \hat s = \frac{1}{N-1}\sum_i (x^{(i)}-\hat\mu)^2 $$ References - https://www.khanacademy.org/math/ap-statistics/quantitative-data-ap/measuring-spread-quantitative/v/review-and-intuition-why-we-divide-by-n-1-for-the-unbiased-sample-variance - (detail derivation) http://www.visiondummy.com/2014/03/divide-variance-n-1/