central limit theorem
1. Statement
Let \(X_1,...,X_n\) be a collection of \(n\) random variables, where \(X_1,...,X_n\) all have mean \(\mu\) and variance \(\sigma^2\). Let \(\bar{X}_n = \frac{1}{n}\sum_{i=1}^{n}X_i\). Then, \[ \frac{\sqrt{n}}{\sigma} (\bar{X}_n - \mu) \underset{n\rightarrow\infty}{\overset{d}{\longrightarrow}} \mathcal{N}(0,1) \]
That is, \(\frac{\sqrt{n}}{\sigma}(\bar{X}_n - \mu)\) converges in distribution to \(\mathcal{N}(0,1)\) as \(n\) grows. This means that samples from \(\frac{\sqrt{n}}{\sigma}(\bar{X}_n - \mu)\) will form an approximate histogram of \(\mathcal{N}(0,1)\) and this approximation will get better as \(n\) increases.
2. some notes on standardization
Where does the \(\sqrt{n}\) come from? Well, we are trying to get to something that has unit variance.
- An \(X_i\) has variance \(\sigma^2\).
- \(\sum X_i\) has variance \(n\sigma^2\).
- \(\frac{1}{n}\sum X_i\) has variance \(\frac{1}{n}\sigma^2\)
- \(\sqrt{n}\frac{1}{n}\sum X_i\) has variance \(\sigma^2\)
- \(\sqrt{n}\frac{1}{n\sigma}\sum X_i\) has variance \(1\)
3. delta method
Roughly: If \(Y_n - \theta\) converges in distribution to a normal random variable then \(g(Y_n) - g(\theta)\) also converges in distribution to a normal random variable provided \(g'(\theta)\) exists and \(g'(\theta) \neq 0\)
3.1. formally
Suppose \(\sqrt{n}(Y_n - \theta) \overset{d}{\rightarrow} \mathcal{N}(0,\sigma^2)\) and \(g\) is continuously differentiable at \(\theta\). Then, \[ \sqrt{n}(g(Y_n) - g(\theta)) \overset{d}{\rightarrow} \mathcal{N}(0,g'(\theta)^2\sigma^2) \]
3.2. usage
want to use central limit theorem for function of random variable. For instance, the central limit may give us the distribution for error for \(\theta\), but the parameter we are actually interested in is \(g(\theta)\).