estimator

From 18.650 lectures:

given \(X_1,...,X_n\) independent samples from \(P_{\theta}\) for an unknown \(\theta\) (see statistical model), an estimator of \(\theta\) is any function of \(X_1,\dots,X_n\) that does not depend on \(\theta\)

1. consistent

a sequence of estimators \(\hat{\theta}_n\) is strongly consistent if \(\theta_n \rightarrow \theta\) almost surely as \(n\rightarrow \infty\)
- it is weakly consistent if the sequence converges in probability to \(\theta\)

for a given \(n\), the bias of an estimator is \(\mathbb{E}[\hat{\theta}_n] - \theta\)
- is the average estimator at \(\theta\)? NOTE: it could be that none of the estimators are at \(\theta\), but the expectation is still at \(\theta\).
the variance of \(\hat{\theta}\) is: \(\mathbb{E}[\hat{\theta}^2] - \mathbb{E}[\hat{\theta}]^2\)
- what is the spread of the estimators?
the risk of an estimator is \(\mathbb{E}[|\hat{\theta}-\theta|^2]\)
- how far is the estimator from \(\theta\) on average?
it turns out: \(\text{risk} = \text{bias}^2 + \text{variance}\). To see this, expand RHS terms and use linearity of expectation

See example in confidence interval note.