Computational Statistics

2.Nonparametric Density Estimation

Let X1,…,Xn∼iidFX_1, \ldots, X_n \simiid F from a differentiable cdf FF with pdf f=F′f = F'. The task is to estimate ff by some f^\hat{f}. As always, there are many garbage estimators. The question is what is a good one?

We could define the mean-squared error as MSE(x)=E ⁣[(f(x)−f^(x))2] ⁣ \mathrm{MSE}(x) = \E{\pa{f(x) - \hat{f}(x)}^2} which is a bit weird because we do not care about a single x∈Rx \in \R. A better criteria is MISE=∫MSE(x) dx=∫(E ⁣[f^(x)] ⁣−f(x))2+Var⁡ ⁣[f^(x)] ⁣ dx \mathrm{MISE} = \int \mathrm{MSE}(x) \dd x = \int \pa{\E{\hat{f}(x)} - f(x)}^2 + \Var{\hat{f}(x)} \dd x Note that this is still just a surrogate criterion.

Example (Histogram): Choose x0∈Rx_0 \in \R and h>0h > 0. Then ∀x∈R\forall x \in \R we have f^x0,h(x)=∑j∈Zg^j1{x∈Ij} \hat{f}_{x_0,h}(x) = \sum_{j \in \Z} \hat{g}_j \ind{x \in I_j} where ∀j∈Z\forall j \in \Z we have Ij=(x0+jh,x0+(j+1)h)I_j = (x_0 + jh, x_0 + (j+1)h) and gj^=#{i∈{1,…,n}:Xi∈Ij}nh\hat{g_j} = \frac{\# \set{i \in \set{1, \ldots, n}: X_i \in I_j}}{nh}.

f^x0,h\hat{f}_{x_0, h} is not even continuous. Is this a problem?

Example (Kernel density estimator): Fix a kernel k:R→R≥0k: \R \to \R_{\geq 0} such that ∫−∞∞k(x) dx=1\int_{-\infty}^{\infty} k(x) \dd x = 1, kk is bounded and ∀x∈R:k(x)=k(−x)\forall x \in R: k(x) = k(-x). Let h>0h>0 and define f^h(x)=1nh∑i=1nk ⁣(x−Xih) ⁣ \hat{f}_h(x) = \frac{1}{nh} \sum_{i=1}^n k\of{\frac{x-X_i}{h}} where 1nh\frac{1}{nh} ensures that f^h(x)\hat{f}_h(x) is integrable to 11. Common choiche are gaussian kernels k(x)=12πe−x22 k(x) = \frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}} Note that many properties over the kernel carry over to the estimator.