Title: Principles of Statistics

Q: Correctly specified
A: If $X\sim P$, we say that the model $\{ P_\theta: \theta \in\Theta\}$ is correctly specified if $\exists\theta\in\Theta$ such that $P=P_\theta$.

Q: Normalised log-likelihood function
A: We define
 \[
\bar\ell_n := \frac{1}{n}\ell_n = \frac{1}{n}\sum_{i=1}^n \log f(x_i,\theta).
\]

Q: Score Function
A: For $\Theta\subseteq\R^p$, we define the \emph{score function} as
\[
U(\theta, Y) = (U_r(\theta, Y)) = S_n(\theta) = \nabla_\theta \ell_n(\theta) = \left(\frac{\partial\ell_n}{\partial \theta_i}\right)_{i=1}^p.
\]

Q: Expected log-likelihood function
A: \begin{thm}
For a model $\{f(\ph, \theta): \theta\in\Theta\}$ and a variable $X\sim P$ such that
$E| \log(f(X, \theta)| < \infty$, if $f$ is well specified by $\theta_0\in\Theta$, then the
function
\[
\ell(\theta) = \E_{\theta_0} \log(f(X,\theta)) = \E_{\theta_0} \ell_n(\theta)
\]
is maximized at $\theta_0$. If $f$ has strict identifiability in $\theta$, then the max is unique.
\end{thm}
\begin{proof}
For any $\theta\in\Theta$, we can write the Kullback-Leibler divergence, or entropy, between the distributions $P_\theta$ and $P_{\theta_0}$ as
\[
KL(P_{\theta_0}, P_\theta) = \ell(\theta_0) - \ell(\theta) \geq 0.
\]
\end{proof}

Q: Expected score
A: For a sm $\{ f(\ph, \theta): \theta\in\Theta \}$ that is regular enough to exchange integration and differentiation, we have that $\forall\theta\in\Int(\Theta)$ (in particular $\theta=\theta_0$):
\[
\E_\theta \left[ \nabla \log f(X,\theta) \right] = 0.
\]
\begin{proof}
\begin{align*}
\E_\theta\left[\nabla\log f(X,\theta) \right] &= \int_\chi \left(\nabla\log f(x,\theta)\right) f(x,\theta) \; \d x \\
&= \int_\chi \left(\frac{\nabla f(x,\theta)}{ f(x,\theta)}\right) f(x,\theta) \; \d x \\
&= \int_\chi \nabla f(x,\theta) \; \d x \\
&= \nabla \int_\chi f(x,\theta) \; \d x  \quad \text{(regularity)}\\
&= \nabla(1) \\
&= 0.
\end{align*}
\end{proof}

Q: Fisher Information Matrix
A: For a parameter space $\Theta\in\R^p$ and $\theta\in\Int(\Theta)$, we define the positive semi-definite symmetric matrix
\[
I(\theta) = \E_\theta\left[ \nabla\log f(X,\theta) \nabla \log f(X,\theta) ^T \right] = \cov (S_n(\theta)) = \E_\theta[i_n(\theta)].
\]
Intuitively, $I(\theta_0)\sim$ difficulty in estimating $\theta$ given $\theta=\theta_0$.

Q: Fisher matrix as a Hessian
A: Assuming we can swap differentiation/integration:
\[
I(\theta) = -\E\left[ \nabla_\theta^2\log f(X,\theta) \right].
\]
\begin{proof}
\begin{align*}
\nabla_\theta^2\log f(X,\theta) &= \nabla_\theta \left( \frac{\nabla_\theta f(X,\theta) }{f(X,\theta)}  \right) \\
&= \frac{\nabla_\theta^2f(X,\theta)}{f(X,\theta)} - \frac{\nabla_\theta f(X,\theta) \nabla_\theta f(X,\theta)^T}{f(X,\theta)^2} \\
-\E\left[ \nabla_\theta^2\log f(X,\theta) \right] &= -\int_\chi \left[ \frac{\nabla_\theta^2 f(x,\theta)}{f(x,\theta)} - \frac{\nabla_\theta f(x,\theta) \nabla_\theta f(x,\theta)^T}{f(x,\theta)^2}\right] f(x,\theta) \; \d x \\
&= -\nabla_\theta^2 \int_\chi f(x,\theta) \; \d x + \E_\theta\left[ \nabla_\theta\log f(X,\theta) \nabla_\theta \log f(X,\theta)^T\right] \\
&= I(\theta).
\end{align*}
\end{proof}

Q: Fisher information of iid variables
A: When $X=(X_1, \cdots X_n)$ consists of $n$ iid observations: $I_n(\theta)=nI(\theta)$.
\begin{proof}
By independence
\[
\nabla_\theta \log f(X_1, \cdots X_n, \theta) = \sum_{i=1}^n \nabla_\theta \log f(X_i, \theta)
\]
so
\[
I_n(\theta) = \sum_{i=1}^n\sum_{j=1}^n \E_\theta\left[ \nabla_\theta\log f(X_i, \theta) \nabla_\theta \log f(X_j, \theta)^T \right].
\]
For $i\neq j$, by centredness and independence, all the terms vanish. For $i=j$, each term is equal to $I(\theta)$.
\end{proof}

Q: Cram\`er-Rao lower bound
A: Let $\{ f(\ph, \theta): \theta\in\Theta \subseteq\R \}$ be a regular sm. Let $\tilde\theta$ be an unbiased estimator of $\theta$. Then $\forall \theta\in\Int(\Theta)$:
\[
\var_\theta(\tilde\theta) = \E_\theta \left[ (\tilde\theta - \theta)^2\right] \geq I_n(\theta)^{-1}.
\]
\begin{proof}
For $\var_\theta(\tilde\theta)<\infty$, $n=1$, by Cauchy-Schwarz
\[
\var_\theta(\tilde\theta) \geq \frac{\cov_\theta(\tilde\theta, S)^2}{\var_\theta(S)}
\]
where $\var_\theta(S) = I(\theta)$ and
\begin{align*}
\cov_\theta(\tilde\theta, S) &= \E_\theta \left[\tilde\theta S\right] \quad \text{(centred)}\\
&= \int_\chi \tilde\theta(x) \frac{\frac{\partial}{\partial\theta} f(x,\theta)}{f(x,\theta)} f(x,\theta) \; \d x \\
&= \frac{\d}{\d\theta}  \int_\chi \tilde\theta(x)  f(x,\theta) \; \d x \\
&= \frac{\d}{\d\theta} \theta \quad \text{(unbiased)}\\
&= 1.
\end{align*}
For $n>1$, see ES.
\end{proof}

Q: Minimum variance of a functional
A: For $\Phi: \Theta\to\R$ differentiable, $\tilde\Phi$ an unbiased estimator of $\Phi(\theta)$ based on $X_1, \cdots X_n$, we have that $\forall\theta\in\Int(\Theta)$:
\[
\var_{\theta}(\tilde\Phi) \geq \frac{1}{n} \nabla_\theta \Phi(\theta)^T I^{-1}(\theta) \nabla_\theta\Phi(\theta).
\]

Q: Asymptotically unbiased
A: The estimator $\tilde\theta_n$ of $\theta\in\Theta$ based on $X_1, \cdots X_n$ is \emph{asymptotically unbiased} if
\[
\E_\theta[ \tilde\theta_n] \to \theta \text{ as } n\to \infty.
\]

Q: Consistency
A: The estimator $\tilde\theta_n$ of $\theta\in\Theta$ based on $X_1, \cdots X_n$ is \emph{consistent} if
\[
\tilde\theta_n \xrightarrow{p} \theta \text{ as } n\to \infty.
\]

Q: Continuous mapping theorem
A: If $X_n, X$ take values in $\chi\in\R^d$ and $g:\chi\to\R$ is continuous, then
\[
X_n\xrightarrow{a.s./p/d} X \implies g(X_n)\xrightarrow{a.s./p/d} g(X).
\]

Q: Slutsky's Lemma
A: Let $X_n\xrightarrow{d} X, Y_n\xrightarrow{d} c$, where $c$ is deterministic. Then
\begin{enumerate}
\item $Y_n\xrightarrow{p} c$.
\item $\forall g$ cts: $g(X_n,Y_n)\xrightarrow{d} g(X,c)$.
\end{enumerate}
\begin{proof}
(i) is immediate. For (ii), by CMT ets $(X_n,Y_n)\xrightarrow{d} (X, c)$. Indeed, using Levy and BCT
\begin{align*}
|\phi_{X_n,Y_n}(t) - \phi_{X,c}(t) | &= |\E[e^{it_1X_n + it_2Y_n}] -  \E[e^{it_1X + it_2c}]| \\
&\leq |\E[e^{it_1X_n} (e^{it_2Y_n} - e^{it_2c}) ] | + |e^{it_2c}|  |\E[e^{it_1X_n} - e^{it_1X} ] | \\
&\leq \E[ |e^{it_2Y_n} - e^{it_2c}|] + |\E[e^{it_1X_n} - e^{it_1X} ] | \to 0.
\end{align*}
\end{proof}

Q: Convergent $\implies$ bounded
A: If $X_n\xrightarrow{d} X$ as $n\to\infty$, then $(X_n)_{n\geq 0}=O_P(1)$.
\begin{proof}
Given $\varepsilon>0$, pick $M_0\in\R$ such that $\P(|X|\geq M_0)<\varepsilon$.
By convergence, we can pick $N\in\N$ such that $\forall n\geq N$: $\P(|X_n|\geq M_0)<\varepsilon$.
Finally pick $M>M_0$ such that $\forall n\leq N$: $\P(|X_n|\geq M)<\varepsilon$.
In conclusion $\forall n\in\N: \P(|X_n|\geq M)< \min\{\P(|X_n|\geq M_0), \varepsilon \}<\varepsilon$.
\end{proof}

Q: Bounded in probability
A: $(X_n)=O_P(1)$ if $\forall\varepsilon>0$, $\exists M<\infty$ such that $\forall n\in\N$:
\[
\P(\|X_n\| \geq M)<\varepsilon.
\]
More generally, $(X_n)=O_P(a_n)$ if $(X_n/a_n)=O_P(1)$.

Q: Weak law of large numbers
A: Let $X_1, \cdots X_n$ be iid copies of $X$ with $\var X<\infty$, then
\[
\bar X_n = \frac{1}{n} \sum_{i=1}^n X_i \xrightarrow{p} \E[X].
\]
\begin{proof}
Given $\varepsilon >0$, by Chebyshev
\[
\P (\|\bar X_n - \E X \| > \varepsilon)  \leq \frac{\var \bar X_n}{\varepsilon^2} = \frac{1}{n} \frac{\var X_n}{\varepsilon^2} \to 0
\]
as $n\to\infty$.
\end{proof}

Q: Deviation of sample mean
A: Under the conditions of the CLT:
\[
\bar X_n - \E X = O_P\left( \frac{1}{\sqrt n} \right).
\]

Q: Asymptotic efficiency
A: Consistent estimators which attain equality in the CRLB asymptotically, i.e. as $n\to\infty$:
\[
\sqrt{n}(\hat{\theta}_n - \theta) \xrightarrow{d} N(0, I^{-1}(\theta)).
\]

Q: Assumptions for consistency theorem
A: Let $\{f(\ph, \theta): \theta\in\Theta \}$ be a model on $\chi\subseteq\R^d$ such that
\begin{itemize}
\item $f(x,\theta) > 0$ \qquad $(\forall x\in\chi, \theta\in\Theta)$.
\item $\int_\chi f(x,\theta) \; \d x =1$ \qquad $(\forall \theta\in\Theta)$.
\item $\theta\mapsto f(x,\theta)$ is continuous for all $x\in\chi$.
\item $\Theta\subseteq\R^p$ is compact.
\item $\theta\neq\theta' \implies f(\ph, \theta) \neq f(\ph, \theta')$.
\item $\E_\theta\sup_{\theta'} |\log f(X,\theta')| < \infty$.
\end{itemize}
These assumptions guarantee that
\begin{itemize}
\item $\ell_n$ has a unique max at $\theta_0$.
\item $\ell(\theta)=\E_{\theta_0} \log f(X,\theta)$ is continuous.
\end{itemize}
Note we can remove compactness if we add differentiability.

Q: Consistency theorem for MLEs
A: Let $X_1, \cdots X_n$ be iid from $\{ f(\ph, \theta) : \theta\in\Theta \}$ satisfying regularity assumptions. Then the MLE $\hat\theta_n$ exists and is consistent.
\begin{proof}
(E) $\bar\ell_n$ is continuous, $\Theta$ is compact.
(C) Given $\varepsilon>0$, define $\Theta_\varepsilon=\{ \theta\in\Theta: \|\theta - \theta_0 \| \geq \varepsilon \}$ compact. $\ell$ is continuous so is maximised at some $\theta_\varepsilon\in\Theta_\varepsilon$, i.e.
\[
\ell(\theta_\varepsilon) = \sup_{\theta\in\Theta_\varepsilon} \ell(\theta) = M < \ell(\theta_0)
\]
as $\theta_0$ is the unique max of $\ell|_\Theta$. Hence we can pick $\delta>0$ such that $M + \delta < \ell(\theta_0) - \delta$. By triangle inequality
\[
 \sup_{\theta\in\Theta_\varepsilon} \bar\ell_n(\theta) \leq  M  + \sup_{\theta\in\Theta_\varepsilon} |\bar\ell_n(\theta) - \ell(\theta)| .
\]
Define the events
\[
A_n = \left\{ \sup_{\theta\in\Theta} |\bar\ell_n(\theta) - \ell(\theta)| <\delta\right\}
\]
on which
\[
\sup_{\theta\in\Theta_\varepsilon} \bar\ell_n(\theta) < M + \delta < \ell(\theta_0) - \delta< \bar\ell_n(\theta_0)
\]
so $\hat\theta_n\not\in \Theta_\varepsilon$, i.e. $A_n \subseteq \{ \| \hat\theta_n - \theta_0 \| < \varepsilon\}$.
As $n\to\infty$, by the uniform LLN $\P(A_n)\to 1$ so $\P(\| \hat\theta_n - \theta_0 \| < \varepsilon) \to 1$.
\end{proof}

Q: Uniform Law of Large Numbers
A: Suppose $f:\chi\times\Theta\to\R$ is continuous in $\theta$ (for each fixed $x\in\chi$), $\Theta$ is compact, such that $\E[\sup _{\theta \in \Theta } |f(X,\theta)|]<\infty$. Then
\[
\sup _{\theta \in \Theta }\left|{\frac {1}{n}}\sum _{i=1}^{n}f(X_{i},\theta )-\E [f(X,\theta )]\right|{\xrightarrow {p }}\ 0.
\]
\begin{proof}
Follows from finite case by compactness
\end{proof}

Q: Uniform Law of Large Numbers (finite case)
A: Let $X_1, \cdots X_n$ be iid in $\chi\subseteq\R^d$ and $h:\chi\to\R$ such that $\E[h(X)]<\infty$. Then
\[
\max_{1\leq j \leq M} \left|  \frac{1}{n}\sum_{i=1}^n h(X_i) - \E[h(X)] \right| \xrightarrow{a.s.} 0 .
\]
\begin{proof}
Let $h_1, \cdots h_M$ be functions $\chi\to\R$ such that $\E[h_i(X)]<\infty$. Then
\[
\frac{1}{n}\sum_{i=1}^n h_j(X_i) - \E[h_j(X)] \xrightarrow{a.s.} 0.
\]
In particular, on events $A_j$ such that $\P(A_j^C)=0$, we have the convergence
\[
\frac{1}{n}\sum_{i=1}^n h_j(X_i) - \E[h_j(X)] \to 0.
\]
Hence on $A=\bigcap_{j=1}^M A_j$, we have $A^C=\bigcup_{j=1}^M A_j^C$ null such that
\[
\max_{1\leq j \leq M} \left| \frac{1}{n}\sum_{i=1}^n h_j(X_i) - \E[h(X)] \right| \to 0.
\]
\end{proof}

Q: Asymptotic efficiency of MLEs
A: Let $\{ f(\ph, \theta) : \theta\in\Theta \}$ satisfy regularity assumptions and $\hat\theta_n$ be the MLE based on $X_1, \cdots X_n$ iid from $P_{\theta_0}$. Then as $n\to\infty$
\[
\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} N(0, I(\theta_0)^{-1}).
\]
\begin{proof}
By consistency $\hat\theta_n\xrightarrow{P_{\theta_0}} \theta_0$. So $\hat\theta_n\in\Int\Theta$ almost surely.
By MVT, $\exists \bar\theta \in (\theta_0, \hat\theta_n)$ such that
\[
0 = \nabla_\theta \bar\ell_n(\hat\theta_n) = \nabla_\theta\bar\ell_n(\theta_0) + A_n(\hat\theta_n - \theta_0)
\]
where $A_n=\nabla_\theta^2\bar\ell_n(\bar\theta) \xrightarrow{p} -I(\theta_0)$. Rearranging
\[
\sqrt{n}(\hat\theta_n-\theta_0) = \underbrace{-A_n^{-1}}_{\xrightarrow{p}I(\theta_0)^{-1}}\sqrt{n}\nabla_\theta\bar\ell_n(\theta_0)
\]
By CLT
\begin{align*}
\sqrt{n}\nabla_\theta\bar\ell_n(\theta_0) &= \frac{1}{\sqrt{n}} \sum_{i=1} ^ n \left[ \nabla_\theta\log f(X, \theta) - \underbrace{\E_{\theta_0} \nabla_\theta \log f(X, \theta)}_{=0}\right] \\
&\xrightarrow{d} N(0, \underbrace{\cov_{\theta_0} \nabla_\theta \log f(X,\theta)}_{I(\theta_0)} ).
\end{align*}
Applying Slutsky
\[
\sqrt{n}(\hat\theta_n-\theta_0)  \xrightarrow{d} N(0, I(\theta_0)^{-1} (I(\theta_0) I(\theta_0)^{-1}) ) = N(0, I(\theta_0)^{-1}).
\]
\end{proof}

Q: Profile likelihood
A: Define the profile likelihood $L^{(p)}: \Phi(\Theta) \to \R$ by
\[
L^{(p)}(\phi) = \sup_{\theta\in\Theta: \Phi(\theta)=\phi} L(\theta).
\]
This has the property that
\[
\sup_{\phi\in\Phi(\Theta)} L^{(p)}(\phi) = \sup_{\theta\in\Theta} L(\theta).
\]

Q: Delta Method
A: Let $\Phi\in C^1(\Theta)$ satisfy $\nabla\Phi(\theta_0)\neq 0$. Let $\hat\theta_n$ be estimators satisfying $\sqrt{n}(\hat\theta_n-\theta_0) \xrightarrow{d} Z$, $Z$ rv in $\R$. Then as $n\to\infty$
\[
\sqrt{n}\left(\Phi(\hat\theta_n) - \Phi(\theta_0) \right) \xrightarrow{d} \nabla \Phi(\theta_0)^T Z.
\]
In particular, for $\hat\theta_n$ efficient (e.g. plug-in MLE) $Z=N(0,I^{-1}(\theta_0))$.
\begin{proof}
By MVT, $\exists \tilde\theta_n\in[\theta_0, \hat\theta_n]$ such that
\[
\sqrt{n}(\Phi(\hat\theta_n) - \Phi(\theta_0)) = \nabla\Phi(\tilde\theta_n)^T \sqrt{n} (\hat\theta_n - \theta_0).
\]
Since $\sqrt{n}(\hat\theta_n - \theta_0) \xrightarrow{d} Z$, it is bounded in probability and $\|\hat\theta_n - \theta_0 \| = O_p(\frac{1}{\sqrt n})$. So $\hat\theta_n \xrightarrow{P_{\theta_0}} \theta_0$. Thus $\tilde\theta_n\xrightarrow{P_{\theta_0}} \theta_0$. By CMT, $\nabla\Phi(\tilde\theta_n) \xrightarrow{P_{\theta_0}} \nabla\Phi(\theta_0)$. Done by Slutsky.
\end{proof}

Q: Plug-in MLE: definition
A: Let $\Phi:\Theta\to\R^d$. Then the plug-in MLE of $\Phi(\theta_0)$ is the estimator $\Phi(\hat\theta_\text{MLE})$.

Q: Observed Fisher Information Matrix
A: We define the positive semi-definite symmetric $p\times p$ matrix
\[
i_n(\theta) = \frac{1}{n}\sum_{i=1}^n \nabla_\theta \log f(X_i, \theta) \nabla_\theta \log f(X_i, \theta)^T.
\]
In particular, we write the plug-in MLE as $\hat i_n = i_n(\hat\theta_\text{MLE})$. Then $\hat i_n\xrightarrow{P_{\theta_0}} I(\theta_0)$.
\begin{proof}
Write $q(X,\theta)=\nabla_\theta \log f(X,\theta) \nabla_\theta\log f(X,\theta)^T$. Therefore
\begin{align*}
|\hat i_n - I(\theta_0)| &\leq \left| i_n(\hat\theta_\text{MLE}) - I(\hat\theta_\text{MLE}) \right| + \left| I(\hat\theta_\text{MLE}) - I(\theta_0)\right| \\
&\leq \sup_{\theta\in\Theta} \left| \frac{1}{n} \sum_{i=1}^n q(X_i, \theta) - \E_{\theta_0} q(X,\theta)\right| + \left| I(\hat\theta_\text{MLE}) - I(\theta_0)\right| \\
& \xrightarrow{P_{\theta_0}} 0
\end{align*}
by ULLN, consistency of the MLE and continuity of $I(\theta)$.
\end{proof}

Q: Wald statistic
A: For all $\theta\in\Theta$, define
\[
W_n(\theta) = n(\hat\theta_\text{MLE} - \theta) ^T\hat i_n (\hat\theta_\text{MLE} - \theta).
\]
This is a quadratic form as $\hat i_n$ is positive semi-definite. The level sets are ellipsoids.

Q: Wald confidence ellipsoids
A: The set $C_n=\{ \theta: W_n(\theta) \leq \xi_\alpha \}$, where $\P(\chi_p^2 \leq \xi_\alpha)=1-\alpha$, is an $\alpha$-level asymptotic CR for $\theta$.
\begin{proof}
We can write
\[
W_n(\theta_0) = \underbrace{\sqrt{n}(\hat\theta_n - \theta_0)^T}_{\xrightarrow{d}N(0, I(\theta_0)^{-1}) }  I(\theta_0) \sqrt{n} (\hat\theta_n - \theta_0) + \sqrt{n}(\hat\theta_n - \theta_0)^T \underbrace{(\hat i_n - I(\theta_0))}_{\xrightarrow{p}0} \sqrt{n} (\hat\theta_n - \theta_0) \xrightarrow{d} \chi^2_p.
\]
So $ \P(\theta_0 \in C_n) = \P(W_n(\theta_0) \leq \xi_\alpha) =1-\alpha$.
\end{proof}

Q: Wilk's Theorem
A: Suppose $\Theta_0 \subseteq \Theta \subseteq \R^p$ and $\dim \Theta_0=p_0$. Under $H_0: \theta\in\Theta_0$, as $n\to \infty$,
  \[
   w_{LR}(H_0)= 2\log \Lambda_n\xrightarrow{d} \chi_{p-p_0}^2.
  \]
\begin{proof}
Assume $ \Theta_0= \{\theta_0 \}$ for $\theta_0\in \Int\Theta$, i.e. $p_0=0$. Then by Taylor's Theorem
\begin{align*}
w_{LR} &= 2\ell_n(\hat\theta_n) - 2\ell_n(\theta_0) \\
&= -2\ell_n(\theta_0) - (-2\ell_n(\hat\theta_n)) \\
&= -2\underbrace{\nabla \ell_n(\hat\theta_n)^T}_{=0}(\theta_0 - \hat\theta_n) - \underbrace{\sqrt{n}(\theta_0 - \hat\theta_n)}_{\xrightarrow{d} N(0, I^{-1}(\theta_0))} \underbrace{\nabla^2 \bar\ell_n(\tilde\theta)}_{\xrightarrow{p} -I(\theta_0)} \sqrt{n}(\theta_0 - \hat\theta_n) \quad (\tilde\theta\in[\theta_0, \hat\theta_n]) \\
& \xrightarrow{d} \chi_p^2.
\end{align*}
\end{proof}

Q: Improper prior
A: A prior non-negative function over the parameter space with infinite integral.

Q: Jeffreys prior
A: \[
\pi(\theta)\propto \sqrt{\det I(\theta)}
\]

Q: Bayes' Rule
A: Bayes' Rule states
\[
O(\Theta_1: \Theta_2 | X) = O(\Theta_1: \Theta_2) \Lambda(\Theta_1: \Theta_2 | X)
\]
where
\[
\Lambda(\Theta_1: \Theta_2 | X) = \frac{\P(X | \Theta_1)}{\P(X | \Theta_2)}
\]
is the \emph{Bayes factor} or LR,
\[
O(\Theta_1: \Theta_2) = \frac{\pi(\Theta_1)}{\pi(\Theta_2)}
\]
is the prior odds and
\[
O(\Theta_1: \Theta_2 | X) = \frac{\pi(\Theta_1 | X)}{\pi(\Theta_2 | X)}
\]
is the posterior odds.

Q: Bernstein-von Mises
A: Let $\pi$ be cts and positive at $\theta_0$ and $\phi_n\sim N(\hat\theta_n, \frac{1}{n} I^{-1}(\theta_0))$. Then, for $\Theta\subseteq\R$, as $n\to\infty$:
\[
\|\Pi_n - \phi_n \|_1 \xrightarrow{a.s.} 0.
\]
\begin{proof}
Since $\Pi_n, \phi_n$ are distributions
\[
\int_\Theta (\phi_n- \Pi_n) = 0.
\]
As $\phi_n>0$, by DCT:
\[
\int_\Theta |\phi_n- \Pi_n| = 2\int_\Theta (\phi_n- \Pi_n) ^+ =2\int_\Theta \phi_n\left(1 - \frac{\Pi_n}{\phi_n}\right) ^+  \xrightarrow{a.s.} 0.
\]
\end{proof}

Q: Credible sets
A: The set $\mathcal{C}_n$is a $1-\alpha$ credible set if $\Pi_n(\mathcal{C}_n)=1-\alpha$.
\begin{eg}
\[
\mathcal{C}_n = \left\{ v: |v-\hat\theta_n| \leq \frac{R_n}{\sqrt{n}} \right\}
\]
\end{eg}
\begin{eg}
\[
\mathcal{C}_n = \left\{ v: |v-\bar\theta_n| \leq \frac{R_n}{\sqrt{n}} \right\}
\]
\end{eg}

Q: Credible set convergence of radius
A: Under regularity assumptions of B-vM, writing $\Phi_0$ for the cdf of $|N(0, I^{-1}(\theta_0) )|$
\[
R_n\xrightarrow{a.s.} \Phi_0^{-1}(1-\alpha).
\]
\begin{proof}
We compute, writing $\phi_n\sim N(\hat\theta_n, \tfrac{1}{n} I^{-1}(\theta_0))$:
\begin{align*}
\Phi_0(R_n) &= \int_{\hat\theta_n - R_n/\sqrt{n}}^{\hat\theta_n+R_n/\sqrt{n}} \phi_n(\theta) \; \d \theta \\
&= \phi_n(C_n) \\
&= \underbrace{\phi_n(C_n)  - \Pi_n(C_n)}_{\xrightarrow{a.s.} 0 \text{ by B-vM}}  + \underbrace{\Pi_n(C_n)}_{=1-\alpha}  \\
&\xrightarrow{a.s.} 1- \alpha.
\end{align*}
Done by CMT.
\end{proof}

Q: Credible sets are confidence sets
A: Under the regularity assumptions of B-vM, $\forall\alpha\in(0,1)$: $\P_{\theta_0}(\theta_0\in\mathcal{C}_n)\to 1 - \alpha$ as $n\to\infty$.
\begin{proof}
We compute
\begin{align*}
\P_{\theta_0}(\theta_0\in\mathcal{C}_n) &= \P_{\theta_0}(|\hat\theta_n-\theta_0| \leq \tfrac{R_n}{\sqrt{n}}) \\
&= \P_{\theta_0}\left( \underbrace{\frac{\Phi_0^{-1}(1-\alpha)}{R_n}}_{\xrightarrow{a.s.} 1} \underbrace{\sqrt{n} |\hat\theta_n - \theta_0|}_{\xrightarrow{d} N(0, I^{-1}(\theta_0))} \leq \Phi_0^{-1}(1-\alpha)\right) \\
&\to \P_{\theta_0}(|Z| \leq \Phi_0^{-1}(1-\alpha)) \\
&= 1-\alpha.
\end{align*}
\end{proof}

Q: Decision rule
A: A function $\delta:X\to A$ where $X$ is the sample space and $A$ is the action space.

Q: Loss function
A: A function $L:A\times\Theta\to[0,\infty)$.

Q: Risk
A: The risk of a decision rule $\delta$ given parameter $\theta$ is
\[
R(\delta,\theta) = \E_{\theta}[L(\delta(X), \theta)].
\]

Q: Bayes risk
A: Given a prior $\pi$, the $\pi$-Bayes risk is
\[
R_\pi(\delta) = \E_\pi[R(\delta, \theta)] = \int_{\Theta\times\chi} L(\delta(x), \theta) f(x,\theta) \pi(\theta) \; \d\theta \; \d x.
\]

Q: Posterior risk
A: The posterior risk given an observation $x\in\chi$ with posterior $\Pi$ is
\[
R_\Pi(\delta) = \int_{\Theta} L(\delta(x), \theta) \Pi(\theta) \; \d\theta \; \d x.
\]

Q: Posterior risk minimiser is a Bayes rule
A: If $\delta$ is the decision rule which minimises the posterior risk for all $x\in \chi$, then $\delta$ minimises the $\pi$-Bayes risk.
\begin{proof}
The $\pi$-Bayes risk is
\begin{align*}
R_\pi(\delta) &= \int_\chi \int_\Theta L(\delta(x),\theta) f(x,\theta) \pi(\theta) \; \d x \; \d\theta \\
&=  \int_\chi \int_\Theta L(\delta(x),\theta) \frac{f(x,\theta) \pi(\theta)}{\int_\Theta f(x,\theta') \pi(\theta') \; \d\theta'} \underbrace{\int_\Theta f(x,\theta') \pi(\theta') \; \d\theta'}_{m(x)\geq 0} \; \d x \; \d\theta \\
&= \int_\chi \E_\pi[L(\delta(x),\theta) | x] m(x) \; \d x
\end{align*}
which is uniformly minimised by $\delta$.
\end{proof}

Q: Bayes decision rule
A: A decision rule $\delta_\pi$ which minimises the $\pi$-Bayes risk is called a $\pi$-Bayes rule.

Q: Unbiased Bayes rules
A: Let $\delta$ be an unbiased decision rule, i.e. $\E_\theta[\delta(X)]=\theta$ for all $\theta$. Assume further that $\delta$ is the $\pi$-Bayes rule under quadratic risk, i.e. $\delta(X)=\E_\pi[\theta | X]$. Then
\[
\E_Q[(\delta(X)-\theta)^2] = 0
\]
where $Q(x,\theta)=f(x,\theta)\pi(\theta)$. In particular $\delta(X)=\theta$ $Q$-a.s.
\begin{proof}
By tower rule of conditional expectation
\begin{align*}
\E_Q[\theta\delta(X)] &= \E_Q[\E_\pi[\theta\delta(X) | X]] = \E_Q[\delta(X)^2] \\
\intertext{and by unbiasedness}
\E_Q[\theta\delta(X)] &= \E_Q[\theta\E_\theta[\delta(X)]] = \E_Q[\theta^2].
\end{align*}
So
\[
\E_Q[(\delta(X)-\theta)^2] = \E_Q[\delta(X)^2] + \E_Q[\theta^2] - 2\E_Q[\theta\delta(X)] = 0.
\]
\end{proof}

Q: Maximal risk
A: The maximal risk is
\[
R_m(\delta,\Theta) = \sup_{\theta\in\Theta} R(\delta,\theta).
\]

Q: Minimax
A: The minimum of the maximum risk is
\[
\inf_\delta \sup_{\theta\in\Theta} R(\delta, \theta)= \inf_\delta R_m(\delta, \Theta).
\]
A decision rule which attains the minimax risk is called \emph{minimax}.

Q: Bayes risk bounded by maximum risk
A: For any prior $\pi$ and decision rule $\delta$
\[
R_\pi(\delta) \leq R_m(\delta,\Theta).
\]
\begin{proof}
We can compute
\[
R_\pi(\delta)  = \E_\pi [R(\delta, \theta)] \leq \sup_{\theta\in\Theta} R(\delta, \theta) = R_m(\delta, \Theta).
\]
\end{proof}

Q: Least favourable
A: A prior $\lambda$ is \emph{least favourable} if it has the greatest Bayes risk, i.e. for any prior $\lambda'$:
\[
R_\lambda(\delta_\lambda) \geq R_{\lambda'}(\delta_{\lambda'}).
\]

Q: Bayes risk equals maximum risk
A: Let $\lambda$ be a prior on $\Theta$ such that the (unique) Bayes rule $\delta_\lambda$ achieves the maximal risk, i.e.
\[
R_\lambda(\delta_\lambda) = R_m(\delta_\lambda, \Theta).
\]
Then $\delta_\lambda$ is (unique) minimax and $\lambda$ is least favourable.
\begin{proof}
For any $\delta\neq \delta_\lambda$:
\begin{align*}
R_m(\delta, \Theta) & \geq  R_\lambda(\delta) \\
&= R_\lambda(\delta_\lambda) \\
&= R_m(\delta_\lambda, \Theta).
\end{align*}
For any prior $\lambda'$:
\begin{align*}
R_{\lambda'}(\delta_{\lambda'}) &\leq R_{\lambda'}(\delta_{\lambda}) \\
&\leq R_m(\delta_{\lambda}, \Theta) \\
&= R_\lambda(\delta_\lambda).
\end{align*}
\end{proof}

Q: Constant risk Bayes rule
A: If a (unique) Bayes rule $\delta_\lambda$ has risk independent of $\theta$, then is (unique) minimax.
\begin{proof}
Simply observe we have maximality of the Bayes risk:
\[
R_\lambda(\delta_\lambda) =  R_m(\delta_\lambda, \Theta).
\]
\end{proof}

Q: Inadmissible
A: A decision rule $\delta$ is \emph{inadmissible} if $\exists\delta'$ such that $\forall\theta\in\Theta$:
\[
R(\delta',\theta) \leq R(\delta,\theta)
\]
with strict inequality for some $\theta\in\Theta$.

Q: Bayes rule is admissible
A: A unique $\lambda$-Bayes rule $\delta_\lambda$ is admissible.
\begin{proof}
Suppose $\exists \delta$ such that $R(\delta, \ph)\leq R(\delta_\lambda, \ph)$. Taking expectations
\[
R_\lambda(\delta) \leq R_\lambda(\delta_\lambda).
\]
By uniqueness $\delta=\delta_\lambda$.
\end{proof}

Q: Admissible $\implies$ minimax
A: If $\delta$ is admissible and constant risk, then it is minimax.
\begin{proof}
Given decision rule $\delta'$, by admissibility $\exists\theta\in\Theta$ such that
\[
R_m(\delta', \Theta) \geq R(\delta', \theta) \geq R(\delta, \theta) = R_m(\delta, \Theta).
\]
\end{proof}

Q: Minimax estimator of Normal mean
A: Let $X_1\cdots X_n$ be iid $N(\theta,\sigma^2)$ with $\sigma^2$ known. Then $\hat\theta=\bar X_n$ is an admissible and minimax estimator of $\theta$ with quadratic loss.
\begin{proof}
Wlog $\sigma^2=1$. The risk is simply a consant:
\[
R(\hat\theta, \theta) = \E_\theta[(\bar X_n -\theta)^2]  = \frac{1}{n}.
\]
So ets $\hat\theta_n$ is admissible. In general for any decision rule $\delta$
\[
R(\delta, \theta) =  \E_\theta[(\hat\theta -\theta)^2] = B(\theta)^2 + \var_\theta (\delta)
\]
where $B(\theta)=\E_\theta[\delta]-\theta$ is the bias. From the proof of CRLB
\[
\var_\theta(\delta)\geq \frac{(\frac{\d}{\d\theta}\E_\theta[\delta])^2}{nI(\theta)} = \frac{(1+B'(\theta))^2}{n}
\]
If $\delta$ dominates $\hat\theta$, then
\[
B(\theta)^2 + \frac{(1+B'(\theta))^2}{n} \leq \frac{1}{n}.
\]
So $B,B'$ are bounded and $ B'\leq 0$. Thus $\exists(\theta_n)\to \pm\infty$ such that $B'(\theta_n)\to 0$ so $B(\theta_n)\to 0$. Since $B\searrow$ in $\theta$, $B=0$. Hence the CRLB applies: $\var_\theta(\delta) \geq \frac{1}{n}$, contradicting the assumption that $\delta$ dominates $\hat\theta$.
\end{proof}

Q: James-Stein estimator
A: The J-S estimator of $X\in\R^p$ for $p\geq 3$ is
\[
\delta^{\text{JS}}(X) = \left(1- \frac{p-2}{\|X\|^2}\right)X.
\]

Q: Stein's Lemma
A: Let $X\sim N(\theta, \sigma^2)$ and $g:\R\to\R$ be bounded, differentiable with $\E|g'(X)|<\infty$. Then
\[
\E[(X-\theta)g(X)] = \sigma^2\E[g'(X)].
\]
\begin{proof}
We compute
\begin{align*}
\E[(X-\theta)g(X)]  &= \int_\R (x-\theta) g(x) \frac{\exp (-\tfrac{1}{2\sigma^2} (x-\theta)^2)}{\sqrt{2\pi}} \; \d x\\
&= - \sigma^2\int_\R g(x) \frac{\d}{\d x}\left(  \frac{\exp (-\tfrac{1}{2\sigma^2} (x-\theta)^2)}{\sqrt{2\pi}}\right) \; \d x \\
&= -\sigma^2\left[ g(x) \left(  \frac{\exp (-\tfrac{1}{2\sigma^2} (x-\theta)^2)}{\sqrt{2\pi}}\right) \right]_{-\infty}^\infty +  \int_\R g'(x) \frac{\exp (-\tfrac{1}{2\sigma^2} (x-\theta)^2)}{\sqrt{2\pi}}) \; \d x \\
&=  \sigma^2\E[g'(X)].
\end{align*}
\end{proof}

Q: Risk of James-Stein
A: Let $X\sim N(\theta, I_p)$ for $p\geq 3$. Then $\forall\theta\in\R^p$:
\[
R(\delta^{JS},\theta) <p = R(\hat\theta, \theta).
\]
\begin{proof}
Write $\delta_{JS} = X- g(X)$ where $g(x) = \frac{p-2}{\|x\|^2}x$ has derivative
\begin{align*}
\nabla\cdot g(x) &= (p-2)\sum \left( \frac{x\cdot x-2x_i^2}{(x\cdot x)^2} \right) \\
&= \frac{(p-2)}{(x\cdot x)^2} \left(p\|x\|^2-2\|x\|^2 \right) \\
&= \frac{(p-2)^2}{(x\cdot x)}.
\end{align*}
Then by Stein's identity
\begin{align*}
R(\theta, \delta_{JS}) &= \E \|X- g(X)-\theta \|^2 \\
&= \E\|X-\theta\|^2 + \E\|g(X)\|^2 - 2\E[(X-\theta)\cdot g(X)\| \\
&= p + (p-2)^2\E\left[ \frac{1}{\|X\|^2}\right] - 2 (p-2)^2\E\left[ \frac{1}{\|X\|^2}\right] \\
&= p - (p-2)^2\E\left[ \frac{1}{\|X\|^2}\right]  \\
&< p.
\end{align*}
\end{proof}

Q: Classification rule
A: A classification rule is a map $\delta:\chi\to\{1,2\}$ where $X\sim f_1$ or $X\sim f_2$. We write $\delta=\delta_\mathcal{R}$ where $\mathcal{R}=\delta^{-1}(1)$ is the classification region.

Q: Classification error
A: The $\pi$-Bayes classification risk is
\begin{align*}
R_\pi(\delta_\mathcal{R}) &= \pi(1) f_1(\mathcal{R}^c) + \pi(2) f_2(\mathcal{R})\\
& = \P_Q(\delta(X)\neq Y) \\
&= \int_{\mathcal{R}} \Pi (2|x) \; \d P_X(x) + \int_{\mathcal{R}^C} \Pi (1|x) \; \d P_X(x)
\end{align*}
where $P_X(x) = \pi(1)f_1(x) + \pi(2)f_2(x)$ is the marginal distribution of $X$.

Q: Bayes classifier
A: For a prior $\pi=\{q_1,q_2\}$, $q_i\in (0,1)$, $\delta_\pi=\delta_\mathcal{R}$ where
\[
\mathcal{R}=\left\{ x\in\chi: \frac{q_1f_1(x)}{(1-q_1)f_2(x)}\geq 1\right\}
\]
minimises the classification risk. Moreover if
\[
\P\left( \frac{q_1f_1(x)}{(1-q_1)f_2(x)}= 1 \right)=0,
\]
then this is the unique Bayes rule.
\begin{proof}
For any $\mathcal{S}\subseteq\chi$, the risk
\begin{align*}
R_\pi(\delta_\mathcal{S}) &= q_1\int_{\mathcal{S}^C} f_1(x) \; \d x + (1-q_1) \int_{\mathcal{S}} f_2(x) \; \d x \\
&= \int_{\mathcal{S}^C} [q_1f_1(x)-(1-q_1)f_2(x)] \; \d x + (1-q_1)
\end{align*}
is minimised (uniquely) by the largest $\mathcal{S}$ for which the first integrand is negative (and non-zero a.e.).
\end{proof}

Q: Linear discriminant analysis
A: For the classification problem $f_1\sim N(\mu_1, \Sigma)$, $f_2\sim N(\mu_2, \Sigma)$, any Bayes or minimax classifier is a function of the discriminant
\[
D(X) = X^T\Sigma(\mu_1-\mu_2).
\]

Q: Sample correlation coefficient
A: Given observations $(X_1,Y_1),\cdots (X_n,Y_n)$, the \emph{sample correlation coefficient} is
\[
\hat{\rho}_{X,Y} = \frac{\tfrac{1}{n} \sum_{i=1}^n (X_i-\bar{X}_n)(Y_i-\bar{Y}_n)}{\sqrt{\tfrac{1}{n} \sum_{i=1}^n (X_i-\bar{X}_n)^2\tfrac{1}{n} \sum_{i=1}^n (Y_i-\bar{Y}_n)^2}}.
\]
If $0< \var X, \var Y <\infty$, then all the sample covariances and correlations are consistent.

Q: Correlation matrix
A: Let $\cov(X_i, X_j)=\Sigma_{ij}$ where $\Sigma\succ 0$ and
\[
\rho_{ij}=\rho_{X_i,X_j} = \frac{\Sigma_{ij}}{\sqrt{\Sigma_{ii}\Sigma_{jj}}}.
\]
Then $[\rho]\succeq 0$ has $\rho_{ij}\in[-1,1]$ and $\rho_{ii}=1$.

Q: Covariance vs. positive semidefinite matrix
A: Given $\Sigma \text{ (resp. $\rho$ with $\rho_{ii}=1$)} \in\R^{p\times p}\succeq 0$, $\exists X\in\R^p$ such that $\E[X]=0$ and $\cov X = \Sigma$ (resp. $\rho_{X_i,X_j} = \rho_{ij}$).
\begin{proof}
Immediate by orthogonal diagonalisability of real symmetric matrices and taking the square root.
\end{proof}

Q: Principal component analysis
A: For $X\in\R^p$ with $\E[X]=0$ and $\cov(X)=\Sigma$, $\exists V=(\mathbf{v}_1, \cdots \mathbf{v}_p)\in O_p(\R)$ such that $\Sigma=V\Lambda V^T$ where $\Lambda=\diag(\{\lambda_i\})$. Then
\[
\Sigma = \sum_{i=1}^p \lambda_i \mathbf{v}_i \mathbf{v}_i^T.
\]
Wrt $\{\mathbf{v}_i\}$, defining $U=V^TX$, we have
\[
\cov(U) = V^T\Sigma V =\Lambda.
\]
Conversely
\[
X=VU=\sum_{i=1}^p\mathbf{v}_i U_i = \sum_{i=1}^p\mathbf{v}_i \sqrt{\lambda_i} Z_i
\]
where $\cov Z =  I_p$. Hence the greatest variance is along the eigenvectors with largest eigenvalues.

Q: Jackknife bias estimate
A: Let $T_n$ be an estimator of $\theta$ with bias $B(\theta) = \E_\theta[T_n] - \theta$. Let $T_{(-i)}$ be the estimator with the $i$th observation removed. The jackknife bias estimate is defined as
\[
\hat{B}_n = (n-1) \left( \tfrac{1}{n} \sum_{i=1}^n T_{(-i)} - T_n \right)
\]
and the jackknife bias corrected estimate of $\theta$ is
\[
\tilde{T}=\tilde{T}_{JACK} = T_n - \hat{B}_n.
\]
For $B(\theta)$ regular, it holds that
\[
|\E[\tilde{T}_{JACK}] - \theta | = O(\tfrac{1}{n^2}).
\]
\begin{proof}
Assume that $B_n(\theta) = an^{-1}+bn^{-2}+O(n^{-3})$ and deduce that $\hat{B}_n=B_n(\theta)+O(n^{-2})$.
\end{proof}

Q: Bootstrap sample
A: Fix $(X_1, \cdots X_n)$ and generate $\{X_{n,i}^b : 1\leq i \leq n \}$ iid by sampling with discrete uniform law
\[
\P_n(X_n^b = X_i) = \frac{1}{n}
\]
and mean
\[
\E[X_n^b] = \sum_{i=1}^n X_i \P(X_n^b=X_i) = \frac{1}{n}\sum_{i=1}^n X_i = \bar X_n.
\]

Q: Efficiency of the bootstrap
A: Let $X_1, \cdots X_n$ be iid from $X\sim P$ with $\E[X]=\mu$ and $\var X = \sigma^2$. Then
\[
\sup_{t\in\R} \left| \P_n(\sqrt{n}(\bar X_n^b - \bar X_n) \leq \sigma t) - \Phi(t) \right| \xrightarrow{a.s.} 0.
\]
\begin{proof}
$Z_{n,i} = X_{n,i}^{b}$ is a triangular array of iid rvs with $\E_n[Z_{n,i}]=\bar{X}_n$ and
\begin{align*}
\sigma_n^2 = \var (Z_{n,i}) &= \E_n[Z_{n,i}^2] - \E_n[Z_{n,i}]^2 \\
&= \frac{1}{n} \sum_{i=1}^n X_i^2 - \left(\frac{1}{n}\sum_{i=1}^n X_i \right)^2.
\end{align*}
By SLLN, $\sigma_n^2\xrightarrow{a.s.} \sigma^2$ and by CLT for triangular arrays
\[
\sqrt{n}(\bar{X}_n^{b} - \bar{X}_n)\xrightarrow{d} N(0, \sigma^2).
\]
\end{proof}

Q: Convergence in distribution $\implies$ uniformly
A: Let $A_n\sim f_n, F_n$ and $A\sim f, F$ where $F$ is continuous. Then as $n\to\infty$
\[
A_n\sim f_n \xrightarrow{d} A\sim f \implies \sup_{t\in\R} | F_n(t) - F(t) | \to 0.
\]
\begin{proof}
Since $F$ is monotonic and continuous, for each $k\in\N$, $\exists (x_i)_1^{k-1}\subseteq\R$ such that $F(x_i)=i/k$.
Hence $\forall x\in [x_{i-1}, x_i)$:
\begin{align*}
F_n(x) - F(x) &\leq F_n(x_i) - F(x_{i-1}) = F_n(x_i) - F(x_i) + \frac{1}{k} \\
F_n(x) - F(x) &\geq F_n(x_{i-1}) - F(x_i) = F_n(x_{i-1}) - F(x_{i-1}) - \frac{1}{k}.
\end{align*}
Given $\varepsilon>0$, pick $k>2/\varepsilon$ and $n$ large enough that $|F_n(x_i)-F(x_i)|<\varepsilon / 2$ for each $0\leq i \leq k$. Then $\forall x\in\R$:
\[
|F_n(x) - F(x) |\leq \frac{\varepsilon}{2} + \frac{1}{k} < \varepsilon.
\]
\end{proof}

Q: Triangular array
A: $(Z_{n,i}: 1\leq i \leq n)_{n\geq 1}$ is a triangular array of iid rvs if for each row $n\geq 1$: $(Z_{n,i}: 1\leq i \leq n)$ are $n$ iid draws from $Q_n$.

Q: CLT for triangular arrays
A: Let $(Z_{n,i}: 1\leq i \leq n)_{n\geq 1}$ be a triangular array of iid rvs with finite variance such that $\var (Z_{n,i})=\sigma_n^2\to \sigma^2$ as $n\to\infty$. Assume that as $n\to\infty$
\begin{enumerate}
\item $\forall \delta>0$: $nP_n(|Z_{n}|>\delta\sqrt{n}) \to 0$.
\item $\var (Z_{n} \mathbf{1}_{|Z_{n}|\leq\sqrt{n} })\to\sigma^2$.
\item $\sqrt{n}\E[Z_{n}  \mathbf{1}_{|Z_{n}|>\sqrt{n} }] \to 0$.
\end{enumerate}
Then
\[
\sqrt{n}\left(\frac{1}{n} \sum_{i=1}^n Z_{n,i} - \E_n [Z_{n,i}] \right) \xrightarrow{d} N(0, \sigma^2).
\]
\begin{proof}
Non-examinable.
\end{proof}

Q: Bootstrap estimation
A: Compute the MLE $\hat{\theta}_n^b$ based on the bootstrap samples $(X_{n,i}^b: 1\leq i \leq n)$ from $\P_n$. We use $\sqrt{n}(\hat\theta_n^b - \hat{\theta}_n)$ as a pivot for $\sqrt{n}(\hat\theta_n - \theta_0)$. Find $R_n^b$ such that
\[
\P_n\left( \|\hat{\theta}_n^b - \hat{\theta}_n \| \leq R_n^b / \sqrt{n}\right) = 1 - \alpha.
\]
Then the bootstrap confidence set is
\[
\mathcal{C}_n^b = \{ \theta: \| \theta - \hat{\theta}_n \| \leq R_n^b / \sqrt{n} \}
\]
such that in fact $\P_{\theta_0}(\theta_0 \in\mathcal{C}_n^b) = 1 - \alpha$.
In parametric bootstrap, we sample instead from $P_{\hat{\theta}_n}$.

Q: Discrete uniform approximation
A: If $U_1, \cdots U_n$ are iid $U[0,1]$, then
\[
X_i = \sum_{j=1}^n x_j\mathbbm{1} \left\{ U_i\in \bigg(\frac{j-1}{n}, \frac{j}{n}\bigg] \right\}
\]
are iid uniform over $\{x_1, \cdots x_n\}$.
\begin{proof}
We can check that
\[
\P(X_i=x_j) = \P \left( U_i\in \bigg(\frac{j-1}{n}, \frac{j}{n}\bigg] \right)= \frac{1}{n}.
\]
\end{proof}

Q: Generalised inverse
A: For $F:\R\to [0,1]$ non-decreasing, we define
\[
F^-(u) = \inf\left\{ x: u\leq F(x)\right\}.
\]
Then for $U\sim U[0,1]$, $F^-(U)\sim F$.

Q: Importance sampling
A: Let $\supp (h)\supseteq\supp(f)$ and $h$ be easier to simulate from. Then
\[
\E_h\left[ \frac{g(X)}{h(X)} f(X)\right] = \int \frac{g(x)}{h(x)} f(x)h(x) \; \d x = \E_f[g(X)].
\]
Hence for $X_i$ sampled from $h$
\[
\frac{1}{n} \sum_{i=1}^n\left[ \frac{g(X_i)}{h(X_i)} f(X_i)\right] \xrightarrow{a.s.} \E_f[g(X)].
\]

Q: Accept/reject algorithm
A: Let $f\leq Mh$ with $h$ easier to sample.
Step 1: sample $X\sim h$ and $U\sim U[0,1]$.
Step 2: If $U\leq f(X)/Mh(X)$, then take $Y=X$. Else return to step 1.
Then $Y\sim f$.
\begin{proof}
First observe that
\[
\P\left( U\leq f(X)/Mh(X) \right) = \int h(x)[f(x)/Mh(x)] \; \d x = 1/M.
\]
Hence the computation time $T\sim\Geo(1/M)$ and $Y$ has density
\[
\sum_{i=0}^\infty \left( \frac{M-1}{M} \right)^i h(y) [f(y)/Mh(y)] = \frac{f(y)/M}{1-(M-1)/M} = f(y).
\]
\end{proof}

Q: Gibbs sampler
A: To sample from $(X,Y)$, starting from $X_0=x$, for $t\geq 1$ sample from the conditional distributions
\[
Y_t \sim f_{Y|X}(\ph | X=X_{t-1}), \qquad X_t \sim f_{X|Y}(\ph | Y=Y_t).
\]
Then by induction $(X_t, Y_t)$ is a MC with invariant distribution $f_{X,Y}$.
By ergodic theorems
\[
\frac{1}{n}\sum_{i=1}^n g(X_t, Y_t) \xrightarrow{a.s.} \E [g(X,Y)].
\]

Q: Empirical distribution function
A: Given $X_1, \cdots X_n$ iid samples define
\[
F_n(t) = \frac{1}{n} \sum_{i=1}^n \mathbbm{1}_{(-\infty, t]}(X_i) = \frac{|\{i: X_i\leq t\}|}{n}.
\]
By SLLN $F_n(t)\xrightarrow{a.s.} F(t)$. By Glivenko-Cantelli Theorem the convergence is uniform.

Q: Brownian bridge
A: Define $(B_t)_{0\leq t \leq 1}$ by $B_t=W_t-tW_1$ or equivalently a BM conditioned on $B_1=1$. Then $B_t$ satisfies $B_0=B_1=0$ and for $s\leq t$
\[
B_t \sim N(0, t(1-t)), \qquad \cov(B_s, B_t) \sim s(1-t).
\]

Q: Donsker-Kolmogorov-Doob Theorem
A: Write $\mathcal{G}_F = B_{F(t)}$ for the Gaussian process with covariance
\[
\cov(\mathcal{G}_F(s), \mathcal{G}_F(t)) = F(s)(1-F(t)).
\]
This emerges in the limit
\[
\sqrt{n}(F_n-F) \xrightarrow{d} \mathcal{G}_F.
\]

Q: Kolmogorov-Smirnov Theorem
A: As $n\to\infty$
\[
\sqrt{n} \|F_n-F\|_\infty \xrightarrow{d} \|\mathcal{G}_F\|_\infty  = \|B\|_\infty.
\]
This can be used for uniform CIs and HTs of $F$.

Q: Normal sample distribution
A: $\bar X \sim N(\mu, \sigma^2/n)$ and $S_{XX}/\sigma^2 \sim \chi_{n - 1}^2$ are MI
\begin{proof}
Write $\mathbf{X} \sim N_n(\boldsymbol\mu, \sigma^2I)$, where $\boldsymbol\mu = (\mu, \mu, \cdots, \mu)^T$.
Pick $A\in O_n(\R)$ with the first row all $1/\sqrt{n}$, e.g.
  \[
    A =
    \begin{pmatrix}
      \frac{1}{\sqrt{n}} & \frac{1}{\sqrt{n}} & \frac{1}{\sqrt{n}} & \frac{1}{\sqrt{n}} & \cdots & \frac{1}{\sqrt{n}}\\
      \frac{1}{\sqrt{2\times 1}} & \frac{-1}{\sqrt{2\times 1}} & 0 & 0 & \cdots & 0\\
      \frac{1}{\sqrt{3\times 2}} & \frac{1}{\sqrt{3\times 2}} & \frac{-2}{\sqrt{3\times 2}} & 0 \cdots & 0\\
      \vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
      \frac{1}{\sqrt{n(n - 1)}} & \frac{1}{\sqrt{n(n - 1)}} & \frac{1}{\sqrt{n(n - 1)}} & \frac{1}{\sqrt{n(n - 1)}} & \cdots & \frac{-(n - 1)}{\sqrt{n(n - 1)}} \\
    \end{pmatrix}
  \]
works since the rows are ON. Now define $\mathbf{Y} = A\mathbf{X}$ such that
 \[
    Y_1 = \frac{1}{\sqrt{n}}\sum_{i = 1}^n X_i = \sqrt{n} \bar X.
  \]
and
  \[
    \mathbf{Y} \sim N_n(A\boldsymbol\mu, A\sigma^2IA^T) = N_n(A\boldsymbol\mu, \sigma^2 I)
  \]
where
  \[
    A\boldsymbol\mu = (\sqrt{n}\mu, 0, \cdots, 0)^T.
  \]
  So $Y_1 \sim N(\sqrt{n}\mu, \sigma^2)$, $Y_i \sim N(0, \sigma^2)$ for $i \geq 2$ and the $Y_i$ are MI where
\[
    Y_2^2 + \cdots + Y_n^2 = \mathbf{Y}^T\mathbf{Y} - Y_1^2= \mathbf{X}^T\mathbf{X} - n\bar X^2=S_{XX}.
\]
\end{proof}

Q: Hodges Estimator
A: Let $\hat\theta_n$ be the MLE for $N(\theta, 1)$. Define the \emph{Hodges} estimator
\[
\tilde{\theta}_n = \hat{\theta}_n \mathbbm{1}_{|\hat\theta_n|\geq n^{-1/4}}.
\]
This gains \emph{superefficiency} at $\theta=0$:
\[
\sqrt{n}(\tilde\theta_n - \theta) \xrightarrow{d} \begin{cases} N(0,1) \quad &\theta\neq 0 \\
N(0,0) \quad &\theta = 0.
\end{cases}
\]
However, the maximal quadratic risk is $O(n^{-1/2})$ instead of $n^{-1}$.
\begin{proof}
For $\theta=0$, $P(\hat\theta_n\geq n^{-1/4}) = \P(|Z|\geq n^{1/4})\to 0$. For $\theta\neq 0$, since $\hat\theta_n\sim N(\theta, \frac{1}{n})$, by Chebyshev
\begin{align*}
\P(|\hat\theta_n|\leq n^{-1/4}) &= \P( \sqrt{n}(-n^{-1/4}-\theta)\leq Z\leq \sqrt{n}(n^{-1/4}-\theta))\\
&\leq 2n^{-1}(|\theta|-n^{-1/4})^{-2}.
\end{align*}
Hence we have the $L^1$ convergence
\begin{align*}
\E[\sqrt{n}\hat\theta_n \mathbbm{1}_{|\hat\theta_n|\leq n^{-1/4}}] &\leq n^{1/4} \P(|\hat\theta_n|\leq n^{-1/4}) \\
&\leq 2n^{-3/4}(|\theta|-n^{-1/4})^{-2}.
\end{align*}
The quadratic risk for $\theta=\frac{1}{2}n^{-1/4}$ is
\begin{align*}
\E_\theta[(\tilde\theta_n-\theta)^2] &\geq \theta^2\P(|\hat\theta_n| \leq n^{-1/4}) \\
&\geq \theta^2(1-n^{1/2}\E_\theta[\hat\theta_n^2]) \\
& =\theta^2(1-n^{1/2}(n^{-1}+\theta^2)) \\
& =\tfrac{1}{4}n^{-1/2}(1-n^{-1/2}-\tfrac{1}{4})).
\end{align*}
\end{proof}

Q: Maximal risk of James-Stein
A: For $X\sim N(\theta, I_p)$, $\forall\varepsilon>0$
\begin{align*}
\E_\theta[\|X\|^{-2}] &\leq \varepsilon + \E_\theta[\|X\|^{-2}\mathbbm{1}_{\|X\|^{-2} > \varepsilon}] \\
&= \varepsilon + \int_{\|x\|^2<1/\varepsilon} \frac{1}{\|x\|^2} f_\theta(x) \; \d x \\
&\leq \varepsilon + \underbrace{\sup_{\|x\|^2<1/\varepsilon} f_\theta(x)}_{\to 0} \underbrace{\int_{\|x\|^2<1/\varepsilon} \frac{1}{\|x\|^2}  \; \d x}_{<\infty\text{ for }p\geq 3} \\
&\to \varepsilon \text{ as } \|\theta\|\to\infty.
\end{align*}

Q: Integral of $\|x\|^{-\alpha}$
A: Let $S_n(R)=C_nR^{n-1}$ be the surface area of the $n$-sphere. Then
\begin{align*}
\int_{\|x\|\leq 1} \frac{1}{\|x\|^\alpha} \; \d x &= \int_0^1 S_n(r)r^{-\alpha} \; \d r \\
&= C_n\int_0^1 r^{n-1-\alpha} \\
&= \frac{C_n}{n-\alpha}
\end{align*}
is finite provided $\alpha < n$.

Q: James-Stein exact risk
A: Let $X\sim N(\theta, I_p)$ for $p\geq 3$. Then the JS risk is
\[
\E \|X- \frac{p-2}{\|X\|^2}X-\theta \|^2 = \E\|X-\theta\|^2 + (p-2)^2\E\|X\|^{-2} - 2(p-2)\E[(X-\theta)\cdot X \|X\|^{-2} ]
\]
where the cross-term is
\[
\sum_{i=1}^p \E\left[\frac{X_i(X_i-\theta)}{X_i^2+\sum_{j\neq i}X_j^2}\right].
\]
Now $g(x):=\frac{x}{x^2+\sum_{j\neq i}X_j^2}$ is a.s finite with integrable derivative
\[
g'(x) = \frac{(x^2+\sum_{j\neq i}X_j^2) - 2x^2}{(x^2 + \sum_{j\neq i}X_j^2)^2}=O(x^{-2}).
\]
By Stein's Lemma the cross-term becomes
\begin{align*}
\sum_{i=1}^p \E[\E[g'(X_i) | \{X_j: j \neq i\}]] &= \sum_{i=1}^p \E\left[\frac{(\sum X_j^2) - 2X_i^2}{(\sum X_j^2)^2}\right] \\
&= \E \left[ \frac{p(\sum X_j^2) - 2\sum X_j^2}{(\sum X_j^2)^2}\right] \\
&= (p-2) \E[\|X\|^{-2}].
\end{align*}
Substituting in the JS risk gives
\[
p - (p-2)^2\E\left[ \|X\|^{-2}\right] .
\]

Q: Validity of the bootsrap
A: Let $C_n^b=\{ \mu: \sqrt{n}|\mu - \bar{X}_n| \leq R_n^b\}$ such that $\P(\bar{X}_n^b\in C_n^b)=1-\alpha$. Write
\[
\sqrt{n}|\bar{X}_n^b - \bar{X}_n| \sim F_n, \qquad \sqrt{n}|\mu - \bar{X}_n| \sim G_n, \qquad |N(0,\sigma^2)|\sim \Phi_\sigma.
\]
Then $R_n^b\to \Phi_\sigma^{-1}(1-\alpha)$ and $\P(\mu\in C_n^b)\to 1-\alpha$.
\begin{proof}
By uniform convergence
\[
\Phi_\sigma(R_n^b) = F_n(R_n^b) - \left[F_n(R_n^b) - \Phi_\sigma(R_n^b) \right] \to 1-\alpha.
\]
and similarly
\[
G_n(R_n^b) = \left[G_n(R_n^b) - \Phi_\sigma(R_n^b) \right] + \left[\Phi_\sigma(R_n^b) - F_n(R_n^b)\right] + F_n(R_n^b) \to 1-\alpha.
\]
\end{proof}

Q: Generating normal random variables
A: $|Z|$ has density
\[
f(x)=\frac{2}{\sqrt{2\pi}}\exp\left(-\frac{x^2}{2} \right)
\]
and compare to density $h(x)=e^{-x}$ with inverse survival $-\log(y)$. Then
\[
\frac{f(x)}{h(x)} = \frac{2}{\sqrt{2\pi}}\exp\left(x-\frac{x^2}{2} \right) = \frac{2}{\sqrt{2\pi}}\exp\left(-\frac{1}{2}(x-1)^2 + \frac{1}{2} \right)
\]
so $M= \sqrt{\frac{2e}{\pi}}$.
Hence we can generate $U\sim U[0,1]$, then $Y=-\log(U)\sim\Exp(1)$ and $|Z|=Y$ if $U\leq \exp\left(-\frac{1}{2}(x-1)^2\right)$. Finally $Z=\sgn(2U-1)|Z|$.