Probability

1. If $A$ and $B$ are events, and $X$ and $Y$ are the indicator random variables refined with respect to them. What is the minimal $\sigma$-algebra for the random variable $X+Y$.
2. Argue that $\mathbb{E} \exp(jX)$ is well-defined for all random variables $X$, where $j=\sqrt{-1}$ the imaginary number.
3. Find an example of random variables X, Y, Z taking values in {0,1} such that they are pairwise independent but not jointly independent, ie, P(X=1,Y=1,Z=1) ≠ P(X=1) P(Y=1) P(Z=1).
4. For general measures $\mu$ and an $A\in\mathcal{F}$, we say “$A\, \mu$-a.e.” to mean that $\mu(A^c) = 0$. If the measure mu is unambiguous, we just say “A a.e.” (a.e. is short for almost everywhere) example: $f\geq g$ a.e. means $\mu((f-g)^{-1}((-\infty,0)))=0$. similarly we say $f=g$ a.e. if $f\geq g$ a.e. and $f\leq g$ a.e. For probability measures, we say a.s. (almost surely) instead of a.e. Find an example of a sequence of non-negtive random variables for which $lim_{n\to\infty} P(X_n > 0) =0$, but $\lim \sup X_n = 1$ a.s.
5. For a non-negative random variable, show that $\mathbb{E}X = \int_{\mathbb{R}} P(X>t)\mathrm{d}t$. [Hint: First try showing it assuming that $P$ is absolutely continuous, i.e., it has a density function $f_X$.]
6. From One Thousand Exercises in Probability:
   • Chapter 1: 1.2.2, 1.2.4, 1.3.1, 1.4.1, 1.4.3, 1.4.5, 1.5.1, 1.5.4, 1.5.5, 1.8.3, 1.8.4, 1.8.8, 1.8.9
   • Chapter 2: 2.1.1, 2.1.2, 2.1.4, 2.1.5, 2.7.3
   • Chapter 3: 3.2.1, 3.3.1, 3.3.6, 3.7.1, 3.7.4, 3.7.6
   • Chapter 4: 4.5.8
   • Chapter 7: 7.2.6, 7.5.1, 7.9.4, 7.11.2, 7.11.3, 7.11.8, 7.11.25

Statistics

(Many of these questions are from Samory Kpotufe’s Statistical Theory and Methods course)

1. Suppose that $\Theta \subset \mathbb{R}^d$. Let $\mathcal{P}_{\theta} = {P_\theta, \theta \in \Theta}$, where $P_\theta$ has pdf (w.r.t. Lebesgue measure) \(f_\theta(x) = h(x)\, l(\theta)\, e^{\alpha(\theta)^T T(x)}, \, x \in \mathbb{R}^p,\) where $T(x)$ is a vector-valued function. Show that $\mathcal{P}_\theta$ has an equivalent representation $\mathcal{P}_\alpha = {P_\alpha,\ \alpha \in A}$ for some set $A \subset \mathbb{R}^r$, where \(f_\alpha(x) = h'(x)\, l'(\alpha)\, e^{\alpha^T T(x)}, \, x \in \mathbb{R}^p.\)
2. For $n$ i.i.d. random variables $\{x_i\}_1^n$ which are distributed as:
   • Gaussian, i.e. $x_i \sim N(\mu, \sigma^2)$, where $\mu \in \mathbb{R}$ and $\sigma^2 > 0$, find the MLE for $\theta = (\mu, \sigma^2)$. Is it biased?
   • Exponential, i.e. $p_\lambda(x_i)= \lambda\exp^{-\lambda x_i}$, where $\lambda \in \mathbb{R}_{+}$, find the MLE for $\lambda$. Is it biased?
   • Poisson, i.e. $p_\lambda(x_i)=\frac{\lambda^x_i}{x_i!}\exp(-\lambda)$, when $x_i \in \mathbb{N} \cup 0$ and $0$ otherwise, find the MLE for $\lambda$. Is it biased?
3. Let $x = \{x_i\}_1^n$ be i.i.d. drawn from $\text{Uniform}([0,\theta])$, where $\theta > 0$. Show that
   • the MLE for $\theta$ is $\widehat{\theta} = \max_i x_i$
   • $\widehat{\theta}$ has a density w.r.t. Lebesgue measure.
   • $\widehat{\theta}$ is biased.
4. Suppose that $X, Y$ are random variables with densities $f_X(x)$ and $f_Y(y)$, and their joint density is $f_{X,Y}(x,y)$. The entropy of $X$ is defined to be $H(X) = E\log(1/f_X(X))$; and the conditional entropy of $X$ conditioning on $Y$ is \(H(X\mid Y) = E \log\!\left(\frac{f_Y(Y)}{f_{X,Y}(X,Y)}\right).\)
   • Consider a family of probability measures $\mathcal{P} = {P_\theta}$. Assume that all of them have density $f_\theta$ which has the same support. Suppose that $x = \{x_i\}_1^n \sim P_{\theta_0}^n$, $P_{\theta_0}^n \in \mathcal{P}$. What is the expected log-likelihood $E \log L(\theta; x)$, in terms of entropic measures (entropy and relative entropy)?
   • Show that conditioning does not increase entropy, i.e. $H(X \mid Y) \le H(X)$. Assume that the above expectations are finite.
   • Show that the mututal information $I(X;Y):=H(X)-H(X \mid Y)$ is $0$ iff $X$ and $Y$ are independent.
5. Suppose we have data $(x_i, y_i)_{i=1}^n$, where $x_i \in \mathbb{R}^p$. The Ridge estimator of the linear model $Y = X^T\beta + \epsilon$ is defined to be the minimizer of the following problem: \(\min_{\beta \in \mathbb{R}^p} \sum_{i=1}^n (y_i - x_i^T \beta)^2 + \lambda \|\beta\|^2, \, \lambda \ge 0,\) where $||\cdot||$ is the Euclidean norm in $\mathbb{R}^p$.
   • Show that there exists some $\lambda \ge 0$ such that the above minimization problem has a unique minimizer.
   • Derive the minimizer.
6. Reduce the general polynomial model \(Y = \text{poly}(X) + \epsilon, \, E\epsilon = 0,\) to the linear model and derive a solution. Note that $\text{poly}(x)$, $x \in \mathbb{R}^p$, is any polynomial of some degree $k$ ($k \ge 1$), i.e. \(\text{poly}(x) = \sum_{\substack{l \in \mathbb{N}^p:\\ |l| \le k}} w_l\, x^l.\)