Examples of Moment Generating Functions
filed under : #math #probability #analysis
This post is a follow up on the last post on moment generating functions (mgfs), where we showed that if the mgf of a probability measure $\mu$ on $\mathbb{R}$ (also known as a distribution) exists on some neighborhood of $0$, then the moments of $\mu$ uniquely determine $\mu$. Let's quickly review the relevant definitions and theorems from the last post.
Definition.
For a probability measure $\mu$ on $\mathbb{R}$ the $n$th
moment is defined as
$\int_{-\infty}^{\infty}x^n\mathrm{d}\mu(x)$ with the $n$th
absolute moment of $\mu$ being
$\int_{-\infty}^{\infty}|x|^n\mathrm{d}\mu(x)$. Here $n$ is any
non-negative integer (The $0$th moment is just $1$ for every $\mu$).
Definition.
The moment generating function or mgf of a
probability measure $\mu$ on $\mathbb{R}$ is the function
And we had the following proposition that relates mgfs back to
moments:
$$
M_{\mu}(t) = \int_{-\infty}^{\infty}e^{tx}\mathrm{d}\mu(x),
$$
if the integral is well defined for all $t$ in an open interval
$(-R, R)$. If not, then we say that the mgf of $\mu$ doesn't
exists.
Proposition.
Suppose the moment generating function exists for the probability
measure $\mu$, then for all integers $n > 0$, we have that the $n$th
moment of $\mu$ is finite and is given by
And then we proved the following theorem:
$$
\at{\libn[]{t}}{t=0}M_{\mu}(t).
$$
Theorem.
Let $\mu$ be a probability measure on $\mathbb{R}$, and suppose the
mgf $M_{\mu}(t)$ exists in some interval $(-R, R)$. Then the
moments $a_n \defeq \int_{-\infty}^{\infty}x^n\mathrm{d}\mu(x)$ are
finite and uniquely determine $\mu$.
Example Moment Generating Functions
In this section we will compute the moment generating functions of common continuous distributions. We will describe our distributions (i.e. probability measures) with probability density functions, or pdfs. Recall that a function $f \colon \mathbb{R} \to \mathbb{R}$ is a pdf if $f(x) \ge 0$ for all $x \in \mathbb{R}$ and $\int_{-\infty}^{\infty}f(x)\mathrm{d}x = 1$. The Radon-Nikodym stated below gives the conditions for when a probability measure $\mu$ has a pdf.Theorem (Radon-Nikodym).
Suppose that $\mu$ is a probability measure on $\mathbb{R}$ that is
absolutely
continuous. Then there exists a function $f \colon \mathbb{R}
\to [0, \infty)$ such that for any $A \in \mathcal{B}$, we have
$$
\mu(A) = \int_Af(x)\mathrm{d}x.
$$
The Normal Distribution
In the post on the gamma function we showed that$$
\int_{-\infty}^{\infty}e^{-t^2}\mathrm{d}t = \sqrt{\pi}.
$$
Therefore for $a$ and $b$ real numbers with $a > 0$, we see
$$
\int_{-\infty}^{\infty}e^{-(at+b)^2}\mathrm{d}t =
\frac{1}{a}\int_{-\infty}^{\infty}e^{-u^2}\mathrm{d}u =
\frac{\sqrt{\pi}}{a}.
$$
Thus,
$$
f(t; \mu, \sigma^2) \defeq
\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(t-\mu)^2}{2\sigma^2}}
$$
is a pdf with parameters $-\infty < \mu < \infty$ and $0 <
\sigma < \infty$. The associated distribution is called the
\emph{Gaussian (normal) distribution} and is denoted as
$\mathcal{N}(\theta, \sigma^2)$. For the Gaussian distribution
$\mathcal{N}(\mu, \sigma^2)$ we see that the mgf is
$$
\label{eq:norm1}\tag{1}
M(t) = \frac{1}{\sqrt{2\pi}\sigma}
\int_{-\infty}^{\infty}e^{-\frac{(x-\mu)^2}{2\sigma^2}}e^{tx}\mathrm{d}x.
$$
By letting $\tau = \mu + \sigma^2t$ we see
$$
(x-\mu)^2 - 2\sigma^2tx
= x^2 - 2(\mu + \sigma^2t)x + \mu^2
= (x-\tau)^2 + \mu^2 - \tau^2.
$$
Substituting this result into \eqref{eq:norm1}, we get
$$
M(t) = \frac{1}{\sqrt{2\pi}\sigma}
e^{\frac{\tau^2-\mu^2}{2\sigma^2}}
\int_{-\infty}^{\infty}e^{-\frac{(x-\tau)^2}{2\sigma^2}}\mathrm{d}x
= e^{\frac{\tau^2-\mu^2}{2\sigma^2}}.
$$
Replacing $\tau$ with $\mu + \sigma^2t$ we find the mgf of the
normal distribution is
$$
\label{eq:norm}\tag{2}
M(t) = e^{{\mu}t}e^{\frac{1}{2}\sigma^2t^2}.
$$
Gamma Distribution
By $u$-substitution we see$$
\frac{1}{\Gamma(\alpha)\beta^{\alpha}}
\int_{0}^{\infty}x^{\alpha - 1}e^{-\frac{x}{\beta}}\mathrm{d}x
= 1.
$$
Thus we define the gamma pdf as
$$
f(x; \alpha, \beta) = \left\{\begin{array}{lr}
\frac{x^{\alpha-1}e^{-\frac{x}{\beta}}}{\Gamma(\alpha)\beta^{\alpha}} & x > 0 \\
0 & x \le 0 \\
\end{array}
\right.,
$$
with $\alpha, \beta > 0$ being parameters. For $\gammad(\alpha,
\beta)$ the moment generating function is
$$
\label{eq:gamma1}\tag{3}
M(t) = \frac{1}{\Gamma(\alpha)\beta^{\alpha}}
\int_0^{\infty}e^{tx}x^{\alpha-1}e^{-\frac{x}{\beta}}\mathrm{d}x
= \frac{1}{\Gamma(\alpha)\beta^{\alpha}}
\int_0^{\infty}x^{\alpha-1}e^{-\left(\frac{1-\beta{t}}{\beta}\right)x}\mathrm{d}x.
$$
Derived from $u$-substitution for any $\tau > 0$ we have
$$
\int_{0}^{\infty}x^{\alpha - 1}e^{-\tau{x}}\mathrm{d}x
= \tau^{-\alpha}\Gamma(\alpha),
$$
so \eqref{eq:gamma1} gives us
$$
\label{eq:gamma}\tag{4}
M(t) = (1 - \beta{t})^{-\alpha}
$$
for $t < 1/\beta$. Furthermore, we see that the first moment
(expected value) of $\gammad(\alpha, \beta)$ is
$$
\at{\lib[]{t}}{t=0}(1 - \beta{t})^{-\alpha} = \alpha\beta.
$$
Sums of Independent Random Variables
Now that we know some example moment generating functions, lets show their true power. Let's recall some background on random variables first.
Suppose $(S, \mathcal{A},\mu)$ is a probability space. We call any measurable function $X\colon S \to \mathbb{R}$ a random variable, where $\mathbb{R}$ is equipped with the Borel sets $\mathcal{B}$. The random variable $X$ induces a probability measure $\mu_X$ on $\mathbb{R}$ of the form$$
\mu_X(A) = \mu(X^{-1}(A))
$$
where $A \in \mathcal{B}$. The measure $\mu_X$ is call the
pushforward measure. Suppose $\nu$ is a probability measure
on $\mathbb{R}$. We say $X$ has distribution $\nu$ or $X \sim \nu$ if
$\mu_X = \nu$. We call a set $\{X_i\}_{i\in I}$ of random variables
identically distributed if each random variable $X_i$ for $i
\in I$ has the distribution $\mu$. Recall that a collection of
measurable sets $\{A_{i}\}_{i \in I}$ of $S$ is called
independent if for any finite subcollection $\{A_{i_1}, \dots
A_{i_n}\}$ we have
$$
\mu(A_{i_1} \cap \cdots \cap A_{i_n})
= \mu(A_{i_1}) \cdots \mu(A_{i_n}).
$$
Furthermore we say a collection of random variables $\{X_i\}_{i \in
I}$ over $S$ is independent if for any Borel measurable set
$B \subseteq \mathbb{R}$ we have that the collection
$\{X_{i}^{-1}(B)\}_{i \in I}$ is independent. Lastly we say a
collection $\{X_i\}_{i \in I}$ of random variables over $S$ iid
if its both independent and identically distributed.
Proposition.
Let $X_1, \dots, X_n$ be a set of independent random variables over
some probability space $(S, \mathcal{A}, \mu)$, with mgfs
$M_{X_1}(t), \dots, M_{X_n}(t)$. Let $Y = X_1 + \cdots + X_n$ Then
the mgf of $Y$ exists and equals
$$
M_Y(t) = \prod_{i=1}^nM_{X_i}(t).
$$
Proof.
We compute
$$
M_Y(t) = \mathbb{E}(e^{t(x_1 + \cdots + x_n)})
= \mathbb{E}\left(\prod_{i=0}^{n}e^{tx_i}\right)
=\prod_{i=0}^{n}\mathbb{E}(e^{tx_i})
= \prod_{i=1}^nM_{X_i}(t).
$$
Proposition.
Let $X_1, \dots, X_n$ be independent random variables over the
probability space $(S, \mathcal{A}, \mu)$ such that $X_i \sim
\mathcal{N}(\mu_i, \sigma_i^2)$. Let $Y = X_1 + \cdots + X_n$, then
$Y \sim \mathcal{N}(\mu_1 + \cdots + \mu_n, \sigma_1^2 + \cdots +
\sigma_n^2)$.
Proof.
We compute the mgf of $Y$ using \eqref{eq:norm} to see
$$
M_Y(t) = \prod_{i=1}^nM_{X_i}(t)
= \prod_{i=1}^n\left(e^{\mu_i{t}}e^{\frac{1}{2}\sigma_i^2t^2}\right)
= e^{(\mu_1 + \cdots + \mu_n)t}e^{\frac{1}{2}(\sigma_1^2 + \cdots + \sigma_n^2)t^2}.
$$
Uniqueness of mgfs give us $Y \sim \mathcal{N}(\mu_1 + \cdots +
\mu_n, \sigma_1^2 + \cdots + \sigma_n^2)$ and the proof is
complete.
Proposition.
Let $X_1, \dots, X_n$ be independent random variables over the
probability space $(S, \mathcal{A}, \mu)$ such that $X_i \sim
\gammad(\alpha_i, \beta)$ (Note that $\beta$ is fixed). Let $Y =
X_1 + \cdots + X_n$, then $Y \sim \gammad(\alpha_1 + \cdots +
\alpha_n, \beta)$.
Proof.
We compute the mgf of $Y$ using \eqref{eq:gamma} to see
$$
M_Y(t) = \prod_{i=1}^nM_{X_i}(t)
= \prod_{i=1}^n(1 - \beta{t})^{-\alpha_i}
= (1 - \beta{t})^{-(\alpha_1 + \cdots + \alpha_n)}.
$$
Uniqueness of mgfs give us $Y \sim \gammad(\alpha_1 + \cdots +
\alpha_n, \beta))$ and the proof is complete.
The $\chi^2$ Distribution
We define the $\chi^2$ distribution with $p$ degrees of freedom as $\gammad(p/2, 2)$. Before we get to our first examples, let's make an observation. Let $Y = X^2$ for some random variable with pdf $f_X$. For $y > 0$, we see for cdf $F_Y$ of $Y$$$
F_Y(y) = \P_{Y}(X^2 \le y)
= \mu_Y(\{t \in \mathbb{R} \vert -\sqrt{y} \le t \le \sqrt{y}\})
= F_X(\sqrt{y}) - F_X(-\sqrt{y}),
$$
and so
$$
\label{eq:sq}\tag{5}
f_Y(y) = \lib[]{y}F_Y(y)
= \frac{1}{2\sqrt{y}}\left(f_X(\sqrt{y}) +
f_X(-\sqrt{y})\right).
$$
Let's apply \eqref{eq:sq} to the standard normal distribution.
Example.
Take $X$ to be the standard Gaussian distribution, so that $X$ has
pdf
Thus we get the following important result.
$$
f_X(x) = \frac{1}{\sqrt{2\pi}}e^{-\frac{x^2}{2}}
$$
for all $x \in \mathbb{R}$. Let $Y = X^2$, and now we see from
\eqref{eq:sq},
$$
f_Y(y) = \frac{1}{\sqrt{2\pi{y}}}e^{-\frac{y}{2}}
= \frac{y^{-\frac{1}{2}}}{\Gamma(\frac{1}{2})\sqrt{2}}e^{-\frac{y}{2}}
$$
for $y > 0$. Thus $Y \sim \gammad(1/2, 2)$ and so $Y$ has a
$\chi^2$ distribution with one degree of freedom.
Proposition.
Let $X_1, \dots, X_n$ be a set of independent random variables with
the standard Gaussian distribution. Let $Y$ be
$$
Y = X_1^2 + \cdots + X_n^2
$$
then $Y$ has a $\chi^2$ distribution with $n$ degrees of freedom.
Counterexamples
Our first counterexample will be a distribution that does not have a mgf.
Example.
Let $\mu$ be a probability measure on $\mathbb{R}$ with pdf
$$
f(x; \theta) = \frac{1}{\pi}\frac{1}{1 + (x - \theta)^2}
$$
for $x, \theta \in \mathbb{R}$. Here $\mu$ has the so called
Cauchy distribution.
Thus the $n$th absolute moment is
$$
\frac{1}{\pi}\int_{-\infty}^{\infty}
\frac{|x|^n}{1 + (x - \theta)^2}\mathrm{d}x = \infty
$$
for all positive integers $n$. Moreover, the odd number moments are
undefined as the improper integer is of the undefined form $\infty -
\infty$.
Next we will show two distinct probability measures sharing the same moments.
Example.
Suppose $\mu_1$ and $\mu_2$ are probability measures on $\mathbb{R}$
with pdfs
$$
f_1(x) = \frac{1}{\sqrt{2\pi}x}e^{-\frac{(\log{x})^2}{2}},
$$
and
$$
f_2(x) = f_1(x)(1 + \sin{(2\pi\log{x})}),
$$
respectfully with $x > 0$ and $f_1(x) = f_2(x) = 0$ for $x \le 0$.
Note that $\mu_1$ has the lognormal distribution. We see for
fixed non-negative integer $n$
$$
\label{eq:ex1}\tag{6}
\int_{0}^{\infty}x^nf_1(x)\mathrm{d}x
= \frac{1}{\sqrt{2\pi}}\int_0^{\infty}x^ne^{-\frac{(\log{x})^2}{2}}\frac{\mathrm{d}x}{x}.
$$
Taking the $u$-substitution $u = \log{x}$, we see \eqref{eq:ex1}
transforms to
$$
\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}e^{nu}e^{-\frac{u^2}{2}}\mathrm{d}u
= \frac{e^{\frac{n^2}{2}}}{\sqrt{2\pi}}\int_{-\infty}^{\infty}e^{-\frac{(u-n)^2}{2}}\mathrm{d}u
= e^{\frac{n^2}{2}}.
$$
Taking $n = 0$, we verify that $f_1$ is indeed a pdf. Furthermore, we
see that the $n$th moment of $\mu_1$ is $ e^{\frac{n^2}{2}}$.
For $f_2(x)$ we see
$$
\label{eq:ex2}\tag{7}
\int_0^{\infty}f_2(x)\mathrm{d}x
= e^{\frac{n^2}{2}} + \int_{0}^{\infty}f_1(x)(\sin(2\pi\log{x}))\mathrm{d}x.
$$
Taking the $u$-substitution $u = u = \log{x} - n$ and simplifying, we
see the integral in \eqref{eq:ex2} is equal to
$$
\label{eq:ex3}\tag{8}
\frac{1}{\sqrt{2\pi}}\int_{-\infty}^{\infty}
e^{-\frac{(u+n)^2}{2}}e^{n(u+n)}\sin(2\pi(u+n))\mathrm{d}u
=\frac{1}{\sqrt{2\pi}}
\int_{-\infty}^{\infty}e^{\frac{n^2 - u^2}{2}}\sin(2\pi{u})\mathrm{d}u.
$$
Now the integrand in \eqref{eq:ex3} is an odd function and it's
trivial to see that the improper integral is well-defined (i.e. not in the
$\infty - \infty$ form), and so \eqref{eq:ex3} is evaluated as $0$.
Therefore by \eqref{eq:ex2}, we have $\mu_2$ has identical moments
as $\mu_1$.