Skip to main content

Expected Values

Expectation is a location measure which give the location of the center of a random variable.

for discrete random variables, the expectation is E(x)=∑x∈RxP(X=x)\mathbb{E}(x) = \sum_{x\in \R}xP(X=x). Let XX, YY be discrete random variables. Let a,b∈Ra,b\in \R be constants. Let Z=aX+bYZ = aX + bY, then E(Z)=aE(X)+bE(Y)\mathbb{E}(Z) = a\mathbb{E}(X) + b\mathbb{E}(Y)

for continuous random variables, the expectation is E(x)=∫x∈Rxf(x)\mathbb{E}(x) = \int_{x\in \R}xf(x). Let XX, YY be discrete random variables. Let a,b∈Ra,b\in \R be constants. Let Z=aX+bYZ = aX + bY, then E(Z)=aE(X)+bE(Y)\mathbb{E}(Z) = a\mathbb{E}(X) + b\mathbb{E}(Y)

If X,YX, Y are independent random variables, then E(XY)=E(X)E(Y)\mathbb{E}(XY) = \mathbb{E}(X)\mathbb{E}(Y)

If there exists some other function gg, then the expectation of gg is:

  • E(g(x))={∫−∞∞g(x)f(x)dxx is continuous∑x∈Rg(x)P(X=x)x is discrete\mathbb{E}(g(x)) = \begin{cases}\int_{-\infty}^{\infty} g(x)f(x)dx & x \text{ is continuous} \\ \sum_{x\in \R} g(x)P(X = x) & x \text{ is discrete}\end{cases}
  • More complicated, for the joint situation, there exists such function h(x,y):R2→Rh(x,y) : \R^2 \to \R E(h(x,y))={∫−∞∞∫−∞∞h(x,y)f(x,y)dxdyx is continuous∑y∈R∑x∈Rh(x,y)P(X=x,Y=y)x is discrete\mathbb{E}(h(x,y)) = \begin{cases}\int_{-\infty}^{\infty} \int_{-\infty}^{\infty} h(x,y)f(x,y)dxdy & x \text{ is continuous} \\ \sum_{y\in \R}\sum_{x\in \R} h(x, y)P(X = x, Y = y) & x \text{ is discrete}\end{cases}

E(X)=∫0∞P(X>t)dt−∫−∞0P(X<t)dt\mathbb{E}(X) = \int_{0}^{\infty} P(X > t)dt - \int_{-\infty}^{0} P(X < t)dt where both probability <∞< \infty

Linearity property of expectations only work finite where : ∑i=1nE[Xi]=E[∑i=1nXi]\sum_{i=1}^n E[X_i] = E[\sum_{i=1}^n X_i]. When it approaches to infinite we have conditions to make it work.

Variance​

Variance present the spread deviation of a random variable.

The population Variance is Var(X)=E[(xi−μx)2]={∑x∈R(x−μx)2P(X=x)∫x∈R(x−μx)2f(x)Var(X) = \mathbb{E}[(x_i-\mu_x)^2] = \begin{cases}\sum_{x\in \R}(x-\mu_x)^2P(X=x) \\ \int_{x\in \R}(x-\mu_x)^2f(x) \end{cases} where μx=E[X]\mu_x = \mathbb{E}[X]

  • Var[X]≥0Var[X] \ge 0
  • Var[aX+B]=a2V[X],a,B∈RVar[aX +B] = a^2V[X], a, B\in \R
  • Var[X]=E[X2]−(E[X])2Var[X] = E[X^2] - (E[X])^2
  • Var[X]≤E[X2]Var[X] \le E[X^2]
  • Define standard deviation SD(X)=Var[X]SD(X) = \sqrt{Var[X]}

The sample Variance is s2=∑i=12(x−μ)2n−1s^2 = \frac{\sum_{i=1}^2(x-\mu)^2}{n-1}

Covariance​

Let XX and YY be two random variables. Their covariance is defined by Cov(X,Y)=E[X−μX][Y−μY]=E[XY]−μXE[Y]−μYE[X]+μYμX=E[XY]−E[X]E[Y]Cov(X, Y) = \mathbb{E}[X - \mu_X][Y - \mu_Y] = \mathbb{E}[XY] - \mu_X\mathbb{E}[Y]- \mu_Y\mathbb{E}[X] + \mu_Y\mu_X = \mathbb{E}[XY]-\mathbb{E}[X]\mathbb{E}[Y]

  • if X⊥Y,Cov(X,Y)=0X\perp Y, Cov(X,Y) =0
  • Cov(X,X)=Var[X]Cov(X,X) = Var[X]
  • Cov(X,Y)=Cov(Y,X)Cov(X,Y) = Cov(Y,X)
  • Cov(aX,bY)=abCov(X,Y)Cov(aX,bY) = abCov(X,Y)
  • Cov(aX+dW,bY+cZ)=abCov(X,Y)+acCov(X,Z)+dbCov(W,Y)+dc(W,Z)Cov(aX + dW, bY + cZ) = abCov(X, Y) + acCov(X,Z) + dbCov(W,Y) + dc(W, Z)

E[Y12−2Y1Y2+Y22]=E[Y12]−2E[Y1]E[Y2]+E[Y22]=Var[Y1]+(E[Y1])2+Var[Y2]+(E[Y2])2−2E[Y1]E[Y2]=Var[Y1]+Var[Y2]+(E[Y1]−E[Y2])2=2Var[Y]\mathbb{E}[Y_1^2 - 2Y_1Y_2 + Y_2^2] = \mathbb{E}[Y_1^2] - 2\mathbb{E}[Y_1]\mathbb{E}[Y_2] + \mathbb{E}[Y_2^2]=Var[Y_1] + (\mathbb{E}[Y_1])^2 + Var[Y_2] + (\mathbb{E}[Y_2])^2 - 2\mathbb{E}[Y_1]\mathbb{E}[Y_2]\\ = Var[Y_1]+ Var[Y_2] + (\mathbb{E}[Y_1] - \mathbb{E}[Y_2])^2 = 2Var[Y] where two random variables from the same sample space, they should have the same variance and mean value.

Correlation: we can calculate the correlation between two random variables by the formula Corr(X,Y)=Cov(X,Y)Var[X]Var[Y]Corr(X, Y) = \frac{Cov(X,Y)}{\sqrt{Var[X]Var[Y]}}

  • −1≤Corr(X,Y)≤1-1 \le Corr(X,Y) \le 1

Generating Functions​

Let XX be discrete random variable, we define the Probability Generating Function(PGF) rX(t)=E(tX);t∈Rr_X(t) = \mathbb{E}(t^X); t\in \R. If given some PGF, e.g. rX(0)=P(X=0)r_X(0) = P(X=0), then we can have

  • rX′(0)=P(X=1)r'_X(0) = P(X=1)
  • rX′′(0)=2P(X=2)r''_X(0) = 2P(X=2)
  • rXk(0)=k!P(X=k)r^k_X(0) = k! P(X=k)

For any random variable XX, we have Moment Generating Function(MGF) which defined as the raw moment given that the expectation exists: mX(s)=E[esX];s∈Rm_X(s) = \mathbb{E}[e^{sX}]; s\in \R. Assume mX(s)<∞,s∈(−s0,s0),s0>0m_X(s) < \infty, s\in (-s_0, s_0),s_0>0 which leads to:

  • mX(0)=1m_X(0) = 1
  • mX′(0)=E[X]m'_X(0) = \mathbb{E}[X]
  • mX′′(0)=E[X2]m''_X(0) = \mathbb{E}[X^2]
  • mXk(0)=E[Xk]m^k_X(0) = \mathbb{E}[X^k]

Let X,YX,Y be two independent random variables with MGFs mX(t),mY(t)m_X(t), m_Y(t), then the MGF of X+Y=mX+Y(t)=E[eX+Yt]=E[eYt]E[eXt]=mX(t)mY(t)X+Y = m_{X+Y}(t) = \mathbb{E}[e^{X+Y}t] = \mathbb{E}[e^{Yt}]\mathbb{E}[e^{Xt}] = m_X(t)m_Y(t). Result is the same as more independent random variables, and we can use the existed MGF to determine the distribution of those random variables.

  • for s∈(−s0,s0),s0>0s\in (-s_0, s_0), s_0>0, mY(s)=mX(s)  ⟹  X,Ym_Y(s) = m_X(s) \implies X,Y have the same distribution.

Let X,YX,Y be independent random variables. Let Z=aX+bY,mZ(t)=E[eZt]=E[eaXt]E[ebYt]=mX(at)mY(bt)Z = aX + bY, m_Z(t) = \mathbb{E}[e^{Zt}] = \mathbb{E}[e^{aXt}]\mathbb{E}[e^{bYt}] = m_X(at)m_Y(bt)

  • if XX is constant, then we have mZ(t)=E[eat]mY(bt)=eatmY(bt)m_Z(t) = \mathbb{E}[e^{at}] m_Y(bt) = e^{at}m_Y(bt)

Conditional Expectation​

Let XX be discrete random variable, let AA be some event with P(A)>0P(A) > 0. The conditional expectation of XX given by AA is equal to E[X=x∣A]=∑x∈RxP(X=x,A)P(A)\mathbb{E}[X = x|A] = \sum_{x\in \R}x\frac{P(X = x, A)}{P(A)}. For condition expectation given by a joint random variable YY, if YY also discrete and P(Y=y)>0P(Y = y) > 0, then E[X=x∣Y=y]=∑x∈RxP(X=x,Y=y)P(Y=y)\mathbb{E}[X = x| Y = y] = \sum_{x\in \R} x\frac{P(X = x, Y = y)}{P(Y = y)}

Let XX and YY be joint absolutely continuous random variable with joint density function fX,Y(x,y)f_{X,Y}(x,y), the conditional expectation of XX given Y=yY = y is equal to E[X=x∣Y=y]=∫x∈Rxf(x∣Y=y)dx=∫x∈RxfX,Y(x,y)fY(y)dx\mathbb{E}[X = x|Y=y] = \int_{x\in \R} xf(x| Y= y)dx = \int_{x\in \R} x\frac{f_{X,Y}(x,y)}{f_Y(y)}dx

Let X,Y,WX, Y, W be random variables, let Z=aX+bYZ = aX + bY, let AA be an event, then:

  • E[Z∣A]=E[aX+bY∣A]=aE[X∣A]+bE[Y∣A]\mathbb{E}[Z|A] = \mathbb{E}[aX + bY | A] = a\mathbb{E}[X|A] + b\mathbb{E}[Y|A]
  • E[Z∣W]=aE[X∣W]+bE[Y∣W]\mathbb{E}[Z| W] = a\mathbb{E}[X|W] + b\mathbb{E}[Y|W]

We also define total expectation for joint random variable X,YX,Y as EY[E[X∣Y]]=E[X]\mathbb{E}_Y[\mathbb{E}[X|Y]] = \mathbb{E}[X] and EX[E[Y∣X]]=E[Y]\mathbb{E}_X[\mathbb{E}[Y|X]] = \mathbb{E}[Y]

Conditional Variance​

Let X,YX, Y be random variables with finite expectations, the conditional variance is defined as Var[X∣Y]=E[X2∣Y]−(E[X∣Y])2Var[X|Y] = \mathbb{E}[X^2|Y] - (\mathbb{E}[X|Y])^2

  • Var[X∣Y]Var[X|Y] also can be a random variable

Total Variance: Var[X]=Var[E[X∣Y]]+E[Var[X∣Y]]Var[X] = Var[\mathbb{E}[X|Y]] + \mathbb{E}[Var[X|Y]]