Chapter 1

Supporting notes

Scott Alexander

Page 11-12

Events are subsets of the sample space, i.e. (Subscript[E, 1]),(Subscript[E, 2])⊆Ω.

Page 13

Events are elements of the sigma-algebra, i.e. (Subscript[E, 1]),(Subscript[E, 2])∈F.

Page 15

Note that
P((Subscript[E, 1])|(Subscript[E, 2])) = ((P((Subscript[E, 1])⋂(Subscript[E, 2])))/(P((Subscript[E, 2])))) = ((P((Subscript[E, 1])⋂(Subscript[E, 2])))/(P((Subscript[E, 2]))))((P((Subscript[E, 1])))/(P((Subscript[E, 1]))))
  = ((P((Subscript[E, 1])⋂(Subscript[E, 2])))/(P((Subscript[E, 1]))))((P((Subscript[E, 1])))/(P((Subscript[E, 2]))))
  = P((Subscript[E, 2])|(Subscript[E, 1]))((P((Subscript[E, 1])))/(P((Subscript[E, 2]))))
which is Bayes' Theorem.

Page 17

Source: page 93 "Probability and Statistics for Engineers" (9th Edition) by Walpole, Myers, Myers, Ye.

Page 20

(Subscript[F, X])(b):=P(X≤b)
  ≡P({X≤b})
  =P((⋃_((Subscript[x, i])≤b)){X=(Subscript[x, i])})
  =(∑_((Subscript[x, i])≤b))P({X=(Subscript[x, i])})    (the events {X=(Subscript[x, i])} are mutually exclusive)
  ≡(∑_((Subscript[x, i])≤b))P(X=(Subscript[x, i]))
  =(∑_((Subscript[x, i])≤b))(Subscript[p, X])((Subscript[x, i]))

Frame 21

Binomial distribution

n=30; p=1/3; xMax=n;

DiscretePlot[PDF[BinomialDistribution[n,p],x],{x,0,xMax},Filling->None,AxesLabel->{"x","f(x)"},
PlotLabel->"PMF of X~B("<>ToString[n,InputForm]<>","<>ToString[p,InputForm]<>")"]

p1=Plot[CDF[BinomialDistribution[n,p],x],{x,0,xMax}];
x=Table[i,{i,0,xMax}];
p2=ListPlot[Transpose[{x,CDF[BinomialDistribution[n,p],x]}]];
Show[p1,p2,PlotRange->All,AxesLabel->{"x","F(x)"},
PlotLabel->"CDF of X~B("<>ToString[n,InputForm]<>","<>ToString[p,InputForm]<>")"]

Binomial distribution mean and variance

Clear[n,p,x];
PDF[BinomialDistribution[n,p],x]    (* probability mass function *)

Clear[n,p];
Assuming[{n>0, 0<p<1},(∑_(x=0)^n)x Binomial[n,x](p^x)(((1-p))^(n-x))]    (* mean or expected value *)

Clear[n,p,x];
Assuming[{n>0, 0<p<1},(∑_(x=0)^n)(x^2)Binomial[n,x](p^x)(((1-p))^(n-x))-((((∑_(x=0)^n)x Binomial[n,x](p^x)(((1-p))^(n-x))))^2)]//Expand    (* variance *)

Clear[n,p,x];
Expectation[x,x\[Distributed]BinomialDistribution[n,p]]    (* mean or expected value *)

Clear[n,p,x];
Expectation[(x^2),x\[Distributed]BinomialDistribution[n,p]]-(((Expectation[x,x\[Distributed]BinomialDistribution[n,p]]))^2)    (* variance *)

Binomial distribution mean (traditional syntax)

Clear[n,p,x];
Assuming[{n>0, 0<p<1},Sum[x Binomial[n,x]p^x(1-p)^(n-x),{x,0,n}]]    (* mean or expected value *)

Frame 22

Geometric distribution

p=1/3; xMax=20;

DiscretePlot[PDF[GeometricDistribution[p],x],{x,0,xMax},Filling->None,
AxesLabel->{"x","f(x)"},PlotLabel->"PMF of X~Geo("<>ToString[p,InputForm]<>")"]

p1=Plot[CDF[GeometricDistribution[p],x],{x,0,xMax}];
x=Table[i,{i,0,xMax}];
p2=ListPlot[Transpose[{x,CDF[GeometricDistribution[p],x]}]];
Show[p1,p2,PlotRange->All,AxesLabel->{"x","F(x)"},AxesOrigin->{0,0},
PlotLabel->"CDF of X~Geo("<>ToString[p,InputForm]<>")"]

Relationship with exponential distribution

Let    X~Geo(1/n).
Then    (lim_(n->∞)) (X/n)~Exp(1).

Frame 23

Poisson distribution

λ=10; xMax=30;

DiscretePlot[PDF[PoissonDistribution[λ],x],{x,0,xMax},Filling->None,
AxesLabel->{"x","f(x)"},PlotLabel->"PMF of X~Pois("<>ToString[λ,InputForm]<>")"]

p1=Plot[CDF[PoissonDistribution[λ],x],{x,0,xMax}];
x=Table[i,{i,0,xMax}];
p2=ListPlot[Transpose[{x,CDF[PoissonDistribution[λ],x]}]];
Show[p1,p2,PlotRange->All,AxesLabel->{"x","F(x)"},
PlotLabel->"CDF of X~Pois("<>ToString[λ,InputForm]<>")"]

Frame 24

For continuous RV it is always the case that P(X=a)=0. To see this note that
P(X=a) = P(a≤X≤a) = (∫_a^a)(Subscript[f, X])(x)dx = 0.

Frame 25

Uniform distribution

a=2;  b=5; xMin=a-1; xMax=b+2;

p1=Plot[PDF[UniformDistribution[{a,b}],x],{x,xMin,xMax}];
p2=ListPlot[Transpose[{{a,b},PDF[UniformDistribution[{a,b}],{a,b}]}]];
Show[p1,p2,PlotRange->All,AxesLabel->{"x","(Subscript[f, X])(x)"},
PlotLabel->"PDF of X~U("<>ToString[a,InputForm]<>","<>ToString[b,InputForm]<>")"]

Plot[CDF[UniformDistribution[{a,b}],x],{x,xMin,xMax},PlotRange->All,AxesLabel->{"x","(Subscript[F, X])(x)"},
PlotLabel->"CDF of X~U("<>ToString[a,InputForm]<>","<>ToString[b,InputForm]<>")"]

Frame 26

Exponential distribution

λ=2; xMax=5;

Plot[PDF[ExponentialDistribution[λ],x],{x,0,xMax},PlotRange->All,AxesLabel->{"x","(Subscript[f, X])(x)"},
PlotLabel->"PDF of X~Exp("<>ToString[λ,InputForm]<>")"]

Plot[CDF[ExponentialDistribution[λ],x],{x,0,xMax},PlotRange->All,AxesLabel->{"x","(Subscript[F, X])(x)"},
PlotLabel->"CDF of X~Exp("<>ToString[λ,InputForm]<>")"]

Frame 27

Normal distribution

μ=1;  σ=2; xMin=μ-4σ; xMax=μ+4σ;

Plot[PDF[NormalDistribution[μ,σ],x],{x,xMin,xMax},PlotRange->All,AxesLabel->{"x","(Subscript[f, X])(x)"},PlotLabel->"PDF of X~N("<>ToString[μ,InputForm]<>","<>ToString[σ,InputForm]<>")"]

Plot[CDF[NormalDistribution[μ,σ],x],{x,xMin,xMax},PlotRange->All,AxesLabel->{"x","(Subscript[F, X])(x)"},
PlotLabel->"CDF of X~N("<>ToString[μ,InputForm]<>","<>ToString[σ,InputForm]<>")"]

Normal approximation of binomial distribution

n=40;
p=0.5;
p1=DiscretePlot[PDF[BinomialDistribution[n,p],x],{x,0,n,1},PlotRange->All];
p2=Plot[PDF[NormalDistribution[n p,(Sqrt[n p(1-p)])],x],{x,0,n},PlotRange->All,PlotStyle->{Red}];
Show[p1,p2,PlotLabel->"PMF of X~B("<>ToString[n,InputForm]<>","<>ToString[p,InputForm]<>") with normal PDF"]

(* CDF with no continuity correction *)
n=40;
p=0.5;
p1=Plot[CDF[BinomialDistribution[n,p],x],{x,0,n},PlotRange->All];

x=Table[i,{i,0,n}];
p2=ListPlot[Transpose[{x,CDF[BinomialDistribution[n,p],x]}]];
p3=Plot[CDF[NormalDistribution[n p,(Sqrt[n p(1-p)])],x],{x,0,n},PlotRange->All,PlotStyle->{Red}];

Show[p1,p2,p3,PlotRange->All,AxesLabel->{"x","F(x)"},
PlotLabel->"CDF of X~B("<>ToString[n,InputForm]<>","<>ToString[p,InputForm]<>") with normal CDF (without continuity correction)"]

(* CDF with continuity correction *)
n=40;
p=0.5;
p1=Plot[CDF[BinomialDistribution[n,p],x],{x,0,n},PlotRange->All];

x=Table[i,{i,0,n}];
p2=ListPlot[Transpose[{x,CDF[BinomialDistribution[n,p],x]}]];
p3=Plot[CDF[NormalDistribution[n p,(Sqrt[n p(1-p)])],x+0.5],{x,0,n},PlotRange->All,PlotStyle->{Red}];

Show[p1,p2,p3,PlotRange->All,AxesLabel->{"x","F(x)"},
PlotLabel->"CDF of X~B("<>ToString[n,InputForm]<>","<>ToString[p,InputForm]<>") with normal CDF (with continuity correction)"]

Normal distribution mean and variance

Clear[μ,σ,x];
PDF[NormalDistribution[μ,σ],x]    (* probability density function *)

Clear[μ,σ,x];
Assuming[{μ∈Reals,σ>0},(∫_(-∞)^∞)x ((E^(-((((x-μ))^2)/(2 (σ^2)))))/((Sqrt[2 π]) σ))\[DifferentialD]x]    (* mean or expected value *)

Clear[μ,σ,x];
Assuming[{μ∈Reals,σ>0},Integrate[(x^2) ((E^(-((((x-μ))^2)/(2 (σ^2)))))/((Sqrt[2 π]) σ)),{x,-∞,∞}]-(((Integrate[x ((E^(-((((x-μ))^2)/(2 (σ^2)))))/((Sqrt[2 π]) σ)),{x,-∞,∞}]))^2)]    (* variance *)

Clear[μ,σ,x];
Expectation[x,x\[Distributed]NormalDistribution[μ,σ]]    (* mean or expected value *)

Clear[μ,σ,x];
Expectation[(x^2),x\[Distributed]NormalDistribution[μ,σ]]-(((Expectation[x,x\[Distributed]NormalDistribution[μ,σ]]))^2)    (* variance *)

Normal distribution mean (traditional syntax)

Clear[μ,σ,x];
Assuming[{μ∈Reals,σ>0},Integrate[x*Exp[-(x-μ)^2/(2*σ^2)]/(Sqrt[2*Pi]*σ),{x,-Infinity,Infinity}]]    (* mean or expected value *)

Marginal density from joint density

Clear[μ,Σ,x,y];
μ={0,0};
Σ={{1,0},{0,1}};
f[x_,y_]:=PDF[MultinormalDistribution[μ,Σ],{x,y}];
f[x,y]    (* joint density function*)
Integrate[f[x,y],{y,-∞,∞}](* marginal density function of X *)

Plots of 2D normal distribution for different ρ

ParallelTable[
Plot3D[
PDF[MultinormalDistribution[{0,0},{{1,ρ},{ρ,1}}],{x,y}],
{x,-4,4},{y,-4,4},PlotRange->All],
{ρ,-0.5,0.5,0.25}]

Frame 28

Gamma distribution

α=2;β=3;

Plot[PDF[GammaDistribution[α,β],x],{x,0,15},PlotRange->All,AxesLabel->{"x","(Subscript[f, X])(x)"},PlotLabel->"PDF of X~Gamma("<>ToString[α,InputForm]<>","<>ToString[β,InputForm]<>")"]

Plot[CDF[GammaDistribution[α,β],x],{x,0,15},PlotRange->All,AxesLabel->{"x","(Subscript[F, X])(x)"},
PlotLabel->"CDF of X~Gamma("<>ToString[α,InputForm]<>","<>ToString[β,InputForm]<>")"]

Frame 30

Proof of linearity expectation operator

E[(Subscript[c, 1])(Subscript[X, 1])+(Subscript[c, 2])(Subscript[X, 2])] = (∫_Ω)((Subscript[c, 1])(Subscript[X, 1])(ω)+(Subscript[c, 2])(Subscript[X, 2])(ω))dP(ω)
 = (∫_Ω)(Subscript[c, 1])(Subscript[X, 1])(ω)dP(ω)+(∫_Ω)(Subscript[c, 2])(Subscript[X, 2])(ω)dP(ω)    (linearity of integration)
 = (Subscript[c, 1])(∫_Ω)(Subscript[X, 1])(ω)dP(ω)+(Subscript[c, 2])(∫_Ω)(Subscript[X, 2])(ω)dP(ω)
 = (Subscript[c, 1])E[(Subscript[X, 1])]+(Subscript[c, 2])E[(Subscript[X, 2])]

Example - exponential distribution

Integrating against density

𝔼[X] = (∫_(-∞)^∞)x (Subscript[f, X])(x)dx  = (∫_0^∞)x λ (e^(-λ x))dx    ((Subscript[f, X])(x)=0 for x<0)
  = (∫_0^∞)x (d/(dx))(-(e^(-λ x)))dx = -(Subscript[[x (e^(-λ x))], 0]^∞)+(∫_0^∞)(e^(-λ x))(d/(dx))xdx    (integration by parts)
  = -(Subscript[[x (e^(-λ x))], 0]^∞)-(Subscript[(1/λ)[(e^(-λ x))], 0]^∞)
  = -(lim_(x → ∞))(x/(e^(λ x))) +0-(lim_(x → ∞))(1/λ)(e^(-λ x))+(1/λ)
  = 0+0-0+(1/λ)    (application of L'Hospital's Rule omitted)
  = (1/λ)

Using gamma function

Recalling
Γ(z) = (∫_0^∞)(x^(z-1))(e^(-x))dx
we have
𝔼[X] = (∫_0^∞)x λ (e^(-λ x))dx = (1/λ)(∫_0^∞)y (e^(-y))dy    (y=λ x)
  = (1/λ)Γ(2) = (1/λ)1! = (1/λ).

Using Mathematica function “Integrate”

Clear[x];
f[x_]:=PDF[ExponentialDistribution[λ],x];
f[x]
Assuming[λ>0,Integrate[x f[x],{x,0,∞}]]    (* note the space between x and f[x] *)

Using Mathematica function “Expectation”

Clear[x];
Expectation[x,x\[Distributed]ExponentialDistribution[λ]]

Clear[x];
Expectation[x,Distributed[x,ExponentialDistribution[λ]]]

Frame 31

Moments from MGF

(Subscript[M, X])(u):=𝔼[(e^(u X))]

((d^k)/(d(u^k)))(Subscript[M, X])(u) = ((d^k)/(d(u^k)))𝔼[(e^(u X))]
  =  𝔼[((d^k)/(d(u^k)))(e^(u X))]    (assuming one may interchange expectation and differentiation operations)
  = 𝔼[(X^k)(e^(u X))]
so
𝔼[(X^k)] = 𝔼[(X^k)(e^(u X))](Subscript[|, u=0]) = ((d^k)/(d(u^k)))(Subscript[M, X])(u)(Subscript[|, u=0]).

Example - exponential distribution

Let X~Exp(λ).

(Subscript[M, X])(u) = 𝔼[(e^(u X))] = (∫_0^∞)(e^(u x))λ (e^(-λ x))dx = (∫_0^∞)λ (e^((u - λ)x))dx
  = (Subscript[[(λ/(u-λ))(e^((u - λ)x))], 0]^∞) = (λ/(u-λ))(lim_(x->∞))(e^((u - λ)x)) - (λ/(u-λ))
  = {({{(λ/(λ-u)), u<λ}, {undefined, u≥λ}}).

Then
((d^k)/(d(u^k)))(Subscript[M, X])(u) = ((d^k)/(d(u^k)))(λ/(λ-u)) = λ((d^k)/(d(u^k)))(((λ-u))^(-1))
  = λ (((λ-u))^(-1-k))2×3×⋯×k = ((λ k!)/(((λ-u))^(1+k))).

So
𝔼[(X^k)] = ((d^k)/(d(u^k)))(Subscript[M, X])(u)(Subscript[|, u=0]) = ((λ k!)/(((λ))^(1+k))) = ((k!)/(λ^k)).

(* Finding MGF *)
Clear[u,λ];
Assuming[{u<λ},(∫_0^∞)(E^(u x))λ (E^(-λ x))\[DifferentialD]x]

Clear[u,λ];
MomentGeneratingFunction[ExponentialDistribution[λ],u]

(* 3rd moment *)
Clear[u,λ];
D[(λ/(λ-u)),{u,3}]/.u->0

Clear[u,λ];
Moment[ExponentialDistribution[λ],3]

Example - normal distribution

Let (Subscript[X, k])~N((Subscript[μ, k]),(Subscript[σ, k]^2)) be independent iid RVs for k∈{1,…,n}.

The MGF of the sum (∑_(k=1)^n)(Subscript[X, k]) is
𝔼[exp(u(∑_(k=1)^n)(Subscript[X, k]))] = 𝔼[(∏_(k=1)^n)exp(u (Subscript[X, k]))] = (∏_(k=1)^n)𝔼[exp(u (Subscript[X, k]))]    (independence)
  = (∏_(k=1)^n)exp(u (Subscript[μ, k]) + (1/2)(Subscript[σ, k]^2)(u^2))    (MGF of N(μ,(σ^2)) RV - see Chapter 2 Lecture Notes)
  = exp(u(∑_(k=1)^n)(Subscript[μ, k]) + (1/2)(u^2)(∑_(k=1)^n)(Subscript[σ, k]^2))
which is MGF of N((∑_(k=1)^n)(Subscript[μ, k]),(∑_(k=1)^n)(Subscript[σ, k]^2)) RV.
So
(∑_(k=1)^n)(Subscript[X, k])~N((∑_(k=1)^n)(Subscript[μ, k]),(∑_(k=1)^n)(Subscript[σ, k]^2)).

Frame 32

Moments from CF

(Subscript[φ, X])(u):=𝔼[(e^(i u X))]

((d^k)/(d(u^k)))(Subscript[φ, X])(u) = ((d^k)/(d(u^k)))𝔼[(e^(i u X))]
  =  𝔼[((d^k)/(d(u^k)))(e^(i u X))]    (assuming one may interchange expectation and differentiation operations)
  = (i^k)𝔼[(X^k)(e^(u X))]
so
𝔼[(X^k)] =  𝔼[(X^k)(e^(i u X))](Subscript[|, u=0]) = (i^(-k))((d^k)/(d(u^k)))(Subscript[φ, X])(u)(Subscript[|, u=0]).

Example - normal distribution

(* Finding CF *)
Clear[μ,σ,x];
Assuming[{μ∈Reals,σ>0},(∫_(-∞)^∞)(E^(I u x))(1/((Sqrt[2π])σ))(E^(-((((x-μ))^2)/(2(σ^2)))))\[DifferentialD]x]//FullSimplify

Clear[μ,σ,x];
CharacteristicFunction[NormalDistribution[μ,σ],u]

(* 5th moment*)
(I^(-5))D[(E^(I u μ-(((u^2) (σ^2))/2))),{u,5}]/.u->0//FullSimplify

Moment[NormalDistribution[μ,σ],5]//FullSimplify

IMPORTANT.
The function Mathematica NormalDistribution expects σ as a parameter, not (σ^2).

This is in contrast to the function MultinormalDistribution, which takes the covariance matrix Σ.

See help file for details.

Frame 40

Plot of moment generating function for normal distribution

μ={1,1};  Σ={{1,0.5},{0.5,1}};
mgf[u1_,u2_]:=MomentGeneratingFunction[MultinormalDistribution[μ,Σ],{u1,u2}];
Plot3D[mgf[u1,u2],{u1,2,3},{u2,2,3},PlotRange->All]

Plot of characteristic function for normal distribution

μ={1,1};  Σ={{1,0.5},{0.5,1}};
cf[u1_,u2_]:=CharacteristicFunction[MultinormalDistribution[μ,Σ],{u1,u2}];
Plot3D[Re[cf[u1,u2]],{u1,2,3},{u2,2,3},PlotRange->All]
Plot3D[Im[cf[u1,u2]],{u1,2,3},{u2,2,3},PlotRange->All]

Frame 41

Let X,Y be RV on (ℝ^n).

cov(X,Y) = 𝔼[(X-𝔼[X])(((Y-𝔼[Y]))^T)] = 𝔼[X (Y^T)-X ((𝔼[Y])^T)-𝔼[X](Y^T)+𝔼[X]((𝔼[Y])^T)]
  = 𝔼[X (Y^T)]-𝔼[X ((𝔼[Y])^T)]-𝔼[𝔼[X](Y^T)]+𝔼[𝔼[X]((𝔼[Y])^T)]
  = 𝔼[X (Y^T)]-𝔼[X]((𝔼[Y])^T)-𝔼[X]𝔼[(Y^T)]+𝔼[X]((𝔼[Y])^T)
  = 𝔼[X (Y^T)]-𝔼[X]((𝔼[Y])^T)    (another common way to express covariance)
  = 𝔼[(({{(Subscript[X, 1])}, {(Subscript[X, 2])}, {⋮}, {(Subscript[X, n])}}))(({{(Subscript[Y, 1]), (Subscript[Y, 2]), ⋯, (Subscript[Y, n])}}))] - (({{𝔼[(Subscript[X, 1])]}, {𝔼[(Subscript[X, 2])]}, {⋮}, {𝔼[(Subscript[X, n])]}}))(({{𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[Y, n])]}}))
  = 𝔼[(({{(Subscript[X, 1])(Subscript[Y, 1]), (Subscript[X, 1])(Subscript[Y, 2]), ⋯, (Subscript[X, 1])(Subscript[Y, n])}, {(Subscript[X, 2])(Subscript[Y, 1]), (Subscript[X, 2])(Subscript[Y, 2]), ⋯, (Subscript[X, 2])(Subscript[Y, n])}, {⋮, ⋮, ⋱, ⋮}, {(Subscript[X, n])(Subscript[Y, 1]), (Subscript[X, n])(Subscript[Y, 2]), ⋯, (Subscript[X, n])(Subscript[Y, n])}}))]- (({{𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, n])]}, {𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, n])]}, {⋮, ⋮, ⋱, ⋮}, {𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, n])]}}))
  = (({{𝔼[(Subscript[X, 1])(Subscript[Y, 1])], 𝔼[(Subscript[X, 1])(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 1])(Subscript[Y, n])]}, {𝔼[(Subscript[X, 2])(Subscript[Y, 1])], 𝔼[(Subscript[X, 2])(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 2])(Subscript[Y, n])]}, {⋮, ⋮, ⋱, ⋮}, {𝔼[(Subscript[X, n])(Subscript[Y, 1])], 𝔼[(Subscript[X, n])(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, n])(Subscript[Y, n])]}})) - (({{𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, n])]}, {𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, n])]}, {⋮, ⋮, ⋱, ⋮}, {𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, n])]}}))
  = (({{𝔼[(Subscript[X, 1])(Subscript[Y, 1])]-𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 1])(Subscript[Y, 2])]-𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 1])(Subscript[Y, n])]-𝔼[(Subscript[X, 1])]𝔼[(Subscript[Y, n])]}, {𝔼[(Subscript[X, 2])(Subscript[Y, 1])]-𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, 2])(Subscript[Y, 2])]-𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, 2])(Subscript[Y, n])]-𝔼[(Subscript[X, 2])]𝔼[(Subscript[Y, n])]}, {⋮, ⋮, ⋱, ⋮}, {𝔼[(Subscript[X, n])(Subscript[Y, 1])]-𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 1])], 𝔼[(Subscript[X, n])(Subscript[Y, 2])]-𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, 2])], ⋯, 𝔼[(Subscript[X, n])(Subscript[Y, n])]-𝔼[(Subscript[X, n])]𝔼[(Subscript[Y, n])]}}))
  = (({{cov((Subscript[X, 1]),(Subscript[Y, 1])), cov((Subscript[X, 1]),(Subscript[Y, 2])), ⋯, cov((Subscript[X, 1]),(Subscript[Y, n]))}, {cov((Subscript[X, 2]),(Subscript[Y, 1])), cov((Subscript[X, 2]),(Subscript[Y, 2])), ⋯, cov((Subscript[X, 2]),(Subscript[Y, n]))}, {⋮, ⋮, ⋱, ⋮}, {cov((Subscript[X, n]),(Subscript[Y, 1])), cov((Subscript[X, n]),(Subscript[Y, 2])), ⋯, cov((Subscript[X, n]),(Subscript[Y, n]))}}))

Frame 43

Let    a,b∈(ℝ^n).

Recall the dot product
a·b = (a^T)b.

Let    A∈(ℝ^(n×n)).

Recall the property
a·A b = (a^T) A b = ((((A^T)a))^T)b = ((A^T)a)·b.

Frame 47

p1=Plot[2,{x,0,2}];
p2=Plot[x,{x,0,2},Filling->Top];
Show[p1,p2,AxesLabel->{"x","y"},PlotRange->{{0,2},{0,2}}]

Example thanks to Juri Hinz.

Let X,Y be RVs with joint density function
(Subscript[f, X,Y])(x,y) = ({{{, ({{(1/2),, 0≤x≤y≤2}, {0,, otherwise}})}}).

The shaded region is where (Subscript[f, X,Y])=(1/2).
Outside the shaded region is where (Subscript[f, X,Y])=0.

MARGINAL DENSITIES
The marginal density of X is
(Subscript[f, X])(x) = (∫_(-∞)^∞)(Subscript[f, X,Y])(x,y)dy
which for 0≤x≤2 is
  = (∫_x^2)(1/2)dy = (Subscript[[(1/2)y], x]^2) = 1-(1/2)x
and is zero elswhere, i.e.
(Subscript[f, X])(x)  = ({{{, ({{1-(1/2)x,, 0≤x≤2}, {0,, otherwise}})}}).

The marginal density of Y is
(Subscript[f, Y])(y) = (∫_(-∞)^∞)(Subscript[f, X,Y])(x,y)dx
which for 0≤y≤2 is
  = (∫_0^y)(1/2)dx = (Subscript[[(1/2)x], 0]^y) = (1/2)y
and is zero elswhere, i.e.
(Subscript[f, Y])(y)  = ({{{, ({{(1/2)y,, 0≤y≤2}, {0,, otherwise}})}}).

Note that X,Y are not independent as
(Subscript[f, X,Y])(x,y) ≠ (Subscript[f, X])(x)(Subscript[f, Y])(y).


CONDITIONAL DENSITIES
The conditional density X given Y=y is
(Subscript[f, X|Y])(x,y) = (((Subscript[f, X,Y])(x,y))/((Subscript[f, Y])(y)))
which for 0≤x≤y≤2 is
(Subscript[f, X|Y])(x,y) = ((1/2)/((1/2)y)) = (1/y)
and 0 elsewhere, i.e.
(Subscript[f, X|Y])(x,y) = ({{{, ({{(1/y),, 0≤x≤y≤2}, {0,, otherwise}})}}).


The conditional density Y given X=x is
(Subscript[f, Y|X])(x,y) = (((Subscript[f, X,Y])(x,y))/((Subscript[f, X])(x)))
which for 0≤x≤y≤2 is
(Subscript[f, Y|X])(x,y) = ((1/2)/(1-(1/2)x)) = (1/(2-x))
and 0 elsewhere, i.e.
(Subscript[f, Y|X])(x,y) = ({{{, ({{(1/(2-x)),, 0≤x≤y≤2}, {0,, otherwise}})}}).


CONDITIONAL EXPECTATIONS
The conditional expectation of X given Y=y
𝔼[X|Y=y] = (∫_(-∞)^∞)x (Subscript[f, X|Y])(x,y)dx
  = (∫_0^y)x (1/y)dx = (1/y)(∫_0^y)x dx
  = (Subscript[(1/y)[(1/2)(x^2)], 0]^y) = (1/y)((1/2)(y^2)-0)
  = (y/2)    for    0≤y≤2.

The conditional expectation of Y given X=x
𝔼[Y|X=x] = (∫_(-∞)^∞)y (Subscript[f, Y|X])(x,y)dy
  = (∫_x^2)y (1/(2-x))dy = (1/(2-x))(∫_x^2)y dy
  = (Subscript[(1/(2-x))[(1/2)(y^2)], x]^2) = (1/(2-x))((1/2)*4-(1/2)(x^2))
  = ((2+x)/2)    for    0≤x≤2.

Clear[x];
(1/(2-x))((1/2)*4-(1/2)(x^2))//FullSimplify

Frame 52

Note that
𝔼[𝔼[X|Y]|Y] = 𝔼[𝔼[X|Y]*1|Y]
  = 𝔼[X|Y]*𝔼[1|Y]    (Proposition 1 Part 3 - 𝔼[X|Y] is function of Y)
  = 𝔼[X|Y].