Chapter 2

Supporting notes

Scott Alexander

Frame 4

The autocovariance
Q = cov(X,X) = 𝔼[(X-𝔼[X])(((X-𝔼[X]))^T)]
  = 𝔼[(m+L ξ-m)(((m+L ξ-m))^T)]
  = 𝔼[L ξ(((L ξ))^T)] = 𝔼[L ξ (ξ^T)(L^T)] =L 𝔼[ξ (ξ^T)](L^T)
  = L cov(ξ,ξ)(L^T) = L (Subscript[I, k]) (L^T)    ((Subscript[I, k]) k-dimension identity matrix)
  = L (L^T).

So X~N(m,Q=L (L^T)).

Frame 5

Let X~N(m,(σ^2)) be a scalar RV.

Then
(Subscript[φ, X])(u) = 𝔼[(e^(i u X))] = (e^(i m u-(1/2)(σ^2)(u^2)))
and
(Subscript[M, X])(u) = 𝔼[(e^(u X))] = (e^(m u+(1/2)(σ^2)(μ^2))).

Positive semi-definite (non-negative-definite) matrix

Consider the (positive semi-definite) covariance matrix
Q = (({{var(X), cov(X,Y)}, {cov(Y,X), var(Y)}})) = (({{(Subscript[σ, X]^2), ρ (Subscript[σ, X])(Subscript[σ, Y])}, {ρ (Subscript[σ, X])(Subscript[σ, Y]), (Subscript[σ, Y]^2)}}))
with    (Subscript[σ, X]),(Subscript[σ, Y])>0    and    -1≤ρ≤1.

Then for z∈(ℝ^2) the "quadratic form"
z·Q z = (({{(Subscript[z, 1]), (Subscript[z, 2])}}))(({{(Subscript[σ, X]^2), ρ (Subscript[σ, X])(Subscript[σ, Y])}, {ρ (Subscript[σ, X])(Subscript[σ, Y]), (Subscript[σ, Y]^2)}})) (({{(Subscript[z, 1])}, {(Subscript[z, 2])}}))
  = (Subscript[σ, X]^2)(Subscript[z, 1]^2)+(Subscript[σ, Y]^2)(Subscript[z, 2]^2)+2 ρ (Subscript[σ, X])(Subscript[σ, Y])(Subscript[z, 1])(Subscript[z, 2]) ≥ 0.

(({{(Subscript[z, 1]), (Subscript[z, 2])}})).(({{(Subscript[σ, X]^2), ρ (Subscript[σ, X])(Subscript[σ, Y])}, {ρ (Subscript[σ, X])(Subscript[σ, Y]), (Subscript[σ, Y]^2)}})) .(({{(Subscript[z, 1])}, {(Subscript[z, 2])}}))//Expand

Frame 6

Identity matrix

The identity matrix
(Subscript[I, n]) = (({{1, 0, ⋯, 0}, {0, 1, ⋯, 0}, {⋮, ⋮, ⋱, ⋮}, {0, 0, ⋯, 1}}))
which is called the identity matrix because if
A =  (({{(Subscript[a, 11]), (Subscript[a, 12]), (Subscript[a, 13]), ⋯, (Subscript[a, 1n])}, {(Subscript[a, 21]), (Subscript[a, 22]), (Subscript[a, 32]), ⋯, (Subscript[a, 2n])}, {(Subscript[a, 31]), (Subscript[a, 32]), (Subscript[a, 33]), ⋯, (Subscript[a, 3n])}, {⋮, ⋮, ⋮, ⋱, ⋮}, {(Subscript[a, n1]), (Subscript[a, n2]), (Subscript[a, n3]), ⋯, (Subscript[a, nn])}}))
then
(Subscript[I, n])A =(({{1, 0, 0, ⋯, 0}, {0, 1, 0, ⋯, 0}, {0, 0, 1, ⋯, 0}, {⋮, ⋮, ⋮, ⋱, ⋮}, {0, 0, 0, ⋯, 1}}))(({{(Subscript[a, 11]), (Subscript[a, 12]), (Subscript[a, 13]), ⋯, (Subscript[a, 1n])}, {(Subscript[a, 21]), (Subscript[a, 22]), (Subscript[a, 32]), ⋯, (Subscript[a, 2n])}, {(Subscript[a, 31]), (Subscript[a, 32]), (Subscript[a, 33]), ⋯, (Subscript[a, 3n])}, {⋮, ⋮, ⋮, ⋱, ⋮}, {(Subscript[a, n1]), (Subscript[a, n2]), (Subscript[a, n3]), ⋯, (Subscript[a, nn])}})) = (({{(Subscript[a, 11]), (Subscript[a, 12]), (Subscript[a, 13]), ⋯, (Subscript[a, 1n])}, {(Subscript[a, 21]), (Subscript[a, 22]), (Subscript[a, 32]), ⋯, (Subscript[a, 2n])}, {(Subscript[a, 31]), (Subscript[a, 32]), (Subscript[a, 33]), ⋯, (Subscript[a, 3n])}, {⋮, ⋮, ⋮, ⋱, ⋮}, {(Subscript[a, n1]), (Subscript[a, n2]), (Subscript[a, n3]), ⋯, (Subscript[a, nn])}}))  = A.

Independence implies zero covariance

For i≠j
cov((Subscript[ξ, i]),(Subscript[ξ, j])) = 𝔼[(Subscript[ξ, i])(Subscript[ξ, j])] - 𝔼[(Subscript[ξ, i])]*𝔼[(Subscript[ξ, j])]
  =𝔼[(Subscript[ξ, i])]*𝔼[(Subscript[ξ, j])] - 𝔼[(Subscript[ξ, i])]*𝔼[(Subscript[ξ, j])]    (independence property)
  = 0.

Using definition 1

Taking m=0 and L=(Subscript[I, n]) in Definition 1 gives
X = 0+(Subscript[I, n])ξ = ξ
so
m = 0
and
Q = cov(ξ,ξ) = (({{1, 0, 0, ⋯, 0}, {0, 1, 0, ⋯, 0}, {0, 0, 1, ⋯, 0}, {⋮, ⋮, ⋮, ⋱, ⋮}, {0, 0, 0, ⋯, 1}})) = (Subscript[I, n]).

Note that Q=(Subscript[I, n]) is non-negative definite as
z·Q z =z·(Subscript[I, n]) z
  = (({{(Subscript[z, 1]), (Subscript[z, 2]), (Subscript[z, 3]), ⋯, (Subscript[z, n])}}))(({{1, 0, 0, ⋯, 0}, {0, 1, 0, ⋯, 0}, {0, 0, 1, ⋯, 0}, {⋮, ⋮, ⋮, ⋱, ⋮}, {0, 0, 0, ⋯, 1}}))(({{(Subscript[z, 1])}, {(Subscript[z, 2])}, {(Subscript[z, 3])}, {⋮}, {(Subscript[z, n])}}))
  = (Subscript[z, 1]^2)+(Subscript[z, 2]^2)+(Subscript[z, 3]^2)+⋯+(Subscript[z, n]^2) ≥ 0.

Frame 7

Note that if X,Y are independent than
𝔼[X Y] = 𝔼[X]𝔼[Y].

Frame 8

Plots of 2D normal distribution for different ρ

ParallelTable[
Plot3D[
PDF[MultinormalDistribution[{0,0},{{1,ρ},{ρ,1}}],{x,y}],
{x,-4,4},{y,-4,4},PlotRange->All],
{ρ,-0.5,0.5,0.25}]

Frame 9

Covariance matrix Q=0

Note that Q=0 is a covariance matrix as the quadratic form
z·Q z =z·0z
  = (({{(Subscript[z, 1]), (Subscript[z, 2]), (Subscript[z, 3]), ⋯, (Subscript[z, n])}}))(({{0, 0, 0, ⋯, 0}, {0, 0, 0, ⋯, 0}, {0, 0, 0, ⋯, 0}, {⋮, ⋮, ⋮, ⋱, ⋮}, {0, 0, 0, ⋯, 0}}))(({{(Subscript[z, 1])}, {(Subscript[z, 2])}, {(Subscript[z, 3])}, {⋮}, {(Subscript[z, n])}}))
  = 0+0+0+⋯+0 = 0.

Non-random vector is Gaussian

Consider deterministic vector x∈(ℝ^n).

This has mean vector
m = 𝔼[x] = x
and autocovariance matrix
Q = cov(x,x) = 𝔼[(x-𝔼[x])(((x-𝔼[x]))^T)] =𝔼[(x-x)(((x-x))^T)]
  = 0.

DEFINITION 1.
Taking n by k matrix of zeros L=0 we have
x = m+L ξ = m+0 ξ = x.

So    x~N(x,0).

DEFINITION 2.
The CF
𝔼[(e^(i u·x))] = (e^(i u·x))    (x is non-random)
  = (e^(i u·m-(1/2)u·Q u)).

So    x~N(x,0).

Invertible  matrix

If a matrix A is inverstible then det(A)≠0.

2D Example
Let A=(({{a, b}, {c, d}})).

The inverse of  A
(A^(-1)) = (1/(det(A)))adj(A) = (1/(a d-b c))(({{d, -b}, {-c, a}}))
which is only defined if det(A)≠0.

Frame 10

𝔼[(e^(i u ·X))] = 𝔼[(e^(i u ·(m+L ξ)))]
  = (e^(i u·m))𝔼[(e^(i u ·L ξ))]
  = (e^(i u·m))𝔼[(e^(i ((u )^T)L ξ))]
  = (e^(i u·m))𝔼[(e^(i((((L^T)u ))^T)ξ))]
  = (e^(i u·m))𝔼[(e^(i((L^T)u )·ξ))]
  = (e^(i u·m))(e^(-(1/2)((L^T)u )· ((L^T)u )))    (using equation (1))
  = (e^(i u·m))(e^(-(1/2)((((L^T)u ))^T)((L^T)u )))
  = (e^(i u·m))(e^(-(1/2)(u^T)L((L^T)u )))
  = (e^(i u·m))(e^(-(1/2)u·(L (L^T))u))

Frame 13

We have
X = m+(H^T) ξ
so
ξ = ((((H^T)))^(-1))(X-m) = ((((H^(-1))))^T)(X-m).

Then
𝔼[(e^(i u·ξ))] =𝔼[(e^(i u·((((H^(-1))))^T)(X-m)))]
  = 𝔼[(e^(i (u^T)((((H^(-1))))^T)(X-m)))]
  = 𝔼[(e^(i ((((H^(-1))u))^T)(X-m)))]
  = 𝔼[(e^(i ((H^(-1))u)·(X-m)))]
  = (e^(-i ((H^(-1))u)·m))𝔼[(e^(i ((H^(-1))u)·X))]
  = (e^(-i ((H^(-1))u)·m))(e^(i m·((H^(-1))u)-(1/2)((H^(-1))u)·Q((H^(-1))u)))    (by Deinition 2)
  = (e^(-(1/2)((H^(-1))u)·Q((H^(-1))u)))
  = (e^(-(1/2)((((H^(-1))u))^T)Q (H^(-1))u))
 = (e^(-(1/2)(u^T)((((H^(-1))))^T)Q (H^(-1))u))
 = (e^(-(1/2)(u^T)((((H^T)))^(-1))Q (H^(-1))u))
 = (e^(-(1/2)(u^T)((((H^T)))^(-1))(H^T)H (H^(-1))u))
  = (e^(-(1/2)(u^T)I u))
  = (e^(-(1/2)u,·u)).


What if instead we take
L = H
so
X = m+H ξ?

In this case
ξ = (H^(-1))(X-m)
so
𝔼[(e^(i u·ξ))] = 𝔼[(e^(i u·(H^(-1))(X-m)))]
  = 𝔼[(e^(i (u^T)(H^(-1))(X-m)))]
  = 𝔼[(e^(i (((((((H^(-1))))^T)u))^T)(X-m)))]
  = 𝔼[(e^(i (((((H^(-1))))^T)u)·(X-m)))]
  = (e^(-i (((((H^(-1))))^T)u)·m))𝔼[(e^(i (((((H^(-1))))^T)u)·X))]
  = (e^(-i (((((H^(-1))))^T)u)·m))(e^(i m·(((((H^(-1))))^T)u)-(1/2)(((((H^(-1))))^T)u)·Q (((((H^(-1))))^T)u)))    (by Deinition 2)
  = (e^(-(1/2)(((((H^(-1))))^T)u)·Q (((((H^(-1))))^T)u)))
  = (e^(-(1/2)(((((((H^(-1))))^T)u))^T)Q ((((H^(-1))))^T)u))
  = (e^(-(1/2)(u^T)(H^(-1))Q ((((H^(-1))))^T)u))
  = (e^(-(1/2)(u^T)(H^(-1))(H^T)H ((((H^(-1))))^T)u))
  = ?

Frame 14

Example Cholesky Decomposition

2D example

Let the covariance matrix    Q=(({{(Subscript[σ, 1]^2), ρ (Subscript[σ, 1])(Subscript[σ, 2])}, {ρ (Subscript[σ, 1])(Subscript[σ, 2]), (Subscript[σ, 2]^2)}})).
Then    Q=(H^T) H    where    H=(({{(Subscript[σ, 1]), ρ (Subscript[σ, 2])}, {0, (Sqrt[(Subscript[σ, 2]^2)(1-ρ)])}})).

σ1=1;σ2=2;ρ=0.5;
Q={{((σ1)^2),ρ σ1 σ2},{ρ σ1 σ2,((σ2)^2)}};
Q//MatrixForm
H=CholeskyDecomposition[Q];
H//MatrixForm
Transpose[H].H//MatrixForm

Frame 16

Forming sum of RVs from vector RV

Let    b∈ℝ,     A=(({{(Subscript[a, 1]), (Subscript[a, 2])}}))∈(ℝ^2)    and    X=(({{(Subscript[X, 1])}, {(Subscript[X, 2])}}))~N(m=(({{(Subscript[m, 1])}, {(Subscript[m, 2])}})),Q=(({{(Subscript[σ, 1]^2), ρ (Subscript[σ, 1])(Subscript[σ, 2])}, {ρ (Subscript[σ, 1])(Subscript[σ, 2]), (Subscript[σ, 2]^2)}}))).
Then
Y:=b+A X = b+(({{(Subscript[a, 1]), (Subscript[a, 2])}}))(({{(Subscript[X, 1])}, {(Subscript[X, 2])}}))
    = b+(Subscript[a, 1])(Subscript[X, 1])+(Subscript[a, 2])(Subscript[X, 2])
    ~N(A m+b,A Q (A^T))
    ~N((Subscript[a, 1]) (Subscript[m, 1])+(Subscript[a, 2])(Subscript[m, 2])+b,(Subscript[a, 1]^2)(Subscript[σ, 1]^2)+2(Subscript[a, 1])(Subscript[a, 2])ρ (Subscript[σ, 1])(Subscript[σ, 2])+(Subscript[a, 2]^2)(Subscript[σ, 2]^2)).

Frame 22

Combing TNC Parts 2 and 3

From TNC Part 2 we know
η := θ-𝔼[θ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ])
and    ξ    are independent.

From TNC Part 3 we know that
𝔼[θ|ξ] = 𝔼[θ]+cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ]).

So    θ-𝔼[θ|ξ]    and    ξ    are independent.

Remembering OLS regression

Let's say we have a sample (Subscript[{(Subscript[θ, i]),(Subscript[ξ, i])}, i=1,...,n]).
Then
𝔼[θ|ξ] = 𝔼[θ] + cov(θ,ξ)(((cov(ξ,ξ)))^(-1))(ξ - 𝔼[ξ])
  ≈ (1/n)(∑_(i=1)^n)(Subscript[θ, i]) + (((∑_(i=1)^n)((Subscript[θ, i])-(OverBar[θ]))((Subscript[ξ, i])-(OverBar[ξ]))/(n-1))/((∑_(i=1)^n)((((Subscript[ξ, i])-(OverBar[ξ])))^2)/(n-1)))(ξ-(1/n)(∑_(i=1)^n)(Subscript[ξ, i]))
  = (OverBar[θ]) + (((∑_(i=1)^n)((Subscript[θ, i])-(OverBar[θ]))((Subscript[ξ, i])-(OverBar[ξ])))/((∑_(i=1)^n)((((Subscript[ξ, i])-(OverBar[ξ])))^2)))(ξ-(OverBar[ξ]))    (using notation for sample mean)
  = (OverBar[θ])-(((∑_(i=1)^n)((Subscript[θ, i])-(OverBar[θ]))((Subscript[ξ, i])-(OverBar[ξ])))/((∑_(i=1)^n)((((Subscript[ξ, i])-(OverBar[ξ])))^2)))(OverBar[ξ]) + (((∑_(i=1)^n)((Subscript[θ, i])-(OverBar[θ]))((Subscript[ξ, i])-(OverBar[ξ])))/((∑_(i=1)^n)((((Subscript[ξ, i])-(OverBar[ξ])))^2)))ξ
  = (OverBar[θ])-(Subscript[(OverHat[β]), 1])(OverBar[ξ]) + (Subscript[(OverHat[β]), 1])ξ
  = (Subscript[(OverHat[β]), 0]) + (Subscript[(OverHat[β]), 1])ξ
which we remember from  OLS regression.

Frame 24

Proof TNC Part 1

Noting that
z·θ+u·ξ = (∑_(i=1)^m)(Subscript[z, i])(Subscript[θ, i])+(∑_(i=1)^n)(Subscript[u, i])(Subscript[ξ, i])
  = (({{(Subscript[z, 1])}, {⋮}, {(Subscript[z, m])}, {(Subscript[u, 1])}, {⋮}, {(Subscript[u, n])}}))·(({{(Subscript[θ, 1])}, {⋮}, {(Subscript[θ, m])}, {(Subscript[ξ, 1])}, {⋮}, {(Subscript[ξ, n])}})) = (({{z}, {u}}))·(({{θ}, {ξ}}))
we have
𝔼[(e^(i(z·θ+u·ξ)))] = 𝔼[(e^(i(({{z}, {u}}))·(({{θ}, {ξ}}))))]
  = (e^(i 𝔼[(({{θ}, {ξ}}))]·(({{z}, {u}}))-(1/2)(({{z}, {u}}))·cov((({{θ}, {ξ}})),(({{θ}, {ξ}})))(({{z}, {u}}))))    (Definition 2)
  = (e^(i 𝔼[(({{θ}, {ξ}}))·(({{z}, {u}}))]-(1/2)cov((({{z}, {u}}))·(({{θ}, {ξ}})),(({{z}, {u}}))·(({{θ}, {ξ}})))))
  = (e^(i 𝔼[(({{z}, {u}}))·(({{θ}, {ξ}}))]-(1/2)var((({{z}, {u}}))·(({{θ}, {ξ}})))))
  = (e^(i 𝔼[z·θ+u·ξ]-(1/2)var(z·θ+u·ξ))).

Also note that for scalar RVs X and Y
var(X+Y) = cov(X+Y,X+Y)
  = 𝔼[(X+Y)(X+Y)]-𝔼[X+Y]*𝔼[X+Y]
  = 𝔼[(X^2)+2X Y+(Y^2)]-(𝔼[X]+𝔼[Y])(𝔼[X]+𝔼[Y])
  = 𝔼[(X^2)]+2𝔼[X Y]+𝔼[(Y^2)]-((𝔼[X])^2)-2𝔼[X]*𝔼[Y]-((𝔼[Y])^2)
  = (𝔼[(X^2)]-((𝔼[X])^2))+2(𝔼[X Y]-𝔼[X]*𝔼[Y])+(𝔼[(Y^2)]-((𝔼[Y])^2))
  = var(X)+2cov(X,Y)+var(Y).

Using result above we have
var(z·θ+u·ξ) = var((∑_(i=1)^m)(Subscript[z, i])(Subscript[θ, i])+(∑_(i=1)^n)(Subscript[u, i])(Subscript[ξ, i]))
  = var((∑_(i=1)^m)(Subscript[z, i])(Subscript[θ, i]))+2cov((∑_(i=1)^m)(Subscript[z, i])(Subscript[θ, i]),(∑_(i=1)^n)(Subscript[u, i])(Subscript[ξ, i]))+var((∑_(i=1)^n)(Subscript[u, i])(Subscript[ξ, i]))
  = var(z·θ)+2cov(z·θ,u·ξ)+var(u·ξ)
  = var(z·θ)+2z·cov(θ,ξ)·u+var(u·ξ)
  = var(z·θ)+var(u·ξ)    (cov(θ,ξ)=0).

So
𝔼[(e^(i(z·θ+u·ξ)))] = (e^(i 𝔼[z·θ+u·ξ]-(1/2)var(z·θ+u·ξ)))
  = (e^(i 𝔼[z·θ+u·ξ]-(1/2)(var(z·θ)+var(u·ξ))))
  = (e^(i 𝔼[z·θ]-(1/2)var(z·θ)))(e^(i 𝔼[u·ξ]-(1/2)var(u·ξ)))
  = (e^(i 𝔼[θ]·z-(1/2)z·var(θ)z))(e^(i 𝔼[ξ]·u-(1/2)u·var(ξ)u))
  = (e^(i 𝔼[θ]·z-(1/2)z·cov(θ,θ)z))(e^(i 𝔼[ξ]·u-(1/2)u·cov(ξ,ξ)u))
  = 𝔼[(e^(i z·θ))]𝔼[(e^(i u·ξ))].

This shows θ and ξ are independent(one-to-one correspondence between CFs and distributions).

Frame 26

Proof TNC Part 2

Note that
𝔼[η] = 𝔼[θ-𝔼[θ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ])]
  = 𝔼[θ]-𝔼[𝔼[θ]]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(𝔼[ξ]-𝔼[𝔼[ξ]])
  = 𝔼[θ]-𝔼[θ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(𝔼[ξ]-𝔼[ξ])
 = 0.

Frame 27

Proof TNC Part 3

𝔼[η|ξ] =𝔼[θ-𝔼[θ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ])|ξ]
  = 𝔼[θ|ξ]-𝔼[𝔼[θ]|ξ]-𝔼[cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ])|ξ]
  = 𝔼[θ|ξ]-𝔼[𝔼[θ]|ξ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))𝔼[(ξ-𝔼[ξ])|ξ]
  = 𝔼[θ|ξ]-𝔼[𝔼[θ]|ξ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(𝔼[ξ|ξ]-𝔼[𝔼[ξ]|ξ])
  = 𝔼[θ|ξ]-𝔼[θ]*𝔼[1|ξ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ 𝔼[1ξ]-𝔼[ξ]𝔼[1|ξ])
  = 𝔼[θ|ξ]-𝔼[θ]-cov(θ,ξ)cov(((ξ,ξ))^(-1))(ξ-𝔼[ξ]).

Note we have used
𝔼[𝔼[θ]|ξ] = 𝔼[θ]*𝔼[1|ξ]    (𝔼[θ] is a constant)
  = 𝔼[θ]
and
𝔼[ξ|ξ] = ξ 𝔼[1|ξ]    (see properties conditional expectation Ch 1)
  = ξ.

Frame 33

(({{θ}, {ξ}}))~N((({{1}, {-1}})),(({{5, 2}, {2, 1}})))

Another method

Let (Subscript[f, θ,ξ]) be the joint density of θ,ξ, (Subscript[f, θ|ξ]) be the conditional density of θ|ξ and let (Subscript[f, ξ]) be the density of ξ. 

Then
𝔼[θ|ξ=y] = (∫_(-∞)^∞)x (Subscript[f, θ|ξ])(x,y)dx = (1/((Subscript[f, θ])(y)))(∫_(-∞)^∞)x (Subscript[f, θ,ξ])(x,y)dx
  = 2 y+3.    (see code below - life's too short to integrate!)

So more generally
𝔼[θ|ξ] = 2ξ+3.

Then
𝔼[(((θ-𝔼[θ|ξ]))^2)] = 𝔼[(((θ-2ξ-3))^2)]
  = 𝔼[9-6 θ+(θ^2)+12 ξ-4θ ξ + 4(ξ^2)]
  = 9-6𝔼[θ]+𝔼[(θ^2)]+12𝔼[ξ]-4𝔼[θ ξ]+4𝔼[(ξ^2)]

  = 9-6𝔼[θ]+(var(θ)+((𝔼[θ])^2))+12𝔼[ξ]
   -4(cov(θ,ξ)+𝔼[θ]𝔼[ξ])+4(var(ξ)+((𝔼[ξ])^2))    (definition of covariance)

  = 9-6*1+(5+(1^2))+12(-1)-4(2+1(-1))+4(1+(((-1))^2))
  = 9-6+6-12-4+8
  = 1.

fξ[y_]:=PDF[NormalDistribution[-1,1],y];
fθξ[x_,y_]:=PDF[MultinormalDistribution[{1,-1},{{5,2},{2,1}}],{x,y}];
(1/(fξ[y]))(∫_(-∞)^∞)x fθξ[x,y]\[DifferentialD]x

Frame 34

ξ,ζ~N(0,1) and independent.

Let    θ=|ξ|sgn(ζ).

Aim: show θ is Gaussian and cov(θ,ξ)=0 despite θ and ξ being dependent.

First we show that θ is normal.
𝔼[(e^(u θ))] = 𝔼[(e^(u|ξ|sgn(ζ)))] = 𝔼[𝔼[(e^(u|ξ|sgn(ζ)))|ξ]]    (tower law)
  = 𝔼[(1/2)(e^(u|ξ|))+(1/2)(e^(-u|ξ|))]    (50-50 chance of (Subscript[ξ, 2]) being positive or negative)
  = 𝔼[cosh(u|ξ|)] = 𝔼[cosh(u ξ)] = 𝔼[(1/2)(e^(u ξ))+(1/2)(e^(-u ξ))]
  = (1/2)𝔼[(e^(u ξ))]+(1/2)𝔼[(e^(-u ξ))]
  = (1/2)(e^((1/2)(u^2)))+(1/2)(e^((1/2)(u^2)))
=  (e^((1/2)(u^2)))    (MGF of N(0,1) RV - see Definition 2)
so θ~N(0,1).

So both θ and ξ are normal.
However, they are not "joint-normal".

To see this,first note that
cov(θ,ξ) = 𝔼[θ ξ]-𝔼[θ]𝔼[ξ]
  = 𝔼[θ ξ]    (𝔼[ξ]=0, also     𝔼[θ]=𝔼[|ξ|sgn(ζ)] = 𝔼[|ξ|]𝔼[sgn(ζ)] = 𝔼[|ξ|]((1/2)(1)+(1/2)(-1))=0)
  = 𝔼[|ξ|sgn(ζ)ξ] = 𝔼[|ξ|ξ]*𝔼[sgn(ζ)]    (ξ,ζ are independent)
  = 𝔼[|ξ|ξ]((1/2)(1)+(1/2)(-1))
  = 0.

By Part 1 TNC, if θ and ξ are joint normal then cov(θ,ξ)=0 implies θ and ξ are independent.
But as we show below, θ and ξ are dependent.

To demonstrate this it suffices to show that the covariance of any function of θ and
function of ξ is non zero.

Noting that
|θ| = ||ξ|sgn(ζ)| = |ξ||sgn(ζ)| = |ξ|
consider
cov(|θ|,|ξ|) = cov(|ξ|,|ξ|)
  = var(|ξ|) = 𝔼[|ξ|*|ξ|]-((𝔼[|ξ|])^2) = 𝔼[(ξ^2)]-((𝔼[|ξ|])^2) = 1-((𝔼[|ξ|])^2)
 = 1-(2/π) ≠0.

So θ and ξ are dependent despite cov(θ,ξ)=0, implying they are not joint-normal (violation TNC part 1).

Finally, to see that
((𝔼[|ξ|])^2) = (2/π)
observe
𝔼[|ξ|] = (∫_(-∞)^∞)|x|(Subscript[f, ξ])(x)dx = (∫_(-∞)^0)(-x)(Subscript[f, ξ])(x)dx+(∫_0^∞)x (Subscript[f, ξ])(x)dx
  = 2(∫_0^∞)x (Subscript[f, ξ])(x)dx
  = (Sqrt[(2/π)]).    (see code below - life's too short to integrate!)

(* Finding 𝔼[\[LeftBracketingBar]ξ\[RightBracketingBar]] *)
2(∫_0^∞)x(1/(Sqrt[2π]))(E^(-((x^2)/2)))\[DifferentialD]x