Stoch Proc - Lab/Tutorial 8 Solutions

Scott Alexander

Q1

(a)

Possible state transitions:
*RR(1)->RR(1)⟹(Subscript[p, 11])=0.4    or    RR(1)->RN(2)⟹(Subscript[p, 12])=0.6
*RN(2)->NR(3)⟹(Subscript[p, 23])=0.2    or    RN(2)->NN(4)⟹(Subscript[p, 24])=0.8
*NR(3)->RR(1)⟹(Subscript[p, 31])=0.3    or    NR(3)->RN(2)⟹(Subscript[p, 32])=0.7
*NN(4)->NR(3)⟹(Subscript[p, 43])=0.1    or    NN(4)->NN(4)⟹(Subscript[p, 44])=0.9

P = (({{(4/10), (6/10), 0, 0}, {0, 0, (2/10), (8/10)}, {(3/10), (7/10), 0, 0}, {0, 0, (1/10), (9/10)}}))

(b)

We need
𝔼[(Subscript[X, 4])] = (∑_(j=1)^4)j*P((Subscript[X, 4])=j) =(∑_(j=1)^4)j*(Subscript[p, j])(4)

where
p(4) = ((((P^T)))^4)p(0) = (((({{(2/5), 0, (3/10), 0}, {(3/5), 0, (7/10), 0}, {0, (1/5), 0, (1/10)}, {0, (4/5), 0, (9/10)}})))^4)(({{(1/10)}, {(15/100)}, {(2/10)}, {(55/100)}}))
  = (({{(43/625), (123/2500), (159/2500), (441/10000)}, {(159/1250), (247/2500), (74/625), (899/10000)}, {(123/1250), (59/625), (247/2500), (947/10000)}, {(441/625), (947/1250), (899/1250), (7713/10000)}}))(({{(1/10)}, {(15/100)}, {(2/10)}, {(55/100)}})) = (({{(10247/200000)}, {(20133/200000)}, {(19169/200000)}, {(150451/200000)}})).

PP=(({{(4/10), (6/10), 0, 0}, {0, 0, (2/10), (8/10)}, {(3/10), (7/10), 0, 0}, {0, 0, (1/10), (9/10)}}));
p0={(1/10),(15/100),(2/10),(55/100)};
PP4=MatrixPower[PP,4];
p4=Transpose[PP4].p0;
x={1,2,3,4};
x.p4//N

(c)

Need to solve
(P^T)π = π    ⟹    ((P^T)-I)π = 0
s.t.
1·π = 1
That is
(({{(2/5)-1, 0, (3/10), 0}, {(3/5), 0-1, (7/10), 0}, {0, (1/5), 0-1, (1/10)}, {0, (4/5), 0, (9/10)-1}}))(({{(Subscript[π, 1])}, {(Subscript[π, 2])}, {(Subscript[π, 3])}, {(Subscript[π, 4])}})) = (({{0}, {0}, {0}, {0}}))
s.t.
(({{1, 1, 1, 1}}))(({{(Subscript[π, 1])}, {(Subscript[π, 2])}, {(Subscript[π, 3])}, {(Subscript[π, 4])}})) = 1.

This becomes
(({{(2/5)-1, 0, (3/10), 0, 0}, {(3/5), 0-1, (7/10), 0, 0}, {0, (1/5), 0-1, (1/10), 0}, {0, (4/5), 0, (9/10)-1, 0}, {1, 1, 1, 1, 1}})) ~ (({{1, 0, 0, 0, (1/21)}, {0, 1, 0, 0, (2/21)}, {0, 0, 1, 0, (2/21)}, {0, 0, 0, 1, (16/21)}, {0, 0, 0, 0, 0}}))
so
(({{(Subscript[π, 1])}, {(Subscript[π, 2])}, {(Subscript[π, 3])}, {(Subscript[π, 4])}})) = (({{(1/21)}, {(2/21)}, {(2/21)}, {(16/21)}})) = (({{0.047619047619047616`}, {0.09523809523809523`}, {0.09523809523809523`}, {0.7619047619047619`}})).

Alternatively
*find eigenavlue 1 of (P^T)
*then find associated eigenvector
*normal eigenvector to components sum to 1.

RowReduce[(({{(2/5)-1, 0, (3/10), 0, 0}, {(3/5), 0-1, (7/10), 0, 0}, {0, (1/5), 0-1, (1/10), 0}, {0, (4/5), 0, (9/10)-1, 0}, {1, 1, 1, 1, 1}}))]//MatrixForm

eig=Eigensystem[Transpose[PP]];
eig//MatrixForm;
eig[[2,1]]/Total[eig[[2,1]]]//MatrixForm//N

Transpose[PP]//MatrixForm

(d)

Find P(2) matrix    ⟹    all entries non-zero then ergodic.

Alternatively
(lim_(t->∞)) p(t) = (lim_(t->∞)) ((((P^T)))^t)p(0).

Note that
(lim_(t->∞)) (P^t) = (({{(1/21), (2/21), (2/21), (16/21)}, {(1/21), (2/21), (2/21), (16/21)}, {(1/21), (2/21), (2/21), (16/21)}, {(1/21), (2/21), (2/21), (16/21)}})).

Now consider arbitrary initial distribution
p(0) = (({{(Subscript[p, 1])}, {(Subscript[p, 2])}, {(Subscript[p, 3])}, {1-(Subscript[p, 1])-(Subscript[p, 2])-(Subscript[p, 3])}}))
so that
(lim_(t->∞)) p(t) = (lim_(t->∞)) ((((P^T)))^t)p(0)
  = (({{(1/21), (1/21), (1/21), (1/21)}, {(2/21), (2/21), (2/21), (2/21)}, {(2/21), (2/21), (2/21), (2/21)}, {(16/21), (16/21), (16/21), (16/21)}}))(({{(Subscript[p, 1])}, {(Subscript[p, 2])}, {(Subscript[p, 3])}, {1-(Subscript[p, 1])-(Subscript[p, 2])-(Subscript[p, 3])}})) = (({{(1/21)}, {(2/21)}, {(2/21)}, {(16/21)}})).

MatrixPower[PP,2]//MatrixForm

Plim=Limit[MatrixPower[PP,t],t->∞];
Plim//MatrixForm

MatrixPower[PP,25]//N//MatrixForm

Q2

(a)

P = (({{0, 0, 0, 1}, {(1/5), 0, (4/5), 0}, {0, 0, 0, 1}, {0, 1, 0, 0}}))

PP=(({{0, 0, 0, 1}, {(1/5), 0, (4/5), 0}, {0, 0, 0, 1}, {0, 1, 0, 0}}));
Table[MatrixPower[PP,n]//MatrixForm,{n,1,6}]

Graph[DiscreteMarkovProcess[{1/4,1/4,1/4,1/4},PP]]

(b)

P = (({{(1/6), (1/2), 0, 0, (1/3)}, {(1/4), (1/4), 0, 0, (1/2)}, {0, 0, (1/2), (1/2), 0}, {0, 0, (6/7), 0, (1/7)}, {(1/10), (3/10), 0, 0, (3/5)}}))

PP=(({{(1/6), (1/2), 0, 0, (1/3)}, {(1/4), (1/4), 0, 0, (1/2)}, {0, 0, (1/2), (1/2), 0}, {0, 0, (6/7), 0, (1/7)}, {(1/10), (3/10), 0, 0, (3/5)}}));
chain=DiscreteMarkovProcess[{1/5,1/5,1/5,1/5,1/5},PP];Graph[chain]
MarkovProcessProperties[chain,"Periods"]

Q3

Info

A=(({{-4, 3, (4/5), (1/5)}, {(6/4), -5, (11/4), (3/4)}, {1, 2, -7, 4}, {0, 0, 0, 0}}))

(a)

(P^jump) = (({{0, (3/4), (4/20), (1/20)}, {(6/20), 0, (11/20), (3/20)}, {(1/7), (2/7), 0, (4/7)}, {0, 0, 0, 1}}))

Series[Exp[s a],{a,0,4}]

(b)

The transition matrix
P(s) = (e^(s A))

In R use expm function from the expm package.


Consider arbitrary initial distribution
p(0) = (({{(Subscript[p, 1])}, {(Subscript[p, 2])}, {(Subscript[p, 3])}, {1-(Subscript[p, 1])-(Subscript[p, 2])-(Subscript[p, 3])}}))
so that
(lim_(t->∞)) p(t) = (lim_(t->∞)) (P^T)(t)p(0) = (lim_(t->∞)) ((e^(t (A^T))))p(0)
  = (({{1, 1, 1, 1}, {0, 0, 0, 0}, {0, 0, 0, 0}, {0, 0, 0, 0}}))(({{(Subscript[p, 1])}, {(Subscript[p, 2])}, {(Subscript[p, 3])}, {1-(Subscript[p, 1])-(Subscript[p, 2])-(Subscript[p, 3])}})) = (({{0}, {0}, {0}, {1}}))

AA=(({{-4, 3, (4/5), (1/5)}, {(6/4), -5, (11/4), (3/4)}, {1, 2, -7, 4}, {0, 0, 0, 0}}));
Pt[t_]:=MatrixExp[t AA];
Plim=Limit[Pt[t],t->∞]//FullSimplify//MatrixForm

(c)

Plot[Pt[t][[1,4]],{t,0,10}]

(d)

(e)

Q4

Info

(P^jump)=(({{0, (2/3), (1/3)}, {(1/2), 0, (1/2)}, {(3/4), (1/4), 0}}))

𝔼[(Subscript[T, 1])] = 2
var((Subscript[T, 2])) = 4
𝔼[(Subscript[T, 3]^2)] = (1/8)

Mean[ExponentialDistribution[λ]]

Variance[ExponentialDistribution[λ]]

(a)

Solve
(A^T)π = 0
s.t.
1·π = 1.

(b)

(c)