\chapter{Eigenequations}


We have seen that linear maps can exhibit interesting behaviours with geometrical interpretations like scaling, translating, squeezing, reflecting etc. Sometimes it is useful to know the domain elements of a linear transform such that the linear transform is merely scaling the vector; this is the idea that gave birth to \emph{eigenequations}, and there will be an abundance of things to say about them mathemtically too.


\begin{definition}[Eigenequation]
An \emph{eigenequation} is an equation of the following form.
\[\mathbf{A}\mathbf{x}=\lambda \mathbf{x}\]
\[T(\mathbf{x})=\lambda \mathbf{x}\]
\begin{itemize}
	\item $\mathbf{A}$ is a square matrix
	\item $T : V \to V$ is a linear endomorphism
	\item solutions of $\lambda$ are called \emph{eigenvalues}
	\item solutions of $\mathbf{x}$ are called \emph{eigenvectors}
\end{itemize}
\end{definition}

Let $V$ be a linear space over $\mathbb{C}$, then all eigenequations have nontrivial solutions.



\section{Solving eigenequations}


To solve the eigenequation for some matrix, the classical method is to reduce it to finding the zeros of the \emph{characteristic polynomial}.

\[\mathbf{A}\mathbf{x}=\lambda \mathbf{x}\]
\[\mathbf{A}\mathbf{x}=\lambda \mathbf{I} \mathbf{x}\]
\[\mathbf{A}\mathbf{x}=\lambda \mathbf{I} \mathbf{x}\]
\[\mathbf{A}\mathbf{x} - \lambda \mathbf{I} \mathbf{x} = \mathbf{0}\]
\[( \mathbf{A} - \lambda \mathbf{I} ) \mathbf{x} = \mathbf{0}\]

This implies that the linear map specified by $(\mathbf{A}-\lambda \mathbf{I})$ is not invertible (and nor is the matrix). The matrix invertibility theorem therefore means we have the following.
\[\mathrm{det}( \mathbf{A} - \lambda \mathbf{I} )=0\]

The expression obtained on the left is a polynomial in terms of $\lambda$ called the \emph{characteristic polynomial}, and calculating its zeros calculates the eigenvalues of $\mathbf{A}$.


\begin{theorem}[Characteristic polynomial (Linear algebra)]
\[ \mathbf{A}\mathbf{v} = \lambda \mathbf{v} \iff \det (\mathbf{A}-\lambda\mathbf{I})=0 \]
\end{theorem}

Since zeros have a multiplicity, we can associate this multiplicity to the eigenvalues obtained by the characteristic polynomial; the multiplicity of an eigenvalue will prove important later when studying eigenspaces.

\begin{definition}[Multiplicity of an eigenvalue]
The\emph{multiplicity of an eigenvalue} is the multiplicity of the eigenvalue as a zero of the characteristic polynomial.
\end{definition}

We can organize the eigenvalues of a linear map (matrix) by a \emph{spectrum}; a multiset that contains all eigenvalues with repeated membership representing multiplicity.

\begin{definition}[Spectrum of a matrix]
Let $\mathbf{A}$ be a square matrix, then the \emph{spectrum of $\mathbf{A}$} is the multiset of eigenvalues of $\mathbf{A}$, where multiplicity of the is eigenvalue is counted
\[\mathrm{spec}(T)\]
\end{definition}
















\section{Eigenspaces}



With every eigenvalue solution to an eigenequation, the eigenvector solutions associated with that eigenvalue span an \emph{eigenspace}; a linear space of solutions to the eigenequation.

\begin{definition}[Eigenspace]
An \emph{eigenspace of $T$ associated with $\lambda$} is a linear space spanned by the eigenvectors of $T$ associated with the same eigenvalue $\lambda$. These eigenvectors form an \emph{eigenbasis} for this eigenspace.
	\[E (\lambda , T) = \{ \mathbf{x} : T(\mathbf{x})= \lambda \mathbf{x} \} \]
	\[E (\lambda , \mathbf{A}) = \{ \mathbf{x} : \mathbf{A}\mathbf{x} = \lambda \mathbf{x} \} \]
\end{definition}

The span of any set of vectors produces a linear space, so eigenspaces are definitely linear spaces. Furthermore, they can be represented by the kernel of a particular linear map.

\begin{proposition}[Eigenspaces as kernels]
	Let $T(\mathbf{x}) = (\mathbf{A}-\lambda \mathbf{I})\mathbf{x}$, then $\mathrm{ker}(T)= E(\lambda, \mathbf{A})$
\end{proposition}




\subsection{Eigenvalue of an eigenspace}


An eigenspace is determined by the linear map (or equivalently, the matrix of that linear map) as well as the chosen eigenvalue; the eigenvalue strongly characterizes the nature of the eigenspace, particularly its dimension. Later, we will also view what relationships hold between eigenspaces of different eigenvalues.


A major fact is that the multiplicity of an eigenvalue is the dimension of its related eigenspace.

\begin{proposition}[Dimension of an eigenspace]
The dimension of $E(T,\lambda)$ is the multiplicity of $\lambda$
\[\mathrm{mult}(\lambda,T)=\mathrm{dim}(E (\lambda , T))\]
\begin{proposition}



Now let's compare eigenspaces formed from different eigenvalues. One can prove that these spaces are orthogonal to eachother; a vector of $E(T,\lambda_1)$ is orthogonal to a vector of $E(T,\lambda_2)$ if $\lambda_1 \neq \lambda_2$!


\begin{lemma}
Let $\mathbf{A}$ be a symmetric matrix. Then the eigenspaces of each distinct eigenvalue are orthogonal subspaces to eachother.
\end{lemma}

Let $\mathbf{v} \in E(T, \lambda_1), \mathbf{u} \in E(T,\lambda_2)$ be arbitrary vectors from the distinct eigenspaces, then consider the following calculation.

\[\lambda_1 ( \mathbf{v_1} \cdot \mathbf{v}_2 ) = (\lambda_1 \mathbf{v})^{\intercal} \mathbf{v}_2\]
\[= (\mathbf{A} \mathbf{v})^{\intercal} \mathbf{v}_2\]
\[= (\mathbf{v}^{\intercal} \mathbf{A}^{\intercal}  \mathbf{v}_2\]

Here we apply the fact that $\mathbf{A}$ is symmetric to continue.

\[= (\mathbf{v}^{\intercal} \mathbf{A}  \mathbf{v}_2\]
\[= (\mathbf{v}^{\intercal} (\lambda_2 \mathbf{v}_2) \]
\[= (\mathbf{v}^{\intercal} (\lambda_2 \mathbf{v}_2) \]
\[= \lambda_2 ( \mathbf{v_1} \cdot \mathbf{v}_2 )\]

Since $\lambda_1 \neq \lambda_2$, we must have $\mathbf{u} \cdot \mathbf{v} =0$, proving orthogonality of any pair of arbitrary vectors from eigespaces of different eigenvalues.


This lemma will prove important in the \emph{spectral theorem} developed in the following section. 



















\section{Eigendecomposition and matrix diagonalization}


Matrix decompositions are extremely useful for numerical analysis, and are occasionally used for proofs within matrix theory. One prominent type of matrix decomposition is that of \emph{eigendecomposition}, also known as \emph{spectral decomposition}. As both names hint, this type of decomposition is intimately linked with the eigenvalues and eigenvectors associated with the matrix.


We first give the definition of a diagonalizable matrix as a matrix that has a spectral decomposition.


\begin{definition}[Diagonalizable matrix]
	An \emph{diagonalizable matrix} is a square matrix $\mathbf{A}$ such that there exists matrixes $\mathbf{P},\mathbf{D}$ such that $\mathbf{A}=\mathbf{P}\mathbf{D}\mathbf{P}^{-1}$
\begin{itemize}
\item $\mathbf{P}$ is an invertible matrix 
\item $\mathbf{D}$ is a diagonal matrix
\end{itemize}
This is called the \empg{spectral decomposition} of a matrix.
\end{definition}


This definition is perhaps not immediately obvious in its relation to a matrix's eigenequations, though we will explore an equivalent condition that makes this relationship much more visible.

Notice that this equivalent to saying that $\mathbf{A}\mathbf{P}=\mathbf{P}\mathbf{D}$ with invertible $\mathbf{P}$; this form shows that we are really just considering multiple eigenequations at once! This is the expresion that originally motivated the idea of a diagonalizable matrix.

The $\mathbf{D}$ has its diagonals as the spectrum, while $\mathbf{P}$ holds the eigenvectors.

It is crucial to note that we require $\mathbf{P}$ to be invertible in order to algebraically reverse this expression back to a decomposition of $\mathbf{A}$, and since $\mathbf{P}$ is made of eigenvectors of $\mathbf{A}$, the invertible matrix theorem implies that $\mathbf{A}$ must have linearly independent eigenvector solutions, since these eigenvector solutions are the columns of $\mathbf{P}$ and must be linearly independent to make $\mathbf{P}$ invertible.



\begin{theorem}[Equivalent definition of a diagonalizable matrix]
An $n \times n$ square matrix $\mathbf{A}$ is diagonalizable iff 
\begin{itemize}
	\item $\mathbb{A}$ has spectrum of cardinality $n$
	\item The eigenequation of $\mathbb{A}$ has $n$ linearly independent eigenvector solutions.
\end{itemize}
\end{theorem}



Proving that diagonalizable matrixes have $n$ linearly independent eigenvector solutions follows by reducing $\mathbf{P}^{-1}\mathbf{D}\mathbf{P}\mathbf{x} = \lambda \mathbf{x}$ to $\mathbf{D}(\mathbf{P}\mathbf{x}) = \lambda (\mathbf{P}\mathbf{x})$, then noting that this eigenequation has $n$ linearly independent eigenvectors. To prove the other direction, we use our $n$ eigenvectors and eigenvalues to construct such a diagonalizable matrix equal to $\mathbf{A}$.

Due to the constructive nature of this proof, we also obtain a method to calculate the spectral decomposition of a diagonalizable matrix.
%This is known as \emph{diagonalization}.





\subsection{Spectral theorem}


We have seen that whether a matrix is diagonalizable is completely dependent on linear independence of eigenvectors, however if we further restrict our attention to the case where the $\mathbf{P}$ is an orthonormal matrix (call such matrixes \emph{orthonormally diagonalizable matrixes}), we will end up with a familiar class of matrixes.


\begin{definition}[Orthogonally diagonalizable matrix]
An \emph{orthogonally diagonalizable matrix} is a square matrix $\mathbf{A}$ such that there exists matrixes $\mathbf{P},\mathbf{D}$ such that $\mathbf{A}=\mathbf{P}^{\intercal}\mathbf{D}\mathbf{P}$
\begin{itemize}
\item $\mathbf{P}$ is an orthogonal matrix 
\item $\mathbf{D}$ is a diagonal matrix
\end{itemize}
\end{definition}



\begin{theorem}
A real square matrix $\mathbf{A}$ is orthonormally diagonalizable iff $\mathbf{A}$ is a real symmetric matrix.
\end{theorem}

This is proven by considering the following calculation assuming that $\mathbf{A}$ is orthonomrally diagonalizable.

\[ \mathbf{A}^{\intercal} = (\mathbf{P}^{\intercal}\mathbf{D}\mathbf{P})^{\intercal}  \]
\[  = \mathbf{P}\mathbf{D}^{\intercal}\mathbf{P}^{\intercal}\]
\[  = \mathbf{P}^{\intercal}\mathbf{D}\mathbf{P}\]
\[  = \mathbf{A} \]

This calculation proves one direction, and the other direction is proved by the previous lemma about real symmetric matrixes producing orthogonal eigenspaces.

This result, cleaned up, produces the \emph{spectral theorem}.


\begin{theorem}[Spectral theorem (Symmetric matrix)]
$\mathbf{A}$ is an $n \times n$ real symmetric matrix then any of the following hold.
\begin{itemize}
\item $\mathrm{spec}(\mathrm{A})$ has cardinality $n$ and is a subset of $\mathbb{R}$
\item Eigenvectors associated with different eigenvalues of $\mathbf{A}$ are orthogonal
\item $\mathrm{A}$ is orthogonally diagonalizable
\end{itemize}
\end{theorem}


In functional analysis, the spectral theorem applies more generally to objects called "self-adjoint operators"; a class of objects which includes the symmetric matrixes (or more technically, the linear maps associated with them). Indeed, there is a whole subfield of functional analysis called \emph{spectral theory} which generalizes the theory of eigenequations to infinite dimensional linear spaces.





\subsection{Singular value decomposition}


Let's return back to the realm of matrixes. Although only diagonalizable matrixes have a spectral decomposition, there exists a more general decomposition for any real matrix, of which the spectral decomposition is a special case applying to diagonalizable matrixes.

It is derived from considering the eigenequations. Despite the fact that a general real matrix may not display the desirable properties of a symmetric matrix or even a diagonalizable matrix, a slightly more complicated decomposition is still possible; the \emph{singular value decomposition}.


\begin{theorem}[Singular value decomposition (SVD)]
Let $\mathbf{A}$ be a real matrix, then there exists the following decomposition.
\[\mathbf{A} = \mathbf{W} \Sigma \mathbf{U}^{\intercal} \]
\end{theorem}









\subsection{Results by using SD and SVD}

Diagonalizable matrixes have various properties with the determinant.

\[ \mathrm{det} (\mathbf{P}^{-1}\mathbf{D}\mathbf{P}) = \prod^{n}_{i=1} [\mathbf{D}]_{i,i} \prod^{n}_{i=1} \lambda_{i}\]