\chapter{Orthogonality}

An inner product space uses its inner product to give a notion of angle and therefore orthogonality, regardless of geometric intuition. 


\begin{definition}[Orthogonal pair of vectors]
\[\mathbf{v} \perp \mathbf{u} \iff \langle \mathbf{v} , \mathbf{u} \rangle = 0\]
\end{definition}

\begin{definition}[Orthogonal set]
An \emph{orthogonal set} is a of vectors that are all orthogonal to eachother.
\end{definition}


\begin{definition}[Orthogonal basis]
An \emph{orthogonal basis for $V$} is a basis for $V$ that is an orthogonal set.
\end{definition}

An even more standardized case of orthogonality is \emph{orthonormality}, where all vectors are orthogonal to eachother and have unit norm.

\section{Orthonormal set}
\begin{definition}[Orthogonal set]
An \emph{orthonormal set } is an orthogonal set where all vectors have norm 1.
\end{definition}

\begin{definition}[Orthonormal basis]
An \emph{orthonormal basis for $V$} is a basis for $V$ that is an orthonormal set.
\end{definition}





\section{Orthogonal subspaces}


\begin{definition}[Orthogonal subspaces]
Two subspaces $U,W \leq V$ are orthogonal iff for any $\mathbf{u} \in U$ and $\mathbf{w} \in W$, $\langle \mathbf{u} , \mathbf{w} \rangle = 0$
\end{definition}



\subsection{Orthogonal complement}

It is known that one can decompose any linear space into 2 linear spaces $U,W$ such that $U \cap W = \mathbf{0}$; this is seen by creating a partition on the (Hamel) basis, and using the span of each partition to create the respective linear subspaces. When considering finite dimensional inner product linear spaces, one may decompose the linear space into 2 orthogonal subspaces; this is the \emph{orthogonal complement}.

\begin{definition}[Orthogonal complement]
Let $V$ be an inner product space and $U \leq V$, then \emph{orthogonal complement of $U$} is the linear subspace $ U^{\perp} \leq V$
	\[U^{\perp} = \{\mathbf{v} \in V : \forall \mathbf{u} \in U [ \langle \mathbf{v}, \mathbf{u} \rangle = 0 ]  \}\]
\end{definition}

\begin{proposition}
	\[U \leq V \land \mathrm{dim}(V) < \infty \implies V = U \oplus U^{\perp}\]
\end{proposition}

This is proven by showing that for any $\mathbf{v}$


\begin{proposition}
\[(U^{\perp})^{\perp} = U\]
\end{proposition}




\subsection{Vector projection}

Can any vector in $V$ can be represented by the sum of a vector of $U$ and a vector of $U^{\perp}$ (so $\mathbf{v} = \mathbf{y} + \mathbf{z}$)? We can prove this for finite dimensional spaces by finding an explicit construction for such a $\mathbf{y}$. Let's remind ourselves of the following 2 properties we have to satisfy for such a construction; they will be the key for actually constructing such a $\mathbf{y}$.

\begin{itemize}
	\item $\mathbf{y} \in U$
	\item $(\mathbf{v} - \mathbf{y}) \in U^{\perp}$
\end{itemize}

Furthermore, let $\mathcal{U}$ be an orthonormal basis for $U$, then $\mathbf{y}$ has the following form.

\[\mathbf{y} = \sum_{n}_{k=1} c_{k} \mathbf{u}_k \]

We also want the following for every $\mathbf{u} \in \mathcal{U}$.
\[(\mathbf{v}-\mathbf{y}) \cdot \mathbf{u} = 0\]

By substituting the form of $\mathbf{y}$, we can caluclate the coefficients.

\[\mathbf{v} \cdot \mathbf{u}_k - (\sum_{n}_{j=1} c_{j} \mathbf{u}_j) \cdot \mathbf{u}_k =  0\]
\[\mathbf{v} \cdot \mathbf{u}_k -  c_{k} \mathbf{u}_k \cdor \mathbf{u}_k =  0\]
\[ c_k = \frac{\mathbf{v} \cdot \mathbf{u}_k}{\mathbf{u}_k \cdor \mathbf{u}_k} \]

Therefore our $\mathbf{y}$ equals the following vector.

\[\mathbf{y} = \sum_{n}_{k=1} \frac{\mathbf{v} \cdot \mathbf{u}_k}{\mathbf{u}_k \cdor \mathbf{u}_k}  \mathbf{u}_k \]

In $\mathbb{R}^3$ and $\mathbb{R}^2$ this looks like a "shadow" or "projection" made by the vector $\mathbf{v}$ standing on $\mathrm{span}(\mathcal{U})$, hence it is called the projection.


\begin{definition}[Vector projection onto $\mathrm{span}(\mathcal{U})$ (finite dimensional linear space)]
	Let $\mathcal{U}$ be an orthonormal basis for a linear subspace of $V$, then the \emph{projection of $\mathbf{v}$ onto $\mathrm{span}(\mathcal{U})$} is the following function $\mathrm{proj}_{\mathcal{U}} : V \to \mathrm{span}(\mathcal{U})$.
	\[ \mathrm{proj}_{\mathcal{U}} ( \mathbf{v}) = \sum_{\mathbf{u} \in \mathcal{U}} \frac{\mathbf{u}\cdot\mathbf{v}}{\mathbf{u}\cdot\mathbf{u}} \mathbf{u} \]

\end{definition}

It is common to only want to project onto one vector, hence the following notation is also used.

\[ \mathrm{proj}_{\mathbf{u}} ( \mathbf{v}) = \frac{\mathbf{u}\cdot\mathbf{v}}{\mathbf{u}\cdot\mathbf{u}} \mathbf{u} \]


It is worth noting that this book has assumed that we are working in a finite dimensional linear space; the notion of vector projection does generalize easily to infinite dimensional linear spaces by the same construction, but considering a Schauder basis rather than a Hamel basis.



\subsection{Orthogonal decomposition theorem}

Now that projections are defined, we can propose the \emph{orthogonal decomposition theorem}, a theorem that holds in finite dimensional linear spaces which is essentially a more concrete upgrade of the fact that $V= U \oplus U^{\perp}$.

\begin{theorem}[Orthogonal decomposition theorem]
	Let $V$ be a finite dimensional linear space and $\mathcal{U}$ an orthogonal basis for a subspace $U \leq V$, then for any $\mathbf{v} \in V$ there exists a unique vector $\mathbf{z} \in U^{\perp}$ such that the following holds.
	\[\mathbf{v} = \mathrm{proj}_{\mathcal{U}}(\mathbf{v}) + \mathbf{z}\]
\end{theorem}





\subsection{Best approximation theorem}

This theorem has major implications in mathematical optimization, statistics and many other branches of mathematics.

\begin{theorem}[Best approximation theorem]
Let $V$ be a finite dimensional linear space and let $U \leq V$, then the following inequality holds for any $\mathbf{u} \in U$.
\[\| \mathbf{v} - \mathrm{proj}_{U} (\mathbf{v})\| \leq \| \mathbf{v} - \mathbf{u} \| \]
\end{theorem}

it is generalized by the \emph{Hilbert projection theorem} in functional analysis.





\section{Gram-Schmidt process}




In practice, one may want to orthogonalize their basis; find an orthonormal basis that spans the same space. There exists an algorithm that constructs such a basis called the \emph{Gram-Schmidt process}. It works by projecting each basis vector onto the space spanned by the current orthogonal set; the result of this identifies the component of the basis element that isn't orthogonal to the set and is subsequenty minused from the vector to form a set orthonormal with the current basis, while still spanning the same space.

\begin{theorem}[Correctness of Gram-Schmidt process]
\[\mathbf{u}_{k} = \mathbf{b}_{k} - \sum^{k-1}_{j=1} \mathrm{proj}_{\mathbf{u}_{j}}(\mathbf{b}_k) \]
\[\mathrm{span}(\mathcal{B}) = \mathrm{span}(\mathcal{U}) \]
\[\mathcal{U} \text{ is an orthogonal set}\]
\end{theorem}

The Gram-Schmidt process not only gives a method to calculate orthogonal basis, but due to its constructive nature also proves that any finite dimensional space has an orthogonal basis.

\begin{corollary}
Finite linear spaces always have an orthogonal basis.
\end{corollary}

Unfortunately the same cannot be said for infinite dimension linear spaces.


\section{Orthonormal matrix}

Given an orthonormal basis, concatenating its vectors into a matrix creates an \emph{orthonomral matrix}.
By noting that orthonormal basis is the only kind of set satisfying $\mathbf{u}_i \cdot \mathbf{u}_j = \delta_{ij}$, we can define orthonormal matrixes in the following way.

\begin{definition}[Orthonormal matrix]
	An \emph{orthonormal matrix} is an invertible square matrix $\mathbf{U}$ that satisfies the following relation.
\[ \mathbf{U}^{-1}= \mathbf{U}^{\intercal}\]
\end{definition}


\[\mathrm{det}(\mathbf{U}) = \pm 1\]


$T(\mathbf{x})=\mathbf{U}\mathbf{x}$ correspond to some rotation and/or reflection