%display latex
latex.matrix_delimiters(left='[', right=']')

$\newcommand{\la}{\langle}$ $\newcommand{\ra}{\rangle}$ $\newcommand{\vu}{\mathbf{u}}$ $\newcommand{\vv}{\mathbf{v}}$ $\newcommand{\vw}{\mathbf{w}}$ $\newcommand{\nc}{\newcommand}$ $\nc{\Cc}{\mathbb{C}}$ $\nc{\Rr}{\mathbb{R}}$ $\nc{\sp}[1]{\la #1 \ra}$ $\nc{\ol}[1]{\overline{#1} }$ $\nc{\norm}[1]{\left\| #1 \right\|}$ $\nc{\abs}[1]{\left| #1 \right|}$ $\nc{\vzz}{\mathbf{z}}$ $\nc{\va}{\mathbf{a}}$ $\nc{\vb}{\mathbf{b}}$ $\nc{\cv}[1]{\begin{bmatrix} #1 \end{bmatrix}}$

Geometry of Vectors IV

Cauchy-Schwarz inequality

The cauchy-schwarz inequality is one of the most important inequality in Mathematics. Check the wikipedia page for its history and more information.

Theorem. (Cauchy-Schwarz inequality) For any $\vu, \vv \in \Cc^n$,

$$\abs{\sp{\vu,\vv}} \le \norm{\vu}\norm{\vv}.$$

Moreover, the equality holds if and only if one of them is a multiple of the other.

Proof. One checks that the equality holds if one of the vectors is a scalar multiple of the other. So we can assume neither $\vu$ nor $\vv$ is a scalar multiple of the other. In particular, neither of them is the zero vector. So, the inequality is equavilent to

$$ \abs{\langle \frac{\vu}{\norm{\vu}}, \frac{\vv}{\norm{\vv}}\rangle} = \frac{\abs{\sp{\vu,\vv}}}{\norm{\vu}\norm{\vv}} \le 1.$$

Therefore, it suffices to prove the theorem for unit vectors $\vu$ and $\vv$. Let $\vw = \vu - \sp{\vv,\vu}\vv$. Then

\begin{align*} \sp{\vv,\vw} &= \sp{\vv, \vu-\sp{\vv,\vu}\vv} = \sp{\vv,\vu} - \sp{\vv,\sp{\vv,\vu}\vv}\\ &= \sp{\vv,\vu} - \sp{\vv,\vu}\sp{\vv,\vv} \quad (\text{since}\ \sp{\vv,\vv}=1)\\ &= \sp{\vv,\vu} -\sp{\vv,\vu} = 0 \end{align*}

And so,

\begin{align*} 1 &= \sp{\vu,\vu} = \sp{\sp{\vu,\vv}\vv+\vw,\sp{\vu,\vv}\vv+\vw} \\ &=|\sp{\vu,\vv}|^2\sp{\vv,\vv} + \sp{\vu,\vv}^* \sp{\vv,\vw} + \sp{\vu,\vv}\sp{\vw,\vv} + \sp{\vw,\vw} \\ &=|\sp{\vu,\vv}|^2 + \norm{\vw}^2 \\ &\ge |\sp{\vu,\vv}|^2. \end{align*}

Since $|\sp{\vu,\vv}|$ is nonnegative, we conlcude that $|\sp{\vu,\vv}| \le 1$. Moreover, the equality holds if and only if $\vw = \mathbf{0}$, i.e. $\vu = \sp{\vu,\vv}\vv$.

Angle

In the real case, $\sp{\vu,\vv} \in \Rr$, so the Cauchy-Schwarz inequality asserts that $\dfrac{\sp{\vu,\vv}}{\norm{\vu}\norm{\vv}}$ is a real number between $1$ and $-1$.

We define the angle between $\vu$ and $\vv$ ($\vu,\vv \in \Rr^n$) to be the unique $\theta \in [-\pi,\pi]$. $$ \cos(\theta) = \frac{\sp{\vu,\vv}}{\norm{\vu}\norm{\vv}}. $$

In other words, $\sp{\vu,\vv} = \norm{\vu}\norm{\vv}\cos(\theta)$.

This definition of angle is compatible with the Law of Cosine.

The angle between the zero vector and any other vector is undefined.

Orthogonality

For nonzero vectors $\vu$ and $\vv$, it follows from the above equation that $\sp{\vu,\vv} =0$ if and only if the angle between $\vu$ and $\vv$ is $\pi/2$.

Definition Two vector $\vu,\vv$ are orthogonal if and only if $\sp{\vu,\vv} = 0$.

So the zero vector is orthogonal to every vector.

Since the norm of a nonzero vector is a positive real number, so $\sp{\vu,\vv}$ and $\cos(\theta)$ has the same sign (or both equal 0) (see definition of angle). Thus, we conclude that

$$ \sp{\vu,\vv} \begin{cases} > 0 & \theta \le \pi/2 \\ =0 & \theta = \pi/2 \\ < 0 & \theta \ge \pi/2. \end{cases} $$

they have the same (resp. opposite) direction if and only if $\theta = 0$ (resp. $\pi$) if and only if $\sp{\vu,\vv} = \norm{\vu}\norm{\vv}$ (resp. $=-\norm{\vu}\norm{\vv}$).

Projections

Given nonzero vectors $\vu$ and $\vv \in \Rr^n$, $\vu$ can be resolved uniquely as $\vu = \vw +\vzz$ where $\vw$ is a vector parallel to $\vv$ and $\vzz$ is a vector that is orthogonal (aka perpendicular) to $\vv$.

The vector $\vw$ is called the projection of $\vu$ to $\vv$ and the vector $\vzz$ is called the orthogonal projection of $\vu$ to $\vv$.

Since $\vw$ is parallel to $\vv$, $\vw = \lambda \vv$ for some $\lambda \in \Rr$. So $\vu = \lambda \vv + \vzz$ and by taking inner product with $\vv$ on both sides, we get

$$ \sp{\vv,\vu} = \lambda \sp{\vv,\vv} + \sp{\vv,\vzz} = \lambda\sp{\vv,\vv} \qquad (\sp{\vv,\vzz} = 0, \text{since}\ \vzz \perp \vv) $$

Therefore, $\lambda = \dfrac{\sp{\vv,\vu}}{\sp{\vv,\vv}}$ and hence

$$ \vw = \lambda \vv = \frac{\sp{\vv,\vu}}{\sp{\vv,\vv}}\vv = \frac{\sp{\vv,\vu}}{\norm{\vv}^2}\vv = \frac{\sp{\vv,\vu}}{\norm{\vv}}\frac{\vv}{\norm{\vv}} = \frac{\sp{\vv,\vu}}{\norm{\vv}}\hat{\vv}. $$

So $\vzz$ must be $\vu - \vw = \vu - \dfrac{\sp{\vv,\vu}}{\sp{\vv,\vv}}\vv$.

The inner product of $\vzz$ and $\vv$ is

$$ \sp{\vv,\vzz} = \sp{\vv,\vu} - \frac{\sp{\vv,\vu}}{\sp{\vv,\vv}}\sp{\vv,\vv} = \sp{\vv,\vu} - \sp{\vv,\vu} = 0. $$

This verifies that $\vzz$ is orthogonal to $\vv$.

Example

Find two vectors $\vv_1$ and $\vv_2$ whose sum is ⟨3,−2,5⟩, where $\vv_1$ is parallel to ⟨−5,−2,5⟩ while $\vv_2$ is perpendicular to ⟨−5,−2,5⟩

Ans: Let $\va = (3,-2,5)$ and $\vb = (-5,-2,5)$ (we identify $\va$ and $\vb$ with $\cv{3 \\ -2 \\5}$ and $\cv{-5 \\-2 \\5}$, respectively.)

Then the question is asking for $\vv_1$ which is the projection of $\va$ to $\vb$ and $\vv_2$ which is the orthogonal projection of $\va$ to $\vb$.

def inner_product(uu,vv):
    return((uu.conjugate_transpose()*vv)[0,0])

aa = column_matrix([3,-2,5]); bb = column_matrix([-5,-2,5])

Then according to the discuss above, $\vv_1 = \dfrac{\sp{\vb,\va}}{\sp{\vb,\vb}}\vb$.

Since $\sp{\vb,\va}$ and $\sp{\vb,\vb}$ are

inner_product(bb,aa), inner_product(bb,bb)

\(\displaystyle \left(14, 54\right)\)

respectively, so $\vv_1 = \frac{14}{54}\cv{-5 \\ -2 \\5} =$

vv1 = inner_product(bb,aa)/inner_product(bb,bb)*bb; vv1

\(\displaystyle \left[\begin{array}{r} -\frac{35}{27} \\ -\frac{14}{27} \\ \frac{35}{27} \end{array}\right]\)

And $\vv_2 = \va - \vv_1$, so $\vv_2$ is

vv2 = aa -vv1

latex(aa)+str('-')+latex(vv1)+str('=')+latex(vv2)

\(\displaystyle \left[\begin{array}{r} 3 \\ -2 \\ 5 \end{array}\right] - \left[\begin{array}{r} -\frac{35}{27} \\ -\frac{14}{27} \\ \frac{35}{27} \end{array}\right] = \left[\begin{array}{r} \frac{116}{27} \\ -\frac{40}{27} \\ \frac{100}{27} \end{array}\right]\)

N.B. This is HW on inner product question 6. Vectors there is respresented by <,...,> (using less than and bigger than sign for pointy brackets)