Chapter III
Orthogonality
65
III Orthogonality
III.1 Orthogonality and Projections
III.1.1 Orthogonal vectors
Recall that the dot product, or inner product of two vectors
x
=
2
6
6
6
4
x
1
x
2
.
.
.
x
n
3
7
7
7
5
y
=
2
6
6
6
4
y
1
y
2
.
.
.
y
n
3
7
7
7
5
is denoted by
x
·
y
or
h
x
,
y
i
and defined by
x
T
y
=
⇥
x
1
x
2
· · ·
x
n
⇤
2
6
6
6
4
y
1
y
2
.
.
.
y
n
3
7
7
7
5
=
n
X
i
=1
x
i
y
i
Some important properties of the inner product are symmetry
x
·
y
=
y
·
x
and linearity
(
c
1
x
1
+
c
2
x
2
)
·
y
=
c
1
x
1
·
y
+
c
2
x
2
·
y
.
The norm, or length, of a vector is given by
k
x
k
=
p
x
·
x
=
v
u
u
t
n
X
i
=1
x
2
i
An important property of the norm is that
k
x
k
= 0 implies that
x
=
0
.
The geometrical meaning of the inner product is given by
x
·
y
=
k
x
kk
y
k
cos(
✓
)
where
✓
is the angle between the vectors. The angle
✓
can take values from 0 to
⇡
.
The Cauchy–Schwarz inequality states

x
·
y

k
x
kk
y
k
.
It follows from the previous formula because

cos(
✓
)

1. The only time that equality occurs
in the Cauchy–Schwarz inequality, that is
x
·
y
=
k
x
kk
y
k
, is when cos(
✓
) =
±
1 and
✓
is either
0 or
⇡
. This means that the vectors are pointed in the same or in the opposite directions.
66
III.1 Orthogonality and Projections
The vectors
x
and
y
are orthogonal if
x
·
y
= 0. Geometrically this means either that one
of the vectors is zero or that they are at right angles. This follows from the formula above,
since cos(
✓
) = 0 implies
✓
=
⇡
/
2.
Another way to see that
x
·
y
= 0 means that vectors are orthogonal is from Pythagoras’
formula. If
x
and
y
are at right angles then
k
x
k
2
+
k
y
k
2
=
k
x
+
y
k
2
.
But
k
x
+
y
k
2
= (
x
+
y
)
·
(
x
+
y
) =
k
x
k
2
+
k
y
k
2
+ 2
x
·
y
so Pythagoras’ formula holds
exactly when
x
·
y
= 0.
To compute the inner product of (column) vectors
X
and
Y
in MATLAB/Octave we use
the formula
x
·
y
=
x
T
y
. Thus the inner product can be computed using
X’*Y
. (If
X
and
Y
are row vectors, the formula is
X*Y’
.)
The norm of a vector
X
is computed by
norm(X)
. In MATLAB/Octave inverse trig functions
are computed with
asin(), acos()
etc. So the angle between column vectors
X
and
Y
could
be computed as
> acos(X’*Y/(norm(X)*norm(Y)))
III.1.2 Orthogonal subspaces
Two subspaces
V
and
W
are said to be orthogonal if every vector in
V
is orthogonal to every
vector in
V
. In this case we write
V
?
W
.
In this figure
V
?
W
and also
S
?
T
.
67
III Orthogonality
A related concept is the orthogonal complement. The orthogonal complement of
V
, denoted
V
?
is the subspace containing all vectors orthogonal to
V
. In the figure
W
=
V
?
but
T
6
=
S
?
since
T
contains only some of the vectors orthogonal to
S
.
If we take the orthogonal complement of
V
?
we get back the original space
V
: This is
certainly plausible from the pictures. It is also obvious that
V
✓
(
V
?
)
?
, since any vector in
V
is perpendicular to vectors in
V
?