MAT247H1/Unit 7

Unit 7 · Algebra II

Polar & Singular Value Decomposition

The spectral theorem needs a hypothesis. This unit removes it. Every operator — normal or not, square or not — factors as a rotation composed with a stretch. The price is two different orthonormal bases instead of one.

By the end you can

• Show $T^*T$ is positive semi-definite and extract $|T|=\sqrt{T^*T}$.
• State and prove the polar decomposition $T=U|T|$.
• Compute a singular value decomposition of a small matrix.
• Explain why the singular values, not the eigenvalues, measure size.

Why it matters & where it comes from

Why it exists

Eigenvalues describe an operator badly when it is not normal, and are not defined at all when the matrix is rectangular. Singular values are defined for every matrix, are always real and non-negative, and behave well under perturbation — they are the honest measure of size.

In the real world

Truncating an SVD gives the best low-rank approximation of a matrix, which is the mathematics behind image compression, latent semantic analysis, and recommender systems. The condition number that decides whether a linear system is numerically safe is the ratio of largest to smallest singular value.

Story & open questions

Beltrami and Jordan found the SVD independently in the 1870s for bilinear forms; it was rediscovered repeatedly before numerical analysts made it central in the 1960s. The polar decomposition is the matrix version of writing a complex number as $re^{i\theta}$.

Positive operators and the square root

Definition

A self-adjoint operator $P$ is positive semi-definite if $\langle Pv,v\rangle\ge0$ for all $v$, and positive definite if the inequality is strict for $v\ne0$.

Theorem Characterisation

A self-adjoint $P$ is positive semi-definite iff all its eigenvalues are $\ge0$.

By the spectral theorem take an orthonormal eigenbasis. Then $\langle Pv,v\rangle=\sum_i\lambda_i|\langle v,e_i\rangle|^2$, which is $\ge0$ for all $v$ exactly when every $\lambda_i\ge0$ — test with $v=e_i$ for the converse.

Theorem Unique positive square root

Every positive semi-definite $P$ has a unique positive semi-definite square root $\sqrt P$, and it is a polynomial in $P$.

Existence: from the spectral resolution $P=\sum\lambda_iP_i$, set $\sqrt P=\sum\sqrt{\lambda_i}P_i$, legitimate since $\lambda_i\ge0$. Uniqueness needs that any positive square root must act as $\sqrt{\lambda_i}$ on each eigenspace of $P$.

Theorem $T^*T$ is always positive

For any $T$, the operator $T^*T$ is self-adjoint and positive semi-definite, since $(T^*T)^*=T^*T$ and $\langle T^*Tv,v\rangle=\langle Tv,Tv\rangle=\|Tv\|^2\ge0$.

Define $|T|=\sqrt{T^*T}$. It satisfies $\||T|v\|=\|Tv\|$ for every $v$: both squares equal $\langle T^*Tv,v\rangle$.

Polar decomposition

Theorem Polar decomposition

Every operator $T$ on a finite-dimensional inner product space factors as $T=U|T|$ with $U$ unitary and $|T|=\sqrt{T^*T}$ positive semi-definite. If $T$ is invertible, $U$ is unique.

Proof Define $U$ on the image first

The rule $U(|T|v)=Tv$ is well defined: if $|T|v=|T|w$ then $\||T|(v-w)\|=0$, so $\|T(v-w)\|=0$ and $Tv=Tw$.

On $R(|T|)$ this $U$ preserves norms, since $\|U(|T|v)\|=\|Tv\|=\||T|v\|$. So it is an isometry from $R(|T|)$ onto $R(T)$.

These two subspaces have equal dimension, so their orthogonal complements do too. Extend $U$ by any isometry $R(|T|)^\perp\to R(T)^\perp$; the result is unitary on all of $V$ and satisfies $T=U|T|$. ∎

The complex number analogy is exact. $z=re^{i\theta}$ splits a complex number into a non-negative magnitude and a unit-modulus phase. Here $|T|$ is the magnitude — positive semi-definite — and $U$ is the phase — unitary. Both decompositions are unique exactly when the object is invertible.

Example A small polar decomposition

$A=\left(\begin{smallmatrix}0&2\\0&0\end{smallmatrix}\right)$.

$A^*A=\left(\begin{smallmatrix}0&0\\0&4\end{smallmatrix} \right)$, so $|A|=\left(\begin{smallmatrix}0&0\\0&2\end{smallmatrix} \right)$.

$U$ must send $(0,2)\mapsto(2,0)$, that is the second standard vector to the first. Taking $U=\left(\begin{smallmatrix}0&1\\1&0\end{smallmatrix}\right)$ gives $U|A|=\left(\begin{smallmatrix}0&2\\0&0\end{smallmatrix}\right)=A$. ✓

Note $A$ is not invertible and indeed $U$ is not unique: $\left(\begin{smallmatrix}0&1\\-1&0\end{smallmatrix}\right)$ works too.

Singular value decomposition

Definition

The singular values of $T$ are the eigenvalues of $|T|$, that is the non-negative square roots of the eigenvalues of $T^*T$, listed with multiplicity in decreasing order $\sigma_1\ge\sigma_2\ge\cdots\ge\sigma_n\ge0$.

Theorem Singular value decomposition

For any linear $T:V\to W$ between finite-dimensional inner product spaces there are orthonormal bases $\{v_1,\dots,v_n\}$ of $V$ and $\{u_1,\dots,u_m\}$ of $W$ and scalars $\sigma_1\ge\cdots\ge\sigma_r>0$ with

$Tv_i=\sigma_iu_i$ for $i\le r$, and $Tv_i=0$ for $i>r$, where $r=\operatorname{rank}(T)$.

In matrix terms, $A=U\Sigma V^*$ with $U,V$ unitary and $\Sigma$ rectangular diagonal carrying the singular values.

Proof Spectral theorem on $T^*T$

$T^*T$ is positive semi-definite, so the spectral theorem gives an orthonormal eigenbasis $\{v_i\}$ of $V$ with eigenvalues $\sigma_i^2\ge0$, ordered decreasingly.

For $i\le r$ set $u_i=Tv_i/\sigma_i$. These are orthonormal: $\langle u_i,u_j\rangle =\dfrac{\langle Tv_i,Tv_j\rangle}{\sigma_i\sigma_j} =\dfrac{\langle T^*Tv_i,v_j\rangle}{\sigma_i\sigma_j} =\dfrac{\sigma_i^2\langle v_i,v_j\rangle}{\sigma_i\sigma_j}$, which is $1$ when $i=j$ and $0$ otherwise.

For $i>r$, $\|Tv_i\|^2=\sigma_i^2=0$, so $Tv_i=0$. Extend $\{u_i\}$ to an orthonormal basis of $W$. ∎

Example Compute an SVD

$A=\left(\begin{smallmatrix}1&1\\0&1\end{smallmatrix}\right)$, which is not normal.

$A^*A=\left(\begin{smallmatrix}1&1\\1&2\end{smallmatrix} \right)$ with characteristic polynomial $t^2-3t+1$, so the eigenvalues are $(3\pm\sqrt5)/2$ and the singular values are their square roots: $\sigma_1\approx1.618$, $\sigma_2\approx0.618$.

Their product is $\sigma_1\sigma_2=\sqrt{\det(A^*A)} =|\det A|=1$, a useful check.

The eigenvalues of $A$ itself are both $1$ — which tells you nothing about how much $A$ stretches. The singular values do: the worst-case stretch is about $1.618$.

Theorem What singular values compute

$\sigma_1=\max_{\|v\|=1}\|Tv\|$, the operator norm; $\operatorname{rank}(T)$ is the number of nonzero singular values; and $\sigma_1/\sigma_n$ is the condition number when $T$ is invertible.

Common pitfalls

• Confusing singular values with eigenvalues. They coincide only when $T$ is positive semi-definite; in general $\sigma_i=\sqrt{\lambda_i(T^*T)}$.
• Forgetting singular values are non-negative and real, even when the eigenvalues are complex or the matrix is rectangular.
• Assuming $U=V$ in $A=U\Sigma V^*$. Two different bases are needed unless $A$ is positive semi-definite.
• Expecting uniqueness. $\Sigma$ is unique, but $U$ and $V$ are not when singular values repeat or vanish.

Practice

Original problems written for this guide in the style of Friedberg, Insel & Spence exercises. Write a full solution before opening the answer.

Practice 1 · Singular values

Find the singular values of $\left(\begin{smallmatrix}3&0\\0&-2\end{smallmatrix}\right)$.

Answer

$A^*A =\left(\begin{smallmatrix}9&0\\0&4\end{smallmatrix}\right)$, so the singular values are $3$ and $2$ — the absolute values of the eigenvalues, as always for a diagonal matrix.

Practice 2 · Rectangular

Find the singular values of the $2\times1$ matrix with entries $3$ and $4$.

Answer

$A^*A=(25)$, a $1\times1$ matrix, so the only singular value is $5$. Eigenvalues are not even defined here, which is the point.

Practice 3 · Positive check

Is $\left(\begin{smallmatrix}2&1\\1&2\end{smallmatrix}\right)$ positive definite?

Answer

Yes. It is symmetric with eigenvalues $3$ and $1$, both positive.

Practice 4 · Square root

Find $\sqrt A$ for $A=\left(\begin{smallmatrix}4&0\\0&9\end{smallmatrix}\right)$.

Answer

$\left(\begin{smallmatrix}2&0\\0&3\end{smallmatrix}\right)$. It is the unique positive square root; sign changes on the diagonal give other square roots that are not positive.

Practice 5 · Polar

Polar decompose $A=\left(\begin{smallmatrix}0&3\\-3&0\end{smallmatrix}\right)$.

Solution

$A^*A=9I$, so $|A|=3I$ and $U=\tfrac13A =\left(\begin{smallmatrix}0&1\\-1&0\end{smallmatrix}\right)$, which is orthogonal. So $A=U\cdot3I$: a rotation composed with a uniform scaling by $3$.

Practice 6 · Determinant identity

Show $|\det A|=\prod_i\sigma_i$.

Solution

From $A=U\Sigma V^*$, $|\det A|=|\det U||\det\Sigma||\det V^*|=\det\Sigma=\prod\sigma_i$, since unitary matrices have determinant of modulus $1$.

Practice 7 · Operator norm

Prove $\max_{\|v\|=1}\|Tv\|=\sigma_1$.

Solution

Expand $v$ in the orthonormal basis $\{v_i\}$ from the SVD: $\|Tv\|^2 =\sum_i\sigma_i^2|c_i|^2\le\sigma_1^2\sum_i|c_i|^2=\sigma_1^2$, with equality at $v=v_1$.

Practice 8 · When they coincide

For which operators do the singular values equal the moduli of the eigenvalues?

Solution

Exactly the normal ones. If $T=UDU^*$ then $T^*T=U|D|^2U^*$, so $\sigma_i=|\lambda_i|$. Conversely equality forces normality — which is why non-normal matrices like $\left(\begin{smallmatrix}1&1\\0&1\end{smallmatrix}\right)$ have eigenvalues $1,1$ but singular values far from $1$.

Expanded topic checklist

Course coverage. This unit is part of the complete MAT247H1 topic map. It should cover the following items before you move on:

• Positive operators. Characterise them by their spectrum and construct the unique positive square root.
• Polar decomposition. Prove $T=U|T|$ and identify when $U$ is unique.
• SVD. Derive it from the spectral theorem applied to $T^*T$ and compute small examples.
• Interpretation. Relate $\sigma_1$ to the operator norm and the count of nonzero singular values to rank.

Textbook/source alignment

Use the current UofT MAT247H1 calendar topics as the coverage spine, and pair lecture notes with a Friedberg-Insel-Spence-style linear algebra text for extra reading. The exact assigned textbook should still be checked against the instructor syllabus.

Core theorem map

$T^*T$ positive gives $|T|$; $|T|$ gives the polar decomposition; the spectral theorem on $T^*T$ gives the SVD. Everything here is the spectral theorem applied to an operator manufactured to satisfy its hypothesis.

Study routine. Check every SVD with $\prod\sigma_i=|\det A|$ and with the count of nonzero singular values against the rank. Both catch arithmetic slips immediately.

Deep practice extensions

Use these after the main practice set. They are designed to force definition-level reasoning, theorem selection, and mixed application rather than routine calculation only.

Extension 1 · Definition

How are singular values defined?

Guide

As the non-negative square roots of the eigenvalues of $T^*T$, equivalently the eigenvalues of $|T|=\sqrt{T^*T}$.

Extension 2 · Why they always exist

Why does every matrix have singular values, even non-square ones?

Guide

$T^*T$ is always a square, self-adjoint, positive semi-definite operator on the domain, so the spectral theorem always applies to it regardless of the shape of $T$.

Extension 3 · Polar uniqueness

When is the unitary factor in $T=U|T|$ unique?

Guide

Exactly when $T$ is invertible. Otherwise $U$ is only determined on $R(|T|)$ and can be anything isometric on the complement.

Extension 4 · Low rank approximation

How does the SVD give the best rank-$k$ approximation?

Guide

Keep the $k$ largest singular values and zero the rest. The error is $\sigma_{k+1}$ in operator norm, and no rank-$k$ matrix does better — the Eckart–Young theorem.

← Unit 06The spectral theorem Unit 08 →Dual & quotient spaces